autorelay #5785

vyzo · 2018-11-19T14:06:33Z

Updates the node construction to automatically construct a routed host and enable autorelay.

Note that instances of the autonat service are required to be (passively) discovered in the network for the autorealy functionality to test for NAT presence, before advertising relay addresses.
This needs to run in the boostrappers and a plus if it is available in other public hosts, so an option to enable it is required.

Depends on

TBD:

resolve test failures
enable autorelay gating with config option
instantiate autonat service with config option.
use explicit autorelay libp2p option.

whyrusleeping · 2018-11-19T23:25:33Z

Where is autorelay actually being enabled? I don't see any new options

vyzo · 2018-11-20T08:30:05Z

it's enabled by libp2p when relay is enabled and you pass the new Routing option, which also constructs a routed host.
Should I add an option so that users have to explicitly enable it (or can disable it)?

vyzo · 2018-11-20T09:16:26Z

Presumably we do want an option in ipfs to enable or disable it, instead of unconditionally enabling.
What should the default state be though?
Perhaps we should start with it disabled as this is a new and experimental feature, and will also require a release cycle to be effective (because new addresses are propagated with identify push, which older nodes do not support).

vyzo · 2018-11-20T10:55:41Z

options in ipfs/go-ipfs-config#21

core/core.go

-	// setup routing service
-	r, err := routingOption(ctx, host, n.Repo.Datastore(), n.RecordValidator)
-	if err != nil {
-		return err


Stebalien · 2018-11-21T00:42:50Z

~~Whatever happened to fixing libp2p immediately after merging the autorelay change? I thought the entire point of that was to get some code to refactor.~~

edit: bah humbug. There's no point in introducing an arbitrary dependency to block a really, frickin, awesome feature just because I'm grumpy about something the user will never see.

vyzo · 2018-11-21T07:58:18Z

The libp2p refactoring is still on, but I don't think it will change the user interface -- so we might as well get the feature out first!

* AutoRelay options for #5785. * Badger truncate-by-default option for #5275, #5625. License: MIT Signed-off-by: Steven Allen <steven@stebalien.com>

vyzo · 2018-11-28T12:57:33Z

added autorelay gating with config option.

vyzo · 2018-11-28T13:10:38Z

instantiated the autonat service as well.

vyzo · 2018-11-28T13:11:07Z

@Stebalien this is complete feature-wise, take another look?

vyzo · 2018-11-30T08:57:44Z

rebased on master.

Stebalien · 2018-12-01T01:08:57Z

What is the difference between a RelayHost and an AutoRelayHost?
Why does passing the routing option to libp2p enable auto relay? They seem totally unrelated (well, the former seems like a dependency of the latter).

This patch should be able to avoid the routing option dance by always using the libp2p option. We'd need to modify the mock host constructor (a cleaner solution would be to add a Host (constructor) option to libp2p itself but that may be overkill given that we're planning on changing this all up).

vyzo · 2018-12-01T10:14:25Z

What is the difference between a RelayHost and an AutoRelayHost?

RelayHost is an actual relay, while AutoRelayHost is a host that connects to relays and advertises relay addrs (when NATed).

Why does passing the routing option to libp2p enable auto relay? They seem totally unrelated (well, the former seems like a dependency of the latter).

Well, the routing is necessary for auto relay, but there is no other explicit option. Perhaps we should add one.

We'd need to modify the mock host constructor

I guess we can do that. We'll need the option for explicit autorelay though, otherwise it will be unconditionally enabled.

vyzo · 2018-12-04T10:13:57Z

Added explicit option in libp2p/go-libp2p#500; we'll have to bubble up, and also merge libp2p/go-libp2p#495

Stebalien · 2018-12-13T19:08:36Z

Actually, we can totally test this (I think). We can't simulate NATs but we can, carefully, configure our transports:

Peer A: Listens on QUIC only (but can still dial with TCP).
Peer B: Has QUIC disabled, has AutoNATService enabled.
Peer C: Has QUIC enabled, has RelayHop enabled.
Peer D: Has QUIC disabled.

Start all peers.
Connect peer A to peer B and peer C.
Try to connect peer D to peer A, keep retrying every 3 seconds until it works (may take a while).
Check ipfs id on peer A to see if it's announcing a relay address.

Think that will work?

vyzo · 2018-12-13T19:12:02Z

there are still some obsene delays involved; there is about a 90s delay until the autonat detection boots and fires, which we don't have a way to configure in sharness.
we could add some parameters just for testing though, but it's ugly.

Stebalien · 2018-12-13T19:28:03Z

Honestly, I'm fine just having a really slow test for now. We can mark it as EXPENSIVE. We just really should have something.

(we really need to get rid of those delays but we'll probably need an event system to do that)

vyzo · 2018-12-13T19:30:41Z

ok, let me cook up a ridiculously slow sharness test then -- how do i mark as expensive?

Stebalien · 2018-12-13T19:38:41Z

test_expect_success EXPENSIVE "some description" ' ... test ... '

(take a look at the add-and-cat tests).

vyzo · 2018-12-17T12:32:45Z

So I've cooked up a test as you suggested, but unfortunately it can't work: autorelay only advertises public addresses and there are no public addresses in my NATed laptop :(

magik6k · 2018-12-17T18:24:40Z

This is ~~probably~~ definetly a really bad idea, but - are there any solutions that would allow us to setup fake userspace network and hook ipfs it into using something like LD_PRELOAD?

Stebalien · 2018-12-17T19:40:24Z

are there any solutions that would allow us to setup fake userspace network and hook ipfs it into using something like LD_PRELOAD?

Maybe but probably not given that go tends to make raw syscalls. We could also use a syscall filter.

However... we could test this from go with a plugin. That is, we could drop in a special plugin that modifies go-multiaddr-net's Unroutable/Private network variables on load. Thoughts? Is that even possible?

Stebalien · 2018-12-17T21:48:46Z

Status: Blocked on at least demonstrating that this works (even if we can't write a test that runs on CI).

vyzo · 2018-12-18T11:23:37Z

I have confirmed that autorelay works with manual testing.
I started an AutoNAT/RelayHop node in a vps, and tested using a double-NATed laptop node.
I connected the laptop node to the vps node, and after a couple of minutes it started advertising relay addrs.

Transcript:

$ ipfs id

{
        "ID": "QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
        "PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDSG2Zlqp
oEyWa498W3NzaiOz642nwkP9jgnggpHWrJ3lL6cv2OzM+n9EREshG4TIHuxjY8unvBby8weDzWPYXO9o
YSR052VKSNzZgsMZ1cjuL6YrNgf9t2l9JeHvHZlEdodYKFwwcWgrAVRjAsVMOpo3zylrz19lYk7b5I2O
QgaX4dSNpjxUuZH1M97IokegHnIKxCqdFdQ03WotlQ08ppzQ/kG0UYiZvLKTxbrFw8wR915oBH9ZQf0t
RVaB+5kbJiVG/UmHyAyCmqpxrhW73UrldU9kIVmFxHFoeqP50stiWVtu/ROmiy040EL/krjv88ftQVvMLtBTUUsnyt2wX5AgMBAAE=",
        "Addresses": [
                "/ip4/127.0.0.1/tcp/4001/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
                "/ip4/10.0.1.30/tcp/4001/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
                "/ip6/::1/tcp/4001/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
                "/ip4/79.167.218.87/tcp/2647/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE"
        ],
        "AgentVersion": "go-ipfs/0.4.19-dev/dad0fa54a",
        "ProtocolVersion": "ipfs/0.1.0"
}

# connect to the relay/autonat node:
$ ipfs swarm connect /ip4/172.104.9.124/tcp/4001/ipfs/QmchgNzyUFyf2wpfDMmpGxMKHA3PkC1f3H2wUgbs21vXoh

# after a couple of minutes relay addrs are advertised as expected:
$ ipfs id

{
        "ID": "QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
        "PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDSG2ZlqpoEyWa498W3NzaiOz642nwkP9jgnggpHWrJ3lL6cv2OzM+n9EREshG4TIHuxjY8unvBby8weDzWPYXO9oYSR052VKSNzZgsMZ1cjuL6YrNgf9t2l9JeHvHZlEdodYKFwwcWgrAVRjAsVMOpo3zylrz19lYk7b5I2OQgaX4dSNpjxUuZH1M97IokegHnIKxCqdFdQ03WotlQ08ppzQ/kG0UYiZvLKTxbrFw8wR915oBH9ZQf0tRVaB+5kbJiVG/UmHyAyCmqpxrhW73UrldU9kIVmFxHFoeqP50stiWVtu/ROmiy040EL/krjv88ftQVvMLtBTUUsnyt2wX5AgMBAAE=",
        "Addresses": [
                "/ip4/127.0.0.1/tcp/4001/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
                "/ip4/10.0.1.30/tcp/4001/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
                "/ip6/::1/tcp/4001/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
                "/ip4/79.167.218.87/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
                "/ip6/2600:3c03::f03c:91ff:fe87:a3b4/tcp/4001/ipfs/QmchgNzyUFyf2wpfDMmpGxMKHA3PkC1f3H2wUgbs21vXoh/p2p-circuit/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE",
                "/ip4/172.104.9.124/tcp/4001/ipfs/QmchgNzyUFyf2wpfDMmpGxMKHA3PkC1f3H2wUgbs21vXoh/p2p-circuit/ipfs/QmRozfTYbjdktgjkC9XNjQNKHz4kui5sfNRzkaGqHmRouE"
        ],
        "AgentVersion": "go-ipfs/0.4.19-dev/dad0fa54a",
        "ProtocolVersion": "ipfs/0.1.0"
}

vyzo · 2018-12-18T11:30:12Z

Steps to reproduce:

Create a public node with AutoNAT/RelayHop Enabled:

ipfs config --json Swarm.EnableRelayHop true
ipfs config --json Swarm.EnableAutoNATService true
ipfs config --json Swarm.EnableAutoRelay true

Start a node behind a bad NAT with autorelay enabled (ipfs config --json Swarm.EnableAutoRelay true), and connect to your relay using ipfs warm connect ...
Wait a couple of minutes, and check the output of ipfs id; it should change the public addresses to the naked ip plus the public relay addresses.

vyzo · 2018-12-18T19:28:02Z

we also need to update go-libp2p-autonat-svc to 1.0.4 for go-libp2p 6.0.29 (once this has bubbled up).

…utorelay License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

PeerHost is now initialized before calling it. License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

vyzo requested a review from Kubuxu as a code owner November 19, 2018 14:06

ghost assigned vyzo Nov 19, 2018

ghost added the status/in-progress In progress label Nov 19, 2018

vyzo requested a review from Stebalien November 19, 2018 14:06

vyzo mentioned this pull request Nov 19, 2018

Relay Infrastructure Integration #4990

Closed

vyzo force-pushed the feat/autorelay branch 2 times, most recently from 9d4cfee to 3700c51 Compare November 19, 2018 14:49

vyzo mentioned this pull request Nov 19, 2018

How do I test autorelay? libp2p/go-libp2p#487

Closed

vyzo mentioned this pull request Nov 20, 2018

autorelay options ipfs/go-ipfs-config#21

Merged

Stebalien reviewed Nov 21, 2018

View reviewed changes

core/core.go

// setup routing service

r, err := routingOption(ctx, host, n.Repo.Datastore(), n.RecordValidator)

if err != nil {

return err

This comment was marked as resolved.

Sign in to view

Stebalien added a commit that referenced this pull request Nov 27, 2018

gx: update go-ipfs-config

cef6459

* AutoRelay options for #5785. * Badger truncate-by-default option for #5275, #5625. License: MIT Signed-off-by: Steven Allen <steven@stebalien.com>

Stebalien mentioned this pull request Nov 27, 2018

gx: update go-ipfs-config #5798

Merged

vyzo force-pushed the feat/autorelay branch from 3700c51 to 13778cd Compare November 28, 2018 12:55

vyzo force-pushed the feat/autorelay branch 2 times, most recently from d7a7386 to ab07fe3 Compare November 30, 2018 08:56

Stebalien requested a review from djdv December 1, 2018 00:30

vyzo mentioned this pull request Dec 4, 2018

explicit option to enable autorelay libp2p/go-libp2p#500

Merged

Stebalien added status/blocked Unable to be worked further until needs are met and removed status/in-progress In progress labels Dec 17, 2018

Stebalien removed the status/blocked Unable to be worked further until needs are met label Dec 18, 2018

Stebalien force-pushed the feat/autorelay branch from dad0fa5 to e8a94b6 Compare December 18, 2018 19:27

ghost added the status/in-progress In progress label Dec 18, 2018

vyzo added 8 commits December 18, 2018 11:30

use libp2p.Routing in constructor for routing host construction and a…

b14907b

…utorelay License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

gate autorelay with config option

b514820

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

instantiate autonat service

19ab124

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

add support for QUIC in autonat service

2969c88

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

gx update go-libp2p-autonat-svc

38da46e

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

explicitly enable autorelay

30e4cc9

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

remove host parameter from StartOnlineServicesWithHost

67a6b14

PeerHost is now initialized before calling it. License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

document autorelay

526d619

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

Stebalien force-pushed the feat/autorelay branch from e8a94b6 to 526d619 Compare December 18, 2018 20:11

Stebalien merged commit 526d619 into master Dec 18, 2018

ghost removed the status/in-progress In progress label Dec 18, 2018

Stebalien deleted the feat/autorelay branch December 18, 2018 20:12

Stebalien mentioned this pull request Feb 1, 2019

More Thorough Autorelay Tests libp2p/go-libp2p#528

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

autorelay #5785

autorelay #5785

vyzo commented Nov 19, 2018 •

edited

Loading

whyrusleeping commented Nov 19, 2018

vyzo commented Nov 20, 2018 •

edited

Loading

vyzo commented Nov 20, 2018

vyzo commented Nov 20, 2018

This comment was marked as resolved.

Stebalien commented Nov 21, 2018 •

edited

Loading

vyzo commented Nov 21, 2018

vyzo commented Nov 28, 2018

vyzo commented Nov 28, 2018

vyzo commented Nov 28, 2018

vyzo commented Nov 30, 2018

Stebalien commented Dec 1, 2018

vyzo commented Dec 1, 2018

vyzo commented Dec 4, 2018

Stebalien commented Dec 13, 2018

vyzo commented Dec 13, 2018

Stebalien commented Dec 13, 2018

vyzo commented Dec 13, 2018

Stebalien commented Dec 13, 2018

vyzo commented Dec 17, 2018

magik6k commented Dec 17, 2018

Stebalien commented Dec 17, 2018

Stebalien commented Dec 17, 2018

vyzo commented Dec 18, 2018

vyzo commented Dec 18, 2018 •

edited

Loading

vyzo commented Dec 18, 2018

autorelay #5785

autorelay #5785

Conversation

vyzo commented Nov 19, 2018 • edited Loading

whyrusleeping commented Nov 19, 2018

vyzo commented Nov 20, 2018 • edited Loading

vyzo commented Nov 20, 2018

vyzo commented Nov 20, 2018

This comment was marked as resolved.

Stebalien commented Nov 21, 2018 • edited Loading

vyzo commented Nov 21, 2018

vyzo commented Nov 28, 2018

vyzo commented Nov 28, 2018

vyzo commented Nov 28, 2018

vyzo commented Nov 30, 2018

Stebalien commented Dec 1, 2018

vyzo commented Dec 1, 2018

vyzo commented Dec 4, 2018

Stebalien commented Dec 13, 2018

vyzo commented Dec 13, 2018

Stebalien commented Dec 13, 2018

vyzo commented Dec 13, 2018

Stebalien commented Dec 13, 2018

vyzo commented Dec 17, 2018

magik6k commented Dec 17, 2018

Stebalien commented Dec 17, 2018

Stebalien commented Dec 17, 2018

vyzo commented Dec 18, 2018

vyzo commented Dec 18, 2018 • edited Loading

vyzo commented Dec 18, 2018

vyzo commented Nov 19, 2018 •

edited

Loading

vyzo commented Nov 20, 2018 •

edited

Loading

Stebalien commented Nov 21, 2018 •

edited

Loading

vyzo commented Dec 18, 2018 •

edited

Loading