Header Synchronization Improvement

[humaite]

Added a way to know if a peer is up or down
in ar_peers by calling ar_peers:connected_peer/1
and ar_peers:disconnected_peer/1 functions. To
know if a peer is activate, one can call
ar_peers:is_connected_peer/1 function. The full
list of active peers (present in the ranking) can
be returned by calling ar_peers:get_peers/1
function.

% returns the list of all connected peers.
% this list is highly unstable, it will depend
% on the gun HTTP connection
ar_peers:get_peers(connected)

All connected peers are now being tagged with a
timestamp with one connection is successful. This
timestamp can be retrieved using ar_peers:get_connection_timestamp/1
function. To filter peers based on their timestamps,
ar_peers:get_peers/1 can be used like this:

% returns the list of peers connected during the
% last 2 days
ar_peers:get_peers({current, 2}).

Arweave can be started with a new option, controlling
the nodes returned by /peers. Only current value
can be filtered at this time of writing, and only
a number of positive days can be configured.

peers_list_filter current
peers_list_filter current:365

The default peers filtering is set to 30 days: {current, 30}.

"tags" are used as simple key in ar_peers
ets table, prefixed with the ar_tags atom. They
are saved in the peers file maintained with
ar_storage function.

Modified ar_http_iface_client:get_block_shadow/2
function to add more parameters, in particular
to modify default timeout and the value used to
pick a peer from peer list.

Modified ar_header_sync module to only select
active peer when synchronizing headers.

Fixed a typo in bin/console.

[ldmberman]

I suggest we do not immediately deactivate peers on communication failure but allow them some slack within a certain time window. The motivation is to stay connected to very useful peers. I would also consider using the existing issue_warning mechanism.

[ldmberman]

We should be getting here on a regular connection shutdown (occurs after a minute) too.

[humaite]

if we get here, that means the connection is down and the peer is seen inactive by the node.

[humaite]

I suggest we do not immediately deactivate peers on communication failure but allow them some slack within a certain time window. The motivation is to stay connected to very useful peers.

If the connection is cut, the server is not connected anymore.

I would also consider using the existing issue_warning mechanism.

If I understand correctly the code, it's when a peer is rejected. I will add it as well.

[humaite]

After a conversation with @JamesPiechota yesterday, I think this PR can be confusing, I will try to do a quick summary of the problem(s) we encounter with the current ar_header_sync module and by extension, ar_peers/ar_http_iface_client ones.

1. The current implementation (2.7.4) when deployed, download one block every minute (sometimes 20 minutes). A full blockchain synchronization can take more than a year.

2. The distribution of the blocks across the network is really sparse. During my first test with a brute-force scanner across peers available from arweave.net, I've found only 0.7% of the peers got the first block (0).

3. The algorithm used to fetch a block (defined in ar_http_iface_client:get_block_shadow/2 and ar_http_iface_client:get_block_shadow/3 functions) are only taking a small subset of the lifetime peers. This number is set to 5 by default, this means only 1 peer from the 5 firsts peers of the lifetime ranking list will be taken.

4. The peers available in the lifetime list have a ranking, but this ranking include, dead peers (not seen on the network for many months or years), unavailable peers (temporarily down nodes) and "inactive" peers (without active http connection from the node point of view). This can lead to many other issue, for example timeout issues.

5. The ranking used by ar_peers is not accurate to defined which peers is a "good peer" from blocks/txs point of view. This means a "good peer" can be above all peers but could have practically no old blocks or txs. Even if the success rate decreases over time if the peer fails to fetch an old block or tx, its position in the peer list can stay for a while because of the other successful requests.

6. Increasing the number of jobs don't increase the speed of downloading with the current implementation. In fact, the 5 first peers in the list will probably slowly be moved down, but the number of failure will still be high. This behavior is also due to one thing: we are reusing always the same list of peers, with the same dead/unavailable/bad peers. One worker will always have the same chance to fetch a block.

Here the strategies I have deployed/tested on my test server:

1. The first one was simply to try downloading a block from an active server (with an HTTP connection UP from gun point of view). This implementation is also the one available in this PR, the easiest one at least, because the previous one used another method to deal with the connection status. Without modifying the peer list (including deadlist) and just by filtering nodes by their connection status, we have a speed increase

2. Another strategy was to create a dedicated score board for block shadow. The idea was to follow the activity of each peers (connected or not) with the success/failure of a wanted block. By using this method, I was able to download ~10 blocks per second in the best case. Unfortunately, this method also introduce another way to list peers when it comes to download blocks/txs.

3. Modifying the random function (or at least its parameters), to increase the number of host selected by the worker. This method was quite successful, and improved a bit the performance. It was added in parallel of the 2 previous strategies.

What about the success/failures ratio? In fact, my test server is currently running a patched version of arweave used to collect this information¹ with the first strategy and 400 jobs. With only the active peers, after more than 12 hours of data collection, we have this ratio: 159741 failures and only 28920. A job has ~18% chance to find a block in the network right now, and this value decrease over time.

What about the discussion we had with @JamesPiechota? The current implementation does not bet on performance/speed but more on distribution, but the fact we are keeping dead peers can lead to problem. The next update to this PR should include:

1. a way to list http active nodes (up/down connection), like we have right now (e.g. ar_peers:get_peers(active)
2. a way to tag/mark all peers with a date of their last connection
3. a way to filter peers by their last connection (e.g. ar_peers:get_peers({days, 5}).)
4. don't purge old peers for the moment, just tag them with this information

I will probably rename things to avoid confusion.

peer state	collected information
{connection, active, boolean()}	gun connection state
{connection, last, integer()}	system_time() from the last connection

In the end, this PR/issues led me to a way to improve our knowledge of the network. At this time, the data collected to create the ranking per peer does not offer enough information regarding its whole state. My next idea is to implement ar_tags, used to add any kind of data to one peer and then offer a way to have fine grained filtering per peers. We could then collect, for example, the number of success/failure per end-point per peer or add dynamic counter based on some information. It can also extend our metrics by adding different information than pure numbers. Here an example how to use this future module (will be available on another PR):

% main way to tag with ar_tags module
ar_tags:set_tag(ar_peers, Peer, {connection, active}, true}).
ar_tags:set_tag(ar_peers, Peer, {connection, active}, false}).

% a frontend to ar_tags
ar_peers:set_tag(Peer, {connection, active}, true}).
ar_peers:set_tag(Peer, {connection, active}, false}).

Footnotes

1. the idea was to increase 2 counters, one on success, another on failure, for each peers when downloading a block using get_block_shadow function. ↩

[ldmberman]

The peers we have established connections with are peers we attempted to make requests to in the first place; it's not like we are establishing connections with every peer we know (so that filtering by active connections would be useful) but we are simply reusing connections we have already established for efficiency. Node being temporarily down or time-outing excessively, on the other hand, might be a useful metric to track.

Consequently we should not purge peers we are not making requests to, we should purge peers we cannot connect to for a certain amount of time.

[humaite]

Consequently we should not purge peers we are not making requests to, we should purge peers we cannot connect to for a certain amount of time.

I've added a timestamp parameter, to filter only active peers during a period of time, say 3 days, 5 months and so on. Those tags are saved and restored from the peers file. All useful information in the commit message (and now in the PR description as well).

[humaite]

Note: this following text is just to give you some example of what's happening on other projects based on academic publication.

In 2019, a paper called Exploring the Monero Peer-to-Peer Network was pusblished. It explores the Monero network to extract information regarding peers and their activities. The mechanism used by Monero is described in section 2.2:

each node maintains a peer list consisting of two parts, i.e., a
white list, and a gray list. In the peer list of a peer A, the information of
each recorded peer not only contains its identity, its IP address, and the TCP
port number it uses, but also a special last seen data field, which is the time at
which the peer has interacted with peer A for the last time. All the peers in the
lists are ordered chronologically according to their last seen data, i.e., the most
recently seen peers are at the top of the list.
Each time a node receives information about a set of peers, this information is
inserted into its gray list. Nodes update their white list and gray list through
a mechanism called “graylist housekeeping”, which periodically pings randomly
selected peers in the gray list. If a peer from the gray list is responsive, then
its information will be promoted to the white list with an updated last seen
field, otherwise it will be removed from the gray list [...] Nodes also periodically
handshake their current connections, and update the last seen field of the
associated responsive peers. If a peer does not respond to the handshake request,
then the requesting node will disconnect from this neighbor, and connect to a
new neighbor chosen from the white list. The disconnected peers will stay in
the white list. The maximum sizes of the white list, and of the gray list, are
equal to 1,000 and 5,000, respectively. If the number of peers in these lists grow
over the maximum allowed size, then the peers with the oldest last seen fields
will be removed from the list.

On bitcoin side, a description of the protocol and how it manages peers can be seen in Information Propagation in the Bitcoin Network in section 3.

When a node joins the network it queries a number
of DNS servers. These DNS servers are run by volunteers
and return a random set of bootstrap nodes that are currently
participating in the network. Once connected, the joining node
learns about other nodes by asking their neighbors for known
addresses and listening for spontaneous advertisements of new
addresses. There is no explicit way to leave the network. The
addresses of nodes that left the network linger for several
hours before the other nodes purge them from their known
addresses set. At the time of writing approximately 16000
unique addresses were advertised, of which approximately
3500 were reachable at a time.

Now, IPFS, the protocol used is described in Design and Evaluation of IPFS: A Storage Layer for the Decentralized Web section 2 and 3.

Upon joining the IPFS network by connecting to a set of canonical
bootstrap peers, peers generate a public-private key pair. Every peer
in the IPFS network is identified by its unique PeerID [...] The PeerID
remains the same, unless the node operator chooses to change
it manually. [...] the structure of a Multiaddress, showing the
network and transport protocols for the communication (IPv4 and
TCP) their corresponding location-based address information (IP address 1.2.3.4 and TCP
port number 3333) followed by the protocol
to address one particular peer (p2p) and its PeerID [...]

any peer that later
retrieves the data becomes a temporary (or permanent, if they
so choose) content provider themselves by publishing a provider
record pointing to their own node to the DHT. Peers retrieving the
content do not need to trust the new providing peer but only verify
that the data they were served matches the requested CID. [...]
provider records are
associated with two other parameters: (i) the republish interval, by
default set to 12 h, to make sure that even if the original 20 peers
previously responsible for keeping the provider record go offline,
the provider will assign new ones within 12 h; and (ii) the expiry
interval, by default set to 24 h, to make sure that the publisher has
not gone offline and is still willing to serve the content. These
settings aim to prevent the system from storing and providing
stale records.

To further streamline the process, each IPFS node maintains an
address book of up to 900 recently seen peers. Nodes check whether
they already have an address for the PeerID they have discovered
before performing any further lookups.

It seems peers are being purged in any case. So, keeping peers ad vitam aeternam is perhaps not the right solution. The only way to know if we can do that is to collect information about active/inactive peers in the network. At this time, I don't think we have this information, and I'm not sure if we have tools for doing that.

Because arweave goal is different than bitcoin or monero, it could be perhaps good to store those nodes longer than expected and collect information about them. In fact, I was thinking to completely refactorize ar_peers module and store information not in a raw ETS file on the disk but on rocksdb. Stats could then be collected and we will be able to have a better understanding of the network based on facts.

[ldmberman]

We do collect this information; active peers end up higher on the list whereas inactive end up lower. We can introduce rocksdb any time, off course; I would not do it earlier than we really need it, so far the data volume is small.

[JamesPiechota]

LGTM!