Add ReplicaStore

[lxfind]

We can reuse the AuthorityStore for the store in a replica.
There can be more differences but the major difference would be storing all history versions of objects.
This PR adds a type template to the store, so that we could use it for both authority and replica.
Since it's type template, the cost should be minimum.

Open to suggestions on using a boolean flag field instead of type template if there is a good reason.

[gdanezis]

Love the approach of using a type template!

But instead of adding a new table for all objects, rather could we not just not run the following two lines in the store?
https://github.com/MystenLabs/fastnft/blob/c72a4aea98dc4fe76de8515fe94f0023524829ef/sui_core/src/authority/authority_store.rs#L429-L430

[lxfind]

@gdanezis That table is keyed on ObjectID. So we would still end up overwriting objects.

[velvia]

Added hopefully useful comments 😄

Regarding one table vs two tables: The only way I could see a unified single table would be a table based on (ObjectID, Version). The Version field would be ignored or set to 0 always in the authority version of the store -- OR you use the right version, but delete the older versions. That would create more churn though, but it might be OK since it fits the LSM pattern.

Actually I kinda like that idea slightly better than two tables. The difference in behaviour would then be if you kept older versions of objects. But maybe it would mess up the read patterns in authorities.

[velvia]

Instead of calling it authority, can we instead make it tied to the function which differentiates the store itself - ie the all object versions map. So I'd call it ALL_OBJ_VER: bool instead. The reason is that down the line, another user of the store could need this functionality.

[lxfind]

I was also thinking about that this flag can be used down the line, but in a different way: I think there will be other differences between authorities and non-authorities, not just whether the object versions are kept.

[velvia]

Right, that is true - my point mainly was that "Authority" might become a misnomer down the line as then the functionality of the store becomes unclear, then I have to look up what differences an authority might have. We can always add more types for diff features, though that adds, well, more type complexity.

[velvia]

An ObjectRef contains the ObjectID, version, and Object Ref.
Isn't the object ref redundant here? There should never be two objects in the system with the same version but different object refs correct?

I would create a new type which is just the ObjectID and version, or just use that as a tuple. Dropping the object ref saves a ton of space in the key, enabling faster object lookups and less space used.

[velvia]

I'm still worried about this part. An ObjectID is 20 bytes, sequence number 28 bytes. An Object digest, the last part of an ObjectRef, is 32 bytes. If we don't need the ObjectRef, that shaves the key size down to less than half of what an entire ObjectRef is. When you multiply this by many many millions of objects it makes a big difference.

[gdanezis]

@gdanezis That table is keyed on ObjectID. So we would still end up overwriting objects.

What about changing its key to ObjectRef?

[lxfind]

Re: using one table instead of two tables

Why do we prefer one over two here? Is it just for code clarity?
There are quite some consequences of merging them to 1 table. For instance, for shared objects, we only have an object ID in the input. If we only have a table keyed by ObjectRef, we would need another table to map from object ID to objectRef.
We could consider setting a dummy version for authorities in this merged table, but the complexity of dealing with that is definitely more than having two tables, unless I am missing something.

[gdanezis]

I do not want to block this forever, but just to clarify what the DBMap BTree allows us to do efficiently:

If we only have a table keyed by ObjectRef, we would need another table to map from object ID to objectRef

We can efficiently lookup the latest ObjectID in in a table indexed by ObjectRef by using the .skip_prev_to() method and a key (ObjectID, u64::MAX, Digest::MAX). This is the reason why we use the compound keys with ObjectID being the first element.

[gdanezis]

Lets start somehwere, and this is a fine place to start -- the refactor in the direction we want.

[velvia]

Would shared objects not be stored in the all_object_versions table?

The tradeoffs for two vs one table are:

1. Size and storage needs
2. consistency, potential for differences

For a thin client 1) is not a concern, maybe for a full replica that wants to keep both tables it could be a concern, but the size of the all_objects table would quickly outgrow the other one, so probably not.

I'm more worried about 2) since there are many more places we have to get it right. I believe you use batches which is a good start though.

[lxfind]

Let me iterate on this a bit

stale

[lxfind]

Made the following changes:

1. Renamed the type template to ALL_OBJ_VER to be specific. We can get back to the naming if we need more different tables.
2. The all_object_versions table is indexed on (ObjectID, SequenceNumber), which saves some space.

I am still using two tables instead of one. I think that trying to unify them has too many implications to the implementation of AuthorityStore that would complicate the authority implementation. For example, having to delete the old version as well as adding the new version is going to be much more expensive/slower than simply overriding the table by id.
We can try to simplify things down the road if there is a need.

[velvia]

@lxfind thanks for the changes