Add API to regenerate index from CARv1 or CARv2 #309
Conversation
The index generation APIs either allowed reading an existing index from a CARv2 or explicitly required a CARv1 to generate index. Introduce APIs to make it easier for users that want to regenerate the index regardless of whether it exists in a CAR file or not. The index generation APIs are changed to accept either of the formats and re-generate the index from the data payload unless `ReadOrGenerate` is called. Adjust the tests to run for all flavours of index generation with both CARv1 and CARv2 payload.
| if v2h.DataOffset < HeaderSize { | ||
| return fmt.Errorf("malformed CARv2; data offset too small: %d", v2h.DataOffset) | ||
| } | ||
| if v2h.DataSize < 1 { |
There was a problem hiding this comment.
I think a zero-length payload is still valid isn't it? it would just produce an empty index. not a big deal, zero payload probably means an error but strictly speaking I think 0 should be acceptable.
There was a problem hiding this comment.
I thought about this when writing this statement. I actually think 1 might be too small. The rationale is that according to CARv2 spec the data size refers to the inner data payload size which should be a valid CARv1; as in the entire thing including the CARv1 header. At the danger of sounding pedantic (apologies if I do 🙂) accepting zero as DataSize would then mean an empty file is technically a valid CARv1?
That's why I think that value should be the minimum possible size for a CARv1, which is a CARv1 header with version value 1 and no roots which I think would come to 18 bytes.
WDYT?
There was a problem hiding this comment.
Looks like the CARv1 should have at least one root CID, which increases the minimum acceptable CARv1 size.
The smallest CID I can think of is one with multihash code IDENTITY and empty data, which brings the total minimum CARv1 size to 26 bytes if my math is right.
There was a problem hiding this comment.
Moving the discussion out to a separate issue: #310
The index generation APIs either allowed reading an existing index from
a CARv2 or explicitly required a CARv1 to generate index.
Introduce APIs to make it easier for users that want to regenerate the
index regardless of whether it exists in a CAR file or not. The index
generation APIs are changed to accept either of the formats and
re-generate the index from the data payload unless
ReadOrGenerateiscalled.
Adjust the tests to run for all flavours of index generation with both
CARv1 and CARv2 payload.