Estargz integrates Dragonfly P2P

[gaius-qi]

I am a Maintainer of Dragonfly. Does estargz need to combine Dragonfly's P2P transfer function to speed up image loading? Integrate like Nydus and Dragonfly, refer to https://d7y.io/docs/setup/integration/nydus.

If estargz and dragonfly P2P integration is feasible, our two communities can work together to do a good job in the image acceleration ecosystem.

@ktock @AkihiroSuda

[ktock]

@gaius-qi Thanks for the proposal. SGTM 👍

[gaius-qi]

Whether estargz can support a mirror mode similar to nydus? Estargz proxy download requests to Dragonfly.

[ktock]

@gaius-qi Yes, stargz-snapshotter supports registry mirrors:

https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/overview.md#registry-mirrors-and-insecure-connection

[[resolver.host."exampleregistry.io".mirrors]]
host = "mirrorhost.io"

[gaius-qi]

Does it support custom headers?

Registry that controls the final Dragonfly access by adding the X-Dragonfly-Registry Header:

[gaius-qi]

Does it support custom headers?

Registry that controls the final Dragonfly access by adding the X-Dragonfly-Registry Header:

cc @ktock

[gaius-qi]

Please help us supplement the documentation for the stargz usage section.

PR: dragonflyoss/d7y.io#178

@ktock

[ktock]

@gaius-qi

PR: dragonflyoss/d7y.io#178

Added a comment

custom headers

PTAL #983 Does this work for your use-case?

[gaius-qi]

#983

Sure, this is work for my use-case #983

This is the document and perf test. If there is any part of stargz that can be optimized, you can tell me, and I will retest the new data.

Document: https://d7y.io/docs/setup/integration/stargz

[gaius-qi]

There is another question that can describe the situation of mirror mode to me, how does stargz exchange auth token.

I have a problem now that the first time get blobs from dragonfly mirror returns 401, then stargz goes to the auth service to exchange for the auth token, and the second time stargz request dragonfly mirror with the token, it still returns 401.

How can I get the stargz logs to find the problem?

[ktock]

@gaius-qi

Which auth method are you using? There are 3 methods are supported as of now: https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/overview.md#registry-related-configuration

Stargz-snapshotter reports failed to prepare remote snapshot warning log when it fails to connect to the registry. Could you provide the log around that message?

[gaius-qi]

@gaius-qi

Which auth method are you using? There are 3 methods are supported as of now: https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/overview.md#registry-related-configuration

Stargz-snapshotter reports failed to prepare remote snapshot warning log when it fails to connect to the registry. Could you provide the log around that message?

There should be no problem now, it was caused by dirty data before.

[gaius-qi]

@gaius-qi

PR: dragonflyoss/d7y.io#178

Added a comment

custom headers

PTAL #983 Does this work for your use-case?

@ktock

Which version can be released?

Maybe v0.12.2?

Stargz return custom header, refer to dragonflyoss/d7y.io#179.

Can you put a link to the documentation for dragonfly and stargz integration (https://d7y.io/docs/setup/integration/stargz) into the stargz documentation?

[ktock]

@gaius-qi

There should be no problem now, it was caused by dirty data before.

👍

Maybe v0.12.2?

We'll release v0.13.0 in days.

Can you put a link to the documentation for dragonfly and stargz integration (https://d7y.io/docs/setup/integration/stargz) into the stargz documentation?

Sure. It would be great if you can address dragonflyoss/d7y.io#178 (comment).

[gaius-qi]

@ktock

Change eStargz version to v0.13.0: dragonflyoss/d7y.io#180

Stargz document: #988

[gaius-qi]

I'm going to post this article to the cncf.io blog, do you have any suggestions?

[ktock]

@gaius-qi

article

I'm curious about that lazy pulling of eStargz node:19.0 image takes 10s+ in the "Performance testing" section in the linked document. What can be the major bottleneck there (too many prefetch bytes)?

[gaius-qi]

@gaius-qi

article

I'm curious about that lazy pulling of eStargz node:19.0 image takes 10s+ in the "Performance testing" section in the linked document. What can be the major bottleneck there (too many prefetch bytes)?

OCIv1:

eStargz:

I am running node --version.

[gaius-qi]

I'm closing this issue. If you have any other questions, let's keep in touch. If you have any questions, you can send me an email at gaius.qi@gmail.com, or you can submit an issue in dragonfly's issue.