Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose set_pledged_src_size as a public API for zstd compression #329

Open
shrekris-openai opened this issue Feb 27, 2025 · 1 comment
Open

Comments

@shrekris-openai
Copy link

Issue

async-compression offers no way to call set_pledged_src_size() on the zstd encoder. Could we provide a public API in async-compression to call this?

Context

zstd encoding can include an optional Frame_Content_Size value. Decompression libraries such as Python's zstandard provide APIs (e.g. decompress) that use this value to decompress the data in a single operation. Without it, these libraries must treat the encoded bytes as a stream.

Writing Frame_Content_Size requires two steps (from the zstd manual):

  1. Enable ZSTD_c_contentSizeFlag. async-compression already offers a CParameter method to do this.
  2. Call ZSTD_CCtx_setPledgedSrcSize() to set the decompressed data size. The zstd crate already exposes the public method set_pledged_src_size() for this, but it's not exposed by the async-compression crate.

When set_pledged_src_size is not called, the encoded bytes won't have a Frame_Content_Size value.

@NobodyXu
Copy link
Collaborator

Yes that sounds reasonable, I will review and merge a PR for it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants