Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Reader max window size and Writer window size #150

Open
wants to merge 4 commits into
base: 1.x
Choose a base branch
from

Conversation

cce
Copy link

@cce cce commented Mar 6, 2025

Adds NewWriterLevelDictWindowSize and NewReaderDictMaxWindowSize to limit memory usage during compression and decompression.

  • NewWriterLevelDictWindowSize added to set specific window size (in bytes, not as log base 2)
  • NewReaderDictMaxWindowSize added to set a maximum window size that this decompressor will allow, also in bytes.
  • Tests exercising the Writer can set window sizes, and that limiting them in the Reader works.

More documentation on window sizes are in RFC 9659 and in zstd.h:

/*! ZSTD_DCtx_setMaxWindowSize() :
 *  Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
 *  This protects a decoder context from reserving too much memory for itself (potential attack scenario).
 *  This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
 *  By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
 * @return : 0, or an error code (which can be tested using ZSTD_isError()).
 */
    ZSTD_c_windowLog=101,    /* Maximum allowed back-reference distance, expressed as power of 2.
                              * This will set a memory budget for streaming decompression,
                              * with larger values requiring more memory
                              * and typically compressing more.
                              * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
                              * Special: value 0 means "use default windowLog".

Resolves #123.

cce added 3 commits March 6, 2025 14:03
…mit memory

usage during compression and decompression. More details available in RFC 9659.
- NewWriterLevelDictWindowSize added to set specific window size (in bytes)
- NewReaderDictMaxWindowSize added to set a maximum window size (in bytes) that
  this decompressor will allow.
Copy link
Collaborator

@Viq111 Viq111 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution and the thorough test cases !

Unfortunately as this is using experimental (per zstd's documentation) features and disallow dynamic linking, we wouldn't be able to merge to the 1.x. I'll leave the PR open and if the upstream zstd repositority makes this parameter non-experimental, we can revisit and merge it

@@ -1,6 +1,7 @@
package zstd

/*
#define ZSTD_STATIC_LINKING_ONLY 1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🫤 Unfortunately this means only static linking is allowed which is a deal breaker for many users of this library that are using dynamic linking. As this would break compatibility, we can't merge it to 1.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Setting decompression WindowSize?
2 participants