Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetching from a log compacted topic pulls all new messages with same key on first call #803

Closed
vordimous opened this issue Feb 20, 2024 · 1 comment
Closed

Fetching from a log compacted topic pulls all new messages with same key on first call #803

vordimous opened this issue Feb 20, 2024 · 1 comment
Assignees
@vordimous
Labels
bug Something isn't working needs_triage Unknowns need to be addressed

Comments

@vordimous
Copy link
Contributor

Describe the bug
The first time all messages are fetched from a log compacted topic, after multiple messages have been produced, all messages are returned. Every subsequent request returns only the most recent key.

To Reproduce
Steps to reproduce the behavior:
using the https://github.com/vordimous/zilla-examples/tree/main/http.kafka.crud example

  1. start
./setup.sh
  1. Post multiple messages
% curl -k -v -X POST https://localhost:7143/items -H 'Idempotency-Key: 1'  -H 'Content-Type: application/json' -d '{"greeting":"Hello, world1"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:7143...
* Connected to localhost (127.0.0.1) port 7143 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=localhost
*  start date: Dec 21 23:04:14 2021 GMT
*  expire date: Dec 19 23:04:14 2031 GMT
*  issuer: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=Test CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: POST]
* h2h3 [:path: /items]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:7143]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* h2h3 [idempotency-key: 1]
* h2h3 [content-type: application/json]
* h2h3 [content-length: 28]
* Using Stream ID: 1 (easy handle 0x12180ec00)
> POST /items HTTP/2
> Host: localhost:7143
> user-agent: curl/7.86.0
> accept: */*
> idempotency-key: 1
> content-type: application/json
> content-length: 28
> 
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 204 
< 
* Connection #0 to host localhost left intact

% curl -k -v -X POST https://localhost:7143/items -H 'Idempotency-Key: 1'  -H 'Content-Type: application/json' -d '{"greeting":"Hello, world1"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:7143...
* Connected to localhost (127.0.0.1) port 7143 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=localhost
*  start date: Dec 21 23:04:14 2021 GMT
*  expire date: Dec 19 23:04:14 2031 GMT
*  issuer: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=Test CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: POST]
* h2h3 [:path: /items]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:7143]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* h2h3 [idempotency-key: 1]
* h2h3 [content-type: application/json]
* h2h3 [content-length: 28]
* Using Stream ID: 1 (easy handle 0x14c00cc00)
> POST /items HTTP/2
> Host: localhost:7143
> user-agent: curl/7.86.0
> accept: */*
> idempotency-key: 1
> content-type: application/json
> content-length: 28
> 
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 204 
< 
* Connection #0 to host localhost left intact

% curl -k -v -X POST https://localhost:7143/items -H 'Idempotency-Key: 1'  -H 'Content-Type: application/json' -d '{"greeting":"Hello, world1"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:7143...
* Connected to localhost (127.0.0.1) port 7143 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=localhost
*  start date: Dec 21 23:04:14 2021 GMT
*  expire date: Dec 19 23:04:14 2031 GMT
*  issuer: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=Test CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: POST]
* h2h3 [:path: /items]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:7143]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* h2h3 [idempotency-key: 1]
* h2h3 [content-type: application/json]
* h2h3 [content-length: 28]
* Using Stream ID: 1 (easy handle 0x14280bc00)
> POST /items HTTP/2
> Host: localhost:7143
> user-agent: curl/7.86.0
> accept: */*
> idempotency-key: 1
> content-type: application/json
> content-length: 28
> 
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 204 
< 
* Connection #0 to host localhost left intact

% curl -k -v -X POST https://localhost:7143/items -H 'Idempotency-Key: 1'  -H 'Content-Type: application/json' -d '{"greeting":"Hello, world1"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:7143...
* Connected to localhost (127.0.0.1) port 7143 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=localhost
*  start date: Dec 21 23:04:14 2021 GMT
*  expire date: Dec 19 23:04:14 2031 GMT
*  issuer: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=Test CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: POST]
* h2h3 [:path: /items]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:7143]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* h2h3 [idempotency-key: 1]
* h2h3 [content-type: application/json]
* h2h3 [content-length: 28]
* Using Stream ID: 1 (easy handle 0x15700bc00)
> POST /items HTTP/2
> Host: localhost:7143
> user-agent: curl/7.86.0
> accept: */*
> idempotency-key: 1
> content-type: application/json
> content-length: 28
> 
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 204 
< 
* Connection #0 to host localhost left intact
  1. get multiple
% curl -k -v https://localhost:7143/items
*   Trying 127.0.0.1:7143...
* Connected to localhost (127.0.0.1) port 7143 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=localhost
*  start date: Dec 21 23:04:14 2021 GMT
*  expire date: Dec 19 23:04:14 2031 GMT
*  issuer: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=Test CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /items]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:7143]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x12f00bc00)
> GET /items HTTP/2
> Host: localhost:7143
> user-agent: curl/7.86.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200 
< content-type: application/json
< etag: AQIABg==
< 
* Connection #0 to host localhost left intact
[{"greeting":"Hello, world1"},{"greeting":"Hello, world1"},{"greeting":"Hello, world1"}]%
  1. get one
% curl -k -v https://localhost:7143/items
*   Trying 127.0.0.1:7143...
* Connected to localhost (127.0.0.1) port 7143 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=localhost
*  start date: Dec 21 23:04:14 2021 GMT
*  expire date: Dec 19 23:04:14 2031 GMT
*  issuer: C=US; ST=California; L=Palo Alto; O=Aklivity; OU=Development; CN=Test CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /items]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:7143]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x13080bc00)
> GET /items HTTP/2
> Host: localhost:7143
> user-agent: curl/7.86.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200 
< content-type: application/json
< etag: AQIABg==
< 
* Connection #0 to host localhost left intact
[{"greeting":"Hello, world1"}]%

Expected behavior
Zilla returns only the latest message
@vordimous vordimous added the bug Something isn't working label Feb 20, 2024
@vordimous vordimous added the needs_triage Unknowns need to be addressed label Aug 22, 2024
@vordimous vordimous self-assigned this Aug 22, 2024
@vordimous
Copy link
Contributor Author

unable to reproduce with zilla 0.9.93

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vordimous vordimous

Labels
bug Something isn't working needs_triage Unknowns need to be addressed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@akrambek @vordimous