-
Notifications
You must be signed in to change notification settings - Fork 674
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to rejoin a group when consumer poll timeout in confluent-kafka-go v2.1.0
#980
Unable to rejoin a group when consumer poll timeout in confluent-kafka-go v2.1.0
#980
Comments
If your client takes too much time to process messages, for example, Be aware that if there is a run time error while processing messages, you must produce the message back to Kafka for retrying because the previous message is marked as consumed. As documented at https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#max-poll-interval-ms,
|
By any chance, are you using "go.logs.channel.enable"? I can't reproduce it locally given the pseudocode, it seems to work fine for me (I can rejoin the group). |
Thank for your reply. I'm not using “go.logs.channel.enable”, I tried using this parameter in my test code, but the results were the same as before. None of them rejoined the group after the error, but the consumer is still polling empty messages. When I looked at the group, there were no active consumers. |
Here are my test code :
|
I'm facing the same issue when I upgraded from 2.0.2 to 2.1.0. |
Detailed logs:
|
Thanks for this report. I was able to reproduce it locally as well, I'll look into it further. |
+1 having the same issue when upgraded from 2.0.2 to 2.1.0.
|
I am also getting this, despite poll() definitely being called. Steps to reproduce:
Despite poll being called (and logs showing the fetch), the consumer is still kicked out of the group after max.poll.interval.ms. |
+1 same issue. Where is @edenhill ? |
This is a librdkafka issue, we're fixing it here: confluentinc/librdkafka#4256 |
ok thanks @milindl but what I don't understand is that we should have detected this kind of problem easily with a basic test in the confluent go lib, no ? |
And for the tests, yes, we should have ideally detected it. |
OK. |
@fkarakas , yep, that's what we did, we added a failing test to librdkafka, where the actual issue exists. I was talking about extending the go test suite to include more tests, too. Fix is available in the v2.1.1-RC1 and we expect to have it in v2.1.1 when it is released after some soak testing. |
Description
When I upgraded
confluent-kafka-go
fromv2.0.2
tov2.1.0
, it was not possible to rejoin consumers back to the consumer group in2.1.0
when the time interval between two consumerpoll
operations exceededmax.poll.interval.ms
, but it was feasible inv2.0.2
.When Group myTopicName
join state
changedwait-unassign-to-complete -> init
:v2.0.2
: Group myTopicName join with 1 subscribed topic(s)...v2.1.0
: Requseting metadata for 1/1 topics: periodic topic and broker list refresh...client config:
consumer config:
How to reproduce
pseudo-code:
Checklist
Please provide the following information:
v2.0.2
tov2.1.0
):ConfigMap{...}
"debug": ".."
as necessary)The text was updated successfully, but these errors were encountered: