-
Notifications
You must be signed in to change notification settings - Fork 11.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] Load balancing cache in POP consumption mode #9131
base: develop
Are you sure you want to change the base?
Conversation
Hello, please fix the code running test and output the test document |
okok |
The cache should be updated when the client is offline or the number of queues changes, but I don't see the logic for that |
private static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME); | ||
|
||
// Cache with pop mode rebalancing under the condition that consumer groups and queues remain unchanged | ||
private final ConcurrentMap<String, ConcurrentHashMap<String, Set<MessageQueue>>> loadBalanceDateTable = new ConcurrentHashMap<>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When the number of clients changes or the number of queues changes.Should clear cache
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I've added a cleanup thread for PopCacheManager and tested it to completely clear the cache every 24 hours (the cache expiration only happens when the topic goes offline, and I'll modify "Brief Description" to simply explain the cache logic). This cleanup interval can be added to the configuration if needed later
Which Issue(s) This PR Fixes
Add cache for each client request broker end rebalancing in the pop consumption mode to avoid wasting computing resources of broker end due to repeated calculation of load balancing without changes. The effect is more obvious in theory when there are a large number of queues or consumers
Fixes #9123
Brief Description
How Did You Test This Change?
Ten consumers were registered for the test consumer group, and mock 15 queues were set up to test the correctness and performance of the broker-end doLoadBalance method under the test pop consumption mode. All QueryAssignmentProcessorTest (including the original 5 methods) passed, and the performance is as follows (in the local environment)