Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Load balancing cache in POP consumption mode #9131

Open
wants to merge 8 commits into
base: develop
Choose a base branch
from

Conversation

YoWuwuuuw
Copy link

@YoWuwuuuw YoWuwuuuw commented Jan 14, 2025

Which Issue(s) This PR Fixes

Add cache for each client request broker end rebalancing in the pop consumption mode to avoid wasting computing resources of broker end due to repeated calculation of load balancing without changes. The effect is more obvious in theory when there are a large number of queues or consumers

Fixes #9123

Brief Description

  1. Cache location: Add cache before the rebalancing distribution queue
  2. Cache logic: Since only the number of consumer groups, the number of queues and clientId can affect the rebalancing calculation, the topic is used as the first layer cache key, and the clientId is used as the second layer cache key, and the consumer groups and queues of the topic are stored as the verification basis
  3. Cache hit: The topic exists in the cache, and the number of consumer groups and queues remain the same as before
  4. Cache update: When the cache is not hit, the cache is updated after the rebalancing calculation
  5. Details: Use reentrant lock to maintain consistency and efficiency, set 24-hour clean-up thread to clean up the cache, and prevent the cache from failing due to the offline of topics

How Did You Test This Change?

Ten consumers were registered for the test consumer group, and mock 15 queues were set up to test the correctness and performance of the broker-end doLoadBalance method under the test pop consumption mode. All QueryAssignmentProcessorTest (including the original 5 methods) passed, and the performance is as follows (in the local environment)

32a8f585dfbee87e9d7e2cbdb32317a

d6e52c5a9e21da12f5038f1b1c14df1

c446c0d2f326bc8a11a36836447db2d

@3424672656
Copy link
Contributor

Hello, please fix the code running test and output the test document

@YoWuwuuuw
Copy link
Author

Hello, please fix the code running test and output the test document

okok

@3424672656
Copy link
Contributor

The cache should be updated when the client is offline or the number of queues changes, but I don't see the logic for that

private static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);

// Cache with pop mode rebalancing under the condition that consumer groups and queues remain unchanged
private final ConcurrentMap<String, ConcurrentHashMap<String, Set<MessageQueue>>> loadBalanceDateTable = new ConcurrentHashMap<>();
Copy link
Contributor

@3424672656 3424672656 Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the number of clients changes or the number of queues changes.Should clear cache

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've added a cleanup thread for PopCacheManager and tested it to completely clear the cache every 24 hours (the cache expiration only happens when the topic goes offline, and I'll modify "Brief Description" to simply explain the cache logic). This cleanup interval can be added to the configuration if needed later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] Load balancing cache in POP consumption mode
3 participants