Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

batch-system: use different default max batch size between master and release branch #9917

Merged
merged 6 commits into from
Mar 30, 2021
Merged

batch-system: use different default max batch size between master and release branch #9917

merged 6 commits into from
Mar 30, 2021

Conversation

gengliqi
Copy link
Member

@gengliqi gengliqi commented Mar 29, 2021 •
edited
Loading

Signed-off-by: gengliqi gengliqiii@gmail.com

What problem does this PR solve?

Problem Summary:
On master, the hibernate region is enabled by default so the max-batch-size can be larger.
However it is disabled by default on the release branch, the max-batch-size is too small which may cause many problems, e.g. too small write batch, hunger problem.
You can also see #9020 for more details.

What is changed and how it works?

What's Changed:
use different default max batch size between master and release branch.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Tests

  • No code

Side effects

  • No

Release note

  • If hibernate-regions is enabled, the default value of store-max-batch-size is 256, otherwise it's 1024.
  • Change the default value of apply-max-batch-size from 1024 to 256.

@ti-chi-bot ti-chi-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Mar 29, 2021
@gengliqi gengliqi added the needs-cherry-pick-release-5.0 Type: Need cherry pick to release 5.0 label Mar 29, 2021
BusyJay
BusyJay reviewed Mar 29, 2021
View reviewed changes
components/batch-system/src/config.rs Outdated
@@ -13,8 +13,15 @@ pub struct Config {

impl Default for Config {
fn default() -> Config {
// FIXME: if hibernate regions is enabled by default on release branch, we can use
// the same configuration here.
let max_batch_size = if tikv_util::build_on_master_branch() {
Copy link
Member

@BusyJay BusyJay Mar 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about changing it at validate method according to the value of hibernate-region?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if users set the max-batch-size? Use a Option?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

@gengliqi
address comment
4ba4cba 
Signed-off-by: gengliqi <gengliqiii@gmail.com>
@gengliqi gengliqi force-pushed the change-default-max-batch-size branch from 6c4f3f0 to 4ba4cba Compare March 30, 2021 04:16
@ti-chi-bot ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 30, 2021
@gengliqi
make clippy happy
3b59390 
Signed-off-by: gengliqi <gengliqiii@gmail.com>
@gengliqi gengliqi force-pushed the change-default-max-batch-size branch from 55301b0 to 3b59390 Compare March 30, 2021 04:47
BusyJay
BusyJay reviewed Mar 30, 2021
View reviewed changes
Copy link
Member

@BusyJay BusyJay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

@gengliqi
address comments
56dd95f 
Signed-off-by: gengliqi <gengliqiii@gmail.com>
@ti-chi-bot ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 30, 2021
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 30, 2021
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • BusyJay
  • NingLin-P

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 30, 2021
@gengliqi
Copy link
Member Author

/merge

@ti-chi-bot
Copy link
Member

@gengliqi: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 56dd95f

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 30, 2021
@gengliqi
Copy link
Member Author

/run-all-tests

@gengliqi
Copy link
Member Author

/run-all-tests

@zhouqiang-cl
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

@zhouqiang-cl: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit ab32ddc into tikv:master Mar 30, 2021
ti-srebot pushed a commit to ti-srebot/tikv that referenced this pull request Mar 30, 2021
@gengliqi @ti-srebot
cherry pick tikv#9917 to release-5.0
3b829f2 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Mar 30, 2021
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-5.0 in PR #9930

ti-chi-bot pushed a commit that referenced this pull request Mar 30, 2021
@ti-srebot @gengliqi
batch-system: use different default max batch size between master and…
3795f09 
… release branch (#9917) (#9930)

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: Liqi Geng <gengliqiii@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BusyJay BusyJay BusyJay approved these changes

@NingLin-P NingLin-P NingLin-P approved these changes

@5kbpers 5kbpers Awaiting requested review from 5kbpers

@Little-Wallace Little-Wallace Awaiting requested review from Little-Wallace

Assignees
No one assigned
Labels
needs-cherry-pick-release-5.0 Type: Need cherry pick to release 5.0 size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@gengliqi @ti-chi-bot @zhouqiang-cl @ti-srebot @BusyJay @NingLin-P