Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update create track #1846

Merged
merged 4 commits into from
May 23, 2024

Conversation

gareth-ellis
Copy link
Member

I seem to mess up my old branch - this is the same as PR #1836
Add option to increase batch size when creating a track to speed up download of data. Update track layout to adhere to best practices

Creating tracks from large corpus can take quite a bit of time, so I have added the option to increase the batch size of scan, so if a user is running with a stable enough network connection and enough hardware resources on the rally instance, they can increase the batch size to speed up download.

I have also updated the track layout to better match how we tend to layout our tracks

There was feedback to add more docs, so that's done too

@gareth-ellis gareth-ellis requested review from inqueue and ebadyano May 17, 2024 10:46
@gareth-ellis gareth-ellis added the enhancement Improves the status quo label May 17, 2024
@gareth-ellis gareth-ellis added this to the 2.x milestone May 17, 2024
@gareth-ellis gareth-ellis requested a review from a team May 22, 2024 08:33
@ebadyano
Copy link
Contributor

ebadyano commented May 22, 2024

@gareth-ellis Thank you for updating the docs. LGTM.

One additional thing, since you are alreday updating the docs do you mind adding an option for data streams? During last review when I tested the change with datastreams I didn't realize I needed to specify --data-streams to actually make track create to include them in the track. See https://github.com/elastic/rally/pull/1531/files Nevermind I somehow missed that it was in the docs already
I will test it shortly to confirm it works

@gareth-ellis
Copy link
Member Author

The data-stream option is already in the docs - https://esrally.readthedocs.io/en/stable/command_line_reference.html#data-streams - or were you meaning something different?

@ebadyano
Copy link
Contributor

The data-stream option is already in the docs - https://esrally.readthedocs.io/en/stable/command_line_reference.html#data-streams - or were you meaning something different?

You are right, I somehow missed it.

@gareth-ellis
Copy link
Member Author

I'd like to add the option to setup writing back to the datastream too - that isn't currently in place, but I thought that could be in a seperate PR

Copy link
Contributor

@ebadyano ebadyano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gareth-ellis gareth-ellis merged commit 31a0eb7 into elastic:master May 23, 2024
17 checks passed
@gareth-ellis gareth-ellis modified the milestones: 2.x, 2.11 May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improves the status quo
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants