Added parameters for Clickhouse flush interval and max buffer size #1073
Conversation
Currently, both are hard-coded to relatively small values. This may cause an unnecessary high load on the Clickhouse instance. It makes sense to make those configurable.
|
I think this is a good addition. I missed the fact that we flush sessions every second instead of every 5-10 seconds which seems more reasonable to me. What values are you using? Or is this PR in preparation for testing on your end? The values that are hard-coded at the moment are pretty arbitrary so I'm curious to hear what settings work well in the wild. |
|
Would you be so kind as to document the changes as well? https://github.com/plausible/docs/blob/master/docs/self-hosting-configuration.md |
|
Hi @ukutaht, thanks for accepting the PR, I'll prepare the documentation changes asap.
Tbh, we had to configure clickhouse-bulk to mitigate this issue without the need to fork Plausible. Regarding the acceptable values, a lot depends on your Clickhouse instance's overall load and how many clients write to it in parallel. In our case, we have a pretty huge instance under heavy load from several applications so adding writes from Plausible every second was problematic. Clickhouse devs suggest this:
If Plausible is the only client of Clickhouse and there's just a single Plausible node, I guess 5 secs / 10k buffer would be okay. But once you start scaling it and adding more nodes, you should probably consider using higher values. |
|
Ah OK that makes sense. Indeed once every second is probably too fast, even for a single node. I'll see what this update does to our Clickhouse host metrics and I'll probably try a 10 second interval as well. |
Changes
Currently, both "flush interval" and "max buffer size" are hard-coded to relatively small values.
This may cause an unnecessarily high load on the Clickhouse instance.
It makes sense to make those configurable.
Tests
Changelog
Documentation
Docs haven't been updated, but if you're okay with this PR, I'll open a PR there.