Collector API to refresh sampling strategies

[yurishkuro]

Requirement - what kind of business use case are you trying to solve?

Update sampling strategies in the collector that are passed to clients without restarting the collector.

Problem - what in Jaeger blocks you from solving the requirement?

Collectors only read sampling configuration on restart.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Issue #1058 suggests reloading configuration on file change. An alternative approach is to expose an API in the collector to allow updating the configuration through an external request. This way the actual method of managing configuration is abstracted away, e.g. a file watching solution can be implemented externally to the collector itself.

Any open questions to address

When collectors run in a cluster (usual prod deployment), the API call should update all of them, which may be difficult to support since collectors don't know about each other.

[pavolloffay]

Would this API reload the change or accept the sampling strategies file?

I like this feature, but it seems orthogonal to watching for filesystem changes.

This way the actual method of managing configuration is abstracted away, e.g. a file watching solution can be implemented externally to the collector itself.

This adds a certain complexity to the deployment for folks who want to detect just the filesystem change.

Is there any unix utility that can watch the file and create a request on a change?

[vprithvi]

Is there any unix utility that can watch the file and create a request on a change?

In Linux, one might do this with inotifywait and bash, but then you'd have to encapsulate it in some sort of linux service and ensure that it is running.

I'm not sure what the unix equivalent is, but I feel like this approach doesn't decrease complexity.

[yurishkuro]

Would this API reload the change or accept the sampling strategies file?

It would accept the full set of sampling strategies as payload.

The point here is to implement a control plane API first. How that payload is generated and sent is a separate concern for the control plane implementation. Having an API opens up many possibilities for those implementations (including writing them in different languages, as sidecars, etc).

[bourbonkk]

I am now coding strategy.json read again by sending a post request to
http://jaeger-collector/api/-/reload
However, the strategies of the collector's httpserver are easily changed,
but the strategies of the grpcserver are difficult to change.
It is possible to have the jaeger-client bring the strategy from the jaeger-collector(collector's httpserver),
but I want to make the jaeger-agent also bring it(collector's grpcserver).

Your requirement is that both httpserver and grpcserver want strategies to be updated?

[yurishkuro]

@bourbonkk what is your objective? We recently merged #2188 which allows collector to reload the sampling config file on a timer. Triggering the reload via HTTP request is less useful than the API to post the whole sampling config (but ok if you want to implement it).

I don't quite follow your comment about "strategies of the grpcserver are difficult to change" - the strategies are controlled by the strategystore, http and grpc servers simply server whatever the store provides.

[jpkrohling]

Was this implemented via #2188, or are we still considering an API to refresh the sampling strategies?

[bourbonkk]

I'm sorry. I couldn't follow up. I decided to use the sampling reload implemented in #2188.