APM agent configuration

[mshustov]

Problem

Kibana allows users to adjust APM agent configuration via elastic.apm.* settings in the kibana.yml file.
Kibana supplies these configuration values to both the APM nodejs and the APM RUM agents "as is". Kibana does not separate the configuration object into more specific properties for each agent, so we hope both agents have compatible configuration interfaces.
Unfortunately, this is not the case:

• an agent might have a unique configuration value that doesn't make sense for another agent (logUnhandledExceptions makes sense in elastic-apm-node only)
• agent interfaces might diverge ( elastic-apm-node: disableInstrumentations?: string | string[], elsatic/apm-rum: disableInstrumentations?: Array<InstrumentationTypes>)

Solutions

Separate configurations

To make the APM configuration bullet-proof, we can consider separating the APM nodejs configuration elastic.apm.node from the APM RUM agent one elastic.apm.rum.
Pros: clear separation of responsibility, the ability to fine-tune each agent
Cons: it's easy to misconfigure an agent (set active: false for nodejs agent, active: true for APM RUM agent), the option is not compatible with the current elastic.apm.* interface.

Separate configuration options in Kibana

Kibana can support an allow-list of configuration options for each agent to apply it to elastic.apm.* configuration space.
Pros: compatible with the current interface, Kibana controls allowed configuration options to prevent misconfiguration.
Cons: less flexibility for the end-users, the support burden of maintaining an allow-list in Kibana.

cc @trentm @lizozom @vigneshshanmugam

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[trentm]

Not sure if this is a crazy "solution" idea.

merge elastic.apm.{node,rum}.* into elastic.apm.*

Allow one to use elastic.apm.* to configure both agents if it doesn't matter, and/or elastic.apm.{node,rum}.* if specifically needed (e.g. for disableInstrumentations).

it's easy to misconfigure an agent

Is there a way to query a running Kibana for its configuration?

[mshustov]

Not sure if this is a crazy "solution" idea.

Added it as a 3rd option.

Is there a way to query a running Kibana for its configuration?

Not yet, but we need something like this.

[pgayvallet]

Kibana can support an allow-list of configuration options for each agent to apply it to elastic.apm.* configuration space

How would that work if the configuration of both agent diverge for the same option? e.g your example with disableInstrumentations (or a more extreme example where both agent have settings with the same name but with totally different values).

Also, my main question is: is this just a 'technical' issue, or do we think we may want to allow end user to configure both agents differently (e.g enabling the node agent but not the rum agent, or having different server, or IDK?)

[lizozom]

As this is an early stage for this integration and we're not promoting it to users, I think we can go with a very light weight allow-list solution.

We can always add additional flexibility later.

I'm adding a simple fix to #127892 - lets see how it's accepted

[lizozom]

I've merged the fix that omits irrelevant configurations from the server side config, before passing them down to the client.
It doesn't solve the whole issue, but I think it can serve as a reasonable solution while we keep developing APM capabilities in Kibana and the rest of the stack.

[pgayvallet]

It is now possible to specify per-service overrides via the serviceOverrides part of the APM config. Closing.