Remove 'prog' label from metrics

[JamesBarwell]

Hi, when metrics are exported by mtail, I notice that it automatically adds a reference to the mtail program as a label for the metric, e.g.: mymetric{prog="foo.mtail",instance="276d82f2092c"} 1

I was just wondering if it's possible to disable or remove this label via a configuration option. I ask because I am writing a generic exporter and I don't think it's relevant to expose the internals of how my mtail programs are set up to external consumers (or the fact that I am using mtail at all - as great as it is!). In my situation, there is no scenario where a user would want to query by the "progs" label.

Cheers.

[dannyk81]

You can always drop the label in Prometheus during relabel process, using metrics_relabel_configs.

This way these labels will not be ingested and the end-user will not see them.

There's a great blog about this here

But, in general - this should do the trick:

   metric_relabel_configs:
   - target_label: prog
     replacement: ''

[jaqx0r]

Thanks Danny, that's a good idea. James, I'm considering adding a flag to let you choose to not export the prog label, but the intention was to add a namespace to mtail variables which otherwise don't exist, there's nothing preventing two programs exporting the same name, with hilarious consequences, so I'm not inclined to remove them totally from mtail itself -- and with Danny's suggesting I think I can't support this without some other way of solving the namespace problem.

…

On 24 May 2017 at 07:01, Danny Kulchinsky ***@***.***> wrote: You can always drop the label in Prometheus during relabel process, using metrics_relabel_configs. This way these labels will not be ingested and the end-user will not see them. There's a great blog about this here <https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs/> But, in general - this should do the trick: metric_relabel_configs: - target_label: prog replacement: '' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC5b-wp9l4ZcolZIzUCZi2BR4IbuKr8oks5r80k1gaJpZM4NgTGi> .

[JamesBarwell]

Not sure how complex it would be to implement, but from a user point of view, I'd suggest that conflicting metrics should generate a warn when detected. The second (conflicting) metric - or entire program that it resides in - should be dropped. If this can occur at startup then all the better. Perhaps it can be flagged when using the --one_shot option for testing?

Another thought is whether each program could define and export its own namespace, which would then be prefixed to each metric it writes to. That might make it easier to spot and reject a conflicting namespace. But perhaps that would rather limit how people name their metrics and force them into putting everything into single programs. I've got something similar going on with a library that I wrote for a customer to provide a generic way to monitor their services. The library requires instantiation with a namespace, which is then prefixed on all metrics, logs, notifications, etc.

A config switch would be useful for me, even if it does come with a big caveat!

@dannyk81 Thanks, that is a helpful workaround.

[jaqx0r]

There's no deterministic ordering for program loads, so while it's totally reasonable to catch and warn, will probably result in more confused users. The prefixing option is one I considered when starting out, and decided against it, because it imposes form on the variable name, that would limit users and program authors! There's some irony of course that now the limit is people can't write programs that use prog as a label :) Having a commandline switch would be best, yeah. I'll do that, and default to the current behaviour.

…

On 25 May 2017 at 17:51, James Barwell ***@***.***> wrote: Not sure how complex it would be to implement, but from a user point of view, I'd suggest that conflicting metrics should generate a warn when detected. The second (conflicting) metric - or entire program that it resides in - should be dropped. If this can occur at startup then all the better. Perhaps it can be flagged when using the --one_shot option for testing? Another thought is whether each program could define and export its own namespace, which would then be prefixed to each metric it writes to. That might make it easier to spot and reject a conflicting namespace. But perhaps that would rather limit how people name their metrics and force them into putting everything into single programs. I've got something similar going on with a library that I wrote for a customer to provide a generic way to monitor their services. The library requires instantiation with a namespace, which is then prefixed on all metrics, logs, notifications, etc. A config switch would be useful for me, even if it does come with a big caveat! @dannyk81 <https://github.com/dannyk81> Thanks, that is a helpful workaround. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC5b-wQgCo1iYScdNeXJaGIdjszIP0a4ks5r9TMWgaJpZM4NgTGi> .

[jaqx0r]

I'm going to have to do some refactoring in the metric storage, but I am seeing that it is totally necessary (because it's a mess) and will make doing this feature request much easier.

So the only unknown is how do we resolve the namespace problem. I'm happy to drop the prog label entirely but:

if two programs define the same metric name, do they have to have the same type and labelsets?

For type, yes: Prometheus' TYPE annotation is only allowed once per metric. For other collection systems, I don't think they care. I think there's no good coercion rule that allows us to choose either counter or gauge to resolve it automatically.

For labelsets, I think they can be whatever they want, so there could be two metrics with unique, nonintersecting labels that happily sit together with the same name.

[dannyk81]

We were hitting a similar issue here -> warmans/prometheus-aggregate-exporter#1

Maybe this can help.

[SuperQ]

One thing that is now possible, is that the Prometheus client library supports custom registries. This should make it possible to switch the whole Prometheus endpoint in mtail to the official library. This would also be nice because it exposes some Go internal metrics for monitoring the health of a running mtail instance.

[jaqx0r]

Neat! I do want to get away from expvar because it is ironic that it's the least accessible metric in mtail. Thanks for the tip.

…

On Thu., 13 Jul. 2017, 11:46 Ben Kochie, ***@***.***> wrote: One thing that is now possible, is that the Prometheus client library supports custom registries. This should make it possible to switch the whole Prometheus endpoint in mtail to the official library. This would also be nice because it exposes some Go internal metrics for monitoring the health of a running mtail instance. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC5b-xNcGk9UJBeh8edLRpOyfxSsqUo7ks5sNfWXgaJpZM4NgTGi> .

[jaqx0r]

I'm not really happy with implementing this feature, because of the namespacing issue (and that there's a workaround for Prometheus in #59 (comment)) but I've gone and done it anyway. Let's see what technical debt this creates :)