Endpoint called, but "Failed to scrape Prometheus endpoint"

[lee-campbell]

The Prometheus sidecar is calling the /metrics endpoint on my service, as demonstrated by the logs screenshot, below, but I'm still seeing "Failed to scrape Prometheus endpoint".

The lack of stack trace makes it difficult to provide any additional information.

The service I'm running is MoleculerJS, which produces a lot of metric output, but here is an excerpt:

# HELP process_arguments Process arguments
# TYPE process_arguments gauge
process_arguments{namespace="hub",nodeID="HUB-5c8dbbcd-d8c5-4fdc-888c-f96324e3d3cf",index="0",value="/usr/local/bin/node"} 1
process_arguments{namespace="hub",nodeID="HUB-5c8dbbcd-d8c5-4fdc-888c-f96324e3d3cf",index="1",value="/app/node_modules/.bin/moleculer-runner"} 1
process_arguments{namespace="hub",nodeID="HUB-5c8dbbcd-d8c5-4fdc-888c-f96324e3d3cf",index="2",value="-c"} 1
process_arguments{namespace="hub",nodeID="HUB-5c8dbbcd-d8c5-4fdc-888c-f96324e3d3cf",index="3",value="/app/dist/moleculer.config.js"} 1
process_arguments{namespace="hub",nodeID="HUB-5c8dbbcd-d8c5-4fdc-888c-f96324e3d3cf",index="4",value="/app/dist/services/*.js"} 1

# HELP process_pid Process PID
# TYPE process_pid gauge
process_pid{namespace="hub",nodeID="HUB-5c8dbbcd-d8c5-4fdc-888c-f96324e3d3cf",value="1"} 1

# HELP process_ppid Process parent PID
# TYPE process_ppid gauge
process_ppid{namespace="hub",nodeID="HUB-5c8dbbcd-d8c5-4fdc-888c-f96324e3d3cf",value="0"} 1

# HELP process_memory_heap_size_total Process heap size
# TYPE process_memory_heap_size_total gauge
process_memory_heap_size_total{namespace="hub",nodeID="HUB-5c8dbbcd-d8c5-4fdc-888c-f96324e3d3cf"} 74477568

In the absence of any guidance on how best to proceed, could the sidecar be enhanced such that it includes more stack output, so that I can diagnose why the scrape failed?

[lee-campbell]

There are some un-parseable metric entries, as it turns out. Is it at all possible to bubble that error up or have it logged under some "verbose" setting?

[m0ps]

@lee-campbell
I have the same problem... how did you identify those "un-parseable metric entries"?

[lee-campbell]

I used this online service:

https://o11y.tools/metricslint/

[m0ps]

Thanks... so I found a bunch of: counter metrics should have "_total" suffix issues. Curious if it's could be a reason

[m0ps]

Ah..., I just realized that I am affected by this #38
I totally forgot that the public endpoint is protected with basic auth