[chain] Add OTEL tracing functionality + Jaeger

[zivkovicmilos]

Description

This task concerns setting up otel tracing functionality (off by default) for the gnoland chain binary.
We should have tracing in:

• VM execution flows
• Transaction processing workflows
• JSON-RPC layer
• Mempool processing layer
• ...

We should set up a workflow that gathers tracing metrics on each master push, during a supernova run, or during the execution of the testing suite.

Additionally, we should be able to utilize Jaeger (suggestions welcome) for viewing the tracing results.

cc @ajnavarro @mvertes

[mazzy89]

So excited to see observability to the next levels.

[sw360cab]

OTEL and Jaeger appear overlapping to me.

Basically we would use Jaeger only for viewing traces?

[ajnavarro]

@sw360cab AFAIK jaeger is only the visualization tool right now, instead of being the entire framework for tracing. The client for tracing is OTEL now.

[sw360cab]

Part of #2618

[sw360cab]

Blocked by @gfanton working on pure tracing in Gno.land

[hthieu1110]

FYI: I'm integrating with Tempo instead of Jaeger, wdyt ? (it's more lightweight and simpler than Jaeger if we don't need strictly specific functionalities from Jaeger) @zivkovicmilos

[hthieu1110]

@zivkovicmilos I've setup the tracing system using otel/tempo and configure a grafana dashboard for tracing.
Now I'm trying to integrate the flows mentioned above.

VM execution flows
Transaction processing workflows
JSON-RPC layer
Mempool processing layer

Could you please clarify a bit more about what should we trace for each of those flows ? thanks

[sw360cab]

FYI: I'm integrating with Tempo instead of Jaeger, wdyt ? (it's more lightweight and simpler than Jaeger if we don't need strictly specific functionalities from Jaeger) @zivkovicmilos

Due to the fact that they should be both (Jaeger and Grafana Tempo) aligned with OTEL, I don't have a preference.
The OPS part in the monorepo is just a POC. What we are trying to demonstrate is that our codebase is aligned with OTEL tracing functionality.

The way anyone will use the feature is not tied to any other technology. In this POC I would rather show how much flexible OTEL could be, so that Tempo and Jaeger could be interchangeable.

[hthieu1110]

@sw360cab I see. Thanks :)

[hthieu1110]

@sw360cab @zivkovicmilos I've integrated OTEL tracing for Consensus flow in #3815

Could you pls take a look to see if it's ok that way ? Thanks.