test: e2e testing for OpenTelemetry Tracing

[ParkMyCar]

Motivation

Fixes MaterializeInc/database-issues#4896

This PR adds an e2e test for OpenTelemetry Tracing, based off of the initial attempt in 2166011.

Specifically it does the following:

• Adds a with_enable_tracing option to the testing Config builder in environmentd
• If tracing is enabled for a test, create a TracingHandle, which is then used by environmentd when running
• Adds a test test_emit_tracing_notice, which asserts that after running a SELECT statement, a NOTICE is emitted, that contains a trace_id

Note: The issue with the original attempt in 2166011 is it immediately dropped the TracingGuard. What's tricky is at first glance it seems like a TracingGuard only pertains to Sentry, but if you look at the Drop impl you'll see that when Drop-ed the TracingGuard also shuts down the OpenTelemetry subscriber, which is why in the initial attempt we didn't receive the NOTICE.

To prevent similar issues with TracingGuard in the future, I marked it as must_use. So if someone ignores it/drops it immediately, it'll throw a warning.

Tips for reviewer

To test this I ran it against a local PostgreSQL DB, and as such I had to comment out ALTER TABLE ... CONFIGURE ZONE in persist and stash. I don't think this should effect the test, but if it does, I'd be more than happy to run it against CockroachDB, if you could provide some instructions on how to setup a dev environment :)

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered.
• This PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way) and therefore is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• This PR includes the following user-facing behavior changes:
  • N/a

[CLAassistant]

All committers have signed the CLA.

[maddyblue]

This is so good, thanks!

[maddyblue]

Can this be None?

[ParkMyCar]

Are you referring to the opentelemetry field? That cannot be None because otherwise we wouldn't setup the opentelemetry tracing layer which is necessary for the test

[maddyblue]

I hit rebuild in buildkite. Some clippy warnings going on now.

[ParkMyCar]

I hit rebuild in buildkite. Some clippy warnings going on now.

Thanks! Fix those warnings, now locally running cargo clippy --tests passes :)

[guswynn]

@ParkMyCar nice find on TracingGuard needing to be held, I missed that when I tried this!

[ParkMyCar]

Thanks @guswynn :)

[benesch]

Thanks very much for pushing this over the finish line! 🎉

[benesch]

Tiniest little nit, for future reference: the leading underscore is meant to silence the unused field checker! If this is pub it doesn't need the leading underscore, because Rust is smart enough to realize that some downstream crate might use it. (But probably this shouldn't be pub?)

[ParkMyCar]

Great catch! Definitely doesn't need to be pub, let me fix that.

It also feels like this should be a Clippy lint, so I put up rust-lang/rust-clippy#10283, we'll see what the team thinks :)

[ParkMyCar]

Since I know you're interested :) that Clippy lint finally got merged!

[benesch]

To test this I ran it against a local PostgreSQL DB, and as such I had to comment out ALTER TABLE ... CONFIGURE ZONE in persist and stash. I don't think this should effect the test, but if it does, I'd be more than happy to run it against CockroachDB, if you could provide some instructions on how to setup a dev environment :)

Did anyone point you at our guide for this? https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide.md#cockroachdb

[ParkMyCar]

Did anyone point you at our guide for this? https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide.md#cockroachdb

Ah ha! I hadn't seen that yet, thanks for linking it