[BUG]: Faraday spans pulled before active span

[HoneyryderChuck]

Tracer Version(s)

2.9.0

Ruby Version(s)

ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [x86_64-linux]

Relevent Library and Version(s)

faraday 2.12.2, httpx 1.4.0 (or gh-70 branch)

Bug Report

I got a bug report related to the datadog integration for httpx, which I've narrowed down to the faraday datadog integration (where faraday uses httpx as a backend, and httpx integration is itself turned off). In the report, the reporter shows a screenshot of a request span not propagating to the next service, which also integrates with datadog and has distributed tracing turned on.

While reproducing the issue, as per one of my last messages in the thread, I was able to spot some different behaviour in the faraday datadog integration (compared to the httpx integration) where, when under an active trace, the faraday span appears before the top span (see linked tests).

Something tells me that this is all related, and a bug perhaps in the faraday integration, as the span should have been appended instead, and this lack of ordering is what may cause the distributed tracing not to work in this case.

Any help appreciated.

Reproduction Code

No response

Configuration Block

No response

Error Logs

No response

Operating System

No response

How does Datadog help you?

No response

[marcotc]

Thank you for the report @HoneyryderChuck!

To bring all the information here, these two tests fail:
https://gitlab.com/os85/httpx/-/merge_requests/364/diffs#c2506ec54b7385f0f8975e9adee8b092d5cdba9d
https://gitlab.com/os85/httpx/-/merge_requests/364/diffs#c2506ec54b7385f0f8975e9adee8b092d5cdba9d_0_108

with the output:

  1) Failure:
FaradayDatadogTest#test_faraday_datadog_distributed_headers_sampling_priority [integration_tests/datadog_helpers.rb:10]:
Expected false to be truthy.
  2) Failure:
FaradayDatadogTest#test_faraday_datadog_distributed_headers_disabled [integration_tests/datadog_helpers.rb:10]:
Expected false to be truthy.

The failed assertion is: https://gitlab.com/os85/httpx/-/merge_requests/364/diffs#c553c19ac82d8947396b4c6604d3eb53a5306755_0_10

And this is the Faraday + httpx setup: https://gitlab.com/os85/httpx/-/merge_requests/364/diffs#e143de6e3e6362dc545300af801eb66c707bfcbe_0_11

One question I have about the setup @HoneyryderChuck: is c.tracing.instrument :library being called for both :faraday and :httpx in the failing Faraday integration test? I'm not sure I can find two calls to it: https://gitlab.com/os85/httpx/-/merge_requests/364/diffs#c553c19ac82d8947396b4c6604d3eb53a5306755_0_77

[HoneyryderChuck]

To bring all the information here, these two tests fail:

That's correct @marcotc ! FWIW you don't need to setup the docker based test env, you can just clone the repo, bundle install, and run each of those individually with bundle exec ruby -Itest integration_tests/faraday_datadog_test.rb --name /name_of_test/ .

is c.tracing.instrument :library being called for both :faraday and :httpx in the failing Faraday integration test?

No. In the integration_tests folder you'll find two test files: datadog_test.rb, which tests plain httpx with the datadog tracing adapter :httpx enabled; and faraday_datadog_test.rb, which tests faraday-using-httpx, with the datadog tracing adapter :faraday enabled.

LMK if you need more help 🙏