[BUG]: Trace propagation errors in conjunction with nginx-datadog

[kelvin-acosta]

Tracer Version(s)

2.8.0

Ruby Version(s)

ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

Relevent Library and Version(s)

nginx-datadog 1.5.0

Bug Report

I'm seeing a bug related to using datadog-ruby while having nginx as a proxy nginx-datadog. Below are some of error logs we are currently seeing from our ruby app coming from dd-trace-rb while extracting the trace data coming from the nginx-datadog integration. It's difficult to tell whether the issue is on the datadog-nginx side producing some non conformant x-datadog-tags or if the issue is coming from dd-trace-rb being unable to process those tags.

Reproduction Code

No response

Configuration Block

No response

Error Logs

Failed to extract x-datadog-tags: Datadog::Tracing::Distributed::DatadogTagsCodec::DecodingError Invalid key: _dd.p.tid at /app/vendor/bundle/ruby/3.2.0/gems/datadog-2.8.0/lib/datadog/tracing/distributed/datadog_tags_codec.rb:60:in `block (2 levels) in decode'

Error extracting distributed trace data. Cause: undefined method delete' for true:TrueClass Location: /app/vendor/bundle/ruby/3.2.0/gems/datadog-2.8.0/lib/datadog/tracing/distributed/datadog.rb:118:in extract_trace_id!'

Operating System

Linux 5.10.233-223.887.amzn2.x86_64

How does Datadog help you?

No response

[marcotc]

Hey @kelvin-acosta, I've seen this issue recently. When this happened, the HTTP header x-datadog-tags was sent twice in the same HTTP request, can you confirm if that's the case for you as well?

[kelvin-acosta]

Yeah after some talks with datadog support, they also determined that nginx is sending the HTTP header twice in the same HTTP request. We have not yet found the cause though. What was the cause in the issue you've seen recently, maybe that can point me in the right direction? Thanks