Fix APM Disablement V1 (ASM Standalone) #4479

vpellan · 2025-03-11T15:59:03Z

What does this PR do?

This PR fixes multiple bugs on APM Disablement V1 (aka ASM Standalone) when it was activated:

When HTTP frameworks (Net::HTTP, RestClient, Excon, Faraday...) were instrumented, the sampling_priority was set to MANUAL_REJECT if tags contained _dd.p.appsec (which was used to check for upstream propagated appsec events), then not sampled later due to a change in the sampler in this PR This was fixed by making this check during the building step of a trace from a digest (which also means that it now works even if an HTTP framework is not instrumented)
Distributed tracing was skipped only for Net::HTTP (in case of upstream or local security event). It now skips it on every HTTP frameworks supported, but also Sidekiq and GRPC.
The APM Tracing disabled tag (_dd.apm.enabled=0) was not added when appsec was disabled. It is now fixed and the code related to it was moved to Tracing
Added tests that actually tests APM disablement heartbeat traces (ASM Standalone too) and cleaned related tests

Motivation:

When I tried to activate SCA Standalone system-tests, I discovered that disabling AppSec and disabling APM Tracing would result in heartbeat traces not sending dd.apm.enabled=0 tag. During my investigation, I discovered more bugs (listed above).
This is also a step towards APM Disablement V2. For now, the variables are still called after appsec standalone, but this will change to APM disablement in the future and lose all reference to asm standalone, as it is in fact an APM feature.

Change log entry

None.

Additional Notes:

This PR should be followed soon by APM Disablement V2, which will replace the env var by DD_APM_TRACING_ENABLED, and every config/variable referencing asm standalone to apm disablement.

I understand that there might be some concerns about AppSec stuff being in APM territory but this is mostly a matter of naming that will be fixed when APM Disablement V2 will be released. In the V2, I will replace any instance of Datadog.configuration.appsec.standalone.enabled by Datadog.configuration.tracing.apm.disabled, Datadog::AppSec::Ext::TAG_DISTRIBUTED_APPSEC_EVENT by something like Datadog::Tracing::Ext::TAG_DISTRIBUTED_TRACE_SOURCE (_dd.p.ts, which will be a bitmask). The only exception might be in the propagate_sampling_priority? method, in tracer.rb, where we check if AppSec is activated.

How to test the change?

CI and system-tests

…frameworks

…t the rate limiter)

…no effect on them)

spec/datadog/tracing/contrib/rack/integration_test_spec.rb

codecov-commenter · 2025-03-11T16:14:40Z

Codecov Report

Attention: Patch coverage is 99.71751% with 1 line in your changes missing coverage. Please review.

Project coverage is 97.69%. Comparing base (2f427f8) to head (def4da4).
Report is 65 commits behind head on master.

Files with missing lines	Patch %	Lines
lib/datadog/tracing/distributed/circuit_breaker.rb	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4479      +/-   ##
==========================================
- Coverage   97.70%   97.69%   -0.01%     
==========================================
  Files        1375     1382       +7     
  Lines       83812    83987     +175     
  Branches     4251     4249       -2     
==========================================
+ Hits        81887    82054     +167     
- Misses       1925     1933       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

datadog-datadog-prod-us1 · 2025-03-11T16:14:50Z

Datadog Report

Branch report: vpellan/fix-sca-standalone
Commit report: def4da4
Test service: dd-trace-rb

✅ 0 Failed, 20565 Passed, 1370 Skipped, 3m 18.38s Total Time

pr-commenter · 2025-03-11T17:08:59Z

Benchmarks

Benchmark execution time: 2025-03-12 14:46:19

Comparing candidate commit def4da4 in PR branch vpellan/fix-sca-standalone with baseline commit 2f427f8 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 31 metrics, 2 unstable metrics.

TonyCTHsu

FYI: Currently, there is a karafka integration under review which contains distributed tracing mechanism

TonyCTHsu · 2025-03-12T09:05:45Z

lib/datadog/tracing/contrib/ethon/easy_patch.rb

+
+            # Skips distributed tracing if disabled for this instrumentation
+            # or if APM is disabled unless there is an AppSec event (from upstream distributed trace or local)
+            def should_skip_distributed_tracing?(trace)


I feel like I am seeing this piece of logic repeatedly several times, perhaps it make sense to encapsulate this from an AppSec module and be reused at multiple places?

The method can be stateless or depends on configuration and the contract is basically returning a boolean.

Thanks, I will investigate that, although I don't think it should be in an AppSec module, as in the V2, this will not contain any reference to AppSec (appsec.standalone will be replaced by tracing.apm and APPSEC_EVENT will be replaced by TRACE_SOURCE)

Fixed in 91f2f68

vpellan · 2025-03-12T09:29:40Z

FYI: Currently, there is a karafka integration under review which contains distributed tracing mechanism

What is the process for this kind of situation ? If I merge to master first, should I myself add a commit on the Karafka integration to add APM Disablement capabilities ? (Of course if the other way around happens I could just rebase and add it here)

TonyCTHsu · 2025-03-12T10:01:35Z

What is the process for this kind of situation ?

I would suggest not to wait on it. However, it would be much welcomed to encapsulate the logic so that the pattern can be adopted easily.

p-datadog · 2025-03-12T14:28:35Z

lib/datadog/tracing/distributed/circuit_breaker.rb

+        # Skips distributed tracing if disabled for this instrumentation
+        # or if APM is disabled unless there is an AppSec event (from upstream distributed trace or local)
+        def should_skip_distributed_tracing?(client_config: nil, datadog_config: nil, trace: nil)
+          if ::Datadog.configuration.appsec.standalone.enabled &&


If the method takes configuration as arguments, why is this line reading global config?

The method would be much more understandable if the types of parameters were listed.

p-datadog · 2025-03-12T14:30:08Z

lib/datadog/tracing/tracer.rb

@@ -545,10 +550,30 @@ def skip_trace(name)
        end
      end

+      # Decide whether upstream sampling priority should be propagated, by taking into account
+      # the upstream tags and the configuration.
+      # We should always propagate if there APM is not disabled.


"there APM" is a typo or broken grammar?

p-datadog · 2025-03-12T14:33:12Z

lib/datadog/tracing/tracer.rb

+      def propagate_sampling_priority?(upstream_tags:)
+        return true unless apm_tracing_disabled
+
+        if upstream_tags&.key?(Datadog::AppSec::Ext::TAG_DISTRIBUTED_APPSEC_EVENT)


This method is hard to understand not knowing the type of upstream_tags.

vpellan added 18 commits March 5, 2025 17:51

Fix apm disablement tag when appsec is disabled

4046576

Add method to skip upstream sampling priority

8ed08f3

Add APM disablement integration tests

ac8c014

Fix propagate_sampling_priority with no upstream_tags

411d428

Fix asm standalone integration tests

5d55d8a

Remove appsec_standalone_reject method

e491c6f

Skip distributed tracing if APM is disabled on all HTTP/GRPC/Sidekiq …

0b261d2

…frameworks

Delete unnecessary default sample rate (as we match all traces agains…

f8acd8f

…t the rate limiter)

Add test to check heartbeat traces

360761a

Fix Datadog-Computed-Client-Stats tests

09646d1

Simplify Datadog-Client-Computed-Stats test

6e82099

Add separate method to generate APM disablement post_sampler

5c73ff8

Cleaned appsec standalone tests for rack

0389f6b

Delete ASM Standalone tests on Rails and Sinatra (ASM Standalone has …

3ec6ff9

…no effect on them)

Remove forced-tests from git conflict resolution

6c863cd

Rename build_above_second_post_sampler to build_rate_limit_post_sampler

ce8a363

Move before block in context

3df2d61

Rubocop

c2ff1b6

vpellan requested review from a team as code owners March 11, 2025 15:59

vpellan self-assigned this Mar 11, 2025

datadog-datadog-prod-us1 bot reviewed Mar 11, 2025

View reviewed changes

spec/datadog/tracing/contrib/rack/integration_test_spec.rb Show resolved Hide resolved

github-actions bot added integrations Involves tracing integrations appsec Application Security monitoring product tracing labels Mar 11, 2025

vpellan added this to the 2.13.0 milestone Mar 11, 2025

Changed system-tests scenarios to Experimental mode

a9473aa

TonyCTHsu reviewed Mar 12, 2025

View reviewed changes

vpellan added 2 commits March 12, 2025 13:32

Add should_skip_distributed_tracing? method

91f2f68

Fix circuit breaker spec

def4da4

p-datadog reviewed Mar 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix APM Disablement V1 (ASM Standalone) #4479

Fix APM Disablement V1 (ASM Standalone) #4479

vpellan commented Mar 11, 2025 •

edited

Loading

codecov-commenter commented Mar 11, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Mar 11, 2025 •

edited

Loading

pr-commenter bot commented Mar 11, 2025 •

edited

Loading

TonyCTHsu left a comment

TonyCTHsu Mar 12, 2025

vpellan Mar 12, 2025

vpellan Mar 12, 2025

vpellan commented Mar 12, 2025

TonyCTHsu commented Mar 12, 2025

p-datadog Mar 12, 2025

p-datadog Mar 12, 2025

p-datadog Mar 12, 2025

Fix APM Disablement V1 (ASM Standalone) #4479

Are you sure you want to change the base?

Fix APM Disablement V1 (ASM Standalone) #4479

Conversation

vpellan commented Mar 11, 2025 • edited Loading

codecov-commenter commented Mar 11, 2025 • edited Loading

Codecov Report

datadog-datadog-prod-us1 bot commented Mar 11, 2025 • edited Loading

Datadog Report

pr-commenter bot commented Mar 11, 2025 • edited Loading

Benchmarks

TonyCTHsu left a comment

Choose a reason for hiding this comment

TonyCTHsu Mar 12, 2025

Choose a reason for hiding this comment

vpellan Mar 12, 2025

Choose a reason for hiding this comment

vpellan Mar 12, 2025

Choose a reason for hiding this comment

vpellan commented Mar 12, 2025

TonyCTHsu commented Mar 12, 2025

p-datadog Mar 12, 2025

Choose a reason for hiding this comment

p-datadog Mar 12, 2025

Choose a reason for hiding this comment

p-datadog Mar 12, 2025

Choose a reason for hiding this comment

vpellan commented Mar 11, 2025 •

edited

Loading

codecov-commenter commented Mar 11, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Mar 11, 2025 •

edited

Loading

pr-commenter bot commented Mar 11, 2025 •

edited

Loading