Expand number of files that require thorough testing #6504

andrewlock · 2025-01-06T16:28:06Z

Summary of changes

Try to catch more integration issues in PRs

Reason for change

We don't run all TFMs on all PRs, because it increases the load on CI, and doesn't add a huge amount of value. Unfortunately, that means we also occasionally merge PRs that break an untested-in-PR TFM which then fails on master. We attempt to mitigate that, but running all TFMs for "high risk of breakage" PRs. This increases the scope of what constitutes "high risk"

Implementation details

Previously, we were only counting PRs which change the calltarget or callsite integration points. This PR updates that to include the implementations too

Test coverage

Tested locally to confirm it works as expected

e-n-0

Thanks 😄

datadog-ddstaging · 2025-01-06T17:09:02Z

Datadog Report

Branch report: andrew/ci/include-more-paths-in-checks
Commit report: 546a542
Test service: dd-trace-dotnet

✅ 0 Failed, 235712 Passed, 2104 Skipped, 18h 53m 35.75s Total Time

andrewlock · 2025-01-06T17:14:30Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6504) - mean (69ms)  : 65, 73
     .   : milestone, 69,
    master - mean (68ms)  : 65, 72
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6504) - mean (977ms)  : 949, 1005
     .   : milestone, 977,
    master - mean (971ms)  : 949, 993
     .   : milestone, 971,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6504) - mean (107ms)  : 105, 110
     .   : milestone, 107,
    master - mean (107ms)  : 104, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6504) - mean (674ms)  : 660, 689
     .   : milestone, 674,
    master - mean (674ms)  : 658, 690
     .   : milestone, 674,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6504) - mean (91ms)  : 89, 92
     .   : milestone, 91,
    master - mean (90ms)  : 88, 92
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (6504) - mean (632ms)  : 616, 648
     .   : milestone, 632,
    master - mean (631ms)  : 615, 648
     .   : milestone, 631,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6504) - mean (194ms)  : 190, 198
     .   : milestone, 194,
    master - mean (194ms)  : 189, 198
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6504) - mean (1,099ms)  : 1072, 1125
     .   : milestone, 1099,
    master - mean (1,102ms)  : 1065, 1139
     .   : milestone, 1102,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6504) - mean (279ms)  : 274, 284
     .   : milestone, 279,
    master - mean (279ms)  : 275, 283
     .   : milestone, 279,

    section CallTarget+Inlining+NGEN
    This PR (6504) - mean (871ms)  : 842, 901
     .   : milestone, 871,
    master - mean (870ms)  : 844, 896
     .   : milestone, 870,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6504) - mean (268ms)  : 263, 272
     .   : milestone, 268,
    master - mean (268ms)  : 263, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6504) - mean (853ms)  : 823, 882
     .   : milestone, 853,
    master - mean (855ms)  : 820, 890
     .   : milestone, 855,

andrewlock · 2025-01-06T18:29:56Z

Benchmarks Report for tracer 🐌

Benchmarks for #6504 compared to master:

1 benchmarks are faster, with geometric mean 1.117
2 benchmarks are slower, with geometric mean 1.123
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.99μs	44.6ns	303ns	0.016	0.00399	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	55.8ns	325ns	0.0152	0.00507	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	68.2ns	264ns	1.04	0.307	0.0914	6.2 KB
#6504	`StartStopWithChild`	net6.0	8.03μs	45.6ns	319ns	0.0162	0.00809	0	5.61 KB
#6504	`StartStopWithChild`	netcoreapp3.1	10.1μs	56.5ns	375ns	0.0151	0.00503	0	5.8 KB
#6504	`StartStopWithChild`	net472	16.5μs	61ns	236ns	1.03	0.297	0.0988	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	503μs	624ns	2.41μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	660μs	515ns	2μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	858μs	757ns	2.93μs	0.431	3.3 KB
#6504	`WriteAndFlushEnrichedTraces`	net6.0	489μs	282ns	1.02μs	0	2.7 KB
#6504	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	670μs	486ns	1.88μs	0	2.7 KB
#6504	`WriteAndFlushEnrichedTraces`	net472	856μs	362ns	1.36μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	154μs	905ns	8.24μs	0.145	14.47 KB
master	`SendRequest`	netcoreapp3.1	177μs	1.02μs	7.96μs	0.163	17.27 KB
master	`SendRequest`	net472	0.00113ns	0.000522ns	0.00202ns	0	0 b
#6504	`SendRequest`	net6.0	157μs	1.05μs	10.5μs	0.144	14.47 KB
#6504	`SendRequest`	netcoreapp3.1	175μs	1.03μs	9.83μs	0.171	17.27 KB
#6504	`SendRequest`	net472	0.00172ns	0.000832ns	0.00311ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	571μs	3.23μs	24.4μs	0.543	0	0	41.6 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	662μs	3.78μs	27μs	0.391	0	0	41.71 KB
master	`WriteAndFlushEnrichedTraces`	net472	852μs	3.98μs	15.4μs	8.39	2.52	0.419	53.33 KB
#6504	`WriteAndFlushEnrichedTraces`	net6.0	578μs	3.05μs	15.2μs	0.59	0	0	41.72 KB
#6504	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	670μs	3.51μs	17.9μs	0.347	0	0	41.76 KB
#6504	`WriteAndFlushEnrichedTraces`	net472	834μs	4.27μs	20μs	8.08	2.55	0.425	53.27 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.2μs	1.4ns	5.43ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	1.29ns	4.99ns	0.0132	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.1μs	3.38ns	13.1ns	0.157	0.00105	987 B
#6504	`ExecuteNonQuery`	net6.0	1.33μs	1.84ns	7.12ns	0.0141	0	1.02 KB
#6504	`ExecuteNonQuery`	netcoreapp3.1	1.83μs	1.55ns	5.99ns	0.0136	0	1.02 KB
#6504	`ExecuteNonQuery`	net472	2.12μs	2.05ns	7.96ns	0.157	0.00104	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.22μs	1.79ns	6.92ns	0.014	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.5μs	2.37ns	8.21ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.59μs	2.4ns	9.29ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.35μs	0.501ns	1.81ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	1.03ns	3.97ns	0.014	1.02 KB
master	`CallElasticsearchAsync`	net472	2.57μs	2.36ns	9.13ns	0.167	1.05 KB
#6504	`CallElasticsearch`	net6.0	1.16μs	0.637ns	2.38ns	0.0135	976 B
#6504	`CallElasticsearch`	netcoreapp3.1	1.6μs	4.47ns	17.3ns	0.0128	976 B
#6504	`CallElasticsearch`	net472	2.58μs	2.63ns	10.2ns	0.157	995 B
#6504	`CallElasticsearchAsync`	net6.0	1.4μs	0.846ns	3.28ns	0.013	952 B
#6504	`CallElasticsearchAsync`	netcoreapp3.1	1.62μs	0.647ns	2.33ns	0.0138	1.02 KB
#6504	`CallElasticsearchAsync`	net472	2.45μs	1.61ns	6.02ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.37μs	0.772ns	2.89ns	0.013	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	1.19ns	4.44ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.77μs	0.331ns	1.24ns	0.145	915 B
#6504	`ExecuteAsync`	net6.0	1.27μs	0.657ns	2.46ns	0.0134	952 B
#6504	`ExecuteAsync`	netcoreapp3.1	1.65μs	0.744ns	2.88ns	0.0132	952 B
#6504	`ExecuteAsync`	net472	1.85μs	0.549ns	2.13ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.52μs	1.57ns	5.88ns	0.0316	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.2μs	2.32ns	9ns	0.0388	2.85 KB
master	`SendAsync`	net472	7.47μs	1.93ns	7.47ns	0.496	3.12 KB
#6504	`SendAsync`	net6.0	4.6μs	1.34ns	4.82ns	0.0322	2.31 KB
#6504	`SendAsync`	netcoreapp3.1	5.43μs	2.49ns	9.64ns	0.0383	2.85 KB
#6504	`SendAsync`	net472	7.41μs	7.8ns	28.1ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.47μs	0.695ns	2.6ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.23μs	1.01ns	3.78ns	0.0221	1.64 KB
master	`EnrichedLog`	net472	2.82μs	1.59ns	5.93ns	0.249	1.57 KB
#6504	`EnrichedLog`	net6.0	1.47μs	0.862ns	3.22ns	0.0235	1.64 KB
#6504	`EnrichedLog`	netcoreapp3.1	2.19μs	2.34ns	8.43ns	0.0219	1.64 KB
#6504	`EnrichedLog`	net472	2.76μs	1.93ns	7.46ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	117μs	110ns	425ns	0.0582	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	250ns	968ns	0	0	4.28 KB
master	`EnrichedLog`	net472	152μs	106ns	398ns	0.683	0.228	4.46 KB
#6504	`EnrichedLog`	net6.0	117μs	135ns	523ns	0.0589	0	4.28 KB
#6504	`EnrichedLog`	netcoreapp3.1	121μs	104ns	404ns	0	0	4.28 KB
#6504	`EnrichedLog`	net472	150μs	129ns	499ns	0.677	0.226	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.04μs	1.18ns	4.4ns	0.0299	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.26μs	2.26ns	8.77ns	0.0298	2.2 KB
master	`EnrichedLog`	net472	4.9μs	2.9ns	10.8ns	0.319	2.02 KB
#6504	`EnrichedLog`	net6.0	2.79μs	0.915ns	3.3ns	0.0308	2.2 KB
#6504	`EnrichedLog`	netcoreapp3.1	4.23μs	1.09ns	4.08ns	0.0296	2.2 KB
#6504	`EnrichedLog`	net472	4.87μs	0.766ns	2.97ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.31μs	0.763ns	2.86ns	0.0164	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.82μs	0.67ns	2.42ns	0.0154	1.14 KB
master	`SendReceive`	net472	2.1μs	1.22ns	4.71ns	0.183	1.16 KB
#6504	`SendReceive`	net6.0	1.33μs	0.566ns	2.19ns	0.0159	1.14 KB
#6504	`SendReceive`	netcoreapp3.1	1.73μs	1.52ns	5.9ns	0.0156	1.14 KB
#6504	`SendReceive`	net472	2.13μs	1.56ns	5.84ns	0.184	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.72μs	0.802ns	3ns	0.0218	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.98μs	1.3ns	5.05ns	0.0217	1.65 KB
master	`EnrichedLog`	net472	4.39μs	3.91ns	15.1ns	0.322	2.04 KB
#6504	`EnrichedLog`	net6.0	2.66μs	1.26ns	4.9ns	0.0227	1.6 KB
#6504	`EnrichedLog`	netcoreapp3.1	3.84μs	1.68ns	6.5ns	0.0229	1.65 KB
#6504	`EnrichedLog`	net472	4.25μs	4.35ns	16.9ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6504

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.125	561.99	631.99

Faster 🎉 in #6504

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.117	901.61	807.45

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	409ns	0.257ns	0.995ns	0.00802	576 B
master	`StartFinishSpan`	netcoreapp3.1	564ns	1.45ns	5.61ns	0.00768	576 B
master	`StartFinishSpan`	net472	636ns	0.721ns	2.79ns	0.0917	578 B
master	`StartFinishScope`	net6.0	549ns	0.265ns	0.993ns	0.00987	696 B
master	`StartFinishScope`	netcoreapp3.1	681ns	0.888ns	3.44ns	0.00929	696 B
master	`StartFinishScope`	net472	901ns	1.03ns	3.97ns	0.104	658 B
#6504	`StartFinishSpan`	net6.0	399ns	0.215ns	0.831ns	0.00801	576 B
#6504	`StartFinishSpan`	netcoreapp3.1	632ns	0.27ns	1.01ns	0.00763	576 B
#6504	`StartFinishSpan`	net472	660ns	0.468ns	1.81ns	0.0918	578 B
#6504	`StartFinishScope`	net6.0	524ns	0.182ns	0.705ns	0.00972	696 B
#6504	`StartFinishScope`	netcoreapp3.1	676ns	0.576ns	2.23ns	0.00951	696 B
#6504	`StartFinishScope`	net472	806ns	0.853ns	3.3ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6504

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.122	656.09	736.07

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	656ns	0.399ns	1.54ns	0.00976	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	932ns	0.652ns	2.53ns	0.00932	696 B
master	`RunOnMethodBegin`	net472	1.11μs	0.355ns	1.33ns	0.104	658 B
#6504	`RunOnMethodBegin`	net6.0	737ns	0.403ns	1.56ns	0.00956	696 B
#6504	`RunOnMethodBegin`	netcoreapp3.1	943ns	0.995ns	3.85ns	0.00961	696 B
#6504	`RunOnMethodBegin`	net472	1.08μs	0.44ns	1.59ns	0.104	658 B

bouwkast

Nice 👍

expand number of files that require thorough testing

546a542

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 6, 2025

andrewlock requested review from a team as code owners January 6, 2025 16:28

andrewlock force-pushed the andrew/ci/include-more-paths-in-checks branch from 91b4e6b to 546a542 Compare January 6, 2025 16:40

e-n-0 approved these changes Jan 6, 2025

View reviewed changes

bouwkast approved these changes Jan 7, 2025

View reviewed changes

andrewlock merged commit b34e73c into master Jan 7, 2025
111 of 114 checks passed

andrewlock deleted the andrew/ci/include-more-paths-in-checks branch January 7, 2025 13:28

github-actions bot added this to the vNext-v3 milestone Jan 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expand number of files that require thorough testing #6504

Expand number of files that require thorough testing #6504

andrewlock commented Jan 6, 2025 •

edited

Loading

e-n-0 left a comment

datadog-ddstaging bot commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025

andrewlock commented Jan 6, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6504

Faster 🎉 in #6504

Raw results

Slower ⚠️ in #6504

Raw results

bouwkast left a comment

Expand number of files that require thorough testing #6504

Expand number of files that require thorough testing #6504

Conversation

andrewlock commented Jan 6, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

e-n-0 left a comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Jan 6, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 6, 2025

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 6, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6504

Faster 🎉 in #6504

Raw results

Slower ⚠️ in #6504

Raw results

bouwkast left a comment

Choose a reason for hiding this comment

andrewlock commented Jan 6, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 6, 2025 •

edited

Loading