[Test Optimization] Add support to xUnit v3 2.0.0 #6738

tonyredondo · 2025-03-04T12:00:25Z

Summary of changes

This PR adds support for xUnit v3 latest major release 2.0.0

Reason for change

A new major version of xUnit v3 has been released.

Implementation details

Bumped the integration to this major version.

Test coverage

New samples also bump the version to 2.0.0

Other details

datadog-datadog-prod-us1 · 2025-03-04T12:23:23Z

Datadog Report

All test runs a48452d 🔗

✅ 2 Total Test Services: 0 Failed, 2 Passed

Test Services

Service Name	Failed	Known Flaky	New Flaky	Passed	Skipped	Total Time	Test Service View
`dd-trace-dotnet`	0	0	0	563454	5415	47h 46m 29.18s	Link
`exploration_tests`	0	0	0	22085	3	1h 27m 44.8s	Link

andrewlock · 2025-03-04T12:50:06Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6738) - mean (76ms)  : 71, 82
     .   : milestone, 76,
    master - mean (70ms)  : 65, 74
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6738) - mean (1,043ms)  : 998, 1087
     .   : milestone, 1043,
    master - mean (1,003ms)  : 977, 1029
     .   : milestone, 1003,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6738) - mean (114ms)  : 106, 122
     .   : milestone, 114,
    master - mean (102ms)  : 100, 105
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6738) - mean (731ms)  : crit, 698, 763
     .   : crit, milestone, 731,
    master - mean (688ms)  : 664, 711
     .   : milestone, 688,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6738) - mean (100ms)  : 93, 107
     .   : milestone, 100,
    master - mean (89ms)  : 87, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6738) - mean (707ms)  : crit, 675, 739
     .   : crit, milestone, 707,
    master - mean (640ms)  : 626, 654
     .   : milestone, 640,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6738) - mean (191ms)  : 187, 195
     .   : milestone, 191,
    master - mean (191ms)  : 187, 196
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6738) - mean (1,114ms)  : 1084, 1144
     .   : milestone, 1114,
    master - mean (1,107ms)  : 1081, 1133
     .   : milestone, 1107,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6738) - mean (270ms)  : 265, 275
     .   : milestone, 270,
    master - mean (271ms)  : 266, 275
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (6738) - mean (875ms)  : 846, 903
     .   : milestone, 875,
    master - mean (877ms)  : 842, 913
     .   : milestone, 877,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6738) - mean (263ms)  : 259, 266
     .   : milestone, 263,
    master - mean (262ms)  : 259, 266
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (6738) - mean (859ms)  : 825, 894
     .   : milestone, 859,
    master - mean (855ms)  : 817, 892
     .   : milestone, 855,

bouwkast

Awesome thanks!

andrewlock · 2025-03-04T14:29:49Z

Benchmarks Report for tracer 🐌

Benchmarks for #6738 compared to master:

1 benchmarks are faster, with geometric mean 1.113
1 benchmarks are slower, with geometric mean 1.148
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.19μs	46.8ns	338ns	0.0119	0.00398	0	5.6 KB
master	`StartStopWithChild`	netcoreapp3.1	10.7μs	60.3ns	435ns	0.015	0.00499	0	5.81 KB
master	`StartStopWithChild`	net472	16.2μs	47.8ns	185ns	1.04	0.299	0.0969	6.2 KB
#6738	`StartStopWithChild`	net6.0	8.34μs	47.2ns	337ns	0.0241	0.012	0	5.61 KB
#6738	`StartStopWithChild`	netcoreapp3.1	10.3μs	59.1ns	426ns	0.0204	0.0102	0	5.8 KB
#6738	`StartStopWithChild`	net472	16.3μs	40.8ns	158ns	1.04	0.308	0.0869	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	483μs	358ns	1.39μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	644μs	635ns	2.46μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	849μs	397ns	1.54μs	0.422	3.3 KB
#6738	`WriteAndFlushEnrichedTraces`	net6.0	511μs	502ns	1.94μs	0	2.7 KB
#6738	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	638μs	304ns	1.18μs	0	2.7 KB
#6738	`WriteAndFlushEnrichedTraces`	net472	847μs	316ns	1.18μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	133μs	411ns	1.59μs	0.199	14.47 KB
master	`SendRequest`	netcoreapp3.1	151μs	212ns	794ns	0.227	17.27 KB
master	`SendRequest`	net472	0.000146ns	9.84E‑05ns	0.000368ns	0	0 b
#6738	`SendRequest`	net6.0	133μs	456ns	1.77μs	0.196	14.47 KB
#6738	`SendRequest`	netcoreapp3.1	146μs	683ns	2.56μs	0.217	17.27 KB
#6738	`SendRequest`	net472	0.000617ns	0.000245ns	0.000916ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	549μs	2.62μs	10.5μs	0.543	0	0	41.47 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	683μs	3.34μs	13.8μs	0.324	0	0	41.84 KB
master	`WriteAndFlushEnrichedTraces`	net472	878μs	3.73μs	13.9μs	8.19	2.59	0.431	53.32 KB
#6738	`WriteAndFlushEnrichedTraces`	net6.0	566μs	2.12μs	8.23μs	0.571	0	0	41.46 KB
#6738	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	673μs	3.33μs	15.6μs	0.331	0	0	41.88 KB
#6738	`WriteAndFlushEnrichedTraces`	net472	868μs	3.42μs	13.2μs	8.25	2.6	0.434	53.26 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.33μs	1.28ns	4.97ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.82μs	1.88ns	7.28ns	0.0133	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.12μs	1.29ns	4.81ns	0.156	0.00106	987 B
#6738	`ExecuteNonQuery`	net6.0	1.37μs	1.46ns	5.65ns	0.0144	0	1.02 KB
#6738	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	2ns	7.75ns	0.0135	0	1.02 KB
#6738	`ExecuteNonQuery`	net472	2.12μs	2.24ns	8.69ns	0.157	0.00105	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.26μs	0.7ns	2.62ns	0.0139	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.55μs	3.3ns	12.3ns	0.0131	976 B
master	`CallElasticsearch`	net472	2.47μs	0.758ns	2.73ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.24μs	1.08ns	4.18ns	0.0131	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	1.06ns	3.95ns	0.0131	1.02 KB
master	`CallElasticsearchAsync`	net472	2.64μs	1ns	3.62ns	0.166	1.05 KB
#6738	`CallElasticsearch`	net6.0	1.16μs	0.322ns	1.12ns	0.0141	976 B
#6738	`CallElasticsearch`	netcoreapp3.1	1.56μs	0.859ns	3.33ns	0.0134	976 B
#6738	`CallElasticsearch`	net472	2.5μs	1.87ns	7ns	0.158	995 B
#6738	`CallElasticsearchAsync`	net6.0	1.25μs	0.391ns	1.46ns	0.0135	952 B
#6738	`CallElasticsearchAsync`	netcoreapp3.1	1.76μs	0.752ns	2.81ns	0.014	1.02 KB
#6738	`CallElasticsearchAsync`	net472	2.56μs	1.46ns	5.67ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6738

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0	1.113	1,333.69	1,198.32

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.34μs	1.58ns	5.9ns	0.0133	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.69μs	1.57ns	6.08ns	0.0128	952 B
master	`ExecuteAsync`	net472	1.85μs	0.672ns	2.6ns	0.145	915 B
#6738	`ExecuteAsync`	net6.0	1.2μs	0.334ns	1.29ns	0.0133	952 B
#6738	`ExecuteAsync`	netcoreapp3.1	1.73μs	1.5ns	5.61ns	0.0129	952 B
#6738	`ExecuteAsync`	net472	1.82μs	0.821ns	3.18ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.36μs	1.04ns	3.75ns	0.0325	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.22μs	2.2ns	7.94ns	0.0393	2.85 KB
master	`SendAsync`	net472	7.45μs	2.36ns	9.15ns	0.492	3.12 KB
#6738	`SendAsync`	net6.0	4.29μs	1.35ns	5.06ns	0.0321	2.31 KB
#6738	`SendAsync`	netcoreapp3.1	5.25μs	2.8ns	10.8ns	0.037	2.85 KB
#6738	`SendAsync`	net472	7.44μs	1.85ns	7.16ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.47μs	0.781ns	3.03ns	0.0234	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.14μs	5.9ns	22.9ns	0.0223	1.64 KB
master	`EnrichedLog`	net472	2.63μs	2.01ns	7.8ns	0.25	1.57 KB
#6738	`EnrichedLog`	net6.0	1.49μs	0.554ns	2.15ns	0.0231	1.64 KB
#6738	`EnrichedLog`	netcoreapp3.1	2.19μs	0.714ns	2.67ns	0.0219	1.64 KB
#6738	`EnrichedLog`	net472	2.56μs	0.888ns	3.44ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	158ns	612ns	0.0563	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	179ns	692ns	0	0	4.28 KB
master	`EnrichedLog`	net472	149μs	211ns	817ns	0.668	0.223	4.46 KB
#6738	`EnrichedLog`	net6.0	112μs	190ns	735ns	0.0559	0	4.28 KB
#6738	`EnrichedLog`	netcoreapp3.1	116μs	216ns	836ns	0	0	4.28 KB
#6738	`EnrichedLog`	net472	149μs	210ns	814ns	0.668	0.223	4.46 KB

Benchmarks.Trace.NLogBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6738

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.NLogBenchmark.EnrichedLog‑net6.0	1.148	2,903.61	3,332.58

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.9μs	0.691ns	2.59ns	0.0318	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.16μs	3.18ns	11.5ns	0.029	2.2 KB
master	`EnrichedLog`	net472	5.01μs	1.69ns	6.54ns	0.32	2.02 KB
#6738	`EnrichedLog`	net6.0	3.33μs	5.77ns	22.4ns	0.03	2.2 KB
#6738	`EnrichedLog`	netcoreapp3.1	4.2μs	1.71ns	6.64ns	0.0294	2.2 KB
#6738	`EnrichedLog`	net472	4.88μs	1.62ns	6.28ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.36μs	0.605ns	2.34ns	0.0156	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.74μs	0.624ns	2.41ns	0.0155	1.14 KB
master	`SendReceive`	net472	2.11μs	1.34ns	5.2ns	0.183	1.16 KB
#6738	`SendReceive`	net6.0	1.43μs	0.578ns	2.24ns	0.0157	1.14 KB
#6738	`SendReceive`	netcoreapp3.1	1.84μs	1.18ns	4.42ns	0.0156	1.14 KB
#6738	`SendReceive`	net472	2.09μs	0.718ns	2.69ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.74μs	0.565ns	2.19ns	0.022	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.9μs	2.99ns	11.6ns	0.0216	1.65 KB
master	`EnrichedLog`	net472	4.46μs	2.17ns	8.11ns	0.324	2.04 KB
#6738	`EnrichedLog`	net6.0	2.86μs	1.69ns	6.56ns	0.0227	1.6 KB
#6738	`EnrichedLog`	netcoreapp3.1	3.85μs	0.748ns	2.8ns	0.0211	1.65 KB
#6738	`EnrichedLog`	net472	4.25μs	2.73ns	10.6ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	392ns	0.434ns	1.68ns	0.00798	576 B
master	`StartFinishSpan`	netcoreapp3.1	606ns	1.38ns	5.36ns	0.0079	576 B
master	`StartFinishSpan`	net472	589ns	0.892ns	3.46ns	0.0916	578 B
master	`StartFinishScope`	net6.0	481ns	0.798ns	3.09ns	0.00986	696 B
master	`StartFinishScope`	netcoreapp3.1	725ns	1.74ns	6.72ns	0.00921	696 B
master	`StartFinishScope`	net472	842ns	1.98ns	7.67ns	0.104	658 B
#6738	`StartFinishSpan`	net6.0	409ns	0.503ns	1.95ns	0.00808	576 B
#6738	`StartFinishSpan`	netcoreapp3.1	569ns	0.751ns	2.81ns	0.00782	576 B
#6738	`StartFinishSpan`	net472	580ns	1.21ns	4.68ns	0.0916	578 B
#6738	`StartFinishScope`	net6.0	515ns	0.783ns	3.03ns	0.00971	696 B
#6738	`StartFinishScope`	netcoreapp3.1	728ns	1.45ns	5.62ns	0.00914	696 B
#6738	`StartFinishScope`	net472	788ns	1.49ns	5.79ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	615ns	1.18ns	4.58ns	0.00981	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	922ns	1.49ns	5.78ns	0.00937	696 B
master	`RunOnMethodBegin`	net472	1.06μs	2.1ns	8.12ns	0.104	658 B
#6738	`RunOnMethodBegin`	net6.0	616ns	0.791ns	3.06ns	0.00977	696 B
#6738	`RunOnMethodBegin`	netcoreapp3.1	957ns	1.74ns	6.73ns	0.00936	696 B
#6738	`RunOnMethodBegin`	net472	1.05μs	1.06ns	4.12ns	0.104	658 B

[Test Optimization] Add support to xUnit v3 2.0.0

a48452d

tonyredondo self-assigned this Mar 4, 2025

tonyredondo marked this pull request as ready for review March 4, 2025 13:11

tonyredondo requested review from a team as code owners March 4, 2025 13:11

bouwkast approved these changes Mar 4, 2025

View reviewed changes

daniel-mohedano approved these changes Mar 4, 2025

View reviewed changes

tonyredondo requested a review from daniel-mohedano March 4, 2025 13:31

daniel-mohedano approved these changes Mar 4, 2025

View reviewed changes

tonyredondo merged commit 86f940f into master Mar 4, 2025
145 of 150 checks passed

tonyredondo deleted the tony/add-support-for-xunitv3-2.x.x branch March 4, 2025 14:54

github-actions bot added this to the vNext-v3 milestone Mar 4, 2025

andrewlock added type:enhancement Improvement to an existing feature area:automatic-instrumentation Automatic instrumentation managed C# code (Datadog.Trace.ClrProfiler.Managed) area:ci-visibility labels Mar 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Test Optimization] Add support to xUnit v3 2.0.0 #6738

[Test Optimization] Add support to xUnit v3 2.0.0 #6738

tonyredondo commented Mar 4, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Mar 4, 2025 •

edited

Loading

andrewlock commented Mar 4, 2025

bouwkast left a comment

andrewlock commented Mar 4, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6738

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6738

Raw results

Raw results

Raw results

Raw results

Raw results

[Test Optimization] Add support to xUnit v3 2.0.0 #6738

[Test Optimization] Add support to xUnit v3 2.0.0 #6738

Conversation

tonyredondo commented Mar 4, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-datadog-prod-us1 bot commented Mar 4, 2025 • edited Loading

Datadog Report

andrewlock commented Mar 4, 2025

Execution-Time Benchmarks Report ⏱️

bouwkast left a comment

Choose a reason for hiding this comment

andrewlock commented Mar 4, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6738

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6738

Raw results

Raw results

Raw results

Raw results

Raw results

tonyredondo commented Mar 4, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Mar 4, 2025 •

edited

Loading