Move the aspnetcore benchmark #6563

andrewlock · 2025-01-16T12:27:54Z

Summary of changes

Move a benchmark from agent 1 to agent 2

Reason for change

Currently benchmarks are basically never completing for PRs because there's a backlog on agent 1. Digging into this:

agent 1 runs consistently for ~45 mins, runs 4 benchmarks
agent 2 runs for 2 mins, runs 0 benchmarks :awkward-monkey:
agent 3-5 runs for 25 mins, runs 3 benchmarks generally, 2 on 5
agent 6 runs for 32 mins, runs 3 benchmarks
agent 7 (ASM) varies somewhat wildly, runs 4 benchmarks

Given that

Agent 1 is running more benchmarks
Agent 2 is running no benchmarks
We're going to move to a new benchmark platform soon anyway which will invalidate previous perf values

This PR moves the slowest benchmark currently running on agent 1 to agent 2

Implementation details

[BenchmarkAgent1] => [BenchmarkAgent2]

Test coverage

This PR is the test

Other details

We basically should never do this, but otherwise noone gets to see the results of the benchmarks for their PRs, rending the whole benchmark stage pointless 😬

datadog-ddstaging · 2025-01-16T13:01:40Z

Datadog Report

Branch report: andrew/ci/move-benchmarks
Commit report: 65f6ab6
Test service: dd-trace-dotnet

✅ 0 Failed, 246141 Passed, 1995 Skipped, 19h 20m 19.12s Total Time

andrewlock · 2025-01-16T13:22:13Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6563) - mean (70ms)  : 64, 76
     .   : milestone, 70,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6563) - mean (985ms)  : 957, 1013
     .   : milestone, 985,
    master - mean (981ms)  : 959, 1004
     .   : milestone, 981,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6563) - mean (108ms)  : 105, 110
     .   : milestone, 108,
    master - mean (108ms)  : 106, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6563) - mean (685ms)  : 661, 710
     .   : milestone, 685,
    master - mean (680ms)  : 662, 698
     .   : milestone, 680,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6563) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6563) - mean (636ms)  : 613, 659
     .   : milestone, 636,
    master - mean (637ms)  : 617, 657
     .   : milestone, 637,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6563) - mean (190ms)  : 186, 194
     .   : milestone, 190,
    master - mean (190ms)  : 186, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6563) - mean (1,087ms)  : 1062, 1113
     .   : milestone, 1087,
    master - mean (1,091ms)  : 1062, 1119
     .   : milestone, 1091,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6563) - mean (275ms)  : 269, 280
     .   : milestone, 275,
    master - mean (276ms)  : 271, 280
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6563) - mean (864ms)  : 829, 899
     .   : milestone, 864,
    master - mean (865ms)  : 838, 892
     .   : milestone, 865,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6563) - mean (264ms)  : 260, 267
     .   : milestone, 264,
    master - mean (263ms)  : 259, 267
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    This PR (6563) - mean (848ms)  : 814, 883
     .   : milestone, 848,
    master - mean (844ms)  : 813, 875
     .   : milestone, 844,

andrewlock · 2025-01-16T19:28:18Z

Benchmarks Report for tracer 🐌

Benchmarks for #6563 compared to master:

2 benchmarks are faster, with geometric mean 1.184
2 benchmarks are slower, with geometric mean 1.146
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.09μs	45.7ns	303ns	0.016	0.004	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	55.4ns	332ns	0.0192	0.00479	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	58.6ns	211ns	1.04	0.297	0.0989	6.21 KB
#6563	`StartStopWithChild`	net6.0	7.99μs	45ns	312ns	0.0153	0.00764	0	5.61 KB
#6563	`StartStopWithChild`	netcoreapp3.1	10.1μs	53.1ns	291ns	0.0254	0.0102	0	5.8 KB
#6563	`StartStopWithChild`	net472	16.4μs	45ns	174ns	1.05	0.318	0.0977	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	486μs	383ns	1.48μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	663μs	498ns	1.93μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	853μs	391ns	1.51μs	0.425	3.3 KB
#6563	`WriteAndFlushEnrichedTraces`	net6.0	483μs	1.07μs	4μs	0	2.7 KB
#6563	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	662μs	385ns	1.49μs	0	2.7 KB
#6563	`WriteAndFlushEnrichedTraces`	net472	860μs	1.16μs	4.48μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6563

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑net6.0	1.187	151,904.80	127,941.46	several?
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑netcoreapp3.1	1.181	168,445.68	142,669.68

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	153μs	923ns	9μs	0.159	14.47 KB
master	`SendRequest`	netcoreapp3.1	170μs	986ns	8.13μs	0.133	17.27 KB
master	`SendRequest`	net472	0.000249ns	0.000222ns	0.000802ns	0	0 b
#6563	`SendRequest`	net6.0	127μs	420ns	1.63μs	0.192	14.47 KB
#6563	`SendRequest`	netcoreapp3.1	143μs	473ns	1.83μs	0.212	17.27 KB
#6563	`SendRequest`	net472	0.00152ns	0.00132ns	0.00509ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	576μs	3.2μs	20.3μs	0.558	0	0	41.56 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	2.02μs	7.29μs	0.327	0	0	41.74 KB
master	`WriteAndFlushEnrichedTraces`	net472	860μs	3.59μs	13.9μs	8.25	2.6	0.434	53.27 KB
#6563	`WriteAndFlushEnrichedTraces`	net6.0	565μs	3.01μs	16.5μs	0.546	0	0	41.7 KB
#6563	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	688μs	3.73μs	26.1μs	0.327	0	0	41.76 KB
#6563	`WriteAndFlushEnrichedTraces`	net472	852μs	4.2μs	17.8μs	8.28	2.48	0.414	53.28 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.32μs	1.08ns	3.91ns	0.0145	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.76μs	1.42ns	5.52ns	0.0131	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.09μs	1.67ns	6.24ns	0.156	0.00105	987 B
#6563	`ExecuteNonQuery`	net6.0	1.31μs	0.859ns	3.33ns	0.0145	0	1.02 KB
#6563	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	0.739ns	2.66ns	0.0132	0	1.02 KB
#6563	`ExecuteNonQuery`	net472	2.09μs	1.19ns	4.59ns	0.156	0.00105	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.2μs	0.492ns	1.84ns	0.0138	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	1.23ns	4.61ns	0.0128	976 B
master	`CallElasticsearch`	net472	2.68μs	1.67ns	6.48ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.28μs	0.504ns	1.95ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	0.398ns	1.43ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.66μs	1.83ns	7.08ns	0.166	1.05 KB
#6563	`CallElasticsearch`	net6.0	1.13μs	0.601ns	2.25ns	0.0138	976 B
#6563	`CallElasticsearch`	netcoreapp3.1	1.66μs	0.6ns	2.25ns	0.0133	976 B
#6563	`CallElasticsearch`	net472	2.68μs	2.82ns	10.9ns	0.157	995 B
#6563	`CallElasticsearchAsync`	net6.0	1.3μs	0.64ns	2.39ns	0.013	952 B
#6563	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	0.73ns	2.73ns	0.0136	1.02 KB
#6563	`CallElasticsearchAsync`	net472	2.67μs	1.69ns	6.55ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.31μs	0.718ns	2.69ns	0.0131	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.69μs	0.974ns	3.77ns	0.0126	952 B
master	`ExecuteAsync`	net472	1.85μs	0.604ns	2.26ns	0.145	915 B
#6563	`ExecuteAsync`	net6.0	1.32μs	0.207ns	0.773ns	0.0132	952 B
#6563	`ExecuteAsync`	netcoreapp3.1	1.66μs	1.49ns	5.59ns	0.0125	952 B
#6563	`ExecuteAsync`	net472	1.78μs	0.568ns	2.2ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.43μs	1.49ns	5.77ns	0.031	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.28μs	1.72ns	6.65ns	0.0371	2.85 KB
master	`SendAsync`	net472	7.32μs	1.41ns	5.45ns	0.494	3.12 KB
#6563	`SendAsync`	net6.0	4.38μs	1.68ns	6.51ns	0.0328	2.31 KB
#6563	`SendAsync`	netcoreapp3.1	5.25μs	2.21ns	8.56ns	0.0368	2.85 KB
#6563	`SendAsync`	net472	7.33μs	1.05ns	3.92ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.58μs	1ns	3.76ns	0.0231	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.18μs	1.44ns	5.37ns	0.0218	1.64 KB
master	`EnrichedLog`	net472	2.74μs	1.18ns	4.58ns	0.249	1.57 KB
#6563	`EnrichedLog`	net6.0	1.55μs	1.01ns	3.78ns	0.0233	1.64 KB
#6563	`EnrichedLog`	netcoreapp3.1	2.19μs	1.35ns	5.23ns	0.0222	1.64 KB
#6563	`EnrichedLog`	net472	2.63μs	2.18ns	8.44ns	0.25	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	180ns	696ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	123μs	263ns	1.02μs	0	0	4.28 KB
master	`EnrichedLog`	net472	152μs	120ns	448ns	0.691	0.23	4.46 KB
#6563	`EnrichedLog`	net6.0	115μs	79.2ns	307ns	0	0	4.28 KB
#6563	`EnrichedLog`	netcoreapp3.1	120μs	179ns	693ns	0.0602	0	4.28 KB
#6563	`EnrichedLog`	net472	151μs	79.2ns	296ns	0.679	0.226	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.16μs	1.46ns	5.66ns	0.0303	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.14μs	1.55ns	5.36ns	0.0295	2.2 KB
master	`EnrichedLog`	net472	4.78μs	0.719ns	2.69ns	0.32	2.02 KB
#6563	`EnrichedLog`	net6.0	3.18μs	1.2ns	4.64ns	0.0316	2.2 KB
#6563	`EnrichedLog`	netcoreapp3.1	4.28μs	1.06ns	3.82ns	0.0296	2.2 KB
#6563	`EnrichedLog`	net472	4.96μs	0.859ns	3.33ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.37μs	1.04ns	4.02ns	0.0159	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.86μs	0.598ns	2.32ns	0.0158	1.14 KB
master	`SendReceive`	net472	2.2μs	1.56ns	6.03ns	0.183	1.16 KB
#6563	`SendReceive`	net6.0	1.46μs	2.26ns	8.77ns	0.016	1.14 KB
#6563	`SendReceive`	netcoreapp3.1	1.85μs	1.02ns	3.81ns	0.0158	1.14 KB
#6563	`SendReceive`	net472	2.03μs	0.712ns	2.66ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.78μs	0.857ns	3.21ns	0.0226	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.89μs	1.97ns	7.64ns	0.0212	1.65 KB
master	`EnrichedLog`	net472	4.5μs	1.74ns	6.26ns	0.322	2.04 KB
#6563	`EnrichedLog`	net6.0	2.81μs	0.938ns	3.63ns	0.0225	1.6 KB
#6563	`EnrichedLog`	netcoreapp3.1	4.04μs	2.18ns	8.43ns	0.0223	1.65 KB
#6563	`EnrichedLog`	net472	4.31μs	3.35ns	13ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6563

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.164	478.26	556.63

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	410ns	0.344ns	1.33ns	0.00805	576 B
master	`StartFinishSpan`	netcoreapp3.1	553ns	0.481ns	1.86ns	0.00772	576 B
master	`StartFinishSpan`	net472	733ns	0.207ns	0.802ns	0.0918	578 B
master	`StartFinishScope`	net6.0	478ns	0.228ns	0.885ns	0.00985	696 B
master	`StartFinishScope`	netcoreapp3.1	729ns	0.517ns	2ns	0.00945	696 B
master	`StartFinishScope`	net472	896ns	0.369ns	1.38ns	0.104	658 B
#6563	`StartFinishSpan`	net6.0	407ns	0.193ns	0.72ns	0.00818	576 B
#6563	`StartFinishSpan`	netcoreapp3.1	573ns	0.357ns	1.38ns	0.00778	576 B
#6563	`StartFinishSpan`	net472	663ns	0.399ns	1.55ns	0.0915	578 B
#6563	`StartFinishScope`	net6.0	557ns	0.181ns	0.676ns	0.0098	696 B
#6563	`StartFinishScope`	netcoreapp3.1	729ns	0.95ns	3.55ns	0.00956	696 B
#6563	`StartFinishScope`	net472	829ns	0.744ns	2.68ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6563

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1	1.128	853.29	962.91

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	617ns	0.573ns	2.22ns	0.00965	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	853ns	0.454ns	1.76ns	0.0094	696 B
master	`RunOnMethodBegin`	net472	1.11μs	0.583ns	2.26ns	0.105	658 B
#6563	`RunOnMethodBegin`	net6.0	637ns	0.332ns	1.28ns	0.00989	696 B
#6563	`RunOnMethodBegin`	netcoreapp3.1	963ns	0.498ns	1.93ns	0.0096	696 B
#6563	`RunOnMethodBegin`	net472	1.07μs	0.914ns	3.54ns	0.104	658 B

Move the aspnetcore benchmark

65f6ab6

andrewlock requested a review from a team as a code owner January 16, 2025 12:27

andrewlock added area:builds project files, build scripts, pipelines, versioning, releases, packages area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc labels Jan 16, 2025

github-actions bot added the area:tests unit tests, integration tests label Jan 16, 2025

kevingosse approved these changes Jan 16, 2025

View reviewed changes

andrewlock merged commit 9514e2c into master Jan 17, 2025
63 of 66 checks passed

andrewlock deleted the andrew/ci/move-benchmarks branch January 17, 2025 09:03

github-actions bot added this to the vNext-v3 milestone Jan 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move the aspnetcore benchmark #6563

Move the aspnetcore benchmark #6563

andrewlock commented Jan 16, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 16, 2025 •

edited

Loading

andrewlock commented Jan 16, 2025

andrewlock commented Jan 16, 2025

Raw results

Raw results

Faster 🎉 in #6563

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6563

Raw results

Slower ⚠️ in #6563

Raw results

Move the aspnetcore benchmark #6563

Move the aspnetcore benchmark #6563

Conversation

andrewlock commented Jan 16, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Jan 16, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 16, 2025

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 16, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Faster 🎉 in #6563

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6563

Raw results

Slower ⚠️ in #6563

Raw results

andrewlock commented Jan 16, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 16, 2025 •

edited

Loading