Wait for background task in ConfigureCiCommand #6535

kevingosse · 2025-01-13T10:01:14Z

Summary of changes

After ConfigureCiCommand completes, wait for the "upload repository changes" task to complete.

Reason for change

We already do this in RunCiCommand, not doing it in ConfigureCiCommand was an oversight.

Other details

Discovered while trying to randomize the order of the tests.

datadog-ddstaging · 2025-01-13T10:22:53Z

Datadog Report

Branch report: kevin/configurecicommand
Commit report: 82f0bfc
Test service: dd-trace-dotnet

✅ 0 Failed, 241784 Passed, 1951 Skipped, 19h 49m 28.95s Total Time
❄️ 2 New Flaky

New Flaky Tests (2)

EarlyFlakeDetection - Datadog.Trace.ClrProfiler.IntegrationTests.CI.MsTestV2EvpTests - Last Failure
Expand for error
```
 The sample did not exit in 600000ms. Memory dump taken: True. Killing process.
```
EarlyFlakeDetection - Datadog.Trace.ClrProfiler.IntegrationTests.CI.TcpXUnitEvpTests - Last Failure
Expand for error
```
 The sample did not exit in 600000ms. Memory dump taken: True. Killing process.
```

andrewlock · 2025-01-13T11:31:37Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6535) - mean (69ms)  : 65, 73
     .   : milestone, 69,
    master - mean (69ms)  : 63, 75
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6535) - mean (978ms)  : 949, 1006
     .   : milestone, 978,
    master - mean (973ms)  : 953, 993
     .   : milestone, 973,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6535) - mean (108ms)  : 106, 110
     .   : milestone, 108,
    master - mean (107ms)  : 104, 110
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6535) - mean (674ms)  : 660, 688
     .   : milestone, 674,
    master - mean (675ms)  : 658, 692
     .   : milestone, 675,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6535) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 88, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6535) - mean (628ms)  : 614, 641
     .   : milestone, 628,
    master - mean (632ms)  : 618, 646
     .   : milestone, 632,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6535) - mean (194ms)  : 189, 198
     .   : milestone, 194,
    master - mean (194ms)  : 190, 198
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6535) - mean (1,099ms)  : 1067, 1131
     .   : milestone, 1099,
    master - mean (1,096ms)  : 1070, 1123
     .   : milestone, 1096,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6535) - mean (278ms)  : 275, 282
     .   : milestone, 278,
    master - mean (278ms)  : 272, 284
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6535) - mean (872ms)  : 839, 905
     .   : milestone, 872,
    master - mean (870ms)  : 843, 897
     .   : milestone, 870,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6535) - mean (267ms)  : 261, 272
     .   : milestone, 267,
    master - mean (266ms)  : 262, 270
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (6535) - mean (842ms)  : 813, 872
     .   : milestone, 842,
    master - mean (854ms)  : 816, 892
     .   : milestone, 854,

andrewlock · 2025-01-13T15:57:20Z

Benchmarks Report for tracer 🐌

Benchmarks for #6535 compared to master:

3 benchmarks are faster, with geometric mean 1.137
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.01μs	44.9ns	280ns	0.015	0.0075	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	52.5ns	246ns	0.0208	0.0104	0	5.8 KB
master	`StartStopWithChild`	net472	16.3μs	31.3ns	121ns	1.03	0.296	0.0961	6.22 KB
#6535	`StartStopWithChild`	net6.0	8.01μs	43.1ns	236ns	0.0156	0.00778	0	5.61 KB
#6535	`StartStopWithChild`	netcoreapp3.1	10.1μs	56.5ns	391ns	0.0204	0.0102	0	5.8 KB
#6535	`StartStopWithChild`	net472	16.3μs	59.8ns	232ns	1.04	0.308	0.0973	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	503μs	219ns	821ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	635μs	277ns	1.03μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	856μs	374ns	1.4μs	0.428	3.3 KB
#6535	`WriteAndFlushEnrichedTraces`	net6.0	491μs	204ns	734ns	0	2.7 KB
#6535	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	650μs	387ns	1.4μs	0	2.7 KB
#6535	`WriteAndFlushEnrichedTraces`	net472	854μs	847ns	3.28μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	154μs	900ns	7.85μs	0.18	14.47 KB
master	`SendRequest`	netcoreapp3.1	176μs	1.13μs	11.2μs	0.17	17.27 KB
master	`SendRequest`	net472	0.00287ns	0.00089ns	0.00345ns	0	0 b
#6535	`SendRequest`	net6.0	153μs	895ns	8.54μs	0.144	14.47 KB
#6535	`SendRequest`	netcoreapp3.1	166μs	964ns	8.29μs	0.173	17.27 KB
#6535	`SendRequest`	net472	0.00199ns	0.000811ns	0.00314ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	588μs	1.98μs	7.15μs	0.604	0	0	41.72 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	678μs	3.54μs	21.8μs	0.316	0	0	41.75 KB
master	`WriteAndFlushEnrichedTraces`	net472	842μs	3.75μs	14μs	8.62	2.59	0.431	53.31 KB
#6535	`WriteAndFlushEnrichedTraces`	net6.0	564μs	3.04μs	16.4μs	0.273	0	0	41.71 KB
#6535	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	663μs	2.94μs	13.5μs	0.326	0	0	41.68 KB
#6535	`WriteAndFlushEnrichedTraces`	net472	862μs	3.45μs	13.4μs	8.08	2.55	0.425	53.28 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.4μs	1.78ns	6.89ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.85μs	1.1ns	4.11ns	0.0139	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.08μs	1.71ns	6.4ns	0.156	0.00103	987 B
#6535	`ExecuteNonQuery`	net6.0	1.32μs	1.39ns	5.36ns	0.0141	0	1.02 KB
#6535	`ExecuteNonQuery`	netcoreapp3.1	1.71μs	2.86ns	10.7ns	0.0135	0	1.02 KB
#6535	`ExecuteNonQuery`	net472	2.2μs	1.54ns	5.56ns	0.157	0.0011	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.24μs	0.817ns	3.17ns	0.0137	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.57μs	0.726ns	2.71ns	0.0134	976 B
master	`CallElasticsearch`	net472	2.55μs	1.53ns	5.71ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.29μs	0.754ns	2.92ns	0.0129	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.978ns	3.79ns	0.0139	1.02 KB
master	`CallElasticsearchAsync`	net472	2.57μs	1.83ns	7.08ns	0.167	1.05 KB
#6535	`CallElasticsearch`	net6.0	1.27μs	0.488ns	1.89ns	0.0139	976 B
#6535	`CallElasticsearch`	netcoreapp3.1	1.64μs	1.43ns	5.54ns	0.013	976 B
#6535	`CallElasticsearch`	net472	2.5μs	2.57ns	9.96ns	0.158	995 B
#6535	`CallElasticsearchAsync`	net6.0	1.3μs	0.553ns	2.14ns	0.0129	952 B
#6535	`CallElasticsearchAsync`	netcoreapp3.1	1.62μs	0.904ns	3.5ns	0.0138	1.02 KB
#6535	`CallElasticsearchAsync`	net472	2.71μs	2.29ns	8.55ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.27μs	1.16ns	4.34ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.53μs	1.64ns	5.91ns	0.0133	952 B
master	`ExecuteAsync`	net472	1.93μs	0.401ns	1.39ns	0.144	915 B
#6535	`ExecuteAsync`	net6.0	1.24μs	0.669ns	2.5ns	0.0136	952 B
#6535	`ExecuteAsync`	netcoreapp3.1	1.57μs	1.15ns	4.45ns	0.0127	952 B
#6535	`ExecuteAsync`	net472	1.82μs	0.671ns	2.6ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.38μs	2.19ns	8.5ns	0.031	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.45μs	1.83ns	6.85ns	0.0379	2.85 KB
master	`SendAsync`	net472	7.39μs	3.2ns	12.4ns	0.492	3.12 KB
#6535	`SendAsync`	net6.0	4.52μs	0.791ns	2.85ns	0.0316	2.31 KB
#6535	`SendAsync`	netcoreapp3.1	5.28μs	1.27ns	4.58ns	0.0368	2.85 KB
#6535	`SendAsync`	net472	7.37μs	1.77ns	6.85ns	0.493	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.52μs	0.687ns	2.66ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.21μs	0.989ns	3.57ns	0.0225	1.64 KB
master	`EnrichedLog`	net472	2.86μs	1.97ns	7.63ns	0.249	1.57 KB
#6535	`EnrichedLog`	net6.0	1.55μs	0.797ns	2.98ns	0.0232	1.64 KB
#6535	`EnrichedLog`	netcoreapp3.1	2.19μs	0.876ns	3.16ns	0.0219	1.64 KB
#6535	`EnrichedLog`	net472	2.77μs	0.597ns	2.07ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	258ns	932ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	232ns	900ns	0.0613	0	4.28 KB
master	`EnrichedLog`	net472	152μs	118ns	458ns	0.683	0.228	4.46 KB
#6535	`EnrichedLog`	net6.0	116μs	127ns	492ns	0.0575	0	4.28 KB
#6535	`EnrichedLog`	netcoreapp3.1	123μs	118ns	459ns	0	0	4.28 KB
#6535	`EnrichedLog`	net472	152μs	181ns	701ns	0.684	0.228	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.97μs	15.8ns	79ns	0.0304	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.18μs	1.68ns	6.28ns	0.0292	2.2 KB
master	`EnrichedLog`	net472	5.18μs	1.24ns	4.8ns	0.319	2.02 KB
#6535	`EnrichedLog`	net6.0	3.04μs	0.723ns	2.7ns	0.0304	2.2 KB
#6535	`EnrichedLog`	netcoreapp3.1	4.17μs	2.09ns	8.08ns	0.0292	2.2 KB
#6535	`EnrichedLog`	net472	4.87μs	0.76ns	2.84ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.35μs	0.594ns	2.3ns	0.0161	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.68μs	0.561ns	2.17ns	0.0149	1.14 KB
master	`SendReceive`	net472	2.14μs	0.793ns	3.07ns	0.183	1.16 KB
#6535	`SendReceive`	net6.0	1.32μs	0.424ns	1.47ns	0.0158	1.14 KB
#6535	`SendReceive`	netcoreapp3.1	1.73μs	1.02ns	3.94ns	0.0158	1.14 KB
#6535	`SendReceive`	net472	2.03μs	1.58ns	6.1ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.73μs	0.698ns	2.7ns	0.0219	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	4.1μs	1.84ns	7.13ns	0.0226	1.65 KB
master	`EnrichedLog`	net472	4.34μs	1.98ns	7.15ns	0.322	2.04 KB
#6535	`EnrichedLog`	net6.0	2.82μs	1.53ns	5.93ns	0.0225	1.6 KB
#6535	`EnrichedLog`	netcoreapp3.1	4.04μs	2.01ns	7.8ns	0.0223	1.65 KB
#6535	`EnrichedLog`	net472	4.43μs	3.15ns	12.2ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6535

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.165	569.09	488.40
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.131	615.20	544.01

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	452ns	0.246ns	0.951ns	0.00818	576 B
master	`StartFinishSpan`	netcoreapp3.1	615ns	1.03ns	3.98ns	0.00804	576 B
master	`StartFinishSpan`	net472	672ns	0.322ns	1.25ns	0.0916	578 B
master	`StartFinishScope`	net6.0	569ns	0.439ns	1.7ns	0.0097	696 B
master	`StartFinishScope`	netcoreapp3.1	720ns	0.908ns	3.52ns	0.00955	696 B
master	`StartFinishScope`	net472	943ns	0.769ns	2.88ns	0.104	658 B
#6535	`StartFinishSpan`	net6.0	460ns	0.243ns	0.941ns	0.00804	576 B
#6535	`StartFinishSpan`	netcoreapp3.1	544ns	0.277ns	1.07ns	0.00756	576 B
#6535	`StartFinishSpan`	net472	628ns	0.148ns	0.554ns	0.0915	578 B
#6535	`StartFinishScope`	net6.0	488ns	0.275ns	1.06ns	0.00977	696 B
#6535	`StartFinishScope`	netcoreapp3.1	670ns	0.345ns	1.29ns	0.00923	696 B
#6535	`StartFinishScope`	net472	910ns	0.595ns	2.3ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6535

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.116	658.52	590.01

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	659ns	0.43ns	1.67ns	0.00961	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	934ns	0.414ns	1.43ns	0.00906	696 B
master	`RunOnMethodBegin`	net472	1.04μs	0.671ns	2.42ns	0.104	658 B
#6535	`RunOnMethodBegin`	net6.0	589ns	0.382ns	1.48ns	0.00994	696 B
#6535	`RunOnMethodBegin`	netcoreapp3.1	917ns	0.678ns	2.62ns	0.00919	696 B
#6535	`RunOnMethodBegin`	net472	985ns	0.461ns	1.72ns	0.104	658 B

## Summary of changes Randomize the order of the tests. ## Reason for change Flaky tests are much harder to fix when we discover them long after they have been written. By randomizing the order of the tests, I'm hoping to make them fail earlier. In practice, this could temporarily increase the overall flakiness, but I expect this will reduce the overall effort spent on fixing tests. ## Implementation details In `CustomTestFramework`, randomize the list of all tests in each collections, and the collections themselves. The seed is displayed in the output. When a test order causes tests to fail, this allows to deterministically reproduce that test order. ## Other details Four other issues were found thanks to this: #6535, #6532, #6511, #6509

await UploadRepositoryChangesTask

82f0bfc

kevingosse requested a review from a team as a code owner January 13, 2025 10:01

kevingosse added area:ci-visibility type:bug labels Jan 13, 2025

andrewlock approved these changes Jan 13, 2025

View reviewed changes

tonyredondo approved these changes Jan 13, 2025

View reviewed changes

kevingosse merged commit 96af331 into master Jan 13, 2025
122 of 124 checks passed

kevingosse deleted the kevin/configurecicommand branch January 13, 2025 17:06

github-actions bot added this to the vNext-v3 milestone Jan 13, 2025

kevingosse mentioned this pull request Jan 14, 2025

Randomize the order of tests #5831

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wait for background task in ConfigureCiCommand #6535

Wait for background task in ConfigureCiCommand #6535

kevingosse commented Jan 13, 2025

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading

andrewlock commented Jan 13, 2025

andrewlock commented Jan 13, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6535

Raw results

Faster 🎉 in #6535

Raw results

Wait for background task in ConfigureCiCommand #6535

Wait for background task in ConfigureCiCommand #6535

Conversation

kevingosse commented Jan 13, 2025

Summary of changes

Reason for change

Other details

datadog-ddstaging bot commented Jan 13, 2025 • edited Loading

Datadog Report

New Flaky Tests (2)

andrewlock commented Jan 13, 2025

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 13, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6535

Raw results

Faster 🎉 in #6535

Raw results

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading