Build the test sample projects in a separate CI stage #6498

andrewlock · 2025-01-03T14:25:08Z

Summary of changes

Simplify the sample build, and move the building of the samples to a separate stage.

Reason for change

Currently, the samples are built at the start of each integration stage (i.e. for each OS/arch/framework). This typically takes ~10+ minutes, depending on the details. If a test fails due to flake, the whole job must be restarted, which means you must pay that cost again.

Additionally, building the samples at this point puts them in the critical path for the whole CI run. By moving them to a separate stage, which can start immediately, we can have the samples built ready by the time the integration test stage starts

Finally, this is an incremental step in attempting to overhaul the Nuke build by reducing the number of different targets, and the complexity of the build in general.

Implementation details

One of the key factors is creating a dedicated (autogenerated) "samples" solution, that you can just run dotnet build on. This should be a lot faster than iterating through all the projects to find them, but it also just simplifies the build stage in general. It's built rom the "main" Datadog.Trace.sln, and can easily be updated by running .\tracer\build.ps1 RegenerateSolutions. I have some other ideas around this approach

Note that I looked at trying to use Microsoft.Build.Traversal but I just couldn't get it to work on our solution, and this seemed easier.

In the pipeline, I chose to build all the samples for all TFMs (in parallel). This mostly for simplicity, as some stages test other TFMs to the "default ones" (e.g. arm64 tests .NET 5 on PRs), but we can look to improve that later. The sample build runs on Windows, as I suspected that to give the fewest issues longer term. I also added a dedicate macos sample build to make sure we don't commit a sample that only builds on Windows! 😄

The biggest problem was CI Visibility. There some weirdness that happens with the Microsoft.TestPlatform.TestHost package. Namely, that on Windows, the package copies an arch-specific binary into the output folder. Unfortunately, this causes the tests to fail when it runs on arm64 (i.e. a different arch). I spent a long time trying to work around this, and in the end, just disabled the default copying for Windows. It looks like the CI Vis tests don't currently run on Windows (⚠️ @tonyredondo) so this "works" but will need to be addressed somehow later if we want to run them on Windows (we should).

Finally, I removed a lot of the old Nuke targets around building samples. This isn't complete yet, but is a step in the right direction I think.

Test coverage

I've done a lot of runs of the CI, including checking it still works for ASM and debugger etc. This is one of the latest runs.

I have also tried to keep the local dev experience the same for now, i.e. BuildAndRunWindowsIntegrationTests etc still work the same as they did before.

Other details

There's a lot of additional work and caveats about this work:

The fully built samples are big: 20GB. We probably could/should dig into that at some point...
Currently the default samples (i.e. not the multi-version samples) are only built not published. That means they don't contain all their dependencies, and rely on the packages folder.
- We don't currently restore that folder between stages.
- That doesn't matter currently, because we're doing a "full solution" restore in the build stage, and are restoring it there
- In the (near) future I'd like to stop doing the global restore (and simplify the "normal projects" build using a similar generated sln). At that point we'll need to look into handling the samples differently (either restoring the packages folder, or doing a publish instead of build)
The samples don't depend on Datadog.Trace which is why we can take this approach. However, some do depend on Datadog.Trace.Manual. Consequently, that project is rebuilt in the samples stage. It's not ideal, but not a big deal IMO
We don't build all the samples that we currently reference in the project. Some of this is for simplicity (e.g. aspnet samples) and some is because we simply don't test some of the samples (ExpenseItDemo).
Some stages are still doing their "own" building of samples. This is fine for now, but it would be nice to remove that rebuilding as we further simplify the Nuke build in the future.
We can now relatively easy start doing the multi-version testing on Windows if we want, as the samples are already built and available.
We need to start testing ci Visibility on Windows 😬
In general, you'll need to run RegenerateSolutions when you add a new sample. We could/should add a github action that flags this for you, but can do that in a follow up PR

https://datadoghq.atlassian.net/browse/AIT-6624
https://datadoghq.atlassian.net/browse/LANGPLAT-5

andrewlock · 2025-01-03T15:07:30Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (70ms)  : 64, 76
     .   : milestone, 70,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (987ms)  : 952, 1021
     .   : milestone, 987,
    master - mean (979ms)  : 957, 1000
     .   : milestone, 979,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (108ms)  : 105, 111
     .   : milestone, 108,
    master - mean (108ms)  : 105, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (677ms)  : 659, 695
     .   : milestone, 677,
    master - mean (682ms)  : 668, 696
     .   : milestone, 682,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (92ms)  : 90, 93
     .   : milestone, 92,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (636ms)  : 618, 654
     .   : milestone, 636,
    master - mean (639ms)  : 625, 654
     .   : milestone, 639,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (195ms)  : 189, 200
     .   : milestone, 195,
    master - mean (195ms)  : 190, 200
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (1,103ms)  : 1077, 1129
     .   : milestone, 1103,
    master - mean (1,099ms)  : 1062, 1136
     .   : milestone, 1099,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (279ms)  : 274, 283
     .   : milestone, 279,
    master - mean (278ms)  : 274, 283
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (875ms)  : 836, 913
     .   : milestone, 875,
    master - mean (869ms)  : 838, 899
     .   : milestone, 869,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6498) - mean (268ms)  : 264, 271
     .   : milestone, 268,
    master - mean (266ms)  : 262, 270
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (6498) - mean (854ms)  : 824, 884
     .   : milestone, 854,
    master - mean (845ms)  : 814, 877
     .   : milestone, 845,

datadog-ddstaging · 2025-01-03T15:19:06Z

Datadog Report

Branch report: andrew/ci/build-samples-separately
Commit report: c89b2c6
Test service: dd-trace-dotnet

✅ 0 Failed, 244974 Passed, 2116 Skipped, 19h 31m 55.83s Total Time
❄️ 38 New Flaky

New Flaky Tests (38)

IntegrationDisabled - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Last Failure
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Last Failure
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Last Failure
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Last Failure
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.SystemDataSqlClientTests - Last Failure
Expand for error
```
 Expected exit code: 0, actual exit code: 134.
```

andrewlock · 2025-01-03T15:23:09Z

Benchmarks Report for appsec 🐌

Benchmarks for #6498 compared to master:

1 benchmarks are faster, with geometric mean 1.270
2 benchmarks are slower, with geometric mean 1.145
1 benchmarks have fewer allocations
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6498

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑net472	1.175	3,596.97	4,224.67
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0	1.116	190,992.32	213,167.91

Faster 🎉 in #6498

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0	1.270	178.26	140.37

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	185μs	51.2ns	185ns	2.6	0	188.9 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	287μs	153ns	591ns	2.59	0	196.23 KB
master	`AllCycleSimpleBody`	net472	250μs	92.8ns	335ns	35.8	2	225.95 KB
master	`AllCycleMoreComplexBody`	net6.0	191μs	97.2ns	376ns	2.68	0	192.41 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	290μs	142ns	532ns	2.61	0	199.65 KB
master	`AllCycleMoreComplexBody`	net472	254μs	79.5ns	287ns	36.5	2.03	229.46 KB
master	`ObjectExtractorSimpleBody`	net6.0	178ns	0.0979ns	0.379ns	0.00394	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	195ns	0.125ns	0.469ns	0.00364	0	272 B
master	`ObjectExtractorSimpleBody`	net472	163ns	0.219ns	0.849ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.02μs	9.11ns	35.3ns	0.0538	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.85μs	2.81ns	10.5ns	0.0502	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.6μs	2.76ns	10.7ns	0.603	0.0054	3.8 KB
#6498	`AllCycleSimpleBody`	net6.0	205μs	62.4ns	225ns	2.66	0	188.76 KB
#6498	`AllCycleSimpleBody`	netcoreapp3.1	301μs	72.6ns	281ns	2.7	0	196.09 KB
#6498	`AllCycleSimpleBody`	net472	276μs	1.12μs	4.33μs	35.8	2.05	225.78 KB
#6498	`AllCycleMoreComplexBody`	net6.0	213μs	189ns	708ns	2.65	0	192.26 KB
#6498	`AllCycleMoreComplexBody`	netcoreapp3.1	315μs	146ns	528ns	2.68	0	199.5 KB
#6498	`AllCycleMoreComplexBody`	net472	280μs	750ns	2.7μs	36.4	2.02	229.3 KB
#6498	`ObjectExtractorSimpleBody`	net6.0	140ns	0.148ns	0.552ns	0.00391	0	280 B
#6498	`ObjectExtractorSimpleBody`	netcoreapp3.1	194ns	0.122ns	0.456ns	0.0037	0	272 B
#6498	`ObjectExtractorSimpleBody`	net472	164ns	0.138ns	0.498ns	0.0446	0	281 B
#6498	`ObjectExtractorMoreComplexBody`	net6.0	2.92μs	2.07ns	7.46ns	0.0525	0	3.78 KB
#6498	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.69μs	2.09ns	8.11ns	0.0495	0	3.69 KB
#6498	`ObjectExtractorMoreComplexBody`	net472	4.22μs	1.5ns	5.41ns	0.601	0.00638	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	35.9μs	16.1ns	60.4ns	0.453	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	54.1μs	28.1ns	109ns	0.43	0	32.4 KB
master	`EncodeArgs`	net472	67.7μs	84.2ns	326ns	5.15	0.0674	32.5 KB
master	`EncodeLegacyArgs`	net6.0	78.3μs	56ns	217ns	0	0	2.14 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	107μs	282ns	1.09μs	0	0	2.14 KB
master	`EncodeLegacyArgs`	net472	155μs	56ns	209ns	0.31	0	2.15 KB
#6498	`EncodeArgs`	net6.0	35.8μs	24ns	89.8ns	0.461	0	32.4 KB
#6498	`EncodeArgs`	netcoreapp3.1	53.9μs	27.2ns	102ns	0.432	0	32.4 KB
#6498	`EncodeArgs`	net472	67.1μs	59.2ns	229ns	5.15	0.0668	32.5 KB
#6498	`EncodeLegacyArgs`	net6.0	73.3μs	13.2ns	49.5ns	0	0	2.14 KB
#6498	`EncodeLegacyArgs`	netcoreapp3.1	107μs	388ns	1.5μs	0	0	2.15 KB
#6498	`EncodeLegacyArgs`	net472	155μs	119ns	461ns	0.308	0	2.15 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	179μs	245ns	950ns	0	2.44 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	186μs	286ns	1.11μs	0	2.39 KB
master	`RunWafRealisticBenchmark`	net472	200μs	66ns	247ns	0.3	2.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	115μs	80.4ns	311ns	0	1.47 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	124μs	114ns	428ns	0	1.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	133μs	106ns	412ns	0.2	1.49 KB
#6498	`RunWafRealisticBenchmark`	net6.0	178μs	458ns	1.77μs	0	2.44 KB
#6498	`RunWafRealisticBenchmark`	netcoreapp3.1	187μs	266ns	997ns	0	2.39 KB
#6498	`RunWafRealisticBenchmark`	net472	200μs	123ns	478ns	0.301	2.46 KB
#6498	`RunWafRealisticBenchmarkWithAttack`	net6.0	116μs	256ns	991ns	0	1.47 KB
#6498	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	123μs	46.9ns	182ns	0	1.46 KB
#6498	`RunWafRealisticBenchmarkWithAttack`	net472	133μs	50.2ns	174ns	0.199	1.48 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6498

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	252.85 KB	260.67 KB	7.82 KB	3.09%

Fewer allocations 🎉 in #6498

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	263.9 KB	253.87 KB	-10.03 KB	-3.80%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	53.2μs	259ns	1.4μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	53μs	237ns	853ns	42.64 KB
master	`StringConcatBenchmark`	net472	37.7μs	103ns	371ns	59.07 KB
master	`StringConcatAspectBenchmark`	net6.0	316μs	1.79μs	13μs	263.9 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	349μs	1.98μs	14.4μs	252.85 KB
master	`StringConcatAspectBenchmark`	net472	279μs	6μs	57.5μs	278.53 KB
#6498	`StringConcatBenchmark`	net6.0	58.8μs	727ns	7.24μs	43.44 KB
#6498	`StringConcatBenchmark`	netcoreapp3.1	53.7μs	262ns	1.11μs	42.64 KB
#6498	`StringConcatBenchmark`	net472	38.4μs	190ns	851ns	59.04 KB
#6498	`StringConcatAspectBenchmark`	net6.0	305μs	1.63μs	11.8μs	253.87 KB
#6498	`StringConcatAspectBenchmark`	netcoreapp3.1	333μs	1.94μs	17.4μs	260.67 KB
#6498	`StringConcatAspectBenchmark`	net472	289μs	7.39μs	71.7μs	278.53 KB

andrewlock · 2025-01-03T15:49:21Z

Benchmarks Report for tracer 🐌

Benchmarks for #6498 compared to master:

3 benchmarks are slower, with geometric mean 1.156
1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.01μs	43.3ns	256ns	0.0163	0.00817	0	5.6 KB
master	`StartStopWithChild`	netcoreapp3.1	10.4μs	51ns	210ns	0.0152	0.00506	0	5.8 KB
master	`StartStopWithChild`	net472	16.3μs	45ns	174ns	1.04	0.32	0.0984	6.22 KB
#6498	`StartStopWithChild`	net6.0	7.96μs	34.3ns	133ns	0.0205	0.0082	0	5.61 KB
#6498	`StartStopWithChild`	netcoreapp3.1	10.1μs	51.4ns	230ns	0.0235	0.0094	0	5.8 KB
#6498	`StartStopWithChild`	net472	16.5μs	58.5ns	227ns	1.06	0.323	0.0994	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	507μs	851ns	3.18μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	659μs	398ns	1.54μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	847μs	467ns	1.81μs	0.422	3.3 KB
#6498	`WriteAndFlushEnrichedTraces`	net6.0	485μs	225ns	872ns	0	2.7 KB
#6498	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	636μs	336ns	1.3μs	0	2.7 KB
#6498	`WriteAndFlushEnrichedTraces`	net472	855μs	609ns	2.36μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	149μs	876ns	8.22μs	0.142	14.47 KB
master	`SendRequest`	netcoreapp3.1	169μs	978ns	8.18μs	0.16	17.27 KB
master	`SendRequest`	net472	0.00138ns	0.000661ns	0.00256ns	0	0 b
#6498	`SendRequest`	net6.0	152μs	891ns	8.68μs	0.155	14.47 KB
#6498	`SendRequest`	netcoreapp3.1	175μs	1.03μs	9.96μs	0.197	17.27 KB
#6498	`SendRequest`	net472	0.00584ns	0.0015ns	0.00539ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #6498

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.89 KB	41.6 KB	-294 B	-0.70%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	565μs	2.79μs	12.5μs	0.561	0	0	41.64 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	678μs	3.73μs	21.1μs	0.345	0	0	41.89 KB
master	`WriteAndFlushEnrichedTraces`	net472	858μs	4.03μs	15.6μs	8.33	2.5	0.417	53.34 KB
#6498	`WriteAndFlushEnrichedTraces`	net6.0	561μs	3.2μs	23μs	0.532	0	0	41.58 KB
#6498	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	645μs	2.2μs	7.94μs	0.331	0	0	41.6 KB
#6498	`WriteAndFlushEnrichedTraces`	net472	859μs	4.5μs	22.1μs	8.39	2.52	0.419	53.3 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.38μs	1.21ns	4.37ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.86μs	2.05ns	7.94ns	0.0141	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.13μs	1.66ns	6.42ns	0.156	0.00106	987 B
#6498	`ExecuteNonQuery`	net6.0	1.3μs	1.68ns	6.5ns	0.0145	0	1.02 KB
#6498	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	1.34ns	5.02ns	0.0132	0	1.02 KB
#6498	`ExecuteNonQuery`	net472	2.06μs	2.71ns	10.5ns	0.157	0.00102	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6498

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net6.0	1.139	1,258.24	1,432.97

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.32μs	0.636ns	2.46ns	0.0134	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	0.51ns	1.91ns	0.0132	976 B
master	`CallElasticsearch`	net472	2.58μs	1.63ns	6.3ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.26μs	0.619ns	2.32ns	0.0132	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.82ns	3.18ns	0.0131	1.02 KB
master	`CallElasticsearchAsync`	net472	2.53μs	0.988ns	3.7ns	0.167	1.05 KB
#6498	`CallElasticsearch`	net6.0	1.28μs	3.03ns	11.7ns	0.014	976 B
#6498	`CallElasticsearch`	netcoreapp3.1	1.58μs	0.711ns	2.75ns	0.0127	976 B
#6498	`CallElasticsearch`	net472	2.58μs	1.53ns	5.53ns	0.158	995 B
#6498	`CallElasticsearchAsync`	net6.0	1.43μs	0.553ns	2.07ns	0.0129	952 B
#6498	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	1.11ns	4.14ns	0.0135	1.02 KB
#6498	`CallElasticsearchAsync`	net472	2.62μs	1.25ns	4.66ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.27μs	1.09ns	4.09ns	0.0133	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.63μs	1.29ns	4.83ns	0.0122	952 B
master	`ExecuteAsync`	net472	1.82μs	0.782ns	3.03ns	0.145	915 B
#6498	`ExecuteAsync`	net6.0	1.26μs	2.05ns	7.68ns	0.0135	952 B
#6498	`ExecuteAsync`	netcoreapp3.1	1.7μs	2.05ns	7.92ns	0.0128	952 B
#6498	`ExecuteAsync`	net472	1.79μs	0.409ns	1.47ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.52μs	4.21ns	16.3ns	0.0312	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.48μs	3ns	11.2ns	0.0382	2.85 KB
master	`SendAsync`	net472	7.44μs	2.27ns	8.79ns	0.493	3.12 KB
#6498	`SendAsync`	net6.0	4.54μs	1.86ns	6.96ns	0.0316	2.31 KB
#6498	`SendAsync`	netcoreapp3.1	5.31μs	1.54ns	5.96ns	0.0372	2.85 KB
#6498	`SendAsync`	net472	7.22μs	2.28ns	8.85ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.51μs	0.682ns	2.64ns	0.0234	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.12μs	0.913ns	3.54ns	0.0227	1.64 KB
master	`EnrichedLog`	net472	2.75μs	2.02ns	7.82ns	0.249	1.57 KB
#6498	`EnrichedLog`	net6.0	1.5μs	0.818ns	3.17ns	0.0233	1.64 KB
#6498	`EnrichedLog`	netcoreapp3.1	2.18μs	0.996ns	3.73ns	0.0217	1.64 KB
#6498	`EnrichedLog`	net472	2.7μs	1.15ns	4.29ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	117μs	214ns	828ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	122μs	293ns	1.13μs	0	0	4.28 KB
master	`EnrichedLog`	net472	151μs	299ns	1.16μs	0.67	0.223	4.46 KB
#6498	`EnrichedLog`	net6.0	116μs	232ns	898ns	0	0	4.28 KB
#6498	`EnrichedLog`	netcoreapp3.1	120μs	96.6ns	374ns	0.0596	0	4.28 KB
#6498	`EnrichedLog`	net472	153μs	158ns	610ns	0.687	0.229	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.01μs	1.12ns	4.02ns	0.0314	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.24μs	1.7ns	6.58ns	0.0297	2.2 KB
master	`EnrichedLog`	net472	5.02μs	1.84ns	7.13ns	0.32	2.02 KB
#6498	`EnrichedLog`	net6.0	2.97μs	1.18ns	4.57ns	0.03	2.2 KB
#6498	`EnrichedLog`	netcoreapp3.1	4.09μs	1.83ns	6.84ns	0.0286	2.2 KB
#6498	`EnrichedLog`	net472	4.98μs	1.75ns	6.3ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.44μs	0.707ns	2.74ns	0.0158	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	1.01ns	3.66ns	0.0149	1.14 KB
master	`SendReceive`	net472	2.08μs	1.09ns	4.07ns	0.183	1.16 KB
#6498	`SendReceive`	net6.0	1.32μs	0.494ns	1.85ns	0.0158	1.14 KB
#6498	`SendReceive`	netcoreapp3.1	1.81μs	0.948ns	3.55ns	0.0155	1.14 KB
#6498	`SendReceive`	net472	2.09μs	1.98ns	7.68ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.64μs	0.624ns	2.25ns	0.0223	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.93μs	2.9ns	10.5ns	0.0218	1.65 KB
master	`EnrichedLog`	net472	4.34μs	2.6ns	10.1ns	0.322	2.04 KB
#6498	`EnrichedLog`	net6.0	2.65μs	1.08ns	4.03ns	0.0226	1.6 KB
#6498	`EnrichedLog`	netcoreapp3.1	3.86μs	1.99ns	7.72ns	0.0229	1.65 KB
#6498	`EnrichedLog`	net472	4.34μs	1.21ns	4.37ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6498

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.191	780.81	930.24

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	408ns	0.287ns	1.11ns	0.00812	576 B
master	`StartFinishSpan`	netcoreapp3.1	611ns	0.372ns	1.39ns	0.00763	576 B
master	`StartFinishSpan`	net472	682ns	0.331ns	1.28ns	0.0916	578 B
master	`StartFinishScope`	net6.0	501ns	0.351ns	1.36ns	0.00975	696 B
master	`StartFinishScope`	netcoreapp3.1	729ns	0.469ns	1.82ns	0.00952	696 B
master	`StartFinishScope`	net472	781ns	0.979ns	3.79ns	0.104	658 B
#6498	`StartFinishSpan`	net6.0	454ns	0.15ns	0.563ns	0.008	576 B
#6498	`StartFinishSpan`	netcoreapp3.1	576ns	0.262ns	0.98ns	0.00773	576 B
#6498	`StartFinishSpan`	net472	694ns	0.437ns	1.69ns	0.0916	578 B
#6498	`StartFinishScope`	net6.0	535ns	0.323ns	1.25ns	0.00969	696 B
#6498	`StartFinishScope`	netcoreapp3.1	683ns	0.511ns	1.98ns	0.00944	696 B
#6498	`StartFinishScope`	net472	930ns	0.4ns	1.55ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6498

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.138	592.16	673.76

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	592ns	0.328ns	1.23ns	0.00987	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.01μs	0.769ns	2.98ns	0.00913	696 B
master	`RunOnMethodBegin`	net472	1.1μs	0.563ns	2.18ns	0.104	658 B
#6498	`RunOnMethodBegin`	net6.0	674ns	0.455ns	1.76ns	0.00977	696 B
#6498	`RunOnMethodBegin`	netcoreapp3.1	949ns	0.558ns	2.09ns	0.00916	696 B
#6498	`RunOnMethodBegin`	net472	1.19μs	0.496ns	1.92ns	0.104	658 B

bouwkast · 2025-01-06T15:26:18Z

Datadog.Trace.Samples.g.sln

Are we supposed to open/interact with this sln file?

I opened this up in VS and just rebuilt a random sample and it worked, but I do get NuGet restore errors for others, wasn't sure if these were known:

CallTargetNativeTest / Datadog.Tracer.Native.Checks both have references to Datadog.Trace.csproj

Samples.TraceAnnotations.VersionMismatch.* all fail due to Datadog.Trace.Annotations.csproj

And VS adds a lot of changes to this sln file as well
Mostly just the links to the Shared helpers: tracer\test\test-applications\Samples.Shared\Samples.Shared.projitems*{021efba6-c4ba-4de5-bf3f-c263ee9e20db}*SharedItemsImports = 5

Are we supposed to open/interact with this sln file

That wasn't really my intention, no 🤔 That said, it would be nice if we could...

CallTargetNativeTest / Datadog.Tracer.Native.Checks both have references to Datadog.Trace.csproj

Urgh I forgot that 💀 That's a big problem, going to have to figure that one out, thanks 😭

Samples.TraceAnnotations.VersionMismatch.* all fail due to Datadog.Trace.Annotations.csproj

😕 I don't understand why these would fail and yet the ones referencing Datadog.TRace.Manual wouldn't 🤔

And VS adds a lot of changes to this sln file as well
Mostly just the links to the Shared helpers: tracer\test\test-applications\Samples.Shared\Samples.Shared.projitems*{021efba6-c4ba-4de5-bf3f-c263ee9e20db}*SharedItemsImports = 5

I can add that one to the solution if it simplifies the VS experience - it shouldn't impact the build at all I think...

Ah we aren't removing them from the main sln I was thinking they were gone and we'd need to open up/interact with this one

I've added a workaround for the CallTargetNativeTest / Datadog.Tracer.Native.Checks checks, as those were potentially problematic. The other issues I'm not too worried about currently, as this solution isn't intended to be opened directly. That may change in the future, once I've tidied some other things, but I think it's ok to just have this be an implementation detail for now 🙂

bouwkast

Thanks!

I ran stuff locally and it all seemed to work as normal for building/testing/running samples

bouwkast · 2025-01-06T17:22:32Z

Datadog.Trace.Samples.g.sln

Ah we aren't removing them from the main sln I was thinking they were gone and we'd need to open up/interact with this one

bouwkast · 2025-01-06T18:21:00Z

tracer/build/_build/Build.Steps.cs

-                    patterns.Add($"test/test-applications/integrations/{dir}/obj");
-                }
+                // Explicitly compiles the prerequisites which MSbuild doesn't
+                // (no, I don't know why, I can't seem to convince it to do this automatically)


not really important, but did you figure out what caused the projects to duplicate themselves when running with TestAllPackageVersions it doesn't seem to happen anymore with this so that is 👍

No, I didn't see it happening so tbh, I didn't try to figure it out 😅

…er" solution file. This makes it easier to build the samples without complex globbing etc in a single shot.

… platforms, and built standalone Note that _currently_ this will require calling the dedicated CompileSamples target until I can tidy it up further

Ensure we build to the dedicated artifacts directory

We now have the same location for each platform (which makes it easier to reason about in general)

…d on Windows

…t broken

…m the samples solution, as they can't be built ahead of time

…mpileIntegrationTests stage

andrewlock · 2025-01-10T06:51:50Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6498) (11.041M)   : 0, 11040563
    master (11.183M)   : 0, 11183411
    benchmarks/2.9.0 (11.045M)   : 0, 11045405

    section Automatic
    This PR (6498) (7.279M)   : 0, 7278587
    master (7.317M)   : 0, 7317474
    benchmarks/2.9.0 (7.885M)   : 0, 7885346

    section Trace stats
    master (7.653M)   : 0, 7652583

    section Manual
    master (11.180M)   : 0, 11180467

    section Manual + Automatic
    This PR (6498) (6.697M)   : 0, 6696942
    master (6.733M)   : 0, 6732804

    section DD_TRACE_ENABLED=0
    master (10.140M)   : 0, 10140020

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6498) (9.550M)   : 0, 9549986
    master (9.596M)   : 0, 9595590
    benchmarks/2.9.0 (9.586M)   : 0, 9586476

    section Automatic
    This PR (6498) (6.203M)   : 0, 6203330
    master (6.493M)   : 0, 6492883

    section Trace stats
    master (6.886M)   : 0, 6885636

    section Manual
    master (9.786M)   : 0, 9785888

    section Manual + Automatic
    This PR (6498) (6.041M)   : 0, 6040846
    master (6.061M)   : 0, 6061169

    section DD_TRACE_ENABLED=0
    master (9.070M)   : 0, 9069859

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6498) (9.922M)   : 0, 9922326

    section Automatic
    This PR (6498) (6.566M)   : 0, 6566195

    section Manual + Automatic
    This PR (6498) (6.192M)   : 0, 6191928

tonyredondo · 2025-01-10T13:07:11Z

tracer/test/test-applications/integrations/Samples.XUnitTests/Samples.XUnitTests.csproj

@@ -40,7 +47,7 @@
    </None>
  </ItemGroup>

-  <Target Name="CodeCoverageFixTarget" AfterTargets="Build" Condition=" '$(OS)' == 'Windows_NT' AND '$(TargetFramework)' != '' ">
+  <Target Name="CodeCoverageFixTarget" AfterTargets="Build" Condition=" '$(TargetFramework)' != '' AND '$(ManuallyCopyCodeCoverageFiles)' != 'false' ">


can you help me understand this change?

The previous condition was making assumptions based on the specifics of the build. Namely, that we only ever do a dotnet build on Windows, and don't run a publish.

After this PR, that's no longer the case. The platform is not a factor in how things are built (we always build on Windows in this PR, but we should be able to build on anything using the same targets after this).

However, the fact remains, we do need this target if we're doing a dotnet build run, but not if we're doing a publish. So this ensures that the files are still copied when we want them to, and not when we don't.

It took a lot of trial and error to figure out, not gonna lie 😅

oh god 🙈 ✅

## Summary of changes Attempt to fix flakiness in Azure Functions multi-sample build ## Reason for change #6472 added multi-version building for Azure Functions samples. Unfortunately, this caused the build stage to be flaky, because Azure Functions apparently generates and builds a [completely separate _.csproj_](Azure/azure-functions-dotnet-worker#1252). Unfortunately, it also ignores all the artifact output stuff, which means when we restore/build multiple versions of the app (with different api versions, it stomps over itself. e.g. we can see the sample app build itself doing the right thing: ``` 08:59:05 [DBG] Samples.AzureFunctions.V4Isolated -> /project/artifacts/bin/Samples.AzureFunctions.V4Isolated/release_net6.0_1.6.0/Samples.AzureFunctions.V4Isolated.dll 08:59:06 [DBG] Samples.AzureFunctions.V4Isolated -> /project/artifacts/publish/Samples.AzureFunctions.V4Isolated/release_net6.0_1.6.0/ ``` But there's also the "WorkerExtensions" thing which is going completely off the rails, ignores all of the Directory.Build.props etc and just does its own thing ``` 08:59:07 [DBG] WorkerExtensions -> /tmp/oh4iajvh.kcj/publishout/ 08:59:07 [DBG] WorkerExtensions -> /tmp/zv30zzl2.cxz/publishout/Microsoft.Azure.Functions.Worker.Extensions.dll 08:59:08 [DBG] WorkerExtensions -> /tmp/zv30zzl2.cxz/publishout/ ``` Unfortunately, this shows it's trying to build two separate versions to the same location, and things break. ## Implementation details My initial attempt in #6521 failed. In this attempt I split the v1 SDK and v2 SDK into two separate projects. The SDK versions are tracked separately. I was thinking that as long as we only build a _single_ version per project we _should_ be ok. It was not OK. So in the end I tore out the Azure Functions version tracking. We could/should consider adding it back in some way, but right now this is causing too many issues An alternative is to just remove the Azure Functions samples from the "auto-updating multi version". I kept the split between v1 and v2 of the Azure Functions SDK to give us a _little_ more coverage; I think it's about the best we can do. ## Test coverage If this all finally passes, we should be good. ## Other details An important thing to note is that _currently_, even though we will now get dependabot notifactions about updates to the azure functions libs, these _aren't_ tested automatically. That's because currently our testing on Windows does _not_ run the "multi version" tests like we do on Linux. And we _only_ test Azure functions on Windows. This is something I'd like to address longer term with PRs like #6498, but for the meantime, we'll need to manually keep the samples up to date with the latest SDK version when there's a dependabot PR

robertpi

Thanks for this!

I don't claim to understand every part of the PR, but I approve that:

this won't negatively impact ASM functionality
this is definitely a useful optimization

## Summary of changes Adds missing dependency ## Reason for change Without it, the stage can start before the samples are available ## Implementation details Add the missing value ## Test coverage This is the test ## Other details Broke in #6498, didn't notice before because it _shouldn't_ be done first, unless we're under a lot of CI pressure

… the samples (#6536) ## Summary of changes Adds a GitHub action that ensures you run `RegenerateSolutions` after adding a new sample to the main solution ## Reason for change In #6498, we now have a separate samples solution that we use to build the samples in a separate stage. However, that means when you add a new sample to `Datadog.Trace.sln` you should run the `RegenerateSolutions` target, otherwise you could get unexpected failures. This GitHub action ensures that you do this. ## Implementation details Added a github action check, same as we have for other similar issues ## Test coverage Can't test it until it's merged. Once I'm sure it's working correctly, will mark it as required

## Summary of changes ## Reason for change We previously had `RunWindowsIntegrationTests`, `RunLinuxIntegrationTests`, `RunOsxIntegrationTests`, each with _subtle_ differences. Overall that adds complexity which isn't really necessary ## Implementation details Combine everything into a single target, and just specialize the filter instead to reduce the code ## Test coverage Tested locally on Windows, and ran a CI test, but OSX side is untested currently. ## Other details Stacked on - #6498 https://datadoghq.atlassian.net/browse/LANGPLAT-66

## Summary of changes Disables the `netcoreapp2.1` integration tests ## Reason for change #6498 broke the .NET Core 2.1 integration tests, due to a strange "missing dependency" issue. I believe this is due to the somewhat frankenstein approach we have to building and restoring right now. I hope to improve this in the near future, but fixing it for 2.1 right now is too big of a cost. ## Implementation details Remove 2.1 from the tests ## Test coverage A lot less now unfortunately, but it's not "supported" anyway ## Other details This is the quickest, easiest, way to remove the 2.1 tests to fix the build. Later we can decide if this is the permanent solution we want (in which case we can stop _building_ for 2.1 too) or if we want/can reenable some of the basic tests at least.

andrewlock requested review from a team as code owners January 3, 2025 14:25

github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 3, 2025

andrewlock mentioned this pull request Jan 6, 2025

Consolidate all integration test targets into a single target #6501

Merged

bouwkast reviewed Jan 6, 2025

View reviewed changes

bouwkast approved these changes Jan 6, 2025

View reviewed changes

andrewlock force-pushed the andrew/ci/build-samples-separately branch 4 times, most recently from 0e3bc1f to 0c3dab6 Compare January 9, 2025 13:17

andrewlock added 13 commits January 9, 2025 14:58

Add a Nuke task that creates a "samples only" sln file from the "mast…

2e5bca4

…er" solution file. This makes it easier to build the samples without complex globbing etc in a single shot.

Add new, dedicated CompileSamples Nuke targets that are shared across…

4f29dd1

… platforms, and built standalone Note that _currently_ this will require calling the dedicated CompileSamples target until I can tidy it up further

Build the samples in a separate, dedicated stage

7b1faa2

Ensure we build to the dedicated artifacts directory

Fix the integration test sample search location

cd2cc0f

We now have the same location for each platform (which makes it easier to reason about in general)

Fix line-ending issue in ASM test

194a16b

Fix sample compilations that break when doing full multi-package buil…

3954be5

…d on Windows

Work around CI Visibility horribleness

a8efca4

Add a test for building the samples on macos, to ensure it doesn't ge…

94ec285

…t broken

Remove now-unnecessary targets

a8b651c

Exclude 'Datadog.Tracer.Native.Checks' and 'CallTargetNativeTest' fro…

fbb9b04

…m the samples solution, as they can't be built ahead of time

Build projects that require direct datadog trace references in the Co…

dc000c1

…mpileIntegrationTests stage

I forgot we have a separate build for linux. Urgh

bc0fc1d

No idea why this would be required, but trying it

95eeb97

more hacking around

c89b2c6

andrewlock force-pushed the andrew/ci/build-samples-separately branch from 0c3dab6 to c89b2c6 Compare January 9, 2025 15:02

andrewlock mentioned this pull request Jan 10, 2025

Fix flakiness in Azure Functions #6524

Merged

tonyredondo reviewed Jan 10, 2025

View reviewed changes

tonyredondo approved these changes Jan 10, 2025

View reviewed changes

robertpi approved these changes Jan 13, 2025

View reviewed changes

andrewlock merged commit cb9a113 into master Jan 13, 2025
105 of 107 checks passed

andrewlock deleted the andrew/ci/build-samples-separately branch January 13, 2025 12:32

github-actions bot added this to the vNext-v3 milestone Jan 13, 2025

This was referenced Jan 13, 2025

Add a Github action that ensures you correctly persist any changes to the samples #6536

Merged

Add missing build dependency #6538

Merged

andrewlock mentioned this pull request Jan 16, 2025

Don't run integration tests on netcoreapp2.1 #6560

Merged

Build the test sample projects in a separate CI stage #6498

Build the test sample projects in a separate CI stage #6498

Conversation

andrewlock commented Jan 3, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Jan 3, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Jan 3, 2025 • edited Loading

Datadog Report

New Flaky Tests (38)

andrewlock commented Jan 3, 2025 • edited Loading

Benchmarks Report for appsec 🐌

Benchmark details

Slower ⚠️ in #6498

Faster 🎉 in #6498

Raw results

Raw results

Raw results

More allocations ⚠️ in #6498

Fewer allocations 🎉 in #6498

Raw results

andrewlock commented Jan 3, 2025 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #6498

Raw results

Raw results

Slower ⚠️ in #6498

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6498

Raw results

Slower ⚠️ in #6498

Raw results

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bouwkast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewlock commented Jan 10, 2025

Throughput/Crank Report ⚡

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertpi left a comment

Choose a reason for hiding this comment

andrewlock commented Jan 3, 2025 •

edited

Loading

andrewlock commented Jan 3, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 3, 2025 •

edited

Loading

andrewlock commented Jan 3, 2025 •

edited

Loading

andrewlock commented Jan 3, 2025 •

edited

Loading