Run Azure Functions integration tests on .NET 8 and .NET 9 #6517

andrewlock · 2025-01-08T16:08:51Z

Summary of changes

Run the azure function integration tests on .NET 8 and .NET 9

Reason for change

We currently only run them on .NET 6 and .NET 7, we should run them on all supported frameworks

Implementation details

Update azure functions to build and run on .NET 8 and .NET 9
~~set DOTNET_ROOT to the latest location of our installed SDKs (not technically necessary for this, but probably still a good idea to head off other issues)~~ This seemed to cause other issues 🙄 And isn't necessary AFAICT, so reverted it
Manually remove the "built-in" dotnet install in the azure functions test stage. Yes this is horrible. But they have code to explicitly use it if it's there, and couldn't find an easy way to override it, so here we are.

Test coverage

More now!

Other details

I'm not proud of what I've done

bouwkast

Thanks!

datadog-ddstaging · 2025-01-08T16:41:30Z

Datadog Report

Branch report: andrew/test-functions-net8
Commit report: 9c97bc5
Test service: dd-trace-dotnet

✅ 0 Failed, 243036 Passed, 2139 Skipped, 18h 42m 53.51s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

HttpClient_SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.HttpMessageHandlerTests - Last Failure
Expand for error
```
 Expected exit code: 0, actual exit code: -1073741819.
```

andrewlock · 2025-01-08T18:23:58Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6517) - mean (69ms)  : 65, 72
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6517) - mean (980ms)  : 948, 1011
     .   : milestone, 980,
    master - mean (978ms)  : 957, 998
     .   : milestone, 978,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6517) - mean (108ms)  : 105, 111
     .   : milestone, 108,
    master - mean (108ms)  : 106, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6517) - mean (680ms)  : 661, 700
     .   : milestone, 680,
    master - mean (678ms)  : 666, 690
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6517) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6517) - mean (635ms)  : 620, 650
     .   : milestone, 635,
    master - mean (635ms)  : 619, 651
     .   : milestone, 635,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6517) - mean (194ms)  : 188, 201
     .   : milestone, 194,
    master - mean (194ms)  : 189, 198
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6517) - mean (1,102ms)  : 1075, 1128
     .   : milestone, 1102,
    master - mean (1,107ms)  : 1071, 1143
     .   : milestone, 1107,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6517) - mean (279ms)  : 273, 285
     .   : milestone, 279,
    master - mean (280ms)  : 276, 284
     .   : milestone, 280,

    section CallTarget+Inlining+NGEN
    This PR (6517) - mean (872ms)  : 843, 901
     .   : milestone, 872,
    master - mean (871ms)  : 842, 900
     .   : milestone, 871,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6517) - mean (267ms)  : 262, 273
     .   : milestone, 267,
    master - mean (267ms)  : 263, 270
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6517) - mean (858ms)  : 826, 890
     .   : milestone, 858,
    master - mean (852ms)  : 819, 886
     .   : milestone, 852,

andrewlock · 2025-01-09T00:55:45Z

Benchmarks Report for tracer 🐌

Benchmarks for #6517 compared to master:

2 benchmarks are faster, with geometric mean 1.151
1 benchmarks are slower, with geometric mean 1.125
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.11μs	45.6ns	329ns	0.0166	0.0083	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	50ns	320ns	0.0157	0.00524	0	5.8 KB
master	`StartStopWithChild`	net472	16.6μs	67.3ns	261ns	1.06	0.331	0.0889	6.21 KB
#6517	`StartStopWithChild`	net6.0	8.19μs	45.5ns	273ns	0.0164	0.0082	0	5.61 KB
#6517	`StartStopWithChild`	netcoreapp3.1	10.1μs	56ns	340ns	0.0202	0.0101	0	5.81 KB
#6517	`StartStopWithChild`	net472	16.3μs	58.1ns	225ns	1.03	0.299	0.0888	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	480μs	324ns	1.26μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	654μs	328ns	1.18μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	855μs	841ns	3.26μs	0.422	3.3 KB
#6517	`WriteAndFlushEnrichedTraces`	net6.0	520μs	233ns	902ns	0	2.7 KB
#6517	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	644μs	226ns	816ns	0	2.7 KB
#6517	`WriteAndFlushEnrichedTraces`	net472	857μs	515ns	2μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	156μs	957ns	9.52μs	0.152	14.47 KB
master	`SendRequest`	netcoreapp3.1	172μs	1.01μs	9.35μs	0.165	17.27 KB
master	`SendRequest`	net472	0.00178ns	0.000725ns	0.00281ns	0	0 b
#6517	`SendRequest`	net6.0	157μs	1.04μs	10.3μs	0.16	14.47 KB
#6517	`SendRequest`	netcoreapp3.1	180μs	1.18μs	11.6μs	0.171	17.27 KB
#6517	`SendRequest`	net472	0.0022ns	0.000824ns	0.00319ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	581μs	3.27μs	22μs	0.587	0	0	41.6 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	671μs	3.72μs	22μs	0.322	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	867μs	3.9μs	15.1μs	8.19	2.59	0.431	53.33 KB
#6517	`WriteAndFlushEnrichedTraces`	net6.0	578μs	3.24μs	22μs	0.616	0	0	41.72 KB
#6517	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	3.45μs	16.9μs	0.327	0	0	41.7 KB
#6517	`WriteAndFlushEnrichedTraces`	net472	833μs	3.96μs	15.9μs	8.28	2.48	0.414	53.27 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.29μs	1.29ns	4.84ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	1.33ns	4.97ns	0.0135	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.13μs	2.18ns	8.44ns	0.157	0.00106	987 B
#6517	`ExecuteNonQuery`	net6.0	1.25μs	1.32ns	5.1ns	0.0143	0	1.02 KB
#6517	`ExecuteNonQuery`	netcoreapp3.1	1.74μs	0.927ns	3.34ns	0.0129	0	1.02 KB
#6517	`ExecuteNonQuery`	net472	2.03μs	1.99ns	7.7ns	0.156	0.00102	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.16μs	0.665ns	2.49ns	0.0139	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	1.66ns	6.21ns	0.0132	976 B
master	`CallElasticsearch`	net472	2.53μs	2.45ns	9.16ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.23μs	0.618ns	2.31ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	0.65ns	2.43ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.68μs	1.79ns	6.94ns	0.167	1.05 KB
#6517	`CallElasticsearch`	net6.0	1.19μs	0.722ns	2.7ns	0.0137	976 B
#6517	`CallElasticsearch`	netcoreapp3.1	1.51μs	0.742ns	2.77ns	0.0128	976 B
#6517	`CallElasticsearch`	net472	2.47μs	1.64ns	6.34ns	0.158	995 B
#6517	`CallElasticsearchAsync`	net6.0	1.33μs	0.607ns	2.27ns	0.0133	952 B
#6517	`CallElasticsearchAsync`	netcoreapp3.1	1.58μs	1.09ns	3.93ns	0.0142	1.02 KB
#6517	`CallElasticsearchAsync`	net472	2.72μs	2.53ns	9.45ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.28μs	0.864ns	3.23ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	0.581ns	2.18ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.82μs	1.09ns	3.78ns	0.145	915 B
#6517	`ExecuteAsync`	net6.0	1.34μs	0.97ns	3.63ns	0.0134	952 B
#6517	`ExecuteAsync`	netcoreapp3.1	1.62μs	0.582ns	2.18ns	0.013	952 B
#6517	`ExecuteAsync`	net472	1.88μs	0.407ns	1.47ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.45μs	1.73ns	6.68ns	0.0315	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.29μs	2.82ns	10.9ns	0.0371	2.85 KB
master	`SendAsync`	net472	7.36μs	3.05ns	11ns	0.494	3.12 KB
#6517	`SendAsync`	net6.0	4.49μs	1.28ns	4.81ns	0.0313	2.31 KB
#6517	`SendAsync`	netcoreapp3.1	5.31μs	0.939ns	3.51ns	0.0372	2.85 KB
#6517	`SendAsync`	net472	7.32μs	1.95ns	7.57ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.54μs	0.653ns	2.45ns	0.0231	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.09μs	0.997ns	3.73ns	0.0225	1.64 KB
master	`EnrichedLog`	net472	2.69μs	2.29ns	8.57ns	0.249	1.57 KB
#6517	`EnrichedLog`	net6.0	1.51μs	0.932ns	3.49ns	0.0228	1.64 KB
#6517	`EnrichedLog`	netcoreapp3.1	2.27μs	1.41ns	5.08ns	0.0229	1.64 KB
#6517	`EnrichedLog`	net472	2.59μs	0.771ns	2.88ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	195ns	756ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	164ns	635ns	0.0603	0	4.28 KB
master	`EnrichedLog`	net472	151μs	173ns	648ns	0.671	0.224	4.46 KB
#6517	`EnrichedLog`	net6.0	117μs	125ns	469ns	0.0583	0	4.28 KB
#6517	`EnrichedLog`	netcoreapp3.1	123μs	335ns	1.3μs	0.0611	0	4.28 KB
#6517	`EnrichedLog`	net472	150μs	228ns	855ns	0.67	0.223	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.92μs	1.13ns	4.38ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.2μs	1.84ns	6.88ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	4.88μs	1.69ns	6.54ns	0.319	2.02 KB
#6517	`EnrichedLog`	net6.0	3.17μs	0.871ns	3.38ns	0.0312	2.2 KB
#6517	`EnrichedLog`	netcoreapp3.1	4.31μs	2.1ns	8.12ns	0.0281	2.2 KB
#6517	`EnrichedLog`	net472	5.15μs	1.88ns	7.26ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.818ns	3.17ns	0.0162	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	0.633ns	2.28ns	0.0152	1.14 KB
master	`SendReceive`	net472	2.13μs	1.58ns	5.92ns	0.184	1.16 KB
#6517	`SendReceive`	net6.0	1.4μs	0.706ns	2.74ns	0.016	1.14 KB
#6517	`SendReceive`	netcoreapp3.1	1.83μs	0.906ns	3.51ns	0.0155	1.14 KB
#6517	`SendReceive`	net472	2.13μs	2.33ns	9.01ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.79μs	0.881ns	3.3ns	0.0223	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.95μs	1.6ns	6.19ns	0.0219	1.65 KB
master	`EnrichedLog`	net472	4.24μs	3.28ns	11.8ns	0.322	2.04 KB
#6517	`EnrichedLog`	net6.0	2.64μs	1.18ns	4.56ns	0.0225	1.6 KB
#6517	`EnrichedLog`	netcoreapp3.1	3.83μs	1.19ns	4.6ns	0.0209	1.65 KB
#6517	`EnrichedLog`	net472	4.54μs	2.99ns	11.6ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6517

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.174	562.66	479.36
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.128	883.78	783.64

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	402ns	0.511ns	1.98ns	0.0081	576 B
master	`StartFinishSpan`	netcoreapp3.1	539ns	0.687ns	2.66ns	0.00777	576 B
master	`StartFinishSpan`	net472	650ns	1.43ns	5.56ns	0.0917	578 B
master	`StartFinishScope`	net6.0	563ns	0.836ns	3.24ns	0.00971	696 B
master	`StartFinishScope`	netcoreapp3.1	713ns	1.11ns	4.31ns	0.00965	696 B
master	`StartFinishScope`	net472	885ns	1.17ns	4.53ns	0.104	658 B
#6517	`StartFinishSpan`	net6.0	392ns	0.46ns	1.78ns	0.00819	576 B
#6517	`StartFinishSpan`	netcoreapp3.1	553ns	0.988ns	3.83ns	0.00775	576 B
#6517	`StartFinishSpan`	net472	671ns	1.01ns	3.91ns	0.0918	578 B
#6517	`StartFinishScope`	net6.0	479ns	0.528ns	2.04ns	0.00982	696 B
#6517	`StartFinishScope`	netcoreapp3.1	723ns	1.33ns	5.15ns	0.00943	696 B
#6517	`StartFinishScope`	net472	784ns	1.34ns	5.19ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6517

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.125	600.43	675.64

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	600ns	1.14ns	4.4ns	0.00973	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	896ns	1.28ns	4.96ns	0.00933	696 B
master	`RunOnMethodBegin`	net472	1.09μs	1.93ns	7.46ns	0.104	658 B
#6517	`RunOnMethodBegin`	net6.0	676ns	0.978ns	3.79ns	0.00994	696 B
#6517	`RunOnMethodBegin`	netcoreapp3.1	928ns	1.6ns	6.2ns	0.00935	696 B
#6517	`RunOnMethodBegin`	net472	1.14μs	1.61ns	6.23ns	0.105	658 B

This reverts commit 7a21336.

andrewlock added area:builds project files, build scripts, pipelines, versioning, releases, packages area:tests unit tests, integration tests area:serverless labels Jan 8, 2025

andrewlock requested review from a team as code owners January 8, 2025 16:08

kevingosse approved these changes Jan 8, 2025

View reviewed changes

bouwkast approved these changes Jan 8, 2025

View reviewed changes

andrewlock added 4 commits January 9, 2025 08:12

Run Azure functions tests on .NET 8 and .NET 9

115a048

set DOTNET_ROOT to avoid ambiguiity

e82cedd

:evil-laugh:

716bbe0

Revert "set DOTNET_ROOT to avoid ambiguiity"

9c97bc5

This reverts commit 7a21336.

andrewlock force-pushed the andrew/test-functions-net8 branch from aa1ca3c to 9c97bc5 Compare January 9, 2025 08:12

andrewlock merged commit f8a75eb into master Jan 9, 2025
99 of 104 checks passed

andrewlock deleted the andrew/test-functions-net8 branch January 9, 2025 16:54

github-actions bot added this to the vNext-v3 milestone Jan 9, 2025

lucaspimentel changed the title ~~Run .NET Functions on .NET 8 and .NET 9~~ Run Azure Functions integration tests on .NET 8 and .NET 9 Jan 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run Azure Functions integration tests on .NET 8 and .NET 9 #6517

Run Azure Functions integration tests on .NET 8 and .NET 9 #6517

andrewlock commented Jan 8, 2025 •

edited

Loading

bouwkast left a comment

datadog-ddstaging bot commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 9, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6517

Raw results

Slower ⚠️ in #6517

Raw results

Run Azure Functions integration tests on .NET 8 and .NET 9 #6517

Run Azure Functions integration tests on .NET 8 and .NET 9 #6517

Conversation

andrewlock commented Jan 8, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

bouwkast left a comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Jan 8, 2025 • edited Loading

Datadog Report

New Flaky Tests (1)

andrewlock commented Jan 8, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 9, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6517

Raw results

Slower ⚠️ in #6517

Raw results

andrewlock commented Jan 8, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 8, 2025 •

edited

Loading