[IAST] Propagation tests reorg #6487

daniel-romano-DD · 2024-12-27T11:13:52Z

Summary of changes

Moved propagation tests to their right folder and namespace in instrumented tests

Reason for change

Namespaces were not standard and tests location made them difficult to locate

Implementation details

Moved propagation tests from Vulnerabilities to Propagation folder and namespace

Test coverage

Other details

andrewlock · 2024-12-27T11:58:49Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6487) - mean (68ms)  : 66, 70
     .   : milestone, 68,
    master - mean (68ms)  : 65, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6487) - mean (976ms)  : 955, 996
     .   : milestone, 976,
    master - mean (979ms)  : 954, 1003
     .   : milestone, 979,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6487) - mean (107ms)  : 105, 109
     .   : milestone, 107,
    master - mean (107ms)  : 105, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6487) - mean (679ms)  : 658, 701
     .   : milestone, 679,
    master - mean (678ms)  : 663, 693
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6487) - mean (91ms)  : 88, 94
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6487) - mean (632ms)  : 617, 647
     .   : milestone, 632,
    master - mean (633ms)  : 618, 647
     .   : milestone, 633,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6487) - mean (193ms)  : 189, 196
     .   : milestone, 193,
    master - mean (195ms)  : 190, 200
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6487) - mean (1,100ms)  : 1070, 1129
     .   : milestone, 1100,
    master - mean (1,099ms)  : 1071, 1127
     .   : milestone, 1099,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6487) - mean (278ms)  : 274, 282
     .   : milestone, 278,
    master - mean (278ms)  : 273, 283
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6487) - mean (870ms)  : 844, 897
     .   : milestone, 870,
    master - mean (876ms)  : 839, 912
     .   : milestone, 876,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6487) - mean (266ms)  : 261, 271
     .   : milestone, 266,
    master - mean (268ms)  : 263, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6487) - mean (855ms)  : 820, 890
     .   : milestone, 855,
    master - mean (853ms)  : 809, 896
     .   : milestone, 853,

datadog-ddstaging · 2024-12-27T12:18:01Z

Datadog Report

Branch report: dani/iast/instrumented_tests_reorg
Commit report: d203bce
Test service: dd-trace-dotnet

✅ 0 Failed, 454874 Passed, 2821 Skipped, 19h 37m 14.32s Total Time

andrewlock · 2024-12-27T13:05:12Z

Benchmarks Report for tracer 🐌

Benchmarks for #6487 compared to master:

2 benchmarks are faster, with geometric mean 1.175
1 benchmarks are slower, with geometric mean 1.151
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.15μs	45.7ns	306ns	0.0198	0.00794	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	56.5ns	348ns	0.0253	0.0101	0	5.8 KB
master	`StartStopWithChild`	net472	16.3μs	40.8ns	158ns	1.03	0.298	0.0968	6.22 KB
#6487	`StartStopWithChild`	net6.0	8.01μs	43.8ns	266ns	0.0161	0.00803	0	5.61 KB
#6487	`StartStopWithChild`	netcoreapp3.1	10.3μs	56ns	326ns	0.0206	0.0103	0	5.8 KB
#6487	`StartStopWithChild`	net472	16.2μs	41.1ns	148ns	1.04	0.318	0.0977	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	480μs	411ns	1.59μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	659μs	190ns	712ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	855μs	573ns	2.14μs	0.428	3.3 KB
#6487	`WriteAndFlushEnrichedTraces`	net6.0	496μs	252ns	975ns	0	2.7 KB
#6487	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	664μs	636ns	2.38μs	0	2.7 KB
#6487	`WriteAndFlushEnrichedTraces`	net472	853μs	857ns	3.09μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	157μs	1.01μs	10.1μs	0.147	14.47 KB
master	`SendRequest`	netcoreapp3.1	183μs	1.15μs	11.2μs	0.192	17.27 KB
master	`SendRequest`	net472	0.00354ns	0.00107ns	0.00414ns	0	0 b
#6487	`SendRequest`	net6.0	151μs	898ns	8.85μs	0.162	14.47 KB
#6487	`SendRequest`	netcoreapp3.1	178μs	1.26μs	12.3μs	0.181	17.27 KB
#6487	`SendRequest`	net472	0.00139ns	0.000918ns	0.00344ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	562μs	2.8μs	15.1μs	0.556	0	0	41.67 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	700μs	3.79μs	26.8μs	0.44	0	0	41.96 KB
master	`WriteAndFlushEnrichedTraces`	net472	837μs	3.62μs	16.6μs	8.33	2.5	0.417	53.26 KB
#6487	`WriteAndFlushEnrichedTraces`	net6.0	562μs	2.96μs	19.4μs	0.571	0	0	41.62 KB
#6487	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	703μs	4.11μs	40.5μs	0.329	0	0	41.84 KB
#6487	`WriteAndFlushEnrichedTraces`	net472	842μs	3.99μs	15.5μs	8.17	2.45	0.408	53.31 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.34μs	0.918ns	3.31ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.82μs	1.97ns	7.62ns	0.0136	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.09μs	2.62ns	9.81ns	0.157	0.00104	987 B
#6487	`ExecuteNonQuery`	net6.0	1.29μs	1.33ns	4.97ns	0.0142	0	1.02 KB
#6487	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	1.95ns	7.56ns	0.0131	0	1.02 KB
#6487	`ExecuteNonQuery`	net472	2.12μs	2.24ns	8.67ns	0.156	0.00106	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.28μs	0.51ns	1.91ns	0.0135	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.61μs	1.25ns	4.84ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.54μs	2.54ns	9.83ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.3μs	0.834ns	3.12ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.68μs	1.93ns	7.48ns	0.0134	1.02 KB
master	`CallElasticsearchAsync`	net472	2.62μs	2.87ns	10.7ns	0.166	1.05 KB
#6487	`CallElasticsearch`	net6.0	1.24μs	0.982ns	3.54ns	0.0137	976 B
#6487	`CallElasticsearch`	netcoreapp3.1	1.54μs	0.398ns	1.44ns	0.0132	976 B
#6487	`CallElasticsearch`	net472	2.5μs	2.33ns	9.04ns	0.157	995 B
#6487	`CallElasticsearchAsync`	net6.0	1.32μs	1.57ns	5.89ns	0.0132	952 B
#6487	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	1.25ns	4.67ns	0.0135	1.02 KB
#6487	`CallElasticsearchAsync`	net472	2.64μs	2ns	7.76ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.31μs	3.34ns	12.5ns	0.0131	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.56μs	1.35ns	4.86ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.81μs	0.341ns	1.23ns	0.145	915 B
#6487	`ExecuteAsync`	net6.0	1.27μs	4.38ns	17ns	0.0138	952 B
#6487	`ExecuteAsync`	netcoreapp3.1	1.63μs	0.677ns	2.53ns	0.0123	952 B
#6487	`ExecuteAsync`	net472	1.75μs	0.629ns	2.44ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.46μs	1.75ns	6.77ns	0.031	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.25μs	2.15ns	8.33ns	0.0369	2.85 KB
master	`SendAsync`	net472	7.22μs	1.16ns	4.5ns	0.493	3.12 KB
#6487	`SendAsync`	net6.0	4.45μs	1.45ns	5.42ns	0.0312	2.31 KB
#6487	`SendAsync`	netcoreapp3.1	5.4μs	4.13ns	16ns	0.0374	2.85 KB
#6487	`SendAsync`	net472	7.36μs	1.54ns	5.95ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.44μs	0.662ns	2.48ns	0.0232	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.36μs	1.04ns	3.88ns	0.0212	1.64 KB
master	`EnrichedLog`	net472	2.8μs	1.18ns	4.57ns	0.248	1.57 KB
#6487	`EnrichedLog`	net6.0	1.6μs	2.53ns	9.8ns	0.0237	1.64 KB
#6487	`EnrichedLog`	netcoreapp3.1	2.23μs	0.801ns	3ns	0.0224	1.64 KB
#6487	`EnrichedLog`	net472	2.76μs	1.57ns	6.1ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	117μs	191ns	739ns	0.0586	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	122μs	210ns	757ns	0	0	4.28 KB
master	`EnrichedLog`	net472	153μs	77.9ns	281ns	0.692	0.231	4.46 KB
#6487	`EnrichedLog`	net6.0	118μs	186ns	722ns	0.0588	0	4.28 KB
#6487	`EnrichedLog`	netcoreapp3.1	121μs	161ns	601ns	0	0	4.28 KB
#6487	`EnrichedLog`	net472	153μs	160ns	620ns	0.688	0.229	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.92μs	0.994ns	3.85ns	0.0307	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.39μs	2.02ns	7.82ns	0.0306	2.2 KB
master	`EnrichedLog`	net472	4.9μs	0.713ns	2.57ns	0.319	2.02 KB
#6487	`EnrichedLog`	net6.0	2.97μs	0.874ns	3.15ns	0.0314	2.2 KB
#6487	`EnrichedLog`	netcoreapp3.1	4.25μs	2.05ns	7.95ns	0.0287	2.2 KB
#6487	`EnrichedLog`	net472	4.86μs	1.46ns	5.25ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.44μs	0.62ns	2.4ns	0.0159	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.83μs	1.79ns	6.93ns	0.0156	1.14 KB
master	`SendReceive`	net472	2.11μs	1.45ns	5.43ns	0.183	1.16 KB
#6487	`SendReceive`	net6.0	1.42μs	0.894ns	3.34ns	0.0162	1.14 KB
#6487	`SendReceive`	netcoreapp3.1	1.71μs	0.887ns	3.2ns	0.0149	1.14 KB
#6487	`SendReceive`	net472	2.07μs	1.52ns	5.7ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.81μs	0.832ns	3.22ns	0.0224	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.91μs	1.98ns	7.65ns	0.0215	1.65 KB
master	`EnrichedLog`	net472	4.4μs	3.23ns	12.1ns	0.322	2.04 KB
#6487	`EnrichedLog`	net6.0	2.74μs	1.04ns	3.88ns	0.0218	1.6 KB
#6487	`EnrichedLog`	netcoreapp3.1	3.85μs	1.36ns	5.09ns	0.0212	1.65 KB
#6487	`EnrichedLog`	net472	4.52μs	2.91ns	11.3ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6487

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.187	557.92	469.91
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.162	742.85	639.35

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	431ns	0.455ns	1.76ns	0.00811	576 B
master	`StartFinishSpan`	netcoreapp3.1	593ns	0.48ns	1.79ns	0.00793	576 B
master	`StartFinishSpan`	net472	743ns	0.662ns	2.48ns	0.0915	578 B
master	`StartFinishScope`	net6.0	558ns	0.323ns	1.25ns	0.0097	696 B
master	`StartFinishScope`	netcoreapp3.1	679ns	0.428ns	1.6ns	0.00946	696 B
master	`StartFinishScope`	net472	872ns	1.11ns	4.31ns	0.104	658 B
#6487	`StartFinishSpan`	net6.0	406ns	0.21ns	0.815ns	0.00801	576 B
#6487	`StartFinishSpan`	netcoreapp3.1	638ns	0.56ns	2.17ns	0.00775	576 B
#6487	`StartFinishSpan`	net472	640ns	0.278ns	1.08ns	0.0917	578 B
#6487	`StartFinishScope`	net6.0	470ns	0.286ns	1.11ns	0.00984	696 B
#6487	`StartFinishScope`	netcoreapp3.1	674ns	0.421ns	1.63ns	0.00946	696 B
#6487	`StartFinishScope`	net472	899ns	0.717ns	2.78ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6487

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.151	582.30	670.34

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	582ns	0.426ns	1.65ns	0.00985	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	863ns	0.584ns	2.26ns	0.00926	696 B
master	`RunOnMethodBegin`	net472	1.1μs	1.16ns	4.48ns	0.104	658 B
#6487	`RunOnMethodBegin`	net6.0	670ns	0.388ns	1.5ns	0.00975	696 B
#6487	`RunOnMethodBegin`	netcoreapp3.1	921ns	0.998ns	3.87ns	0.00965	696 B
#6487	`RunOnMethodBegin`	net472	1.08μs	0.683ns	2.64ns	0.104	658 B

Propagation tests reorg

d203bce

daniel-romano-DD added area:tests unit tests, integration tests area:asm-iast labels Dec 27, 2024

daniel-romano-DD requested a review from a team as a code owner December 27, 2024 11:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IAST] Propagation tests reorg #6487

[IAST] Propagation tests reorg #6487

daniel-romano-DD commented Dec 27, 2024

andrewlock commented Dec 27, 2024

datadog-ddstaging bot commented Dec 27, 2024 •

edited

Loading

andrewlock commented Dec 27, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6487

Raw results

Slower ⚠️ in #6487

Raw results

[IAST] Propagation tests reorg #6487

Are you sure you want to change the base?

[IAST] Propagation tests reorg #6487

Conversation

daniel-romano-DD commented Dec 27, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Dec 27, 2024

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Dec 27, 2024 • edited Loading

Datadog Report

andrewlock commented Dec 27, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6487

Raw results

Slower ⚠️ in #6487

Raw results

datadog-ddstaging bot commented Dec 27, 2024 •

edited

Loading