chore(llmobs): add span.finished telemetry metric #5444
Conversation
Overall package sizeSelf size: 9 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.5.0 | 29.83 MB | 29.83 MB | | @datadog/native-appsec | 8.5.1 | 19.26 MB | 19.27 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.6.0 | 9.79 MB | 10.16 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.4.0 | 2.77 MB | 5.42 MB | | @datadog/native-iast-rewriter | 2.8.0 | 2.6 MB | 2.74 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.13.1 | 117.64 kB | 839.26 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | dc-polyfill | 0.1.6 | 24.56 kB | 24.56 kB | | shell-quote | 1.8.2 | 23.54 kB | 23.54 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #5444 +/- ##
===========================================
- Coverage 79.57% 57.10% -22.47%
===========================================
Files 510 37 -473
Lines 22898 1527 -21371
===========================================
- Hits 18220 872 -17348
+ Misses 4678 655 -4023 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Datadog ReportBranch report: ✅ 0 Failed, 802 Passed, 0 Skipped, 9m 2.77s Total Time |
| @@ -48,6 +50,7 @@ class LLMObsSpanProcessor { | |||
|
|
|||
| try { | |||
| const formattedEvent = this.format(span) | |||
| telemetry.incrementLLMObsSpanFinishedCount(span) | |||
There was a problem hiding this comment.
it's possible we'll increment this count, but the appending could fail if for some reason there's a json-serializable issue when calling _writer.append (as that will JSON.stringify and get the byte length, as noted in the comment below it's rare but i don't wanna rule it out). are we OK with incrementing this metric if the appending fails, or do we only want to increment after we have successfully appended?
There was a problem hiding this comment.
Yup, we're fine to increment it here, we'll have separate metrics for spans submitted and payload errors so we won't miss any gaps here
BenchmarksBenchmark execution time: 2025-03-19 21:36:21 Comparing candidate commit c5db57b in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 918 metrics, 15 unstable metrics. |
What does this PR do?
Adds the
span.finishedtelemetry count metric to LLMObs spans before submitting to Datadog, in theLLMObsSpanProcessor.process()method. Also adds a minor_ml_obs.integrationinternal span store key-value pair to keep track of which integration a span originates from.Todo: add
_ml_obs.decoratorinternal span store key-value pair to keep track whether a span is created from a decorator.Motivation
Plugin Checklist
Additional Notes