Skip to content
This repository was archived by the owner on Dec 6, 2024. It is now read-only.

Commit 1f4fa6e

Browse files
cyrille-leclerccyrille-leclerc
authored
Merge branch 'main' into main
2 parents 0d0211a + d226b67 commit 1f4fa6e

4 files changed

+460
-0
lines changed

‎text/0201-scope-attributes.md

+209
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
# Introduce Scope Attributes
2+
3+
This OTEP adds attributes to the Scope of a telemetry emitter (e.g. Tracer, Meter, LogEmitter).
4+
5+
## Motivation
6+
7+
There are a few reasons why adding Scope attributes is a good idea:
8+
9+
- There are 2 known use cases where Scope attributes can solve specific problems:
10+
- Add support for [Meter "short_name"](https://github.com/open-telemetry/opentelemetry-specification/pull/2422),
11+
represented as an attribute of Meter's Scope.
12+
- Add support for differentiating the type of data emitted from the scopes that belong
13+
to different data domains, e.g. profiling data emitted as log records or client-side
14+
data emitted as log records needs to be differentiated so that it can be easily
15+
routed and processed differently in the backends. We don't have a good way to handle
16+
this today. The type of the data can be recorded as an attribute Logger's Scope.
17+
- It makes Scope consistent with the other primary data types: Resource, Span, Metric,
18+
LogRecord.
19+
20+
See additional [discussion here](https://github.com/open-telemetry/opentelemetry-specification/issues/2450).
21+
22+
## Summary
23+
24+
The following is the summary of proposed changes:
25+
26+
- We will extend OpenTelemetry API to allow specifying Scope attributes when obtaining a
27+
Tracer, Meter or LogEmitter. Scope attributes will be optional.
28+
- We will add `attributes` field to the [InstrumentationScope](https://github.com/open-telemetry/opentelemetry-proto/blob/88faab1197a2a105c7da659951e94bc951d37ab9/opentelemetry/proto/common/v1/common.proto#L83)
29+
message of OTLP.
30+
- We will specify that Telemetry emitted via a Scope-ed Tracer, Meter or LogEmitter will
31+
be associated with the Scope's attributes.
32+
- We will specify that OTLP Exporter will record the attributes in the
33+
InstrumentationScope message.
34+
- We will create a section for Scope attributes' semantic conventions in
35+
the specification.
36+
37+
## Internal details
38+
39+
### API Changes
40+
41+
#### Tracer
42+
43+
`Get a Tracer` API will be extended to add the following parameter:
44+
45+
```
46+
- `attributes` (optional): Specifies the instrumentation scope attributes to associate
47+
with emitted telemetry.
48+
```
49+
50+
Since the attributes are optional this is a backwards compatible change.
51+
52+
We will modify the following clause:
53+
54+
```
55+
It is unspecified whether or under which conditions the same or different
56+
`Tracer` instances are returned from this functions.
57+
```
58+
59+
and replace it by:
60+
61+
```
62+
The implementation MUST NOT return the same `Tracer` when called repeatedly with
63+
different values of parameters. The only exception to this rule is no-op `Tracer`, the
64+
implementation MAY return the same instance regardless of parameter values.
65+
66+
It is unspecified whether or under which conditions the same or different
67+
`Tracer` instances are returned from this functions when the same
68+
(name,version,schema_url,attributes) parameters are used.
69+
```
70+
71+
Since we are defining more precisely previously undefined behavior this is a
72+
backwards compatible change.
73+
74+
#### Meter
75+
76+
`Get a Meter` API will be extended to add the following parameter:
77+
78+
```
79+
- `attributes` (optional): Specifies the instrumentation scope attributes to associate
80+
with emitted telemetry.
81+
```
82+
83+
We will modify the following clause:
84+
85+
```
86+
It is unspecified whether or under which conditions the same or different
87+
`Meter` instances are returned from this functions.
88+
```
89+
90+
and replace it by:
91+
92+
```
93+
The implementation MUST NOT return the same `Meter` when called repeatedly with
94+
different values of parameters. The only exception to this rule is no-op `Meter`, the
95+
implementation MAY return the same instance regardless of parameter values.
96+
97+
It is unspecified whether or under which conditions the same or different
98+
`Meter` instances are returned from this functions when the same
99+
(name,version,schema_url,attributes) parameters are used.
100+
```
101+
102+
#### LogEmitter
103+
104+
`Get LogEmitter` SDK call will be altered to the following:
105+
106+
```
107+
Accepts the instrumentation scope name and optional version and attributes and
108+
returns a LogEmitter associated with the instrumentation scope.
109+
110+
The implementation MUST NOT return the same `LogEmitter` when called repeatedly with
111+
different values of parameters. The only exception to this rule is no-op `LogEmitter`, the
112+
implementation MAY return the same instance regardless of parameter values.
113+
114+
It is unspecified whether or under which conditions the same or different
115+
`LogEmitter` instances are returned from this functions when the same
116+
(name,version,attributes) parameters are used.
117+
```
118+
119+
### OTLP Changes
120+
121+
The InstrumentationScope message in OTLP will be modified to add 2 new fields:
122+
attributes and dropped_attributes_count:
123+
124+
```protobuf
125+
message InstrumentationScope {
126+
string name = 1;
127+
string version = 2;
128+
repeated KeyValue attributes = 3;
129+
uint32 dropped_attributes_count = 4;
130+
}
131+
```
132+
133+
This change is backwards compatible from OTLP's interoperability perspective. Recipients
134+
of old OTLP versions will not see the Scope attributes and will ignore them, which we
135+
consider acceptable from interoperability perspective. This is aligned with our general
136+
stance on what happens when telemetry sources _add_ new data which old recipients
137+
don't understand: we expect the new data to be safely ignored.
138+
139+
## Attribute Value Precedence
140+
141+
If the same attribute is specified both at the Span/Metric/LogRecord and at the Scope
142+
then the attribute value at Span/Metric/LogRecord takes precedence.
143+
144+
This rule applies to non-OTLP exporters in SDKs, to conversions from OTLP to non-OTLP
145+
formats in the Collector and to OTLP recipients of data that need to interpret the
146+
attributes in the received data.
147+
148+
## Exporting to non-OTLP
149+
150+
SDK's non-OTLP Exporters and Collector's exporter to formats that don't have a concept
151+
that is equivalent to the Scope will record the attributes at the most suitable place
152+
in their corresponding format, typically at the Span, Metric or LogRecord equivalent.
153+
154+
## Prior art and alternatives
155+
156+
The [Meter "short_name" PR](https://github.com/open-telemetry/opentelemetry-specification/pull/2422)
157+
had an alternate approach where the "short_name" was added as the only attribute to the
158+
InstrumentationScope. This OTEP's proposal generalizes this and allows arbitrary
159+
attributes which allows them to be used for use cases.
160+
161+
Differentiating the type of data emitted from the scopes that belong to different data
162+
domains can be alternatively done by recording attributes on the Span, Metric or LogRecord.
163+
However, this will be less efficient since it will require the same attributes to be
164+
specified repeatedly on the wire. It will be also cumbersome to require the callers
165+
to always specify such attributes when creating a Span, Metric or a LogRecord as
166+
opposed to specifying them once when obtaining the Trace, Meter or LogEmitter.
167+
168+
## Examples
169+
170+
### Usage in Code
171+
172+
The following is an example usage where LogEmitter is used to emit client-side
173+
log records (pseudocode follows):
174+
175+
```
176+
// Obtain loggers once, at startup.
177+
appLogger = LogEmitterProvider.GetLogEmitter("mylibrary", "1.0.0")
178+
loggerForUserEvents = LogEmitterProvider.GetLogEmitter("mylibrary", "1.0.0", KeyValue("otel.clientside", true))
179+
180+
// Somewhere later in the code when the user clicks a UI element. This should
181+
// export telemetry with otel.clientside=true Scope attribute set.
182+
loggerForUserEvents.emit(LogRecord{Body:"click", Attributes:...})
183+
184+
// Somewhere else in the code, not related to user interactions. This should
185+
// export telemetry without any Scope attributes.
186+
appLogger.emit(LogRecord{Body:"Error occurred while processing the file", Attributes:...})
187+
```
188+
189+
### LogRecord Multiplexing
190+
191+
Here is an example usage where LogRecords are used to represent profiling data,
192+
client-side events and regular logs. The Scope attribute is used for multiplexing
193+
and routing the LogRecords:
194+
195+
![LogRecord Multiplexing](img/0201-scope-multiplexing.png)
196+
197+
## Open questions
198+
199+
- Should we allow/encourage recording Span/Metric/LogRecord attributes at the Scope level?
200+
The alternate is to disallow this and have completely separate set of semantic
201+
conventions that are allowed for Scope attributes only.
202+
- Can all existing APIs in all languages be safely modified to ensure the addition
203+
of the optional attributes is not a breaking change? (It should be safe, since we did
204+
a very similar change when we [introduced the Scope](https://github.com/open-telemetry/opentelemetry-specification/pull/2276))
205+
206+
## Future possibilities
207+
208+
If this OTEP is accepted we need to then introduce the relevant semantic conventions
209+
that will make the 2 use cases [described earlier](#motivation) possible.

‎text/0202-events-and-logs-api.md

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Introducing Events and Logs API
2+
3+
We introduce an Events and Logs API that is based on the OpenTelemetry Log signal, backed by LogRecord data model and LogEmitter SDK.
4+
5+
## Motivation
6+
7+
In OpenTelemetry's perspective Log Records and Events are different names for the same concept - however, there is a subtle difference in how they are represented using the underlying data model that is described below. We will describe why the existing Logging APIs are not sufficient for the purpose of creating events. It will then be evident that we will need an API in OpenTelementry for creating events. Note that the Events here refer to standalone Events and not to be confused with Span Events which occur only in the context of a span.
8+
9+
The Logs part of the API introduced here is supposed to be used only by the Log Appenders and end-users should continue to use the logging APIs available in the languages.
10+
11+
### Subtle differences between Logs and Events
12+
13+
Logs have a mandatory severity level as a first-class parameter that events do not have, and events have a mandatory name that logs do not have. Further, logs typically have messages in string form and events have data in the form of key-value pairs. It is due to this that their API interface requirements are slightly different.
14+
15+
### Who requires Events API
16+
17+
Here are a few situations that require recording of Events, there will be more. Note that the Trace API provides capability to record Events but that is only when a span is in progress. We need a separate API for recording standalone Events.
18+
19+
- RUM events (Client-side instrumentation)
20+
- Standalone events that occur when there is no span in progress, such as errors, user interaction events and web vitals.
21+
- Recording kubernetes events
22+
- Collector Entity events [link](https://docs.google.com/document/d/1Tg18sIck3Nakxtd3TFFcIjrmRO_0GLMdHXylVqBQmJA/edit)
23+
- Few other event systems described in [example mappings](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#appendix-a-example-mappings) in the data model.
24+
25+
### Can the current Log API interfaces be used for events?
26+
27+
- The log level is fundamental to the Log APIs in almost all the languages; all the methods in the Log interface are named after the log level, and there is usually no generic method to submit a log entry without log level.
28+
- In JavaScript for Web, the standard method of logging is to use console.log. Events can be created using [Event/CustomEvent](https://developer.mozilla.org/en-US/docs/Web/Events/Creating_and_triggering_events) interfaces. However, there is no option to define custom destination for these logs and events. Logs go only to console and event listeners are attached to the DOM element that dispatches it.
29+
- In Android, android.util.Log has methods Log.v(), Log.d(), Log.i(), Log.w(), and Log.e() to write logs. These methods correspond to the severity level.
30+
- Swift on iOS has Logger interface that has methods corresponding to severity level too.
31+
- The current Log APIs do not have a standard way to pass event attributes.
32+
- It may be possible to use the interpolation string args as the parameter to pass event attributes. However, the logging spec seems to map the human readable message (which is obtained after replacing the args in the interpolated string) to the Body field of LogRecord.
33+
- Log4j has an EventLogger interface that can be used to create structured messages with arbitrary key-value pairs, but log4j is not commonly used in Android apps as it is not officially supported on Android as per this [Stack Overflow thread](https://stackoverflow.com/questions/60398799/disable-log4j-jmx-on-android/60407849#60407849) by one of log4j’s maintainers.
34+
- In Python, logging.LogRecord's extra field is mapped to Otel LogRecord's attributes but this field is a hidden field and not part of the public interface.
35+
- The current Log APIs have a message parameter which could map to the Body field of LogRecord. However, this is restricted to String messages and does not allow for structured logs.
36+
37+
For the above reasons we can conclude that we will need an API for creating Events.
38+
39+
## Explanation
40+
41+
We propose a structure for Events for the purpose of distinguishing them from Logs and also propose having an API to ensure the structure is followed when creating Events using `LogRecord` data model.
42+
43+
### Events structure
44+
45+
All Events will have a name and a domain. The name is MANDATORY. The domain will serve as a mechanism to avoid conflicts with event names and is OPTIONAL. With this structure, an event name will be unique only in the context of a domain. It allows for two events in different domains to have same name, yet be unrelated events. When the domain is not present in an Event no claim is made about the uniqueness of event name.
46+
47+
### Events and Logs API
48+
49+
We also propose having an API interface for creating Events and Logs. Currently, there is only an SDK called [LogEmitterProvider](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/logging-library-sdk.md#logemitterprovider) for creating `LogRecord`s.
50+
51+
However, there is a question of whether OTel should have an API for logs. A part of the OTel community thinks that we should not have a full-fledged logging API unless there is a language that doesn't already have a plethora of logging libraries and APIs to choose from where it might make sense to define one. Further, we will not be able to have the [rich set of configuration options](https://logging.apache.org/log4j/2.x/manual/configuration.html) that some popular logging frameworks provide so the logging API in OTel will only become yet another API. However, it was noted that the Log Appender API is very similar to the API for Events and so instead of having API for Events and API for Log Appenders separately it was agreed to have one API for Events and Logs, and that the API for Logs is targeted only to Log Appenders. This will also keep it consistent with Traces and Metrics in having one API for each signal.
52+
53+
## Internal Details
54+
55+
The event name and domain will be attributes in the `LogRecord` defined using semantic conventions.
56+
57+
For the Events and Logs API, it will be very similar to the Trace API. There will be LoggerProvider and Logger interfaces analogous to TracerProvider and Tracer. The Logger interface will then be used to create Events and Logs using the LogRecord data model.
58+
59+
## Trade-offs and mitigations
60+
61+
There could be confusion on whether the Logs part of the API is end-user callable. While it can eventually be used in the languages that do not have a popular logging library, it is not recommended to be used in the languages where there are other popular logging libraries and APIs and this fact must emphasized in different forums.
62+
63+
## Prior art and alternatives
64+
65+
For client-side instrumentation, it was suggested initially that we use 0-duration spans to represent Events to get the benefit of Spans providing causality. For example, Splunk's RUM sdk for Android implements Events using [0-duration span](https://github.com/signalfx/splunk-otel-android/blob/main/splunk-otel-android/src/main/java/com/splunk/rum/SplunkRum.java#L213). However, 0-duration spans are confusing and not consistent with standalone Events in other domains which are represented using `LogRecord`s. Hence, for consistency reasons it will be good to use `LogRecord`s for standalone Events everywhere. To address the requirement of modeling causality between Events, we can create wrapper spans linked to the `LogRecord`s.
66+
67+
## Open questions
68+
69+
None.
70+
71+
## Future possibilities
72+
73+
1. As noted in the `Trade-offs and mitigation` section, we could allow the API to be used by end-users in the languages that do not have a popular logging library.
74+
2. There is a possibility that we may want to record the Span Events using `LogRecord`s in future. In this case, they will be correlated wth the Spans using the `TraceId` and `SpanId` fields of the `LogRecord`. If this is desired, we may add a configuration option to the `TracerProvider` to create LogRecords for the Span Events.

‎text/img/0201-scope-multiplexing.png

54.3 KB
Loading

0 commit comments

Comments
 (0)