You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Dec 6, 2024. It is now read-only.
Copy file name to clipboardexpand all lines: text/0000-support-elastic-common-schema-in-opentelemetry.md
+12-14
Original file line number
Diff line number
Diff line change
@@ -16,9 +16,9 @@ Adding the coverage of ECS to OTEL would provide guidance to authors of OpenTele
16
16
17
17
In addition to the use case of structured logs, the maturity of ECS for SIEM (Security Information and Event Management) is a great opportunity for OpenTelemetry to expand its scope to the security use cases.
18
18
19
-
Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using ‘content-type’ to identify event types.
19
+
Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using ‘content-type' to identify event types.
20
20
21
-
We’d like to see different categories of structured logs being well-supported in the [OTEL Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy).
21
+
We'd like to see different categories of structured logs being well-supported in the [OTEL Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy).
22
22
23
23
24
24
## Customer Motivation
@@ -39,27 +39,27 @@ OpenTelemetry has the potential to grow exponentially if the data from these oth
39
39
40
40
### What is ECS?
41
41
42
-
The [Elastic Common Schema (ECS)](https://github.com/elastic/ecs) is an open source specification, developed with support from Elastic’s user community. ECS defines a common set of fields to be used when storing data in Elasticsearch, such as logs, metrics, and security and audit events. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Learn more at: [https://www.elastic.co/guide/en/ecs/current/ecs-reference.html](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)
42
+
The [Elastic Common Schema (ECS)](https://github.com/elastic/ecs) is an open source specification, developed with support from Elastic's user community. ECS defines a common set of fields to be used when storing data in Elasticsearch, such as logs, metrics, and security and audit events. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Learn more at: [https://www.elastic.co/guide/en/ecs/current/ecs-reference.html](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)
43
43
44
44
The coverage of ECS is very broad including in depth support for logs, security, and network events such as "[logs.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-log.html)" , "[geo.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html)", "[tls.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-tls.html)", "[dns.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-dns.html)", or "[vulnerability.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-vulnerability.html)".
45
45
46
46
ECS has the following guiding principles:
47
47
48
48
* ECS favors human readability in order to enable broader adoption,
49
49
* ECS events include metadata to enable correlations across any dimension (host, data center, docker image, ip address...),
50
-
* ECS doesn’t differentiate the metadata fields that are specific to each event of the event source and the metadata that are shared by all the events of the source the same way OTEL differentiates Resource Attributes and Log/Span Attributes),
50
+
* ECS doesn't differentiate the metadata fields that are specific to each event of the event source and the metadata that are shared by all the events of the source the same way OTEL differentiates Resource Attributes and Log/Span Attributes),
51
51
* ECS groups fields in namespaces in order to:
52
52
* Offer consistency and readability,
53
53
* Enable reusability of namespaces in different contexts,
54
-
*Example the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces
54
+
*For example, the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces
55
55
* Enable extensibility adding fields to namespaces and adding new namespaces,
56
56
* Prevent field name conflicts
57
-
* ECS covers a broad spectrum of events with 40+ namespaces including detailed coverage of security and network events. It’s much broader than simple logging use cases.
57
+
* ECS covers a broad spectrum of events with 40+ namespaces including detailed coverage of security and network events. It's much broader than simple logging use cases.
58
58
59
59
60
60
### Example of a log message structured with ECS: NGINX access logs
61
61
62
-
Example of an NGinx Access Log entry structured with ECS
62
+
Example of a Nginx Access Log entry structured with ECS
63
63
64
64
65
65
```json
@@ -128,8 +128,6 @@ Example of an NGinx Access Log entry structured with ECS
128
128
129
129
## Principles
130
130
131
-
132
-
133
131
| Description |[OTel Logs and Event Record](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#log-and-event-record-definition)|[Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)|
134
132
|-------------|-------------|--------|
135
133
| Metadata shared by all the Log Messages / Spans of an application instance | Resource Attributes | ECS fields |
@@ -146,7 +144,7 @@ Example of an NGinx Access Log entry structured with ECS
| Numbers | number | <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html">long</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html">double</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html">scaled_float</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/boolean.html">boolean</a>… |
149
-
| Objects |Uint32, uint64… | <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/object.html">object</a> (JSON object), <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/flattened.html">flattened</a> (An entire JSON object as a single field value) |
147
+
| Objects |uint32, uint64… | <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/object.html">object</a> (JSON object), <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/flattened.html">flattened</a> (An entire JSON object as a single field value) |
150
148
| Structured Objects | No complex semantic data type specified for the moment (e.g. string is being used for ip addresses rather than having an "ip" data structure in Otel). <br/> Note that Otel supports arrays and nested objects. | <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/ip.html">ip</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/geo-point.html">geo_point</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/geo-shape.html">geo_shape</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/version.html">version</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/range.html">long_range</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/range.html">date_range</a>, <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/range.html">ip_range</a> |
151
149
| Binary data | Byte sequence | <ahref="https://www.elastic.co/guide/en/elasticsearch/reference/master/binary.html">binary</a> |
152
150
@@ -241,13 +239,13 @@ As the markdown code of the tables is hard to read and maintain with very long l
241
239
242
240
Prometheus is a de facto standard for observability metrics and OpenTelemetry already provides full interoperability with the Prometheus ecosystem.
243
241
244
-
It would be useful to get interoperability between metrics collected by an [official Prometheus exporter](https://prometheus.io/docs/instrumenting/exporters/) (e.g. the [Node/system metrics exporter](https://github.com/prometheus/node_exporter) or the [MySQL server exporter](https://github.com/prometheus/mysqld_exporter)) and the equivalent OpenTelemetry Collector receivers (e.g. Otel Collector [Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) or [MySQL Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mysqlreceiver))
242
+
It would be useful to get interoperability between metrics collected by [official Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/) (e.g. the [Node/system metrics exporter](https://github.com/prometheus/node_exporter) or the [MySQL server exporter](https://github.com/prometheus/mysqld_exporter)) and their equivalent OpenTelemetry Collector receivers (e.g. Otel Collector [Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) or [MySQL Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mysqlreceiver)).
245
243
246
-
Note that one of the challenges with Prometheus metrics naming conventions is that these are implicit conventions defined by each integration author which don’t enable correlation due to the lack of consistency across integrations. For example, this inconsistency increases the complexity that an end-user has to deal with when configuring and monitoring alerts.
244
+
Note that one of the challenges with Prometheus metrics naming conventions is that these are implicit conventions defined by each integration author which doesn't enable correlation due to the lack of consistency across integrations. For example, this inconsistency increases the complexity that an end-user has to deal with when configuring and monitoring alerts.
247
245
248
-
Prometheus conventions are restricted to the style of the name of the metrics (see [Metric and label naming | Prometheus](https://prometheus.io/docs/practices/naming/) ) but don’t specify unified metric names.
246
+
Prometheus' conventions are restricted to the style of the name of the metrics (see [Metric and label naming | Prometheus](https://prometheus.io/docs/practices/naming/) ) but don't specify unified metric names.
249
247
250
248
251
249
# Other areas that need to be addressed by OTEL (the project)
252
250
253
-
Some areas that need to be addressed in the long run as ECS is integrated into OTEL include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTEL specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md))
251
+
Some areas that need to be addressed in the long run as ECS is integrated into OTEL include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTEL specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)).
0 commit comments