Skip to content

Commit 5d6c73c

Browse files
hacdiaslidel
hacdias
and
lidel
authored
feat(gateway): trace context header support (#256)
* feat(examples): wrap handler with OTel propagation * feat: add tracing sub-package based on Kubo * docs: update tracing according to comments * docs(tracing): clarify that kubo is an example Context: ipfs-inactive/bifrost-gateway#68 https://www.w3.org/TR/trace-context/ --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>
1 parent 999d939 commit 5d6c73c

14 files changed

+636
-28
lines changed

‎docs/tracing.md

+142
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Tracing
2+
3+
Tracing across the stack follows, as much as possible, the [Open Telemetry]
4+
specifications. Configuration environment variables are specified in the
5+
[OpenTelemetry Environment Variable Specification].
6+
7+
We use the [opentelemtry-go] package, which currently does not have default support
8+
for the `OTEL_TRACES_EXPORTER` environment variables. Therefore, we provide some
9+
helper functions under [`boxo/tracing`](../tracing/) to support these.
10+
11+
In this document, we document the quirks of our custom support for the `OTEL_TRACES_EXPORTER`,
12+
as well as examples on how to use tracing, create traceable headers, and how
13+
to use the Jaeger UI. The [Gateway examples](../examples/gateway/) fully support Tracing.
14+
15+
- [Environment Variables](#environment-variables)
16+
- [`OTEL_TRACES_EXPORTER`](#otel_traces_exporter)
17+
- [`OTLP Exporter`](#otlp-exporter)
18+
- [`Jaeger Exporter`](#jaeger-exporter)
19+
- [`Zipkin Exporter`](#zipkin-exporter)
20+
- [`File Exporter`](#file-exporter)
21+
- [`OTEL_PROPAGATORS`](#otel_propagators)
22+
- [Using Jaeger UI](#using-jaeger-ui)
23+
- [Generate `traceparent` Header](#generate-traceparent-header)
24+
25+
## Environment Variables
26+
27+
For advanced configurations, such as ratio-based sampling, please see also the
28+
[OpenTelemetry Environment Variable Specification].
29+
30+
### `OTEL_TRACES_EXPORTER`
31+
32+
Specifies the exporters to use as a comma-separated string. Each exporter has a
33+
set of additional environment variables used to configure it. The following values
34+
are supported:
35+
36+
- `otlp`
37+
- `jaeger`
38+
- `zipkin`
39+
- `stdout`
40+
- `file` -- appends traces to a JSON file on the filesystem
41+
42+
Default: `""` (no exporters)
43+
44+
### `OTLP Exporter`
45+
46+
Unless specified in this section, the OTLP exporter uses the environment variables
47+
documented in [OpenTelemetry Protocol Exporter].
48+
49+
#### `OTEL_EXPORTER_OTLP_PROTOCOL`
50+
Specifies the OTLP protocol to use, which is one of:
51+
52+
- `grpc`
53+
- `http/protobuf`
54+
55+
Default: `"grpc"`
56+
57+
### `Jaeger Exporter`
58+
59+
See [Jaeger Exporter](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md#jaeger-exporter).
60+
61+
### `Zipkin Exporter`
62+
63+
See [Zipkin Exporter](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md#zipkin-exporter).
64+
65+
### `File Exporter`
66+
67+
#### `OTEL_EXPORTER_FILE_PATH`
68+
69+
Specifies the filesystem path for the JSON file.
70+
71+
Default: `"$PWD/traces.json"`
72+
73+
### `OTEL_PROPAGATORS`
74+
75+
See [General SDK Configuration](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md#general-sdk-configuration).
76+
77+
## Using Jaeger UI
78+
79+
One can use the `jaegertracing/all-in-one` Docker image to run a full Jaeger stack
80+
and configure the Kubo daemon, or gateway examples, to publish traces to it. Here, in an
81+
ephemeral container:
82+
83+
```console
84+
$ docker run --rm -it --name jaeger \
85+
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
86+
-p 5775:5775/udp \
87+
-p 6831:6831/udp \
88+
-p 6832:6832/udp \
89+
-p 5778:5778 \
90+
-p 16686:16686 \
91+
-p 14268:14268 \
92+
-p 14269:14269 \
93+
-p 14250:14250 \
94+
-p 9411:9411 \
95+
jaegertracing/all-in-one
96+
```
97+
98+
Then, in other terminal, start the app that uses `boxo/tracing` internally (e.g., a Kubo daemon), with Jaeger exporter enabled:
99+
100+
```
101+
$ OTEL_TRACES_EXPORTER=jaeger ipfs daemon
102+
```
103+
104+
Finally, the [Jaeger UI] is available at http://localhost:16686.
105+
106+
## Generate `traceparent` Header
107+
108+
If you want to trace a specific request and want to have its tracing ID, you can
109+
generate a `Traceparent` header. According to the [Trace Context] specification,
110+
the header is formed as follows:
111+
112+
> ```
113+
> version-format = trace-id "-" parent-id "-" trace-flags
114+
> trace-id = 32HEXDIGLC ; 16 bytes array identifier. All zeroes forbidden
115+
> parent-id = 16HEXDIGLC ; 8 bytes array identifier. All zeroes forbidden
116+
> trace-flags = 2HEXDIGLC ; 8 bit flags. Currently, only one bit is used. See below for details
117+
> ```
118+
119+
To generate a valid `Traceparent` header value, the following script can be used:
120+
121+
```bash
122+
version="00" # fixed in spec at 00
123+
trace_id="$(cat /dev/urandom | tr -dc 'a-f0-9' | fold -w 32 | head -n 1)"
124+
parent_id="00$(cat /dev/urandom | tr -dc 'a-f0-9' | fold -w 14 | head -n 1)"
125+
trace_flag="01" # sampled
126+
traceparent="$version-$trace_id-$parent_id-$trace_flag"
127+
echo $traceparent
128+
```
129+
130+
**NOTE**: the `tr` command behaves differently on macOS. You may want to install
131+
the GNU `tr` (`gtr`) and use it instead.
132+
133+
Then, the value can be passed onto the request with `curl -H "Traceparent: $traceparent" URL`.
134+
If using Jaeger, you can now search by the trace with ID `$trace_id` and see
135+
the complete trace of this request.
136+
137+
[Open Telemetry]: https://opentelemetry.io/
138+
[opentelemetry-go]: https://github.com/open-telemetry/opentelemetry-go
139+
[Trace Context]: https://www.w3.org/TR/trace-context
140+
[OpenTelemetry Environment Variable Specification]: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md
141+
[OpenTelemetry Protocol Exporter]: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md
142+
[Jaeger UI]: https://github.com/jaegertracing/jaeger-ui

‎examples/gateway/car/main.go

+14
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package main
22

33
import (
4+
"context"
45
"flag"
56
"io"
67
"log"
@@ -17,16 +18,29 @@ import (
1718
)
1819

1920
func main() {
21+
ctx, cancel := context.WithCancel(context.Background())
22+
defer cancel()
23+
2024
carFilePtr := flag.String("c", "", "path to CAR file to back this gateway from")
2125
port := flag.Int("p", 8040, "port to run this gateway from")
2226
flag.Parse()
2327

28+
// Setups up tracing. This is optional and only required if the implementer
29+
// wants to be able to enable tracing.
30+
tp, err := common.SetupTracing(ctx, "CAR Gateway Example")
31+
if err != nil {
32+
log.Fatal(err)
33+
}
34+
defer (func() { _ = tp.Shutdown(ctx) })()
35+
36+
// Sets up a block service based on the CAR file.
2437
blockService, roots, f, err := newBlockServiceFromCAR(*carFilePtr)
2538
if err != nil {
2639
log.Fatal(err)
2740
}
2841
defer f.Close()
2942

43+
// Creates the gateway API with the block service.
3044
gwAPI, err := gateway.NewBlocksGateway(blockService)
3145
if err != nil {
3246
log.Fatal(err)

‎examples/gateway/common/handler.go

+8-1
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import (
66
"github.com/ipfs/boxo/gateway"
77
"github.com/prometheus/client_golang/prometheus"
88
"github.com/prometheus/client_golang/prometheus/promhttp"
9+
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
910
)
1011

1112
func NewHandler(gwAPI gateway.IPFSBackend) http.Handler {
@@ -58,10 +59,16 @@ func NewHandler(gwAPI gateway.IPFSBackend) http.Handler {
5859
var handler http.Handler
5960
handler = gateway.WithHostname(mux, gwAPI, publicGateways, noDNSLink)
6061

61-
// Finally, wrap with the withConnect middleware. This is required since we use
62+
// Then, wrap with the withConnect middleware. This is required since we use
6263
// http.ServeMux which does not support CONNECT by default.
6364
handler = withConnect(handler)
6465

66+
// Finally, wrap with the otelhttp handler. This will allow the tracing system
67+
// to work and for correct propagation of tracing headers. This step is optional
68+
// and only required if you want to use tracing. Note that OTel must be correctly
69+
// setup in order for this to have an effect.
70+
handler = otelhttp.NewHandler(handler, "Gateway.Request")
71+
6572
return handler
6673
}
6774

‎examples/gateway/common/tracing.go

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
package common
2+
3+
import (
4+
"context"
5+
6+
"github.com/ipfs/boxo/tracing"
7+
"go.opentelemetry.io/contrib/propagators/autoprop"
8+
"go.opentelemetry.io/otel"
9+
"go.opentelemetry.io/otel/sdk/resource"
10+
"go.opentelemetry.io/otel/sdk/trace"
11+
semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
12+
)
13+
14+
// SetupTracing sets up the tracing based on the OTEL_* environment variables,
15+
// and the provided service name. It returns a trace.TracerProvider.
16+
func SetupTracing(ctx context.Context, serviceName string) (*trace.TracerProvider, error) {
17+
tp, err := NewTracerProvider(ctx, serviceName)
18+
if err != nil {
19+
return nil, err
20+
}
21+
22+
// Sets the default trace provider for this process. If this is not done, tracing
23+
// will not be enabled. Please note that this will apply to the entire process
24+
// as it is set as the default tracer, as per OTel recommendations.
25+
otel.SetTracerProvider(tp)
26+
27+
// Configures the default propagators used by the Open Telemetry library. By
28+
// using autoprop.NewTextMapPropagator, we ensure the value of the environmental
29+
// variable OTEL_PROPAGATORS is respected, if set. By default, Trace Context
30+
// and Baggage are used. More details on:
31+
// https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md
32+
otel.SetTextMapPropagator(autoprop.NewTextMapPropagator())
33+
34+
return tp, nil
35+
}
36+
37+
// NewTracerProvider creates and configures a TracerProvider.
38+
func NewTracerProvider(ctx context.Context, serviceName string) (*trace.TracerProvider, error) {
39+
exporters, err := tracing.NewSpanExporters(ctx)
40+
if err != nil {
41+
return nil, err
42+
}
43+
44+
options := []trace.TracerProviderOption{}
45+
46+
for _, exporter := range exporters {
47+
options = append(options, trace.WithBatcher(exporter))
48+
}
49+
50+
r, err := resource.Merge(
51+
resource.Default(),
52+
resource.NewSchemaless(
53+
semconv.ServiceNameKey.String(serviceName),
54+
),
55+
)
56+
if err != nil {
57+
return nil, err
58+
}
59+
options = append(options, trace.WithResource(r))
60+
return trace.NewTracerProvider(options...), nil
61+
}

‎examples/gateway/proxy/blockstore.go

+8-10
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,11 @@ import (
55
"fmt"
66
"io"
77
"net/http"
8-
"net/url"
98

109
"github.com/ipfs/boxo/exchange"
1110
blocks "github.com/ipfs/go-block-format"
1211
"github.com/ipfs/go-cid"
12+
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
1313
)
1414

1515
type proxyExchange struct {
@@ -19,7 +19,9 @@ type proxyExchange struct {
1919

2020
func newProxyExchange(gatewayURL string, client *http.Client) exchange.Interface {
2121
if client == nil {
22-
client = http.DefaultClient
22+
client = &http.Client{
23+
Transport: otelhttp.NewTransport(http.DefaultTransport),
24+
}
2325
}
2426

2527
return &proxyExchange{
@@ -29,17 +31,13 @@ func newProxyExchange(gatewayURL string, client *http.Client) exchange.Interface
2931
}
3032

3133
func (e *proxyExchange) fetch(ctx context.Context, c cid.Cid) (blocks.Block, error) {
32-
u, err := url.Parse(fmt.Sprintf("%s/ipfs/%s?format=raw", e.gatewayURL, c))
34+
urlStr := fmt.Sprintf("%s/ipfs/%s?format=raw", e.gatewayURL, c)
35+
req, err := http.NewRequestWithContext(ctx, http.MethodGet, urlStr, nil)
3336
if err != nil {
3437
return nil, err
3538
}
36-
resp, err := e.httpClient.Do(&http.Request{
37-
Method: http.MethodGet,
38-
URL: u,
39-
Header: http.Header{
40-
"Accept": []string{"application/vnd.ipld.raw"},
41-
},
42-
})
39+
req.Header.Set("Accept", "application/vnd.ipld.raw")
40+
resp, err := e.httpClient.Do(req)
4341
if err != nil {
4442
return nil, err
4543
}

‎examples/gateway/proxy/main.go

+12
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package main
22

33
import (
4+
"context"
45
"flag"
56
"log"
67
"net/http"
@@ -15,10 +16,21 @@ import (
1516
)
1617

1718
func main() {
19+
ctx, cancel := context.WithCancel(context.Background())
20+
defer cancel()
21+
1822
gatewayUrlPtr := flag.String("g", "", "gateway to proxy to")
1923
port := flag.Int("p", 8040, "port to run this gateway from")
2024
flag.Parse()
2125

26+
// Setups up tracing. This is optional and only required if the implementer
27+
// wants to be able to enable tracing.
28+
tp, err := common.SetupTracing(ctx, "CAR Gateway Example")
29+
if err != nil {
30+
log.Fatal(err)
31+
}
32+
defer (func() { _ = tp.Shutdown(ctx) })()
33+
2234
// Sets up a blockstore to hold the blocks we request from the gateway
2335
// Note: in a production environment you would likely want to choose a more efficient datastore implementation
2436
// as well as one that has a way of pruning storage so as not to hold data in memory indefinitely.

0 commit comments

Comments
 (0)