Metrics don't ship to Datadog after 0.34.0 upgrade

[a26nine]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I upgraded from 0.33.1 to 0.34.0, and the metrics stopped on Datadog.

I see this in the changelog:

The datadog_agent source now records the “interval” on any incoming metrics that have it set rather than just rate. This is useful as metrics can be interpreted as rates later when viewing the data in Datadog, where the interval field will be used.

Config file loads fine. Healtcheck is passed. However, metrics don't appear on Datadog —they do appear on other sink destinations.

Configuration

sink_dd:
    type: datadog_metrics
    inputs:
      - source_dd
    default_api_key: xxx
    batch:
      timeout_secs: 5
      max_bytes: 4194304
    buffer:
      type: disk
      max_size: 4294967296
      when_full: drop_newest
    request:
      rate_limit_duration_secs: 5
      rate_limit_num: 100
      retry_max_duration_secs: 600

Version

0.34.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[StephenWakely]

Ok, we have figured out what is likely going on. This release updated Vector to use a new endpoint in Datadog which unfortunately accepts a much smaller payload size. Vector is optimised for the larger size. The result is now Datadog is rejecting the batches being sent in.

You have two options:

1. Don't use v0.34.0. It looks like you have downgraded already. Keep it like that, we will get a patch release out shortly.
2. Use v0.34.0, but configure Vector to use the old V1 endpoint. This can be done by setting an environment variable VECTOR_TEMP_USE_DD_METRICS_SERIES_V1_API=1

Sorry for the inconvenience.

[a26nine]

Ok, we have figured out what is likely going on. This release updated Vector to use a new endpoint in Datadog which unfortunately accepts a much smaller payload size. Vector is optimised for the larger size. The result is now Datadog is rejecting the batches being sent in.

You have two options:

1. Don't use v0.34.0. It looks like you have downgraded already. Keep it like that, we will get a patch release out shortly.
2. Use v0.34.0, but configure Vector to use the old V1 endpoint. This can be done by setting an environment variable VECTOR_TEMP_USE_DD_METRICS_SERIES_V1_API=1

Sorry for the inconvenience.

Thanks for looking into this promptly.

I have a couple of questions:

• Will the patch have any impact on the batch size limits? And will the patch be released anytime soon?
• Will the v1 endpoint work fine without breaking anything? Was v1 being used till 0.33.x?

[jszwedko]

Just a small update: we are digging into this now to come up with a correct fix. Thanks for the report!

I have a couple of questions:

• Will the patch have any impact on the batch size limits? And will the patch be released anytime soon?

The patch will likely change the batch limits for the v2 endpoints

• Will the v1 endpoint work fine without breaking anything? Was v1 being used till 0.33.x?

The v1 endpoints were in-use up until v0.34.0 so the flag just reverts the behavior back to what you were using prior to the upgrade.

[a26nine]

Just a small update: we are digging into this now to come up with a correct fix. Thanks for the report!

I have a couple of questions:

• Will the patch have any impact on the batch size limits? And will the patch be released anytime soon?

The patch will likely change the batch limits for the v2 endpoints

• Will the v1 endpoint work fine without breaking anything? Was v1 being used till 0.33.x?

The v1 endpoints were in-use up until v0.34.0 so the flag just reverts the behavior back to what you were using prior to the upgrade.

Okay. Looking forward to the patch.

[neuronull]

These are the steps to repro this issue:

• Generate the input events file
• Start vector v0.34.0 using the vector.toml config which accepts statsd metrics and sends to Datadog.
• Send metrics to Vector
• Observe the 413 errors reported in the vector debug logs , and lack of metrics in the DD UI

Generating the input file

The metrics must be unique so as not to be aggregated.

for i in {1..6666000}
do
  echo "foo:1|c|#tag:$i" >> events.txt
done

Vector config

[sources.in]
type = "statsd"
address = "0.0.0.0:9000"
mode = "tcp"
path = "/tmp/tmp.socket"

[sinks.dd]
inputs = [ "in" ]
type = "datadog_metrics"
default_api_key="${TEST_DATADOG_API_KEY}"
batch.max_events = 8000
batch.timeout_secs = 999999
buffer.type =  "disk"
buffer.max_size = 4294967296
buffer.when_full = "drop_newest"

Start Vector

(Debug log level needs to be set in order to confirm the 413 HTTP status code)

vector -c ./vector.toml -v

Sending the metrics

socat -dd OPEN:events.txt TCP:localhost:9000

[neuronull]

👋 hi @a26nine , have an update on this- the fix/patch will roll back to using the v1 endpoint. We have remaining work to do to sort out some strange performance we saw with the v2 endpoint payload limits. We will plan to get that ironed out and well tested for v0.35.0.

[a26nine]

👋 hi @a26nine , have an update on this- the fix/patch will roll back to using the v1 endpoint. We have remaining work to do to sort out some strange performance we saw with the v2 endpoint payload limits. We will plan to get that ironed out and well tested for v0.35.0.

Thanks, @neuronull! Awaiting the patch release.

[a26nine]

@jszwedko @neuronull 0.34.1 is still broken. The metrics are not shipped to Datadog.

[neuronull]

@jszwedko @neuronull 0.34.1 is still broken. The metrics are not shipped to Datadog.

Hi @a26nine, would you mind opening a new issue with your config and include debug logs? I just re-tested 0.34.1 again with the payload limits test and am still seeing it go through to Datadog (and on the v1 endpoint).

[a26nine]

@jszwedko @neuronull 0.34.1 is still broken. The metrics are not shipped to Datadog.

Hi @a26nine, would you mind opening a new issue with your config and include debug logs? I just re-tested 0.34.1 again with the payload limits test and am still seeing it go through to Datadog (and on the v1 endpoint).

I noticed that the uri in 0.34.1 changed to uri=https://0-34-1-vector.agent.datadoghq.com/api/v1/series. This doesn't look right. Also, it's passing API key from the Datadog Agent source in the header instead of reading it from the sink config.

0.33.1

vector[33763]: 2023-11-23T11:40:41.029584Z DEBUG sink{component_kind="sink" component_id=dd_sink_dd component_type=datadog_metrics component_name=dd_sink_dd}:request{request_id=28}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://0-33-1-vector.agent.datadoghq.com/api/v1/series method=POST version=HTTP/1.1 headers={"dd-api-key": "VECTOR_SINK_CONFIG_KEY", "dd-agent-payload": "4.87.0", "content-type": "application/json", "content-encoding": "deflate", "user-agent": "Vector/0.33.1 (x86_64-unknown-linux-gnu 3cc27b9 2023-10-30 16:50:49.747931844)", "accept-encoding": "identity"} body=[19568 bytes]

0.34.1

vector[31541]: 2023-11-23T11:30:40.904288Z DEBUG sink{component_kind="sink" component_id=dd_sink_dd component_type=datadog_metrics}:request{request_id=14}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://0-34-1-vector.agent.datadoghq.com/api/v1/series method=POST version=HTTP/1.1 headers={"dd-api-key": "DD_AGENT_CONFIG_KEY", "dd-agent-payload": "4.87.0", "content-type": "application/json", "content-encoding": "deflate", "user-agent": "Vector/0.34.1 (x86_64-unknown-linux-gnu 86f1c22 2023-11-16 14:59:10.486846964)", "accept-encoding": "identity"} body=[21792 bytes]

[a26nine]

The configuration and everything else is same as shared in the OP. It's just the difference of versions. I toggled between versions multiple times to confirm the issue.

[bruceg]

I think I see what you are referencing, @a26nine, but that seems like a trivial difference:

uri=https://0-33-1-vector.agent.datadoghq.com/api/v1/series
uri=https://0-34-1-vector.agent.datadoghq.com/api/v1/series

The only other difference between the two is the size in bytes, which could easily be explained by different underlying events going through.

[neuronull]

Also, it's passing API key from the Datadog Agent source in the header instead of reading it from the sink config

That is intended/expected behavior that has been unchanged for a few releases.

https://vector.dev/docs/reference/configuration/sinks/datadog_metrics/#default_api_key