Back off on gRPC error 16 UNAUTHENTICATED

[jwulf]

At the moment, worker polling backs off on 14 UNAVAILABLE and 8 RESOURCE EXHAUSTED (Backpressure).

Other errors, such as 16 UNAUTHENTICATED, should also cause a backoff. Immediate retries of an unauthenticated request DOS the gateway.

This should be implemented here: https://github.com/camunda/camunda-8-js-sdk/blob/main/src/zeebe/lib/ZBWorkerBase.ts#L511

The question is whether all error conditions should trigger a backoff, or if 16 should just be added to the list.

Intermittent network disruption, or waiting for a gateway to come up are examples where backing off reduces the recovery time of the system, but without protecting a gateway from DOS.

I think in the first instance, I'll add 16 UNAUTHENTICATED to the backoff conditions.

Related: SUPPORT-25519

[jwulf]

At the moment, client operations retry once on unauthenticated, and then fail:

camunda-8-js-sdk/src/zeebe/zb/ZeebeGrpcClient.ts

Line 1520 in fe2abfe

if (isAuthError && authFailures === 0) {

So, making the worker progressively backoff is the way to go.

I'm not sure that there is a scenario where an unauthenticated worker can become authenticated without restarting the application.

But the log messages from a failing worker may help a developer / production understand what is happening. So continuing to emit them, but a decreasing rate is probably useful (and easier to implement than any other behaviour, because we already have backoff).

[jwulf]

By default, the job worker will back off exponentially to a maximum of 16s (16000 ms). You can tune that limit by setting CAMUNDA_JOB_WORKER_MAX_BACKOFF_MS via environment variable or the Camunda 8 client constructor parameters.

The Job Worker also emits streamError and backoff events.

The backoff event lets you know what the backoff period is.