Workers do not reconnect after TRANSIENT_FAILURE

[jwulf]

It looks like workers do not reconnect after a TRANSIENT_FAILURE with the pure JS Grpc implementation.

My hypothesis about this is that the channel goes to TRANSIENT_FAILURE, and it takes another gRPC call before the channel state switches.

In order to deal with this, I am changing the GrpcClient failure logic to wait for 5 seconds on a TRANSIENT_FAILURE, then resolve to reconnected and unblock further operations.

[jwulf]

Yeah, this is still an issue. It looks like this with DEBUG level logging:

00:06:50.898 | zeebe |  [cloud-test-task] DEBUG: Long poll loop. this.longPoll: 30s
00:06:50.900 | zeebe |  [cloud-test-task] INFO: Stalled on Grpc Error
00:06:50.900 | zeebe |  [cloud-test-task] INFO: Grpc Error: 14 UNAVAILABLE: Connection dropped
00:06:50.900 | zeebe |  [cloud-test-task] DEBUG: Start watching Grpc channel...
00:06:50.900 | zeebe |  [cloud-test-task] INFO: Grpc Channel State: IDLE
00:06:50.901 | zeebe |  [cloud-test-task] DEBUG: {"code":14,"details":"Connection dropped","metadata":{"internalRepr":{},"options":{}}}
00:06:50.901 | zeebe |  [cloud-test-task] DEBUG: Stream ended after 27.449 seconds
00:06:50.901 | zeebe |  [cloud-test-task] DEBUG: Activating Jobs...
00:06:50.901 | zeebe |  [cloud-test-task] DEBUG: Grpc Channel state: IDLE
00:06:50.901 | zeebe |  [cloud-test-task] DEBUG: Long poll loop. this.longPoll: 30s
00:07:22.152 | zeebe |  [cloud-test-task] DEBUG: {"code":0,"details":"","metadata":{"internalRepr":{},"options":{}}}
00:07:22.152 | zeebe |  [cloud-test-task] DEBUG: Stream ended after 31.254 seconds
00:07:22.152 | zeebe |  [cloud-test-task] DEBUG: Activating Jobs...
00:07:22.153 | zeebe |  [cloud-test-task] DEBUG: Long poll loop. this.longPoll: 30s

[gizmo84]

Same here. I try to deploy a WorkflowInstance with the zeebe brocker being down:

11:57:03.917 | zeebe |  INFO: [topology]: 14 UNAVAILABLE: No connection established
11:57:03.920 | zeebe |  INFO: [createWorkflowInstance]: 14 UNAVAILABLE: No connection established
11:57:04.923 | zeebe |  INFO: [topology]: Attempt 2 (max: 50).
11:57:04.926 | zeebe |  INFO: [topology]: 14 UNAVAILABLE: No connection established
11:57:04.933 | zeebe |  INFO: [createWorkflowInstance]: Attempt 2 (max: 50).
11:57:04.937 | zeebe |  INFO: [createWorkflowInstance]: 14 UNAVAILABLE: No connection established
...

Then I start the brocker but zeebe-node never reconnect.

[jwulf]

Fixed in 0.23.0