feat: Add graceful shutdown timer to GRPC frontend

[mattwittwer]

What does the PR do?

This PR adds a shutdown timer to the gRPC endpoint for both infer and streaming infer requests. Inflight requests will be allowed to complete before the timer expires and new requests made after shutdown has started will be rejected.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

N/A

Where should the reviewer start?

Start with the new functions in src/grpc/grpc_server.cc and the flow in src/main.cc
GracefulStop();
unload the models in core
Stop();

Test plan:

Review L0_lifecycle and grpc related tests.
Updated shutdown behavior changes the tested responses for some lifecycle tests

• CI Pipeline ID: 24730709

Caveats:

The updated shutdown process includes three responses. The CANCELLED state cannot be removed as it is part of the GRPC endpoint shutdown behavior:

• "GRPC server is shutting down and has stopped accepting new requests."
• "CANCELLED"
• "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: connect: Connection refused (111)"

Background

Shutdown behavior for the gRPC endpoint could lead to unexpected errors when unloading the models from the core

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

[krishung5]

Do we need a lock for accepting_new_conn_?

[mattwittwer]

Added lock: conn_mtx_
For both accepting_new_conn_ and cq_shutdown_

[tanmayv25]

Alongwith reading requests and returning error to the client, I would advice we also prevent calling

service_->RequestModelInfer(
      state->context_->ctx_.get(), &state->request_,
      state->context_->responder_.get(), cq_, cq_, state);

in L671.
The idea is that we should avoid additional activity on the completion queue once the server shutdown is detected. We should just go through the inflight requests and drain their state objects before exiting the server to prevent any memory leak.

When the “Graceful shutdown function” is called, the server immediately notifies all clients to stop sending new RPCs. Then after the clients have received that notification, the server will stop accepting new RPCs.

Source: https://grpc.io/docs/guides/server-graceful-stop/

[mattwittwer]

Moved the forceful shutdown until after the core unloads. This prevents responses from returning from the core once the forceful shutdown has initiated. Completion queues are unloaded only after the forceful shutdown is complete

[krishung5]

Question - why would we still need the forceful shutdown of GRPC while we have the graceful shutdown?

[mattwittwer]

The graceful shutdown GracefulStop() rejects any new GRPC requests but does not stop ongoing requests. It waits until the timeout completes and then returns but the GRPC server may still exist. Unloading the triton server core first prevents writing back to unloaded states in the GRPC server.

The forceful shutdown in Stop() then cancels any remaining requests and stops the GRPC server if it hasn't stopped already.