feat: add timeouts, max payload size

[ramonpetgrave64]

Summary

• Adds timeouts for the GRPC server.
• Adds max payload size, and metrics for the payload sizes

The timeout can be configured at CLI with --grpc-timeout=30s, for example.

We use this value for both when establishing a connection and when the connection is established but idle.

• https://grpc.io/docs/guides/keepalive/#keepalive-configuration-specification

Testing Process

No new unit tests added against these changes, specifically. It is difficult to test the timeouts, but we can manually test the max payload size.

payload_size=$((5 * 1024 * 1024)) 
payload_file="payload.json"
random_data=$(head -c "$payload_size" /dev/urandom | base64)
json_payload=$(cat <<EOF
{
  "data": "$random_data"
}
EOF
)
echo "$json_payload" > "$payload_file"
curl -X POST \
  -H "Content-Type: application/json" \
  -d @- \
  http://localhost:3000/api/v2/log/entries < payload.json 

{"code":3, "message":"http: request body too large", "details":[]}%   

TODO:

• since the grpc server is fronted by an http proxy , determine if it makes sense to set the grpc server's connection and idle timeouts.
• determine a good MaxRecvMsgSize
• A way to test the timeouts and payload sizes

Release Note

Adds timeouts, max payload size, and payload size metrics.

Documentation

[codecov]

Codecov Report

Attention: Patch coverage is 85.71429% with 8 lines in your changes missing coverage. Please review.

Project coverage is 34.51%. Comparing base (faee49c) to head (b1def17).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/rekor-server/app/serve.go	0.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #153      +/-   ##
==========================================
+ Coverage   33.95%   34.51%   +0.55%     
==========================================
  Files          35       35              
  Lines        2668     2703      +35     
==========================================
+ Hits          906      933      +27     
- Misses       1675     1683       +8     
  Partials       87       87              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[haydentherapper]

Discussion from 4/1 meeting: HTTP proxies to gRPC, so we should test what the error is when gRPC times out for an HTTP request. We can also configure the idle timeout for HTTP to match the gRPC timeout here, which is what we do in Fulcio here.

re: max receiving size, gRPC defaults to 4Mb. We have a hard limit of 128Mb at the load balancer - https://github.com/sigstore/public-good-instance/blob/33d9d6f53441ead73b2ba60dfa15c5d2329634b6/terraform/environments/staging/1-infrastructure/ingress.tf#L229

[haydentherapper]

For HTTP setting the max byte size - https://github.com/sigstore/fulcio/blob/0c254bdd4be4fa736d1e3e7e1a26616cd5734b48/cmd/app/http.go#L90

Whatever we set in gRPC should be matched in the HTTP server.

[haydentherapper]

can this be a flag value, max-request-body-size, and passed through configuration?

[ramonpetgrave64]

There is also the DefaultMaxHeaderBytes, which by default is 1MB, but some providers like GCP load balancer limit it to 8KB. Should we also make that configurable?

• https://pkg.go.dev/net/http#pkg-constants
• https://cloud.google.com/load-balancing/docs/https/custom-headers#:~:text=the%20same%20name.-,Limitations,KB%20or%2016%20request%20headers.

[haydentherapper]

That default seems fine and not something we'd need to modify.