Mistake in a BackendTrafficPolicy causes all routes to return 404

[dghubble]

Description:

A colleage and I found that a subtle mistake in a single BackendTrafficPolicy can make envoy proxy instances return 404's for ALL routes.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: hellogo
  namespace: default
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: hello
  retry:
    numRetries: 1
    perRetry:
      backOff:
        baseInterval: 0s      # 0s breaks everything, 1s is ok

Repro steps:

Create a BackendTrafficPolicy as shown above. Nothing stops a developer setting baseInterval: 0s.

At first, nothing is wrong. Then, if you restart envoy proxies, you'll find ALL httproutes return 404s immediately. Logs show route_not_found for all requests but no mention of why or which resources causs this. Inspecting the raw envoy config via the admin portal, the dynamic_route_configs section is never generated (usually its populated).

To find the offending resource, we had to delete resources until discovering the problematic thing was this one BackendTrafficPolicy and this one value within it. Pretty scary to us. Questions:

• What values should be allowed in baseInterval?
• What validations can be done to stop misconfigurations like this?
• Supposing there are other (perhaps future) resource misconfigs / validation issues, how can those be scoped to avoid breaking all routes?
• How can a user identify the problematic resources, either in envoy-gateway or the envoy proxy? Here we had to guess and test

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Environment:

Include the environment like gateway version, envoy version and so on.

envoy-gateway: v1.2.5

Logs:

Include the access logs and the Envoy logs.

[arkodg]

looks like this one managed to escape all the checks, here's the error from the envoy proxy

[2025-01-25 03:01:44.281][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.route.v3.RouteConfiguration rejected: Proto constraint validation failed (RouteConfigurationValidationError.VirtualHosts[0]: embedded message failed validation | caused by VirtualHostValidationError.Routes[0]: embedded message failed validation | caused by RouteValidationError.Route: embedded message failed validation | caused by RouteActionValidationError.RetryPolicy: embedded message failed validation | caused by RetryPolicyValidationError.RetryBackOff: embedded message failed validation | caused by RetryBackOffValidationError.BaseInterval: **value must be greater than 0s**): name: "default/eg/http"

We have 3 levels of validation

1. Apply Time

• Validated by Kube API Server based on the CRD OpenAPI schema based off Kube-Builder and CEL tags , and config that fails validation is rejected

2. Runtime

• Validated by the gateway-api runner in Envoy Gateway
• More complex validations happen here and if any validation fails
  • A negative status is added to the Policy resource
  • A 500 Direct Response is attached to the targeted Route, implementing a fail closed state, so requests targeting this route will fail, and should be able to debug these faster

3. xDS Translation

• We run Validate on the xDs Resource which executes the proto validations for the envoy proxy defined resources, if this fails it should be logged in envoy-gateway. We plan on bubbling this up as a status too in the future.
  It should have been caught here, because we didn't add the must be greater than 0s validation anywhere but it wasn't and the config was pushed to envoy proxy which failed the entire route config

[arkodg]

@zhaohuabing any idea why the xDS validate didn't kick in ?

this issue can be fixed by adding a CEL validation for this case

[zhaohuabing]

@arkodg the validation is done in the ResourceVersionTable.AddXdsResource function, but Routes are added to the RouteConfiguration after tCtx.AddXdsResource(resourcev3.RouteType, xdsRouteCfg) is called. So the validations for RouteConfiguration are skipped.

gateway/internal/xds/translator/translator.go

Lines 427 to 441 in 549fdde

	if xdsRouteCfg == nil {
	xdsRouteCfg = &routev3.RouteConfiguration{
	IgnorePortInHostMatching: true,
	Name: httpListener.Name,
	}
	if err = tCtx.AddXdsResource(resourcev3.RouteType, xdsRouteCfg); err != nil {
	errs = errors.Join(errs, err)
	}
	}
	// Generate xDS virtual hosts and routes for the given HTTPListener,
	// and add them to the xDS route config.
	if err = t.addRouteToRouteConfig(tCtx, xdsRouteCfg, httpListener, metrics, http3Enabled); err != nil {
	errs = errors.Join(errs, err)

This may happen in other xDS validation as well. I'm going to send a PR to fix it.

[zhaohuabing]

Created #5148 to add missing validations. The CEL validation/Gateway API translator validation for baseInterval can be addressed in a separate PR.

[dda104]

Same problem with all routes 404 when using filters

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: something
spec:
  parentRefs:
    - name: eg
      namespace: something
      sectionName: something
  hostnames:
    - something.com
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: something
          port: 123
          weight: 1
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: Host
                value: something.com
              - name: X-Forwarded-Proto
                value: https
              - name: X-Forwarded-Host
                value: something.com
      matches:
        - path:
            type: PathPrefix
            value: /

I understand that filters are not needed here and maybe they are written incorrectly, I'm just making a report that the problem in one httproute affects all httroutes in the cluster.