Update Audit Logging documentation for 1.10

Signed-off-by: Mik Vyatskov <vmik@google.com>

Author: Mik Vyatskov

docs/tasks/debug-application-cluster/audit.md

@@ -72,6 +72,9 @@ In both cases, audit events structure is defined by the API in the
 `audit.k8s.io` API group. The current version of the API is
 [`v1beta1`][auditing-api].
 
+**Note:** in case of patches, request objects are arrays, containing operations, not the appropriate
+API objects. You should design your system to behave correctly in this case.
+
 ### Log backend
 
 Log backend writes audit events to a file in JSON format. You can configure
@@ -91,14 +94,56 @@ audit backend using the following kube-apiserver flags:
 
 - `--audit-webhook-config-file` specifies the path to a file with a webhook
   configuration. Webhook configuration is effectively a [kubeconfig][kubeconfig].
-- `--audit-webhook-mode` define the buffering strategy, one of the following:
-  - `batch` - buffer events and asynchronously send the set of events to the external service
-    This is the default
-  - `blocking` - block API server responses on sending each event to the external service
+- `--audit-webhook-initial-backoff` specifies the amount of time to wait after the first failed
+  request before retrying. From the second request and so on, exponential backoff is used.
 
 The webhook config file uses the kubeconfig format to specify the remote address of
 the service and credentials used to connect to it.
 
+### Batching
+
+Both log and webhook backends support batching. Using webhook as an example, here's the list of
+available flags. To get the same flag for log backend, replace `webhook` with `log` in the flag
+name. By default, batching is enabled in `webhook` and disabled in `log`. Similarly, by default
+throttling is enabled in `webhook` and disabled in `log`.
+
+- `--audit-webhook-mode` define the buffering strategy, one of the following:
+  - `batch` - buffer events and asynchronously process then in batches. This is the default
+  - `blocking` - block API server responses on processing each individual event
+
+The following flags are only used in the `batch` mode.
+
+- `--audit-webhook-batch-buffer-size` defines the size of the buffer to store events before batching.
+  If the rate of incoming events is too high and the buffer is overflown, events are dropped
+- `--audit-webhook-batch-max-size` defines the maximum number of events in one batch
+- `--audit-webhook-batch-max-wait` defines the maximum amount of time to wait before unconditionally
+  batching events in the queue
+- `--audit-webhook-batch-throttle-qps` defines the maximum average number of batches generated
+  per second
+- `--audit-webhook-batch-throttle-burst` defines the maximum number of batches generated per second
+  if the allowed QPS was underutilized previously
+
+#### How to set up parameters
+
+Parameters should be set to accommodate your load on the apiserver.
+
+For example, if you receive 100 requests to the kube-apiserver each second, and each request is
+audited only on `StageResponseStarted` and `StageResponseComplete` stages, you should account for
+~200 audit events being generated each second. Assuming that you have up to 100 events in a batch,
+you should set throttling level at at least 2 QPS. Assuming that the backend can take up to
+5 seconds to write events, you should set the buffer size to hold up to 5 seconds of events, i.e.
+10 batches, i.e. 1000 events.
+
+In most cases however, the default parameters should be sufficient and you don't have to worry about
+setting them manually. You can look at the Prometheus metric `apiserver_audit_error_total` and
+in the logs to monitor the state of the auditing subsystem.
+
+## Multi-cluster setup
+
+If you're extending the Kubernetes API with the [aggregation layer][kube-aggregator], you can also
+set up audit logging for the aggregated apiserver. To do this, pass the same configuration options
+to the aggregated apiserver and set up the log ingesting pipeline to pick up audit logs.
+
 ## Log Collector Examples
 
 ### Use fluentd to collect and distribute audit events from log file
@@ -250,8 +295,8 @@ plugin which supports full-text search and analytics.
 
 ## Legacy Audit
 
-__Note:__ Legacy Audit is deprecated and is disabled by default since Kubernetes 1.8.
-To fallback to this legacy audit, disable the advanced auditing feature
+__Note:__ Legacy Audit is deprecated and is disabled by default since Kubernetes 1.8. Legacy Audit
+will be removed in 1.12. To fallback to this legacy audit, disable the advanced auditing feature
 using the `AdvancedAuditing` feature gate in [kube-apiserver][kube-apiserver]:
 
 ```
@@ -299,3 +344,4 @@ and `audit-log-maxage` options.
 [fluentd_install_doc]: http://docs.fluentd.org/v0.12/articles/quickstart#step1-installing-fluentd
 [logstash]: https://www.elastic.co/products/logstash
 [logstash_install_doc]: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
+[kube-aggregator]: /docs/concepts/api-extension/apiserver-aggregation