Skip to content

Commit 5751bff

Browse files
jeropjerop
committed
TEP-0127: Larger Results via Sidecar Logs
Today, `Results` have a size limit of 4KB per `Step` and 12KB per `TaskRun` in the best case - see [issue][4012]. The goal of [TEP-0086][tep-0086] is to support larger `Results` beyond the current size limits. TEP-0086 has many alternatives but no proposal. This TEP is proposes experimenting with one of the alternatives - `Sidecar` logs. This allows us to support larger `Results` that are stored within `TaskRun` CRDs. [4012]: tektoncd/pipeline#4012 [4808]: tektoncd/pipeline#4808 [tep-0086]: https://github.com/tektoncd/community/blob/main/teps/0086-changing-the-way-result-parameters-are-stored.md
1 parent 524ca13 commit 5751bff

File tree

2 files changed

+288
-0
lines changed

2 files changed

+288
-0
lines changed

‎teps/0127-larger-results-via-sidecar-logs.md

+287
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,287 @@
1+
---
2+
status: implementable
3+
title: Larger Results via Sidecar Logs
4+
creation-date: '2022-11-30'
5+
last-updated: '2022-12-01'
6+
authors:
7+
- '@chitrangpatel'
8+
- '@jerop'
9+
see-also:
10+
- TEP-0086
11+
---
12+
13+
# TEP-0127: Larger Results via Sidecar Logs
14+
15+
<!-- toc -->
16+
- [Summary](#summary)
17+
- [Motivation](#motivation)
18+
- [Goals](#goals)
19+
- [Non-Goals](#non-goals)
20+
- [Use Cases](#use-cases)
21+
- [Proposal](#proposal)
22+
- [Feature Gate](#feature-gate)
23+
- [Logs Access](#logs-access)
24+
- [Size Limit](#size-limit)
25+
- [PipelineRun Status](#pipelinerun-status)
26+
- [Design Details](#design-details)
27+
- [Design Evaluation](#design-evaluation)
28+
- [Reusability](#reusability)
29+
- [Simplicity](#simplicity)
30+
- [Performance](#performance)
31+
- [Security](#security)
32+
- [Experiment Questions](#experiment-questions)
33+
- [References](#references)
34+
<!-- /toc -->
35+
36+
This TEP builds on the hard work of many people who have been tackling the problem over the past couple years,
37+
including but not limited to:
38+
- '@abayer'
39+
- '@afrittoli'
40+
- '@bobcatfish'
41+
- '@dibyom'
42+
- '@imjasonh'
43+
- '@pritidesai'
44+
- '@ScrapCodes'
45+
- '@skaegi'
46+
- '@tlawrie'
47+
- '@tomcli'
48+
- '@vdemeester'
49+
- '@wlynch'
50+
51+
## Summary
52+
53+
Today, `Results` have a size limit of 4KB per `Step` and 12KB per `TaskRun` in the best case - see [issue][4012].
54+
55+
The goal of [TEP-0086][tep-0086] is to support larger `Results` beyond the current size limits. TEP-0086 has many
56+
alternatives but no proposal. This TEP is proposes experimenting with one of the alternatives - `Sidecar` logs.
57+
This allows us to support larger `Results` which are stored within `TaskRun` CRDs.
58+
59+
## Motivation
60+
61+
`Results` are too small - see [issue][4012]. The current implementation of `Results` involves parsing from disk and
62+
storing as part of the `Termination Message` which has a limit of 4KB per `Container` and 12KB per `Pod`. As such,
63+
the size limit of `Results` is 12KB per `TaskRun` and 4KB per `Step` at best.
64+
65+
To make matters worse, the limit is divided equally among all `Containers` in a `Pod` - see [issue][4808]. Therefore,
66+
the more the `Steps` in a `Task` the less the size limit for `Results`. For example, if there are 12 `Steps` then each
67+
has 1KB in `Termination Message` storage to produce `Results`.
68+
69+
[TEP-0086][tep-0086] aims to support larger `Results`. [TEP-0086][tep-0086] has many [alternatives][tep-0086-alts] with
70+
no proposal because there's no obvious "best" solution that would meet all the requirements.
71+
72+
This TEP proposes experimenting with `Sidecar` logs to support larger `Results` that are stored within `TaskRun` CRDs.
73+
This allows us to provide an immediate solution to the urgent needs of users, while not blocking pursuit of the other
74+
alternatives.
75+
76+
In addition, the [documented guidance][docs] is that `Results` are used for outputs less than 1KB while `Workspaces`
77+
are used for larger data. Supporting larger `Results` up to the CRD limit allows users to reuse `Tasks` in more
78+
scenarios without having to change the specification to use `Workspaces` upon hitting the current low size limit of
79+
4KB per `TaskRun`.
80+
81+
> As a general rule-of-thumb, if a `Result` needs to be larger than a kilobyte, you should likely use a `Workspace`
82+
to store and pass it between `Tasks` within a `Pipeline`.
83+
84+
### Goals
85+
86+
The main goal is to support larger `Results` that are limited by the size of a `TaskRun` CRD (1.5MB) via `Sidecar` logs.
87+
88+
### Non-Goals
89+
90+
The following are out of scope for this TEP:
91+
92+
1. Solving use cases that requires really large `Results` beyond the size limit of a CRD (1.5MB).
93+
94+
2. Addressing other [alternatives][tep-0086-alts] for larger `Results` that are listed in [TEP-0086][tep-0086].
95+
96+
### Use Cases
97+
98+
1. Support signing `Results` using SPIRE for non-falsifiable provenance that's required for [SLSA L3][L3]. As described
99+
in [TEP-0089][tep-0089], the signatures and certificates used to verify `Results` are stored alongside the `Results`
100+
themselves in the `Termination Message` which further exacerbates the size limit issues. The size of the certificate
101+
is 800 bytes and the size of signatures is approximately 100 bytes per `Result`.
102+
103+
> For now, signatures of the `Results` will be contained within the `Termination Message` of the `Pod`, alongside
104+
any additional material required to perform verification. One consideration of this is the size of the additional
105+
fields required. The size of the certificate needed for verification is about 800 bytes, and the size of the
106+
signatures is about 100 bytes * (number of `Results` + 1). The current `Termination Message` size is 4K, but there
107+
is TEP-0086 looking at supporting larger results.
108+
109+
2. Support emitting structured `Results`. For example, store the images produced by `ko` and all their copies to the
110+
various regional registries in a `TaskRun`. The release `Pipeline` has this use case, and it ran into the current
111+
size limit of 4096 bytes. As described in the [issue][4282], the images produced were 4572 bytes which didn't fit
112+
into `Results`. For further details about structured `Results`, see [TEP-0075][tep-0075] and [TEP-0076][tep-0076].
113+
114+
## Proposal
115+
116+
We propose using stdout logs from a dedicated `Sidecar` to return a json `Result` object to support larger `Results`.
117+
The `Pipeline` controller would wait for the `Sidecar` to exit and then read the logs based on a particular query and
118+
append `Results` to the `TaskRun`.
119+
120+
We propose injecting the dedicated `Sidecar` alongside other `Steps`. The `Sidecar` will watch the `Results` paths of
121+
the `Steps`. This `Sidecar` will output the name of the `Result` and its contents to stdout in a parsable pattern. The
122+
`TaskRun` controller will access the stdout logs of the `Sidecar` then extract the `Results` and its contents.
123+
124+
After the `Steps` have terminated, the `TaskRun` controller will write the `Results` from the logs of the `Sidecar`
125+
instead of using the `Termination Message`, hence bypassing the 4KB limit. This method keeps the rest of the current
126+
functionality identical and does not require any external storage mechanism.
127+
128+
For further details, see the [demonstration][demo] of the [implementation][poc].
129+
130+
This provides an opportunity to experiment with this solution to provide `Results` within the CRDs as we continue to
131+
explore other alternatives, including those that involve external storage.
132+
133+
### Feature Gate
134+
135+
This feature will be gated using a `result-extraction-method` feature flag.
136+
137+
Users have to set `result-extraction-method` to `"sidecar-logs"` to enable the feature:
138+
```shell
139+
kubectl patch cm feature-flags -n tekton-pipelines -p '{"data":{"result-extraction-method":"sidecar-logs"}}'
140+
```
141+
142+
This feature is disabled by default. When disabled, `Results` continue to pass through `Termination Message`.
143+
144+
### Logs Access
145+
146+
This feature requires that the `Pipeline` controller has access to `Pod` logs.
147+
148+
Users have to grant `get` access to all `pods/log` to the `Pipeline` controller:
149+
```shell
150+
kubectl apply -f config/enable-log-access-to-controller/
151+
```
152+
153+
### Size Limit
154+
155+
The size limit per `Result` can be configured using the `max-result-size` feature flag, which takes the integer value
156+
of the bytes.
157+
158+
The `max-result-size` feature flag defaults to 4096 bytes. This ensures that we support existing `Tasks` with only one
159+
`Result` that uses up the whole size limit of the `Termination Message`.
160+
161+
If users want to set the size limit per `Result` to be something other than 4096 bytes, they can set `max-result-size`
162+
by setting `max-result-size: <VALUE-IN-BYTES>`. The value set here cannot exceed the CRD size limit of 1.5MB.
163+
164+
```shell
165+
kubectl patch cm feature-flags -n tekton-pipelines -p '{"data":{"max-result-size":"<VALUE-IN-BYTES>"}}'
166+
```
167+
168+
Even though the size limit per `Result` is configurable, the size of `Results` are limited by CRD size limit of 1.5MB.
169+
If the size of `Results` exceeds this limit, then the `TaskRun` will fail with a message indicating the size limit has
170+
been exceeded.
171+
172+
### PipelineRun Status
173+
174+
In [TEP-0100][tep-0100], we proposed changes to `PipelineRun` status to reduce the amount of information stored about
175+
the status of `TaskRuns` and `Runs`. As a result, the `PipelineRun` status is set up to handle larger `Results` without
176+
exacerbating the storage issues that were there before.
177+
178+
For `ChildReferences` to be populated, the `embedded-status` must be set to `"minimal"`. We recommend that the minimal
179+
embedded status - `ChildReferences` - is enabled while migration is ongoing until it becomes the only supported status.
180+
This will ensure that larger `Results` from all its `TaskRuns` will not bloat the `PipelineRun` CRD.
181+
182+
## Design Details
183+
184+
The `Sidecar` will run a binary that:
185+
- receives argument for `Results`' paths and names which are identified from `taskSpec.results` field. This allows the
186+
`Sidecar` to know the `Results` it needs to read.
187+
- has `/tekton/run` volume mounted as [read-only][4227] where status of each `Step` is written.
188+
- periodically checks for `Step` status in the path `/tekton/run`.
189+
- when all `Steps` have completed, it immediately parses all the `Results` in paths and prints them to stdout in a
190+
parsable pattern.
191+
192+
For further details, see the [demonstration][demo] of the [implementation][poc].
193+
194+
## Design Evaluation
195+
196+
### Reusability
197+
198+
This proposal does not introduce any API changes to specification `Results`. The changes are in implementation details
199+
of `Results`. The existing `Tasks` will continue to function as they are, only that they can support larger `Results`.
200+
201+
Even more, supporting larger `Results` upto the CRD limit allows users to reuse `Tasks` in more scenarios without
202+
having to change the specification to use `Workspaces` upon hitting the current low size limit of 4KB per `TaskRun`.
203+
This allows users to control execution as needed by their context without having to modify `Tasks` and `Pipelines`.
204+
205+
### Interoperability
206+
207+
Users may write `Tasks` that assume larger `Results` support. These `Tasks` would only work on *Tekton Pipelines*
208+
installations that are configured to support it. This is a risk to `Task` interoperability which is mitigated by:
209+
1. We still have a hard limit on the size of `Results` which is the CRD size limit.
210+
2. We still plan to support larger `Results` in the long run regardless of the implementation details.
211+
212+
### Simplicity
213+
214+
This proposal provides a simple solution that solves most use cases:
215+
- Users don't need additional infrastructure, such as server or object storage, to support larger `Results`.
216+
- Existing `Tasks` will continue to function as they do now, while supporting larger `Results`, without any API changes.
217+
218+
### Performance
219+
220+
Performance benchmarking with 20-30 `PipelineRuns`, each with 3 `TaskRuns` each with two `Steps`:
221+
- Average `Pipeline` controller's CPU difference during `PipelineRun` execution: 1%
222+
- Average `Pipeline` controller's Memory usage difference during `PipelineRun` execution: 0.2%
223+
- Average `Pod` startup time (time to get to running state) difference: 3s per `TaskRun`
224+
225+
In the experiment, we will continue to measure the startup overhead and explore ways to improve it.
226+
227+
For further details, see the [performance metrics][performance].
228+
229+
### Security
230+
231+
This approach requires that the `Pipeline` controller has access to `Pod` logs. The `Pipeline` controller already has
232+
extensive permissions in the cluster, such as read access to `Secrets`. Expanding the access even further is a concern
233+
for some users, but is also acceptable for some users given the advantages. We will document the extended permissions
234+
so that users can make the right choice for their own use cases and requirements.
235+
236+
When users have enabled larger `Results` via `Sidecar` logs, there's a validation check to ensure that users cannot
237+
inject their own `Sidecar` overtop of the one specified by Tekton.
238+
239+
## Experiment Questions
240+
241+
These are some questions we plan to answer in the experiment:
242+
243+
- What impact does this change have on the startup and execution time of `TaskRuns` and `PipelineRuns`? Can we
244+
improve the performance impact?
245+
246+
- How reliable is using `Sidecar` logs to process `Results`?
247+
248+
- How many users adopt this solution? How many are satisfied with it given the advantages and disadvantages?
249+
We will conduct a user survey soon after the feature has been released.
250+
251+
## References
252+
253+
- Tekton Enhancement Proposals
254+
- [TEP-0075: Object Parameters and Results][tep-0075]
255+
- [TEP-0076: Array Results][tep-0076]
256+
- [TEP-0086: Larger Results][tep-0086]
257+
- [TEP-0089: Non-Falsifiable Provenance][tep-0089]
258+
- [TEP-0100: PipelineRun Status][tep-0100]
259+
- Issues:
260+
- [Changing the way Results are stored][4012]
261+
- [Results, TerminationMessage and Containers][4808]
262+
- [Publish task fails, IMAGES result too large][4282]
263+
- Implementation:
264+
- [Implementation Pull Request][poc]
265+
- [Demonstration by Chitrang][demo]
266+
- Prior Work:
267+
- [TEP-0086: Using logs emitted by the Task][tep-0086-logs]
268+
- [TEP-0086: Larger Results via Sidecar Logs][745]
269+
270+
[docs]: https://tekton.dev/docs/pipelines/tasks/#emitting-results
271+
[4012]: https://github.com/tektoncd/pipeline/issues/4012
272+
[4808]: https://github.com/tektoncd/pipeline/issues/4808
273+
[4282]: https://github.com/tektoncd/pipeline/issues/4282
274+
[4227]: https://github.com/tektoncd/pipeline/issues/4227
275+
[745]: https://github.com/tektoncd/community/pull/745
276+
[L3]: https://slsa.dev/spec/v0.1/levels
277+
[crd-size]: https://github.com/kubernetes/kubernetes/issues/82292
278+
[demo]: https://drive.google.com/file/d/1NrWudE_XBqweomiY24DP2Txnl1yN0yD9/view
279+
[poc]: https://github.com/tektoncd/pipeline/pull/5695
280+
[performance]: https://github.com/tektoncd/community/pull/745#issuecomment-1206668381
281+
[tep-0075]: ./0075-object-param-and-result-types.md
282+
[tep-0076]: ./0076-array-result-types.md
283+
[tep-0086]: ./0086-changing-the-way-result-parameters-are-stored.md
284+
[tep-0086-alts]: ./0086-changing-the-way-result-parameters-are-stored.md#alternatives
285+
[tep-0086-logs]: ./0086-changing-the-way-result-parameters-are-stored.md#using-logs-emitted-by-the-task
286+
[tep-0089]: ./0089-nonfalsifiable-provenance-support.md
287+
[tep-0100]: ./0100-embedded-taskruns-and-runs-status-in-pipelineruns.md

‎teps/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -292,3 +292,4 @@ This is the complete list of Tekton teps:
292292
|[TEP-0123](0123-specifying-on-demand-retry-in-pipelinetask.md) | Specifying on-demand-retry in a PipelineTask | proposed | 2022-10-11 |
293293
|[TEP-0124](0124-distributed-tracing-for-tasks-and-pipelines.md) | Distributed tracing for Tasks and Pipelines | implementable | 2022-10-16 |
294294
|[TEP-0125](0125-add-credential-filter-to-entrypoint-logger.md) | Add credential filter to entrypoint logger | proposed | 2022-10-27 |
295+
|[TEP-0127](0127-larger-results-via-sidecar-logs.md) | Larger Results via Sidecar Logs | implementable | 2022-12-01 |

0 commit comments

Comments
 (0)