Integrate custom tasks into Pipelines

A PipelineTask now can reference a custom task.  This causes the PipelineRun reconciler to create a Run instead of a TaskRun.
Pipeline parameters can be passed to the custom task.  The custom task's results can be used by other Pipeline tasks and by the Pipeline results.

Author: GregDritschler

config/config-feature-flags.yaml

@@ -82,3 +82,8 @@ data:
   # This is an experimental feature and thus should still be considered
   # an alpha feature.
   enable-tekton-oci-bundles: "false"
+  # Setting this flag to "true" enables the use of custom tasks from
+  # within pipelines.
+  # This is an experimental feature and thus should still be considered
+  # an alpha feature.
+  enable-custom-tasks: "false"

docs/install.md

@@ -353,6 +353,9 @@ not running.
 and use Workspaces to mount credentials from Secrets instead.
 The default is `false`. For more information, see the [associated issue](https://github.com/tektoncd/pipeline/issues/3399).
 
+- `enable-custom-tasks`: set this flag to `"true"` to enable the
+use of custom tasks in pipelines.
+
 For example:
 
 ```yaml

docs/pipelines.md

@@ -25,6 +25,7 @@ weight: 3
   - [Configuring the `Task` execution order](#configuring-the-task-execution-order)
   - [Adding a description](#adding-a-description)
   - [Adding `Finally` to the `Pipeline`](#adding-finally-to-the-pipeline)
+  - [Using Custom Tasks](#using-custom-tasks)
   - [Code examples](#code-examples)
 
 ## Overview
@@ -888,6 +889,90 @@ In this example, `PipelineResults` is set to:
 ],
 ```
 
+## Using Custom Tasks
+
+**Note: This is only allowed if `enable-custom-tasks` is set to
+`"true"` in the `feature-flags` configmap, see [`install.md`](./install.md#customizing-the-pipelines-controller-behavior)**
+
+[Custom Tasks](https://github.com/tektoncd/community/blob/master/teps/0002-custom-tasks.md)
+can implement behavior that doesn't correspond directly to running a workload in a `Pod` on the cluster.
+For example, a custom task might execute some operation outside of the cluster and wait for its execution to complete.
+
+A PipelineRun starts a custom task by creating a [`Run`](https://github.com/tektoncd/pipeline/blob/master/docs/runs.md) instead of a `TaskRun`.
+In order for a custom task to execute, there must be a custom task controller running on the cluster
+that is responsible for watching and updating `Run`s which reference their type. 
+If no such controller is running, those `Run`s will never complete and Pipelines using them will time out.
+
+Custom tasks are an **_experimental alpha feature_** and should be expected to change
+in breaking ways or even be removed.
+
+### Specifying the target Custom Task
+
+To specify the custom task type you want to execute, the `taskRef` field
+must include the custom task's `apiVersion` and `kind` as shown below:
+
+```yaml
+spec:
+  tasks:
+    - name: run-custom-task
+      taskRef:
+        apiVersion: example.dev/v1alpha1
+        kind: Example
+```
+
+This creates a `Run` of a custom task of type `Example` in the `example.dev` API group with the version `v1alpha1`.
+
+You can also specify the `name` of a custom task resource object previously defined in the cluster.
+
+```yaml
+spec:
+  tasks:
+    - name: run-custom-task
+      taskRef:
+        apiVersion: example.dev/v1alpha1
+        kind: Example
+        name: myexample
+```
+
+If the `taskRef` specifies a name, the custom task controller should look up the
+`Example` resource with that name and use that object to configure the execution.
+
+If the `taskRef` does not specify a name, the custom task controller might support
+some default behavior for executing unnamed tasks.
+
+### Specifying `Parameters`
+
+If a custom task supports [`parameters`](tasks.md#parameters), you can use the
+`params` field to specify their values:
+
+```yaml
+spec:
+spec:
+  tasks:
+    - name: run-custom-task
+      taskRef:
+        apiVersion: example.dev/v1alpha1
+        kind: Example
+        name: myexample
+      params:
+      - name: foo
+        value: bah
+```
+
+### Using `Results`
+
+If the custom task produces results, you can reference them in a Pipeline using the normal syntax,
+`$(tasks.<task-name>.results.<result-name>)`.
+
+### Limitations
+
+Pipelines do not directly support passing the following items to custom tasks:
+* Pipeline Resources
+* Workspaces
+* Service account name
+* Pod templates
+
+
 ## Code examples
 
 For a better understanding of `Pipelines`, study [our code examples](https://github.com/tektoncd/pipeline/tree/master/examples).

docs/runs.md

@@ -35,10 +35,6 @@ will have no `.status` value and no further action will be taken.
 `Run`s are an **_experimental alpha feature_** and should be expected to change
 in breaking ways or even be removed.
 
-`Run`s are not currently integrated with Pipelines, and require a running
-third-party controller to actually perform any work. Without a third-party
-controller, `Run`s will just exist without a status indefinitely.
-
 ## Configuring a `Run`
 
 A `Run` definition supports the following fields:

go.sum

@@ -58,6 +58,11 @@ ${PREFIX}/deepcopy-gen \
   --go-header-file ${REPO_ROOT_DIR}/hack/boilerplate/boilerplate.go.txt \
 -i github.com/tektoncd/pipeline/pkg/apis/resource/v1alpha1/storage
 
+${PREFIX}/deepcopy-gen \
+  -O zz_generated.deepcopy \
+  --go-header-file ${REPO_ROOT_DIR}/hack/boilerplate/boilerplate.go.txt \
+-i github.com/tektoncd/pipeline/pkg/apis/run/v1alpha1
+
 # Knative Injection
 # This generates the knative injection packages for the resource package (v1alpha1).
 # This is separate from the pipeline package for the same reason as client and all (see above).

hack/update-codegen.sh

@@ -32,13 +32,15 @@ const (
 	runningInEnvWithInjectedSidecarsKey     = "running-in-environment-with-injected-sidecars"
 	requireGitSSHSecretKnownHostsKey        = "require-git-ssh-secret-known-hosts" // nolint: gosec
 	enableTektonOCIBundles                  = "enable-tekton-oci-bundles"
+	enableCustomTasks                       = "enable-custom-tasks"
 	DefaultDisableHomeEnvOverwrite          = false
 	DefaultDisableWorkingDirOverwrite       = false
 	DefaultDisableAffinityAssistant         = false
 	DefaultDisableCredsInit                 = false
 	DefaultRunningInEnvWithInjectedSidecars = true
 	DefaultRequireGitSSHSecretKnownHosts    = false
 	DefaultEnableTektonOciBundles           = false
+	DefaultEnableCustomTasks                = false
 )
 
 // FeatureFlags holds the features configurations
@@ -51,6 +53,7 @@ type FeatureFlags struct {
 	RunningInEnvWithInjectedSidecars bool
 	RequireGitSSHSecretKnownHosts    bool
 	EnableTektonOCIBundles           bool
+	EnableCustomTasks                bool
 }
 
 // GetFeatureFlagsConfigName returns the name of the configmap containing all
@@ -99,6 +102,9 @@ func NewFeatureFlagsFromMap(cfgMap map[string]string) (*FeatureFlags, error) {
 	if err := setFeature(enableTektonOCIBundles, DefaultEnableTektonOciBundles, &tc.EnableTektonOCIBundles); err != nil {
 		return nil, err
 	}
+	if err := setFeature(enableCustomTasks, DefaultEnableCustomTasks, &tc.EnableCustomTasks); err != nil {
+		return nil, err
+	}
 	return &tc, nil
 }
 

pkg/apis/config/feature_flags.go

@@ -47,6 +47,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				RunningInEnvWithInjectedSidecars: false,
 				RequireGitSSHSecretKnownHosts:    true,
 				EnableTektonOCIBundles:           true,
+				EnableCustomTasks:                true,
 			},
 			fileName: "feature-flags-all-flags-set",
 		},

pkg/apis/config/feature_flags_test.go

@@ -24,3 +24,4 @@ data:
   running-in-environment-with-injected-sidecars: "false"
   require-git-ssh-secret-known-hosts: "true"
   enable-tekton-oci-bundles: "true"
+  enable-custom-tasks: "true"

pkg/apis/config/testdata/feature-flags-all-flags-set.yaml

@@ -18,14 +18,12 @@ package v1alpha1
 
 import (
 	"fmt"
-	"time"
 
 	"github.com/tektoncd/pipeline/pkg/apis/pipeline"
 	v1beta1 "github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1"
+	runv1alpha1 "github.com/tektoncd/pipeline/pkg/apis/run/v1alpha1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/runtime/schema"
-	"k8s.io/apimachinery/pkg/util/json"
 	"knative.dev/pkg/apis"
 	duckv1 "knative.dev/pkg/apis/duck/v1"
 )
@@ -75,85 +73,16 @@ func (rs RunSpec) GetParam(name string) *v1beta1.Param {
 	return nil
 }
 
-type RunStatus struct {
-	duckv1.Status `json:",inline"`
+const (
+	// RunReasonCancelled must be used in the Condition Reason to indicate that a Run was cancelled.
+	RunReasonCancelled = "RunCancelled"
+)
 
-	// RunStatusFields inlines the status fields.
-	RunStatusFields `json:",inline"`
-}
+// RunStatus defines the observed state of Run.
+type RunStatus = runv1alpha1.RunStatus
 
 var runCondSet = apis.NewBatchConditionSet()
 
-// GetCondition returns the Condition matching the given type.
-func (r *RunStatus) GetCondition(t apis.ConditionType) *apis.Condition {
-	return runCondSet.Manage(r).GetCondition(t)
-}
-
-// InitializeConditions will set all conditions in runCondSet to unknown for the PipelineRun
-// and set the started time to the current time
-func (r *RunStatus) InitializeConditions() {
-	started := false
-	if r.StartTime.IsZero() {
-		r.StartTime = &metav1.Time{Time: time.Now()}
-		started = true
-	}
-	conditionManager := runCondSet.Manage(r)
-	conditionManager.InitializeConditions()
-	// Ensure the started reason is set for the "Succeeded" condition
-	if started {
-		initialCondition := conditionManager.GetCondition(apis.ConditionSucceeded)
-		initialCondition.Reason = "Started"
-		conditionManager.SetCondition(*initialCondition)
-	}
-}
-
-// SetCondition sets the condition, unsetting previous conditions with the same
-// type as necessary.
-func (r *RunStatus) SetCondition(newCond *apis.Condition) {
-	if newCond != nil {
-		runCondSet.Manage(r).SetCondition(*newCond)
-	}
-}
-
-// MarkRunSucceeded changes the Succeeded condition to True with the provided reason and message.
-func (r *RunStatus) MarkRunSucceeded(reason, messageFormat string, messageA ...interface{}) {
-	runCondSet.Manage(r).MarkTrueWithReason(apis.ConditionSucceeded, reason, messageFormat, messageA...)
-	succeeded := r.GetCondition(apis.ConditionSucceeded)
-	r.CompletionTime = &succeeded.LastTransitionTime.Inner
-}
-
-// MarkRunFailed changes the Succeeded condition to False with the provided reason and message.
-func (r *RunStatus) MarkRunFailed(reason, messageFormat string, messageA ...interface{}) {
-	runCondSet.Manage(r).MarkFalse(apis.ConditionSucceeded, reason, messageFormat, messageA...)
-	succeeded := r.GetCondition(apis.ConditionSucceeded)
-	r.CompletionTime = &succeeded.LastTransitionTime.Inner
-}
-
-// MarkRunRunning changes the Succeeded condition to Unknown with the provided reason and message.
-func (r *RunStatus) MarkRunRunning(reason, messageFormat string, messageA ...interface{}) {
-	runCondSet.Manage(r).MarkUnknown(apis.ConditionSucceeded, reason, messageFormat, messageA...)
-}
-
-// DecodeExtraFields deserializes the extra fields in the Run status.
-func (r *RunStatus) DecodeExtraFields(into interface{}) error {
-	if len(r.ExtraFields.Raw) == 0 {
-		return nil
-	}
-	return json.Unmarshal(r.ExtraFields.Raw, into)
-}
-
-// EncodeExtraFields serializes the extra fields in the Run status.
-func (r *RunStatus) EncodeExtraFields(from interface{}) error {
-	data, err := json.Marshal(from)
-	if err != nil {
-		return err
-	}
-	r.ExtraFields = runtime.RawExtension{
-		Raw: data,
-	}
-	return nil
-}
-
 // GetConditionSet retrieves the condition set for this resource. Implements
 // the KRShaped interface.
 func (r *Run) GetConditionSet() apis.ConditionSet { return runCondSet }
@@ -165,24 +94,10 @@ func (r *Run) GetStatus() *duckv1.Status { return &r.Status.Status }
 // RunStatusFields holds the fields of Run's status.  This is defined
 // separately and inlined so that other types can readily consume these fields
 // via duck typing.
-type RunStatusFields struct {
-	// StartTime is the time the build is actually started.
-	// +optional
-	StartTime *metav1.Time `json:"startTime,omitempty"`
-
-	// CompletionTime is the time the build completed.
-	// +optional
-	CompletionTime *metav1.Time `json:"completionTime,omitempty"`
-
-	// Results reports any output result values to be consumed by later
-	// tasks in a pipeline.
-	// +optional
-	Results []v1beta1.TaskRunResult `json:"results,omitempty"`
+type RunStatusFields = runv1alpha1.RunStatusFields
 
-	// ExtraFields holds arbitrary fields provided by the custom task
-	// controller.
-	ExtraFields runtime.RawExtension `json:"extraFields,omitempty"`
-}
+// RunResult used to describe the results of a task
+type RunResult = runv1alpha1.RunResult
 
 // +genclient
 // +genreconciler