RFE: Cluster builds with kaniko: allow to set securityContext / serviceAccountName

[Andrei-Stepanov]

Hello.

It would be good to allow user specify securityContext / serviceAccountName for Kaniko builds.

Expected behavior

1. Kaniko pod can build images on OpenShift.
2. Modify base-images. Build as root.
   For example have a Dockerfile with: RUN yum install --assumeyes openssl-devel

Actual behavior

1. Kaniko-pod fails to build any image on OpenShift with error:

  | November 18th 2019, 12:42:02.074 | �[37mDEBU�[0m[0000] Copying file /kaniko/buildcontext/Dockerfile to /kaniko/Dockerfile
  | November 18th 2019, 12:42:02.074 | �[37mDEBU�[0m[0000] Getting source context from dir:///kaniko/buildcontext
  | November 18th 2019, 12:42:02.074 | Error: error resolving dockerfile path: copying dockerfile: open /kaniko/Dockerfile: permission denied

Information

skaffold version
f2038fa 
on Centos 7
kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1+5115d708d7", GitCommit:"fff65cf", GitTreeState:"clean", BuildDate:"2017-11-03T15:44:55Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-03-19T15:41:07Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
    - image: XXX
      kaniko:
        flags:
          - --verbosity=debug
        buildContext:
          localDir: {}
        cache: {}
  cluster:
    namespace: YYY
deploy:
  kubectl:
    manifests:
      - k8s-*

➜  skaffold -v debug build
INFO[0000] Skaffold &{Version:f2038fa ConfigVersion:skaffold/v1 GitVersion: GitCommit:f2038fa0592045e7dee0a57f896eb07ef02d9e89 GitTreeState:clean BuildDate:2019-11-13T18:12:53Z GoVersion:go1.12.13 Compiler:gc Platform:linux/amd64} 
DEBU[0000] validating yamltags of struct SkaffoldConfig 
DEBU[0000] validating yamltags of struct Metadata       
DEBU[0000] validating yamltags of struct Pipeline       
DEBU[0000] validating yamltags of struct BuildConfig    
DEBU[0000] validating yamltags of struct Artifact       
DEBU[0000] validating yamltags of struct ArtifactType   
DEBU[0000] validating yamltags of struct KanikoArtifact 
DEBU[0000] validating yamltags of struct KanikoBuildContext 
DEBU[0000] validating yamltags of struct LocalDir       
DEBU[0000] validating yamltags of struct KanikoCache    
DEBU[0000] validating yamltags of struct TagPolicy      
DEBU[0000] validating yamltags of struct GitTagger      
DEBU[0000] validating yamltags of struct BuildType      
DEBU[0000] validating yamltags of struct ClusterDetails 
DEBU[0000] validating yamltags of struct DeployConfig   
DEBU[0000] validating yamltags of struct DeployType     
DEBU[0000] validating yamltags of struct KubectlDeploy  
DEBU[0000] validating yamltags of struct KubectlFlags   
INFO[0000] Using kubectl context: osci-installability/privileged-psi-redhat-com:443/system:serviceaccount:osci-installability:priv 
DEBU[0000] Using builder: cluster                       
DEBU[0000] setting Docker user agent to skaffold-f2038fa 
Generating tags...
 - gcr.io/k8s-skaffold/skaffold-example -> DEBU[0000] Running command: [git describe --tags --always] 
DEBU[0000] Command output: [v1.0.0-29-g8b870e2e
]       
DEBU[0000] Running command: [git status . --porcelain]  
DEBU[0000] Command output: [ M examples/kaniko/skaffold.yaml
] 
gcr.io/k8s-skaffold/skaffold-example:v1.0.0-29-g8b870e2e-dirty
INFO[0000] Tags generated in 62.662794ms                
Checking cache...
DEBU[0000] FIXME: Got an status-code for which error does not match any expected type!!!: -1  module=api status_code=-1
INFO[0000] update check failed: get latest and current Skaffold version: parsing current semver, skipping update check: parsing semver: No Major.Minor.Patch elements found 
DEBU[0001] FIXME: Got an status-code for which error does not match any expected type!!!: -1  module=api status_code=-1
DEBU[0002] Found dependencies for dockerfile: [{main.go /go true}] 
 - gcr.io/k8s-skaffold/skaffold-example: Not found. Building
INFO[0002] Cache check complete in 2.705445769s         
Building [gcr.io/k8s-skaffold/skaffold-example]...
DEBU[0002] Found dependencies for dockerfile: [{main.go /go true}] 
Storing build context at /tmp/context-8250de0b3c64c1ea2f4a254d7943e4f0.tar.gz
WARN[0002] The additionalFlags field in kaniko is deprecated, please consult the current schema at skaffold.dev to update your skaffold.yaml. 
DEBU[0002] getting client config for kubeContext: ``    
DEBU[0003] getting client config for kubeContext: ``    
INFO[0003] Waiting for kaniko-klhvz to be initialized   
DEBU[0007] Running command: [kubectl --context osci-installability/privileged-psi-redhat-com:443/system:serviceaccount:osci-installability:priv exec -i kaniko-klhvz -c kaniko-init-container -n osci-installability -- tar -xzf - -C /kaniko/buildcontext] 
DEBU[0009] Running command: [kubectl --context osci-installability/privileged-psi-redhat-com:443/system:serviceaccount:osci-installability:priv exec kaniko-klhvz -c kaniko-init-container -n osci-installability -- touch /tmp/complete] 
INFO[0011] Waiting for kaniko-klhvz to be complete      
DEBU[0011] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-klhvz" is waiting to start: PodInitializing 
DEBU[0012] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-klhvz" is waiting to start: PodInitializing 
FATA[0012] build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-example]: kaniko build for [gcr.io/k8s-skaffold/skaffold-example]: waiting for pod to complete: condition error: pod already in terminal phase: Failed 

[balopat]

Thank you for opening @Andrei-Stepanov! We have a lot of issues from @prary as well, that are around the inflexibilities of configuring kaniko from Skaffold. I am actively thinking about these issues - so that we don't have to one-by-one add a new supported field.

[prary]

Hi @Andrei-Stepanov @balopat ,
IMHO better approach would be using something like OPA and mutating pod object with service account which has root podSecurityPolicy attach, by doing so you are having more control than user manually attaching service account. BTW I am talking from a platform owner's perceptive :D .

[diist]

Kaniko is now compatible with AWS IAM Role for Service Accounts, so being able to set the service account used by the Kaniko pod is a must.

[nkubala]

since we don't have a better solution right now, I think adding this field to the Kaniko configuration in the skaffold.yaml would be fine. is anyone interested in submitting a PR?

[prary]

@nkubala I will raise the pr for allowing user to configure service account for kaniko in skaffold.yaml

[tejal29]

@prary Do you have an example of how the PSP should be configured ?

[tvvignesh]

Hi. I am still seeing this error when trying to deploy on a cluster with restricted PSP set with no root privileges - I am using this PSP with no changes - https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml

and I get this error:

And this is my skaffold sample:

apiVersion: skaffold/v2beta8
kind: Config
profiles:
- name: dev-svc
  build:
    artifacts:
    - image: asia.gcr.io/<project>/<app>
      kaniko:
        cache: {}
        dockerfile: Dockerfile
    cluster:
      pullSecretName: kaniko-secret
      namespace: default
      timeout: 20m
      runAsUser: 1000
      concurrency: 0

Using v1.15.0 of skaffold

[tejal29]

Unfortunately, kaniko needs to run as root. There isn't a way to run it with restricted PSP