Skip rotate dex/gangway cert if signer unknown

[jenting]

Waiting PR #858 got merged and re-test, now it's ready for review.

Why is this PR needed?

User might manually replace dex+gangway server certificate signed by their trusted CA certificate, no letting skuba controls the certificate rotation.

User could replace the server certificate at:

• after deployment, edit oidc-dex-cert and oidc-gangway-cert secret resource
• before deployment, but it does not support now until PR kustomize Add kustomize support for addons #858 got merged

This PR address after deployment issue. One the PR kustomize #858 got merged, it needs to double-check the behavior is the same for the before deployment case, and also update doc-caasp PR.

Fixes SUSE/doc-caasp#619

Reminder: Add the "fixes bsc#XXXX" to the title of the commit so that it will
appear in the changelog.

What does this PR do?

Check the dex+gangway server certificate is signed by our CA file (/etc/kubernetes/pki/ca.crt) during cluster bootstrap and node upgrade:

• if yes, create a server certificate and create/update to the corresponding secret resource
• if no, bump output a log that the server certificate is not signed by our CA file, skip the process

Anything else a reviewer needs to know?

N/A

Info for QA

Please test two cases:

1. The user does not replace existing dex+gangway certificate, after skuba cluster upgrade from 1.15.2 to 1.16.2, the server certificate should be rotated and signed by the CA certificate (/etc/kuberntes/pki/ca.crt).

• Check signer

openssl s_client -connect <IP/FQDN>:32000 -CAfile pki/ca.crt <<< "Q"
openssl s_client -connect <IP/FQDN>:32001 -CAfile pki/ca.crt <<< "Q"

• Check certificate got rotate

echo | openssl s_client -connect <IP/FQDN>:32000 | openssl x509 -noout -dates
echo | openssl s_client -connect <IP/FQDN>:32001 | openssl x509 -noout -dates   

2. The user replace existing dex+gangway certificate signed by custom CA certificate, after skuba cluster upgrade from 1.15.2 to 1.16.2, the server certificate should keep the same. (Follows the below test steps)

Related info

N/A

Status BEFORE applying the patch

1. Bootstrap a cluster with version 1.15.2
2. Sign dex certificate by the custom CA certificate [1]

• Get the dex server SAN IP/DNS

mkdir -p pki.bak
kubectl get secret oidc-dex-cert -n kube-system -o yaml | tee pki.bak/oidc-dex-cert.yaml > /dev/null
cat pki.bak/oidc-dex-cert.yaml | grep tls.crt | awk '{print $2}' | base64 --decode | tee pki.bak/oidc-dex.crt > /dev/null

openssl x509 -noout -text -in pki.bak/oidc-dex.crt | grep -oP '(?<=IP Address:)[^,]+'
openssl x509 -noout -text -in pki.bak/oidc-dex.crt | grep -oP '(?<=DNS:)[^,]+'

• Sign the dex server certificate with mkcert tool

mkcert <IP Address> <DNS>

• Replace dex certificate, prepare file oidc-dex-cert.yaml

apiVersion: v1
kind: Secret
metadata:
  name: oidc-dex-cert
  namespace: kube-system
type: kubernetes.io/tls
data:
  ca.crt: cat <TRUSTED_CA_CERT_PATH> | base64 | awk '{print}' ORS='' && echo
  tls.crt: cat <SIGNED_OIDC_DEX_SERVER_CERT_PATH> | base64 | awk '{print}' ORS='' && 
echo
  tls.key: cat <SIGNED_OIDC_DEX_SERVER_KEY_PATH> | base64 | awk '{print}' ORS='' && 
echo

• Replace dex secret resource and restart pod

kubectl replace -f oidc-dex-cert.yaml
kubectl delete pod -lapp=oidc-dex -n kube-system

3. Sign gangway certificate by the custom CA certificate [2]

• Get the gangway server SAN IP/DNS

mkdir -p pki.bak
kubectl get secret oidc-gangway-cert -n kube-system -o yaml | tee pki.bak/oidc-gangway-cert.yaml > /dev/null
cat pki.bak/oidc-gangway-cert.yaml | grep tls.crt | awk '{print $2}' | base64 --decode | tee pki.bak/oidc-gangway.crt > /dev/null

openssl x509 -noout -text -in pki.bak/oidc-gangway.crt | grep -oP '(?<=IP Address:)[^,]+'
openssl x509 -noout -text -in pki.bak/oidc-gangway.crt | grep -oP '(?<=DNS:)[^,]+'

• Sign ther server certificate with mkcert tool

mkcert <IP Address> <DNS>

• Replace gangway certificate, prepare file oidc-gangway-cert.yaml

apiVersion: v1
kind: Secret
metadata:
  name: oidc-gangway-cert
  namespace: kube-system
type: kubernetes.io/tls
data:
  ca.crt: cat <TRUSTED_CA_CERT_PATH> | base64 | awk '{print}' ORS='' && echo
  tls.crt: cat <SIGNED_OIDC_GANGWAY_SERVER_CERT_PATH> | base64 | awk '{print}' ORS='' && 
echo
  tls.key: cat <SIGNED_OIDC_GANGWAY_SERVER_KEY_PATH> | base64 | awk '{print}' ORS='' && 
echo

• Replace gangway secret resource and restart pod

kubectl replace -f oidc-gangway-cert.yaml
kubectl delete pod -lapp=oidc-gangway -n kube-system

4. Perform cluster upgrade to version 1.16.2, the dex+gangway certificate would be resigned by the CA (/etc/kubernetes/pki/ca.crt) [3]

openssl s_client -connect <IP/FQDN>:32000  -CAfile pki/ca.crt <<< "Q"
openssl s_client -connect <IP/FQDN>:32001  -CAfile pki/ca.crt <<< "Q"

Status AFTER applying the patch

1. Bootstrap a cluster with version 1.15.2
2. Sign dex certificate by the custom CA certificate as [1]
3. Sign gangway certificate by the custom CA certificate [2]
4. Perform cluster upgrade to version 1.16.2, the dex+gangway certificate should not be resigned by the CA (/etc/kubernetes/pki/ca.crt), keep it the same as the user manually replace certificate

openssl s_client -connect <IP/FQDN>:32000  -CAfile <custom-certificate-path> <<< "Q"
openssl s_client -connect <IP/FQDN>:32001  -CAfile <custom-certificate-path> <<< "Q"

Docs

SUSE/doc-caasp#625

Merge restrictions

(Please do not edit this)

We are in v4-maintenance phase, so we will restrict what can be merged to prevent unexpected surprises:

What can be merged (merge criteria):
    2 approvals:
        1 developer: code is fine
        1 QA: QA is fine
    there is a PR for updating documentation (or a statement that this is not needed)

Signed-off-by: JenTing Hsiao jenting.hsiao@suse.com