Store secrets in GitOps Repository using SealedSecrets

For security reasons, Kubernetes secrets are usually the only resource that cannot be managed with a GitOps workflow. Instead of managing secrets outside of GitOps and having to use a third-party tool like Vault, SealedSecrets provides a way to keep all the advantages of using a GitOps workflow while avoiding exposing secrets. SealedSecrets is composed of two main components:

A CLI (Kubeseal) to encrypt secrets.
A cluster-side controller to decrypt the sealed secrets into regular Kubernetes secrets. Only this controller can decrypt sealed secrets, not even the original author.

This tutorial describes how to install these two components, configure the controller, and add or remove sealed secrets.

Setup

These instructions are used as an example. For instructions on the latest release, see the release page. For full documentation on SealedSecrets, see the GitHub repo.

Install Kubeseal CLI to encrypt your secrets

On MacOS
```
brew install kubeseal
```

On Linux

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.17.1/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Install the SealedSecrets controller on your cluster

This controller will be able to decrypt SealedSecrets and create Kubernetes secrets.

Create the controller:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.17.1/controller.yaml

Fetch the certificate that you will use to encrypt your secrets into sealed secrets:
```
kubeseal --fetch-cert > mycert.pem
```
Commit mycert.pem to your git repo.

Adding a secret

Secrets can be securely added to Git using sealed secrets:

Create a Kubernetes secret and pipe it into kubeseal using the certificate mycert.pem that you fetched from the controller in the setup:

echo '---' >> secrets.yaml
kubectl create secret -n dispatch generic mysecret --dry-run -o yaml --from-literal=my-secret=value | \
    kubeseal --format yaml --cert mycert.pem >> secrets.yaml

Go to the end of secrets.yaml where you just added your new sealed secret. Remove any “creationTimestamp” fields from the yaml.
Run kubectl apply -f secrets.yaml. If you do not have permission, commit your changes to the repo and let FluxCD apply the changes for you.
The sealed secret controller will then decrypt the sealed secret and generate a Kubernetes secret from it. Your secret got successfully created by running:
```
kubectl get secret mysecret -n dispatch -o yaml
```
If your sealed secret got created successfully but did not generate the matching secret, look at the logs of the controller:
```
kubectl logs -l=name=sealed-secrets-controller -n kube-system
```
Commit secrets.yaml to your repo if you have not already done so in step 3.

Removing a secret

Following the same example from above in “Adding a secret”, now remove the manifest for mysecret in secrets.yaml and commit those changes to the repo.
Delete the sealed secret in the cluster:
```
kubectl delete SealedSecret mysecret
```
Delete the secret itself:
```
kubectl delete secret mysecret
```

Rotating the controller’s sealing key

For added security, it is a good practice to rotate the key the controller uses to decrypt sealed secrets. By default, the controller generates a new key every 30 days. When this happens, you need to update the certificate you use to create sealed secrets by fetching the latest one:

   kubeseal --fetch-cert > mycert.pem

NOTE: Do not forget to commit it back to the repo!

In a disaster case, let’s say your cluster gets destroyed, you would lose all your sealing keys, so you would not be able to recreate all the secrets from the sealed secrets in your GitOps repo. For this reason, you might want to back up the sealing keys. To do this every time a new sealing key is generated, run:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealing-key

Then store sealing-key with the others in a safe location such as OneLogin Notes or Vault. To restore from a backup after a disaster, recreate all of the sealing keys with kubectl apply -f sealing-key1 sealing-key2 ... before starting the controller. If the controller was already started, restart it: kubectl delete pod -n kube-system -l name=sealed-secrets-controller

To disable sealing key rotation For example, configure the controller’s command in the pod template with --key-renew-period=0. See the following yaml file.

Pod Template:
  Labels:           name=sealed-secrets-controller
  Service Account:  sealed-secrets-controller
  Containers:
   sealed-secrets-controller:
    Image:      quay.io/bitnami/sealed-secrets-controller:v0.9.8
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:    controller
      --key-renew-period=0

If required, edit the controller’s manifest with:

kubectl edit deployment.apps/sealed-secrets-controller -n kube-system