Why Sealed Secrets?
Since I discovered Argo CD I hardly look back to the old days of storing YAML files on the server, and deploying them locally like peasants. I now store 99% of my configuration in GitLab and just refer Argo CD to it to deploy and maintain state of it. This was one of the most transformative tool for me for Kubernetes that I come across so far.
But what is the 1% you ask. Well
secrets, secrets are the one thing I could not just plainly store in the git, because they are plain text (base64 encoded). Anybody can read them. And you really do not want to store, for example, credentials from your database in git just like that.
You have multiple options how to deal with this, of course. I know of two:
And they both have they own advantages and disadvantages. Just to sum them up.
Uses operator and custom resources on your cluster to encrypt and decrypt secrets. In essence, you prepare the secret as you would normally and pipe it through
kubeseal command to encrypt it. As a result, you have an YAML file with encrypted secret that you can safely store in git. If that YAML files is applied, it's automatically decrypted and stored in your cluster as any secret would be. This is a very simple solution to a common problem. But I don't think this can be used for anything else, but secrets. I can confirm it also supports arm64.
Now, this is also interesting solution, but more complicated to set up and maintain (depends on what you gonna use as secret store). This requires some external secret store, like AWS Secrets Manager, Azure Key Vault or HashiCorp Vault. To be the single source of truth. This operator lives in your cluster and fetches secrets from an external secret store and inject them back into your cluster. I can confirm HashiCorp Vault is supported on arm64, and you can install it on your cluster together with
external secrets operator (also works on arm64). Now you have to manage secret store though and your secrets lives away from your deployments. Backups and other things are now on the menu. However, once you have, for example, HashiCorp Vault, you can use it for more than a secret store. You can use it in CI/CD for app deployment and its secrets, encrypt / decrypt files and so on.
Sealed Secrets install
In this guide I will focus on
Sealed Secrets and Vault + External Secrets will be done in another guide later. Personally, I used Argo CD to install this operator from helm chart, look at my Argo CD guide for more information Argo CD.
It's pretty simple deployment, nothing special that you would need to do.
- REPO URL: https://bitnami-labs.github.io/sealed-secrets
- CHART: sealed-secrets
- VERSION: 2.2.0 (This is a pre-release at the time of writing this guide)
- NAMESPACE: kube-system
Deploy / Sync and it should look like this:
kubeseal is a command line tool to interact with
sealed-secrets operator. It is used to encrypt secrets. So we need to install it on the controller node, the same place where you run
kubectl from. Look at the GIT in
releases for the correct version. Helm chart have version 0.2.2 which seems to be pre-release, and the actual release is 0.18.0... It's confusing to be honest, but the helm cart is using
sealed-secrets-controller:v0.18.0, so I guess it's ok.
curl -sSL https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/kubeseal-0.18.0-linux-arm64.tar.gz | tar -xz mv kubeseal /usr/local/bin/kubeseal chmod +x /usr/local/bin/kubeseal root@control01:~# kubeseal --version kubeseal version: 0.18.0
How do we encode our Kubernetes secret? Well, we need to use
kubeseal command with combination of
kubectl. We will create a dry run and pass it to
kubeseal. Secret will name will be
foo, and secret will be
bar. You can create the initial clean secret any way you like, but keep in mind that
kubeseal uses JSON by default. I, however, prefer to use
yaml format. Refer to Documentation.
#Create plain secret: echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml >mysecret.yaml
mysecret.yaml will look like this:
apiVersion: v1 data: foo: YmFy kind: Secret metadata: creationTimestamp: null name: mysecret
YmFy is Base64 encoded
Encrypt the secret:
kubeseal --controller-name=sealed-secrets --controller-namespace=kube-system --format yaml <mysecret.yaml>mysealedsecret.yaml
You might want to use also parameter --namespace to specify namespace where you want the secret to exist. Together with
--scope to specify if the secret can be decrypted in any namespace, or only in specified one. Check the possible scopes in documentation.
mysealedsecret.yaml will look like this:
apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: creationTimestamp: null name: mysecret namespace: default spec: encryptedData: foo: AgDEPWfG7iH8p2DBSqRGe+hVpRa1+d06hWffB1krTyF2iBpxTPY/rZw6Ba26dA+txlWYZN5uw/CxLyk+zs1WqU64qskHptC5dcbEuCPwXnZQbUL6x/HBzkr4sXwAcYGFKPXtCSG98o5E5F/Mx7PtFQAMcZ0Jo1e2OZt4vH07QMaDdTLwwPFrWGOiIcyOGJX/XFOeW/s7wGj31loIHi50uljGxCGns4l2DiU29mo7VSq4aHAOEWAM8jiyGPC8eapdrmYU2NpEBJJMAWwwsO6WkF6jAMIiDvKXMC1alYIYxIFJB7OcEyuLvddLbmn0fvh9hcQJe2bRSe/yT7AVvYoCWsSX1zji3IIPCzLTOHDzf74gsi2Gbt+FCyTYJNu2dzQpl5jIBctMwVOTF1H1154RFHyRoAGl5R3jgwbQ2kn1pK4O1w23FuLKHfLr8ExllPeoiSDykYetrvRuKcV2BUeAztbDs+aKXn4yfVHNpvryhyzEbm7804CyjDpSRkjC2tKcnrv1mEoCD5EyAqWvfIbFVnoj7mp8cOhLlDWz0cp32u1KAHxg21dK3K0XQfUDaqOiXa1TiBmGFedjyo/MpMKMbZTqs6TPpiEPiP6Eso9fr5u/9LQQY2V1eYpTChI8e824U3YUmo2ooC+GOGarUk5en1VQLC9yGf5XcppeZh23NAp9Egzlo3j+B25P2IEuLqiiqfyLLHc= template: data: null metadata: creationTimestamp: null name: mysecret namespace: default
Bam! That value for
foo is now encrypted. You can store this even in public repo, nobody will be able to get the value back without the
sealed-secrets operator that is running on your server. This is because the
sealed-secrets operator creates a decryption key that is unique to it.
This also means if you erase your
sealed-secrets operator, you will not be able to decrypt the secrets anymore. You can, however, backup the master key and restore it later. See
backup section down below.
You can apply the encrypted secret to your cluster with
root@control01:~# kubectl create -f mysealedsecret.yaml sealedsecret.bitnami.com/mysecret created
And when you check it out in Kubernetes, you will see it as normal secret:
root@control01:~# kubectl get secret mysecret -o yaml apiVersion: v1 data: foo: YmFy kind: Secret metadata: creationTimestamp: "2022-06-28T09:46:36Z" name: mysecret namespace: default ownerReferences: - apiVersion: bitnami.com/v1alpha1 controller: true kind: SealedSecret name: mysecret uid: 32154559-18ea-4abd-b4db-deb9a83bd05a resourceVersion: "8130084" uid: be1d1888-2fcc-429a-9176-a5102e0bfbef type: Opaque
It's important to back up your master key. Hide it, hide it well. If your cluster blows up, you will need it to decrypt the secrets again.
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >master.key
Just redeploy the
sealed-secrets operator. And then apply the master key:
kubectl apply -f master.key #delete the running pod, to restart it and read the new key kubectl delete pod -n kube-system -l name=sealed-secrets-controller
Done and done. I needed to write this down, so I can get back to it when needed. I hope it's useful for you as well.