Skip to main content

K3s Kubernetes

Sealed Secrets

Why Sealed Secrets?

Since I discovered Argo CD I hardly look back to the old days of storing YAML files on the server, and deploying them locally like peasants. I now store 99% of my configuration in GitLab and just refer Argo CD to it to deploy and maintain state of it. This was one of the most transformative tool for me for Kubernetes that I come across so far.

But what is the 1% you ask. Well secrets, secrets are the one thing I could not just plainly store in the git, because they are plain text (base64 encoded). Anybody can read them. And you really do not want to store, for example, credentials from your database in git just like that.

You have multiple options how to deal with this, of course. I know of two:

  • Sealed Secrets GIT
  • External Secrets GIT

And they both have they own advantages and disadvantages. Just to sum them up.

Sealed Secrets

Uses operator and custom resources on your cluster to encrypt and decrypt secrets. In essence, you prepare the secret as you would normally and pipe it through kubeseal command to encrypt it. As a result, you have an YAML file with encrypted secret that you can safely store in git. If that YAML files is applied, it's automatically decrypted and stored in your cluster as any secret would be. This is a very simple solution to a common problem. But I don't think this can be used for anything else, but secrets. I can confirm it also supports arm64.

External Secrets

Now, this is also interesting solution, but more complicated to set up and maintain (depends on what you gonna use as secret store). This requires some external secret store, like AWS Secrets Manager, Azure Key Vault or HashiCorp Vault. To be the single source of truth. This operator lives in your cluster and fetches secrets from an external secret store and inject them back into your cluster. I can confirm HashiCorp Vault is supported on arm64, and you can install it on your cluster together with external secrets operator (also works on arm64). Now you have to manage secret store though and your secrets lives away from your deployments. Backups and other things are now on the menu. However, once you have, for example, HashiCorp Vault, you can use it for more than a secret store. You can use it in CI/CD for app deployment and its secrets, encrypt / decrypt files and so on.

Sealed Secrets install

In this guide I will focus on Sealed Secrets and Vault + External Secrets will be done in another guide later. Personally, I used Argo CD to install this operator from helm chart, look at my Argo CD guide for more information Argo CD.

It's pretty simple deployment, nothing special that you would need to do.

Deploy / Sync and it should look like this:


kubeseal is a command line tool to interact with sealed-secrets operator. It is used to encrypt secrets. So we need to install it on the controller node, the same place where you run kubectl from. Look at the GIT in releases for the correct version. Helm chart have version 0.2.2 which seems to be pre-release, and the actual release is 0.18.0... It's confusing to be honest, but the helm cart is using sealed-secrets-controller:v0.18.0, so I guess it's ok.

curl -sSL | tar -xz
mv kubeseal /usr/local/bin/kubeseal
chmod +x /usr/local/bin/kubeseal

root@control01:~# kubeseal --version
kubeseal version: 0.18.0

Sealing secrets

How do we encode our Kubernetes secret? Well, we need to use kubeseal command with combination of kubectl. We will create a dry run and pass it to kubeseal. Secret will name will be foo, and secret will be bar. You can create the initial clean secret any way you like, but keep in mind that kubeseal uses JSON by default. I, however, prefer to use yaml format. Refer to Documentation.

#Create plain secret:
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml >mysecret.yaml

mysecret.yaml will look like this:

apiVersion: v1
  foo: YmFy
kind: Secret
  creationTimestamp: null
  name: mysecret

YmFy is Base64 encoded bar string.

Encrypt the secret:

kubeseal --controller-name=sealed-secrets --controller-namespace=kube-system --format yaml <mysecret.yaml>mysealedsecret.yaml
You might want to use also parameter --namespace to specify namespace where you want the secret to exist. Together with --scope to specify if the secret can be decrypted in any namespace, or only in specified one. Check the possible scopes in documentation.

mysealedsecret.yaml will look like this:

kind: SealedSecret
  creationTimestamp: null
  name: mysecret
  namespace: default
    foo: AgDEPWfG7iH8p2DBSqRGe+hVpRa1+d06hWffB1krTyF2iBpxTPY/rZw6Ba26dA+txlWYZN5uw/CxLyk+zs1WqU64qskHptC5dcbEuCPwXnZQbUL6x/HBzkr4sXwAcYGFKPXtCSG98o5E5F/Mx7PtFQAMcZ0Jo1e2OZt4vH07QMaDdTLwwPFrWGOiIcyOGJX/XFOeW/s7wGj31loIHi50uljGxCGns4l2DiU29mo7VSq4aHAOEWAM8jiyGPC8eapdrmYU2NpEBJJMAWwwsO6WkF6jAMIiDvKXMC1alYIYxIFJB7OcEyuLvddLbmn0fvh9hcQJe2bRSe/yT7AVvYoCWsSX1zji3IIPCzLTOHDzf74gsi2Gbt+FCyTYJNu2dzQpl5jIBctMwVOTF1H1154RFHyRoAGl5R3jgwbQ2kn1pK4O1w23FuLKHfLr8ExllPeoiSDykYetrvRuKcV2BUeAztbDs+aKXn4yfVHNpvryhyzEbm7804CyjDpSRkjC2tKcnrv1mEoCD5EyAqWvfIbFVnoj7mp8cOhLlDWz0cp32u1KAHxg21dK3K0XQfUDaqOiXa1TiBmGFedjyo/MpMKMbZTqs6TPpiEPiP6Eso9fr5u/9LQQY2V1eYpTChI8e824U3YUmo2ooC+GOGarUk5en1VQLC9yGf5XcppeZh23NAp9Egzlo3j+B25P2IEuLqiiqfyLLHc=
    data: null
      creationTimestamp: null
      name: mysecret
      namespace: default

Bam! That value for foo is now encrypted. You can store this even in public repo, nobody will be able to get the value back without the sealed-secrets operator that is running on your server. This is because the sealed-secrets operator creates a decryption key that is unique to it.

This also means if you erase your sealed-secrets operator, you will not be able to decrypt the secrets anymore. You can, however, backup the master key and restore it later. See backup section down below.

You can apply the encrypted secret to your cluster with kubectl apply:

root@control01:~# kubectl create -f mysealedsecret.yaml created

And when you check it out in Kubernetes, you will see it as normal secret:

root@control01:~# kubectl get secret mysecret -o yaml
apiVersion: v1
  foo: YmFy
kind: Secret
  creationTimestamp: "2022-06-28T09:46:36Z"
  name: mysecret
  namespace: default
  - apiVersion:
    controller: true
    kind: SealedSecret
    name: mysecret
    uid: 32154559-18ea-4abd-b4db-deb9a83bd05a
  resourceVersion: "8130084"
  uid: be1d1888-2fcc-429a-9176-a5102e0bfbef
type: Opaque


It's important to back up your master key. Hide it, hide it well. If your cluster blows up, you will need it to decrypt the secrets again.

kubectl get secret -n kube-system -l -o yaml >master.key


Just redeploy the sealed-secrets operator. And then apply the master key:

kubectl apply -f master.key
#delete the running pod, to restart it and read the new key
kubectl delete pod -n kube-system -l name=sealed-secrets-controller

Done and done. I needed to write this down, so I can get back to it when needed. I hope it's useful for you as well.