Sealed Secrets
Why Sealed Secrets?
Since I discovered Argo CD I hardly look back to the old days of storing YAML files on the server, and deploying them locally like peasants. I now store 99% of my configuration in GitLab and just refer Argo CD to it to deploy and maintain state of it. This was one of the most transformative tool for me for Kubernetes that I come across so far.
But what is the 1% you ask. Well secrets, secrets are the one thing I could not just plainly store in the git, because they are plain text (base64 encoded). Anybody can read them. And you really do not want to store, for example, credentials from your database in git just like that.
You have multiple options how to deal with this, of course. I know of two:
And they both have they own advantages and disadvantages. Just to sum them up.
Sealed Secrets
Uses operator and custom resources on your cluster to encrypt and decrypt secrets. In essence, you prepare the secret as you would normally and pipe it through kubeseal command to encrypt it. As a result, you have an YAML file with encrypted secret that you can safely store in git. If that YAML files is applied, it's automatically decrypted and stored in your cluster as any secret would be. This is a very simple solution to a common problem. But I don't think this can be used for anything else, but secrets. I can confirm it also supports arm64.
External Secrets
Now, this is also interesting solution, but more complicated to set up and maintain (depends on what you gonna use as secret store). This requires some external secret store, like AWS Secrets Manager, Azure Key Vault or HashiCorp Vault. To be the single source of truth. This operator lives in your cluster and fetches secrets from an external secret store and inject them back into your cluster. I can confirm HashiCorp Vault is supported on arm64, and you can install it on your cluster together with external secrets operator (also works on arm64). Now you have to manage secret store though and your secrets lives away from your deployments. Backups and other things are now on the menu. However, once you have, for example, HashiCorp Vault, you can use it for more than a secret store. You can use it in CI/CD for app deployment and its secrets, encrypt / decrypt files and so on.
Sealed Secrets install
In this guide I will focus on Sealed Secrets and Vault + External Secrets will be done in another guide later. Personally, I used Argo CD to install this operator from helm chart, look at my Argo CD guide for more information Argo CD.
It's pretty simple deployment, nothing special that you would need to do.
- REPO URL: https://bitnami-labs.github.io/sealed-secrets
- CHART: sealed-secrets
- VERSION: 2.2.0 (This is a pre-release at the time of writing this guide)
- NAMESPACE: kube-system
Deploy / Sync and it should look like this:
kubeseal
kubeseal is a command line tool to interact with sealed-secrets operator. It is used to encrypt secrets. So we need to install it on the controller node, the same place where you run kubectl from. Look at the GIT in releases for the correct version. Helm chart have version 0.2.2 which seems to be pre-release, and the actual release is 0.18.0... It's confusing to be honest, but the helm cart is using sealed-secrets-controller:v0.18.0, so I guess it's ok.
curl -sSL https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/kubeseal-0.18.0-linux-arm64.tar.gz | tar -xz
mv kubeseal /usr/local/bin/kubeseal
chmod +x /usr/local/bin/kubeseal
root@control01:~# kubeseal --version
kubeseal version: 0.18.0Sealing secrets
How do we encode our Kubernetes secret? Well, we need to use kubeseal command with combination of kubectl. We will create a dry run and pass it to kubeseal. Secret will name will be foo, and secret will be bar. You can create the initial clean secret any way you like, but keep in mind that kubeseal uses JSON by default. I, however, prefer to use yaml format. Refer to Documentation.
#Create plain secret:
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml >mysecret.yamlmysecret.yaml will look like this:
apiVersion: v1
data:
foo: YmFy
kind: Secret
metadata:
creationTimestamp: null
name: mysecretYmFy is Base64 encoded bar string.
Encrypt the secret:
kubeseal --controller-name=sealed-secrets --controller-namespace=kube-system --format yaml <mysecret.yaml>mysealedsecret.yaml--scope to specify if the secret can be decrypted in any namespace, or only in specified one. Check the possible scopes in documentation.mysealedsecret.yaml will look like this:
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: mysecret
namespace: default
spec:
encryptedData:
foo: AgDEPWfG7iH8p2DBSqRGe+hVpRa1+d06hWffB1krTyF2iBpxTPY/rZw6Ba26dA+txlWYZN5uw/CxLyk+zs1WqU64qskHptC5dcbEuCPwXnZQbUL6x/HBzkr4sXwAcYGFKPXtCSG98o5E5F/Mx7PtFQAMcZ0Jo1e2OZt4vH07QMaDdTLwwPFrWGOiIcyOGJX/XFOeW/s7wGj31loIHi50uljGxCGns4l2DiU29mo7VSq4aHAOEWAM8jiyGPC8eapdrmYU2NpEBJJMAWwwsO6WkF6jAMIiDvKXMC1alYIYxIFJB7OcEyuLvddLbmn0fvh9hcQJe2bRSe/yT7AVvYoCWsSX1zji3IIPCzLTOHDzf74gsi2Gbt+FCyTYJNu2dzQpl5jIBctMwVOTF1H1154RFHyRoAGl5R3jgwbQ2kn1pK4O1w23FuLKHfLr8ExllPeoiSDykYetrvRuKcV2BUeAztbDs+aKXn4yfVHNpvryhyzEbm7804CyjDpSRkjC2tKcnrv1mEoCD5EyAqWvfIbFVnoj7mp8cOhLlDWz0cp32u1KAHxg21dK3K0XQfUDaqOiXa1TiBmGFedjyo/MpMKMbZTqs6TPpiEPiP6Eso9fr5u/9LQQY2V1eYpTChI8e824U3YUmo2ooC+GOGarUk5en1VQLC9yGf5XcppeZh23NAp9Egzlo3j+B25P2IEuLqiiqfyLLHc=
template:
data: null
metadata:
creationTimestamp: null
name: mysecret
namespace: defaultBam! That value for foo is now encrypted. You can store this even in public repo, nobody will be able to get the value back without the sealed-secrets operator that is running on your server. This is because the sealed-secrets operator creates a decryption key that is unique to it.
sealed-secrets operator, you will not be able to decrypt the secrets anymore. You can, however, backup the master key and restore it later. See backup section down below.You can apply the encrypted secret to your cluster with kubectl apply:
root@control01:~# kubectl create -f mysealedsecret.yaml
sealedsecret.bitnami.com/mysecret createdAnd when you check it out in Kubernetes, you will see it as normal secret:
root@control01:~# kubectl get secret mysecret -o yaml
apiVersion: v1
data:
foo: YmFy
kind: Secret
metadata:
creationTimestamp: "2022-06-28T09:46:36Z"
name: mysecret
namespace: default
ownerReferences:
- apiVersion: bitnami.com/v1alpha1
controller: true
kind: SealedSecret
name: mysecret
uid: 32154559-18ea-4abd-b4db-deb9a83bd05a
resourceVersion: "8130084"
uid: be1d1888-2fcc-429a-9176-a5102e0bfbef
type: OpaqueBackup
It's important to back up your master key. Hide it, hide it well. If your cluster blows up, you will need it to decrypt the secrets again.
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >master.keyRestore
Just redeploy the sealed-secrets operator. And then apply the master key:
kubectl apply -f master.key
#delete the running pod, to restart it and read the new key
kubectl delete pod -n kube-system -l name=sealed-secrets-controllerDone and done. I needed to write this down, so I can get back to it when needed. I hope it's useful for you as well.