Without the private key that is managed by the controller, there is no way to decrypt the encrypted data within a SealedSecret. In the event that you are trying to restore the original state of a cluster after a disaster or you want to leverage GitOps workflow to deploy the Kubernetes resources, including SealedSecrets, from a Git repository and stand up a separate instance of an EKS cluster, the controller deployed in the new cluster must use the same private key to be able to unseal the SealedSecrets.
Run the following command in order to retrieve the private key from the cluster. In a production environment, you will typically make use of Kubernetes RBAC to grant the permissions required to perform this operation to restricted set of clients.
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > master.yaml
To test how this works, let’s first delete the installation of the controller, the Secret that it created which contains the private key, the SealedSecret resource named database-credentials as well as the Secret that was unsealed from it.
kubectl delete secret database-credentials -n octank
kubectl delete sealedsecret database-credentials -n octank
kubectl delete secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
kubectl delete -f controller.yaml
Now, put the Secret containing the private key back into the cluster using the master.yaml file.
kubectl apply -f master.yaml
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
Output:
Next, redeploy the SealedSecret CRD, controller and RBAC artifacts on your EKS.
kubectl apply -f controller.yaml
kubectl get pods -n kube-system | grep sealed-secrets-controller
Output:
View the logs of the newly launched controller pod. Note that the name of the controller pod will be different in your cluster.
kubectl logs sealed-secrets-controller-84fcdcd5fd-ds5t6 -n kube-system
Output:
As you can see from the logs, the controller was able to find the existing Secret sealed-secrets-keyvk4pr in the kube-system namespace and therefore does not create a new key pair. Now, let’s redeploy the SealedSecret and verify that the controller is able to successfully unseal it.
kubectl apply -f sealed-secret.yaml
kubectl logs sealed-secrets-controller-84fcdcd5fd-ds5t6 -n kube-system
Output:
If the file master.yaml which contains the public/private key pair generated by the controller is compromised, then all the SealedSecret manifests can be unsealed and the encrypted sensitive information they store revealed. Hence, this file must be guarded by granting least privilege access. For additional guidance on sealing key renewal, manual sealing key management etc., please consult the documentation.
One option to secure the private key is to store the master.yaml file contents as a SecureString parameter in AWS Systems Manager Parameter Store. The parameter could be secured using a KMS Customer managed key (CMK) and you can use the Key policy to restrict the set of IAM principals who can use this key in order to retrieve the parameter. Additionally, you may also enable automatic rotation of this CMK in KMS. Note that Standard tier parameters support a maximum parameter value of 4096 characters. Hence, given the size of the master.yaml file, you will have to store it as a parameter in the Advanced tier.