In the previous post we looked at how Kubernetes implements PKI and some of the security consequences of that. In this post, we'll be exploring secrets management in Kubernetes. Securely handling secrets is essential for any cluster operator, and there are several important nuances to keep in mind.
Secret-handling threat models
Before we get into the details of how Kubernetes handles secrets natively and some of the available external options, it's a good idea to think about the threats we're aiming to mitigate. Quite often when this topic is discussed, options might be dismissed as "insecure" without detailing the specific attack scenarios involved.
The goal of Kubernetes secret management is to enable applications and system components to access sensitive information, such as necessary credentials, that they need for their operation. While there's a variety of secret types, like API keys for external systems or login credentials for a database, the threats and mitigations tend to be the same.
Attacks on secrets tend to fall into three categories: attacks on secrets at rest, attacks on secrets in transit, and attacks on APIs that provision secrets. Secrets are at rest where they're stored in the etcd database before being used and then again when they're provided to an application to use they are at rest on the worker node, and they're in transit when moving from the store to the application.
Attacking secrets at rest could involve gaining access to the etcd host, the node OS, or physical disks and backups in order to extract information.
Attacking secrets in transit involves intercepting traffic as it flows between the database and nodes, or by accessing the Kubernetes or etcd APIs (with sufficient rights) to request the secrets directly.
API attacks focus on bypassing the authentication and authorization controls of Kubernetes and etcd themselves, but always with the same goal of secret extraction.
Using Kubernetes secrets
Kubernetes provides a built-in Secret object that can be used with applications running in the cluster. The secrets are stored in the cluster's etcd database and come in a variety of different types. For example, if an API key needs to be accessible by an application running in the cluster, you could construct a Secret object with a type of opaque that looks like the manifest below. Note that the secret has been base64-encoded; this is not for security but to prevent the secret's content from interfering with the YAML object.
apiVersion: v1
kind: Secret
metadata:
name: my-api-secret
type: Opaque
data:
API_KEY: aHVudGVyMg==You can then reference the secret in the application's Pod manifest, making it available as an environment variable in the application:
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
spec:
replicas: 1
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
containers:
- name: webapp
image: nginx:latest
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: my-api-secret
key: API_KEYWhen this deployment and secret are run in a cluster, the decoded secret is visible in the application's environment variable list.
An important security consideration is whether to make the secret available as an environment variable (as shown in the previous example) or as a file mounted into the container. From a security perspective, a mounted file is better because an application's environment variables are often captured for debugging, which could result in secrets being leaked to systems that shouldn't have access to them.
Why not use ConfigMaps instead?
A common question is why one should use dedicated Secret objects rather than ConfigMaps. While both store data in etcd and provide it to running containers, the difference lies in security intent.
Simply put, using a dedicated Secret object type signals to external systems that the data is intended to be secret. For example, systems might avoid logging secrets while routinely capturing general ConfigMaps for debugging. Furthermore, if your cluster uses generic Role-Based Access Control (RBAC) roles like view or edit, these typically provide access to manage configmaps but not secrets; this makes it easier to restrict access when using dedicated object types.
This risk was highlighted in 2023 by a critical CVE in the RKE distribution of Kubernetes, where a large amount of sensitive data was inappropriately stored in a ConfigMap in the cluster.
Securing Kubernetes secrets at rest
Securing secrets at rest involves protecting them where they are stored: in the cluster's database (etcd) and on the cluster's worker nodes. For secrets stored in etcd, we need to consider the differences between managed and unmanaged Kubernetes.
For managed Kubernetes, securing the etcd database and the underlying systems is the responsibility of the Cloud Service Provider (CSP). While the CSP may expose options for this, you ultimately rely on their processes and procedures. The CSP is responsible for securing the etcd hosts and also managing access to physical disks, so they will need to ensure the security of those systems.
For unmanaged Kubernetes, cluster operators need to consider how they'll protect the secrets at rest in etcd. Kubernetes provides an option to encrypt data at rest by using a key held by the API server. It's worth considering whether this provides enough of a benefit, as it does introduce overhead in managing the encryption keys. For example, if your API server and etcd database are on the same node, an attacker with node access can retrieve both the encrypted data and the key needed to decrypt it, negating the security improvement.
Encrypting data at rest is more beneficial in larger clusters where etcd and API server processes run on separate nodes or where a requirement to encrypt data at rest is in place for regulatory or compliance reasons. The risk you're mitigating here is where an attacker gains access to either the etcd nodes or the disks running those nodes (or potentially backup files). If that's a realistic scenario for you, then it makes sense to enable this feature.
Secrets are also at rest on cluster worker nodes. An attacker with privileged shell access to the underlying node can likely pull secrets from any container running on that node. In multi-tenant clusters, using node pools to separate tenants' workloads can reduce the risk of secrets leaking inappropriately. It is also important to restrict access to the node's filesystem from pods running on it, particularly through features like hostPath volumes. Attackers with the ability to create arbitrary hostPath mounts can access the secret volumes of any other container running on that node.
Securing Kubernetes secrets in transit
Protecting secrets in transit primarily involves encrypting connections between Kubernetes components. Since traffic between etcd, the API, and the Kubelet is encrypted by default, sniffing is only possible if an attacker steals private keys, which would generally allow many other attacks as a result.
Securing access to secrets
Access to secrets is primarily controlled through two Kubernetes APIs: the main Kubernetes API and etcd. End users generally do not access the etcd API directly, and this is often blocked in managed Kubernetes. In unmanaged Kubernetes, it's important to protect the private keys that the API server uses to authenticate to the etcd service.
Managing access to secrets via the Kubernetes API requires careful consideration of some lesser known points:
- List and watch rights: If you provide a user or service account with rights to list or watch secrets, this also provides access to the contents of all of those secrets. For example, with list rights, the user can't do
kubectl get secret SECRETto see one secret, but runningkubectl get secrets -o yamlwill show all the secrets and their contents. - Secrets Access in a namespace: Any user or service that can create pods in a given Kubernetes namespace can access any secret in that namespace. This is possible even without direct secret access because the Kubelet uses its own elevated rights to populate the secret into the container when the workload is started.
Using external secret managers
In addition to Kubernetes native support for secret objects, there are a number of external solutions for managing secrets that might be more appropriate depending on the organization's needs and threat model. A key drawback for these solutions is that it's another system that needs to be managed and patched, and which may incur additional cost if you're using a cloud provider service.
External providers can offer a more granular set of access control and auditing options, which could be useful in some environments. They may also support features like hardware-backed encryption; this could be relevant in some threat models but more likely are useful when complying with external regulations that stipulate the use of hardware-backed encryption.
Conclusion
Secrets management is fundamental to most Kubernetes clusters, as applications running in the cluster require access to credentials and other forms of sensitive data for their operations. In many cases, the built-in Kubernetes secret support is perfectly adequate. When considering external secrets solutions, it's important to understand your risk model and precisely which attacks you expect them to mitigate.
