In the previous post in this series, we continued our look at unpatchable Kubernetes vulnerabilities with an examination of CVE-2020-8562, which can allow attackers to bypass filters that the Kubernetes project put in place to avoid misuse of the API server proxy feature.
For this post, we're going to look at the last of the four unpatchable Kubernetes CVEs, CVE-2021-25740, which relates to how Kubernetes ingress or LoadBalancer features can be abused to bypass network security controls in a cluster. The core of this issue is that users who can modify Endpoint or EndpointSlice objects, which define which Pod objects back services, can redirect traffic to addresses they don't otherwise have access to.
How Kubernetes Services and EndpointSlices route traffic
To understand this vulnerability and its impact, we first need to look at how Kubernetes Services and EndpointSlices work. Services are used to provide a stable network identity to a set of pods. They're an important part of Kubernetes, as they enable entities like load balancers and proxies to direct traffic to an application with multiple replicas, without them directly needing to track the IP addresses of the Kubernetes pods.
While the service object provides a stable network identity, services don't actually track the IP addresses of pods that provide applications. Instead, this information is stored in EndpointSlice (and, in older clusters, Endpoint) objects, and those objects are kept up to date by controllers running as part of the kube-controller-manager.
As an example, let's say we create a simple Kubernetes deployment of three replicas of an NGINX-based application and a service that provides a stable identity:
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.27
ports:
- containerPort: 80Service
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
app: nginx
ports:
- port: 80
targetPort: 80Once these deployment and service configurations have been applied to the cluster, an EndpointSlice is automatically created that provides the IP addresses of the three pods in the deployment. This EndpointSlice can then be used by the service object to send traffic to one of the pods, when requested.
kubectl get endpointslices.discovery.k8s.io
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kubernetes IPv4 6443 172.18.0.2 12m
nginx-tkrxf IPv4 80 10.244.0.6,10.244.0.5,10.244.0.7 87sCVE-2021-25740 technical details
The mechanism we've just described can cause security issues in some cases, especially in multi-tenant clusters that have shared ingress or load balancer services, and it's this risk that's captured in CVE-2021-25740. In a cluster with multiple tenants that do not have direct network access to each other's workloads, a malicious application owner can modify their own EndpointSlice objects, causing shared ingress and load balancers to redirect traffic to endpoints in other tenants' namespaces.
For example, consider a cluster that has two namespaces for different tenants. Each tenant has a service that they own and can access, but they're prevented from accessing each other's services by network policies and load balancer rules. The load balancer routes traffic based on URLs, which then use services in the two tenants' respective namespaces to work out which pod to send traffic to. The load balancer must have network access to both namespaces in order to fulfill its function, so network policies don't affect traffic coming into the cluster via the load balancer.
In this cluster, each tenant has full access to their own namespace, including rights to edit EndpointSlice objects.
If our attacker wants to access the pods in the victim deployment, they can edit the "attacker EndpointSlice" in their own namespace and enter the IP addresses of the victim's service. (We're not addressing the question of how the attacker gets those IP addresses, as it wasn't covered in the CVE.) After those values have been updated, the load balancer will send requests for http://attacker-service on to the victim deployment, allowing the attacker unauthorized network access to that application.
This vulnerability can be mitigated at a couple of different levels. First, from an architectural standpoint, avoiding shared load balancers and ingress services mitigates the risk of traffic being misdirected in multi-tenant clusters. Users of ingress services might consider migrating to the new Gateway API architecture, which doesn't have the same reliance on namespace scoped objects.
Another mitigation in clusters where this risk is present is to control what access each tenant has in their namespace. Generally, EndpointSlice objects should not be directly modified by cluster users. They're managed by Kubernetes controllers, so removing privileges to those objects can be an effective mitigation for this issue.
If you'd like to try this exploit out, there's a full proof of concept available.
Conclusion
As we close out our series of posts on unpatchable vulnerabilities in Kubernetes, some clear themes have emerged across all of the issues we've looked at. First, they all relate to Kubernetes networking, which shows how complex this topic is. In addition, each of these vulnerabilities is only important in specific cluster architectures, so not every cluster operator needs to think about how to mitigate them. It's important that every cluster operator thinks about what their threat model is and which risks they need to focus on.
