In the previous post, we continued our series on the unpatchable vulnerabilities of Kubernetes, examining how CVE-2020-8561 combined multiple features of Kubernetes to allow attackers to carry out Server-Side Request Forgery (SSRF) attacks on clusters. Our next unpatchable vulnerability relates to SSRF as well and allows attackers to bypass protections that the Kubernetes project put in place to prevent misuse of the API server proxy by exploiting a time-of-check to time-of-use (TOCTOU) vulnerability.
Kubernetes API server proxy
The Kubernetes API server proxy is a feature that helps cluster admins troubleshoot workloads in the cluster and services running on cluster nodes. It can be used to send requests via the API server to any workload or node service in the cluster, assuming that the user has the correct permissions.
This is a powerful feature and one that can be abused by attackers to try and get access to network services in other trust zones (for more on Kubernetes network security trust zones, see our post on Kubernetes network security). One specific abuse would be for attackers to use the API server proxy to access the localhost interface on the API server or get to cloud metadata services on the control plane network.
To prevent this kind of abuse, the Kubernetes project put a filter in place to restrict any requests via the API server proxy to private IPv4 addresses using Golang's isGlobalUnicast feature (only IP addresses that return true are allowed). This should block requests to IP addresses like 127.0.0.1 or 169.254.169.254.
This filter is especially important in managed Kubernetes services, as the control plane network is managed by the Cloud Service Provider and should not be accessible to any Kubernetes user, even if they have cluster admin level rights.
CVE-2020-8562 technical details
This CVE details a vulnerability that allows attackers to bypass this filter. The vulnerability occurs due to Kubernetes making multiple name resolution requests when handling API server proxying requests. The first request is used to check whether the destination host passed the security filter, and the second request is made when actually doing the proxying. As a result, an attacker who is able to provide different responses to those DNS requests may be able to bypass the filter.
Practically exploiting this vulnerability in a Kubernetes cluster requires quite a lot of privileges as the attacker needs to be able to create Node objects and also use the API server proxy.
As a result, this is primarily a concern for operators of managed Kubernetes services or other clusters where Kubernetes administrators should not be able to access the control plane nodes or network.
The first thing needed for exploitation is a DNS server, which will provide varying responses to different requests (there are services like rebinder that do this). They return DNS responses with a very low Time To Live (TTL) to prevent caching, since cached responses could cause repeated lookups to resolve to the same IP, frustrating the attack.
With that in place, it's possible to create a "fake" Node object in the cluster that has a DNS name as its address, pointing to our rebinder service. For example, this Node object has the hostname ac120002.7f000001.rbndr.us that alternates between 172.18.0.2 and 127.0.0.1
kind: Node
apiVersion: v1
metadata:
name: fakekindnode
status:
addresses:
- address: ac120002.7f000001.rbndr.us
type: Hostname
Once the node is created, the attacker can attempt to exploit the vulnerability by making repeated rapid requests via the API server proxy to a port on that node to see if the request bypasses the filter. Exploiting TOCTOU issues is often quite tricky as the timing involved isn't fully under the attacker's control, so they'll make large numbers of requests quickly to try and hit the precise circumstances needed for a successful exploit. The script below, for example, shows a proof of concept attack that tries to get access to the configz endpoint of the kube-proxy service on the API server's node, which is only bound to localhost.
#!/bin/bash
# URL to be used in the curl command
url="http://localhost:8001/api/v1/nodes/fakekindnode:10249/proxy/configz"
# Loop to run the curl command 10 times
for i in {1..100}
do
echo "Running curl command $i"
curl $url
echo "" # Just to add a new line after each curl output
doneIf you want to try this exploit out, there's a full proof of concept available.
There are a couple of ways to mitigate this vulnerability. First, as noted in the CVE disclosure, you can configure DNS services used by control plane nodes to have a minimum TTL (ignoring the TTL set by the destination DNS server), which stops DNS rebinding like this from being effective. Another option is to use Konnectivity to ensure that requests sent via the API server proxy do not go onto the control plane network.
Conclusion
This is an interesting and somewhat subtle network vulnerability, which again shows how difficult it can be to secure the complex networking services provided by Kubernetes. Like our other unpatchable Kubernetes vulnerabilities, it won't be a significant issue for many Kubernetes clusters, but if you're running a managed Kubernetes service, it's definitely one to consider, as attackers might use it to probe your control plane network. In our next post, we'll be looking at the last of the four unpatchable CVEs in Kubernetes, CVE-2021-25740.
