Hack a Cloud and Kubernetes

Okay, let's go. Yeah. My name is Patrick. I'm the CSO and co-founder of mondu. In my previous job I was a pen tester, so hacking hot software, discovered some zero days, and also during my time by S v A created from scratch the incident response team. So we did some forensics, reversing more there, and also ransomware negotiations to get the Bitcoins, the mon errors and transfer all the stuff so that we get the decryptor. Let's come to the problem. What we are seeing out there is that's a survey from props. It's 1100. IT and security professions were asked about ransomware stuff.

80% of them were attacked from ransomware group, and 60% out of this 80 paid the ransomware money. So that means what you see in the press is just the dip of the iceberg. Why companies care about security at first to stop the hackers and also to pass your audits, your ISO 27 0 1, your, your PCI, I D S S and so on. But all the compliance frameworks have one target to protect your data and to stop hackers. So it's the same goal from my knowledge, or we as small do we noticed we can reduce the main problem to two or three main reasons why hackers also successful.

One is no, not patched hardware, software and hardware. So no patch management in place. That is also for counts for, for docker containers, for the Kubernetes and so on.

Also, there are missing hardening. Hardening means to apply secure configuration on the system for your Kubernetes cluster underlying easy two instance or whatever.

And also, surprise, surprise, Microsoft noted that in the cyber ignorance report that 80% of the atex can be attributed to outdated software and misconfigurations. What we also see in the industry at the moment is that we have a budget misallocation. Let's focus at first on the circle here means get information about your asset inventory and also identify your risks. Then harden your system petrol system to protect that. Then you need something to detect the, the attackers in your environment and then that you respond to such an attack from, from ransomware, gangs or hackers.

And also the next step is to recover to the normal operations. What we are seeing is that most of the companies allocate the budget in the wrong, or from my perspective or model perspective, a little bit in the wrong direction. Because if you see, they're invested in security incident event monitoring, edr, hdr. So most of the capital is in detect and response.

From our perspective, that is too late because you can only detect if you, they had the attackers already in your or attacking your infrastructure, but you can do something before patch and harden your system that you can really close the doors and that is the misallocation, what we are seeing right now. Then also, let's focus a little bit on compliance frameworks. Which compliance frameworks are require hardening and secure configurations. That's new in the update of the ISO 27 0 1.

That's speaking about including security configuration also in the BSI and also cyber risk insurance questionnaires asking if you have guidance to secure, to apply secure configuration to a server and endpoints to make it a little bit more visible. You see that you have open doors and you put a camera or 10 cameras on to survive or to surveil the open door. That means you are investing in a 24 7 security team.

I, I don't want to say that is wrong, but maybe you have to, to switch a little bit of the budget in the protect page because for me it makes more sense to close the fucking door and close it and, and, and lock it. And don't put the camera on that. What is the solution? It's extensively security poster management. What is poster management at first? Get data about, about your infrastructure, collect all the stuff, then do a risk assessment to identify your potential risks for vulnerabilities and misconfiguration. Then put counter measurements in place to mitigate your risks.

Have a system in place that monitors and reviews all the time, your counter measurements and yeah, for, for, for ransomware attack. Be prepared with incident response plan. XPM monitors the whole infrastructure because what we are seeing at the moment, you have tools for the cloud native applications or the cloud security poster management. They are really good in working for the cloud, but you cannot really use that tools in your on-prem infrastructure for a VMware, for your endpoints and so on.

For that, you have other tools on-prem that are not work, really working in DC ICD pipeline that you really can support your developers and, and do the shift left action. And also it's not working really with Kubernetes cluster and your SaaS services such service like M 365, Google Workspace and so on. So that means at least you have two tools to monitor infrastructure that mean you get a fragmented view on your infrastructure and with fragmented data, you cannot really decide on a, on a good counter measurement or risk as evaluation too.

That means you need a unified view on a comple complete infrastructure stack for every layer in your infrastructure. And also you, you have to, because we as small to develop open policy as code and these policies can be used already locally for the developer to scan the Terraform template.

The Kubernetes manifest, although you can also put it in C I C D, you can during the build time to run that in your pipeline processes, you can scan with this open source policy as code now open source tools already the cloud, the Kubernetes, and all the stuff that you really get all the information together. Let's do a little bit of hacking. I deployed and Kubernetes cluster in aws. It's just one note and I deployed one pot with one container with the dam vulnerable application. This web application has remote code execution. I will show you that and we exploit that.

The green rectangle is the dam vulnerable application, so that means on my attacker machine I de started a web server. That web server is delivering two malwares or two binaries. One is we using the remote code execution to download from our, from my web server. The first binary execute that, and then we get, we call it as a pen tester or an hacker, a reverse shell.

That means this container connects back to my mission and then I have shell access to that and after that we do a privilege escalation because I really need route rights was in the container to do the container escape to compromise the underlying Kubernetes node. And if you as hecker have the underlying Kubernetes node, you lost your, the complete control over your complete Kubernetes cluster. It does matter. It's just one node or 10, it's just a scaling factor. So that is, is it big enough?

Hopefully this is this one Kubernetes node and this one pot that is running and this one is running this web application. And to check that you have remote code execution just through a semicolon and ls, and then you get listed the underlying files from the file system within the container itself. Then here I have started my web server. Then I'm listening for the container shell. That is the reverse shell that connects back to me from my container and already I owned also an connector to listen for the reverse shell for the underlying Kubernetes node. Let's do that.

At first I use this command, this cold command to download the binary met container for my web server. So let's open the web server and put that in. Now you see here that the container connects to my web server and downloads this binary. Now I have to make this binary executable and execute the binary and hopefully everything's perfect. Go right. So that is the reverse shell.

Now I, my process is running as ww dash data. That is an unprivileged user to perform and container escape. You need root rights within this container. Let's get a minimal shell. Right now I can type commands like be commands and to gain root access. I pre-com compiled on my three bucket. It's called the paw kit vulnerability. It's a C code. So I pre-com compiled that for AMD 64 architecture and put it in my S3 bucket. Let's download that one.

Execute, Make it executable to binary, execute a binary and get the shell itself. And now just check we have route rights. That means that's a privilege escalation. So right now I am the, the God within this container, let's perform the container escape itself.

I copy, I will discuss that here. Oh wait, I, I copy it in the shell at first. Perfect. Here. I'm mounting DC groups and then I enable within DC groups to notify un rerelease agent. Then with this set command, because every container lifts under via lip dock or per default, you can configure it to that is other location that are store.

But with this command, I find out from the container where I live in the underlying fire system of the Kubernetes node in the fire system and this pass, I will save that with a little extension CMD, because that is my shell script that we really want to, to, to put some commands in, and I will save this pass in the release agent. It is shell script has the shebang, so that's such a shell script.

Again, download the second binary for my web server, make the second binary executable and run that and make my shells script executable and to put a dollar dollar into DC groups. I trigger that the C groups loading the past that I previous put in in the release agent. That means if you execute the staff you see on my web server that the underlying Kubernetes node was connecting to my web server downloads binary and execute that. And here I have Elle to confirm that I have rude rights on the underlying Kubernetes note. Don't trust verify. Let's verify the host name. It's 5 2 0 1.

It's 5 2 0 1. So we compromised to complete Kubernetes node. Every workload that is running in this Kubernetes cluster is compromised.

Okay, let's see what happened here. The first problem was let's go.

Oh, With our open source tools, you can already scan the Kubernetes manifest itself. That means the, the, the manifest is the yamble fire. You can come to our booth and we will get in detail in this stuff. So that means we have our query language and here we have a problem because this container was a privileged container. That's a misconfiguration. I see that from time to time as a pen tester because the, the, the guys are thinking, I have to put that as a privilege container to consume the in vita graphic cards for my AI calculations. That's not true anymore.

Five years old or four years ago. You need that, but for that you have special interfaces, so use that please. And then also we are giving you also the policies are open source giving you how to, to mitigate the stuff. The next question, so that is the problem why we did the containers kept that we get the underlying Kubernetes note. What was the problem that we get root rights within the container itself. That was an unpatched vulnerability within the container. That means because you are using containers, you are isolated, not really.

So you really have to keep an eye that every container in Kubernetes node is really patched. And where the lip Paul kit, here we go. That is this vulnerability that's unpatched. So that means that a couple of advisories in CVE belongs to this package. You have just to update that. Yeah. Any question? We have three minutes, Anyone?

Oh, then I have a pretty trivial question. So how do we fix this problem? Petro container? So create the, or start the pipeline again that this pipeline creates the container and update all the stuff that is running within the container and adjust the, the, the what is called the Kubernetes manifest itself. Let me show you this is the misconfiguration just put privileged from true to false.

Well, the question is will your solution or any similar solution actually help me that will it automate it? Will it tell it to an appropriate person in explainable terms? Because that's I think the biggest challenge. Vendors have been talking for years, so we are not doing prevention, right? We are missing, so things for we have to do a lot of detection. Yes.

Saying no, that's the wrong approach and I tend to agree with you, but like how do we fix it? How do we put it back? At first you need transparency because if you, if you go to the DevOps guy and want that, he's also a half pen tester. That's not possible. So that means as policy, as code, you put the, the knowledge, the security knowledge in this policy and run it in DC i c d pipeline. If the developer de commits such a Kubernetes manifest in the pipeline, then the pipeline is red and he cannot really push the commit in the domain or the master branch.

So detected early to, to really reduce the cost because if you just detect the stuff in the production, then you have to open a share a ticket. Four, six weeks later, the developer gets, then he do some adjustment. Then you notice, oh, that's not correct, another ticket and another iteration. If you put this policy as code in front of the developers to the pipeline as close as possible to the developer, then he knows which, which standard he has to fulfill. Right. Okay.

Well, if you have no further questions of the audience, thank you very much. That was a really practical and impressive demonstration of how things can go wrong. Thank you very much.