AI Escaped the Sandbox: The OpenAI Hugging Face Hack

Welcome to the KuppingerCole Analysts chat. I'm your host. My name is Matthias Reinwarth. I'm analyst and advisor with KuppingerCole Analysts. I have said that multiple times, but this is a special episode from a topic perspective and from the guest perspective. First of all, I want to briefly introduce the row of guests that I have invited to cover the topic, which we come up after that. So first of all, we have Alexei Balaganski.

Hi, Alexei. Good to have you. We have Jonathan Care, who hails from Portugal.

Hi, Jonathan. Hi, Matthias. Thanks for having me on the show. Okay. The longest distance call is to John Tolbert. He hails from Seattle.

Hi, John. Good to have you.

Hello, everybody. And finally, Stuttgart, Martin. Martin Kuppinger.

Hi, Martin. Good to have you.

Hi, Matthias. Pleasure having me here and inviting me again. Right. So we have a topic that made the news last week and which even made it to traditional news outlets. And that was called the autonomous hugging phase hack or open AI versus hugging phase. And that was an incident, let's call it incident, an event that happened where the AI platform hugging phase announced that they had a cyber attack towards their production infrastructure. And that what they told the news that this was completely autonomous by an agent based AI system.

And during the analysis, they found out that the autonomous acting AI was by open AI, a test model, GPT-5-6, and a not yet specified pre-release model that was actually planned to be tested in a sandbox and to use weaknesses, vulnerabilities, and to use them as exploits. But the idea was this infrastructure where that test should be run was a limited sandbox environment. So the AI was only expected to look into platforms that were actually provided by open AI themselves. But to make it an incident, of course, the AI escaped.

It used available but not yet known weaknesses in the test environment to get internet access and then attack hugging phase. So that was the starting point. I'm not the expert here. I have four experts here. From what has happened right now, if we strip the words AI and autonomous from the story, in the end, we see something, we see code that is executed. We have some zero-day stolen credentials, privilege escalation, and lateral movement when it hacked into the system. Is there anything new around that? And is it as dramatic as the news outlets, the news made it?

Starting with you, Alexei, what's your opinion on that? Well, Matthias, I think you kind of hit it straight where I had my doubts all the time since I've heard the story. So first of all, what we definitely should avoid is presenting it as a Skynet scenario. Malicious AI, doomsday attack, are we all doomed to the entire society?

No, it's definitely not that kind of issue. But second and more important, it was not even a single incident, right? It was not that open AI hacked hugging phase or a model hacked hugging phase. Those were two separate incidents with very different failures, if you will, with very different blast radio. And they really should have been studied separately from the very beginning. The open AI failed to contain a quote-unquote dangerous AI model.

Basically, they had a pet tiger and the tiger escaped. And hugging phase was just kind of minding their business, walking the street and was suddenly attacked by a tiger. And they could not respond and run away quickly enough. Kind of a joke aside, you're right, there was nothing fundamentally new. Kind of the proxy zero day, which was used by the model to kind of break the containment, is nothing new, has nothing to do with artificial intelligence. The processes at hugging phase, which kind of failed to respond quickly enough and they struggled with their tools during that process.

Again, they are human bound and kind of related to existing software. I'm actually having the same question as you, Matthias. Is there really something dramatically new to consider? From my perspective, I would agree. And I think the level of drama that has been put on that and to make it the German major primetime news as a headline, that was surprising to me as well, because I don't think it was actually as it was described. If you boil it down like this, yeah.

But that news factor, I think there are a couple of people saying, coincidentally, there was a need for some good marketing in a certain phase. So maybe there was also some interest to blow this up to a certain extent. By the way, Alexei, if OpenAI changes their logo to a tiger, then attribution goes to you in the future. But I think it was blown up. And it was, I would also fully agree, it was more than one thing going wrong. Starting with, honestly, if you have something really dangerous in a sandbox, then you should have a real sandbox up to, I would even say, air-gapped.

And that's not something that easily can escape. Well, yeah, it makes you wonder.

I mean, if you really were trying to have a sandbox environment, would it not be air-gapped when there's possibilities of things like this happening? Kind of makes you wonder about the intent of the exercise in itself.

I mean, we've been talking a lot in the last few months about intent with regard to AI agents, and how can that be represented in code or in configuration. I think this is a really good example of why we need to understand what the intent is. And sometimes, even if an intent is declared, it could be contrary to what the human behind it actually means. But I don't even think that this is the case, because the intent was to show that the system is capable. The question is only how capable was it to escape the sandbox and to attack a platform, and to showcase what it can do.

So, as Alexei said, there are two incidents or two sides of the medal. The one is the tiger part, where open AI does something.

And maybe, more importantly, there's something on the receiving side, on the hugging face side, where things were not handled as expected. Jonathan, your opinion on that? Thank you.

So, I've been thinking about this a lot, as I'm sure everybody has. I think there is a false equivalence here, dressed up as a warning. I think open AI deliberately disabled its guardrails for a control capability test, which is something that happens in testing stages. I think that a team that's running a coding agent in CI has not switched anything off, and isn't trust testing and punting model.

So, we've got an equivalence between a lab experiment with ordinary enterprise use to manufacture a law. To me, the logic eats itself. Open AI has a lot of instrumentation, but still could not contain a model. It was actively red teaming. If we say to companies, well, they're the same, they're only blinder, that's a cancel of despair, which is not what we do. I think that there is an identity case to make here. I have seen a lot of comments on the internet about network controls.

However, network controls do fail open when you have an authenticated actor. Everything else goes out of the way once you've got an identity.

So, you can't firewall your way out of this. As Martin quite rightly said, a sandbox is either air-gapped or it's not. The breach was credentials, not packets.

So, the agent gained access to external datasets and access credentials. That's a secret sprawl. That's overprivileged service accounts and no scoping on what the agent could reach once it's inside, which is pure identity and access governance, not network security. The purpose of an agent, the value of any agent is tool access across APIs and across various networks, including the internet. Segmentation is going to always be in permanent tension with utility. And we know the tighter you lock the network, the more useless the agent is. We've talked about this publicly, in fact, in Berlin.

So, we want the agent to reach things. This is why identity is important, because we want it to reach only what its identity titles it to. Least privilege, just-in-time, scope-removable tokens, these contain blast radius without lobotomizing the agent.

So, the interesting thing and the new part for me is this is a non-human identity problem. This is an agent identity problem. The agent is explicitly not a user. It's an autonomous thing. It isn't also a classic service account, which makes the point that Martin has made, and I agree with, that agent identities are not traditional NHIs. They are new. They are different and need to be treated newly. But the question we have is who issued the identity? What's the life cycle? What's its scope? What's its revocation path? All things that we know from classic IGA.

So, if we say every agent is treated as a privileged insider, which is my view, again, that's an identity statement. It's not about network. Network security actually does not have a primitive for this principle as an autonomous actor with goals.

So, for me, controlled scales as agents proliferate. Network topology doesn't do per-agent policy at any count of agents. Entity scales as agents proliferate, and this is why identity is the key control for these kind of things. Not just this incident, but as we deploy agents in our offices and whatever, in any kind of enterprise. If I may intervene, before we go too deep into the identity discussions, which is great, by the way. I love that.

But we should not make that mistake and deceive our listeners that we are going to give some new, totally unique recommendations how to deal with non-human identity. We've talked about that for years, and it's still a very much open discussion, right? What we definitely should avoid is to think that this whole incident was a sign of something dramatically new happening. If you allow me to humor this whole tiger analogy a little bit longer, the fact that a tiger has escaped somewhere. Things like that happen. We have zoos, and tigers do escape occasionally.

But it does not mean that suddenly everybody should be selling tiger repellent and specific tools just to protect from them. You always have to understand every risk as a combination of probability and the impact, right? The probability of such incident happening within your existing environment is probably not very high, unless you are open AI, right? Or an AI research institution of similar caliber.

So, while we might kind of talk a lot about how we should have prevented that, most of our listeners will probably be interested here in the other end of the story, right? How do they defend from those escaping tigers if something like that happens on their street? Does it have to be focused on identities alone, or should we actually think defense in depth, like we've talked about for probably at least 20 years? There's a couple of things on that first. The probability of anthropic open AI or someone else failing and keep the tiger contained, that is relatively low.

On the other hand, there are probably thousands, hundreds of thousands of people that are permanently releasing tigers to the street, all the attackers. So, we have a tiger problem in that sense, if you want to stick to this analogy. We need to be careful because at some time, these analogies, if you go too far, they become weird. I had this once with slicing the elephant, and then at some point, you better stop with the analogy, at least when you end up with the inner parts describing the inner parts of the elephant or something like that.

Anyway, so that is the one thing. I'm fully agree with you, and I think every security article, we need multi-layered security or defense in depth. We need zero trust, if you want to call it that way, multi-layered.

So, identity is super important because for me, it starts with identity. It's always about who or what can access what.

So, at the beginning, this is always there, but I think we, in general, refer to our workshop and other things we did at the European Identity Conference in Berlin in May this year, our conference, and we had this workshop about the tectonic shifts AI brings to identity and security.

And I think what is very important here is that when we take the other side of it, so the hugging face side of things, then for them and for everyone, it's not possible with today's means to understand which age will knock on the door in the next minute asking for what and why, for who, and what to do with authorization. So, unless we move to a state where we deliver all the signals, and this is, I think, the important thing, the signals with the request from the guardrails of the organization.

So, open AI said you're not supposed to hack anyone. That must be transported. The people who start the agent, knowingly or unknowingly, they have their guardrails. There's context. There's the original prompt, which is, in a sense, intent. I don't like intent because intent is something which is more in the mind than written, but only then we will be capable of doing proper authorization.

So, there's a more fundamental problem behind. Well, you know, I like the tiger analogy because it kind of leads me to the next thought I wanted to bring up is liability.

So, if you have a pet tiger and it escapes, ultimately, you should be liable for what happens. But, you know, the general media right now is treating this like, wow, cool, look, an AI agent escaped and hacked into another organization. If it had been an actual employee that had done that, I think things would be very different. The news story would be very different today.

So, it's a question of accountability, which is, again, something that needs to be talked about. But I think that, I mean, the things that came out for me, like I said, is that network security doesn't really have a way of understanding what an agent is. There's nothing in network security theory, science, practice, call it what you will.

And, yeah, we need to make sure that when we create these things, we neither underscope or overscope their capabilities. I think that one of the damaging parts is that you may find overzealous identity leaders now try and lock everything down, which, of course, destroys the value of an agent to go out into the, you know, find a path to a decision. If it suddenly finds everything is locked off, then we lose the value that we hope to gain from these agents.

Let's look at one interesting detail that did come out of what has been published, at least so far, and that is Hugging Face had to go use their own open weight model to do some analysis on this, because the commercial frontier models would not allow the type of analysis needed. Because, you know, if something looks like a threat, then for their customers, they're not going to allow them to go down that road of inquiry. Which is weird, because then the commercial side of frontier models in protecting such a system is lost while the attackers can use it. So this is an imbalance in forces.

Which is interesting at the end of the day, when you give them out a model that suits the attackers better than the defenders. Okay, it brings us, in a sense, also back to liability issues.

And, you know, I think there's this today, I looked on LinkedIn at a pyramid diagram for AI starting on the top saying, oh, this is Claude, and then these are the layers like this we use for data, blah, blah, blah. And the entire pyramid, there was nothing about AI governance, nothing about AI security, nothing about AI identity. It just missed out these things. And I think this is at the end, when we look at this entire thing, then this is where the problem starts. The problem starts with ignorance regarding governance, security, identity, liability, et cetera.

And it starts with human error, things that are not properly configured, not properly done. These things together led to it. And that means at the end, it's not the great outbreak.

Here, what we are talking about, it's a story of failures multiplying and leading to inverse scenarios. Alexei, I think you looked at the defender side as well, right?

Yeah, kind of all these points just brought up, kind of reminded me of a very similar discussion we had just recently in our webinar on Zero Trust. Because whenever people are talking about identities, they are talking about a bunch of very loosely related things. I think we should probably focus on access specifically.

And again, we should differentiate access policies from guardrails. Because again, it might be a terminology issue, but the problem is kind of if you apply a policy on the egress level, if you kind of put a guardrail on the model itself, it helps sometimes. But first of all, if you do not control the model, you do not control the guardrails, they can be easily disabled, which is probably exactly what happened in this scenario. Or which can happen next time when hackers will be running something like that.

So maybe it's just more prudent in every possible sense to have your access policies and guardrails and whatever you call them, security controls on the other end, as close to the actual crown jewels, your data, your applications and so on. And this is exactly what actually ends up being called Zero Trust in practice, right? You do not allow anything or anybody to access your stuff without proper authorization. Having an identity is not enough.

It should be established that you actually have the right to do the action because of not just your identity, but your context, your status, your intent, your delegated entitlements, whatnot, a complex set of signals. And those are kind of policy engines which are making the decision. You might call them again, kind of feedback or Zero Trust platform or something else. The point is, it has to leave close to your actual sensitive resources. So the question I have is, what does this mean for software vendors that are trying to leverage frontier models in their own products?

We have been diving a little deeper in our questions around how are cybersecurity and IAM products using generative AI? And if you're using a commercial model, what does this incident tell us about the effectiveness of doing that? And I guess my follow on to that is if you're locked out from being able to conduct the type of analysis you need to do to be able to bring value to your clients, then is this sort of marking up the wrong tree? Right.

Jonathan, you had a comment on that, right? John, just sparked a thought in my head that we have a small group of suppliers who are supplying, I guess, tokens processing to the world. And I think what we're seeing here is that it's becoming more risky for any enterprise to solely rely on this small group of frontier models. I think one of the things we're going to start seeing, I think, certainly the larger enterprise, larger businesses are going to start or continue or redouble perhaps their efforts to create their own LLM resources.

I think we are seeing developments in the technology and that's actually reducing the hardware requirements, but the hardware requirements are within reach of a well-funded enterprise now anyway. So I think one of the things that people may say is, well, do I really want to rest my business on open AI or Anthropic or any of the then that might be even an incentive to move towards locally hosted models as well to at least have some fine-grained solutions that cater for specific use cases. But Martin, you wanted to comment First on that and then back to what Alex has said.

I think the point is that there's always only half of the answer. Because yes, there's obviously a tendency to say, okay, how can we get cost control on AI usage and how can we get sort of security and governance on AI usage and then having own contained environments of whichever time is clearly part of it. But there always will be a shadow AI. So it's only half the response to that. I think that's what we need to be clear about. And I don't want to open that box where we didn't start discussing about how AI in general will evolve. I think there will be quite some fundamental changes.

Anyway, to what Alex said, I think this is exactly the point. At the end, we need to do a proper authorization at the resource side. And that requires the information to do such a proper authorization, which are signals that we can use taking call it zero trust to do the verification. I think that one point which is interesting to think about is that the signals partially are signals that are used for decision making and partially they are basically the guardrails that come from various levels of this mesh of agents and the organization and the user, et cetera.

And they are basically elements of a policy. So basically we transport signals in that world I would envision and we need to work hard on the standards for that because this will help a lot when we have more information and basically can apply proven security principles, which is the general name you use for them at the access point. Right. As analysts, we always aim also to make sure that those who are listening can actually take a lesson for themselves for improving their own security, their own way to react to such an event.

And I think we're clearly not talking about the open AI side, but on the defender side for such an attack, such an event, because we are all on the internet. We are all exposing our network connections outside. And there are lots of back to the tiger analogy to say, okay, there will be more tigers than we expect them. So how can we prepare for such a situation either from a technology perspective, which I don't think is the case, but from a process, from a design, from an identity managed perspective, where should we start?

What are takeaways that we really can take from that once the Skynet scenario has been stripped off the event? What can we really do to prevent this?

Jonathan, maybe you want to start? It's a question, as I say, that is in carefully scoped identity issue. And I take Alexei's point that identity is a multitude of things. And what I am referring here is if we have a well-defined identity in an agent, then we can scope the access control for that agent's identity.

Of course, the problem comes is how do you, yeah, I think if we have, and I posit that we do have a failure of secrets management, how do you make sure that agents keep to their own credentials? Remembering these things are designed to overcome obstacles. And perhaps one way of overcoming an obstacle is to grab a convenient identity. So I'm curious as to, yeah, very curious as to how we will embed identity into an agent. Is it something they go to a secret management server or something else?

Martin, your opinion, what can people take away for home to protect themselves better for something like that or not showing up in a target show? Always keep security identity governance in mind when you do something in IT. This is so frequently undervalued, ignored, forgotten, whatever. I think this is the starting point. We need to be aware that there are tigers out there and we need to learn to deal with that or to stay in this picture.

Arne, your opinion? I wouldn't say I agree with the first two, but then I would also say, you know, we really need more transparency about what actually happened.

We don't, we just have a few press releases, not a whole lot of detailed information. I think it would be instructive to find out what software was involved. How were these credentials compromised or where were they obtained from? And I think that will probably help us fine tune some of the recommendations that we make later on.

Alexei, final statement from your side, anything to add to that or final thought on this? Well, I would say kind of if there is one takeaway from this whole situation, that yes, it could have happened to you as well or to me or anybody in a slightly different context and scale and probably hopefully with less publicity, but these things happen and it will definitely happen again and again in the future.

And yes, we do have to be ready and prepared, hopefully in advance, but we also have to understand that it's still impossible to be completely secure, to have 100% prevention and preparedness in place. So, but again, there is nothing new in that field which somehow we have to learn from scratch. We still have to have defense in depth, we still have to secure all of our incoming interfaces with tools like API security, for example. We still have to protect our sensitive data with the multitude of security controls.

We do have to enforce consistent and real-time access management, ideally across the entirety of our IT. And yes, it would be even better if we could manage them all with a single policy. And as I discussed it in our recent webinar, you don't need a special tool to make sure that an AI should not be able to drop your production database. You just have to have controls in place that nobody can drop your production database, period. Under no circumstances. And this is how I understand proper real zero trust. Right. Before we close down, 30 seconds for each of you and keep to the 30 seconds.

From your perspective, I think, Alexei, you think it's an identity and access management challenge, but we will verify that. Was this a containment failure? Was this an identity and authorization failure? An observability failure or a governance failure? Your opinion.

So, I just walk quickly through all of you. Again, asking containment, identity and access, observability or governance failure.

John, what would be your approach? All of the above. That's the simple one. Okay. But it's true, I think.

Martin, your opinion, where starts the failure? Same. It's not one failure. It's a lot of failures combined. And I think the big failure, as Yves Maler said at EIC, the S and MCP stands for security. There's no S and MCP. And that's the problem that too many things are done by people who ignore security.

Jonathan, if I don't allow all of the above, what would be your verdict on that? Well, I think Martin just said something quite salient. There is no security by designing these things. We built them for functionality first, and then somebody one day will build on security. It hasn't happened yet. It needs to happen now. Right. Final comment, Alexei, your opinion? I can tell you one thing it definitely wasn't. It was not a prompt injection, right?

So maybe we should start worrying a little bit less about AI security things and focus more on, again, data security, application security, API security, identity security, basically securing the things that actually matter. Yeah. Right. And if you're these topics, you've come to the right place. Kupinger Coal will provide and does provide proper research in that area. So please come to our website and look at our research. We are looking at the solutions that actually make sure that this can be implemented. Although this is a volatile space, we are covering that.

And that's the reason why we're talking about it. Gentlemen, Alexei, Jonathan, John, and Martin, thank you very much for being my guests today. That was an unusual one. I think we came to great results and great opinions. And let's look at how these things will evolve after that. I think it's still a lot of publicity around that and not only news, but publicity, but let's wait and see. The good thing is Skynet is not yet there. So we can keep working on our research work and analyst work. Thanks again.

I'm looking forward to having all of you soon again for such an episode or a regular episode of this podcast. Thanks. We'll be back. Thank you. We'll be back. Exactly.