When Models Run the Business: Risk, Trust, and the New AI Identity Economy

Thank you for attending today. Thank you for having me here at this conference. This is my first time being at this conference. But what I'm going to talk to you today about is the following. It's when AI models actually run your business. We're getting to a point where if you've been listening at all today, the conversation has been all about AI, about frameworks, about how AI agents are actually taking on roles that were previously not anticipated for.

So really, the question for me when I think about this is this, is by a show of hands, if you can, how many of you believe AIG has actually arrived? No one?

Sebastian, come on. No. Okay. So then the follow-up question actually doesn't have any bearing on it. It's now keep your hand up if your company uses AI for more than 30% of its knowledge work.

Well, maybe that's the question I should have asked first. How many of you have 30% of your business managed by AI? Not a lot here. Interesting.

So, you know, this is a thing that I've said and argued many, many times, and that is if you just told me that AIG isn't here while simultaneously admitting that you're already deploying it at scale, that's a cognitive dissonance. And that's not a technical problem. That's an identity crisis. And it's about to become your biggest risk vector. So really what I start to want to manage for this is this, is that there's really two data points that are destroying this waiting game. I argue that if your company already is deploying AI agents, that you have AGI in your company.

It knows everything about your company. It knows all of your customers. It knows what your products are. It actually answers questions through chatbots. You have AGI for your company.

Now, we can disagree on the different definitions of AGI, but when we start to really sit down and think about what is the opportunity here and what are we really discussing, this is an important discussion that we need to make sure that we are on the same page. So I'm going to give you two case studies here. The first is Anthropic's reality check.

Anthropic, when I first put together this slide deck, was announcing at that time that 60% of all the code that Anthropic is writing for themselves was generated by their own AI. Subsequently, that number has now risen to 90%. If you can't imagine that AGI is not already here, then you have to step back and ask yourself, well, when these AI or these frontier AI agents start creating the tools that we're all now reliant on, haven't we already received this and haven't we already gotten there?

The other, of course, is McKinsey. So McKinsey had a workforce shift, so to speak, and what's interesting about what they did is this, is that they used to have about 40,000 people.

No, they had 65,000 people that worked for them. In the last year alone, they've dropped the number of their headcount down to 40,000 and they've actually hired 25,000 AI agents. So when you start to look at this and you say to yourself, oh, and by the way, McKinsey is actually estimating that the number of agents will be on a one-to-one ratio within the next 12 months. You can't deny that we're already in a situation where AI is actually starting to impact us in ways that we have never, ever thought about.

So there's a, you know, this is the argument that I was making just a moment ago, and that is, is that are you preparing for AGI? And if you're not preparing for AGI, then you're just pretending. Because 25,000 uncredentialed, unidentifiable entries running McKinsey's operations suddenly don't count as general intelligence because we can't agree upon a definition.

Now, what I just said, and this is actually a quote from McKinsey, and that is that when you start to think about this, they're admitting that their AI agents are uncredentialed and unidentifiable entities. Everything that you've listened to today has been every agent has to be in a registry. Every agent has to have guardrails. Every agent has to function within its capabilities. What we're seeing is that is not true. That is not what we're seeing out there. So really what it comes down to is this, the risk of AI agents is not theoretical. It is operational, it is legal, and it's reputational.

So what we have here is this thing that we refer to as an identity inversion, and that is when your workforce stops having names. Now, this is an identity conference, and we're talking a lot in this conference about AI agents, and yet at the same time, we really don't give identity. If you go out there and every one of these absolutely amazing organizations out there, all these vendors with their booths are telling you about the magnificent solutions that they're having, the problem is is no one's adopting it, and that's a problem. It's a huge, huge problem.

So really we're kind of caught in this trap. It's a language trap where what we have is this, is we have these moving goalposts, and really what's happening is is that we're now looking at a situation where reality is much different. So AI performs already 60% of the top work at top engineering firms.

And again, McKinsey agent to human ratio will reach one to one. When I first created this, it was 18 months. It's now about 12 months away. So the real question is not that is AI agents deployment in your organization AGI, but can you identify who just approved a transaction? It was last week that I was reading about EY, so it's the consulting firm. They're actually going through a process of where they're encouraging all of their staff to start to create AI agents.

And you think about it, you go in a consulting firm, they have an accounting background, they know what they're doing, they know what they're going to have. They're creating agents at such a rapid pace that there's no way that there is anyone within the organizations that has control, mastery, or even guardrails for what these agents are doing. There's conversations and stories about how deep mind Google engineers are creating agents and then coming back after lunch and realizing that the agent has deleted everything that they've been working on for the last 18 months.

Or where an agent is released into the wild to optimize the company's database and the optimization was we're going to delete everything, including all the backups. So what we're seeing is this, is we're seeing that there's a lot of changes. So we have this traditional model out there. It's humans have credentials, tools have version numbers, and regulatory compliance is built on this idea of identity. But we're in a new reality. Agents make decisions autonomously. Models evolve continuously.

You know, we're looking at credentials that do not exist. This is an ideal. We should have frameworks where there are agents that have credentials and those credentials can be revoked. Not only just granted, but revoked. We're in a new reality where that is still the ideal. It's a framework that we need to be working on.

So really, again, when I'm talking about this, this McKinsey study is really quite interesting. So they have these 25,000 agents. And the questions that you would normally want to ask, these are all risk questions. Which agent analyzed that Fortune 500 restructuring plan? Do we know which agent did it? Was it a multiple set of agents? Was there an orchestra of agents? Everyone likes Clodbot, you know, or Paperclip. These are orchestrated agents that manage groups of people. The question is really going to be is, who actually did that analysis?

Or there's another one, and that is, what was its training data? Is there a bias in that training, that agent's training data? And then the third thing is when it hallucinates a financial model, who is actually liable for it?

Well, this is an important quote right there. I mean, statistics. It's 85.6% of all agents deployed are deployed without formal security approval. This is done by a study last year, just six months ago. When you start hearing that everybody's agents are all registry, and everybody's agents are being deployed with the full knowledge and consent of an organization, I'm telling you that that's not the real story. The real story is that you almost have 90% of all agents that are being released are being done without the knowledge of the organization.

So, we really have a couple of reality checks here. So, I like to ask questions.

So, here's the question that I have is, does your organization currently track which specific AI agent's models make decisions? By a show of hands, how many of you have that?

Yes, full tracking. Anybody? How about partial tracking? We have a partial tracking back here. Awesome. How many of you have no tracking in place? There's a brave man. He raised his hand for that. Or even the worst, which is ignorance. How many of you have no clue? Absolutely no clue. Another brave person.

So, really, there's five risk vectors that nobody's pricing into this. The first one is, of course, a compliance collapse. Without the ability to actually audit what your agent is actually teaching or doing or transacting, you have no way that you can meet any compliance standard that exists in the European Union, in the United States, it doesn't really matter where. I'm sure there are some areas where no compliance is the compliance, so maybe that's a good thing. The second thing is the attribution failure.

Well, the first question that you would be looking at is, is there hallucinations? We know about hallucinations. We hear about it all the time.

Last week, a Supreme Court submission, a submission to the U.S. Supreme Court, was prepared by an AI that quoted cases that didn't exist. You would think that the people who were trying to do something would not make that kind of obvious mistake, but they do.

So, we have all these things. You know, is there fine-tuning of the data?

So, is it contaminated? Is there a bias in there? Is there a way for you to actually determine where that particular agent comes from? We also have now a model drift and ghost updates. When you create an agent, are you creating it with the latest version of the frontier AI? Are you? Are the agents that you created, are they actually working on older versions or legacy versions of AI? It's really kind of a funny thing. We have this linguistic problem about what is new and what is old.

Previously, when you thought of, say, like GPUs or CPUs, there was this 18-month cycle between these growths in terms of the capacity, the number of transistors, the amount of calculations they could take. And that was something that scared people in this world since the early 70s when George Moore actually created this philosophy. But really, what you're starting to now see is this, is you're seeing that we're having AI agent updates at a six-week interval. What was state-of-the-art six months ago is now a legacy system that you can't even access oftentimes.

So you have this drift where week one, you have an agent possesses compliance check. Week two, the model updates silently. Week three, now you have this behavioral drift for the agent. And then by week four, four weeks, we're seeing an unvetted version that's now released into production. Our risk sector four is this synthetic identity fraud. Simply because an AI agent is out there doesn't mean that organizations, nefarious organizations or malicious individuals can't create a synthetic version of it and actually create fraud.

The last few speakers that are up here spoke about fraud, so I'm not going to really go into that. But the idea behind it is that there is a lot out there that you need to be cognizant about. Here's one that is oftentimes neglected. It's your reputational clarity. How is it that you're being, how's your reputation being managed? When I think of McKinsey creating 25,000 AI agents, I'm going like smartest people in the world, creating AI agents using the latest tools, and it's got to be good.

But I look at it and go, wait until one of these little agents makes a huge mistake and now McKinsey is having to pay out a multi-million dollar penalty or a settlement. These are huge, huge problems. You wouldn't have that necessarily with a human, but again, scaling a human is difficult. So really what I'm trying to say is this, is that we're in an AI identity economy where verifiable credentials are this new trust layer. So we have this paradigm shift. So the old question was, what or can AI actually do the job? And the answer was very straightforwardly, yes, decisively, even at scale.

But really the new question is, can we prove which AI actually did the job and under what constraints and under whose permission, whose authority? That is now the entire game. I talk about these things as auditable. If your AI agent isn't completely fully auditable, then you really don't have a structured framework that manages your AI agents. So this audit test is this, and again, if you want to vote with your hand, that would be great. Can your company currently prove which AI agent produced a specific output from six months ago? I would dare say no one can do that.

Because you weren't even logging some of the things that we're doing. You know, there was a discussion today about logging, what an AI agent does.

You know, they're so fast, they're doing so much that they're creating these massive logs that you almost need an AI agent to actually help you go through the logs to find what's relevant. So there's really three pillars of AI identity infrastructure that I believe that needs to be out there. And that is, of course, and this is the mantra of this conference, and of course, the word of the day is signals. I don't know how many times I've heard that today. These are the signals that I want you to understand. And that is cryptographic agents require credentials.

The second is that behavioral signal architecture. You really want to make sure that when you've created an orchestra of AI agents, that there's an architecture to it, and that it's not slapped together. And of course, there should be delegated authority frameworks. So in asking yourself to self-diagnose where you're coming from, the first thing that I would ask is this, is what should you focus on first? Which pillar represents your biggest gap?

Now, this is going to be different for every single one in this room. Is it going to be that you need cryptographic agent credentials? Or do you need an immutable audit trail? Or do you need an AI governance framework? Or even worse, I need all of the above.

So really, there's a couple things that I want to say. And that is that we're at a point where we'll have an operational tipping point, where there is going to come up a moment where there's a 100% regulatory mandate for organizations to adhere to. We're not there yet. We're kind of in this suggestive phase.

You know, if you look at GDPR, you look at DORA, you look at MECA, you look at all the US regulations, you're seeing these things as kind of like highly recommended frameworks, excuse me, or suggestions. And really, what we need to understand is that these aren't suggestions. Once the regulatory framework changes, are you going to even be ready or even close to ready to understand what you need to do for compliance? So there is a practical roadmap. And that's, of course, two tracks.

One is an immediate identity hygiene, you need to go through and inventory your agents, you need to assign unique identifiers to them, you need to have now massive logs that manage this. And of course, most importantly, define authority ceilings. We know that guardrails don't always work on AI agents. So that's an important thing to do. And of course, track two is this strategic trust architecture. You need to actually partner with identity platforms, all these guys out there, go and talk to them, please. They are desperate to tell you about their products.

Their products may or may not be relevant to you. And your particular case study, but it is something you should be paying attention to. And then really what it comes down to is, is you need to find a way that if you if 40% of your workforce doesn't have a birth certificate or an employment contract or a background check, really, then the question is, what are you going to do?

With that, I would say thank you very much. Thank you for having me here. And it was a pleasure speaking. Thanks very much. Thanks for picking up on the themes.

Yeah, signals is a very important theme this year. We're sort of lost time again, but a quick question. Looking at the AI agent traceability, how much of this should be solved by legal diversification or instead of technical traceability?

Well, I think really what happens in most technologies is that the technical situations emerge first and then legal response to it. It's really hard for a lawyer to actually construct something from a technology standpoint, but they're very, very adept at finding what's wrong with the current implementation. So it's always going to be technology. And then legal will lag. Great. Thanks. Give it up for Brian Nielsen.