Hello everyone, EIC26, fantastic to be here, To Bot or Not to Bot. I want to explore some of the things that you've been hearing over the last couple of days, provide a very simple reference architecture for simple questions to try and answer and also point you in the direction of some fantastic keynote speakers and session speakers through the week that you can take a deep dive and have a look at their presentations or re-watch to help answer the question. So I'm looking at this question of To Bot or Not to Bot as kind of the Shakespearean question of our time.
So in Hamlet, there was this question to be or not to be, this idea of wrestling with the weight of consequences. And I think right now, if we think about this tectonic shift from a society point of view, we're facing that same question. But instead of thinking about a prince and power, we're thinking about what power we want to give to an algorithm. And before we deploy an AI agent, which is a super simple thing to do in seconds, to act on our behalf, do we really understand what we are unleashing?
Are we taking the pause to think about the consequences and the downstream actions that may happen? And the aim of the talk today is to just start thinking about how we can shift some of that intentionality further upstream. I don't need to go into this. We've heard this pattern discussed on stage over the last couple of days. A year ago, when Martin first started talking about AI from an identity point of view, the aim of being able to monitor security, fast forward a year, and we have agents doing absolutely everything for us. So the question now is, this is not really just a tech problem.
This shift in such a short period of time is something that's really impacting the whole of society. Just minutes ago, before I came on stage, I got a notification on my watch, New York Times, that Governor Newsom in California has just issued an order to look at the impact of AI displacing the workforce in California and the downstream impacts that's going to have to society, health, affordability, income. In the article, it quotes Anthropic's CEO saying that he thinks in the next few years, we could lose as much as 50% of white collar workers.
I mean, that is significant. So let's have a look at this question of bot to not to bot in four acts. So the first one is, who is the agent and the identity problem? How do you give an AI agent a verifiable, auditable identity? How do you bind that identity to a human or an organization who will bear that ultimate responsibility? And I've heard debates on this stage during the course of this week around, you know, to what degree that is or isn't required.
But giving an agent identity, we know, we already have the tools in terms of credential certificates, OAuth scopes, and acknowledging that there are gaps right now. There are standards emerging, but we've got gaps. So we're starting to join up some of those things. But we don't necessarily have an end-to-end capability.
What I would encourage you on this, rather than going into this, is to go back and have a look at this fantastic workshop that was done by the Coop & Go Coal team on day one, where the team have mapped these 15 fundamental changes that are going to impact and listening to the speakers that are just on stage, I think we're seeing some of this best practice coming through. So that's the first thing. We've got to be aware of how this is going to fit with our identity infrastructure.
The thing that scared me about Jonathan in one of his discussions was that he was thinking that this idea of human-to-one agent could very, very quickly become one to 10,000 agents. So what does that look like from an identity infrastructure in an organization? So let's jump to Act 2, governance. Accountability risk and human in the loop.
Again, I've heard both sides of the argument during the course of this week in terms of whether or not we must have humans in the loop, I think we do, versus the fact that if you have 10,000, 100,000, millions of agents, it's just not going to be possible. So if it's not going to be possible, binding a human to some sort of mandate chain as upstream as possible is going to be critical. An agent identity has to be traceable back to some natural or legal person. Why?
Because we haven't even started to map and understand the downstream risks and issues, and without that clarity, it's going to be very difficult to work out who is responsible. One of the problems that we already know, and this is happening every day in sort of everyday use cases, like booking airlines and shopping, is credential sharing. Humans handing over their credentials to agents, their passwords, and this is a liability time bomb for organizations and for regulators. Something goes wrong and the human says, well, I didn't do that, the agent did that.
So operating without identity, what happens when agents act autonomously, there's no audit trail, audit trails vanish, liability becomes unclear, breach attribution is impossible. So the things that seem really simplistic in deploying an agent upstream and maybe a developer thinks, hey, this is a cool thing to do, if that's not mapped against the downstream risk for an organization, then we haven't even started to see what the cost of that risk is going to be. So this argument now about human in the loop. So who's responsible when an agent causes harm?
Human in the loop is not, from my perspective, a limitation. I agree that it's going to become an overwhelming task if we can't work out how to upstream that and have the right surfacing so that a human steps in at the right time, but if we don't have that active relationship during this phase of designing, then we're going to find out the hard way through the downstream consequences.
And one of the ways that we can do that is to have these narrow mandates in the framework right up front, define what an agent can and can't do, what they must not do, and how we can enforce those technical boundaries. So if it's a case that it's not possible because of the velocity and the number to be able to have a human overseeing every action, then we've got to work out what things are surfaced and the mandate to do that, because what's not clear right now is when something goes wrong, is it the developer? Is it who has deployed it? Is it the model? Is it the user?
Where is this liability going to sit? Doing the risk assessment before the deployment, working out how to narrow the mandate, what's the governance model that's going to sit above it, what's the liability profile that you can tolerate, this has got to be defined before the agent is given access. Once this is done after an incident, it's going to be too hard to put the genie back in the bottle.
So when we're looking at human in the loop, the reason it matters is that this is not just about ethics or agents today, it's often because the agent or the deployment may not have been trained, tested or validated against every specific use case. So the autonomous high-stakes decision-making without oversight, unless that mandate is very clear, and we've seen this play out in the courts, in the media, over the last month or two, with Anthropic and the Department of Defense in the United States.
And so Anthropic is saying, look, the reason we're trying to limit the use of our model is that two of the reasons that you want to use it, either in autonomous weapons or for surveillance, we haven't necessarily built the model or tested against those specific use cases. And I think this is something we're not necessarily talking about. Just because an agent can doesn't mean it should. And so one of the critical factors before delegating is looking at whether or not the agent or the model is fit for purpose for that use case.
If not, and it's a case of experimenting or designing or looking to see if you can retrain the model, what's the override mechanism? In terms of how this happens in the organization, I'd encourage you to go and look at Nat's, if you didn't see it, Nat's talk from this week. Two key thoughts from that. He was looking at agents from the perspective of agents as employees. And when you think about it from an employee point of view, there's just no way that we would allow an employee to do many of the things that we're allowing an agent to do. My key takeaway from his talk was this idea of UBO.
I'm very familiar with that from a banking, financial perspective, but the ultimate bot owner. If you can't identify in your organization who the ultimate bot owner is, then there is a significant issue that is likely to eventuate downstream and it'll be very hard to identify.
Next act, to bot or not to bot, the bigger picture, geopolitics, children, society. Zooming out, one of the things that is a risk is that we see that we're living in a very interesting time geopolitically. Countries that we used to have strong alliances with, things are reshaping, there's rupture in trust, and we can't just treat this like it's just a technological shift. It's more than that. We have three superpowers that have very different ways of serving society, and this is very generalized, but we know in the EU we spend a lot of time on trying to protect rights.
The US is racing on innovation, and China is saying, look, the state's going to decide. How, if your organization is working across different boundaries, do you start to reconcile those diverging regulatory philosophies? And I know it's very easy to point often at China and say, you know, the state is involved in things and we think there should be more freedoms, but one thing that China has done very recently in the last month is they've made it very clear that they're going to ban synthetic relationships for children under 18.
Now, what's behind that, whether or not that's about identity, emotional dependency, manipulation The one thing I would say, while we criticize sometimes the idea of a state making those decisions is, what we all know is that we're ten years into mobile and social networks or more, the results are starting to come out in terms of the harm to young people. I read a study yesterday of the number of children that are getting their phone out in bed after midnight, and they're online between midnight and 4am, and their family has no idea.
What happens when it's not just the phone, but it's a synthetic relationship? And so this idea of being really clear about what the relationship is and the downstream harms is that this is something that we haven't...we don't have the evidence for it yet, we don't have the data, but if we're not thinking there is going to be consequences, I think a generation of harm, it's too late if we find out at that point. Data sovereignty, agents can cross borders.
If you have different regulatory regimes, if you have different philosophies, an agent operating across jurisdictions, it may unknowingly violate data residency, privacy, surveillance laws, and again, upstream, who's responsible for the consequences of that? And mass surveillance risk, again, if you have models that are competing with each other, agents that are acting without identity, they can become instruments of surveillance by default, or by design, baked in. So who's watching the watcher?
What's the governance model to understand that something that you have deployed with a very narrow purpose and mandate, and you're clear about that, but in what ways can that possibly be weaponised? So the regulatory horizon. I would hope, expect, that identity accountability frameworks to become mandatory, not optional. Certainly here in the EU. Whether or not we see those things in other jurisdictions, who knows? And then the problem we've got is how do we reconcile where we have those? I see Bryant here in the audience.
One thing that I found really interesting, I would encourage you to look at his keynote, and the thing that I took away from it from an accountability point of view is this quote from McKinsey's and deploying 25,000 agents without, if they are continuing to do this without those credentials in place, a workforce of that size, starting to offer to a multinational strategic advice and not being able to understand from an audit trail, I mean, we haven't even begun to understand what the downstream impacts of that's going to be, particularly for a company that's about to go public, not go public.
Things don't go according to plan, and it's going to be very difficult to say, OK, that's not traced back to a 30-year seasoned partner with great expertise. That's been downstream to a bot that read something on the internet.
So, Act 4, a framework for responsible deployment, four simple questions, a simple, a very simple framework, and of course it's not simple, there's a lot of complexity, but four simple questions to get started on answering to bot or not to bot. First of all, does it have a verified identity? Really simple. If something goes wrong, if something goes right, do you know who that agent, can you trace that agent?
Secondly, back to Nat's talk, UBO, is it bound to an accountable human or entity or division or department or somebody that can ultimately say, hey, I'm responsible for that outcome? Is the mandate explicitly scoped and enforceable? And if it's not, is it really simple to override that really quickly? Do we know what that red button is? Getting those four questions into a framework then needs to be supported by a model of constantly monitoring that governance framework within the organisation. You can start with doing something really simple, low autonomy, medium autonomy, high autonomy.
Obviously, high autonomy is the place to start. And then looking at how you can create that framework of relaxing the posture as you start to understand what the downstream consequences might be. And credential hygiene as a policy.
Again, we've seen some fantastic sessions on that this week, but making sure that there's never anyone in your organisation sharing their credentials in any way with an agent. And then mandate governance and making that a living document. Understanding that a mandate might start here, it may move to there. And so what's the cadence to make sure that that mandate remains narrow or expands as your monitoring is, where you have confidence that the monitoring is working?
And again, a great session on this that I'd encourage you to go and look at from DMP Kewit Miko earlier this week, looking at a practical implementation, experimenting with the mandate to get it right. And most importantly, focused on how do you make sure that things are fair when you have a human and you have an agent trying to get an outcome. So I'd encourage you to also look at that session.
Final act, the question, our Shakespearean question, the question of our time. To bot, yes. If you can confidently go away, apply those four questions, and you are confident that you understand what the downstream consequences, the adjacent outcomes will be in your organisation, then sure, get going. If you can't, then you need to pause.
If it's unclear, if the mandate is undefined, if there's no one that you can point to in terms of accountability, then I would encourage someone in your organisation to be able to say, hey, great idea, go back, work on it a little bit, come back with this framework in place. Organisations that are able to get a framework like this in place and get it right, they'll gain speed and scale and all the fantastic things that are possible. Organisations that don't are going to face liability, regulatory action and reputational harm.
It's going to be too hard in some sectors with some agents, with life-endangering consequences to put the genie back in the bottle. So I'd like to finish with one thought to borrow from Martin's keynote at the beginning of the week. Everything we didn't solve is going to backfire with force now.
So everything that we've been working on together for the last decade, everything that we've been talking about here for 19 years at EIC, everything that we kind of left to work on with standards, with the evolution of technology, it is going to come back to bite us with the velocity of change that we see with AI. So to bot or not to bot? Maybe. I'd encourage you to go back to your place of work, your teams, start auditing against those four simple questions, embed them in everything you do and start that now. Thank you. Katharina Do.
Thanks so much for bringing us back to what it all needs and summarising it so nicely. I like the phrase that you said. Easy framework, but not that easy. So thanks a lot.
And yeah, then see you next time again. Thank you.
