Road to EIC: Automated Identity Access Decisions: Building Trustworthy AI Systems

Good morning, good afternoon, welcome wherever you may be. My name is Jonathan Care, I'm a lead analyst for KuppingerCole and today we are going to be talking about building trustworthy AI systems, part of our seminar webinar series on the Road to EIC. With me I have Massimo Flore and Martin Sandren, gentlemen.

Welcome, thank you for joining me. Massimo, would you like to quickly introduce yourself, tell the audience all about you.

Yeah, so good morning everyone and what can I say, I'm a strategic analyst, I'm working on digital trust and AI governance and my background is on disinformation and hybrid threats but in recent years my work has focused on how AI systems interact with users and interact with users over time and what and how that affects trust, decision making and the system behaviour. Excellent, Martin, could you possibly introduce yourself in 30 seconds.

So my name is Martin Sandren, I'm the product lead for Inter, for IEM, for Inter IKEA, which is the mother company in the IKEA group and we do around meatballs, flat packages and emotional support animals for monkeys and other needy people. Okay, excellent and so and to the audience thank you for joining us and this webinar will be recorded and will be available from the KuppingerCole website following this, following it being prepared obviously for presentation.

So I'd like to kick off with a question, we're talking about building trustworthy AI systems and what does trustworthy actually mean when we apply it to an AI system and who gets to decide? Martin, what are your thoughts? Well depending a little bit of where we operate, obviously the European Union has some opinions about this and most likely we'll see regulations following in other places and it is of course a big as a lot of the GDPR and a lot of the regulations from European Union it's about the individual's rights.

So one part of trustworthy is of course that ensures that it has fair treatment of of customers and other end users. There's also of course the cyber security aspect to this to ensure that it's resilient and it's able to, it's hard to bring into trouble. Okay and then of course it needs to run as well from an operational perspective like any IT system there has to be a operational residence. Interesting and Massimo what are your thoughts?

Well the thing is that this is an interesting question because we need to define what is trustworthy because in AI identity system usually refer to security, to reliability, or to compliance of the AI system and these are essential dimension but they tell us that the system is functioning as expected at any point in time but what becomes more complex is when AI it's what it is about like trust. So when trust is not only about whether the system works correctly now but whether it continues to behave in line with what this intended function is and how it evolves.

So in this sense trustworthiness is not only a property of the system at deployments it becomes something that needs to be sustained over time. I think that's interesting and I think that leads us on to as you say the EU has defined some baselines for trustworthiness and I think we'll start seeing obviously whether that survives international competition or not. I think that ultimately who gets to decide who gets to decide well and one would hope that the users of the AI system have a certainly a stake in whether it is trustworthy or not.

Obviously the owners, the operators of the system and ultimately as you say governance also playing an interest in this as well. So the next question as I say these things are already widespread and only likely to become more so. So how do we distinguish between an AI system that is safe and genuinely trustworthy? How do we know when something is safe and indeed when it's genuinely trustworthy? Are they the same thing or do we mean different things by safety and trustworthiness?

Well the problem here is that safe and trustworthy this is the thing because what we usually do was just to define if the check that we usually do is a point in time check. So basically it's something that we check if the system is the credential when we access the system are okay. If everything is in place when is the starting point but then we don't monitor anymore what is happening when the user is start to interact with the system and what happens in time.

Because what happens in time is that the system evolves because it takes data, there are changes, there are updates in the system itself. So the problem here is that the system becomes safe when we can just we can assure that the system keeps working for what is intended to do. So we design the system to be trustworthy but only usage over time tells us whether it's safe or not. Yeah that's the point. Fascinating. Martin do you have a thought on that? It is a very interesting difference.

So if you go back to my university days for 30 years or 25 years ago when you had data systems at that point you had memory that contained data and you had memory that contained instructions and if you were really advanced you on the fly rewrote your instructions part of your memory with new things and then you could have very advanced software that usually kind of blew itself up at some point. With normal programming with a given set of input you will expect that another set of output and you should always get the same.

But with the advantage with the AI coming into the picture you get to a non-deterministic results. So like Massimo highlighted the system might start to move in its outcomes might start picking up behaviors. So what we need to do is to put in the right kind of guardrails on the outside to see that making sure that the outcomes are still reasonable and keeping some form of an intent. What was the intent of this original request which also gets possible to provide some form of a guidance on what data the AI should be able to access in order to execute its mission.

We're talking about guidance and so forth. I also think about baseline and so again a question that I am looking at in my research and is what baseline properties must an AI system demonstrate and how can we yeah so what's the what's the proof that an organization needs before an organization to deploy in production. As we talk about the the IT system operations and as you say the secured software development life cycle process. But again specifically looking at the AI system what baseline properties does an AI system need to demonstrate. So I think that these are the really good questions.

I think that the kind of the risk control frameworks that you need are being developed as we speak. Obviously being an AI specialist I am mostly focused on the setup for the ensuring that the AI has the right level of energy and the controls on that side. But you do need to add also controls on what it's actually the output from the AI and making sure that it's within certain parameters. I think the question of identity is absolutely fundamental and I think that it's important to first of all be able to determine the identity in AI and possibly also that leads to its provenance.

So again who is you know who is operating it and what and depending on the level of detail you go into but you can even say well what data was it trained on. So you know for example I saw an AI system that was being put into place in banks and this is in America and not here in Europe and I saw the AI vendor had trained it on the popular so it's facial recognition. So they trained the facial samples they used were from a university.

The university had a decided orientation towards young obviously and Caucasian with the result being that this AI system for facial recognition was going into banks and was actually very poor at recognizing the people of color. I suggested to the vendor that they really ought to warn the bank of the public relations nightmare that awaited them if they adopted this and I don't know if you've got any thoughts towards that as well Massimo. Well the thing is that what we have to realize that this is not a static system. Those are dynamic systems.

They change over time and the main problem here that is one of the things that you highlighted is like behavioral drift. So the thing is that is how the system makes decision and frame its option and learn from the interaction that he has with user or with other machines. So the thing is that what we are just what we are seeing here is that there is not a sudden failure. It's a slow drift and the slow drift is much more difficult to detect because it's gradual and it's slowly.

So we might see like a system that just function correctly but then the outputs just slightly turn into a direction that was not the originally intended and that's that's I think that is the main problem. Good. I'm listening you lead on something interesting and sorry Martin we're going to make. Yeah I think there's a very quite interesting part there when the example of the US that in the US so I'm originally from Sweden one lived in Germany UK and US and now Netherlands. But in the US you have a tradition of protected categories and also legal framework around protected categories.

That means that there are certain things that you legally may not discriminate against and that is why if you for example they if you're in an area where they may do through searches on you for weapons or other things that you shouldn't have in the subway for example. They will ask you to self-identify with your box on the your self-identified race the box on the birth certificate and the idea there is that at least if they should not drift too far away what you would expect.

So if the population in a given area should have a certain composition then the people who get checked should also have a similar similar composition to ensure that there's no discrimination of a protected category. And I think certain of these types of controls could be very useful because it will mean that if it turns out that your AI is drifting you will detect it. You won't detect it immediately but at some point you at least see that you're outside of the acceptable range of outcomes.

I like that and I think it is yes it does become important and it actually leads on to I think the next point is we've been talking about responsibility and accountability in training which I think is important and turning to the output. So if we have a system which as Massimo says we built it to be trustworthy and unfortunately for whatever reason it produces an untrustworthy output. Who bears that liability? Is it the vendor the deployer or is it the model itself?

When I mentioned this to a CEO of a very well-known software company he said well who can you grab by the scruff of the neck and drag them into court? You can't really drag a model into court. So I don't know what your thoughts are but I'm very interested. Massimo have you got any thoughts on that? Well you know this is a question that I got asked like a lot and very often because it's the thing as you said like you cannot just bring a computer into court because the computer is not a juridical persona.

So it cannot just be tried and there is another thing because when I every time I spoke with like some engineers that just they are developing these AI systems what they keep repeating is that we are not sure we don't know precisely why the machine took a specific decision or why the machine gave a specific answer because we gave we just designed we just designed how it's to work. We gave the database but we don't control the output. So the problem here is that is the vendor it depends I think on specific cases.

So the thing is that what are the algorithms of the instruction that were given to the machine? So if the machine just the instruction that were given to the machine points to a specific drifting then probably the person who is liable is the vendor. But if there is also a problem that these machines are designed usually to please the user. So probably sometimes the machine is not doing any harm it's only reinforcing what the user wants to hear. So it's like the user is extracting meaning from something that the machine has said but maybe they don't mean.

So it's a gray area and it will be like kind of difficult to assess. I must admit when you said that engineers aren't sure I felt a shudder pass through our audience especially ones who are designing anything safety critical. So and you're right as you say these this is something that I mean you touch on two things. First of all as I used to be an expert witness when I lived in the UK. I'm now enjoying the sunshine in Portugal but when I lived in the UK I worked as an expert witness and of course the job of the expert witness is to speak for the evidence within their area of expertise.

So if you say you cannot ask questions or well you can perhaps now ask questions but you cannot ask questions normally of a computer system it's an inanimate object. So as you say that's where the expert comes in. The other side of it is talking with traditional IT managers and even in systems such as fraud systems and so on and you say to them look you are now moving into a territory where identity is no longer a black and white thing. It's on the balance of probabilities. We think it's Jonathan Kerr.

It's no longer so we no longer have an absolute no one have an absolute assurance of identity. We don't have an absolute assurance of risk or anything because as you say these systems are non-deterministic and and it does make me wonder sort of if we have these non-deterministic systems and we are building AI systems so obviously they don't yet exist but what governance structures do we need to establish at the outset for these AI systems to again safely interact with traditional software architectures. Martin your thoughts?

So this is a great question I think the first part is that there are certain areas where it's more the non-deterministic nature of these systems is more acceptable. So I used to work within a federal drug administration enforced company in the pharma and biotech world and if you look at GXP of course there you have an extreme determinism like you are supposed to write down all the steps in your configurations and you're supposed to execute them in the right order exactly as they're written.

You may not make any changes and that is to minimize the risk that there is contamination in the in the drugs which of course in the end could result in serious health risks or even death for the patients. On the other hand so it's in a system like that or in a life-saving system you probably not put an AI that makes decisions initially at least. On the other hand if you look at something like ITDR or identity security where you're trying to determine if you never to with 100% certainty know if any given person at any given time it's been compromised or not.

You can look at indications such as if you come from a different country to where you normally are or where your peer groups are or where we operate that would be an indication of compromise.

If you use a device that you haven't seen before and it is not used by it's non-IKEA device especially we can see that that's also an indication of compromise and then you put all this lots of other factors as well let me look at and you leverage an AI for that to determine okay should we put in some methods to increase our confidence in your identity for example we could ask to do a self-service password reset which requires MFA which is a quite hard thing for an attacker to have.

There you're totally fine as long as it doesn't cost your users too much effort and it has likely to cost the attacker a lot of effort that they will lose their compromised identity basically there it's a good usage so I think a lot of this is also about how you use it used in the right places where the cost of non-determinism is acceptable.

I want to come back to something that actually both of you touched on and this is the black box problem which we've talked about and Massimo called it out and said you know builders can't fully explain why a model produced to give an answer and again what is the you know what what advice can we offer organizations who are looking at these systems you know how do they how do they handle you know that an answer has been given which could be a risk decision in a transaction it could be an identity decision it could be you know medical diagnosis it could be all sorts of different things and as we know the the use cases keep expanding every day and how do organizations handle that the builders the engineers can't explain the why there's no they cannot say why the model boosts the answer in the early days we had you know the 10 years ago when I was researching this you know we had the head of AI at Google and you know other sort of senior luminaries saying well of course we need to have explainability it feels that ship has sailed somewhat now so how do we handle it?

Well the thing is is that's okay I mean if we are talking like about like large language models which are the most the most diffused in in today world and the most adopted even in the workflow of of of companies the problem is that we have to shift the mindset from like a one-time verification to a continuous observation so the thing is that at this time at this point in time we don't have machines and this is all those who are are developing this that are completely 100% reliable we will get there one day it can be like in one year in two years in five years we will get there to a machine that can give us like a complete and accurate answer and it's completely trustworthy so for now on we need to have this continuous observation so the thing is that what what companies or organization can do is just keep asking if the systems are behaving differently over time compared to what they were originally intended because if we start only asking that question that will change our relationship with the machine because we know what to look for we know what to monitor.

Okay so even if drift appears pleasing it is nevertheless still drift and it should be noted right. Martin I don't know if you've got any thoughts or should we get onto the good stuff around adversarial resilience? I think let's go into the adversarial interesting area.

Right yeah so for those of you this is the hacky bit um for the audience here so um props injection, data poisoning and again many years ago um I talked with Googlers who were yeah looking at this in you know in now what's the infancy of this art or science and I said well what can you do and they said you can very broadly you can map the oracle so you can understand what particular set of inputs generate a particular output and the second thing of course you can poison the oracle so you can poison the thing and then traditional example back then was sees a picture of a cat and it says it's a dog.

Now of course we have much more sophisticated attacks it's something I'm actually just writing a white paper on now so we have we know prompt injection we know we have data poisoning. How do these undermine trust and and what practical defenses do you folks see being enacted today? I'm genuinely interested because this is you know close to my heart and indeed close to what I do in my day job. Martin what are the thoughts you have?

I think it's a it's a great question um the usual way of trying to solve new questions to seek is there have we solved a similar problem before and what did we do then and of course the prompt injection is quite close to SQL injection for example so validation of input and again having some form of an idea of of intent so in most use cases that you have in a company you're not expecting that the oracle should be able to answer anything and it might be able to tell you how to you know make a good stew but that is probably not what you expected to do so if you can have some form of a pre-processor and what kind of questions that the user asks and I think initially now at least you also have to have an ability to kind of call a human once you get outside of the the realms of the pre-processor that it where the question starts being outside of the intent the the stated intent of the AI.

Right Massimo any thoughts? Well the thing is that we can just think about like three possible scenarios the first one is the one that you just mentioned is like or poisoning of the data sets or the machine how it works or or even without like any malicious intent it can be like an upgrade in the model so just imagine a model that's been first trained on data that will just well curated and then this data is integrated with with public data sets so it's a forum socials and that's those that kind of data can be also poisoned.

So the things that you will see that the machine will start moving from like a certain behavior and will incorporate like bias or toxic narratives and in this case the provider does not realize what is going on because the internal auditing are nowadays are testing performance and non-coherence on moral or semantic.

Then there is a second scenario which is a drift in fine-tuning so is the system learns from the users and it's like what is called like the reinforce reinforcement learning from user feedback and in this way the user just the machine just learn what how the the user feels like and and just start reacting in this way and the third one is that the drift is made by like some company objectives so the the company just changes the the the templates on how the machine has to work.

So for the first thing is that what we can do is that there can be like an ex ante declaration so that there is an intent credential.

So the system declares what you have to do and then this becomes like the baseline that we can check and monitor what is going on and then it can be like an ongoing this is what I'm saying just the constant monitoring just to see if the machine is start just drifting or changing the behavior and then just a sort of validation that's to see and also that integrate also with the user feedback that can be like some I mean I'm very quick this is a very long things to to say.

No no this is this is really interesting and I think while I was listening to you both the a concept occurred to me that actually drift that you were talking about is just as much an exploitable condition as anything else.

I mean one thing we do know certainly that is true I think of most AI systems anything that's certainly based on any type of neural structure is it's very hard to get things to unlearn and and any we've all well I'm sure and people in the audience as well we've all experienced that you know trying to get something to unlearn once it's learned something is problematic and you also run into things like catastrophic forgetting and so on and so you only increase the hallucination but all of these things are you know exploitable conditions which leads me on to this idea of penetration testing.

We are now getting red teaming firms and offering penetration testing services. We are all as security practitioners and obviously everybody on listening on this webinar we've all engaged or even performed penetration testings and and the question then comes when we have this AI should we be testing them to the same rigor as other critical infrastructure? Should we be saying yeah can we get this thing to learn wrong? Can we get it to provide biased output?

And also all of the other things that we can do to an additional IT software stack what we're adding is actually another you know almost another layer on the on the stack it's but it's certainly it's another attack surface that should I think should be penetration but I'm you know interested in your thoughts Martin. Yes I think that the main difference is in the nature of the AI that normally when you do a pen test you know you do it either always have the challenge about exactly when in the software lifecycle you do a pen test. Do you do it after UAT?

Okay if you find problems then you have an issue with your timelines or do you do it earlier well then you might not have already a software then you of course you don't pen test on the latest version that you're supposed to go live on. With an AI that that problem becomes a bit even more tricky that perhaps you should do a you know every month or perhaps every week depending on the importance system do an AI pen test to see if the model has drifted.

And then of course any AI system also uses you know standard IT infrastructure at the end so you have to protect that one and figure out how to ensure that that is a sufficiently hardened against attacks. Massimo your thoughts please.

The thing is that it depends on so also on what is the nature of the machine that we are using it's something that is only transactional so it's a machine that is only giving information or is a companion so it's something that is just providing a continuous relationship with the user so and these are just made like two different different kind of risks because on one side if it's only transactional and you're just giving something there is one time or based on a specific target we have a specific kind of risk if it's a companion that it's more it's more it's a deeper risk because it starts to create a relationship with the user and that's a completely different risk so we have first to understand which one we are talking to because there are two different solutions for for each each one of the problems.

And I think I mean I think we've we've covered off without starting to delve into particular techniques but I think we which is you know heading into the weeds in itself and I think we've covered sort of adversarial systems and I think we've got a general consensus across the panel which is great thank you for that. I want to go back to well actually something which again as a former auditor I was constantly looking for ways to measure and make qualitative sorry quantitative assessments and trustworthiness in AI. Can it be measured?

Can we have a trust score or is it inherently a qualitative aspect of an AI system? So can we say yeah this system scores eight on a trust score or can we just say yes it is qualitative in that it does this and so on?

Well the problem here is that if we are talking about trust and because this is the main the main issue is that if this system has like both qualities but it's easy to just measure the quantitative part so it's performance and accuracy and risk scores and this is something that organizations already use because they are easy to measure but if we are just getting into the qualitative dimension that is kind of harder to capture because it relates on how the system behaves and how consistent the machine is in its intended roles and how it adapts and interacts over time and this is kind of more difficult to just to monitor because it will be also individually because we have to remember that this machine if they just start to interact heavily with the user they start to personalize the experience.

Martin your thoughts please. It's a great really really great question but I believe it's a little bit like asking the question is the system secure? You can't really answer that in a a kind of fully quantitative way you can say that okay we applied a certain risk control framework on it and we it either met or did not meet the controls and if they didn't meet the controls there's either compensatory controls or there is a return on risk that has been accepted by the business and I think that that kind of thinking is what we would need to look at the AI's trustworthiness.

What are your controls that protects your trustworthiness of your AI? What are the guardrails? I think that's true and I think again we come back to we establish as you say a set of guardrails and then the measure is does this system does it align with those guardrails or does it veer away from them which to me I think we're getting I think we're getting more into qualitative territory there. But you mentioned guardrails and we we're all now familiar with what these things do because we're all now familiar the concepts of bias, hallucination and drift.

Massim you said something earlier which I thought was tremendously insightful you said we build these AI systems to please us. That's an interesting thing actually that's a audience let's let that one sink in for a minute we build these AI systems to please us even if we are pleased by them being contrarian we still build these systems to satisfy a need and we therefore have you know the capabilities for we therefore have you know the capabilities for bias. An AI system that I built may well reflect the unconscious bias that I hold. It may well hallucinate in order to please me.

We know I don't think it's unequivocal that they will drift over time. So how do we audit these capabilities and how do we say well this has drifted beyond a certain safety margin? Well the thing is that it's the first thing is it's not that easy and it will be not only time consuming but also really expensive because we are not talking about like a machine just talking and behaving with everyone the same way. If those machines are just personalized machines they will just respond differently and behave personalizing. They personalize their answer to all the users.

So this is one big problem that we have in front. What we can do is that we can just put as Martin said like guardrails that just warn the user on what is going on. We can have like some checks in how the machine is working just to try to see if a drift is happening or if the machine is just working. So I don't know when there is like a new model that's been integrated we can just have a re-consent prompt. So just to have something easy that the user can say like look the machine has been upgraded and do you want to have the upgraded machine or you want to continue with the old one?

That can be like a guardrail because we don't know. We can have like a trust ledger so it's something that just is transparent and just shows when there are the modification and all the drift that has been seen and the action that's been taken toward that drift. And then we can give like a score on what is going on. I know that I'm sure that one of my colleagues over in the advisory team of Capacico are probably overjoyed to hear you say that auditing will be an expensive and long drawn-out process. One of their favorite words.

But Martin before I move on I'm actually very interested in your thoughts because clearly you work in an industry that needs to have some safety measures and needs to be able to audit those. What are your thoughts? I think on the safety measures is to the most straightforward part is to look at outcomes and see are the outcomes falling within a reasonable category. Because it's very hard to look at every single decision and see if it's a reasonable decision. But as a collective you can see if the decisions are within the expected behavior range.

And I think that's a little bit the canner in the coal mine in many cases. Yeah and then obviously we do have aviation, pharmaceuticals and nuclear power. And they all operate complex systems and they have learnt to establish what trust means. And I think it's interesting to me I think what should the AI industry learn from aviation, from pharmaceuticals and nuclear power? What should you know what can we yeah what can we take from from the practices built up in these industries over time?

I think I used to have a background in chemical engineering which is an interesting field because some of these kind of non-deterministic parts are actually kind of introduced in chemical engineering in the late 19th century. So we have never really been able to fully simulate or mathematically describe something like a distillation column to 100 percent. We started to do it quite recently with some of the more numerical methods. But traditionally you kind of created a model and then you looked at the outcomes and then you figured out you know how do I impact the outcomes in direction I want.

And I think A-B testing is an interesting part you have two independent models and if the results starts vary widely then why is this and why do we think that one of them or perhaps even both are wrong. And also determine certain protective measures what is the unacceptable outcomes and can we detect when we get too many of those. So if you look at the ITDR space for example you probably should have some form of a throttling when you have too many users who start being being asked to do a passive reset. If that's your one of your major steps you take on users that you feel are highly risky.

Okay well in the I mean we're moving towards the home straight now and I would like to say both of you you've been you know excellent and you know very well informed panelists on this webinar as it's one of our road to EIC series. We've got our flagship conference coming up in Berlin in May so this is why we try and do these things. And so part of any identity architecture and the implementation program we have to think about the organizational and cultural change that goes together with technological change.

And I think this is true of AI and something I floated I think just before the webinar started and how do you build a culture where the AI questions AI sorry where the employees question AI beg your pardon and rather than just blindly deferring to it. It seems to me that right now we you know put something into the keyboard and you know when the answer come back we gratefully accept it and we carry it forward. It must be true because chat GPT told me to or you know other other AI models may exist but because you know Claude said so, chat GPT said so, Grok said so.

We need to I feel we need to question it but how would you go about building that culture of actually we should question this we shouldn't just blindly accept it. Martin let's give you first. I think on this one is that so a lot of things you should look at your AIs more as very enthusiastic and happy interns and see what they come out of them and then see is it real reasonable what you're telling me now. And also of course a part of that is to curate what kind of data you put into the the AI so that it has good data to work with.

I don't think it's really that different from you know running a team of junior engineers or similar that you you ask them to work on a task and then you coach them how to improve the quality but you also keep a little bit of an eye on what what they're doing and what their outcomes are because they may not have the full picture and they may not understand all the things that kind of how the the whole works. And I think that's an important part of how we successfully work with AIs as when it's kind of a extension of your your own work.

So again they're not necessarily the wise elder what they actually are is the eager junior. Yes or either I do like the little yellow you know minions as an AI. So speaking of speaking of minions and junior engineers I recall when I built my ISP that two of my engineers wandered in shyly into my office hand in hand and said you know Shona is going to have a baby I was like well I didn't even know you two are going out never mind staying in but there we are. So yes organizational I don't think we've got to worry about that with AI just yet.

There is another side to this and we've all seen the funny things on Reddit when someone gets a customer service AI to sing a song about potatoes that's quite funny. We've all seen you know other similar things where the AI has made some laughable and hilarious mistakes. However we also are now seeing public failures which reflect on organizational reputation and organizational reputation of course is an expression of trust. How do you recover trust when your AI whatever type of ambassadorial role it has fails publicly?

Well if I may reply I mean the thing I like I like your example Martin because I usually when I refer to AI I usually refer to them as a child who knows everything but it's a bit dumb so you have to accompany him in your journey and you have to use properly.

So in these cases the thing is that we should just try to keep the frictions into how we use AI because the AI just give us like a direct very convincing answer and it's very easy to say as you say Jonathan it's like okay that's the answer is convincing and so I will go with that without just checking but the thing is that the AI is not something that is replacing us the AI is an assistant to us. If we use it correctly it will enhance the way our workflow but if we don't use it correctly it will lead to this kind of mistake.

So the thing is just to have to just question what the AI is saying so just to try to identify what are the weak points or even ask the AI to identify itself the weak points and the AI will say oh I just made a mistake these are the weak points of the text that you just gave to me so just learn that it's a continuous process and that probably the first answer is the wrong one maybe when you get to the 20th draft and edits you get like something that is proper. Interesting.

I'm going to actually I'm going to cut now to the audience we've got quite a few questions so gentlemen prepare yourselves because we've got a very eager and alive audience here so I'd like to turn to those questions now if that's okay.

I have the first question from Prathaban sorry I hope I pronounced that correctly and Prathaban asks is access analytics truly quantifiable with the AI explosion or are we still far from achieving scoped permissions for AI systems which seems to encompass a number of things but I think where we're going to there is yeah really sort of NHI and resolution analytics and so on what are your thoughts on that and Martin I didn't think I let you speak enough last time so I'll give that question to you.

Well I think there's a couple of parts to that question so the first part is can we define the intent of a question and that is not always that easy and of course the intent will then drive the type of just-in-time access that you want to provide to your AI. So I think we can do you know it's all about getting to a level where we feel comfortable with the level of risk that we're taking and amount of return on risk. We will never be able to provide a perfect system that is able to fully do an access control and ensures that you have exactly the right amount of privilege.

At least privilege is not a it's not a binary state it's much more of a gray scale. So I think that the answer from my part is that no we will never be perfect but there's certainly a lot of things we can do to be better. Massimo I'm very interested in your thoughts. Well the thing is that now what we have is a framework of actions and governance in this realm and to complement what Martin says that we don't have I think a sort of broadly adopted governance on this topic and we have we still don't have I think a sort of a single operational layer for this intent.

So it's something that we have to build but it's still fragmented and so we have to just collect more data probably to just try to get what is the the right way to coordinate all this framework of this fragmented elements that we have. Intent is a very interesting and I see there's a couple of questions asking about intent which hopefully we'll come to. I've got a question here from Warwick. Where do you see the right balance between automation and human oversight in identity access decision making and what or which decisions should never never be fully delegated to AI?

So now we're touching on human in the loop we're touching on yeah what governance and so on what's okay and not okay. Gentlemen any thoughts you want to go first on that? Well the thing here is that I mean we have to divide this but it's it's not an easy an easy answer because I mean we can have probably automation for I don't know like highly frequency low impact decisions.

So something that where you need speed and consistency and probably it's the the the impact is is low but human oversight remain essential when the difficulties reverse when when something affects like the user rights or it's ambiguous or it can just provoke something that is substantial and in that side human oversight cannot be just and in that side human oversight cannot be just removed. Interesting Martin your thoughts please. So I think there's a couple of different aspects to this. So firstly of course you have the the risk of loss of life.

So I used to work for a big big oil and gas company and it's very interesting you can still see it people have been working for one of these companies if they walk down the stairs they will automatically go out and grab the stair handle if it's more than three steps because you're trained that that safety is the most important thing in my life is the most important thing. So I think a lot of this is to find the right places where the fact that that the AI will most likely be a an enthusiastic intern for quite a while where can we afford to make mistakes.

So if you look at specifically in the ITDR space for example you may say that okay you're given the right to to force the user to do an extra pass or reset we can take that as indirect effect. We may not say that you may delete the accounts even if you think it's an attack. So one part is to look at the impact from the decisions and also the ability to to roll back the decisions in an effective way. Yeah it's interesting and I see we're coming to the last five minutes and so actually something has come up.

Darren has asked on best practices or thoughts on defining the scope of pen tests for AI systems. What considering AI system would be problematic in nature not deterministic in future. So again I think this is restating this and it's certainly something that in our Kaplan and Cole research library our members can access white paper. I think we do have one on this particular subject but again we're coming up to you know the last five minutes or so.

Massimo, Martin any thoughts on that? Martin please go first on this one. It's a good question so I'm not a specialist on pen tests. So I would actually have to admit that this is something I would have to read up on. What is the best way to do pen test against AIs? I've been overseeing pen test conventional ones and there the one usual thing to try to do is to put on the attacker hat to try to figure out how you get from a basic penetration up to really privileged like global admin or equivalent.

Also perhaps you should do pen test in the purple team way so that you also teach the the internal team you know how do you kind of gather get to the compromise of the system. Those will be my primary things to think about but Massimo hopefully you have some more deeper insights into this one. The thing is that I mean usually like those pen tests those penetration tests are about like finding vulnerabilities in code just correct me if I'm wrong and but these systems are kind of different because they are probabilistic systems.

So this is those those tests probably should be extended and to explore how those systems behave under different condition and inputs. So it's just to see what I like the behavioral tests and long-term interaction patterns and just not to test once but just to test how those systems behave in a long in a long frame in a long time frame that could be like interesting. So extending the okay the frame. Right well we have come to just about the end of the webinar and it only remains for me to thank both of you Massimo Flore and Martin Sandrun for your participation.

If you have further questions that have not been answered please do send them to Kapp and Nicole and we'll either send them on to the panelists or if we have an answer ourselves we'll certainly give you that. As I said this has been one of our Road to EIC webinars in preparation for our conference in Berlin in May. It's been all about automated identity across decisions building trust with the AI systems. For Kapp and Nicole I'm Jonathan Kerr and I look forward to seeing you all in Berlin in May. Thank you very much. Thank you.