WEBVTT 00:00:15.300 --> 00:00:20.100 Welcome to the KuppingerCole Analyst Chat. I'm your host. My name is Matthias Reinwarth. 00:00:20.300 --> 00:00:24.800 I'm analyst and advisor with KuppingerCole Analysts. But more importantly for today, 00:00:24.800 --> 00:00:30.000 I'm the host for this podcast and for this podcast episode. And for this, I have invited 00:00:30.000 --> 00:00:33.053 two colleagues of mine. I have invited Jonathan 00:00:33.053 --> 00:00:36.300 Care and I have invited Guillaume Teixeron. I hope 00:00:36.300 --> 00:00:41.300 that I do not butcher the name too much. But first of all, hi, Jonathan. Good to have you. 00:00:42.700 --> 00:00:44.900 Hello again, Matthias. Good to be on the podcast. 00:00:46.200 --> 00:00:51.200 Great to have you. And hi, Guillaume. First of all, surname was correct a bit? 00:00:51.200 --> 00:00:53.800 Well, it's Guillaume Teixeron, but don't worry, 00:00:53.800 --> 00:00:56.400 nobody out of France can pronounce it properly, 00:00:56.500 --> 00:00:59.929 so I'm used to it. Right. Okay. I just don't try 00:00:59.929 --> 00:01:03.500 it anymore. So Guillaume, you joined KuppingerCole 00:01:03.500 --> 00:01:08.900 quite a while ago already, but you're the new kid on the blog, as we tend to say here. So 00:01:09.100 --> 00:01:11.526 if you can give a brief introduction of who you 00:01:11.526 --> 00:01:13.900 are, where you come from and what you're doing 00:01:13.700 --> 00:01:16.536 here, that would be interesting also for the 00:01:16.536 --> 00:01:19.500 audience and for me. Yeah, sure. First, it's a 00:01:19.800 --> 00:01:23.185 pleasure to be on the podcast. So that's 20 00:01:23.185 --> 00:01:27.200 years I'm working in the identity and cybersecurity 00:01:27.200 --> 00:01:30.261 industry, but on the product side. So I used 00:01:30.261 --> 00:01:33.600 to build product in the authentication business, 00:01:33.700 --> 00:01:40.300 in the PKI business, in many other adjacent topic. And indeed, a few months ago, I joined 00:01:40.300 --> 00:01:43.416 KuppingerCole on the other side of the barrier 00:01:43.416 --> 00:01:46.600 to discuss with them though and do the analysis 00:01:46.600 --> 00:01:53.300 this time. So it's interesting to be on both sides to have the different perspective. And 00:01:53.300 --> 00:01:55.941 once again, I'm happy to be there. Great. And 00:01:55.941 --> 00:01:58.700 great to have you and great to have a new voice 00:01:59.400 --> 00:02:03.750 also contributing to this podcast, which is always 00:02:03.750 --> 00:02:08.100 nice to have a colorful palette of guests that can 00:02:08.100 --> 00:02:10.969 join me here. And actually, you also suggested 00:02:10.969 --> 00:02:13.900 the topic for today. And then I realized, okay, 00:02:13.900 --> 00:02:20.200 I need to have Jonathan as well. And so we want to talk about the use of AI in software 00:02:20.200 --> 00:02:22.879 development and what that means from a general 00:02:22.879 --> 00:02:25.500 perspective. And this is more of what I think 00:02:25.500 --> 00:02:29.032 is on your plate, Guillaume, and the security 00:02:29.032 --> 00:02:32.800 aspect of things. This is where I think Jonathan 00:02:32.800 --> 00:02:36.850 can contribute from his vast amount of research 00:02:36.850 --> 00:02:40.900 already in the area of AI security. So first of 00:02:40.900 --> 00:02:44.943 all, maybe to you, Guillaume, if we apply AI to 00:02:44.943 --> 00:02:48.900 software development, if you think back of the 00:02:48.900 --> 00:02:51.381 last 12 months before you joined and now you 00:02:51.381 --> 00:02:54.200 have joined KuppingerCole, what has changed in the 00:02:54.200 --> 00:02:58.558 market? And from an adoption standpoint, is AI 00:02:58.558 --> 00:03:03.200 support in software development still just a hype 00:03:03.200 --> 00:03:06.776 topic or is this something that has settled into 00:03:06.776 --> 00:03:10.500 something structural? Well, I think in the last 12 00:03:10.500 --> 00:03:20.700 and even slightly before that, now we see the AI becoming a structural topic in software 00:03:20.700 --> 00:03:24.314 development. What was indeed hype a few months 00:03:24.314 --> 00:03:28.400 ago become now the norm because of different things. 00:03:28.400 --> 00:03:32.308 First, the product has changed by nature. What 00:03:32.308 --> 00:03:36.300 was assistant to coding a few months ago is now 00:03:36.600 --> 00:03:39.800 agentic development-based and that's 00:03:39.800 --> 00:03:44.600 completely a different paradigm. Then we saw also that 00:03:45.000 --> 00:03:50.000 what was pushed by developers as early adopters 00:03:50.000 --> 00:03:55.000 are now older, if I may say, or driven by CTOs. 00:03:55.300 --> 00:03:59.130 So ideas are different. And also what is 00:03:59.130 --> 00:04:04.300 slightly under the hood is that AI is bundled with any 00:04:04.500 --> 00:04:08.031 development environment now. So any new 00:04:08.031 --> 00:04:13.100 developers come to his desk and as an AI assistant built 00:04:13.100 --> 00:04:19.500 inside his environment. So it's a new baseline for any developer to have it. 00:04:20.200 --> 00:04:29.100 Now, are they all using it? Yes. But are they all using it the same way? Probably not yet, 00:04:29.100 --> 00:04:33.256 but they are pushing to that direction. So your 00:04:33.256 --> 00:04:37.500 question is structural, but it doesn't mean that 00:04:37.500 --> 00:04:41.550 it is mature because adoption is structural, but 00:04:41.550 --> 00:04:45.600 the governance is still improvised in that area. 00:04:46.200 --> 00:04:52.600 Okay. Jonathan, I know you are a hidden developer. I think you are on both sides. You are 00:04:53.300 --> 00:04:56.900 using technology, but you're all looking at it from a security perspective. 00:04:56.900 --> 00:05:02.000 From the security standpoint, was there a specific moment where you said, okay, 00:05:02.000 --> 00:05:09.000 I should cover that also in my AI practice research? I do like the title of the hidden 00:05:09.000 --> 00:05:12.652 developer. Yeah, like typing when nobody's 00:05:12.652 --> 00:05:17.000 looking. So I think that it's not a single breach, 00:05:17.600 --> 00:05:25.400 but it's the pattern of MTP server incidents. And so the failure is the connective tissue 00:05:25.400 --> 00:05:31.100 around the model, not necessarily the model itself. And so that to me is an identity and 00:05:31.100 --> 00:05:34.790 supply chain problem wearing an AI costume, if 00:05:34.790 --> 00:05:38.400 I can put it like that. And you can fold that 00:05:38.400 --> 00:05:44.900 into general AppSec as well on three counts. Providence, you cannot attest to code whose 00:05:44.900 --> 00:05:52.000 origin you did not record. Scale, review capacity did not grow with generational capacity. 00:05:52.800 --> 00:05:56.085 And thirdly, non-human identity, that phrase we 00:05:56.085 --> 00:05:59.300 love to hate. Agents hold credentials and act, 00:05:59.900 --> 00:06:03.460 which AppSec never had to model. So we've got 00:06:03.460 --> 00:06:07.100 three tectonic shifts, if I can coin a phrase, 00:06:07.700 --> 00:06:15.600 that have dramatically changed what the AppSec paradigm is supposed to be managing. 00:06:16.200 --> 00:06:23.200 And the tell for me as an analyst was vendors answering security questions with benchmark 00:06:22.900 --> 00:06:27.447 scores. So an entire market was measuring the 00:06:27.447 --> 00:06:32.600 wrong thing. Therefore, that market needs dedicated 00:06:32.300 --> 00:06:40.200 coverage. Right. And is this also the same angle that you see, Guillaume, when it comes to 00:06:41.500 --> 00:06:46.100 the general perspective? Because I think the security part actually comes to my mind, 00:06:46.300 --> 00:06:49.070 but we more think of what is the outcome? What 00:06:49.070 --> 00:06:51.900 is the code? How do we understand the code that 00:06:51.900 --> 00:06:54.376 has been written? But is this something that you 00:06:54.376 --> 00:06:56.800 see in general or that you maybe have seen when 00:06:56.800 --> 00:06:59.550 you were on the different side of the barrier 00:06:59.550 --> 00:07:02.300 that you've mentioned? Is this something that 00:07:02.300 --> 00:07:08.300 has arrived also in the development offices of large vendors right now? 00:07:09.000 --> 00:07:15.800 Yeah. What I see is that now organizations produce code faster than they can verify it. 00:07:16.700 --> 00:07:20.181 So they are producing code at light speed or 00:07:20.181 --> 00:07:23.900 electronic speed, if I may say. And the problem 00:07:23.900 --> 00:07:26.755 is that this code has to be verified because 00:07:26.755 --> 00:07:30.000 you don't ship code, you don't deploy code without 00:07:30.000 --> 00:07:38.000 verification. And that's, I think, the core of the topic. We shift from securing code at 00:07:38.000 --> 00:07:45.100 the production to securing the code at the validation time, if I may say. And the core 00:07:45.100 --> 00:07:51.900 question is, do you trust AI to verify AI-produced code? Or do you put the verification, 00:07:51.900 --> 00:07:56.109 accountability, and liability still in human 00:07:56.109 --> 00:08:00.700 hands? And that's the core of the topic. Because 00:08:00.700 --> 00:08:08.300 if you produce 10,000 lines of code, but you don't verify them, then you are in trouble. 00:08:09.300 --> 00:08:16.000 Okay. So I think the default state of AI in enterprise software development is unmanaged 00:08:16.000 --> 00:08:23.400 velocity. So something we saw in our conference EIC, but among many other places, is that 00:08:23.700 --> 00:08:29.500 people can tell you that AI is in the pipeline, but not how much, not from which tools, 00:08:29.600 --> 00:08:35.300 or under whose accountability. So adoption is measured and by a degree is accurate, 00:08:35.500 --> 00:08:37.896 but provenance is not. Of course, I'm coming 00:08:37.896 --> 00:08:40.400 to this out of the recently published software 00:08:40.400 --> 00:08:43.683 supply and security leadership compass. But 00:08:43.683 --> 00:08:47.500 what I'd observe is that our security control set, 00:08:47.800 --> 00:08:50.122 and this is nowhere more true than in 00:08:50.122 --> 00:08:53.700 application security, is designed for human authored code 00:08:53.700 --> 00:09:00.700 moving at human speed. Neither assumption is now true. So if I was to sum up one sentence, 00:09:00.900 --> 00:09:09.000 we have industrialized code production. We have put in place AI code delivery, augmenting, 00:09:09.000 --> 00:09:12.198 augmenting production times significantly. What 00:09:12.198 --> 00:09:15.600 we have not done is industrialized code assurance. 00:09:16.400 --> 00:09:20.329 So we are still clinging on to the traditional 00:09:20.329 --> 00:09:24.600 ways of static dynamic code testing and assurance. 00:09:25.300 --> 00:09:28.800 And of course, we are falling behind and getting further behind. 00:09:29.500 --> 00:09:31.874 Right. But if I understand you correctly, that 00:09:31.874 --> 00:09:34.300 is not necessarily a new challenge. It's just a 00:09:34.300 --> 00:09:40.500 change in the dimension and the velocity and the amount of code that is created. 00:09:40.800 --> 00:09:43.304 I think we were not necessarily always that 00:09:43.304 --> 00:09:46.100 brilliant when it came to code checking anyways. 00:09:46.500 --> 00:09:49.788 And now we are in the situation that code is 00:09:49.788 --> 00:09:53.300 produced at a high level and a high, as I said, 00:09:53.400 --> 00:09:56.634 high volume. Is this just amplifying the risks, 00:09:56.634 --> 00:09:59.800 the problems, the challenges, or is this a new 00:10:00.100 --> 00:10:06.100 from what I understand? It is not, but I think Guillaume did contradict there. 00:10:06.800 --> 00:10:14.100 Well, I think there are two things. First is, as Jonathan said, production volume 00:10:15.000 --> 00:10:18.201 increased and verification remains at human 00:10:18.201 --> 00:10:21.700 level. So that's the problem. You have a funnel 00:10:21.700 --> 00:10:29.500 and you cannot verify everything you produce. That's something. Nowadays, the part that is 00:10:29.500 --> 00:10:33.186 beyond the volume, that is the certainty of 00:10:33.186 --> 00:10:37.300 what is produced and the fact that before it was 00:10:37.300 --> 00:10:40.199 produced by a human and the guy, you hire 00:10:40.199 --> 00:10:44.300 yourself, you train yourself. Now you are using a tool and 00:10:44.300 --> 00:10:52.800 basically you switch from verifying code you didn't write yourself to supervising an agent 00:10:52.800 --> 00:10:56.743 you didn't hire yourself. So you were given an 00:10:56.743 --> 00:11:00.600 agent. You don't know how it was trained. You 00:11:00.600 --> 00:11:03.700 don't know anything about that, but you know that 00:11:03.700 --> 00:11:06.800 it produced code. But with the certainty that you 00:11:06.800 --> 00:11:10.377 don't know. So yes, there is volume, but there is 00:11:10.377 --> 00:11:14.100 also uncertainty on the quality, on the groundways, 00:11:14.500 --> 00:11:20.300 on the underlying objective of the agent that may be written by someone else than the one 00:11:20.300 --> 00:11:25.700 developing the code. So there are volumes problem that's of use, but there are uncertainty 00:11:26.200 --> 00:11:32.500 on the code that is produced. Yeah, I think that's correct. I think the provenance piece 00:11:32.500 --> 00:11:35.995 I mentioned at the top of the podcast is new. 00:11:35.995 --> 00:11:39.800 You say that, well, maybe it's not a new problem. 00:11:39.800 --> 00:11:43.541 It's just the velocity. Well, that's also true 00:11:43.541 --> 00:11:47.200 in terms of automobile impact and automobiles 00:11:47.200 --> 00:11:49.718 have impacted from the very first day of their 00:11:49.718 --> 00:11:52.400 inception. However, the difference is now they do 00:11:52.400 --> 00:11:56.370 it at several hundreds of kilometers an hour 00:11:56.370 --> 00:12:00.700 rather than a 10 or 20. And so the damage is far 00:12:00.700 --> 00:12:03.928 greater. And I think, yes, so the 00:12:03.928 --> 00:12:09.700 multiplicative factor of velocity should not be unintended. 00:12:10.300 --> 00:12:14.400 When we're talking about software, of course, we're talking about tremendously complex 00:12:14.400 --> 00:12:17.557 multi-state machines, all the more difficult 00:12:17.557 --> 00:12:21.000 because they are intangible. And so we are going 00:12:21.000 --> 00:12:29.300 to see a significant challenge here. We need to make sure that we can get in front of the 00:12:29.500 --> 00:12:32.710 AI generated code, which as Guillaume says, we 00:12:32.710 --> 00:12:36.200 have no measure for that. Previously, if Guillaume 00:12:36.200 --> 00:12:39.038 submitted code or I submitted code into a code 00:12:39.038 --> 00:12:42.000 base, we could be said, well, they submitted the 00:12:42.200 --> 00:12:45.589 code base, they can be considered responsible. 00:12:45.589 --> 00:12:49.200 That's no longer the case. And in fact, it may be 00:12:49.200 --> 00:12:51.763 code in a language that the submitter is not 00:12:51.763 --> 00:12:54.500 even that familiar with. So we have some unique 00:12:55.000 --> 00:12:58.916 challenges there. What I think that means is 00:12:58.916 --> 00:13:03.100 that architectural assumptions and constraints, 00:13:03.300 --> 00:13:05.655 which previously would have probably been 00:13:05.655 --> 00:13:08.700 tested by a human developer, may not be tested in the 00:13:08.900 --> 00:13:12.203 same way with AI-driven software development, 00:13:12.203 --> 00:13:15.800 as a result of which we can surmise that software 00:13:15.800 --> 00:13:22.700 quality will be more variable. But you've mentioned your research on software supply 00:13:22.700 --> 00:13:29.200 chain security already. And I think this is also getting more visibility, more attention, 00:13:29.800 --> 00:13:32.960 since most of the newly upcoming regulations, 00:13:32.960 --> 00:13:36.400 they're not that new anyways. So if we talk about 00:13:37.000 --> 00:13:41.161 NIST, SP 800218, about DORA, about NIST 2, they 00:13:41.161 --> 00:13:45.500 all demand for supply chain security and software 00:13:45.500 --> 00:13:48.850 is obviously a huge part of the supply chain. 00:13:48.850 --> 00:13:52.200 From a market perspective, Guillaume, is this 00:13:52.200 --> 00:13:55.303 already driving change? Do you see that with the 00:13:55.303 --> 00:13:58.600 vendors that you're talking to with your day-to-day 00:13:58.600 --> 00:14:04.600 research? Or is this, I hate that term, but sometimes it's true, is this just compliance 00:14:04.600 --> 00:14:08.952 theater so far? So it's, there are two sides 00:14:08.952 --> 00:14:13.700 to that. So it's obvious that regulation and the 00:14:13.700 --> 00:14:19.800 CIA, for example, put, and then my study is coming in September, forced to have a better 00:14:19.800 --> 00:14:24.817 traceability. It was branded covered by the 00:14:24.817 --> 00:14:30.300 supply chain agency by Jonathan. So now people, 00:14:30.300 --> 00:14:33.759 software developers, software organization, 00:14:33.759 --> 00:14:37.700 produce evidence with software bills of materials 00:14:37.700 --> 00:14:44.300 and attestation. It becomes a baseline of the production of code. And the problem is that 00:14:44.300 --> 00:14:47.740 on the other side, it still remains a checkbox. 00:14:47.740 --> 00:14:51.400 And the RFP states that, yeah, you need to deliver 00:14:52.000 --> 00:14:54.773 software bills of material, you need to 00:14:54.773 --> 00:14:58.400 deliver attestation, but very few people does or do 00:14:58.400 --> 00:15:01.943 something from that in terms of traceability. 00:15:01.943 --> 00:15:05.800 So I think for the moment we are at the beginning 00:15:05.800 --> 00:15:08.960 of something where we start easily to produce 00:15:08.960 --> 00:15:12.400 traceability, but we don't do anything from that. 00:15:12.500 --> 00:15:16.600 That's something that we'll have to change. That's something the regulation will enforce 00:15:17.500 --> 00:15:20.870 that, yes, you need to take those artifacts, 00:15:20.870 --> 00:15:24.700 you need to add them somewhere, you need to track, 00:15:24.700 --> 00:15:27.986 you need to follow the vulnerability up to the 00:15:27.986 --> 00:15:31.200 root of the provider, and that's something AI 00:15:31.700 --> 00:15:37.300 should have. Also, it's not only a part of the problem, it's part of the solution, but 00:15:37.700 --> 00:15:41.335 yes, that once again, it puts the problem at 00:15:41.335 --> 00:15:45.300 scale and it may also help to solve the problem. 00:15:46.300 --> 00:15:51.700 Well, I don't know if that's necessarily true because I think we aren't seeing 00:15:52.300 --> 00:15:58.100 reliable code testing tools. We've obviously got the tools like Mythos and so forth, 00:15:58.300 --> 00:16:04.200 which are available to a few select organizations, but we have seen, in fact, the 00:16:04.200 --> 00:16:07.631 hugging face incident does tend to indicate 00:16:07.631 --> 00:16:11.700 that AI tools are actually reluctant to do security 00:16:12.500 --> 00:16:18.800 checking. So again, we need to closely examine guardrails there. I think what gets worse 00:16:19.600 --> 00:16:23.032 in general is volume and homogeneity. If a 00:16:23.032 --> 00:16:27.200 flawed pattern is introduced and the flawed pattern 00:16:27.300 --> 00:16:30.438 finds its way into the context that the AI 00:16:30.438 --> 00:16:34.100 model is using, then it will be replicated across 00:16:34.100 --> 00:16:36.539 hundreds of repositories very quickly indeed 00:16:36.539 --> 00:16:39.200 because, of course, the same model suggested it. 00:16:39.900 --> 00:16:44.500 And what we know is that where you have monoculture in code, you will therefore have 00:16:44.500 --> 00:16:48.074 monoculture invulnerability. What this means, I 00:16:48.074 --> 00:16:51.800 think, is that provenance breaks and AI-generated 00:16:51.800 --> 00:16:54.682 code has no upstream to name, so it becomes 00:16:54.682 --> 00:16:58.100 invisible to the tooling built to track third-party 00:16:58.100 --> 00:17:04.900 risk. Yeah, but if vulnerability can be introduced in thousands of repositories, 00:17:06.200 --> 00:17:09.117 a vulnerability can be fixed in thousands of 00:17:09.117 --> 00:17:12.300 repositories on the other side. So if you assume 00:17:12.300 --> 00:17:15.525 that you can introduce, you can assume that 00:17:15.525 --> 00:17:19.200 you can fix also at scale. I don't think that's a 00:17:19.200 --> 00:17:24.400 safe assumption. I think that a dependency of selection is the sharp edge. Models suggest 00:17:24.400 --> 00:17:29.900 packages by frequency in training data, not by current maintenance or security posture, 00:17:30.100 --> 00:17:32.774 and hallucinated package names are a live 00:17:32.774 --> 00:17:36.100 supply chain attack vector. I think tooling has not 00:17:36.100 --> 00:17:41.300 caught up because scanners look for known signatures in known places, and the failure 00:17:41.300 --> 00:17:44.256 mode here is plausible-looking code that is 00:17:44.256 --> 00:17:47.900 subtly wrong in context, which is exactly what static 00:17:47.900 --> 00:17:50.198 analysis is worst at. And as you said at the 00:17:50.198 --> 00:17:52.600 beginning, this is not a new problem. It's the 00:17:52.600 --> 00:17:55.950 reason why we've had buffer overflows in the DES 00:17:55.950 --> 00:17:59.300 algorithm for 60 years or so. Right, but is this 00:17:59.300 --> 00:18:02.200 really, or can this really be considered just a 00:18:02.200 --> 00:18:05.100 tooling gap? Is this something that we can heal 00:18:05.100 --> 00:18:08.900 over time with just tools getting better, or are 00:18:08.900 --> 00:18:12.700 we looking at a conceptual challenge that is not 00:18:12.700 --> 00:18:17.100 easy to be solved? Maybe both of you, Jonathan, 00:18:17.100 --> 00:18:21.500 to start. Well, I think tooling is where we can 00:18:21.500 --> 00:18:26.051 make tangible, observable, measurable changes. 00:18:26.051 --> 00:18:30.800 I think that what we're not necessarily catching 00:18:30.800 --> 00:18:37.800 is this new class of bugs. So as I mentioned, where we have plausible-looking code 00:18:37.900 --> 00:18:41.478 that is subtly wrong in the context in which 00:18:41.478 --> 00:18:45.300 it's being put, that is not something which our 00:18:46.300 --> 00:18:50.366 current methods instantiated in tooling are 00:18:50.366 --> 00:18:55.000 able to pick up. And so what I would hope is that 00:18:55.900 --> 00:19:03.100 AI is useful in software supply chain, and which ultimately is a quality feature, 00:19:04.100 --> 00:19:07.400 is to make sure that supply chain is 00:19:07.400 --> 00:19:12.900 contextually valid, that the plausible-looking code actually 00:19:12.900 --> 00:19:19.200 in the context to which it is put works correctly. And so yeah, that would be my hope. 00:19:19.200 --> 00:19:25.700 But I think that, yeah, well, I've written, yeah, model benchmarks and red teaming do not 00:19:25.700 --> 00:19:29.482 prove security efficacy for AI guardrail. And 00:19:29.482 --> 00:19:33.600 so yeah, I think we have challenges yet to solve. 00:19:34.200 --> 00:19:37.029 Okay. Now that we realize that the world is bad, 00:19:37.029 --> 00:19:39.800 we need to make sure that it's not that bad. We 00:19:39.800 --> 00:19:43.850 need to improve things. We are analysts, we are 00:19:43.850 --> 00:19:47.900 advisors. So what are our recommendations? What 00:19:47.900 --> 00:19:52.900 are things we can say to people that are responsible right now? So maybe Guillaume, 00:19:53.100 --> 00:19:56.650 to start with you, you are coming from the other 00:19:56.650 --> 00:20:00.200 part. And if now a CISO or an engineering leader 00:20:00.200 --> 00:20:04.900 comes to you and says, I understand there are challenges, I've heard your podcast, 00:20:05.200 --> 00:20:09.050 how can I structure my AI adoption in development 00:20:09.050 --> 00:20:12.900 responsibly? What would be your first question to 00:20:12.900 --> 00:20:17.000 them? What would be your first recommendation towards such a CISO or engineering lead? 00:20:17.600 --> 00:20:21.187 Well, the first one is, do you know where AI 00:20:21.187 --> 00:20:25.100 generated code is and where is it committed? And 00:20:25.100 --> 00:20:29.119 can you still track it after weeks, months? Do 00:20:29.119 --> 00:20:33.400 you have an inventory of your AI tools generating 00:20:33.400 --> 00:20:38.300 code? Do you know what, because whatever you decide as a policy around that, if you don't 00:20:38.300 --> 00:20:42.010 know what you have, well, there is no point to 00:20:42.010 --> 00:20:45.800 that. So that's the first thing. The second one 00:20:45.800 --> 00:20:49.510 is that the human is the ultimate owner of the 00:20:49.510 --> 00:20:53.300 code. So when a code is merged, there should be 00:20:53.300 --> 00:20:57.104 a goal by a human and that's obvious. And the 00:20:57.104 --> 00:21:01.500 more sensitive is the code, the more human should be 00:21:01.200 --> 00:21:05.960 there. And at the end, not all the code is 00:21:05.960 --> 00:21:11.400 the same. So there are obviously some parts that 00:21:11.400 --> 00:21:14.236 should not be touched by someone else than a 00:21:14.236 --> 00:21:17.200 senior engineer, where there are tons of lines 00:21:17.200 --> 00:21:21.455 of code that could be written by AI without any 00:21:21.455 --> 00:21:25.800 security concern. So these are the three topics, 00:21:26.000 --> 00:21:32.300 the inventory, the accountability and the proportionality that I would advise anyone to 00:21:32.300 --> 00:21:35.230 consider. Jonathan? And just to correct the words 00:21:35.230 --> 00:21:38.100 you put in my mouth, Matthias, I don't think the 00:21:38.300 --> 00:21:41.432 world is bad. I think that mistakes will be 00:21:41.432 --> 00:21:45.000 found out and will be exploited far more rapidly. 00:21:45.900 --> 00:21:48.076 And that doesn't necessarily imply the world 00:21:48.076 --> 00:21:50.400 is bad. It just means that there is very little 00:21:50.800 --> 00:21:57.100 room for error, whether it be human-generated error, AI-generated error or something else. 00:21:57.800 --> 00:22:05.900 Right. But if you think of AI-generated code governance, and you highlighted that there 00:22:06.000 --> 00:22:09.400 still should be ownership, control, maybe kind 00:22:09.400 --> 00:22:12.800 of risk-based approach towards what code does. 00:22:13.300 --> 00:22:15.574 And I think that's a good starting point. But 00:22:15.574 --> 00:22:18.000 we earlier discussed also this industrialization 00:22:18.000 --> 00:22:20.604 of code production, which might be 00:22:20.604 --> 00:22:25.200 contradicting that when it comes to producing code really at 00:22:25.200 --> 00:22:33.400 scale and at high volume. If we need to let the person, people out of that loop from your 00:22:33.400 --> 00:22:36.533 research, from your expectations as AI practice 00:22:36.533 --> 00:22:39.600 lead, what would a credible AI code governance 00:22:39.700 --> 00:22:45.300 control set actually contain from your perspective? We've talked about provenance, 00:22:45.800 --> 00:22:53.800 attestation. What else do you think should be implemented, even if possible, automated 00:22:53.800 --> 00:23:01.900 and added by humans? What is your perspective right now? I think it's a valid question. 00:23:01.900 --> 00:23:05.056 And I think that, again, the way to examine 00:23:05.056 --> 00:23:08.800 this is to examine the threat model. And so we need 00:23:08.800 --> 00:23:11.218 to expand our view beyond the code that gets 00:23:11.218 --> 00:23:13.800 written. And I think we look at the development 00:23:13.800 --> 00:23:16.539 pipeline, the industrialization, as you quite 00:23:16.539 --> 00:23:19.400 rightly called it. And that becomes the target. 00:23:20.300 --> 00:23:22.891 So if an agent has repository rights access, 00:23:22.891 --> 00:23:25.600 if it has credentials to access the continuous 00:23:25.600 --> 00:23:32.300 integration loop, and if you have a package registry token, that's a privileged insider. 00:23:32.900 --> 00:23:36.065 It never sleeps and it cannot be interviewed. 00:23:36.065 --> 00:23:39.300 Secondly, of course, MCP connections are trust 00:23:39.300 --> 00:23:42.700 delegations and frequently undocumented. Each one 00:23:42.700 --> 00:23:46.100 widens that blast radius that we talked about the 00:23:46.100 --> 00:23:49.871 automobile impact. And what we know is that most 00:23:49.871 --> 00:23:53.800 organizations have little or no inventory of their 00:23:54.300 --> 00:24:00.600 The missing control plane is something we discussed at EIC. Non-human identity. Our 00:24:00.600 --> 00:24:04.050 non-human identities need passports. They need 00:24:04.050 --> 00:24:07.500 a joiner, mover, lever process. We need to see 00:24:07.500 --> 00:24:15.800 weak credentials rotated and we need to see session accountability. If we can attack as a, 00:24:15.800 --> 00:24:18.355 again, sinking ourselves into the mind of an 00:24:18.355 --> 00:24:21.200 attacker, the agent's identity is the weak point. 00:24:21.200 --> 00:24:28.700 If we can subvert that, then the code review becomes actually less important. And again, 00:24:28.700 --> 00:24:30.613 while I don't think the world is bad and 00:24:30.613 --> 00:24:33.100 wrong, I think that's where we need to shine a light 00:24:33.100 --> 00:24:36.378 as analysts and advisors is on the SDLC 00:24:36.378 --> 00:24:41.000 control. The assumption that we make is that a human is 00:24:41.000 --> 00:24:45.657 the actor at each gate of the SDLC. If you 00:24:45.657 --> 00:24:51.200 remove that assumption, gates become advisory, not 00:24:52.100 --> 00:24:54.546 to the end of this episode. And I think there's 00:24:54.546 --> 00:24:57.200 so much more to discuss. And from what I understand 00:24:57.200 --> 00:25:01.357 right now is that will be a tough ride to strike 00:25:01.357 --> 00:25:05.600 the balance between adoption and security because 00:25:05.600 --> 00:25:08.516 everybody understands why it's useful and 00:25:08.516 --> 00:25:12.000 necessary to use these tools for code generation, 00:25:12.300 --> 00:25:14.524 for code production, for writing software. And 00:25:14.524 --> 00:25:16.700 on the other hand, we need the security part. 00:25:16.700 --> 00:25:19.681 And for me, as always, when you're talking 00:25:19.681 --> 00:25:23.300 about AI and agentic AI and about now this software 00:25:23.400 --> 00:25:26.090 generation, this is nothing that happens next 00:25:26.090 --> 00:25:28.900 year or next week or next month. It's happening 00:25:28.900 --> 00:25:31.831 right now. There are people already doing this. 00:25:31.831 --> 00:25:34.700 So the question to both of you, maybe starting 00:25:34.700 --> 00:25:37.250 with you, when it comes to telling a developer 00:25:37.250 --> 00:25:39.800 or a responsible person, the engineering lead, 00:25:40.400 --> 00:25:44.032 what should they look at tomorrow morning with 00:25:44.032 --> 00:25:47.900 their team to improve their security posture when 00:25:47.900 --> 00:25:52.900 it comes to creating that code? What would be something that can be immediately done, 00:25:53.100 --> 00:25:59.000 maybe does not even cost money? What would be such a first step? And no fears, Jonathan, 00:25:59.100 --> 00:26:01.882 I asked the same question. I would say that you 00:26:01.882 --> 00:26:04.900 should, well, once again, start with the inventory, 00:26:04.900 --> 00:26:08.239 check what is writing your code, what code is 00:26:08.239 --> 00:26:11.800 writing by who and what is the level of autonomy 00:26:12.000 --> 00:26:21.700 of the agent, if any. And because everything depends on that. So if agents are limited to 00:26:21.700 --> 00:26:24.503 insensitive code writing, I don't know, 00:26:24.503 --> 00:26:28.600 front-end and so on, you may have no problem. If they are 00:26:28.600 --> 00:26:31.630 touching sensitive code, you need to have policies 00:26:31.630 --> 00:26:34.600 and so on and so on and so on. So the first thing 00:26:34.600 --> 00:26:37.844 you should do is check where the code is 00:26:37.844 --> 00:26:41.900 coming from. Do you have shadow AI? Because that's 00:26:42.600 --> 00:26:47.235 something that is also at stake here. And 00:26:47.235 --> 00:26:53.000 start with knowing what AI or all AI is involved in 00:26:53.000 --> 00:26:55.229 the production of your code, because everything 00:26:55.229 --> 00:26:57.600 starts from there in terms of software development 00:26:57.600 --> 00:27:05.700 lifecycle. Jonathan? I think Guillaume is absolutely spot on. We must inventory our AI 00:27:05.700 --> 00:27:08.907 code paths and our agent identities. We 00:27:08.907 --> 00:27:13.100 cannot govern what we have not counted. And we know 00:27:13.100 --> 00:27:16.888 that most teams will be surprised by that 00:27:16.888 --> 00:27:21.600 count. Then I think we go to our vendors and we ask 00:27:21.600 --> 00:27:26.247 one question. Show me security efficacy evidence, 00:27:26.247 --> 00:27:30.800 not benchmark scores. And I think the quality of 00:27:30.800 --> 00:27:35.900 that answer tells you an awful lot about the product. Right. Two things to cover or three 00:27:35.900 --> 00:27:38.400 things to cover before we close down. First, I 00:27:38.400 --> 00:27:40.900 need to highlight that there will be an event, 00:27:41.500 --> 00:27:45.567 especially looking at AI and NHI. So that will be 00:27:45.567 --> 00:27:49.800 the AI-NHI Impact Day in October. So that will take 00:27:49.800 --> 00:27:56.900 in Munich. So if you want to talk to us as analysts, talk to other practitioners or 00:27:57.200 --> 00:27:59.900 talk to a few vendors, then you are in the right 00:27:59.900 --> 00:28:02.600 place for one day in Munich around these topics. 00:28:02.700 --> 00:28:05.519 And I think code will be an important part of 00:28:05.519 --> 00:28:08.400 the equation. First of all, I did that. So the 00:28:08.400 --> 00:28:11.077 commercial break is done right now. Second 00:28:11.077 --> 00:28:14.200 part is, we've talked about that a bit. Jonathan, 00:28:14.200 --> 00:28:17.315 you have already produced quite a substantial 00:28:17.315 --> 00:28:20.500 amount of research in that area. What would be 00:28:20.500 --> 00:28:23.419 your three recommendations that people can read 00:28:23.419 --> 00:28:26.400 when they come to our website? Either blog posts 00:28:26.400 --> 00:28:28.646 in front of the, I hate that word, paywall. 00:28:28.646 --> 00:28:31.100 Just have a subscription and you have access to 00:28:31.100 --> 00:28:33.778 everything and the stuff that is only available 00:28:33.778 --> 00:28:36.400 to our subscribers. So what would be the three 00:28:36.400 --> 00:28:39.750 things that you would highlight when it comes to 00:28:39.750 --> 00:28:43.100 AI security and that aspect? Well, first of all, 00:28:43.100 --> 00:28:46.181 I can only agree that a Cup and a Coal membership 00:28:46.181 --> 00:28:49.200 is well worth it. And obviously with the holiday 00:28:49.200 --> 00:28:53.000 season coming up, it's an ideal gift for family and friends and indeed colleagues. 00:28:54.500 --> 00:28:57.936 I think that the research that I've written, but 00:28:57.936 --> 00:29:01.300 also many of my colleagues have also written as 00:29:01.300 --> 00:29:03.822 well, shows that the end result fight will be 00:29:03.822 --> 00:29:06.400 over evidence standards. So if you look at our 00:29:06.400 --> 00:29:08.359 blog posts, and indeed, if you look at some 00:29:08.359 --> 00:29:10.500 of our leadership compasses and advisory notes, 00:29:11.100 --> 00:29:14.250 you can see that the people on the side of the 00:29:14.250 --> 00:29:17.400 adoption curve will accept vendor attestation. 00:29:17.700 --> 00:29:20.200 I think we will see security teams wanting this 00:29:20.200 --> 00:29:22.700 independent efficacy testing that we're talking 00:29:22.700 --> 00:29:25.352 about. And I can confidently expect that 00:29:25.352 --> 00:29:28.800 nobody will have agreed who will pay for it. I think 00:29:28.800 --> 00:29:31.968 there's a second flashpoint coming as well, 00:29:31.968 --> 00:29:35.800 which is agent autonomy in production pipelines. And 00:29:36.400 --> 00:29:39.487 we've skirted on that a little bit today. But the 00:29:39.487 --> 00:29:42.700 disagreement will move from can AI write code? Yes, 00:29:42.700 --> 00:29:46.200 it can. To can AI merge and deploy it without 00:29:46.200 --> 00:29:49.700 a human gate? And that, I think, we are still 00:29:50.100 --> 00:29:52.904 waiting for evidence from the field. Where I 00:29:52.904 --> 00:29:55.900 expect convergence, and I think this is, again, 00:29:55.900 --> 00:29:59.066 something that is in the software supply chain, 00:29:59.066 --> 00:30:02.300 the attack surface management leadership compass 00:30:02.300 --> 00:30:05.722 is the name. But two, regulation is going to 00:30:05.722 --> 00:30:09.300 settle this prominent argument for both camps. 00:30:10.200 --> 00:30:13.122 CRA, NIS2, DORA, make this a market access 00:30:13.122 --> 00:30:16.600 question rather than a preference. In other words, 00:30:16.700 --> 00:30:25.400 if you're going to do this, you have to be in the alignment with the legal requirements 00:30:25.400 --> 00:30:30.400 and regulations that are being enacted. Okay, final questions to both of you. Of course, 00:30:30.400 --> 00:30:34.800 analysts do have this crystal ball where they can say what will happen in one year's time. 00:30:35.500 --> 00:30:40.500 So that will be some of these questions that you will be asked. So looking 12 months out, 00:30:40.600 --> 00:30:44.279 what do you expect to be still not yet well 00:30:44.279 --> 00:30:48.300 resolved between the adoption camp and security 00:30:48.300 --> 00:30:51.666 camp? Will they drift far apart or will they work 00:30:51.666 --> 00:30:55.100 together? And what will be the main issue standing 00:30:55.100 --> 00:30:58.011 between them? Maybe starting with Jonathan. 00:30:58.011 --> 00:31:01.600 Well, I feel like I'm out of step because I believe I 00:31:01.600 --> 00:31:04.960 just answered this question now. So again, 00:31:04.960 --> 00:31:09.600 evidence standards, agent autonomy, and I think that in 12 00:31:09.300 --> 00:31:16.500 months' time that regulation will... So this will still not be solved in 12 months' time? 00:31:17.200 --> 00:31:22.100 I don't think so. Okay, get that. So what you've mentioned still will be the case. 00:31:22.100 --> 00:31:26.200 Guillaume, from your perspective, what will be the greatest 00:31:26.700 --> 00:31:30.000 barrier between the two, the adoption and the security camp? 00:31:31.700 --> 00:31:40.300 So I agree with Jonathan. I think that the open fight in 12 months is whether AI-generated 00:31:40.300 --> 00:31:43.988 insurance counts as insurance. Can I verify with 00:31:43.988 --> 00:31:47.600 AI something produced with AI? That will be the 00:31:47.600 --> 00:31:50.253 discussion. And anyway, I think that the 00:31:50.253 --> 00:31:54.100 regulator will answer it before the market agrees on that. 00:31:54.800 --> 00:31:58.400 So regulation will say what you should produce and 00:31:58.400 --> 00:32:02.000 what you should track. And that's what I do expect 00:32:04.200 --> 00:32:06.575 because for the rest, it will be always a fight 00:32:06.575 --> 00:32:09.000 between, yeah, you can trust this, you can trust 00:32:09.000 --> 00:32:15.100 that, or you cannot because of this or because of that. That will be a never-ending fight. 00:32:15.100 --> 00:32:18.189 Right. Maybe if I'm allowed to also share my 00:32:18.189 --> 00:32:21.700 view on that, I think the provenance topic will be 00:32:21.700 --> 00:32:27.500 getting much more important because when we create code based on human-written code, 00:32:27.700 --> 00:32:34.000 based on available libraries, and based on augmented code provided by AI agents that 00:32:34.000 --> 00:32:39.200 write that, and that all goes into our code base, and we use the code base for training 00:32:39.200 --> 00:32:43.358 the machine to create more code, I think it 00:32:43.358 --> 00:32:48.000 will thin down or it will make the human-created 00:32:48.000 --> 00:32:50.424 code less and less important. And the question 00:32:50.424 --> 00:32:52.900 is, what does that mean for provenance when I'm 00:32:52.900 --> 00:32:58.900 training on material that has already been partially or completely created by AI agents? 00:32:59.300 --> 00:33:04.900 What does that mean for provenance, for quality assurance? I think that is a philosophical 00:33:04.900 --> 00:33:09.300 question a bit, but I think it's a highly technical question as well. So I think that 00:33:09.300 --> 00:33:14.100 will be something to discuss in 12 minutes time if we don't do it right now already. 00:33:14.400 --> 00:33:19.500 So that would be my point, unless you completely disagree. Looking at you, 00:33:19.600 --> 00:33:24.100 you don't. I think that we're close down with that as usual. 00:33:25.100 --> 00:33:28.400 I think there's something interesting, I think, that will be hitting us in 12 months, 00:33:28.400 --> 00:33:39.800 which we're seeing nascent risings. Right now, we are starting to see copycat code. 00:33:40.800 --> 00:33:46.132 You have technical standards, you have design 00:33:46.132 --> 00:33:51.700 specifications, you even have a website outline 00:33:51.700 --> 00:33:56.900 or an app UI design. And we are seeing the rise in tools, which just say, well, 00:33:56.900 --> 00:34:01.300 take what is already out there and produce me something that's very, very similar. 00:34:03.100 --> 00:34:05.248 There's a lot of questions there about 00:34:05.248 --> 00:34:08.300 intellectual property rights and so on. But then also, 00:34:08.500 --> 00:34:14.900 what happens when those tools are being used as part of an enterprise mainstream, 00:34:16.100 --> 00:34:22.200 perhaps unknowingly by the stakeholders? And then, yeah, what happens when that sort of 00:34:22.200 --> 00:34:23.300 land in court? 00:34:23.800 --> 00:34:27.640 Right. It's a bit like 15 years or 20 years ago, 00:34:27.640 --> 00:34:31.400 long before AI, when a composer wrote a melody, 00:34:31.600 --> 00:34:33.476 which he completely thought was great and 00:34:33.476 --> 00:34:35.900 didn't know that it was already out there and created 00:34:35.900 --> 00:34:42.500 by somebody else. That would be the unplanned copyright infringement. And that would be 00:34:42.500 --> 00:34:47.400 the same thing on steroids in the end. So that will be interesting to see. 00:34:48.000 --> 00:34:53.600 The good thing is the profession of lawyers will not go away. So this is something that 00:34:53.600 --> 00:34:58.100 which we will need all the time and even more, maybe, unless they are AI supported. 00:34:58.400 --> 00:35:00.791 If there are any questions, if there are any 00:35:00.791 --> 00:35:03.400 comments, if you think this was a great episode, 00:35:03.600 --> 00:35:06.570 let's have Guillaume and Jonathan talk more about 00:35:06.570 --> 00:35:09.600 these topics. Leave your comments below that video 00:35:09.600 --> 00:35:11.984 on YouTube. Send a mail to myself, to 00:35:11.984 --> 00:35:15.400 Jonathan, to Guillaume. We are easy to find. Send the 00:35:15.600 --> 00:35:18.923 talk to us individually. We are happy to do so. 00:35:18.923 --> 00:35:22.600 And apart from that, thank you very much, Guillaume, 00:35:22.900 --> 00:35:27.900 for the first time here. Jonathan, for supporting me again in such an episode. That was an 00:35:28.400 --> 00:35:30.875 interesting conversation and it gave me some 00:35:30.875 --> 00:35:33.800 food for thought and I think and I hope that it also 00:35:33.800 --> 00:35:37.100 did for the audience. So happy to continue the 00:35:37.100 --> 00:35:40.400 conversation in Munich and everywhere where we 00:35:40.400 --> 00:35:46.100 can be found virtually and in real life. Thanks, Guillaume. Thanks, Jonathan. 00:35:46.600 --> 00:35:46.900 Thank you. 00:35:47.700 --> 00:35:48.700 Thank you. Bye-bye.