Can You Trust AI to Verify AI Code?

Welcome to the KuppingerCole Analyst Chat. I'm your host. My name is Matthias Reinwarth. I'm analyst and advisor with KuppingerCole Analysts. But more importantly for today, I'm the host for this podcast and for this podcast episode. And for this, I have invited two colleagues of mine. I have invited Jonathan Care and I have invited Guillaume Teixeron. I hope that I do not butcher the name too much. But first of all, hi, Jonathan. Good to have you. Hello again, Matthias. Good to be on the podcast. Great to have you.

And hi, Guillaume. First of all, surname was correct a bit?

Well, it's Guillaume Teixeron, but don't worry, nobody out of France can pronounce it properly, so I'm used to it. Right. Okay. I just don't try it anymore. So Guillaume, you joined KuppingerCole quite a while ago already, but you're the new kid on the blog, as we tend to say here. So if you can give a brief introduction of who you are, where you come from and what you're doing here, that would be interesting also for the audience and for me.

Yeah, sure. First, it's a pleasure to be on the podcast. So that's 20 years I'm working in the identity and cybersecurity industry, but on the product side. So I used to build product in the authentication business, in the PKI business, in many other adjacent topic. And indeed, a few months ago, I joined KuppingerCole on the other side of the barrier to discuss with them though and do the analysis this time. So it's interesting to be on both sides to have the different perspective. And once again, I'm happy to be there. Great.

And great to have you and great to have a new voice also contributing to this podcast, which is always nice to have a colorful palette of guests that can join me here. And actually, you also suggested the topic for today. And then I realized, okay, I need to have Jonathan as well. And so we want to talk about the use of AI in software development and what that means from a general perspective. And this is more of what I think is on your plate, Guillaume, and the security aspect of things.

This is where I think Jonathan can contribute from his vast amount of research already in the area of AI security. So first of all, maybe to you, Guillaume, if we apply AI to software development, if you think back of the last 12 months before you joined and now you have joined KuppingerCole, what has changed in the market? And from an adoption standpoint, is AI support in software development still just a hype topic or is this something that has settled into something structural?

Well, I think in the last 12 and even slightly before that, now we see the AI becoming a structural topic in software development. What was indeed hype a few months ago become now the norm because of different things.

First, the product has changed by nature. What was assistant to coding a few months ago is now agentic development-based and that's completely a different paradigm. Then we saw also that what was pushed by developers as early adopters are now older, if I may say, or driven by CTOs. So ideas are different. And also what is slightly under the hood is that AI is bundled with any development environment now. So any new developers come to his desk and as an AI assistant built inside his environment. So it's a new baseline for any developer to have it.

Now, are they all using it? Yes. But are they all using it the same way? Probably not yet, but they are pushing to that direction. So your question is structural, but it doesn't mean that it is mature because adoption is structural, but the governance is still improvised in that area. Okay.

Jonathan, I know you are a hidden developer. I think you are on both sides. You are using technology, but you're all looking at it from a security perspective. From the security standpoint, was there a specific moment where you said, okay, I should cover that also in my AI practice research? I do like the title of the hidden developer.

Yeah, like typing when nobody's looking. So I think that it's not a single breach, but it's the pattern of MTP server incidents. And so the failure is the connective tissue around the model, not necessarily the model itself. And so that to me is an identity and supply chain problem wearing an AI costume, if I can put it like that. And you can fold that into general AppSec as well on three counts. Providence, you cannot attest to code whose origin you did not record.

Scale, review capacity did not grow with generational capacity. And thirdly, non-human identity, that phrase we love to hate. Agents hold credentials and act, which AppSec never had to model. So we've got three tectonic shifts, if I can coin a phrase, that have dramatically changed what the AppSec paradigm is supposed to be managing. And the tell for me as an analyst was vendors answering security questions with benchmark scores. So an entire market was measuring the wrong thing.

Therefore, that market needs dedicated coverage. Right. And is this also the same angle that you see, Guillaume, when it comes to the general perspective? Because I think the security part actually comes to my mind, but we more think of what is the outcome? What is the code? How do we understand the code that has been written? But is this something that you see in general or that you maybe have seen when you were on the different side of the barrier that you've mentioned? Is this something that has arrived also in the development offices of large vendors right now? Yeah.

What I see is that now organizations produce code faster than they can verify it. So they are producing code at light speed or electronic speed, if I may say. And the problem is that this code has to be verified because you don't ship code, you don't deploy code without verification. And that's, I think, the core of the topic. We shift from securing code at the production to securing the code at the validation time, if I may say. And the core question is, do you trust AI to verify AI-produced code? Or do you put the verification, accountability, and liability still in human hands?

And that's the core of the topic. Because if you produce 10,000 lines of code, but you don't verify them, then you are in trouble. Okay. So I think the default state of AI in enterprise software development is unmanaged velocity. So something we saw in our conference EIC, but among many other places, is that people can tell you that AI is in the pipeline, but not how much, not from which tools, or under whose accountability. So adoption is measured and by a degree is accurate, but provenance is not.

Of course, I'm coming to this out of the recently published software supply and security leadership compass. But what I'd observe is that our security control set, and this is nowhere more true than in application security, is designed for human authored code moving at human speed. Neither assumption is now true. So if I was to sum up one sentence, we have industrialized code production. We have put in place AI code delivery, augmenting, augmenting production times significantly. What we have not done is industrialized code assurance.

So we are still clinging on to the traditional ways of static dynamic code testing and assurance. And of course, we are falling behind and getting further behind. Right. But if I understand you correctly, that is not necessarily a new challenge. It's just a change in the dimension and the velocity and the amount of code that is created. I think we were not necessarily always that brilliant when it came to code checking anyways. And now we are in the situation that code is produced at a high level and a high, as I said, high volume.

Is this just amplifying the risks, the problems, the challenges, or is this a new from what I understand? It is not, but I think Guillaume did contradict there.

Well, I think there are two things. First is, as Jonathan said, production volume increased and verification remains at human level. So that's the problem. You have a funnel and you cannot verify everything you produce. That's something.

Nowadays, the part that is beyond the volume, that is the certainty of what is produced and the fact that before it was produced by a human and the guy, you hire yourself, you train yourself. Now you are using a tool and basically you switch from verifying code you didn't write yourself to supervising an agent you didn't hire yourself. So you were given an agent. You don't know how it was trained. You don't know anything about that, but you know that it produced code. But with the certainty that you don't know.

So yes, there is volume, but there is also uncertainty on the quality, on the groundways, on the underlying objective of the agent that may be written by someone else than the one developing the code. So there are volumes problem that's of use, but there are uncertainty on the code that is produced.

Yeah, I think that's correct. I think the provenance piece I mentioned at the top of the podcast is new. You say that, well, maybe it's not a new problem. It's just the velocity.

Well, that's also true in terms of automobile impact and automobiles have impacted from the very first day of their inception. However, the difference is now they do it at several hundreds of kilometers an hour rather than a 10 or 20. And so the damage is far greater. And I think, yes, so the multiplicative factor of velocity should not be unintended. When we're talking about software, of course, we're talking about tremendously complex multi-state machines, all the more difficult because they are intangible. And so we are going to see a significant challenge here.

We need to make sure that we can get in front of the AI generated code, which as Guillaume says, we have no measure for that. Previously, if Guillaume submitted code or I submitted code into a code base, we could be said, well, they submitted the code base, they can be considered responsible. That's no longer the case. And in fact, it may be code in a language that the submitter is not even that familiar with. So we have some unique challenges there.

What I think that means is that architectural assumptions and constraints, which previously would have probably been tested by a human developer, may not be tested in the same way with AI-driven software development, as a result of which we can surmise that software quality will be more variable. But you've mentioned your research on software supply chain security already. And I think this is also getting more visibility, more attention, since most of the newly upcoming regulations, they're not that new anyways.

So if we talk about NIST, SP 800218, about DORA, about NIST 2, they all demand for supply chain security and software is obviously a huge part of the supply chain. From a market perspective, Guillaume, is this already driving change? Do you see that with the vendors that you're talking to with your day-to-day research? Or is this, I hate that term, but sometimes it's true, is this just compliance theater so far?

So it's, there are two sides to that. So it's obvious that regulation and the CIA, for example, put, and then my study is coming in September, forced to have a better traceability. It was branded covered by the supply chain agency by Jonathan. So now people, software developers, software organization, produce evidence with software bills of materials and attestation. It becomes a baseline of the production of code. And the problem is that on the other side, it still remains a checkbox.

And the RFP states that, yeah, you need to deliver software bills of material, you need to deliver attestation, but very few people does or do something from that in terms of traceability. So I think for the moment we are at the beginning of something where we start easily to produce traceability, but we don't do anything from that. That's something that we'll have to change.

That's something the regulation will enforce that, yes, you need to take those artifacts, you need to add them somewhere, you need to track, you need to follow the vulnerability up to the root of the provider, and that's something AI should have. Also, it's not only a part of the problem, it's part of the solution, but yes, that once again, it puts the problem at scale and it may also help to solve the problem.

Well, I don't know if that's necessarily true because I think we aren't seeing reliable code testing tools. We've obviously got the tools like Mythos and so forth, which are available to a few select organizations, but we have seen, in fact, the hugging face incident does tend to indicate that AI tools are actually reluctant to do security checking.

So again, we need to closely examine guardrails there. I think what gets worse in general is volume and homogeneity. If a flawed pattern is introduced and the flawed pattern finds its way into the context that the AI model is using, then it will be replicated across hundreds of repositories very quickly indeed because, of course, the same model suggested it. And what we know is that where you have monoculture in code, you will therefore have monoculture invulnerability.

What this means, I think, is that provenance breaks and AI-generated code has no upstream to name, so it becomes invisible to the tooling built to track third-party risk. Yeah, but if vulnerability can be introduced in thousands of repositories, a vulnerability can be fixed in thousands of repositories on the other side. So if you assume that you can introduce, you can assume that you can fix also at scale. I don't think that's a safe assumption. I think that a dependency of selection is the sharp edge.

Models suggest packages by frequency in training data, not by current maintenance or security posture, and hallucinated package names are a live supply chain attack vector. I think tooling has not caught up because scanners look for known signatures in known places, and the failure mode here is plausible-looking code that is subtly wrong in context, which is exactly what static analysis is worst at. And as you said at the beginning, this is not a new problem. It's the reason why we've had buffer overflows in the DES algorithm for 60 years or so.

Right, but is this really, or can this really be considered just a tooling gap? Is this something that we can heal over time with just tools getting better, or are we looking at a conceptual challenge that is not easy to be solved? Maybe both of you, Jonathan, to start.

Well, I think tooling is where we can make tangible, observable, measurable changes. I think that what we're not necessarily catching is this new class of bugs. So as I mentioned, where we have plausible-looking code that is subtly wrong in the context in which it's being put, that is not something which our current methods instantiated in tooling are able to pick up.

And so what I would hope is that AI is useful in software supply chain, and which ultimately is a quality feature, is to make sure that supply chain is contextually valid, that the plausible-looking code actually in the context to which it is put works correctly. And so yeah, that would be my hope. But I think that, yeah, well, I've written, yeah, model benchmarks and red teaming do not prove security efficacy for AI guardrail. And so yeah, I think we have challenges yet to solve. Okay. Now that we realize that the world is bad, we need to make sure that it's not that bad.

We need to improve things. We are analysts, we are advisors. So what are our recommendations? What are things we can say to people that are responsible right now? So maybe Guillaume, to start with you, you are coming from the other part. And if now a CISO or an engineering leader comes to you and says, I understand there are challenges, I've heard your podcast, how can I structure my AI adoption in development responsibly? What would be your first question to them? What would be your first recommendation towards such a CISO or engineering lead?

Well, the first one is, do you know where AI generated code is and where is it committed? And can you still track it after weeks, months? Do you have an inventory of your AI tools generating code? Do you know what, because whatever you decide as a policy around that, if you don't know what you have, well, there is no point to that. So that's the first thing. The second one is that the human is the ultimate owner of the code. So when a code is merged, there should be a goal by a human and that's obvious. And the more sensitive is the code, the more human should be there.

And at the end, not all the code is the same. So there are obviously some parts that should not be touched by someone else than a senior engineer, where there are tons of lines of code that could be written by AI without any security concern. So these are the three topics, the inventory, the accountability and the proportionality that I would advise anyone to consider. Jonathan? And just to correct the words you put in my mouth, Matthias, I don't think the world is bad. I think that mistakes will be found out and will be exploited far more rapidly.

And that doesn't necessarily imply the world is bad. It just means that there is very little room for error, whether it be human-generated error, AI-generated error or something else. Right. But if you think of AI-generated code governance, and you highlighted that there still should be ownership, control, maybe kind of risk-based approach towards what code does. And I think that's a good starting point. But we earlier discussed also this industrialization of code production, which might be contradicting that when it comes to producing code really at scale and at high volume.

If we need to let the person, people out of that loop from your research, from your expectations as AI practice lead, what would a credible AI code governance control set actually contain from your perspective? We've talked about provenance, attestation. What else do you think should be implemented, even if possible, automated and added by humans? What is your perspective right now? I think it's a valid question. And I think that, again, the way to examine this is to examine the threat model. And so we need to expand our view beyond the code that gets written.

And I think we look at the development pipeline, the industrialization, as you quite rightly called it. And that becomes the target. So if an agent has repository rights access, if it has credentials to access the continuous integration loop, and if you have a package registry token, that's a privileged insider. It never sleeps and it cannot be interviewed.

Secondly, of course, MCP connections are trust delegations and frequently undocumented. Each one widens that blast radius that we talked about the automobile impact. And what we know is that most organizations have little or no inventory of their The missing control plane is something we discussed at EIC. Non-human identity. Our non-human identities need passports. They need a joiner, mover, lever process. We need to see weak credentials rotated and we need to see session accountability.

If we can attack as a, again, sinking ourselves into the mind of an attacker, the agent's identity is the weak point. If we can subvert that, then the code review becomes actually less important.

And again, while I don't think the world is bad and wrong, I think that's where we need to shine a light as analysts and advisors is on the SDLC control. The assumption that we make is that a human is the actor at each gate of the SDLC. If you remove that assumption, gates become advisory, not to the end of this episode. And I think there's so much more to discuss.

And from what I understand right now is that will be a tough ride to strike the balance between adoption and security because everybody understands why it's useful and necessary to use these tools for code generation, for code production, for writing software. And on the other hand, we need the security part. And for me, as always, when you're talking about AI and agentic AI and about now this software generation, this is nothing that happens next year or next week or next month. It's happening right now. There are people already doing this.

So the question to both of you, maybe starting with you, when it comes to telling a developer or a responsible person, the engineering lead, what should they look at tomorrow morning with their team to improve their security posture when it comes to creating that code? What would be something that can be immediately done, maybe does not even cost money? What would be such a first step? And no fears, Jonathan, I asked the same question.

I would say that you should, well, once again, start with the inventory, check what is writing your code, what code is writing by who and what is the level of autonomy of the agent, if any. And because everything depends on that. So if agents are limited to insensitive code writing, I don't know, front-end and so on, you may have no problem. If they are touching sensitive code, you need to have policies and so on and so on and so on. So the first thing you should do is check where the code is coming from. Do you have shadow AI? Because that's something that is also at stake here.

And start with knowing what AI or all AI is involved in the production of your code, because everything starts from there in terms of software development lifecycle. Jonathan? I think Guillaume is absolutely spot on. We must inventory our AI code paths and our agent identities. We cannot govern what we have not counted. And we know that most teams will be surprised by that count. Then I think we go to our vendors and we ask one question. Show me security efficacy evidence, not benchmark scores. And I think the quality of that answer tells you an awful lot about the product. Right.

Two things to cover or three things to cover before we close down. First, I need to highlight that there will be an event, especially looking at AI and NHI. So that will be the AI-NHI Impact Day in October. So that will take in Munich. So if you want to talk to us as analysts, talk to other practitioners or talk to a few vendors, then you are in the right place for one day in Munich around these topics. And I think code will be an important part of the equation. First of all, I did that. So the commercial break is done right now. Second part is, we've talked about that a bit.

Jonathan, you have already produced quite a substantial amount of research in that area. What would be your three recommendations that people can read when they come to our website? Either blog posts in front of the, I hate that word, paywall. Just have a subscription and you have access to everything and the stuff that is only available to our subscribers. So what would be the three things that you would highlight when it comes to AI security and that aspect?

Well, first of all, I can only agree that a Cup and a Coal membership is well worth it. And obviously with the holiday season coming up, it's an ideal gift for family and friends and indeed colleagues. I think that the research that I've written, but also many of my colleagues have also written as well, shows that the end result fight will be over evidence standards. So if you look at our blog posts, and indeed, if you look at some of our leadership compasses and advisory notes, you can see that the people on the side of the adoption curve will accept vendor attestation.

I think we will see security teams wanting this independent efficacy testing that we're talking about. And I can confidently expect that nobody will have agreed who will pay for it. I think there's a second flashpoint coming as well, which is agent autonomy in production pipelines. And we've skirted on that a little bit today. But the disagreement will move from can AI write code?

Yes, it can. To can AI merge and deploy it without a human gate?

And that, I think, we are still waiting for evidence from the field. Where I expect convergence, and I think this is, again, something that is in the software supply chain, the attack surface management leadership compass is the name.

But two, regulation is going to settle this prominent argument for both camps. CRA, NIS2, DORA, make this a market access question rather than a preference. In other words, if you're going to do this, you have to be in the alignment with the legal requirements and regulations that are being enacted.

Okay, final questions to both of you. Of course, analysts do have this crystal ball where they can say what will happen in one year's time. So that will be some of these questions that you will be asked. So looking 12 months out, what do you expect to be still not yet well resolved between the adoption camp and security camp? Will they drift far apart or will they work together? And what will be the main issue standing between them? Maybe starting with Jonathan.

Well, I feel like I'm out of step because I believe I just answered this question now. So again, evidence standards, agent autonomy, and I think that in 12 months' time that regulation will... So this will still not be solved in 12 months' time? I don't think so.

Okay, get that. So what you've mentioned still will be the case.

Guillaume, from your perspective, what will be the greatest barrier between the two, the adoption and the security camp? So I agree with Jonathan. I think that the open fight in 12 months is whether AI-generated insurance counts as insurance. Can I verify with AI something produced with AI? That will be the discussion. And anyway, I think that the regulator will answer it before the market agrees on that. So regulation will say what you should produce and what you should track.

And that's what I do expect because for the rest, it will be always a fight between, yeah, you can trust this, you can trust that, or you cannot because of this or because of that. That will be a never-ending fight. Right.

Maybe if I'm allowed to also share my view on that, I think the provenance topic will be getting much more important because when we create code based on human-written code, based on available libraries, and based on augmented code provided by AI agents that write that, and that all goes into our code base, and we use the code base for training the machine to create more code, I think it will thin down or it will make the human-created code less and less important.

And the question is, what does that mean for provenance when I'm training on material that has already been partially or completely created by AI agents? What does that mean for provenance, for quality assurance? I think that is a philosophical question a bit, but I think it's a highly technical question as well. So I think that will be something to discuss in 12 minutes time if we don't do it right now already. So that would be my point, unless you completely disagree. Looking at you, you don't. I think that we're close down with that as usual.

I think there's something interesting, I think, that will be hitting us in 12 months, which we're seeing nascent risings. Right now, we are starting to see copycat code. You have technical standards, you have design specifications, you even have a website outline or an app UI design. And we are seeing the rise in tools, which just say, well, take what is already out there and produce me something that's very, very similar. There's a lot of questions there about intellectual property rights and so on.

But then also, what happens when those tools are being used as part of an enterprise mainstream, perhaps unknowingly by the stakeholders? And then, yeah, what happens when that sort of land in court? Right. It's a bit like 15 years or 20 years ago, long before AI, when a composer wrote a melody, which he completely thought was great and didn't know that it was already out there and created by somebody else. That would be the unplanned copyright infringement. And that would be the same thing on steroids in the end. So that will be interesting to see.

The good thing is the profession of lawyers will not go away. So this is something that which we will need all the time and even more, maybe, unless they are AI supported. If there are any questions, if there are any comments, if you think this was a great episode, let's have Guillaume and Jonathan talk more about these topics. Leave your comments below that video on YouTube. Send a mail to myself, to Jonathan, to Guillaume. We are easy to find. Send the talk to us individually. We are happy to do so. And apart from that, thank you very much, Guillaume, for the first time here.

Jonathan, for supporting me again in such an episode. That was an interesting conversation and it gave me some food for thought and I think and I hope that it also did for the audience. So happy to continue the conversation in Munich and everywhere where we can be found virtually and in real life.

Thanks, Guillaume. Thanks, Jonathan. Thank you. Thank you. Bye-bye.