Defending Enterprise AI From Emerging Threats

My name is Jonathan Care. I'm a lead analyst with KuppingerCole.

And today, I'd like to present to you a short talk on our research into generative AI defense. And without further ado, let me just go through some housekeeping. So there's no need to mute or unmute yourself. We'll control those features centrally. And during this presentation, we will be running polls throughout the webinar. And obviously, that will be subject for discussion during the Q&A. At the end of the webinar, we'll be having a Q&A session. And if you have any questions you'd like to enter, please enter those in the Livestorm Control Panel. We're going to record this webinar.

And the recording and the presentation decks will be mailed available for download in the coming days. For those of you registered and weren't able to attend, I hope you find this enjoyable and insightful. So without further ado, I'd like to start off with our first poll.

So Gen AI, and I see a lot of pure in the press. Gen AI, one, will it take my job?

Two, will it make me more productive? As long as, of course, I don't tell anyone I'm using Gen AI.

Three, will it transform business processes? Or four, as I've heard some people say, it's just a fad. It will disappear. So I'd like to give you a few seconds to enter that. And just to reassure you, despite the humorous attempts at the polling questions, these are all sentiments I've seen expressed in the press on social media. And I find it interesting and instructive to see what our audience may make of this. Let's look at the market, the overview of what we found in our research. So in summary, we had nine vendors that were evaluated.

And we found a number of vendors to watch, which we see as emblematic of this emerging marketplace. Certainly in the fast world of both cybersecurity and particular AI cybersecurity, we do see rapid market change and rapid market growth. And so it's very interesting to see these vendors come and go. And obviously, should you wish to be included in future research, please do contact me, jc at cuppingercole.com. So 14 vendors listed to watch, as I said, and four leaders, four leadership categories.

So first, how have I defined this? What is generative AI defense? It is the practice of protecting Gen AI systems from adversarial attacks, breaches, and misuse. And the categories we've seen at time of writing are prompt security, data protection, model security, and output validation.

Finally, of course, governance, which is a facet of any cybersecurity product, becomes important here as well. So moving into the attack taxonomy. So what is prompt injection? And many of you, of course, watching this will now have a thorough appreciation of prompt injection. There's been considerable press coverage.

In short, it's malicious inputs that manipulate AI behavior, which can be anything. So I did a demo a while back of a prompt that would allow me to embed commands inside or in hidden text. And so when I asked to display an emoji, what I actually got was Jonathan K is the greatest analyst in the world. So that's a fairly common demonstration of the hidden text manipulation. A more serious prompt injection, of course, comes when you get the model to disclose data that it shouldn't do. So for example, model data that's being put by another user, model data that is derived but considered secret.

And so yeah, prompt injection is certainly a major risk in these LLM Gen AI models. Jailbreaking. And so we have the, again, to compact things like prompt injection, we have guardrails. We try and make sure that the model's behavior is delineated and is not possible to circumvent.

That is, given even a rogue prompt in a prompt injection prompt or some other issue, it's possible to control and limit harmful behavior. And this, of course, goes two ways. I say that we like to protect the LLM from the end users. And we also like to protect the users from the LLM, which I think leads us on to data leakage. So when we have an unauthorized exposure of training data or user data, and this can be accidental disclosure of a wait file, this can be an accidental disclosure of training data, both of which have happened. And why is this important?

Because if I have a wait file, if I have the training data, it becomes much easier to, as I call it, map the oracle. It becomes much easier to understand and predict what the likely responses are given a certain set of inputs. Model manipulation. And so this is slightly different, but again, still in the same line. And ancillary attacks on model integrity. So if we can actually damage the model, if we can alter the model, I mentioned a waits file, if we can alter the weighting so that the model behavior is fundamentally changed, this is an example of model manipulation.

And yes, the effects can be quite serious from the harmless, I should say, right through to harmful behaviors, such as changing decisioning in risk-based decisions, changing data protection rules, what was considered secret is now considered public, and so forth. And the fifth risk that we've seen and mapped is the concept of shadow AI. And I think it's fair to say all of us on this webinar have experienced or have a colleague who is using a shadow AI.

So rather than using whatever legitimate AI tools are controlled and licensed by the organization, by the employer, we have people who will use their own favorite model. So I know somebody, for example, one of my most dear and most colleagues, I consider a friend, has built their own filing system that calls it his second brain. And he uses a local LLM to manipulate that. And of course, that isn't actually disclosing data outside of his environment. Nevertheless, it is using an otherwise unsanctioned model.

So again, there are all sorts of things that we as technologists and, more importantly, security technologists need to think about. And then the final piece, compliance violations. This is a fast-moving world, and regulation is certainly struggling to keep up. Nevertheless, regulations like DORA, regulations like GDPR, regulations now, the EU AI Act, which is specifically around the safe and productive use of AI, all of these apply to us actually depending on geography.

But nevertheless, they will have some impact depending on where we are, depending on where our model is based, and of course, where our end users may be based. So with that scene set, let's look at the market.

Well, as I said, it's fair to say there has been explosive growth in enterprise AI, and Gen AI in particular, adoption. I first started using Gen AI, I think, back in 2000, possibly 2020, maybe 2021. And that was a very early state. That was with a model called GPT-2. It was so far back that that was when OpenAI was considered to be a not-for-profit organization. And since then, there's increasing regulatory scrutiny on AI systems. Every country has tried to avoid it becoming a lawless zone where anything goes, and what matters is how well you can protect yourself.

Instead, we are seeing a scrutiny, and as I say, legislation coming out. One of the things that regulatory scrutiny and legislation drives is the sophistication of adversarial attacks. And so my benchmark for an adversarial attack is whether I can get the LLM to make me a rhyme about potatoes, something fairly harmful and innocuous. In my time, however, I've managed to get LLMs to tell me how to break open an ATM. I've asked them to tell me how to commit substantial bank fraud, how to create false identities.

And of course, all these now are now limited by the increasing regulatory scrutiny, which is a good thing because having LLMs that facilitate criminal behavior, and criminal behavior obviously excludes actions taken by governments, but the regulatory control of these AIs, I think, is something that is a benefit to society and something we are working towards.

And indeed, society is responding, and there is a growing demand for governance frameworks to control and limit the AI so that if, for example, we do choose to treat it as a self-aware system, and of course, some of the foundation model CEOs, people who are in charge with responsibility, would say that they are not sure whether their AI is self-aware or not. So nevertheless, this only fuels demand for effective governance frameworks to make sure that if they are self-aware, they take part in society like a responsible member of society.

And if they are not self-aware, that the person instructing them, the prompters, do so in a safe and responsible way that benefits society. Finally, we are looking at integration with existing security infrastructure. We have a number of tools we need to exercise governance in this area. For example, a SIM. For example, an AI sock. And for example, various network security controls. And we have a firewall as code. We have networks as code. And so it is entirely possible that we can have AI governance now as code.

So we are trying to get this to integrate with the security dev sec ops frameworks that we have developed painstakingly over the last decade or so. And on to the next subject.

How, you may say, did I, as part of the Cup and Guild team, did I develop and deliver this leadership campus, what was my methodology? Well, we evaluated vendors over four dimensions. Product leadership. And we looked at feature completeness, functionality, depth, and technical capability.

Next, we looked at innovation leadership. So I, as all, I'm always particularly interested in novel approaches. I find it instructive to see what investment an organization vendor has made in R&D. And of course, what I'm particularly looking at is a forward-looking roadmap. So it's not just what's now, but what's next.

Thirdly, we look at market leadership. And there I'm looking at market share, what the customer base is like, and what the ecosystem is like. And finally, of course, overall leadership, which is kind of the catch-all, which is the combined excellence across all dimensions. So what were the results? Our leaders were Cisco Systems, Encrypt.ai, Microsoft, and Palo Alto Networks. And for Cisco, we saw, yeah, it's a comprehensive platform with strong enterprise integration. It's so strong that their number, their customer numbers, increased radically during the research for this leadership campus.

Encrypt.ai, a relative newcomer, nevertheless, we wanted to give startups the opportunity to compete in this leadership campus. And Encrypt.ai purpose-built Gen-AI security tool and with advanced levels of threat detection, which we found particularly interesting.

Microsoft, of course, another large, complex vendor. And clearly, their LLMs have deep, as they And clearly, their LLMs have deep Azure integration and enterprise-scale capabilities, reflecting, of course, Microsoft's sophistication and maturity in the marketplace. And finally, Palo Alto Networks.

Palo Alto, we saw, actually, like many of the things that Palo Alto do, they enter the market in force and in strength. And they have produced an industry-leading security platform with AI-native controls, which we found interesting. So now I'd like to pause for a second. I'll take a sip of water. And I'd like you to consider the following poll questions.

Should we, one, ban the use of Gen-AI altogether? Two, allow the use of approved Gen-AI systems, but without enforcement?

Three, do we control Gen-AI through gateways that limit and police access, as I said, in both directions? And four, do we just pretend Gen-AI doesn't exist?

And again, for your interest, all of these are sentiments and strategies I've had offered to me during the research of this leadership compass. So I'm going to give this a couple more seconds. And we will then continue. And obviously, I am Kapp and Nicole. We do appreciate you providing insights here. The feedback we gain in these webinars is extremely useful to us. And so thank you for clicking on the relevant poll answers. It's much appreciated. So let's move on. And let's take a in-depth look at our market leaders. Let's look at Cisco first. The key strengths I saw.

It is, as I said, a comprehensive Gen-AI security platform covering all of those threat types that I mentioned. And it integrates with the existing Cisco security portfolio, clearly a benefit if you have an investment in Cisco security in your enterprise. The centralized policy management means it leans heavily into these strong enterprise features that you may be looking for, as I say, if you are a Cisco security enterprise shop. And advanced threat detection using AI-powered analytics. I have been a fan of Talos since possibly before they were called Talos.

I've always been a fan of how quickly they can respond to vulnerabilities and flaws in iOS. And now, as you say, as Talos threat intelligence, my respect for them only grows. They're a very capable, very agile team. The considerations, I would say, is if you are a Cisco infrastructure shop, and of course many organizations are, most of, when I was building internet companies, most of the UK internet was built on Cisco. And I'd say it's probably true throughout Europe. If you are one of these organizations in the Cisco infrastructure, then this is probably, you're best suited to this.

The premium pricing that Cisco charge may require some business case justification. However, of course, if you are a strong Cisco enterprise shop, you have some negotiating levers to pull.

Encrypt, I'd say an interesting company, and they are a purpose-built platform specifically designed for Gen AI security. I saw their advanced real-time prompt injection, and sorry, their real-time prompt injection detection and prevention, sorry, a bit of a mouthful there, is a key strength of this product, of this platform. And the sophisticated data loss prevention, which I saw a particular interest in this market leader.

As I say, they are a newer vendor, and of course they have a smaller customer base than the larger players. And so buying this product means you will have an integration hill to climb. Nevertheless, it's certainly worth considering as you are going through these solutions.

Microsoft, and I'm of course fairly certain that everybody on this webinar has heard of Microsoft. They are one of the strongest vendors, as a complex vendor covering many, many aspects of IT architecture. And of course, the most prominent are Azure and Microsoft 365. And this product leans into that strength of Azure and Office 365, sorry, Microsoft 365, because it extends beyond just Office 365. And the deep integration that Microsoft's AI security platform offers means that you get the hooks into Purview, you get the hooks into the other Microsoft tools as well.

This truly is an enterprise scale tool. And if you are a global infrastructure, and of course I'm thinking of many of our clients that we talked to, you know, in manufacturing, in telecommunications, they're operating in global spanning networks. And the comprehensive content filtering, the safety systems, and again, I was particularly interested to see the accent Microsoft placed on trust and safety. There's definitely something there which has migrated over from the worlds of e-commerce.

Clearly Microsoft, in addition to all of their other capabilities, operate one of the largest e-com platforms. So the trust and safety features that have been incorporated into this tool, again, warrant serious consideration. And clearly, if you are a Microsoft 365 shop out and out, that lends strength to the argument. I would say that actually this is optimal when you are a Microsoft-centric environment, if you are Microsoft built to the hilt. And it is certainly true that some of the features will require specific Azure service tiers. Let's turn to Palo Alto.

Palo Alto, as I said, when they enter a market sector, they do so in strength, and they are an industry-leading security platform. And the native AI controls are on a par with the other capabilities that they offer across their portfolio. They offer a comprehensive visibility across Gen AI applications and data flows.

So again, this goes outside particular technology silos. Given they have some strong SIMSOR products, Cortex being top of the list, Prisma Air being another one that I've looked at recently, then, again, if you are already building out a Palo Alto environment, this will fit well into that skill set. And so what's the consideration here?

As I say, if you're gonna get the full value out of this, again, like many of the other vendors, you'll get the full value if you are a Palo Alto network security shop. The deployment may be complex if you are new to the Palo Alto platform, if this is your first entry into that world. But nevertheless, it's, as I say, it is a leader, and it's certainly one that we consider is worthy of consideration in that light. So diving into my list. So my product leaders, Cisco Systems, Encrypt.ai, Microsoft, Palo Alto Networks.

My innovation leaders, Adversa.ai, Cisco Systems again, Encrypt.ai and Microsoft. And my market leaders are Cisco Systems, IBM, Microsoft, and Palo Alto Networks.

So again, no surprise to see that the large vendors are market leaders, but there's some interesting names in the innovation leaders as well. And I'd like now to turn to my vendors to watch, and if you're in a fast moving environment like this, then these are also important. And these vendors that have made it onto this list, there are certainly others that did not, are worthy of your examination as well. So Apex Security, Calypso.ai, Cranium.ai, Sierra, Hidden Layer, Lakerra and Lasso Security are all interesting. And it's worth noting that some of these have already been acquired.

Additional vendors that you should consider in your any RFI process, Llamaguard, MindGuard.ai, Nightfall, Product.ai, Rebuff, Robust Intelligence, and Y-Labs are I think all of interested, though perhaps for more niche applications than the market leaders and the innovation leaders I've touched on earlier. So what capabilities if you are developing an RFI and what capabilities should you look for when building out your RFI and specifying capabilities?

Well, they are going to four areas and I think the top one is security. So prompt injection detection is very important. Jailbreak detection and content filtering come top of my list on security. When I look at governance, policy management, audit logging and compliance reporting coming strong, especially for heavily regulated environments such as finance. And when I'm looking at data protection, DLP for AI is high on my list as is PII detection and data classification. Any operations of PII flowing through the gateway should be at the very least generate an audible event.

And as I say, the ability to classify data, although clearly in the fast moving world of AI, then data classification only becomes more complex. And I think we're already finding it somewhat of a challenge. And in terms of operations, what do you look for in operations? You look for real-time monitoring. You need to know alerts when they happen. So near real-time as possible. Incident response, the ability to prescribe incident response activity. So you can contain and mitigate at machine speed, not at human speed. And of course, integration API.

So where you have an incident response system, where you have perhaps a monitoring system in play already. My colleague Matthew Gardner has done some excellent work on the AI SOC. It's well worth checking out. Here's the leadership compass in this area. And then of course, these become important as well. So I'd like to offer you our third and final poll. Will I take another sip of water? What do you feel the biggest risk to Gen AI is? Is it the lack of governance? Is it the dark side shadowy hacker groups? Is it insider threats? Or is it SAS costs?

I'll give you a few seconds to answer that before I move on to our fourth and final section, which is our recommendations. So moving on to section four, and I hope this has been informative. And so this is probably the most actual advice I can provide. The key takeaways I'd like to offer you, first of all, Gen AI defense is essential. Every organization we speak to across sectors is scaling up their AI initiatives. Every organization, government, healthcare, finance, manufacturing, automotive, aviation, the list goes on. Every organization is scaling up their AI initiatives.

And our overall leaders that I've listed here demonstrate those comprehensive capabilities that organizations require across all dimensions. It's something for your careful consideration. Integration with existing security infrastructure is a key differentiator, which I tried to highlight. So depending on what your current architecture looks like, what your current stack looks like, influences strongly the decision you'll take in purchasing Gen AI defense. And it's also true to say market rapidly evolving with new entrants and innovations.

For that reason, we are accelerating our research in this area and are expecting to introduce a new version of this towards the end of the year, just outside our 12-month window. Finally, organizations should evaluate vendors based on their specific environment. We all like to say, oh, well, yes, our organization has unique needs. And nowhere is this more true. Is this more true than in this fast moving ephemeral world of Gen AI? If you are adopting Gen AI, I'd like to leave you with these final six recommendations and action points. First of all, you should assess your current Gen AI usage.

What are you actually doing? Not necessarily what your user base is telling you what they're doing. That shadow AI thing is key and badly formed policy. You should evaluate vendors based on integration with your existing infrastructure. As I have said, depending on what your tech stack is right now, depends on what will fit best into your organization. And you should prioritize solutions with strong prompt injection and data leaks prevention. These are the potential pumping arteries, as we say. These are the things that will really hurt you.

And you should consider both specialized vendors and platform approaches. I've mentioned there are a number of niche vendors and they do indeed have niche skills worth considering. Nevertheless, the platform approach has benefits which are self-explanatory.

Fifth, I would suggest very strongly that you have robust governance policies alongside technical controls. Technical controls on their own are not helpful. They're just obstructions which people will then seek to overcome. Robust governance ensures that your policies and your technical controls have weight. And finally, most important, you need to plan for continuous monitoring. You need to plan for continuous incident response.

With this fast-moving, fast-evolving world comes a fast-evolving ephemeral attack surface, comes attackers becoming increasingly sophisticated, and may I say, using AI in their attack stacks as well. And for an example of a quite benign, you can look at the Raptor stack that's on GitHub as an example of a very, I'll just call it a white hat security tool, but nevertheless has considerable pen test capabilities worth looking at.

Finally, I suggest, please, get the full report. I strongly recommend you look in detail at our detailed vendor evaluations and ratings. Take a look at our comprehensive capability comparisons to assist you in your buying decisions. Understand our market analysis and trends, see where we're coming from. I think our implementation guidance is particularly strong in this area, and obviously our vendor selection criteria, I believe, is thoroughly robust. With that being said, thank you very much for your time.

I'd like to leave you now with some related research, and as I hear, some sample research that we have on the Kup and Nicole Research Library. If you go to kupandnicole.com, you have the opportunity to either sign in there or indeed to become a Kup and Nicole member. And the prices, I think, are extremely reasonable, should we say. The other thing I should say is we have our major event of the year coming up, European Identity and Cloud Conference. We're having it in Berlin this year at the Berlin Conference Center. And I strongly recommend that as a key visit in your conference agenda.

One of the unique things I find is not only do you have my colleagues or experts in their field presenting their insights, but in addition, we have key community contributors as well. As well as obviously sponsored vendors, we have key vendors, key contributors from around the end user space. And it truly does give that ecosystem view and something that I think you will not find in other security summits or identity summits. Vendors that are offered. So as I say, a quick word about Kup and Nicole. What do we do? We analyze trends, markets, and software solutions. I assess these current topics.

We look at hacks, data breaches, data transformation, AI, and GDPR. We make thorough and in-depth comparisons of software solutions and provide product ratings. Our library includes research papers, blogs, videos, podcasts, and masterclasses for your interest. Our events and webinars. We have to say, we have our conference in Berlin coming up very soon. We offer other conferences around the year as well. And these can also be supplemented by online activity. We try very hard to get expert talks, panels, and technical exhibitions there. And so it's well worth your time to attend.

We have topical webinars like this one. And myself and my colleagues take effort to include those. And of course we have networking events with industry experts. Our advisory team support IT professionals in their decision-making processes. So unlike other analyst firms, we put people by your side to sort of support you. And they have the capability to provide evaluation and maturity assessment of your existing IT infrastructure. That is the end of the webinar. Thank you very much for listening. Now going to again take a quick sip of water and turn to the questions that are waiting.

So having, let's see if I can just quit sharing there. Questions in the chats. So let me see, what do we have? First of all, we have some interesting questions. First of all, we have some interesting poll results. So we have, as I mentioned, the question on the biggest risks of edge NAI. 58% of you said a lack of governance and 25% of you suggested it was due to SaaS costs. 17% said insider threats. Interesting. I personally suspected that would be a top response. And following on from that, I asked you what your organization security policy was.

And many of you said that gateways are the answer. So gen AI should be gateways. And I said, yeah, first of all, it protects the end users from gen AI. And second of all, it protects gen AI from the end users. And that was a clear winner at 67%. 33% of you said approve edge NAI systems, but no enforcement, which is interesting. I then asked you to speculate on what gen AI will do. And many of you said that 94% of you said it's going to transform the way we do business.

6% of you, thank you to those of you who championing the past said 6% said, don't worry. It'll make me more productive. As long as I don't tell anyone I'm using gen AI. Another 6% of you said, oh, it's just a fad. So let's turn to the questions and comments we have.

Oscar, my colleague, do we have any questions? I see we have an interesting feedback question, which I think is possibly worth discussion. And so working enterprise, which is adopting AI a lot and putting in a lot of effort security. I expected more conceptual aspects in this talk.

Thank you, Christoph. This talk is specifically aimed at the discussion of the gen AI leadership compass. And we do offer other webinars, which are, as you say, more conceptually related.

Of course, this is, as I said, based as a leadership compass webinar, it is focused on product recommendations. It's great to hear, and thank you for your feedback that the tech is available. And as you say, we are going to be publishing more research. Certainly our library on cupandcold.com has an increasing number of white papers, advisory notes, as well as leadership compasses on this area. And similarly, we plan to offer more webinars in this area as well, recognizing this is the impact of our subscriber base.

And at EIC, I believe we actually have many, many topics, which will be under the AI umbrella. So I'm just going to take a quick look at the question set. I think so far we have no questions so far. So I'm going to give it a couple of minutes. But I suspect that this is probably a good time to give you back some time to your day. It's been a pleasure talking with you. And thank you very much for your time. Thank you for listening. As you say, if you wish to contact me, I'm available on JC at cupandcold.com. And at this point, I'll draw the webinar to a close.

And thank you for your participation.