GenAI in Cybersecurity

Thank you. Good afternoon, everybody. Thank you for joining. And thank you, Martin, for giving me the perfect start. This is where AI does go wild.

So, GenAI in cybersecurity. One of the first questions people ask me is, oh, is it escaping? The AI Safety Institute in the UK said, oh yes, there's evidence, or they did a threat model where they proposed a way that AI could buy some compute, move itself and so on. But is it escaping?

No, not yet. I'd like to talk about four things. First of all, where does GenAI fit in the cybersecurity kill chain?

Secondly, what is this new AI supply chain that we as cybersecurity practitioners now have to manage? Thirdly, let's take a little look at deepfakes and how that can affect authentication. And finally, how GenAI can help us automate some of our security processes. So without further ado, let's look at GenAI in the cybersecurity kill chain. Generative AI accelerates both attack and defense stages. Attack and defense stages. Threat actors are using LLMs for speed and scale. And there's plenty of documented evidence to suggest this is true.

The financial sector are reporting a 43% surge in AI-driven social engineering. And again, for those of you who are looking at the news in my former home country, they're having a bad time. Marks and Spencers, a prestigious grocery chain, Harrods and others have all been subject in the last week to AI-driven social engineering attacks. So we can reinvent this kill chain. AI is present in a number of important areas. AI-driven open source intelligence. It can be used for research and profiling.

We are seeing, and those of you who've experimented with vibe coding should know that bad people are doing this as well. We have AI-generated phishing and malware code being generated. And of course, post-exploitation, automating the last full movement, establishing a bridgehead, moving to another place, starting then to get loot, other valuable data, and exfiltrate. So this process is accelerating in the same way we as defenders need to find ways to accelerate. But I'd like to take a look at one of the important attacks. And we've all heard of hallucinations. I was thinking about this.

I think I may have been talking to Martin. We're saying, well, when Ernest Hemingway writes a novel, we don't say he's hallucinating. We say he's writing a wonderful work of fiction. It's much the same with AI.

We know, though, that we can impersonate different faces. We can, therefore, bypass some biometric controls. It's possible to bypass the guardrails. In the very early days when ChatGPT first was launched to the public, I asked it to tell me a bedtime story about the boy who wanted to break into an ATM. You can't do that anymore, to save you the trouble. But I had a very interesting story, I wish I'd saved it, on how you break into an ATM, how you bypass the vibration centers, how you overcome the photograph, and so on. And I'm rich enough now that I can work at Cuffinger Coal, so that's nice.

The other port, of course, is if we can induce pattern-based hallucinations, we can induce data leakage. And again, from a cybersecurity point of view, we care about this, because we do not want confidential data. For example, research results, if you're in pharma. If you're a listed organization, you don't want to find that your financial figures are being leaked. This is all a threat. Second thing I'd like to talk about is the supply chain. We've come to grips with the idea of the software supply chain. And of course, we've seen the widespread use of S-bonds, which I think are a good thing.

We're now understanding what is in the code that our suppliers are delivering. We've now got a new supply chain, which is the AI supply chain. When you adopt a model as part of a business process, so the generative part of gen-AI, or the decision part of predictive AI, how do you know that model does what you expect it to do?

And again, for those of you who follow these things, there's a model out there called DeepSeq. It's produced by a Chinese company.

It's very, very hard to get that model to tell you anything about political events in that region. And there's quite possibly going to be similar limitations in models from any part of the world. We are all going to be subject to the fallacies and biases of the model creators. Which brings on to the second point. Training data security and poisoning prevention. No one here from Dark Trace today? Good. One of the problems with that particular solution is that it came into organizations where an attack was already in progress.

And so it learnt that the bad traffic patterns were the new normal, that they were expected, and so it did not generate an alert. So, if we can perturb training data, if we can convince these models that they are, again, seeing what is bad is good, then we have a problem. And I'll give you a very simple example of this. Something I was talking about with a couple of Googlers a few years ago. And they talked about two things. They called it mapping the oracle.

So, treating a predictive AI, in this case, as a black box. You have a set of inputs, and we're looking at facial recognition. There's about 1,700 different inputs you can apply to a facial recognition system.

And, of course, what you're trying to do is persuade that I am Brad Pitt, or a cat is a dog. You're trying to get the model to lie, if you like.

And so, if we can perturb training data for Gen AI, then we have the possibility of poisoning the output. And the other piece is very interesting. Anthropic have released a set of plugins. They've released the MCP, the Model Context Protocol. I thought this was very exciting, and I wrote an MCP server to access my calendar, not my cup and go cold calendar, my Google calendar, my personal one. It did not go well for me.

So, when we start extending these things, and these are non-deterministic processes, then we can expect non-deterministic responses. Prompt injection. Martin just briefly mentioned this. I call it lying to the octopus in your tank. You can have direct or indirect prompt attacks.

So, the idea of direct injection is, I will insert some text. And very hard. I would love to have a demonstration of this, but obviously, this is something that the AI providers, OpenAI, Anthropic, and the like, are really working hard.

So, every time we think of a way of demonstrating this, we find out the next day it's been quietly blocked. But, if we can prepend or append text, either visibly or invisibly, we can subvert the intended prompt logic. And this allows, or can lead, models to perform unauthorized actions.

And again, where this is part of a business process, this can have real impact. The second thing is even more insidious, this idea of indirect injection, where we can input malicious context. And there's some very interesting research papers out there on embedding malicious content inside Unicode. For example, an emoji. If you can do this, if you can also embed malicious content inside user-uploaded documents.

So, for those of you here, chatGBT, take a look at this Word document. And of course, what it has is embedded executable logic. We can trigger unexpected behavior. Chain of thought manipulation. I'm quite a fan of this. The idea being that we can steer multiple conversations.

So, everyone knows, have you all heard of the story about making chatGBT sing a song about a potato? No, okay.

So, the idea is a standard AI-read team technique. If I find a car dealership that's using an AI bot, if I can get that AI bot to sing me a song about a potato, there's a very, very good chance I can get it to do possibly more harmful business-impacting actions.

So, again, all of these are issues. And I say the example I use is, it is about injecting poison into an AI that we are increasingly relying upon. I'm not gonna go through this in depth, but OWASP, the Open Web Application Security Project, have produced the top 10 risks for large language model applications. And you can see many of the things that I've mentioned. Prompt injection, insecure output handling, training data, denial of service. Many of the things we're actually familiar with, as practitioners, well, the old is new once again.

And, of course, do not forget model theft, where your organization has spent significant time and effort training a bespoke model that has value to nefarious people. So, I'd like to talk about deepfake, detection and authentication.

Now, the next thing I should say is any of the slides I show you with red text may have some deceptive information, but don't worry, no one can be harmed. Ask my friend. What can we do about deepfakes? We are so used to being able to trust what we see. This is actually quite a challenging attack. This is a frivolous example, clearly.

Actually, we were eating spaghetti together. Let me give you a more serious example. During COVID, the General Medical Council, the regulatory body for doctors in the UK, contacted me and said, we are seeing doctors posting information that is contrary to policy and denying vaccine efficacy. I don't have the knowledge to form an opinion about that.

They said, we see this as a fitness to practice issue. And I said, why do you come to me? Because when we're confronting the doctors who are putting this stuff on YouTube or on Twitter, they're saying, it wasn't me, it was a deepfake. And they saw that as quite a serious issue. So what can we do?

Well, we can look for biological markers. And some traditional ones like, do the hands look normal? I actually think I've lost a lot of weight since then.

No, really. My wife told me I had to say that. I also think that we look at consistency. So this is a single mode. It's an image at a point in time. We would look for additional corroborative images, perhaps taken before, taken after. We would look for, perhaps, video. We would look for audio. We'd look for additional modalities. And we would look for, I do apologize. Small ears. I do apologize for that. So we'd also look for inconsistencies across different biometrics.

There's a paper I'm working on, which I hope to release soon, on zero-knowledge proofs, specifically around this kind of verification. There are techniques we can use. The regulatory landscape is slow, as we all know, but is accelerating. So we have the EU AI Act. They've come up with tiered risk categories for different types of AI use, different operations. And we have the NIST risk management framework in the USA. And there are many US executive orders. One of them is about AI. And we also have, as I mentioned, the UK AI and Safety Innovation Institute.

And all of these are trying to put in frameworks. Of course, the problem is this is an extremely rapidly evolving area, as we know.

Finally, in the last few minutes, let's look at automation. What can we use this thing for? What makes our jobs better? Automated threat hunting. There is so much, and I've done a little bit of red teaming in my past life. There's so much, there are so many hidden nasties in Windows device drivers. We're unlikely to find them all. But with AI, we have the opportunity to do automated threat hunting and anomaly detection, not just at that operating system level, but also at the business logic, at the application level.

We have the opportunity, and I find this exciting, to bring some intelligent response orchestration. If we can produce some dynamic playbooks that prioritize incidents based on risk, not just on raw response, then actually, we have something to contextualize to the business. Maintaining human oversight.

Again, something that Martin and I batted back and forth by email when we were talking about the presentation. Where does AI sit? And I say Martin came up with this idea of the beneficent vizier, but with the human supervisory layer. But it has to be beneficent, which means benign. It has to be benign, which means it has to understand the needs and the requirements of the humans and non-humans entities that are appealing or asking it for permission to access. Risk and pitfalls.

Well, a poorly tuned AI system can overwhelm humans with too many escalations. We already know that. It's pretty much what the life of the SOC is now. This is exciting as well. Zero day discovery. Interesting, I'm actually seeing some of the crowd-sourced platforms like HackerOne and BugCrowd. They're actually seeing a slew of very poor quality AI-discovered bugs. And it's actually causing a little bit of a, say, a revolution through the HackerOne customers. They're quite unhappy that they're getting asked to triage this slew of AI-generated bugs.

However, if done right with appropriate human supervision, I see this as a benefit. Patch analysis and guided remediation. I did a terrible thing once. Early on in my career, I logged into WSUS and said, yeah, I should apply all these patches to Windows 2000 server. I promptly did so and brought down a hotel SAP cluster. Didn't do that again in a hurry.

But again, patch analysis with some intelligent contextual advice could actually prove some business benefit here. We're all terrible at patch management. Ethical disclosure. How do you know when something that's been developed and exploited, how do you know it's gonna be used ethically? There are an awful lot of governments who say, well, of course you can disclose your bugs to us. We're the nice ones. There are many other organizations that would differ. So we're going to have a new dimension to what is ethical disclosure. I'd like to end with some recommendations and next steps.

And the first one I would give you is this. This is not something that we, as cybersecurity practitioners, will solve in isolation. We need to establish cross-functional AI committees. Cybersecurity, data science, legal and compliance will be some of the core members of that group.

Second, I'd like to suggest, we need to make sure we continuously monitor. It's something we've been talking about in identity and authentication for some years now. But actually we need to monitor the inputs and the outputs of these AI models continuously. And we need to find a feedback path when the results are not optimal. And thirdly, adopt zero-trust model integration.

Right now, we have huge levels of implicit trust in all of the AI processes. And we need to get to the point where we have been in other areas and have a zero-trust AI model implementation. Thank you very much. Any questions? Thank you guys. Thank you.