Welcome to the KuppingerCole Analyst Chat. I'm your host. My name is Matthias Reinwarth. I'm Advisor and Analyst with KuppingerCole Analysts. And so is Jonathan Care. He is hailing from Portugal, close to Porto. Good to have you.
Hi, Jonathan. Hi, Matthias. And thank you for having me on the chat again. Right. So how is weather in Portugal? How are you doing in Portugal? Doing pretty well, actually. We've just started going into the warmer season, so that basically means it's t-shirt weather once again. And I'm hoping my roofer finishes the roof repairs so I can get my solar panels back and stop paying for electricity.
And so, yeah, the basics also are valid in Portugal, unfortunately. So you need to take care of electricity and weather and the roof.
Yeah, no, it's very agreeable here and looking forward to the season getting warmer. And do I believe you or don't I believe you? That's the question that we actually want to talk about today. We want to talk about misinformation, disinformation, untrust. And this is something that will be a key topic also at our event in May for EIC. But it's a key topic for everybody. So to have a proof that it's relevant, can you give, just for the recent months or years, some examples where misinformation and disinformation really spreads and really does harm?
I don't want to delve into the complex, intense geopolitical situation we all find ourselves in, but let me give you one example. I had a conversation with the General Medical Council, who are the organization in the UK that regulate and can disbar doctors from practicing. So they are the legislative board and have, yeah, they have legal powers. They said to me, this is during COVID, that they saw doctors going on Twitter and on YouTube and saying that, you know, vaccines were not effective.
And, you know, I think one of them was saying we should all just drink chocolate milkshakes and that would make it better. Without delving into the rights and wrongs of vaccines, and I know there are strongly held beliefs on that, the General Medical Council's view was that it is a matter of medical policy that vaccination is effective and that doctors should be following. And failure to do so is a fitness to practice issue. In other words, they can lose their license.
So the reason they contacted me is they said the defense they were getting in a number of cases was that, oh, God said that wasn't me. It must have been a deepfake. And so they were looking for any evidence, anything I could provide in terms of tooling or experiential knowledge to identify a deepfake. And as we know, the quality of deepfake videos has only got better. So last year at EIC, I presented a beautiful photo of me and the Dalai Lama sharing a roller coaster ride together. And obviously, anyone who knows me knows I don't like going on roller coasters anymore.
So that was obviously a fake. Really, the Dalai Lama and I were actually eating spaghetti bolognese together. So in all seriousness, the use of tools such as NanoBanana and even the recent developments with Anthropic Claude and the innovations with OpenAI and Sora 2 have only improved video generation to make it even more lifelike. And so it is going to, you know, these deepfakes are a challenge. If you like the ticks, we would say, you know, like the uncanny valley, which is the perception of a person that doesn't quite look human, raises some, yes, odd ancestral fears within us.
And so we can all spot something that isn't quite human. The hands look wrong or there were too many fingers and so on. All of these errors, all of these ticks are disappearing. And so we are seeing AI-generated imagery, including motion imagery, that is better and better. And I think that if we look at this information, we frame it as a political problem. So we think about elections, propaganda, and social media.
Well, as I said, that's already out of date. AI has industrialized production of false content and moved the attack surface into the enterprise.
So, for example, you know, I would, you know, if I touch up a video to make me look handsome and windswept and interesting, you know, is that false content or is it just me making the best of my appearance? What was once a societal concern is now a direct financial security risk. If you think about the number of processes that rely on us going, oh, yes, that's Matthias. He's allowed to, for example, access the Treasury accounts. That's Jonathan Kaye. He's allowed to access the safe where the company's private keys and certificates are stored. So the numbers make a point.
Deepfake fraud losses in North America alone exceeded $200 million in quarter one of 25. That's one quarter. And it is my belief that's only increased since then.
Yeah, I think so. And I didn't think of deepfakes being the modern version of the dog ate my homework. So this is interesting to hear.
But, yeah, of course, this is something where it wasn't me. It was somebody else who fabricated something. That is even another dimension of this disease that's spreading. When I did the intro a few minutes ago, I said misinformation and disinformation and trust. And this is something that comes out of my mouth as if it was one term, but it's three. Maybe to open the discussion, what is even the difference between misinformation and disinformation? And what should we know about the two of them? I think that's a great point.
For example, misinformation is false information shared without malicious intent. And I use the example of my uncle who will forward me a health claim or will forward me something he genuinely believes. This information, on the other hand, is deliberately crafted to deceive. The tricky part is that disinformation often becomes misinformation once real people pick it up and share it sincerely. For example, someone could share with me that, I don't know, I'm trying to think of a ridiculous example. Most have already happened. Drinking chocolate milkshakes is a cure for COVID.
And I might sincerely believe that I share that with you in good faith. Therefore, my act of sharing with you is misinformation.
However, disinformation are the people that created that false information and shared it. And it may be to further political agenda.
It may be, in that case, to increase the sales of their patent chocolate milkshakes. Who knows? But I think of this as like a counterfeit 10 euro note. A forger knows it's fake. Once it's in circulation, you, me and most people passing it along have no idea. So if we think about this in an organizational constant, this distinction drives our response. So inside a company, misinformation can be, for example, an employee sharing inaccurate threat intelligence. An area I'm very interested in. It can be a large language module, hallucinating facts in a board report.
These things are AI governance and they are accuracy problems. Disinformation targeting the organization is an adversarial security problem. Threat actors, financial fraud, executive impersonation. Each of those has a completely different threat model. And most security teams are only thinking about one of it. And I think as you described, what is it? The question is how it is conveyed to those who are consuming this information. I think an important part is also the way it is presented in a way that it actually evokes emotions, often strong feelings.
So that goes hand in hand to say this is not only something that is surprising or a fact that needs to be considered. Always some kind of, to put it mildly, engaging or it's really making people angry or very engaged. So it's not only the disinformation, the misinformation, but also the human factor, right?
And again, here I'm going to continue to try and walk the political tightrope and avoid anything that's politically sensitive. Because there you go. Emotional resonance is beating factual accuracy. So we used to think of politics as rational people conducting reasoned debates and coming up with a policy, a way forward.
Actually, what we're finding is that reasoned debate is actually less powerful, a convincer and motivator than emotional resonance. So a story that makes you angry or afraid travels faster and sticks harder than a dry fact check.
So yeah, I like my chocolate milkshake thing. It doesn't make anybody angry or afraid. But if I preface it with the COVID vaccine is actually really bad for you. And many people believe this. I have a school friend who is quite convinced that COVID vaccine can give rise to heart conditions, can give rise to neurological issues. And so these are clearly things that you or I, well, certainly me, make me angry or afraid.
Oh, my God, what's this vaccine? And even though I'm the rational part of me says, well, why would we have a vaccine that was designed to make people ill? Why would we have a vaccine that's designed to make people obedient to command? But these are things that make us angry, afraid. And I'm sure you've heard those rumors. I think when I mentioned that, I think I saw you nodding along. So those stories are familiar to us and they travel fast and they are much more than a dry fact check and so on. So it's very hard once these highly emotive stories are out.
Debunking them is like trying to put toothpaste back in the tube. And one of my favorite authors, the late Terry Pratchett, had a phrase.
He said, a lie can travel around the world while the truth is still getting its boots on. And it's this kind of idea. So more interestingly, what if we talk about pre-bunking rather than debunking? How can we inoculate people against these manipulation tactics before they encounter them? How can we teach them the pattern of the trick, not just correcting a specific lie?
And again, this applies within organizations to security awareness training. Both of us have just been through, I think, some security awareness training from our employer. We teach people how deepfake fraud works. If you see a video from our CEO saying, hey, I need you to send me your password or, hey, I'm stuck in a foreign country. I need you to send me 500 euros. Teaching people how these deepfake fraud works before they encounter it is actually more durable than a policy that says, well, if anybody asks you to transfer money or give away your password, then you need to verify it and so on.
Now, actually, the fact, again, that I'm sure many of us are listening to the podcast going, give away your password. I'd never do that. Good. That's an example of pre-bunking, inoculating, training people against that manipulative tactic.
Before, and people like yourself and I guess myself as well, who are security practitioners, before we started engaging in security awareness training, there were a lot of people and there were studies done in Liverpool Street Station in the UK. People were offered a candy bar if they could supply their password, which they happily did.
Now, fortunately, we have a security awareness mindset saying, no, actually, don't just believe a video. Don't believe an email saying, I need you to send me 500 euros. People are now, if you like, engaging in the more critical mindset, a critical thinking faculty. I fully agree. I think education can be one key to solving this.
But you've mentioned it in your introduction earlier to say, OK, there are so many sophisticated tools out there, which can be operated at scale so that they are, or the adversaries, they are really in a position to just throw out that many disinformation and misinformation that even this debunking process is too slow to deal with the sheer volume of disinformation. Isn't that another issue that we are facing when we are consuming all these signals on a daily basis?
Back in my single days, I observed that there are a number of ostensibly young, attractive women who were contacting me and expressing romantic desire and fascination with my somewhat holy, homely features. And, you know, they would be of various different nationalities. And I think a very common one was the ivory skinned Korean, Japanese, Chinese young lady. You mentioned that I'm from Portugal.
Well, actually, perhaps now if someone was trying to romance scam me, they would send a video call of a Portuguese one. Maybe I'm more interested in someone who is local as I am happily married. So hopefully I am not vulnerable to that particular tick. But you bring up another interesting point. If it's not one source, if instead you have a multitude of orchestrated sources flooding with contradictory claims. Chocolate milkshakes can cure COVID.
No, it's not chocolate milkshakes. You need to drink colloidal silver.
No, colloidal silver doesn't work. You need to eat grass.
No, grass doesn't work. You need to drink neat vodka. I could see many analysts being quite fond of that, fond of that last one. And the goal with the fire hose of falsehood, I call it, is to flood our information zone. If you like, the arena which we consume information with all these different contradictory claims, which exhaust our critical faculty and we give up trying to figure out what's true.
And, you know, I've done it myself. I really, really don't want to watch the news. So we have apathy, confusion, and these are the real objectives.
So, for example, in Malaysia, I believe there was a very successful campaign that says, if you don't support the government, then stay home. Don't vote. That'll show them. That's nice. And maybe it really rings a bell for somebody.
But, of course, the outcome is clear. Yeah, the outcome is clear. The activists who are voting or interested in the opposition party go hooray. Everyone get out to the poll booths. So how does this translate for our security, our audience today?
Well, this is the playbook in some nation state influence operations. And if you are a CISO in a large, and even the meat, you don't have to be a large organization anymore. If you're in the supply chain for a large organization, if you're doing something that is a key piece of research, you are quite likely to be a target in some nation state influence operations. Remembering here, the goal isn't to make us believe the false thing. It's to make us uncertain enough to do nothing. Do I put in endpoint detection? Do I need to use an MDR? Do I need to upgrade my firewall?
Do I need to put in place something to prevent agentic pollution? We sometimes think about agentic AI.
So again, the goal isn't to make you believe the false thing. It's to make you uncertain enough to do nothing. I think that's something we should all think about. It's certainly, I think, the reason why places like EIC and in our research companies do try to say, here's some clarity in a very vocal and often confused marketplace. Let's get back to this later. I just wanted to come back to one thing that you've mentioned.
To say, okay, if you use AI to look more handsome, or if I do that, that sounds in the first place a bit ridiculous. But if we look at the bigger social media platform that especially younger people than us use, if you think of Instagram or TikTok, I think there is no image of a single person where there is no filter on it. And what is a filter other than changing reality, than changing the message? And these images, these videos, they are picked up by the algorithms and they are promoted. And they have the visibility and the reach that is required.
So do these platforms, think of X, think of Instagram, think of TikTok, not even ask for this disinformation? Well, as you say, we have to recognize that platform design is not neutral. We have to recognize that the algorithms you said for all of these different platforms, X, Facebook, TikTok, LinkedIn, Blue Sky, Macedon, recommendations optimized for engagement, full stop. Outrage is engaging again, period. Which means that that infrastructure is an accelerant.
So, for example, a music composer who I'm acquainted with, I don't know him very well, not as well as I'd like to, made a statement which was taken to be supporting very controversial views about autism, and was then linked to views regarding gender identity. Without again delving into the rights or wrongs of it, the original comment was rapidly lost in the roar of outrage that occurred across the social media platform. With the result that this particular person had to retire because it was unacceptable to work with them.
Not because of what they said, but because of the perception of what they said. And so we have a real question here. Is the solution better content moderation, or do we actually need to change the incentive structures? Do we need to change the optimizing for engagement? And that's a much harder problem because clearly all of these social media platforms thrive on engagement. The more engagement they have, the more they can sell advertising, the more they can sell data analysis based on their user platforms.
And so nobody has solved this and everyone has gone for, well, let's optimize for engagement, cross our fingers and hope that the right thing happens. There's no evidence to that as yet. Is this also related to the erosion of trust in traditional media outlets, traditional sources of information that is then replaced by the influencers? We live in interesting times and we have our institutional trust.
So I compare my perception of media, government, even of science, and I would say that my institutional trust, how I view the UK government, how I view UK media organizations, how I view various different places is very different from the way that my parents view it. And certainly from the way it was viewed growing up. We now don't evaluate information on its merits alone. We evaluate the source. And I'm sure many of you listening on this podcast will say, well, yes, of course we do. Why wouldn't you? If we lose that institutional trust, it creates a vacuum.
And as we know, nature abhors a vacuum. So it gets filled by influencers, by partisan outlets and peer networks. And so one of the challenges for all of us in the analyst industry right now is that the up and coming generation of technologists tend to rely on peer networks. They're more likely to ask a question on Reddit than they are to call up an analyst firm.
And so we have to change how we are working to make ourselves authentic and to make ourselves accountable through different mechanisms, because these new modes of communication, influencers, partisan outlets, they feel more authentic, but they have fewer accountability mechanisms. And so we need to rebuild trust in this new generation of technologists. This actually does translate to security leaders who are seeing a new generation of employees coming in. When your employees don't trust official internal communications, they fill the gap with informal channels and rumor.
Of course, you would say, well, that's the water cooler. Gossip has been a thing since time immemorial. But it's changed. It is now far more primal. Information security depends on a functioning chain of trust. And this means that if you have a disinformation attack that breaks that chain of trust, if you have disinformation attack that actually tries to replace that chain of trust, because our institutional trust in authority is very different now, and indeed the next generation will be different again, disinformation attacks are quite powerful.
Trusted messenger problem is one that is going to be a problem for all of us in our employment as well as our personal lives. I remember many years ago, there was an initiative by the musician Peter Gabriel who handed out video cameras to make sure that there's proof of things going wrong when it comes to states behaving inadequately, to policemen behaving inadequately, etc. So the video camera and the product of the video camera was the actual proof to say there's something going on.
Today, when we have that good fake information, videos, deepfakes, whatever, does that not render the actual evidence not as important, not as trustworthy? And maybe is this not even a bounce back from the availability of this deepfake?
So to say, okay, I do not trust even legitimate sources because I don't know if it's legitimate. That's very interesting. You mentioned what Peter did many years ago, and the action of truth, the epitome of truth was video evidence. That's where the phrase came from. We are now very aware of deepfakes and AI-generated content. And so again, everybody smiled when I mentioned the Dalai Lama and me, but clearly I was deepfaking. And we now have a paradox, which I would call the liar dividend. Real evidence can now be dismissed as fake.
If you go back to the anecdotes I came in with about the General Medical Council trying to regulate practitioners, they were saying, well, no, that was a deepfake. A politician, when they're caught on video saying something damaging, has just claimed it's AI-generated. So this mere existence of sophisticated fakes gives bad actors plausible deniability for things that in fact are completely real. In my view, this is as dangerous as the fakes themselves. We don't just believe in the false things, we lose confidence in true ones.
And this is actually, I think, the most underappreciated aspect of the whole problem. I mentioned my parents. They're in their 80s. My mother is a retired medical practitioner. And she said to me, I don't know how people of my age cope with some of the challenges with the UK's National Health Service. I wouldn't be able to manage without my training in medicine to make sure that we get the treatment that we need.
Of course, not everybody of my mother's age is a retired medical practitioner. In fact, I'd say they're probably a very small percentage. So this liar's dividend is something we all need to be careful of. And as security leaders, again, you need to make sure that the information you provide, the training and the awareness you provide is unimpeachable. Because the moment somebody says, ah, that's just his opinion or her opinion, and here's evidence to the contrary, the role of security becomes much, much harder. Understood.
And in the end, when you translate that into the organizations, into the enterprise, into business, that is actual risk. But now that we've come full circle, we are back at EIC. We need to understand what is out there. And maybe you can describe this a bit.
But also, how can we make the right steps for preventing our employees, our colleagues, fall victim to such a thing and to get better in identifying that? Because this is a real risk and it will be something that we have to deal for the next years with.
So, first of all, what are the attack vectors and what can we do? Okay.
So, yeah, let's look at that. I said AI is the accelerant. And by that, I mean that generative AI makes it trivially cheap to produce convincing false content. I think that's unarguable.
I said, let's talk about where the enterprise risk is concrete. I think we need to do that because I've talked about lots of theoretical things. But I'm sure, again, our audience is thinking, well, you know, how does this really work?
So, let's talk about how we can use the same technology for detection and content authentication. However, historically in security, offense tends to have the advantage.
Defenders, that's a lot, are always training on yesterday's attacks. So, right now, there are three things that matter for security leaders. Voice cloning.
One, 30 seconds of audio is enough to convince you to clone a voice. And by the way, it's free. Look at voicebox.sh. This can be used for CEO fraud, vishing, and bypassing voice biometrics. Anybody using voice biometrics, your authentication system is at risk.
In fact, I want to actually go further than that. If voice authentication is part of your control set, part of your authentication stack, it's weakening.
Secondly, deepfake video calls. We have recent evidence of an employee at a company called Arup wired $25 million after a live deepfake video call impersonating the CFO. Not a phishing link. A meeting with a face and a voice. I talked about, you know, we've seen this in romance scams, but hey, here we are. It's right in our workplace. Our human instinct is to trust what we see and hear. And that's a vulnerability.
The third, and for those of you who work in fraud, you'll have heard this phrase. Synthetic identity.
By this, I mean AI generated people that don't exist. They're used for account creation fraud, fake employee references, vendor impersonation. The know your customer tools and onboarding controls were frankly not designed for this world. And they are outmatched.
So I said, let's talk about what does work and what we can do. Well, clearly don't rely on technical detection. And I would say the generation side is moving too fast. You need to rely on process controls. We need to rely on out-of-band verification for any high-value request. So if somebody gives you a video call saying, I'd like you to wire $25 million, that to me would be a high-value request. And so it's worth you calling the CFO on their landline, on their mobile, whatever you have available. But out-of-band of the video call and saying, hey, did you really want me to wire $25 million?
Regardless of who appears on the call, even if it's a CFO with a reputation of being grumpy, if it's a CEO who's got the reputation of firing people who don't comply with his demands. It is important, and it's important in our security awareness training, that we empower people with the ability to question what they are being presented with.
Again, dual authorization for wire transfers. It's actually much harder to scam two people than it is one. There's a whole host of psychological reasons why this is. But nevertheless, if you are trying to scam me to send a wire transfer, it's going to be much harder if you have to scam Matthias as well. Code word communications.
Again, this is a shared secret. We know that shared secrets are vulnerable. But they can help in sensitive, objective communications. So all of these tools can help us defend. But the main thing, the most important thing, and I must reemphasize this again, train people on the pattern of the attack before they encounter it. Inoculate. Right. So it's the same thing that we started with. You mentioned the awareness training that we are doing right now when trying to identify phishing mechanisms and checking links in emails.
It's just taking one step back and also not only trusting links, but also not trusting faces, voices, even if they look convincing, just to make sure that this could be easily faked. This is the message to convey to our colleagues, to our family, to our friends? I think so. And I think if I was to leave with a closing thought, we've spent 20 years teaching people not to trust unexpected emails. But I'm not proud to say that when we recently changed our security training provider, I fell foul of the first spoof email they sent me. So if I can fall for it, frankly, anybody can fall for it.
You just need to be sufficiently stressed and you need to be under pressure. And there you go. But we now need to teach people not to trust unexpected voices and faces. That's harder because trust in human perception is deeply wired. We are biologically inclined to believe what we see and hear. And so we need to make sure that one of those at the organization that does adapt and adapt fast, we need to have new process controls, new identity verification and the cultural awareness to question a face on a screen. That will be the real advantage.
And this also demands for some bravery to say, OK, I dare to do so. I do not trust that this is my CEO, that this is my colleague that I've met last year, but he's in the department in the US. So it really demands more standing by those who actually do it. And I think that is maybe also part of the story to tell. There is.
I mean, there's been some psychological research done on the different personality types in organizations, people who are rule makers, rule shakers, rule followers, rule blockers. And we need to, again, tune the message for each of the different personality types.
But yes, ultimately, it's the CEO, the CFO, all the senior executive leadership of a company to say you are empowered to question. If it's me, you will not be fired for questioning this.
In fact, you could even change it. We will reward people who successfully challenge a deepfake. So I think there's a lot of motivational tools we can use here. But the most important thing is to say people need to feel it's OK to ask questions. It's OK to challenge. A new type of second factor being built into the brains of our colleagues. So to really double check whether this is something that is really OK. So thank you very much, Jonathan. That was really interesting and really went full circle. We went from TikTok to filters to your parents up until to how can we protect our enterprise.
And I think that is really the full scope of this topic of misinformation slash disinformation and trust. We will follow up on that at EIC.
Of course, it is baked into the overall fabric of identity because this is where attackers are trying to fool us. Any final words before we close down?
Just, yeah, I echo that. I know there are some vendors who are really working hard to challenge this. And I know that many of them are going to be at EIC. I look forward to talking with all of you and say workshopping, discussing this problem. And maybe we come up with some innovative solutions to this in this area. Thank you. And for the audience, if you have questions, please leave them below this video on YouTube or send Jonathan or me a mail. If you have any suggestions how to work even more in this topic and to have another episode on that going into more detail, let us know.
Send us your questions. If you want to talk to Jonathan just about this topic or to me, please reach out to us. We are happy to do so. And for the time being, thank you very much, Jonathan. Thank you for your time. This has been a longer episode, but it was worth every second. And I'm really looking forward to having you soon on this podcast and see you next time.
Thank you, Jonathan.
