I'll just ad lib while we sort out the technical issues. So, my background is, I've been a practitioner running large regulatory programs around the IAM, PAM and non-human identity space for over 25 years.
I've run, more recently, huge regulatory programs relating to non-human identities where we've addressed in excess of 100,000 NHIs, scanned source code, where we found over a million potential leaks. So, I'm now known in the industry as Mr. NHI. My objective, our mission is to educate, evangelize about non-human identity risks. People say, I'm crazy, why are you focusing on educating, evangelizing about probably the hardest problem you'll find, you'll deal with in your career. About a year ago, I decided to create the Non-Human Identity Management Group.
The goal of this organization was purely to help educate, create awareness, create a community so folks can understand and think about how to tackle one of the hardest challenges you ever face in your careers. Most organizations have this ticking time bomb that they're sitting on.
Clearly, from sort of how many people we have here today, clearly, there's huge interest in this topic. Last year, Gartner called this out as the number one identity risk and trend in the industry. As Heiko said, you know, we're seeing a huge number of breaches that are occurring on a weekly basis.
Last year, we reported 40 breaches that have occurred in the last couple of years. So, this is a real and present danger that you do need to get under control.
Also, in terms of my background, I've written a number of white papers, recently published a huge research report called The Ultimate Guide to Non-Human Identities. If you've got a couple of hours, it's a 70-page report. It'll tell you everything you need to know about NHIs.
Okay, before we go on to sort of the main content, we're also running a huge NHI workshop downstairs in room A01, where we've got some great content. We're going to talk about the secret sprawl problem. We're going to get a great insight on how attackers compromise NHIs. We're then going to sort of have a panel session to talk about how to approach tackling an NHI program, what are the top use cases for machine and workload identity. There's a great topical discussion about agentic AI and NHI intersection.
And there's going to be a great final panel session where we talk about can IGA and PAM solutions address NHI risks. So, please, from 12 o'clock, come down and join us for what should be a fantastic afternoon talking all about NHIs.
Okay, I'm going to start off with a real-life story about why it took three weeks to cycle one NHI password. So, many years ago at some financial institution, I get a call from my CISO and head of IM.
They said, look, we've had a big production issue, large operational impact, and we think it's caused by an NHI, I guess in those days we would call them technical accounts or service accounts, and given my background, my CISO said, look, can you help get this under control? Our CIO said to us, look, we need to cycle this password within 24 hours. That was challenging. It took three weeks, so why was that the case?
So, what was the initial root cause of this incident? There was someone in IT that decided to use a non-human identity account, it was a database account, and they thought they were connecting into UAT and accidentally connected to production.
So, the first issue, humans using non-human identities, this is a major issue. If you think the problem around NHIs is an external threat issue, think again, a lot of your staff within your organisation will be using NHIs to bypass controls.
So, as we started talking to the application team, it turned out that it's going to be very hard to cycle this password. Why? Because the password was hard-coded in many places, in source code scripts, a common problem that I'm sure you're all familiar with, and the password was not sitting in some vault solution.
So, we then had to sort of figure out where were all the places that we needed to change the code, I guess, to an earlier question, operational impact, if you don't fix all your scripts and coordinate it and you cycle the password, you could break parts of your application. The other issue clearly was the password for the QA database and for the production database was the same.
So, there's a lack of environment segregation. So, as we started talking to the application team, they said, oh, in addition to figuring out where we're using this password, we think we may have given the password to other teams in the past. We know there's three or four application teams that may be using our accounts, but there could be more.
So, we then had to turn on monitoring, logging controls on a particular Oracle database to see what traffic was coming in, what IP addresses, who was using this account. And we found roughly eight other applications all sharing the same credential.
So, clearly, sharing a credential is another big issue. Again, we then had to ask all of those teams, you need to move away from this shared credential. You now need to move to your own dedicated account to connect to the database.
So, we had to set up new accounts. All those teams then had to move to these new credentials, find all the places in their source code where they were using this credential, change them, deploy these changes.
So, you get the joke, you know, it took three weeks to basically change hundreds of scripts, all because of one simple password that was internally used inappropriately by an IT staff member. Sweet. And then off the back of this incident, we then started what was the largest cyber NHI program within this investment bank. And I spent three and a half years running that program. And it was probably a $20 million plus program that we embarked on about four or five years ago.
So, look, a quick recap, UAT production database, same password. You've got your primary application, right, that is using this credential, you know, same in prod and non-prod. And then you have a whole bunch of other applications that are sharing the same credential.
So, look, a key recap of what were the issues. Humans using NHI accounts, a lack of environment segregation, hard-coded plain text passwords, sharing of accounts, lack of monitoring controls, and then a lack of password cycling were all root cause issues why we had this issue.
So, I hope this sort of resonates with the crowd here in terms of, you know, just one simple example taking three weeks to resolve. And then we had to go on a journey where we had to fix over 100,000 of these credentials.
So, this is a huge, huge, huge problem, you know, something that you shouldn't underestimate. So, as we started our program, you know, we spent quite a lot of time thinking about our strategy, our risk-based approach. When we started, you know, four or five years ago, some of the vendor solutions that have emerged in the last few years, nothing existed.
So, we had to start and do everything sort of from the ground up and build sort of various solutions, processes, controls. So, just going through some examples of what we had to tackle.
You know, the first thing, you know, everyone talks about when you talk about dealing with NHIs is you need an inventory of all your accounts until you know how many you have. You know, you don't know the size of the problem.
Now, our organization kind of was primarily an on-prem based organization. So, getting an inventory of all your NHIs was very, very challenging because you needed to connect to all the different endpoints, local databases, Windows servers, Linux servers, and collect that information.
So, that took, you know, significant investment to implement. The next thing is ownership, claiming.
You know, it's very rare that organizations have processes around who are the owners of these accounts. Many accounts were created years and years ago. No one knows who owns them. People don't want to claim ownership.
So, another common problem. Look, it took us probably three years to complete a claiming process for 100,000 plus NHI accounts.
You know, really, really challenging problem. Clearly, you then need to sort of think about, you know, finding hard-coded credentials.
So, typically, you'll be looking at some kind of secret scanning solution. Again, this is very challenging, you know, looking for credentials. The rules that you implement, you'll find lots of false positives as you find these credentials.
So, even just trying to understand how many there are, where are they in source code. Again, very, very, you know, challenging problem. At the time, we didn't have a secret vault solution.
So, we had to then onboard. We picked HashiCorp at the time.
But again, just having a vault solution is not enough. You need to think about, you know, how do you integrate that into your CICD processes, what controls do you have for onboarding these credentials, your metadata, namespaces, a lot of things you need to think about, about how you onboard your secrets onto a secret solution. We were probably one of the first organizations sort of in the industry that implemented prevent controls.
So, when you check in your code, we actually would check if you were checking in secrets and actually would prevent and block that check-in at a runtime. So, you know, we've not really heard of many organizations that have done that. But clearly, that's where you want to get to is more of a preventative control than a reactive control. As I mentioned, we had to integrate into our CICD pipelines, our vault solutions, and our prevent solutions. Another thing that's quite challenging is cycling. People hate doing that to the earlier point. If you cycle, there could be operational impact.
There could be unknown dependencies. And then also, you need to have cycling capabilities on all your endpoints, be it a database, your cloud, your servers.
So again, huge infrastructure uplift to support cycling. Do you cycle manually? Do you cycle in an automated way? Some big challenges there. And then we go on to sort of hygiene, posture management. Many of these accounts may have been created 10, 20, 30 years ago.
You know, we typically find that most organizations, these accounts will be unused. We've seen some places where 50, 60% of the accounts are stale and inactive. So clearly, that affects the surface area of risk. So one of the biggest things you can do very quickly is try and identify your stale accounts, unused accounts, and just blow them away. But there's always nervousness about, you know, do you have visibility on usage? Do you have monitoring, logging? People are afraid to remove accounts. But the more you can remove, the more you can then reduce the surface area of risk.
The other thing you will typically see is overprivileged NHIs. NHIs will have, like, write permissions when they only need read.
So again, one of the things we did was we looked at the permissions, looked at what access patterns they were following, and we found many accounts that had write privileges were only doing reads. So we then changed those permissions to read and brought down our risk posture and also then reduced sort of risks from a SOX control standpoint because SOX only cares about write versus read. I think one of the toughest things that we had to deal with was monitoring controls.
Clearly, humans using NHIs, clearly risks of, are there any external threat actors that are sort of compromising your NHIs, your API keys. Again, you know, we actually used a vendor, a UAIBA product for solving this problem, but we really just touched the surface because, again, the amount of data that you get from logs, from networks, from all your endpoint devices is significant. There's a lot of false positives. So trying to understand who's using your NHIs, again, is a really, really difficult problem.
So hopefully that just gives you a bit of a flavor of kind of how we at one organization tackled sort of the NHI challenge. As I said, it was the hardest thing our organization had to deal with, and we probably still only solved 80% of the problem. We haven't solved it completely. So this is a lifecycle diagram that we've sort of put together at the NHI management group that just talks about some of the things Heiko also touched on. You clearly need to look at, you know, your provisioning processes. When you create NHIs, do they go directly into a vault in terms of the static passwords?
You know, you need to make sure you've got proper off-boarding processes, you know, as accounts are no longer used. We've talked about discovery, inventory. That's kind of the core. That's one of the first things you need to do. You then need to classify your accounts, who owns them, what types of privileges do they have, how broad is the access, you know, looking for hard-coded credentials, and then do some posture management, hygiene. And then you've got to go on to securing the credentials, monitoring controls, but clearly you need to move to more of a preventative control.
So there's a big dilemma of how do you deal with your existing estate, lots of static credentials, and that's really the root cause issue of most of the problems we have with NHIs, and really you want to move to sort of a target state, architecture, zero trust, you know, just-in-time credentials. I think for anything new that you do, try and move to sort of dynamic secrets, but most organizations will typically have issues around static secrets.
Okay, so what were the lessons learned? Well, this is clearly, as I said, a huge problem. When my CISO initially asked us to tackle this problem, they said, look, why can't we fix this in a year? And I said, look, this is typically going to be a three- to five-year program, and you can't tackle the elephant in the room kind of, you know, quickly. You've got to take a risk-based approach, understand what are your highest-risk applications, what are your highest-risk accounts, what are the accounts that have the broadest level of privileges, admin accounts.
And as I said, look, this isn't just an internal threat. You've probably got a much bigger internal threat issue that you're going to find. So what Heiko said, really focus on people, process, technology. It's very important to make sure you've got clear policies, standards, and controls. Many organizations skip this part or don't really have good controls and guidance. And then education and training is really important. At our previous organization, we basically said, look, if you check in credentials going forward, it's a disciplinary action and could result in termination.
As I said, focus on static versus dynamic. You've got to get that balance right. And then finally, I would definitely say, look, based on our experience, you know, you should look at vendor solutions. A number of them have come into play in the last sort of two, three years, and there's some good capabilities out there. But you need to start first with your strategy, look at your current maturity, what is your tech stack, and then come up with sort of your target state maturity model, and then start to evaluate some vendor products that can help you along your journey.
Okay, just to wrap up, just sharing some key resources at our non-human identity management group portal. We have the most comprehensive resources on NHIs from an independent organization. You can sort of view our ultimate guide to NHIs, our 40 breach report. You can look at all the lifecycle processes, top 10 issues. And then finally, we just launched nhiforum.org. This is a new discussion forum that we want to, you know, ask people to start contributing. If you've got questions, you want advice, that's the place to come now.
It's the Reddit of NHIs, and we actually just launched a competition. We're looking for people to sort of showcase their top NHI stories that they've got within their organization. And at Identiverse, we're going to announce the top 10 NHI Hall of Fame. So come to our portal and submit your best examples of your NHI challenges.
Okay, thank you. Thank you, Halit, for the additional insights. Sorry for that.
And yeah, if you have any questions, please raise your hand. I will ask mine at the beginning already, and you can maybe think of your question, raise your hand, and Alejandro will come around and pass you the mic to ask your question. My question would be around, Halit, you were talking about people, processes, of course, is always a key topic. So when I look at the past, and of course, the past always helps you to connect the dots, right? So when I saw myself confronted with such a task, I had not only to convince the management, but also to convince the developers.
And their argument was, when we were talking about the lifecycle and so on, we can't work like this, you will slow us down, we will not be able to be that productive anymore. And then they had a very good argument in terms of the management, and my programs were killed, or at least stopped, paused, or slowed down. What are your top arguments that maybe your audience will find as well when talking about this topic? What would you give them as advice to convince the developers to join this journey, but not see us as a burden as they usually see I am? Great question.
Look, as I said, look, it's the whole education and the culture needs to change within the organization that it's just not acceptable to be checking in hard-coded secrets into your source code, and it just has to stop, right? Whether you have the capabilities, you've got a vault solution or some integrated capabilities.
Look, clearly we don't want to slow developers down, so it's critical that your DevSecOp teams in partnership with your IAM teams work on integrated CICD pipeline solutions that integrate into your vaults and can generate credentials on the fly, dynamic credentials. So it's very important that the engineering teams across your IT organizations work together and make it a seamless process for developers.
They shouldn't really have to think about secrets is really the target state, and then if you can get to your more zero-trust model, then that's even better where you don't have to think about it at all. So if I understand correctly, you aim to not only to see like coming again with a big governance flag again and saying you are not allowed to do and say be the bad person, but to take them into the process and to make them part of the journey, right? Yeah. Okay.
Yeah, are there any other questions from the audience? Yeah, we have ten more minutes for questions before a five-minute break. So.
Hi, I'm Rob Byrne from One Identity. Question I have for you as a practitioner is what we're seeing with customers, should organizations been making use of platform native security capabilities for these non-human machine workload accounts? I think that's a rhetorical question.
First, I feel the answer is yes, but then the question for us as identity folks is where is the touchpoint with our IAM infrastructure and how do we harmonize it across Google and Amazon and SAP and Azure and all of that, right? Because they've all got different ways of doing these things, right? So I don't expect you to give a magic answer, but I'm just wondering what you're seeing and I'm curious if other people have thought about that as well, because I think it's an important one. I know it's a great question, and yes, the answer is yes, of course.
We have huge hyperfragmentation within the industry in terms of all the different cloud providers, different IDPs, SaaS, and then obviously on-prem, and then with microservices and containerization and APIs, there's just been a proliferation of secret sprawl. Look, I don't think there's an easy answer to this problem.
Look, the industry does need to step back and think about this, some of the stuff that Kupinga, the identity fabric, how we can sort of harmonize identity management sort of end-to-end, is really something we need the industry to come and solve together. It's not an easy problem, and there aren't any silver magic bullets to address what you've just raised there, but great question.
Okay, Lalit, I would have a question. Oh, no.
Oh, yes? I'm curious about your thoughts. When I look at this market, I think we all know that I have a bit of a different perspective on the term non-human, because I think most of it is more workload, but when you take non-human identity management and KIEM, so the cloud infrastructure and title management, shouldn't that converge? Because I would say that one is identity, the other, KIEM, is actually access. So the one, in that sense, is non-human identity, and the other, non-human access management. So wouldn't there be a lot of convergence to be expected?
I think, clearly, convergence is where we need to head overall, even just from a human, non-human standpoint. Everything fundamentally is an identity. It needs to be authorized. It needs access.
So really, I think all these terms that we have in the industry, that segment, certain capabilities, it's all really the same fundamental problem. You talk about API security.
Again, these are just all extensions of the same problem. Look, I guess the CISO from JP Morgan just called out the mess we have with cloud and SaaS. So I think, again, back to the point, the industry does need to step back and think about how do we get more of a converged identity model and just think of identities overall, and even the distinction between human and non-human should really blur longer term.
Okay, we have five more minutes, maybe for one more question. Yes? Andre from NetBI Brazil. I work with, let's say, this new word, you know, identity accounts for a long time. So if you look for a service account, it exists how long? More than 10 years. And if you look, it's almost the same problem. Like HashiCorp exists more than 10 years, and whatever. Why the market is blowing up this new word, no identity accounts, now? And why we didn't take care about this before? Because this problem is a long-term problem.
Again, look, a great question. Look, my first experience in dealing with non-human identities, or we used to call them service accounts, technical accounts, admin accounts, was 25 years ago where we had to cycle passwords for SOX controls. So clearly the problem's been there all along.
You know, it was always the forgotten problem child. But I think really what's happened is the last five, six, seven years with some of the things I mentioned around cloud, SaaS, containerization, microservices, APIs, we've just seen now an explosion, right, of these accounts, these NHIs. I guess there was no good name before, right, that kind of clubbed this together. I know there's the camps around machine identity or workload or non-human. So I guess there's a few camps out there.
But I think it's really because of all those issues and the hyperfragmentation that we have, we've just got a massive secret sprawl problem. And now it's much, much easier, right, for external threat actors to sort of discover credentials, hard-coded credentials on GitHub repos, 24 million in a report that will be talked about in the afternoon. So we're seeing a lot more breaches now. Before it was an internal threat issue. It's now becoming an external issue. And I think that's part of the reason why the focus has increased so much in the industry now.
When you remind or remember 10 years, 15 years ago, how customers or how users of the internal data center secret communication, that was often by firewall, that this server is allowed to talk to this server. That's it. That was the kind of security paradigm for so many. There have even not been service users or basic principles. It was just segmented by a firewall in a data center. And with all of what Lalit said, with SAS Cloud, it changed the paradigm. So basically the problem was there in the past, but reduced and locked in in some data centers. And no one really cared.
And now those approaches just don't match anymore. Maybe one additional question or one closing point with an opinion from your end. We now saw with your presentation these fancy cyclists where everything seems quite well solved. From your experience, from your gut feelings, what would you say? Do you recognize any organizations where you would say, oh, they have a quite good maturity in this regard? Or would you say, okay, this is for the whole industry, something now to start a movement where no one is really there? Just from your experience, from your gut feelings, what you have seen?
I think the maturity levels in most organizations is pretty low. I think unless you've been hit with sort of regulatory issues or audit issues or a breach, typically organizations have not tackled this problem, but it is a ticking time bomb. So there's a lot of work. I think every organization has this issue and does need to solve it because you are heavily exposed. I can just add on that.
So when you look at those kind of breaches and accidents that happened to so many big brands, where you say those are modern companies with a strong cloud-native mindset with great developer teams, and unfortunately they have been affected as well. So basically, as Lalit said, everyone has to do his or her homework. I guess now with the Gentic AI, the NHI problem is going to get even worse. We still have five minutes until Martin Kupinger's coming session on Identity Fabrics. So if you want to grab some coffee really quick, you can go ahead.
Otherwise, maybe one more question. We still have maybe a couple of minutes. Or not to necessarily compete with Mr.
Kupinger, but we've also got the NHI workshop that's starting in A01. So if you want to learn more, please come to our workshop. Thank you.
