Anchors of Trust - Lessons learned from a Ransomware attack

Thank you. Yeah. Welcome to our presentation with our title Anchors of Trust, lessons Learned from Loren Ransomware Attack. This will be the only slide we are showing today to you, but I'm very happy. My name is Matthias Mark working for Vector ai. We are helping customers with our threat detection platform based on AI to yeah, find a attack signal out of network traffic. And I'm very happy that our beloved customer, Maria Flat from idac will share her knowledge and her lessons learned out of their ransomware attack some years ago. Maybe you can introduce yourself, Maria. Sure,
Thanks Matthias. So my name is Maria Floro. I am the IT security officer of E DAC Engineering. We are the world worldwide biggest independent engineering service provider for the automotive industry.
Yeah. Maria, I bet that you can remember better on what happened on Saturday in the morning, two and a half years ago than six weeks ago. Maybe you can tell us Definitely something about this Saturday.
I'm happy if I remember what was for lunch yesterday, so, but I will never forget this day. It was Saturday morning 2021, so it's already some time ago, but probably will remember to stay for the rest of my life. Saturday morning. I was quite fresh in the position as IT security officer even though I was in with EDA since over 10 years. But I was quite fresh. I started in January and it was March 13th of March, 2021. And I got, I got up, it was very early in the Saturday morning and I saw, oh, there are a lot of missed calls on my phone. That's not good. And I saw that it wasn't my private phone actually, because one of my colleagues is also a friend of mine. So he had my private number, which was a good thing in the end. And I called him back, I was like, it's six in the morning.
What is, what is wrong? Do you need my help? And he was like, Maria, we've been hacked. It's real. And my first thought was, yeah, sure. They're just making a joke on me because I started this new position and maybe he had a few shots to shots to many or a few beers, but it wasn't unfortunately. And it took me a few minutes to realize that, to be honest. And I know back then that we have a problem here. When I started in January, I set up, I wanted to go for new processes, for new services, but they weren't there yet. So I rushed in the company, no shower, no toothbrush used. So there was quite a day definitely.
Yeah. Yeah. But yeah, to, hmm, start from the back here. What is the worst thing about this ransomware?
The worst thing about ransomware in our case was actually not that the data was encrypted or that we had to stop the company for, for some time, of course, that that some, that these are can be issues. But our biggest problem and my feeling is that the biggest problem is the loss of trust that you have for your network, for your whole system landscape. You're standing for your, before your landscape. And you cannot trust it in any case anymore. It's, it's completely broken. So for me, the loss in trust was the more, or the bigger problem than the encrypted data because we were lucky, our backups worked. We didn't had really data loss, but we lost the trust in our processes and our services. It's a little bit, you can compare it maybe to, to a burglary. Like when someone breaks in into your, into your home, you may not trust your home anymore. You have to move. That's, that's something you can compare to I think.
Yeah. As I mentioned, we are thankful that you sh are sharing today some lessons learned. Of course we did some months ago a webinar about what you did during the ransomware attack, how you organized and so on. And today we want to look back to all the lessons learned and the countermeasures you bring in place and brought in place in the past months. Maybe you can share some, yeah, some lessons learned what you've done and what you've experienced.
Sure. Let's start easy. So one of the first things that we had to do that nobody ever talked about beforehand was to disable our backups. Sounds a bit strange. The problem is, as I said, our backup systems still worked and they did backups of the, of the corrupted data. So we really fast run it in that issue that the backup systems would maybe override our last known good state of data. So that's definitely some, some lessons learned. Sounds strange, but disable your backup, save your last known good state of data. And the second thing that I always say, it's maybe it's a little bit old fashioned, but print out your details. We had other systems, we had backup systems with the important information, who to call the numbers. But these systems also didn't work. I didn't have internet access, I wasn't able to reach the cloud at first. So what really helped me was that I could rush into the office. I knew the cupboard where the, the paper was, was storaged or was stored. And I went there and got all the important numbers. I got the number of the cybersecurity insurance of the incident handler of our head of it. This paper was, was like gold for me. It's old fashioned, but it worked.
Yeah. Good old paperwork. Yeah. Yeah. Last week in the evening we had in my area where I live, a big power outage in the evening, it was round about half past nine. And I was sitting on in the living room watching tv and then everything went dark, TV went off, light went off and round about 10 seconds later I heard my children's crying, please turn on the wifi. Yeah. And I told my children, yeah, look outside everything is dark. There's no power. Yeah. But you have round about 8,000 children, employees and the company. And how do they communicate? How did you organize that?
It was actually the first big problem that we had to think about. Because if you want to organize the it or even all the administrative teams that you need to, to get together, you have to communicate. And we are kind of centralized, but a lot of them are in different locations. So how can I communicate if email isn't working? My phone lists aren't there. I mean I have the important numbers, but not from, from everyone And our luck or we were lucky that at this point we had a Microsoft Teams test tenant. It was just a small test tenant for the it. So no one else was, was in there. But we already moved our active directory or copy to to Azure ad. So information were there. And so we switched and got all users from just in within a few hours to using teams, which was the only thing working because it was in the cloud. It was not affected by our ransomware attack. So definitely that's something you have to think of beforehand. How do you want to communicate? How can you reach to people and how can you then orchestrate them? What to do? What do you wanna do with recovery or remediation?
Yeah. Yeah. What about the communication with the outside world now it's like we can see the press release of these times because IDAC is publicly traded. Of course there are some yeah. Regulatory messages you have to send out to the world. But what did you do in case of communication with your customers and so on and so on.
That was, yeah, a little of, a little bit of discussion internally. How do we want to to go on from this point with communication to the outside, you always have, you have to tell your employees of course. And then it will, it will go public. So I suggested and the board followed and I'm very happy about that, that we take a very open approach. So we directly contacted with with a post or an email inbox that was on a separate system and also was just a test system. But it worked. We directly contacted all of, contacted all of our customers, all of our suppliers. We went straightforward with the message, Hey, we think we are, we've been hacked. We are still on the way to finding out what is actually happening and we want you to not trust us anymore. So we took the step on on the front and said, we are not trustworthy anymore.
Wait for us until we say it's, it's okay again, you can communicate to us again. And all of the customers, a lot of the customers were so happy about this approach because back in 2021 is, it was more like hiding if you have been hacked. And of course I don't say, yeah, open up your your hearts and tell everyone everything. That's not the approach to take, but be proactive. And all of the customers were very happy about it. And most of them told me afterwards that they even trust us more now because they saw we are taking respo responsibility for our problems or mistakes or whatever. And we didn't lose any of the customers. Nobody got lost on the way.
Great. Yeah. Yeah. I think we, Germans, we are loving insurances. And of course there are a lot of cyber insurances out there. And you told us at the beginning that Ida of course has a, has had a cyber insurance as well. What kind of learnings did you made roundabout your cyber insurance?
Cyber insurance is, is good. I was happy that we had one because with our cyber insurance came the opportunity to get an incident responder right away. So it was just like a hotline. We had to call them. We didn't know who we would get as incident responder. But to be honest, that didn't matter at at that day. And it's good to have one. But the, a aftermath of this, it took us one and a half years with the cyber insurance. And one of the biggest learning I took from that is that you have to documentate every step from the third day on. We wrote meeting minutes. At the first two days we didn't because nobody thought about, we were doing real stuff, you know, trying to, to help the company. And for, from the third day on, we did the meeting minutes that helped. But our insurance wanted to know every step.
Like when was the first server down, who decided to cut the internet connection? When did it of head of it arrived in the building, they wanted to know everything. And we started around half a year after the incident, we started to talking to the cyber insurance. You will not remember that. You are happy if you remember the last three months or six months at all. It's so stressful. So only I thing I can say is take meeting minutes from day zero on. Maybe you can use someone that is not really needed in the recovery. Like a working student for, for example, for our side, he was there just a few weeks. He wasn't really, we couldn't really get him into working with the recovery, but he could take the meeting notes, the meeting minutes and it was so helpful from day three on.
Yeah. Yeah. Let's get back to the title of our presentation. Where does the term anchor of trust come from? Who founded that? Who invented that?
Anchor of trust is a term that actually came from Deutsche Telecom, I think, which was also one of the companies we worked with. And the most important thing was in theory, after such an incident, you could start again completely new on a green network. It's often called green network. So from scratch, something that is really safe and then you start to transition. Sounds like a good idea in theory, but in the real world you do not have enough time, manpower, money to do everything new. So our approach was to set anchors of trust, which were our EDR, our NDR system. We took a look at the tactics and procedures of the threat actor and defined a profile. And if a section of our network or a section of our landscape was not reacting to that profile, NDR was silent for three days. EDR was silent for three days. This was again labeled as the, as a trusted zone where we could work again. So of course this is still a little risk is left, definitely, but sometimes you have to take an approach to, yeah, save the company, to save all these working places. So definitely an approach that was the right one for us.
Yeah. Yeah. Maybe for the last 30 seconds we can do some merchandising for maybe a short commercial for Vector ai. Why is NDR so important for E dac? Why? Why do you count Vector and NDR to your anchors of trust?
So I would say nobody wants to be blind on one eye. So for EDR, for, for the endpoints, you always would go for an EDR system, you would use an antivirus or whatever. But within a network, I didn't want to go blind either. And we also, from the tactics and procedures from the threat actor, we saw we could use that to implement it into the NDR system and be sure to see if something would come up again, if we see a traffic and communication that was probably malicious. So it was really fitting in our plan of these anchors of trust. And the second part was Vectra was just easy to install. We, the central hardware we needed was shipped within, I don't know, two or three days. So really fast in, in that kind of incident. And then for all as fastest as
FedEx. Yeah.
And for all international locations, we didn't have to ship any hardware. We could use the virtual sensors that can be deployed in just a few minutes. To be honest, it was so easy that we said, okay, this is the right solution for us now because we do not have time to just set everything up to go deep into the settings. And V helped us with, with the implementation. That was just the red partner for that time.
Thanks that you are, yeah. Investing your time today in our presentation and yeah, maybe we have some questions left from, from the audience then we can
Make them. Yes, absolutely. First of all, thank you very much. And of course the round of applause, which I believe is to totally deserved.