Stefan Würtemberger: In the Crosshairs of Cyber Criminals – A Case Study by Marabu

Thank you. Yeah. Welcome. Thank you for, for being here. I want to, I want to introduce you a little bit about the cyber attack. What happened at Mabu in November, 2019? So I'm started at Mabu at October at 19, and yes, we had a bad cyber attack. And so that's what I want to introduce you and show you a little bit. What we can see is that cyber criminal or cyber crime grow up in the last, in the last years. So here a little bit figures and it merged from, from spread attacks to, to target attacks as happened at our site. And we see that the cyber criminals had to, to change their, their targets from, from big companies to small medium enterprises. Well, the, the, the security, there is not that big as in big companies. And, you know, the resources are not there so much.
And that what happened to us also, you can see in the, in the German area where we are since June, we had a lot of cyber attacks, not that big companies. So we have also a big company nearest to us built, which was also hidden really big or, or governments are also really, really hard headed. And yes, when we look a little bit in our case, see there, we, we were hit and almost all service and infrastructure will be offline for long time. And so I want to show you a little bit the Chronicle, what happens and which malware where hit us and what we have learned about our cyber attack. So when we look at day one, it was the black Friday. So everybody was online, shopping, not, we were, we were been heated at half past four in the night, and the virus were rolled out worldwide on all our service.
And there we saw in the night, nothing at the morning as the production started, they reported some mail functions and not availability of service. And then we investigated that problem. So the first time we thought that some virtualized service are not there, or the hosts are, are down. So we tried to switch from the service as we then saw that, that we have been headed by a ransomware. And then we decided to start to shut down all, all infrastructures immediately, which, which was done at 10 30, we are shut down all systems disconnected, all internet lines, so that we are completely separated. You see, we have almost 200 service, 180 are infected. And so we have not, not services from the it, we started immediately of emergency planning to come back online and organized some security consultants, which, which then assists us to, to analyze which malware has he us and what we can do to be back online.
Also, we have lost all backup service. So we started the installation of new backup service to, to bring back the, the backups. And also we have then our telephone conferences with the general management in our countries. But the situation was that we are not able to, to bring back at Friday the it, so we decided then at 2:00 PM to send our employees home and use the weekend to prepare that we are at this point, we back Monday morning online so that we can work. So in our case, it started to decrypt all service in, in 10 seconds in a row, started in Spain, which was the infected main server. And then it spread out over Europe and then it goes over into the world. So all service, all systems were wide. Reds are where we are working was in a time lapse, six hours, 23 minutes to encrypt 90% of all infrastructures, all data, all databases, and that occurs that nothing will be work.
So the second day was analyzes of the, of the, of the malware to be, to be, know what the, what the starting points to recover the systems. At the second day, also service providers began the analysis of the damages. We have our first crisis summit personally, to, to close all plans the next three days. So Monday, Wednesday, and Tuesday, Monday, Tuesday, and Wednesday. And also we decided to report our incident to the police. And yeah, we have initial situation assessment from the police and from the land criminal lumped in, in Germany. And also I was not in Germany itself. My flight from Spain was at the, at the evening. So I, at the first of all, done all with my handy and I also have the crisis management remotely to do, but the result was at, at Saturday evening, everything is encrypted. We are not be online in the next couple of days.
So as I was back in the office, we make a priority planning, analog. We have no it infrastructure. I use my, my office to prioritize what should be done in the next, in the next hours and in the next days, and what are decisions to be, to be done. So we start to get back online, the active directory that we can log on. And we also start to clean all clients, what we have 580 clients and machines, industrial PCs, and so on at Saturday night, day three at half past 10, we have the first success that we are back online and can work again. And then we have two more prioritize that the E P systems for the production sites coming back. And we have then also defined information policy for the employees and for, for our partners and customers. Also, the decision was there on all planes are closed worldwide.
So they for, we have the further systems coming up, start the communications with our subsidiaries. And also the police arrived with a lot of investigators and the team which support us psychologically. You are in trouble when you have that. And that's not, not, you are not happy at the first days. And also you want to, you know, what the criminals want to have. And, but we also, we decided to not go in negotiation with the criminals and we prepared everything, but we thought we should be trying to come back without paying any money for the criminals. So that's all, all. So our internal regulations that we are not pay any money to criminals. So then the days go up, we have updated our fiber infrastructure, we set it up new functionalities. And then also we started the first internet connections to get, to get back online with our subsidiaries.
And we started more power to bring back E P systems worldwide. So you see from there, it's not, it's not a short time event. So we are now in day six. And the first goal was E P systems Bri back online. We have had the target at day six, so we can produce, and the first shipments were going out. And that was a big goal for us, but it's really hard. And you have to work careful while you see, when you install a system here and bring back some systems. There, you have connectors, you have to make updates. It's not that easy while you're not in a normal operation. You are in a, in a extreme situation. And when you then pitch something on the other side, some other systems will not work again. You have to fix that. Also day seven, 60% in Germany are back online. Our phone system is back online. We can our phone again and email, and we realized, then we have lost a complete EAP system in Spain. We lost all file service in Sweden, in Austria, and we have lost, complete our outlook and our mail service.
We then moved from on premise to the cloud. We migrated all mailboxes to Microsoft, oh 365. And at we two, we are with 95% of all our mail addresses back, online, lot of work. And then at the we two, we have then our countries back for Sweden and the us we're back online and have 60% of operational work. And then also we started to restore the data. We have lost 95% of all our backups. And then we started with new weeks software from Australia, with fire car to restore our data, which was really successful. We have had a lot alone with this software three and a half terabyte of data, and also restored OST files, migrated them to PST and migrated them. These PST files to oh three, five that we have our email history back and that with 300, 520 mailboxes. So you see a load alone.
This workload is really heavy and it's not a, it's not a sprint, it's a marathon to bring up a whole it back to normal operation at week three. Also, we have to prepare our E E P migration, which was planned for the first of 2020. So also this operation started migrate all data from the EAP system to the new EAP system. Happy was that the new E P system was not infected by the virus. So that was the big goal to, to come back and go further with this, with this project at week four to eight, we had a lot of work for the detailed restore things. Also the start of the EAP system was to be migrated the old data from the, from the former E P system, the, the new EAP system to launch for fraud and Germany, the new AAP system at the 1st of January, also outlook restoring of the PST files scoring on and for 520 mailboxes to convert OST to PST and bring, then this PSD filed back to office 365, took over nine weeks.
That's also a huge, huge, manually process. Also we have to configure a lot of services and detailed communications to the systems, which is not that easy. Also documentation we have had was really good, but not in that detail, you needed in such a situation. And also more processes were in growing. And then we have had a big, a big lucky day. We found the last old storage system, which contains the Sweden file server on it. We lost the data of three months, but that was better than to lost the server forever. So we restart that and Sweden is now normally operating and they have to work on the last three months data and bring excellent work files back, which were also have had mostly in outlook and sended emails after nine weeks. So end of January, we have stock fire car while we have had all datas back. 95% of all outlook migrations were successfully done, and we are normally operate in a normal, in a normal business. So that was a really, really tough week. And also the, the complete cyber attack was really, really tough, tough issue for us.
We find out what was, was heed us was mini cuts, which collects all credentials of administrators. And then we have Trion double payment, which is the encryption file, encrypt ransomware. So it's, it's really tricky when you have that on it, it, it covers itself as a service process and start to encrypt service. When you have a lot of data, then it took a while. So that was really happy for us from the file service to be, to be not completely encrypted till we have had them shut it down. So we lost partial portions of data. And at all, some in some countries, the situation was much, much dramatically. So you see it, it's not that easy when you have that. No clients were infected by the, by the ransomware. So it was not that that, that we have to reinstall all PCs that we have then afterwards done deploy a new image and everything.
What we have learned from our case was you have to, you have to call specialists, which are, which are, have the expertise and to protect your own resources, their own resources, normally all organizations to, to have the knowledge about the systems not doing itself. So we were normally a team from 10 in the it we have in the nine weeks, 25 in it. So 15 external resources, which support us in the crisis from our side, I can say report incident to the police. They have a lot of knowledge to handle that issue and to proceed and to communicate with the criminals and also to have psychological assistance, which you need in that Christ then prepare, you see nine weeks, you cannot handle one, 100 hours a week in that long period of time. So plan free time plan, everything that you can hand over to the, to the long time event.
What also helps is a well infrastructure, documentation, and process documentation, which helps us that we have really good documentation over our infrastructure. Not that detail we have needed, but we do that afterwards and document also the, the APIs and the communication under the application itself, not the installation of the application, good network specialists are the backbone of any cyber attack. The virus comes through a network and we have to find out the leaks where they are coming in and how they, they use, how they use the network itself to operate. We have windows service with RDS and not patched systems. So over that they come to the network and have access to all our service via remote desktop. A good emergency plan is also necessary. You can handle that before you have a attack. When you clear who is responsible for decisions who is responsible for communication and how to handle the it department communication internally, who can write the it, who can call the it. When you have such an such an incident. Also, what we have learned is to store your backups. Not only in it, we have also make paper backups that we can work without it five days. And so also that's was, was really, was really good for us. And what we can say, we are safe. Can you not say we are sure nobody is 100% protected, and that was what I have to tell you about our cyber attack. Thank you very much.