Lessons Learned: Responding to Ransomware Attacks

Yeah, my name is Stephan Rutenberg and I'm the CIO of Mara Boeings. And I want to give you some impressions of, yeah, fighting against ransomware attack. We have had 2019, if you see this then it might be delayed. So all your things in preparation, prevention, detection has failed and yeah, the hackers or the ransom is in the company and it might be the result can be that your company can't work anymore. And there are four. We have had a lot of circumstances to to come back online and the next next six points should be show you a little bit how to, how to handle a cyber ransom attack.

So first of all, it's necessary to get an overview. How heavy is the spread in the company? How many systems affected, how many servers, clients, OT infrastructure did you have? And that's the first impression you have to collect and to to do everything that you have a big picture. That's then for the later things to to do. The next question is what could it be? What kind of attack has you been hit at the first beginning? You see the files are encrypted but you don't have any kind of information which exact virus it is and how to fight against this virus in the company network.

The next is when you look at question one, how big is the spread? The second thing is how far to detect me or spread in in the company. So is it only a DM set? Is it only a system or is it companywide? In our case it was globally. So we have 18 locations worldwide which are affected in the ransom attack. And so we mainly have to shut down everything. Once you have the overview, the big picture, you should be roll out the emergency plan and create a crisis team which can make decisions and decisions has to be done immediately.

And who can in your company decide to shut down a whole company worldwide for several hours, days or weeks? So this crisis team is elementary necessary to be built that you can make the right decisions in the right time. From my perspective, you will do a lot of decisions in a time of a cyber attack, but you will do also a lot of wrong decisions which you later have to rooke and it could be expansive when you do it, when you don't have a big picture, what have you hit?

So if you have built the crisis team, activated an emergency IT plan, which then describes who has been involved in a cyber crime incident, what should we do, what have to we do, which are the activities, who is coming next? That all has to be done quick as quicker you are just to that last spread you have in the company. So in our case it starts half past five or half past four in the morning. All systems were shut down until 10 and we lost 95% of our IT infrastructure. So as faster you are, that slower or more intensive is the cyber attack.

So, and then if you have global spread as we have, then you need help alone with your team. Normally it's not to be done. To get out of this ransom attacks, you need forensic need analysis, you need consultants to be shut down or recover everything. So that would be also the first things you have to be clear, who can you get in a, in a crisis as from my knowledge is that when you have crisis, other stuff has also crisis how far and how fast you get the help. So if you know how far they are come in the company, then it's the question isolate and let them work or shut down.

Isolate is better than you have for the forensic team. More data shutdown is more efficient other than can't work and they can't spread anymore.

So that's, that's a kind of the decision you have to take when you have, when you have the data from the first one, when you have the overview. So we have done the second thing, We have shut down everything, which was not the bad dec, not the best decision while we lost a lot of data, which the forensic has needed to be figuring out which way the hackers has been taken to come into the company. So when you have isolation then you can contain the spread in this area, but it will won't stop.

When you isolate systems, it will go further and further and further data can be come out of the company. Also the internet connection has to be separated that the commanding control networks are not longer communicating to your systems. And also a big thing is, as I told you, be fast again, also the fastness in cyber attack is the most important thing to do.

Fast, fast, fast as faster you have shut down isolated in an overview. Then the recovery and the online time to bring the company online would be much more efficient later on. So then if you have a shutdown of all systems as we have, we have to prioritize what is needed first, what's second and which processes we have seen that before has to be online and how can can they come back online? So therefore rank the systems, rank the processes in your company that you need to be operate.

Yeah, that can you do before you have random attack. In our case we haven't had that so we have to do it parallel. And also if you have to be fast, that's a a slowness when you then have to prioritize which systems, which processes has to be rolled out back the structure as clearer the structure is in this recovery and prioritization that's more efficient. So if you have also good documentation of your business processes and the relied system to your business processes also that more data you have, then you are quicker to restore the whole infrastructure of your company.

Also then when you have a structure and the overview, you as a CSO or a crisis manager, you don't have to work at the systems, you have to overview and manage the crisis. So don't work with the staff, let them work, let them make their own decisions but keep an overview that all the prioritizations you have made are in the right shape and coordinated to be done. And also keep an overview which tasks are done, which task has to be do done in the future.

And also a really helpful thing in our case was communication internal and external communication as best you as best you communicate internal and external that more the people know what's in the company it's going on. So when you have a of a globally white IT systems, then the people are frustrated, they're frightened about how can I work again, will I work again? Did we have to close the company and that therefore it's needed that you communicate clearly and structured what's going on, what's happened, what hit us, how far we are, the process of recovery.

So if you have done then all the informations you have no, you have knowledge, what have you hit it, you have close to communication to the commanding control service and you know what's happen in your network, then you can start recovery. And also recovery takes longer as you know while all the systems which are relied to business process are there, but the settings and some some processes, communications between systems could be affected. And also did you have backups from all your systems, which we haven't had.

So we have lost some things and we have to recreate them by the knowledge why we have also not have a documentation. Also recovery takes a long time. So as long backup takes, the recovery takes the same time to restore from tape, from hard drive or from the cloud. But the configuration and the reliability for the systems and the communication between business processes could also be take a while if you have recovered one system but the second system it's needed for the business process is not that far in the recovery, then you have also problems to get and deliver a full service of that.

Yeah, and then also the documentation is really informational or necessary. Did you have a documentation of all your business processes as I recommended before? While the environment is not only a system, it's not only a software, it's only the, it's also the knowledge of the people which are operating the business processes and if the log is not working while you have a missed a file which is not there and not recovered and then you can log onto the system. So after that we have then had nine weeks to recover all of our subsidiaries and all our networks.

Then the big and the huge project starts while then you have to do the rework. So I have told you if you see the message in the first page, then all your, your things in cybersecurity has failed and then you have to to ask yourself why it's failed, why have done no preparation, why have done all my prevention has failed, my detection has not shown up that the hacker is in the network. So that's also a real huge project. Then to recreate the whole network and all of the, all of the systems surrounding in the security also you have to rethink and we have done that.

We have rethink our security concept. We have made huge errors in our security but it was a time to then to rethink what we can do better in the future that we are not be a next time victim of a cybersecurity and cybersecurity strategy is then also important while we sw that from the IT CSO to the management board. While it's necessary that the board and the CEO has a overview of cybersecurity and strategy and cybersecurity strategy, don't start at it. It's an organizational thing.

When new employees come into the company, they have to know how the security rules all of the, all of the things are working in the company while we hitting them with thousand of logs, multifactor indication, thousand messages, awareness trainings.

That's hard to understand when a new employee comes to the company, which was not part of the first cyber we have had that he has all the knowledge about how could be cyber attack ransomware and for a company and therefore we have had a lot of communication also done to the new employees awareness trainings onboarding to be understandable how important cybersecurity is for a company. What we have learned cyber security strategy is now port thing, not IT thing I'm the technical advisor of the board, but the decisions has to be done from the board and from the management.

That's also make pressure off the organization that when they want something, they are also going behind the management and say okay, let's do it. A restructure of a security concept is that heart while you have a compromised network and then you restructured by operating that compromised network and yeah, so you have a compromised network and then you restructuring it's hard of work and the heart of things. You won't have outages when you build the new securities concept and new systems. Also we have had a lot of opportunities with our cyber tech.

So we have lot of colleagues which have this legacy software. So the H has eliminate all of the legacy software at our case. So we have to rethink software new. We used it to go into the cloud, but the cloud is also not the wonder web. Also you have to secure the cloud as it's described in your security strategy. You also have to be communication from on premise to the cloud. That has to be secured and that's only informational thing from my side, don't forget to be thinking security in the cloud. It's not built in.

It's also the configuration from your site and adaption of your organizational needs, which the cloud provides us. We have had after our cyber tech 653 sub-tasks to reorganize the whole infrastructure. We changed nearly everything. Every window server, any system also in the facility in the production would be replaced step by step. So that was be the huge approach as the cyber tech itself. We have taken I guess three years to replace all of the IT stuff for new secure and cloud based systems. All what I can say is say never, we are safe.

100% security is not payable, not reliable and not makeable from my point of view. So thank you. That was the journey and the points I want to tell you. Thank you. There are a few, there are a few questions already in the, in the online part, so I, I choose one. How strong was your business involved when it comes to the incident handling and was business continuity part of your rework and also of the lessons learned?

Yeah, absolutely. A cyber, a cyber attack is not only IT stuff, it's organizational thing. So also operations, sales, production colleagues are involved in a cyber attack. You need them also and they have also good ideas to change processes after the cyber tech or to come back to work.

And yes, business continuity, as my boss says, we have now a target five times without it operational as a result of the cyber tech. Right. And a question maybe from me, but I'm really interested in that. We always say, okay, have a plan, prepare, have a backup, have a have a strategy and test it. How often do you test now? Twice a year. Twice a year. And it's faster than the original recovery?

Yeah, yeah, absolutely. We have had in 20 20, 21 second cyber attack. The first one took nine weeks, the second 1 48 hours. So we trained, well. We have a lot of new technology in so we, we saw it much more efficient and and better. Yeah. Great. Thank you very much. Final chance for your question from the room. Otherwise we let him go and introduce the next one. Thank you very much Mr. Wittenberger. That was a great presentation.