Conquering Crisis: Effective Strategies for Incident Management

So welcome to our panel. We will talk about effective strategies for incident management. We have three great speakers and experts here on the stage, and maybe we start with a short introduction. Maybe Aris you start.

Hello, good morning everyone. My name is Aris COOs. I'm a technology strategist at CrowdStrike. Most people know CrowdStrike as an E-D-I-X-D-R platform or threat intelligence company. But in fact, we are a leading instant response company. A lot of other firms are that are doing incident response at scale, are using cross technology. So I'm here to basically share a few of the insights, a few of the learnings from the IR perspective. And I wanna lead up with a statement if you, from Benjamin Franklin, if you fail to prepare, you plan to fail. That's a good statement.

Mark You to quickly think about a good statement actually. So good morning. My name is Mark Wolfman. I'm the Chief security officer at Noir and I've been asked yesterday, what the hell is noir actually, is it a FinTech or something like that?

Actually, Nodia is a universal bank in the Nordics, so Scandinavia and Finland. And we have around 30,000 employees. Before that I was the, the, the CSO at Swift, which by the way, on the incident perspective was super interesting with all the payment fraud incidents that time. Okay. And I have no statement actually for you, unfortunately, Still 80 minutes, 18 minutes left for that.

So good, Good morning Andre Vallet. I had a cybersecurity for Vodafone business.

Vodafone, as you you may know, is a relatively large global telco, about 315 million subscribers around the world. Quite a, quite a few of them in Europe, quite a few of 'em in Sub-Saharan Africa and around with Vodafone business. We look after about 6 million organizations across Europe, ranging from the largest governments all the way down to the smallest sort of soho and small businesses. So I think, you know, understanding this and that, you know, I dunno the quote, no plan survives being punched in the face, I think. Right? So any incident, any, any response, any crisis, right?

Whatever you think you've planned, whatever you think you've understood, you know, will change in that first, you know, first moment as you really understand what's happening. Perfect, thanks.

Okay, so maybe we start with some really hands-on thing here. Like, given your experiences, could you share a recent cybersecurity incident maybe in your sector and the key lessons you learned? Maybe starting with Mark, I must not talk at concrete incidents at Node, obviously, but perhaps I can talk about one case, which was the, the Bank of Bangladesh incident. Perhaps you, you know, this thing that was in 2016, a payment fraud.

The, the Bank of Bangladesh SWIFT systems have been hacked at that times. And actually the, the threat actors, they were very, very diligent in their planning. They've had a reconnaissance phase, which was about nine months or so. They've been in the bang of Bangladesh and just observing and finding out actually what are the, the payment operations, who's in charge for what, sniffing around and doing all the reconnaissance work actually. So plenty of time to detect them actually, and do then proper response on that. But the Bank of Bangladesh failed at that time.

And when they finally found out, so when the threat actors launched the fraudulent payments 35 in a row, the, the reaction from the response from Bank of Bangladesh was super weak because they elect all preparation for that. And just without going into all the details, because if you, if you digest this, and I think there's a super BBC podcast series on the, the Lazarus heist, which is super insightful on this case, but you look at the details, then you find out preparation, preparation, preparation is super key.

So give you an example, when the director in charge on the weekend in Bangladesh found out their fraudulent payments, which went to the Bank of New York, actually the Federalist Reserve in New York, sorry, not the Fed was it, to transfer funds from the Bank of Bangladesh account to somewhere else on the Philippines. Actually he has had no phone number in New York to stop the payments. So he actually was then forced to use the phone number, which was publicly available on the webpage from the Fed, which was monitored in business hours and working days. I think about preparation is key.

Yeah, definitely. Andre, any experience if for sure, any experience, any experience you can share with us?

Yeah, well, and actually I, I, it's always difficult or or to talk about incidents that you've experienced, but we experienced, you may have heard a, a significant and major attack on our Portuguese network in February, 2022. It was pretty catastrophic. It took out majority of our fixed mobile networks, SMS streaming TV services for over 4 million Portuguese citizens, right? There was an immediate impact on blue light services, on banking on NTM networks. And that was pretty, that was a pretty significant thing for us to take out.

You know, a huge portion of a country's communications network really, really brought it home for us. And I think, you know, you talked about preparation, preparation, preparation. That preparation allows you to do a couple of things really. I think that, that for us were the huge learnings. The first was being able to operate at speed, right? When the network goes down and everything in a, not everything but 4 million citizens in the country and the network goes down. Being able to understand and operate at speed means you have to have the right people in place.

Second big thing was for us was the communication. And I know a lot of, you know, us and other organizations sort of admit nothing until it's a little bit too later or until you're legally or regulatory obliged to admit things. But we made a very clear conscious decision to step in front of that and communicate very, very clearly and transparently with both our internal stakeholders and teams, external stakeholders with government. We put our Portuguese CEO on television immediately to talk about the disruption was occurring and, and what we were doing about it.

And then also to communicate very clearly with our employees and the partners to help, you know, rebuild a, a network. And we were able to do that in just under or just over 48 hours, which I think was a, an an immense piece of engineering.

And, and a lot people worked through that. And I think the, you know, the, the final, I know the, the final key learning for us was alongside that, you know, speed of execution, absolute clarity and transparency of communication and continue to communicate. And we had an amazing response from, you know, the Portuguese regulators from the government, from citizens, which was, oh, we understand what's happening. You've told us what's happening and we can, you know, actually our, our NPS scores went up as a result, which is, I think people care when you talk to them.

But the, you know, the, the third, the third really important takeaway that we, you know, that we focused on was the impact, not the impact on our networks. You know, it cost us about 30 to 40 million euros to rebuild those networks that sort of, you know, as some that, you know, to, to many would be massive to us was, you know, not, not significant, but it was the impact on our partners, on people's lives, on their ability to speak to loved ones, to, you know, to access healthcare access online services.

So it really is speed of operations, clarity and transparency of communication and then understanding the impact, not necessarily just on you, but on your, you know, on your, your suppliers, your partners, but also end, end customers. Perfect. Thank you.

And Aris, anything to add from your experience from your customer base? Obviously I cannot speak about our customers because all the big breaches you see typically CrowdStrike or a IR firm is involved that is using CrowdStrike technology.

Now, there's a few learnings here, of course, right? We spoke about preparedness.

I mean, one area that we do quite well is tracking adversaries. One of the learnings is of course here that adversaries are getting faster and faster. So the time it takes on adversary from entering on an organization, the initial access all the way to moving laterally, and this is basically the best opportunity to stop the attack or the adversary in its tracks that is down to 79 minutes on average, right? It's not about mid tempera detection, but I'm just saying that you need to have the right tooling in place.

And when it comes to preparing things, I totally agree with everything that Judge Matia said. It's very important to do exercises also prepare the leadership team. What I mean by this is if you have, you need to have these crisis or incident response plans in place and practice them.

Well, one thing that I, that's, that basically I remember quite vividly is that if your organization and typically adversaries are trying to get hold of the active directory infrastructure, the keys to the kingdom, or they're entering with valid credentials, if they have access to these systems, you cannot rely on your normal communication, which means you need to have out-of-band communication in place, right? And then of course scalability is a big thing. First of all, tooling to do enterprise scale incident response.

If you have a thousands of machines basically compromise or potentially compromise, it's gonna be difficult to do traditional forensics, right? So that's one area. Scalability. And then also thinking about the ability to basically contain, remediate and having the right supply partners in place. What I mean by this is if you're not good in crisis management, find somebody that can help you on a retainer, the legal team, but also our companies that can help you to basically scale up to 24 by seven, follow the sun.

Because the first thing you're going to notice, and I like the Mike Tyson quote earlier, the first thing, regardless how much you prepare, people will burn out quickly. Perfect. So we heard about multiple times and fully agree preparation is key. And also know how to reach out, setting up a communication strategy, internal, external also with the media, depending on how this affected giving you experience, mark is there for companies in general, is there a perfect strategy for incident management?

I mean, the topic of the panel is effective strategies for incident management. Is there something like a blueprint or that would, that would you recommend to the people here in the audience, this is how you need to do incident management besides for sure preparation or how does this really look like? I would, I would say there, there could be something like a checklist of things you need to think about, but at the end, it depends a little bit on your organization. It's complexity how incident management and crisis management actually work.

But I think you cannot stress enough the preparation topic. So what, what we did, we started two years or three years back to do tabletop exercises in our organization where we prepared what, I think it was ransomware scenario. We started with, so from first infection onto all core banking systems down with a significant customer impact. And there was such a, such a significant learning story because we realized actually how many things are not yet prepared and we don't have the time to prepare this now.

So usually I say it's a good thing to learn from experience and from, from practical things. However, for incidents, I think it's a very bad idea. So you need to, you need to prepare it upfront. So I'll give you, give you a couple of examples on my checklist. For example, can I get the relevant people into the room in my crisis, a global crisis management team meeting, actually, do I get them online? So can I communicate to them if our systems are down, do I have their latest phone numbers? Do I have the relevant decision makers?

So that means from present media, from the board, from the business lines, if we are up to making a, a difficult decision. So taking certain systems down actually to protect ourselves from the, from the malware to spread from it. Obviously you need to have the regular updates from the, from the first response team and so on and so forth. And then we found out it takes an, an enormous long time to decide on what do we tell to the customers now. So there's, and then we said, okay, let's prepare this scenario and the different potential developments of the scenario and what do we communicate.

So starting from, at this point in time, we have no indication that we would've been hacked. Yeah. Down to a customer data is not affected down to pants down.

And, and be super honest and then be prepared under which circumstance are we taking which decision? And I know we've discussed that briefly yesterday. There's a super difficult decision, would we pay a ransom? And my immediate answer on this is your official answer for no idea. Of course we will not. Yeah. We are not finance the bad guys obviously. But I've talked to many CSOs and actually, if all your systems are encrypted and you have no other chance, then you are tempted actually to do that.

So the question is, if you consider that for your organization, you need to think about who is taking this decision. So is it the CEO together with the chairman of the board? Who's taking that decision?

Okay, then you need to define this at the beginning. That, that they can decide on that next question is how do the two communicate, if not in the same location? Would you let them have this discussion with a normal cell phone call or with a teams call? I would not. So you need to have something like a secure communication channel for them in place for such things, which must not go to the press afterwards and not talking about the Bitcoin wallet.

Yeah, definitely. And I know Martin, I think it was two years ago at yes. He also answered directly the question of would you pay the ransom? Definitely was No, but the question is, if you're not able to work anymore, then maybe you should, Maybe, I fully agree, I would not pay the ransom. But from a business perspective, what people told me is maybe there's no business left. Right. If you don't yeah.

Don't, if you're not able to, to recover quickly. Right. And that's a business decision at the end of the day, not something that crowd track promotes. Definitely.

So, and and I you, you talk about business decisions and Mark, you, you talked about having how you make a decision as a, as an organization, as a business. And I think that's a big part of the preparation. We are probably most efficient right at first response at root cause analysis, at instant containment, at understanding, you know, the technical aspects of a, of a bridge. What I think we don't understand or have or very seldom give the context for is how to make the decision about whether to pay a ransom or not.

How to make the decision about which customers to communicate with or not, right? It's the context, it's almost a decision tree to help a senior team or a board make those decisions. 'cause they won't have had to make them before in anger and to balance legal advice with regulatory pressures with all of the other aspects. And we shouldn't expect them to understand the technical nature of the attack, but they should understand and be able to really clearly right, define that business impact of those decisions they're making.

And in this, unless we, you know, we train and help people about how to make those decisions, what are the critical data points? What are the thresholds? Why are we making a decision in a moment of stress and, and how do you do that? And what are the rules around which we communicate? I think that's really critical for a, you know, for a significant and catastrophic incident that has wide ranging impact on your organization and others. Definitely. Anything you want to add here? Yeah.

In terms of, of checklists, in fact, we released a leadership checklist when it comes to a framework on how to prepare, what to do during an incident, right? And these are aspects also when to invoke or communicate with insurance, for example, right? That basically can guide you through this. And the other important piece is always the lessons learned. What did you learn from this incident?

I mean, all these incidents do not have to be catastrophic, but typically there is a lessons learned and there should be a continuous feedback loop where you basically try to optimize things and do things better. And again, tabletop exercises, even strengthening the, the, the blue teams by doing red teaming, emulating, adversaries, right? These are things that can really help to basically at least understand the scope of the incident and understand also what it is you need to do from a technical perspective before you then engage with all the other stakeholders. Absolutely.

If I make quickly, I think you have a good point. I think at incident management we are pretty good, right? When I talk about preparation then, I mean the, I mean the escalation on the crisis management level.

Yeah, absolutely. Okay. So given that this panel and here in the audience, any questions? So the chance to ask maybe Wick you can, or can I do it that faster here? This is a, a question about the, the communication when, or communicating when your normal channels have gone down. I heard that the example of Vodafone or Portugal, h how did you communicate? Did they have a telephone of the competitor that they use something else? I'm very curious.

It it, it's a really good question. And what we're able to do as, as Mark said, outta the OUTTA band and, and different communications channels put in place for those, for those moments and elements of sort of the yeah, the traditional fixed network still work at that point.

But I, the bigger you say the, the bigger point is, you know, how do you effectively and efficiently communicate? Who'd you get in the room? How'd you get them in the room?

And the, and then how do you facilitate that conversation, right? You can lose, you know, minutes and hours in discussion where actually what you need to do is make a decision and move on to the next stage. When the Ukraine situation started.

Yeah, we went through all the potential scenarios and even the worst case scenarios and we bought satellite funds for the key role holders for this situation. Definitely. Perfect. Given the time, and I would love to discuss this hours more and probably in the audience as well, what is the most important thing in one sentence, and not only use preparation, please to make it a bit difficult in one sentence. Maybe we start with you. I was hoping I go last, ah, Last I saw this.

Yeah, yeah, Yeah. I think having, basically, I'm not saying prep being prepared. I think we, we, we hammered on this point quite, quite heavily, but just making sure that there is trust in that community. There is basically an understanding of the stakeholders of each and everybody's role. And then there is also an understanding when you start to need help, right? Yep. Because as mentioned before, a few things that I learned, I was there when the SCCM wasn't, was smashed so people couldn't deploy short, please. Okay. Very short.

And, and then again, if you need help prepare this during Christmas time, normally instant response relies on humans and very skilled humans. And we have a lack of humans in our industry, which means get a retainer in place when you can and use it. Then if you don't need it, use it for, yeah, for, for, for tabletop exercise and other things. But use this basically to strengthen your, your team and you can then invoke it when you need it. Right. And then scale up. Perfect.

Mark, if I must not say preparation then as I say, training, training, training, and Learning. Now seriously, I think, I think you need to have, this is a bank wide thing for us. Yeah. So it's not just for a few specialists in the techie team. So when all these senior stakeholders and role holders actually need to be involved and know what they're doing, Communication absolutely key, right? Communication and coordination will be the single determining factor of how efficient your response is.

And I think if you get that right to stakeholders, to employees, to partners, then you have a chance of achieving what you want. If you don't, the whole thing will fall apart. Perfect. Thank you very much. Mark.

Mark, Andre, and ris.