Maintaining OT and Worker Identity Security in Disrupted, Degraded, Intermittent, and Low-Bandwidth (DDIL) Environments [Advanced]

All right, as stated, I'm Jon Lehtinen, and this is Maintaining Operational Technology and Worker Identity Security in DDIL Environments, and you're going to learn some fun acronyms. We'll get into that in a moment.

So, the thesis that drove this exploration and some of the work we've done has been that the internet doesn't seem quite as resilient as it once was. So, you could argue why that is. That's not the purpose of this talk, but you cannot argue with what has happened. I was in Canada when the CrowdStrike incident occurred, and I watched the Rainbow Bridge back to the United States get so full that the entire town by Niagara Falls had all of their surface arteries fully locked up.

And then when we left for the airport from Buffalo, every single kiosk had a broken Windows 11 screen saying, sorry, we can't help you. It was tremendously impactful. Similar with anything using, you know, a handful of issues in AWS with the identity controls, we've seen, you know, admittedly very infrequent, but when it happens, folks who have built on that platform feel it. Same with other providers managing some of the higher-level components of the internet. And I want to be very clear that none of this is insinuating that any of these are bad or they were malicious in intent.

But at the end of the day, the assumption I'm kind of putting forward here is that maybe we should prepare for this being the new operating environment. So, the tenor of my entire career has been getting things to the Cloud.

2014, folk were migrating applications and onboarding SaaS, and the pace of that continued. In 2020, well, we all learned just how important identity was and Cloud-based services would be for continuing business operations as we all enjoyed our work-from-home time. That continued and eventually became the accepted standard that a Cloud-based IDP will be the thing you use to authenticate your users and your business.

However, that also assumes that all the supporting infrastructure of the internet will have 100 percent reliability. Turns out, that may have been a bit of an assumption if these trends do continue.

So, let's say you work at a manufacturer, and impacts to manufacturing are very significant because it is not financial transactions on a spreadsheet, as important as those are. There are real inputs and outputs, true material going in and out of the floor.

So, if a factory floor finds itself with its one link down for any reason, be it something that prevents external connectivity or one of those major players that facilitate the modern internet have another issue, well, the shifts can't start. Well, that's the least of your concerns. Your terminals will fail. Most impactfully, the line will stop. And I learned earlier today from a talk that Scatter Spider saw Land Rover lose 5 million pounds a day while their factory floor stopped. They actually had to get backstopped by the government because of an attack.

So, there's real money involved in the line stopping. Furthermore, lots of audit trails and compliance processes either get locked out or just not logged. That's not great from a compliance perspective. And at least in a place where you don't have very many worker protections, the workers will probably still try to find a way to do their job, including using shared break glass credentials to continue production, maybe propping the door open to receive shipments from people. That will just make your security patterns fall apart really darn quick.

So, this is a niche problem with a niche solution. So, where can we look to try to find patterns that match that? And regardless of your opinion on the military, they have been living this problem for the past 30 years.

So, they are operating inside of environments where connectivity cannot be guaranteed. There's malicious jamming or attacks.

And so, things like forward operating bases won't be able to connect to home base. Convey operations, they're going to pass through areas where they're going to lose connectivity anyway or intentionally remain dark. But they still need to execute their processes within that mission. And imagine a submarine disappearing beneath the waves for 30 days. They're going to maintain radio silence. There are systems and things that must be done inside of that environment regardless. They will not be calling home.

So, they are going to build a detail compliant infrastructure. So, let's finally define detail. Despite sounding spicy on the tongue, it's pretty basic. It's denied. And that's when connectivity is actively blocked, either through malicious intent or otherwise. Degraded is when the quality of the reliability of the traffic is such that you don't have the bandwidth of connectivity that the identity systems expect and all the downstream issues of that will cause. Intermittent indicates intermittent connectivity.

You're going to have to have systems that are capable of completing their transaction. Even if midstream, they lose access to external systems. And limited, the pipe's too narrow. In addition to talking to your doctor, we need to figure out how to make sure that your kilobit level streams in a degraded environment can service flows for manufacturing.

Well, in the military case, for military purposes, when most systems are used to megabit. So, the military has a pattern for structuring identity inside of a detail type architecture.

So, identity is not centralized. It is federated across four different tiers. And each one of these tiers is capable of operating independently. And they synchronize with each other when the links allow or they are no longer inside of a detail type situation.

So, there's no single node. It's a single point of failure. They're all autonomous, but they all sync with each other. We'll get into a little bit more about this in a moment.

So, the most important thing here is that they don't design for connectivity. They design for the absence and then benefit from connectivity when it is available.

So, there are five interlocking principles that allow this kind of solution to work and maintain identity security, even in degraded conditions. And we're going to go through those right now.

So, the first is the principle of offline first authentication. Authentication will happen at the edge against locally resident cache credentials.

So, users authenticate against the local node at a forward operating base, not against a server hosted in the cloud or 500 miles away at, you know, whatever the base that is. I'm honestly not much of a military guy.

So, if I keep saying base and place and places, then so be it. But this still allows them to use their CAC and PIV cards. Those are cryptographic authenticators issued to every member of the military. Revocation lists are pre-populated and refreshed during periods of connectivity.

So, that information on the certificates which have been revoked gets updated as rapidly as possible. The authentication state will persist through extended outages capable of lasting days. And you maintain the same security posture, whether connected or isolated.

So, there's no degraded fallback mode. So, offline isn't considered a degraded experience. It's considered the baseline. You get a boon for connectivity. Pattern two, non-person entity identity. Every single device system in the tactical network uses a cryptographic identity for authentication.

So, that's not just the humans using PIV and CAC cards. So, everything uses it. Machine identities get provisioned prior to deployment, not necessarily spawned in the field.

So, short-lived certificates are designed to automatically rotate within the connectivity windows as they happen. Device access can be revoked independent of the status of the human user using it.

So, that way you're able to isolate a given piece of the technology and zap it there. Mutual TLS for authentication.

And, of course, zero trust, since every connection is authenticated and authorized. So, compromised credentials already have a dramatically reduced blast radius that can be very precisely targeted using those cached revocation lists. Zero trust at the edge. I don't think there's too much interesting here other than you get a copy of the policy local.

So, one of the big themes here is you're going to be replicating everything you have at home base into increasingly smaller deployments. So, that way you can maintain that autonomy under a disconnected condition.

Now, that separated architecture allows store forward and sync. So, that connectivity is now used primarily for synchronization and making sure that certificate revocation lists and other important information from home base flow down when you have it.

However, when you're not connected, everything stays static. So, there's an order of operations in terms of prioritizing for prioritizing for maximizing security while minimizing the amount of bandwidth and capability used inside of this situation.

Obviously, priority is given to credential revocations, access changes, and policy and role updates, and if you have the time, certificate rotations and routine provisioning. This is a delta sync, not a full sync, that helps save about 60% of the bandwidth that would be used generally to keep all of these information systems aligned. And as I already said, this layered deployment tier, each one of these is independent of the other, but they're all copies of each other.

Any one could be chopped off from the top and everything below will continue to operate, potentially even with feeding back up into the upper nose once again when periods of connectivity are available. All right.

So, cool. That's the military. What are we going to do about our factory floor and what can we learn about this? I will say this is a very niche solution, but these are very niche problems.

So, we'll talk through it and invite you to think about the applicability, if possible, for anybody who works inside of a manufacturing concern. All right.

So, as I've already intimated, the same conditions are operating inside of a different arena. Rather than military-based perspective, we're considering factory floors. There's situations in there that align to deny degraded intermittent and limited internet connectivity. It doesn't take much imagination to see them.

So, with that, what would be the five interlocking principles that would allow a factory to continue operating inside degraded conditions? So, first of all, and the provenance of this comes from work we did with a manufacturer to vet these concepts, which means it's very proprietary, and I have to be a little bit vague, but I still think there's a lot to be gleaned here.

So, principle one, offline authentication, first on the factory floor, very similar to that military pattern, which means caching credentials locally and, funny enough, using domain-joined instances and on-prem active directories, potentially. Now, the nature of that deployment is going to change, and we'll get into that in a moment, but, yeah, this allows you to use cryptographic authenticators, including 502 and WebAuthn, and Kerberos for session management across all the domain-connected and shared workstations.

So, the goal is to have a full authentication continuity through a 24-hour degraded period without a fallback to any or regression in the security posture, generally speaking. All right, principle two, very similar to the military's principle two, every device has an identity. Everything gets PLCs, HMIs, SCADAS, et cetera, either through a series of reverse proxies using common protocols or certificates or things like that.

We do make sure that every item on the floor has an identity, which means all the authentication authorization transactions can be validated, even without access to an authoritative or higher-level certificate store or a CRL. So, the third principle is zero-trust lockout, which is enforced entirely within the four walls of the factory.

So, you have a physical badge out, your physical access control system reads that and fires an event, which could then trigger an active directory change in membership, perhaps moving the user to a temporary denied group, which means the identity provider will no longer issue a token for that user. The goal is to have that execute within five seconds without having to call home.

So, that works without your WAN. It applies to the machines and all the other identities we've applied a cryptographic identity to. We preserve the audit trail regardless what happens in terms of connectivity, and it is designed to be compliant.

So, that way, all of the access verification requirements for certain regulations are met by design and not bolted on, and they're all executed within the context of that edge or the four walls of the factory. So, principle four, synchronization and store and forward for OT.

So, the factory environments, our sink is engineered for constrained bandwidth. We use a delta-only transmission, just like the military, to minimize the amount of data required to ship up. When there is an outage, the system falls back to a read-only mode. That is by design.

So, that way, we can continue to authenticate and do everything we need to do, but configurations, net new additions, net new accounts, modifications to certain systems are going to be locked out. There will be no provisioning during this. As the outage status flagged, every audit event is enqueued for reconciliation to make sure everything operating within those four walls eventually makes it to the enterprise sim upstream.

So, that way, you maintain compliance. The target here was to limit our bandwidth and also be able to maintain up to 30 days of those kind of logs and transactions and events to be able to forward them out to the fact.

And once again, and this one is where it's a little different than the military, but we offer a stack in the box, which looks a lot like the other things, but the way the military does it, but the important thing here is that we're aligning all the identity tools and capabilities across a unified Kubernetes and Helm chart, which means we can deploy it centrally and administrate it through things like GitHub Actions and GitLab Actions, Terraform pipelines and whatnot, and provision from the enterprise cloud, the regional hubs into the site nodes, potentially into an edge kit or equivalent for that factory.

These pipeline-driven deployments are able to allow you to scale so that we do not need to go on site. It requires some data infrastructure on site to execute, and it requires aligning those identity tools across the stack to make that easy and scalable, but the benefits seem to speak for themselves, at least for this, once again, surprisingly narrow use case.

So, as I said, this was validated in a real manufacturing environment. I've already mentioned the four walls, ensuring that we're able to execute everything within the context of those disconnected operations inside of the factory itself without requiring a call up to a regional or the central environment.

So, this architecture covers all the worker populations and all the devices, contractors, business-to-business, because it is all locally cached. The legacy OT systems are addressed through a similar caching process using something that will translate an OIDC-style transaction like a proxy to a scatter, and forgive me, I'm an identity guy. I don't work in a factory, but to cover the low-bandwidth control planes for those machineries.

And, of course, physical access, logout events, HR termination events, as they're received, are copied locally and then able to execute within a tightened time frame. So, what that gets is you're able to maintain full authentication in under three seconds without having access to a Cloud Identity Provider. We're able to withstand 24 hours of wide area network outage easily, and we've tested the option for up to 30 days of that kind of outage to make sure that all critical audit events are maintained and queued and sent back to the SIEM.

We maintain a strict badge-out process and policy contained entirely locally. We've reduced the bandwidth needed to operate the factory, and we're servicing them centrally from one location and managing all the instances during periods of connectivity. And I'm making this sound more dramatic than it is, because it's not like we saw a proper issue during this testing. We had to synthesize that, but it was still interesting.

Now, I'm a big fan of always emphasizing what the business gets out of this, because technology is cool, but the business gets productive capability. They're not going to suffer that five million pounds per day outage, because they're able to operate independently. You maintain your regulatory compliance control, reduce the tax service, nothing is actually exposed from the OT networks beyond the boundary. You do some really cool orchestration capabilities with scalable and pipeline-driven operations, and you're able to merge your physical and digital governance.

Now, something that was new to me, but was raised by business partners, was there's often discounts for cyber insurance if you're able to prove these kind of capabilities. You have to prove there won't be a disruption.

So, it's another field in which I'm utterly unprepared to explain or understand, but apparently the auditors and the insurers really like this notion of reliability. So, if you're interested in seeing and sniffing around what it is it would take for your factory floor to do something like this, first you have to understand the scale of the problem. And these three questions or actions will go ahead and give you the exact parameters of the problem you will need to solve, so that way you too could apply these principles specific for your organization, if you feel the value prop is there.

So, question one, what authenticates when you don't have a WAN connectivity? I don't know. My guess is a lot of you don't know either. Good way to find out is probably pull the plug during an outage window, but yeah, good luck. Figure that one out on your own. When was your last time a shared workstation credential was rotated? If there is no record of it and it cannot be validated, then that probably means more than one person has had access to that credential and needs to be rotated.

And of course, how long does it take for operators to lose all access upon either resignation or badging out for the day? I think the HR operation is the much more sensitive one in terms of execution, because there might be a longer load time, but still the time between badge out or even separation is that vulnerable window where somebody could retain access to human machine interface kiosks and other pieces of technology, even though they should no longer have that kind of access. So if you have any questions, go and ask them, but that's my piece.

Okay, one quick question here. Sure. I don't know if I got it right, but how do you disconnect the factory floor from the key infrastructure? From the key, like particularly WAN connectivity or enterprise connectivity through the WAN? Manufacturing floor to the IoT infrastructure. To the IoT infrastructure? Yes. So the specificity of that is going to depend greatly on your control plane interfaces from the OT to the IT. I think I had some blurbs in here about what we do to make that disconnection happen or keep that connection resilient.

But I think the thrust that we were going for here was we never ever expose that OT stuff beyond those four walls of the localized environment anyway. Everything is abstracted through the identity layer. And I don't feel like I'm answering your question quite right, so maybe you and I can talk after this.

But yeah, anything else? I think that's good.

Thanks, John. Thank you.