Let the Robots Help: Runtime Security for Workloads, Services, and APIs [Introductory]

Hello, everybody. My name is Justin Richer and I am here today to talk about API security. Who the heck is clapping?

David, shut up. Anyway, I am here today to talk about API security, which seems to be a pretty straightforward topic, right? You've got an API and you want to keep the bad guys out. So when the bad guys show up, oh, that bad guy's invisible. That's an interesting rendering artifact there. It's an even sneakier bad guy than it's supposed to be. Regardless, when the bad guy shows up, we simply wrap the API in security and the bad guy's plans are foiled, right? This is all that it really takes. It's pretty simple. We've got an API.

We keep the bad guys away from the API and everything's all good, right? Even though that seems pretty simple, I think we all know that the reality is a very different story.

But first, I want to get you guys thinking about a different process that on the surface seems simple and that's biometric validation. Now, that might actually sound complicated just by the terms, but think about when you use it on your phone. You press a fingerprint or it looks at your face, it validates you, lets you in, everything seems to go.

Well, if you look at NIST's SOFA-B paper, and I love the title of this paper, they lay out the five steps of the biometric validation process and the 12 places that those five steps can be compromised by an attacker. This happens because biometric validation is a complex system.

Now, by complex system, I don't just mean that it's complicated. I, in fact, mean complex in the mathematical sense. And those of you coming to my talk know that I was probably going to get into the math at some point, right? We're not going to go deep into this, really, but the whole notion here is that you've got a bunch of independent components and it's the relationships between those components that bring the complexity to the overall system. That's where a lot of the trouble starts to come, right?

So the behavior of complex systems is really, really hard to predict, but they're everywhere around us. They are absolutely everywhere. So I want you guys to think about yet another system, and that is getting a glass of water.

Again, on the surface, this seems pretty simple and we can model it, right? It goes from the tap to the glass to you, and getting a glass of water, it shouldn't actually be that hard. But if you're like me and you grew up with the magic school bus, you know that the water system is incredibly complex. There's a whole lot of different things going on.

Now, I was informed yesterday that apparently you guys unfortunately didn't have the magic school bus here in Germany. And I'm sorry, you're missing out. It's a really great series. But in this, we find out that the water delivery system is really complicated to get clean water into a tap. You have to have a clean source. You have to have a place to store it. You have to have ways to treat it, ways to distribute it, all the way before it gets to you.

But this probably means, even with all of this complication, that, well, it is a complex system, but as long as your delivery is good, then you're done, right? You have solved the problem of getting clean tap water out to people, and that's probably all you need to care about. Except that how many of you have something like this in your home? So I have a water filter in my fridge that further filters and chills my perfectly clean and drinkable tap water. Why? Because it's better. It tastes better, and I use this for a lot of things.

And in fact, I'm a little bit of a tea snob, so I always use additionally filtered water to make my tea. Never tap water, even though, quite honestly, tap water would be fine. But there's an even deeper secret going on here. And that's that even if I have absolutely clean water coming out of the tap, and I totally trust it, and everything's good, would I put it into a dirty glass? How many of you still wash your glasses before you put the water in it?

Like, even though the water source is clean, even though it's been through treatment, even though it is safe through delivery, it is safe into your home, even if you actually filter it after it gets to your home, after all of that, you still have to put it into a clean glass. Because this is a complex system. And every single step of this system, from the source of the water all the way to it getting to you, it affects the result, right? Although I will say some people make different choices about that last mile delivery, and you do, of course, always have the option.

Now, the reason I'm talking about all of this is that software security, and especially as we look at API deployment, this is also a complex system. There's a bunch of different moving parts that have to tie together and that have to work together in order for these things to work.

Now, this is brought into a special focus lately due to the rise of AI-based systems. Protocols like MCP and A2A and all of that, what are they but just another layer of APIs, right? They are APIs that different systems are calling, and they need APIs to power what it is that they're doing. You look at MCP specifically, and you've got two different sets of API that are tied together. These are independent components, like we were talking about with these other complex systems, but they're kind of weirdly intertwined.

Because if I get access to call the MCP server and the MCP server has access to call this upstream API, they're tied together in a way that's a little bit hard to predict ahead of time. And it can fall over in interesting and fun ways. So let's take a look at securing an API. Let's get down into the brass tacks of how we actually do this.

First, assume that you have a good delivery system. Your software is actually good. All of your images are fine. Your OS hasn't been compromised. Everything is deployed okay. Is all of this really hard? Absolutely. But we're assuming that the water got to the tap clean at this point. We need to now make sure that after this, we're actually putting it in a running system and making it available in a way that is still secure. I went into great detail about how to address this part of the problem in another talk, which, you know, we don't have time to go through all of that today.

I only have 107 slides today. So it's really important that you do have good source and that you actually validate and test everything up to this point. Because if you don't, then everything we're about to talk about doesn't really make sense anymore. So you have to assume that you're starting somewhere good. So first step to get everything deployed, figure out which server you're going to deploy it to, right? We'll use that one.

And back in the day, you would install an OS, log in as root, figure out where you're going to put it, configure it, and then go through the, like, you know, week-long process to get a certificate issued to that server after it's been provisioned on the network. And then you install and set it up and probably not touch it for another 10 years. Because it took such a long, painful time to get everything set up, you don't want to go through all of this again. So this is a really, you know, this is an onerous process. It's a manual process. So we do it as little as possible.

Today's world is a whole lot dynamic. And right now an old man is going to yell about the cloud for a little bit. Because the world that we live in, in terms of our computer architecture, doesn't look like what it used to. Back at the beginning of my career, I was installing things on hardware. That went to, you know, get the actual physical computer on your desk, and that's what you're installing to. We then went to virtual machines and blade servers and containers and all of this other stuff, down to these days we have Lambdas, which exist for a couple of microseconds and then disappear.

An entire piece of server functionality can rise and fall within the scope of a single request. And our systems are being architected and built in order to accommodate that. And we can plug them in in all sorts of different ways, and in ways that we didn't actually originally intend those systems to be plugged into. In other words, we've got a whole system of independent components with relationships in between them. And can you start to see the little bit of parallel here to the other complex systems we were just talking about?

Now, we commonly do have a path of something coming in through a gateway, then it's got a host that loads images, but this is far from universal. It is a model that we can often use, but just keep this in the back of your heads as we go through the rest of this. We're probably going through a whole stack of different systems.

Now, once again, find your server. Except you're going to say, our server doesn't look like that anymore.

No, no, no, we run everything on the cloud. So this is what our server looks like. And the thing is, though, because of all of the layers of abstraction, what does it mean to find your server? How do I log into the server to install the keys and the certificates and set up the trust routes, stuff like that? We have workloads, which are pieces of software that run for a specific purpose. They could show up on any one of those nodes at any point, and then be scaled up and down all over the place in ways that we can't predict ahead of time.

That's kind of the whole point of dividing things up like this. But because they're so ephemeral, because they're so focused and potentially super powerful, we still can't treat them like traditional servers. And yet we have the same set of questions that we need to answer about them. How do I get the certs in there? How do I get the trust routes in there? How do I get access to the data and all of my policies? And how do I do this every microsecond?

Clearly, our old manual systems, our old way of thinking about it doesn't apply anymore. And we need to think in terms of automation, which means that we need good tooling and we need good systems to help us do this. So the first of these that I want to talk about is something called Spiffy.

Now, Spiffy is a really cool system because the whole idea is that you get a piece of workload software, and instead of pre-configuring it with all of its keys and its identity and it sort of knows who it is and where it's running and all that, it wakes up and has no clue who it is, and it asks sort of this local agent, like, hi, I just came into existence, what do I do with my life? And the Spiffy agent actually recognizes it.

The platform recognizes that piece of software as having a particular hash to it, a particular configuration to it, and it assigns identifiers, it assigns certificates, and importantly, it assigns a whole bunch of trust routes to it. This identity through recognition is a super, super powerful thing. Because in the world of non-human identities, we don't want to have a human have to figure out, like, oh, this is the piece of software that I am giving these sets of keys to right now. These things come and go so quickly, this has to be automated, this has to be built on policies.

And all of this needs to be run in a space where I can see what's going on, but I don't have to have my hands in it. Now that we have identities and keys in the Spiffy environment, these workloads can actually start to recognize each other. I know when I'm calling from one workload to the next, and that all works pretty well. But we fall into a really simple trap here, and that's thinking that identity solves our security needs. If I can identify the caller, then surely everything else is going to be fine, right?

Well, except at this point, if all I'm doing is identifying the caller, then that caller can do anything that that caller can do at any time it wants to do it, whether or not somebody actually asked the caller to do that in the first place. We expect workloads to be working in a particular context, in a particular transaction. So how do we go about actually limiting things?

Well, we have a mechanism for limiting things already out there, and that is from the OAuth world, we have access tokens. Now, in the OAuth space, this is delegated access rights, generally from a person to a piece of software, in order to call a resource server. So this works pretty well once we get to that boundary. But what am I delegating to? I'm delegating to the resource server, which is this sort of big orange blob here. But inside of that orange blob, this is where all my workloads run. This is where my data lives.

This is where all of those different pieces, that complex system that I need to deal with, this is where all of that lives. And so this is where we're trying to solve some of these problems. So how do we control access to what's inside if we sort of know what's running and we've checked all of that and we have this access token?

Well, we could just replay the access token everywhere. I mean, hey, it's an access token. It should have all of the information about what this thing can do, right? And you see this a lot, actually. You see this a lot out in the world. And it works pretty well. But the thing is, that access token was not made for this internal context. And in fact, a rogue workload or service, anything in here, could actually replay that access token and kick off a whole new transaction under the rights of that access token in the first place.

Now, wait, you might be thinking, why don't we just do sender-constrained tokens? Because if I sender-constrained the token, then only the client can present it, right?

And sure, that stops this attack. But unless we jump through some hoops, it actually stops us from using the access token in this way. Because we're now saying that, okay, this is a sender-constrained access token, except we're not sender-constraining it on the inside because it's not really an access token anymore and it's something else, but we're calling it an access token. And our middleware thinks it's an access token. And things get weird real fast.

Since access tokens are meant to sort of kick off this whole transaction, why on earth are we still using them, this external access token on the inside? Which leads us to the technique called token exchange.

Now, with token exchange, I take the external token and I can swap it for a different token on the inside that I can then play through all of these different systems. The color's a little bit washed out, but the ones on the bottom here are all dark blue instead of the black-shaded token on the way in.

Now, with this setup, we've got some good aspects. Since it's a new token, it can be incredibly short-lived. It can be tightly constrained to sort of this one request, this one transaction that we need to care about. And importantly, we can't replay it externally anymore because that external presentation is only for that external token. And then these internal services never see that external token. They only see this derived token. So they can't reuse it. And that's great.

But if we're not careful, we could end up with the same problem, but on a smaller scale internally, where that internal token could be replayed between a rogue service and something else in order to get something else done that it doesn't need to. Now, we can actually apply Venn diagrams to solve this problem. And I'm sorry that Eve's not in the room here. But she was in the hallway, man. So somebody tell her there's a Venn in here and she'll come running.

But now that we have this notion of who's doing the call and what the token is doing, we can actually start to write policies to scope things down, right? So we have at least a little bit of a foothold to start to think about how we can secure these things in a way that scales into this space.

Now, we could go to the extreme of every single time we go to a new step, we do another token exchange because I don't want this token to be reused elsewhere. So I'm going to get a brand new one every step that I go. And this works pretty well, except as you can probably tell, it leads to some pretty nasty token proliferation. Every time I go down a step in the tree, I end up with a new token, and you end up with some interesting scaling and complexity problems. And I know this because I built it and I deployed it a bunch of years ago. Don't.

Honestly, doing an internal token exchange makes sense in some limited circumstances, but don't go to the extremes that we did. I'm actually glad that we did because we sort of proved that it was possible and why it was a bad idea at the same time. Everything is all about finding a balance. So what if the thing that we're trading for isn't actually an access token at all? So we do a token exchange, except we get this thing called a transaction token.

Now, a transaction token is new work in the ITF that this is not an access token. It's not meant to protect access to things, but it's meant to provide context. It's saying that an access token had come in and somebody at the gateway decided, you know what, this is what this current transaction request is for. And so everybody working on this needs to understand that that's what we're doing. I can combine this with an internal access token if I want to lock that down even more, and that's pretty nice.

Even if I want to keep swapping out the internal access tokens, I can add these layers on independently. And that's a really important thing for me to be able to do in order to build this out in a way that meets the needs of my highly dynamic environments. And this brings us to the workload identity and multi-system environments work in the ITF that's happening right now, also known as WMSI.

Now, WMSI is basically looking at these exact problems in sort of two main thrusts. First, what does security look like in the boundaries if we take all of these techniques together? If we can identify the workload, if we can do token translation, if we can have transaction tokens, if we can carry all of this stuff together, how do we start to reason about that? What does that actually mean to deliver a transaction token or an access token or an identity into one of these workloads as we build things out? But also importantly, what does it look like when we cross security boundaries?

Because everything that I was just talking about kind of blows up in a weird way if your service needs to call some external cloud service, right? We're sort of back into the weird OAuth outside world, except that we started deep inside of a workload call chain. How do we actually even reason about that? And that's one of the things we're tackling at WMSI. There's a bunch of work that's happening right now, but there's still also a lot of open questions.

Like, for example, how do we even carry a transaction token along with an access token in the SPIFFI IDs, and how does that all relate together? This is one of the things that we're trying to actively figure out. Maybe we embed it in a token. Maybe we put it in a separate header. One of the things that I think is important that we as a community do is question whether or not all of these artifacts that we have right here, whether they're actually tokens in the first place.

Are we really still talking about tokens, or is this a convenient artifact that we keep swinging around trying to solve this problem? Because no tool is perfect. No tool is used perfectly. But I do think that we need to build more automation into everything that we're doing in these cloud and API deployment scenarios. And that's to be worth it, of course. You have to think about the long-term value. If you're doing something, if you're automating something that you're doing a lot, then it's probably worth it. There's always going to be a balance.

And more math, XKCD actually went and did the calculations of how long you could invest into something in order for it to be worth automating. And I think this holds pretty decently. And if you're doing many workloads up and down in a single second, then it's worth investing in.

So, of course, it's 2025. I need to ask the question, where does AI fit into this? I've been talking about automation a bunch, right?

So, clearly, he just means go throw Claude at this problem or cursor, and then it'll automate everything for you. Well, first off, AI is always a tool. It's never a solution. And it's never absolutely a replacement for any expertise.

AI, I think, actually helps us in this highly dynamic world to augment our sensemaking. So, for example, look at your security dashboards.

Like, all of those different values and the red lights and everything like that, is that light red because something critical failed? Or is it red because it's a Tuesday and that just happens? Right? I hear enough chuckles that people know about that red light that happens just because it's a Tuesday.

Like, we've all been there. This is where a lot of this sort of, you know, pattern recognition and reasoning type of systems have a potential to really, really help us out in the security space.

So, automation, I really think we need to use a lot of it, but this is follow an action based on a program process. We can automate the things that we did manually, and that'll help us look at the things that we did manually in a different way. But AI gives us a way to make better decisions. Potentially. We're still getting there.

So, we need to make automation common. In order to address complex problems, we need to make the solution common. It needs to be the easy way to go. The process needs to be repeatable in an easy way, especially for developers to apply to new systems. And all of the results need to be auditable. Because if you're not checking the outputs of your system, is it actually secure? You don't know.

And so, you can't say yes. We got to get humans out of the loop of critical systems and instead have these repeatable, auditable systems in place for doing the actual hands-on deployment. Instead of back in the day where I would actually build the binary on my desktop and then just SSH a binary and run it on a production server directly.

Like, that terrifies me now, but that is literally how we used to deploy software. Ultimately, you need a secure path all the way through. Need a good source, but you also need good delivery and you need a good runtime.

You know, you need to put it into a clean glass in order for this to work. So, the biggest takeaway lesson here is, of course, stay hydrated. And thanks to George Fletcher for this photograph. This is a photograph, not AI generated. I want to point out that there is actual artistry involved here. Brilliant photographer. And ultimately, let the robots help.

You know, they're good at a lot of this stuff. Let's use the automated systems for what they're good at. Thank you. Thank you very much, Justin. That was informative and highly entertaining, which doesn't happen that often. Awesome. Unfortunately, we don't have any time left for questions.

So, can you be found somewhere during the conference? Yeah, I will be floating around during the conference. If anybody has any questions, I'll just hang out right outside the room here and happy to talk more.

Okay, thank you very much again. Great, thank you.