Signing in the Rain: HTTP Message Signatures and Web Security

So hello everybody. My name is Justin Richard, and I'm an independent consultant based outta Boston in the us. And today I'm here to talk about a new draft that I've been working on in the ITF with some other really smart people called HGP message signatures.

Now, the reason for this, this draft and this work is that HTD P is absolutely you're everywhere, right? It, it started out as a document sharing protocol, but really that's not what it is now. It is fundamentally a basic part of our society. So much of our world runs across HTD P and security is a good thing. It's something that we all want. We want security to be, you know, part of these things that are so fundamental and so ingrained in our lives. So it makes a lot of sense. We would want HTP to be secure.

And TLS helps that if you ever go to a, a website or call an API, you notice that it's not just HTP, it's HTD, PS. This is something where the HTP message just gets told right along with the secured channel and everything is encrypted and stuff along that channel. So nobody can modify it. Nobody can steal it. And all of that's a really great thing, but TLS doesn't solve all of the problems because these days, when you send an HGV message, what's the first thing that it runs into when it hits your data center, it's gonna find a TLS Terminator, right?

This is the first thing we do to our TLS connections. And the we've got lots of very, very good reasons for this, but it kind of breaks the security model of TLS because TLS is all about protecting something from point a to point B and just not admitting that there might be a lot of other points in the middle H GDP on in contrast really is a protocol that's designed to be passed around to a whole lot of different things. So how can you be sure that the message that you're getting really was sent from the person that claims to be sending it? How do you know that the sender is there now?

TLS and HGS is really great about saying sort of the response, the message that I'm receiving is from the server that I'm expecting to talk to. It's really, really great, but how do we even start to talk about that in the other direction? And if my server is behind a proxy behind a gateway, how do I know that that server is the one that sent that response really and made it all the way across those across all of those points in a way that wasn't modified?

Well, we're going to now look at signing those HTP messages and signatures are a way that allow us to protect a bunch of data so that it can't be modified in transit. And we can also verify that the sender is the sender that we intended, right? Longstanding, cryptographic practice, well known, and they work because any tiny, tiny change in the thing that you're signing completely invalidates the signature, the whole crypto system freaks out.

And for a good reason, it's supposed to freak out because that way, you know, that somebody was messing with stuff in the middle, but HTP is super weird as a protocol like anybody that's gotten down onto the wire with HGP knows that you're gonna send a message and headers are gonna get reordered and all sorts of weird things are gonna happen. And the truth is when I go and create a message, and I say, this is what I'm signing. I'm giving you like, this is what I'm going to send you. This is what I'm trying to tell you.

That's gonna go off across the internet and it's gonna go behind a bunch of different proxies and a bunch of different routers and all sorts of things like that. And so what comes out looks a little bit different, right? You look away for just a second. And the message that you get on the far side is not the same that you sent in. But the funny thing is it's all Tom Holland, right? That message that you sent is still gonna be semantically equivalent. Otherwise your application wouldn't work, even though the actual H GDP message itself, the bites on the wire have changed.

Now, this is a problem for signatures, isn't it? Because if we change any of the bites on the wire, even the smallest ones, then that invalidates the entire signature, which is the entire point of the signatures. So how can we start to combine these concepts together, right? How can I sign something that I expect to change in expected ways? And importantly, I touched on this a second ago, this has to work for both requests and responses. Like I can't have a solution. That's just like, oh, I'm talking to the right server. And I'm pretty sure about that.

I need something that goes both for requests and responses and works in this robust fashion. So what if we just encapsulate the HGP message, I take that entire message, all of those changing bites. And I just kind of wrap that in something else so that my original message doesn't change. And then I wrap that into an envelope that can change.

Well, this idea has been brought up a lot and people tried that. So who here has deployed and used S HTTP, not HTTPS, but S HTTP. That's what I thought, right? This is a great idea from 1999. And it went absolutely nowhere. And people have tried doing this type of encapsulation, think a lot of different times in a lot of different ways. And it doesn't really work for this type of messaging now to be fair, encapsulation has some really, really, really good places.

So if you've ever worked with the Jose suite, Jason web tokens and stuff like that, those are utterly brilliant for self-contained artifacts, tokens, assertions, things like that, because now you have something that is its own self-contained bit, but that has to be carried across within something else. And it doesn't tell you anything about the space that is surrounding it, unless you copy that into the message. And then you've got cross-validation problems and all sorts of other issues.

So instead, we are going to look at a method called a detached signature, and that means that I'm gonna need to be able to derive a signature from the context of the message that I'm signing. And we're going to do that by creating a set of rules, to make a common base, to work from, for our signature. All right, now, people have tried this a few times too. The Cavage signature spec has been implemented and deployed a bunch of times. This was draft 12 of the spec. Each version kind of broke something else in the spec, cuz this wasn't part of a working group.

This was just some well-meaning people's idea to try and solve this problem. And we've been working on this a few times. I have personally been involved with three different efforts, including the current one, but it's gotta be different this time. It's gonna work this time. And I actually do feel some confidence about that though, because now it's an artifact of the HTD P working group.

So whereas before we had a lot of security nerds and a lot of application developers trying to solve this weird problem with the HTP protocol, we now have a bunch of HTTP nerds saying like no HDP has some very specific ways that all of this stuff fits together and you have to take those into account. This is a draft spec in the ITF. It is on its way to RFC. And I think we're gonna, we're actually going to see this thing really roll out and deploy. People are already building this out. It's very exciting stuff.

So I am now gonna walk you through how you sign and verify HG messages using this spec all. So who's ready to do some math. Don't worry. That's all right. Awesome. I have a different presentation for that. Don't worry. We're gonna, we're gonna go things over, go over things rather at a conceptual level. So first off I take my HGB message represented by that bunch of blocks up there. I'm gonna decide which parts of that I wanna sign.

This is a really important step because sometimes I know that I'm gonna be sending it into an application that it's gonna go hit a reverse proxy and it's gonna change the authority section, or it's gonna insert a header that or add to a header that, that I know is there. So I know at the application level that there's gonna be some stuff that I can rely on not being reliable. I can take that into account when I'm deciding which parts of the message that I wanna sign. So I'm gonna decide which bits I wanna sign, lay those out in a very specific order. And in order I get to choose.

But as soon as I choose that order, I keep that order. I'm then going to take that set of they're called covered components. And I'm going to pull those out from the message and lay them out, using a deterministic format to create a signature base. This is in layman's terms. This is just a string, a bunch of characters that I can throw at the crypto algorithm, which is the next step. I take that set of bites. I throw it at the crypto algorithm. It gives me another set of bites. That's how signatures generally work. The hard part is getting to that set of bites.

And that's really the bulk of the HTB signatures protocol. Once we have this signature in this format, we need to attach it to the HTP message. I'm not gonna get into the details, but you send the signature and also the list of things that you signed as part of that message. You do not.

However, send this whole signature base. We are not encapsulating and sending the thing that we signed across. So it is not like Jose. It is not like a jot we do.

However, sign the list of things that we signed and we send the list of things that we signed along with the signature. And we'll see, see why in just a couple of minutes.

All right, now we have assigned H GDP message. That's all well and good, but it's only as good as we can actually get it across the internet and have it make sense.

So, because the important part here is that once we sign something, it, we have to be able to verify it. And that's where things start to get really strange, cuz I take my message and I sign it and then I send that message out over to the internet, right? Something happens to it in transit. I don't even really know what I don't have control over that. The bites and everything of that message can get changed around. So that the thing that I send and the thing that's received are not the same set of bites.

Alright, they're just, they're different. So if I were to run both of these blocks as is through a signature algorithm and a verification algorithm, it would fail. It is designed to fail because things get changed around.

However, at the HGP layer, these can be completely equivalent. These are absolutely expected and totally okay, this is what makes this a hard technological problem. So to verify step one, we pull the signature out of the message. This is a set of, this is a set of headers in the message that we pull out the list of things that we signed and then the value of the signature itself. Well guess what, we've got a list of things that we signed. We've got an HG message. We are going to use that to do our validation.

But first we have to make sure that that list of things that we signed is the right list of things for our application. Because if I have a signer that is not signing a critical header and that gets, and that critical header gets modified or the path parameters get modified or the authority parameter gets modified or the verb gets modified and I haven't signed those, then the application should reject the signature and fail because not because the signature is bad, but because the signature doesn't cover everything that it was supposed to cover. All right.

So now that I have my list of things that I've got my message, I go through the same exact process as the signer did. And I read, derive that signature base.

Now, remember this was not sent across the wire, but I am going through that list. And just bit by bit deterministically taking that string and, and appending to it. An important thing about this and where it's different from XML DSIG. And I'm sorry for bringing back flashbacks of anybody. Who've had to deal with XML DSIG, but an important bit about this is that this is not a parable message. This is not meant to be a structured document here. This is just a basis for the crypto to work on.

And that gives us a lot more power and flexibility and stability than XML D SEG ever did or any document based system. So we run those bits through the verification algorithm against the keys that we know, and we figure out whether or not the message has been verified right now that the signature and verifier agree. We know that the message is valid and everything is just fine. So this brings us to an interesting question at the application level, what should we sign the signature spec? A lot of people have said, well, you need to define a minimum list of things that, that you have to sign.

But the thing is like everybody's got kind of different use cases. And when you go to look at this and you go to apply, this things are gonna look different. For example, I titled this presentation, just, just give you an idea of how different the world can be than what you assume it needs to be. I titled this presentation signing in the rain. And so I was pulling a lot of images from the singing in the rain movie. And I went to Google search things and it turns out that there's a Korean pop singer who goes by the name rain. So he comes up in the list of images for singing in the rain.

There's also a singer named prince who has a movie called purple rain in which he sings. All of these are singing and rain. And it totally makes sense to the people who put these together. That those words would actually be the kind of thing that I was looking for. Right? So we need a system that is flexible enough to allow for these different types of expressions and these different types of applications. Because my application might be really, really, really concerned about which HTD P verb I use. Like the difference between a put and a post could be absolutely critical.

Somebody else could have a reverse proxy that takes all inputs and then just wraps them up in something weird and posts them to a backend system. And that's totally fine for their system. Like they don't need to cover the verb. So if the me, if the signature spec says, you always have to cover the verb, then, then that's a problem for all of those kinds of applications. So there is no universal way to say this is the list of things that you have to cover.

It is however, very important and hopefully laid out very clearly within the signature spec that this is a question that you absolutely have to answer, right? And this is possible because we have decoupled the process of creating the signatures from the application of the signatures up at the application level. This wasn't really done with previous attempts at at message signing. A lot of stuff is really tied together, you know, tied all of the pieces together. We really have made an effort this time to make sure that those are separable.

This also means that we can now do some pretty advanced stuff and no, I can't do that dance move so we can sign a message that's already been signed. Why would I wanna do this? I've got a message coming into a reverse proxy. And I validate the signature at the proxy level. And then the proxy is now the API gateway is now saying that that signature was valid.

And anything else in the message that came in has been validated before it gets passed on to the downstream system, because maybe the API gateway is the one that has access to all of the client's keys and your downstream APIs don't or lots of other reasons to be able to do this. The signature specs explicitly allows multiple signatures, and it explicitly allows you to do different, different parts of the message, including existing signatures. I can also take and sign the message content.

Now, this is not really helpful. If all I can sign is like the verbs and the headers and stuff like that. I need to be able to sign message body because a lot of times the message body itself is the most important semantic content. So you're doing graph API or, or heaven forbid soap.

Well, there's a sister specification that allows you to take the message body and present a digest of that and puts it into a header, which means that we can now take that header and sign that header and add that to the signature and push everything along together. And now we have the message body signed.

Similar thing happens if I want to be able to sign the client TLS certificate on the way in there's another sister specification that says, if you have something coming in with a TLS certificate, here's how you take that certificate, collapse it down and put it into a header and put it into the background. In other words, we are actually embracing that TLS Terminator and allowing it to make clear statements that say, not only have I, not only has the TLS been terminated, but I guarantee you that I am the one that terminated this one, right?

And this is one of the more interesting things that we can do with this spec that previous work didn't do. Now, when I'm sending an HGP response, that response is always made in the context of an HGP request. Some request always triggers every response as the verifier, or sorry, as the signer, as the creator of that response, I have that request context. So I can add pieces of that request into my signature base. In other words, I can say that this response is from an HTP post.

You don't get that information in the response message itself, but that might be really, really important for the security of an application. I need to be able to say that, yes, I validated your access token was in the authorization header.

Now, how do I do that without just echoing all of that information into the verifier or, or into the response message, which is what other protocols have done well, when I'm signing the message, I create this signature base. And I list out that like these two bits actually come from the request. So when the verifier gets it, they need to have access to the request. But guess what? It's the verifier is the party that made the request in this case. So they probably still have access to all of that information at, in the context in which they're doing the signature verification.

And if this is part of your application profiling, then that's something you can build out your toolkit to do. In conclusion, I wanna encourage people to go check this out. If you've got a more nerdy bent, go to HTTP sig.org, this is a live playground. You can actually poke in a, your own HTTP message, your own keys. It will show you how to build the signature and, and it will actually sign it for you and verify it for you. So please go check that out. Thank you all very much. I think we should have time for a few more questions, so thank you.