In fact on this picture, you have the room of a, a tuna fishing boat that is operating in the ocean Indian ocean very quickly, just to, to tell you where I'm coming from. So my company is located in loon in south Britain. So we are, it's an ABO city. We are very nearby. The sea. We have 30 engineers specialized in embedded Linux, and all background is VWA wind river. This is where we're coming from. In fact, the company was created when Intel shut down all this research facility in France, and we are especially known for two things that we've done. The first one is ties, which is the operating system for Samsung TV, where we design most of the operating system and the security framework and automotive great Linux, which is a Linux foundation initiative that was mostly pushed by the German, the, the Japanese automotive manufacturer, where we also did most of the technical contribution. And especially the security framework.
This being said, you know, when we talk about cybersecurity and security at large, I think everyone understands that, you know, security is more or less protecting the machine against human and safety is protecting human against the machine. Okay. More or less. So when we talk about cybersecurity, we mostly talk about security. Even if in the embedded world, you know, both are, are somehow limited. I would say whatever you do, it's always a question of mitigation on, on how, you know, you mitigate the complexity and the security, because obviously the more security you have, the more your process is complex. The less people are ready to accept it. So you have to find good compromise and security is, you know, it, it's always tempting to bypass your security model. Okay. And I think all of us, you know, sometime leaves the key of the entry door and there's a carpet in front of the house, or we, we write a, a long password on, on a PostIt and so on.
So all of us, we also bypass security from time to time. And we should not forget that it is a natural behavior from human to, to bypass the security, especially when the security is painful. In most of the case, people will claim they bypass security for performance reason. Honestly, it's very rarely the case on modern systems security costs few percent, I would say typically in between one and 3% of the performance for security. So it's rarely a, a big issue. Most of the time, people do not implement security because they have a lack of knowledge on how to do so. Another element that is potentially a good candidate as well, to bypass security's time to market and cost concern. Obviously the fact that you want to design your product with security built in, make it a little longer to, to, to put on the market.
And sometimes the, the marketing manager in fact, is the one who is killing the security. Another key element to take into account in this mitigation is the fact that the embedded security expert is very rare animal. Okay. We usually consider that we have a little less than 10 million mobile developer, roughly the same amount of web developer, probably a little less and only half a million of embedded developer. And on this half million, the question is how many embedded security developer we can hire? And I may tell you another is close of zero. Okay. So if you, if you try to hire an embedded expert that has a deep understanding of security process, you're probably not going to find it and further more on top of that. You, you need to make sure that your security's built in, in your process. So it means that you have to include the security from the architecture level, in the development, in QA and in the maintenance.
And it, it has to be, you know, built in API and it has to be transparent of application development. For the two reason I exposed before the fact that first people will try to bypass the security framework if they have to implement it themself. And the second thing is you probably not going to hire the right person to implement your security. So as a result, I would say from a development point of view, the security should not be the responsibility of the developer. Now, if we look for what we have to do, okay. And moving from theory to, to real implementation. When we look for, you know, a, a real embedded device, typically we have a set of software running a target, a target is typically a specific hardware. So we have to deal. Obviously the first thing is how we, we handle the secure boot, which is to know what we are booting, making sure that we are not booting the wrong OS, but on top of that, we also need to handle the identity management and some private secure store, which typically include a TPM, potentially some PKI infrastructure and so on.
And, and to be honest, the secure boot is well understood by a better developer, the identity management and private secure store far less well understood. Another key element in your, in your device is the Linux Nel that need to be up to date with your, you know, the latest batches and as well, we have a difficulty because as we using custom hardware, we have specific bot support package. So typically the low level software that enables the bot to run Linux, and two often zoo's board support package are very badly synchronized with upstream kernel. And here in house, we have some example where we have 500 patches to apply to the mainline kernel before running a board. And if you have this situation, obviously, and you need to be a bump of kernel version to response to a vulnerability, a known vulnerability, you know, reapplying, the 500 patch cannot happen in 24 hours.
Okay. It's probably going to take best case couple of weeks, which means that this delay your capability to respond Tod. Even if you have the answer, then on top of the kernel, obviously you have all the service and you need to harden all your services. Okay. It's and here again, it's not in the tradition of, of AMD software. Most AMD software tend to ship one image that is doing everything. So they ship one image that doing the kernel, the board support package, the security, everything, the communication, blah, blah, blah. And you have to split your, your, your implementation in different services in such a way that you can harden the services independently, one from the other. And obviously all of that needs to be based on, on modern API. Typically, we're talking here about open API model and you have to have a trusted middleware, so you can implement the security at the middleware level, which is the only way to run and trusted application.
Now, if we deep dive a little bit in that, what, in my opinion is the most important part of it. I would say the first thing is to create and adopt the right code. Okay. Even if today, most of the system, we integrate more codes that we write or sell some code. We have to make sure that we either create or adopt the right code. And for that, we need to have a very well integrated C I C D software factor. So content delivery, container integration, continuous integration. And, and we have to make sure that this work for the release management, that it under automatically holds the dependency, but also that it unders the functional and nonfunctional dependency. So typically a nonfunctional dependency is I don't want G PLV three in my configuration. So you have to make sure that if you pull a patch to correct an error, you don't introduce, you know, some, some G PLV three in your code, because this is going to, to taint your code.
Okay. You may also have functional nonfunctional requirement on this code needs to have 80% of test coverage. And so on, the test should be automatic and the reporting as well, not for each release, unfortunately today in the embedded world, most of the people only do test when they have a new release, that's a big error. You have to do the test for each commit. So the commit technically is every time a developer is pushing a new piece of code in, in the system. And you have to do that for all your version and for all your hardware. And unfortunately, it's not, it's rarely the case. And, and at the end, you know, by not doing that for every topic, you discover your a too late, and then it become very, very hard to fix them. When your test are done. Obviously you need to have an automatic image generation, either a full image.
When you want to fully refl the device, which is pretty common in the embedded world. But even more importantly for the future, you need to support incremental update. Because if you have updates that are coming every couple of days, or even every couple of weeks, you cannot reinstall systematically all the system for two reason. First, it's going to be too hard to sum the full image to your devices. And the second thing is going to write too much on the flash, which is the memory, the disc memory of the board. So you're going to kill your flash and your flash is not never going to last 10 years, if you do do that. And the last element is everything has to run in containers. And this is going back to my point with, you know, you have to make sure your service are insulated. So you have to make sure they're run in containers, which means that your software factory should be able to understand what is a container, how they are run and what are the depends in between the different containers.
And you have to do this testing, obviously as much as possible on real board, but because you never have enough real board to do the test, you, you have to do. In the meantime, you have to do things on virtual board. So you, you have to have a mixed where it is relatively transparent to move from virtual board to real board, because we never have enough. Real boards runs. The right code is a different thing. Now we've done the code. And the question is, do we boot a record? And for that, obviously you need to have a trusted boot. That's a must, a feature. Many people in the embedded world are talking about. We want to boot fast, like in the automotive to say, we want to boot in less than two second or less than five seconds, but you have to also trust your, your boot.
So you have to have the fast trust boot and to do that. You have to leverage usually also hardware capabilities that you have on your boat. And you have to make sure that if you have keys, which is very common, like you have signature on your kernel, on your packet and so on, you have to make sure that you have small series, because if one of your keys compromised, you're not willing to have all your fleet to be compromised, but you want to limit that to a subset of your feet at application installation. You want to verify not only the integrity, but also the origin, which is where the package is coming from. And obviously the identity of the person installing the package. And potentially you may also have to request for privacy and permission, to be honest in the embedded world, this is rarely a hard topic.
Okay? Most of the time we don't ask permission to anyone with just do the job, but checking the identity for example, is really done, which is a very important process because you have to make sure that the fact you have the USB key or, you know, the URL of the system is not enough for you to, to update the system on the update side, you more or less have the same issues that you have at installation time. So you also need to have the trusted installation. You need to have the trusted route, and you want to have incremental update. As I said before, because if you do frequent updates, you only impact the subset of your system. So this reduce the amount of tests you have to do. It also reduce balance width because sometime, you know, sending code to, to embedded device is not that easy.
It's not everywhere where you have a 4g or a 5g connection. And also we have some issue on the flash. And if you right too much on the flash, you're going to kill it. And the last point on the update is to make sure that your container model does not create black box. Okay. Sometime I also call them a, you know, diplomatic suitcase, which more or less, you know, the color of the containers, but you have no idea what is running inside. And you have to make sure that also your container don't, you know, import some vulnerabilities that you don't see because you, you have black box container and not white box containers, and you need to have obviously security rules. And if you want them to be accurate, you need to generate the rules automatically, which means that you need to have a higher level of description of your security model and you should generate automatically the rule.
And we are especially talking in this case, the mandatory access control rules for Linux ORM, the, the sec comp or the cgroups, but also the namespace and the container, as I said before, container should be part of your global security model. And it should not be a black box that, that runs something you don't understand on your system in order to do so. The only valid way is to have a consistent, secure microservice architecture. So if you don't have this kind of, of factorization and globalization of your development, it is just impossible because developer are not going to implement for each services or security model or transport layer and identity model. So you want this to be provided by your platform. And for that, you need to have a microservice architecture and your microservice architecture should have some, some specific characteristic. The first one is that it should cross domain.
So you should be able to move from the edge to the cloud, to the real time operat system. If you have some safety devices, this, this should be done by your microservice architecture, you should have a built in security model. So your microservice architecture should be able to leverage the kernel security capability, but also the social or the local identity to have a generic model for debugging and monitoring and model for the installation of the containers. And last but not least, if you want to cross domain, you have to be transport agnostic, which means that you should not make the assumptions that everyone is. Everything is going to be transported as Jaen, or has a, you know, a flat, flat pad buffer. But you need to have a model where you can use the best transport for the system you want to talk to with some specific issue in the embedded world, that sometimes the bandwidth is, is both unstable and very limited.
This is especially through in when we work on boat and ship, where we, we go through satellite link, but generally, you know, we don't have a, a good connectivity. When we start talking about devices. The last point is whatever you're going to do, you will have some bug, some errors. And so you need to monitor, and you need to understand what happened in your system. So you need to have a supervised middleware, which mean you need to have some introspection hook on your API. You need some to have some filtering and some log, some capability to send all the log on the cloud. Here, we have an example where we push further log on, on a local database with red, and then we replicate that database on the cloud. So we can run some AI process. This graphic, in fact, is for predictive maintenance on, on the boat.
You need to have some reporting tools and a very important element here is a time series database, because that's the only way we have to, to aggregate element and issues coming from different component and to make a correlation in between them. Okay. So time 30 database is clearly a very important point. Static security reporting is also something you should try to do as much as, as you can. Security is a complex feature auditing and certification is very important. And so the only way to be able to do some monitoring and some efficient certification is by having some, you know, setting model where you can implement some, some report easily without having to dig to deep dive in, in the code of your application. Obviously. So my conclusion on that is there are three elements that are, I found out extremely important. So first system is you should keep your system under control.
And for that, you have to simplify and standardize your architecture. And because the system are more and more complex, okay. To DEC is 100 million lines of code. And we are talking about 300 to 500 million lines of code for fully autonomous car. So we have a lot of complexity coming in. So you need to have middleware. You need to have standard API standard service, abstraction, the transport attraction and the security. And so on. The second element to keep your system outta control is to develop everything and a strict C I C D control. So continuous integration, continuous delivery. You need to have that every commit of the developer should generate an event on your system. That look for all the dependency, reapply, the test, whether they are functional or nonfunctional, you have to obviously have automatic image release generation. If you want, especially to do partial update, you should be able to understand what's the difference in between version one, one and version one 11.
So you can generate the diff and you can push the diff on your system. And you have to assert that your code is generic. And here, my recommendation is your, your system should at least run on two different hardware. Even if you only have one hardware targeting on your system, it's a very good behavior in the development process to force the developer, to run the code on two hardware, you find a lot of bug by doing that. Then you have to make sure your security is editable. So you need to have an automatic security rules, privilege generation here. Again, do not rely on human. You need to have an automatic mapping of the CBE. So the CBE has the, the well known vulnerability on Linux. So you need to make sure that you can move automatically from a known CBE to the applying of dispatch for the correction.
And in input that in your C CD factory, you need to have dynamic allocation of framework and, and APIs supervision monitoring. Because to be honest, at the beginning, you don't really know what you're looking for. So you probably will have to change the rule of monitoring when the system go live. And, and probably you're going to, to monitor stuff that were not expected to be monitor at the origin. So you need to be able to change that dynamically, and you have to enforce a simple and generic static privilege model for all services. There is one exception of the static privilege model is identity. Okay. When you have to handle the identity of user to an action of, usually this is dynamic because you cannot provision in advance, which user is going to do something. You have to run everything insulated because whatever, and how smart you are to implement your protection, you will always have some failure.
So you have to make sure that if one component goes wrong, it does not kill your entire system. You want your system to be resilient, which potentially is even more important than being secure, which in all case translate in one container per service, we have hardware isolation for safety related tasks. So typically we run to safety task in, in a microcontroller that is often embedded in directly in the OCC, like R seven or M five on arm architecture, potentially it might be an external component, but on modern hardware, it's very often embedded in. You need to have an atomic container update model. So if you have a burger on the container should be able to fix that container without having to fix your full system, which prevent to redo all the test. As I said before, and also also have to make sure that you don't run the same software company twice in your platform. That's a very common issue with containers where you have two or three times the same software component implemented on your, on your platform. And they have different version. And that's very dangerous from a security point of view, because obviously it means that your system is not under control. So to sum up my talk and I'm one minute late, I would say that complexity and adult process are your enemy. And you have to make sure that you don't rely on human for your security, but you rely on process. Okay. And that's my last slide.
