How to Train Your AI to Mis-Identify Dragons

This week Skylight Cyber disclosed that they were able to fool a popular “AI”-based Endpoint Protection (EPP) solution into incorrectly marking malware as safe. While trying to reverse-engineer the details of the solution's Machine Learning (ML) engine, the researchers found that it contained a secondary ML model added specifically to whitelist certain types of software like popular games. Supposedly, it was added to reduce the number of false positives their "main engine" was producing. By dumping all strings contained in such a whitelisted application and simply appending them to the end of a known piece of malware, the researchers were able to avoid its detection completely, as shown in their demo video.

This finding is just another confirmation of inherent challenges of designing ML-based cybersecurity products. Here are some issues:

  1. The advantages that ML-enhanced cybersecurity tools provide can be easily defeated if overrides are used to eliminate false positives rather than proper training of ML algorithms. ML works best when fed as much data as possible, and when products are implemented using the right combination of supervised and unsupervised ML methods. It’s possible that whitelisting would not have been necessary if sufficient algorithmic training had been performed.
  2. ML can be gamed. Constraining data sets or simply not having enough data piped through the appropriate mix of ML algorithms can lead to bias, which can lead to missed detections in the cybersecurity realm. This can be either intentional or unintentional. In cases of intentional gaming, malicious actors select subsets of data with which to train the discriminator, while purposely omitting others. In the unintentional case, software developers may not have access to a full sample set or may simply choose to not use a full sample set during the construction of the model.  
  3. Single-engine EPP products are at a disadvantage compared to multi-engine products. Using “AI” techniques in cybersecurity, especially in EPP products, is an absolute necessity. With millions of new malware variants appearing monthly, human analysts can’t analyze and build signatures fast enough. It is infeasible to rely on signature-based AV alone, and this has been true for years. However, just because signature-based engines are not completely effective doesn’t mean that products should abandon that method in favor of a different single method. The best endpoint protection strategy is to use a mixture of techniques, including signatures, ML-enhanced heuristics, behavioral analysis, sandboxing, exploit prevention, memory analysis, and micro-virtualization. Even with an assortment of malware detection/prevention engines, EPP products will occasionally miss a piece of malicious code. For those rare cases, most endpoint security suite vendors have Endpoint Detection & Response (EDR) products to look for signs of compromise.
  4. Marketing ML-enhanced tools as an “AI” panacea has drawbacks.  ML tools are a commodity now. ML techniques are used in many cybersecurity tools, not just EPP. ML is in most data analytics programs as well. It’s a necessary component to deal with enormous volumes of data in most applications. The use of the term “AI” in marketing today suggests infallibility and internal self-sufficiency. But such tools can make mistakes, and they don’t eliminate the need for human analysts.   

KuppingerCole is hosting an AImpact Summit in Munich in November where we’ll tackle these very issues. The Call for speakers is open.

For an in-depth comparison of EPP vendors, see our Leadership Compass on Endpoint Security: Anti-Malware.


Related Events

AImpact Summit 2019

Conference

AImpact Summit 2019

Holiday Inn Munich - City Centre, Munich

Discover KuppingerCole

KuppingerCole PLUS

Get access to the whole body of KC PLUS research including Leadership Compass documents for only €800 a year

KuppingerCole Select

Register now for KuppingerCole Select and get your free 30-day access to a great selection of KuppingerCole research materials and to live trainings.

Stay Connected

Blog

Spotlight

AI for the Future of Your Business Learn more

AI for the Future of Your Business

AI for the Future of your Business: Effective, Safe, Secure & Ethical Everything we admire, love, need to survive, and that brings us further in creating a better future with a human face is and will be a result of intelligence. Synthesizing and amplifying our human intelligence have therefore the potential of leading us into a new era of prosperity like we have not seen before, if we succeed keeping AI Safe, Secure and Ethical. Since the very beginning of industrialization, and even before, we have been striving at structuring our work in a way that it becomes accessible for [...]

Latest Insights

How can we help you

Send an inquiry

Call Us +49 211 2370770

Mo – Fr 8:00 – 17:00