Here is what matters most regarding artificial intelligence (AI) in cybersecurity: Outcomes. The threat landscape is rapidly evolving, with generative AI now augmenting the toolsets available to defenders and attackers alike. As a result, evaluating the relative effectiveness of AI-based security оfferings has become more essential and complex. Asking the right questions can help you spot solutions that deliver value and ROI instead of just marketing hype. Questions like, “Can your predictive AI tools sufficiently block what’s new?” and “What signals success in a cybersecurity platform powered by artificial intelligence?” As BlackBerry’s AI and ML (machine learning) patent portfolio attests, the vendor is a leader in this field and has developed an exceptionally well-informed perspective on what works and why. This blog will pull back the covers and explore this timely topic.

Evolution of AI in Cybersecurity

Some of the earliest ML and AI uses in cybersecurity date back to the development of the CylancePROTECT® EPP (endpoint protection platform) more than a decade ago. However, predicting and preventing new malware attacks is more crucial today, as generative AI helps threat actors rapidly write and test new code. The most recent BlackBerry Global Threat Intelligence Report uncovered a 13% surge in novel malware attacks, quarter over quarter. Preventing these attacks is an ongoing challenge, but thankfully, technological advancements meet their evolution.
BlackBerry’s data science and machine learning teams continuously improve model performance and efficacy for the company’s predictive AI tools. Recent third-party tests reveal that CylanceENDPOINT® blocks 98.9% of all threats because they can actively predict malware behavior, even if it’s a brand-new variant. Achieving this level of efficacy is difficult, and requires precise model training on the correct indicator types.
Over the past decade, BlackBerry has continuously innovated, experimented, and evolved its AI to produce the best outcomes. Some major shifts include moving solely from supervised human-labeling in our early models to a composite training approach, including unsupervised, supervised, and active learning – both in the cloud and locally on the protected endpoints. The vendor also optimized the attributes and datasets utilized to provide the best predictive outcomes, after examining massive volumes of data over time. The result of this continuous evolution has been a very well-fit model for real-world application: One that can accurately predict and anticipate new threats.

Temporal Advantage: Taking Time Into Account

Today, discussions regarding the quality and efficacy of ML models often revolve around the model's size, number of parameters, and performance in established test data without considering the most crucial outcome good AI can provide: time. In certain domains — like language, vision, object categorization, and identification tasks — time is not a critical attribute for evaluation. However, in cybersecurity, time matters a great deal. It is essential for threat detection in the context of malware pre-execution protection. This is where models identify and block malware before it deploys and executes.
Along with making adversarial behavior predictions through machine learning, model validation must consider temporal resilience, where it proves itself effective against both past and future attacks. One of the most important metrics in this context is the model's predictive advantage over time. Temporal Predictive Advantage (TPA) is a term used within our data science team to assess the performance of our models against future threats. This concept dates to the evaluation of security algorithms or cipher designs that measure cryptographic time invariance — in other words, whether the system's response to input is predictable and correct, regardless of when that input signal occurs. Here is an example: Given that we cannot fast-forward or rewind time, we train the models using malware classes from the past and test them against newer malware from the present. This temporal testing aims to validate generalized performance over time, which is crucial for detecting zero-day protection. It helps us train the model architecture and assess its ability to learn and detect malicious intent.
It is reasonable to ask, “Why does it matter?” After all, models can be updated frequently in the cloud, which is typically where most models are served. However, there are many endpoints — for example, in IoT, regulated industries, or disconnected and even intentionally air-gapped endpoints — that are not cloud-connected. Updating models may not always be feasible in these cases. In ML models that are heavily cloud-dependent, a loss of connectivity can greatly decrease detection rates. However, recent third-party analysis reveals that the way the BlackBerry Cylance model was built, malware detection and ongoing protection occur at the same level regardless of connectivity. It is cloud-enabled, but not cloud-dependent.

And here is a further important note: If your vendor pushes out frequent model updates, it may indicate the immaturity of an ML model. Without the updates, that model might experience faster drift — the loss of predictive power — and experience rapidly increasing malware class misses. In contrast, the chart below illustrates the temporal predictive advantage (TPA) in months, when the fourth-generation Cylance model was tested against newer classes of malware. How long into the future did it detect and block threats without a model update?
The protection continued for up to 18 months without a model update. Again, this reveals model maturity and precise model training. This does not happen accidentally.

Chart 1 — The temporal predictive advantage for the fourth-generation Cylance AI model. It reveals how long into the future the protection will continue without a model update. In this case, for six to 18 months.

Mature AI Predicts and Prevents Future Evasive Threats

CylanceENDPOINT has a novel ML model inference technology that sets it apart. It can deduce, or “infer” whether something is a threat, even when it has never seen it before. Our approach utilizes a unique hybrid method of distributed inference, a concept that was conceived seven years ago, before the availability of ML libraries and model-serving tools. The result of this approach is our latest model, which represents the pinnacle of innovation and improvements over the many generations of this technology.

To see how mature AI detects evasive malware, see the chart below and look at the detection of Sality and Parite malware as examples. These are polymorphic malware variants, which create multiple versions of themselves in an attempt to avoid detection. These are very difficult to detect using conventional methods, such as signatures and heuristics, or with immature machine-learning methods. All AI and ML models are not created equal, regardless of how they are marketed.
RedLine Infostealer emerged in early 2020 as a malware-as-service with high replication. It frequently hides in ChatGPT, Bard, Facebook ads, etc.

Chart 2 — This graph shows 30 days of malware classes observed in our system. Some of these classes are new and updated and have not been seen before. However, the model detects them, regardless.

Could it be readily detected? The charts below show model performance against two classes, polymorphic malware class Sality, and malware-as-service class RedLine. (Note: PE4C, PE6E, and PE7D are model generations, with PE7D being the most recent).

Charts 3a and 3b — These graphs show multiple Cylance model generations and their high conviction rates of polymorphic Sality malware (left) and RedLine Infostealer (right), even when the ML model is several years old.

The models’ temporal predictive advantage comes through quite nicely in these results. The sixth-generation model, released more than three years ago, decisively identifies, or “convicts” the malware, and the newest model update convicts over 99% of RedLine’s newer variations (in this chart, more is better).
Below are some additional charts of model performance against specific classes of interesting ransomware, such as LockBit (left) and Maze ransomware (right). These results show a similar ML model’s predictive advantage. It is also worth noting that our most recent model version, PE7D, convicts more than 99% of these variants.

Charts 4a and 4b — These graphs show multiple Cylance model generations and their high conviction rates of LockBit (left) and Maze malware (right), even when the ML model is several years old.

Now let’s broaden out these results and look at the overall performance of the models against all malware classes observed during a 30-day timespan. See the chart below, where less is better.
The chart above shows the percentage of false omissions. This refers to when the model does not convict a malware strain because it incorrectly identifies it as exhibiting benign behavior. A lower score is better here since it captures the omission rate as a percentage of the total. BlackBerry’s newest generation model’s miss rate is less than 0.005%, and results are below 0.01% for the previous generations, across all categories, observed over 30 days on recent malware types.

Chart 5 — This graph reveals extremely low false omission rates across various Cylance model generations. This is when a model fails to recognize malware as malicious, so lower is better.

Predicting Malware: The Most Mature Cylance Model

With access to vast and diverse datasets, comprising petabytes of data and observed malware behaviors over time, BlackBerry’s most recent model is also the most powerful version to date. It has outperformed all its predecessors across various performance indicators, including temporal predictive advantage.
After well over 500 million samples evaluated across billions of features, and inference results from the previous generations capturing insights over time, BlackBerry Cylance AI continues to deliver highly desirable results. And it boasts impressive speed as it supports distributed inference, both locally and in the cloud. BlackBerry’s application of machine learning in cybersecurity is well established, and the company is excited about the journey to drive innovation in this field. As adversaries continue to adopt AI, it is more important than ever to ensure your defensive cybersecurity posture focuses on outcomes that matter. Cylance AI has protected businesses and governments globally from cyberattacks since its inception, with a multi-year predictive advantage. BlackBerry’s Cylance AI helps customers stop 36% more malware, 12x faster, and with 20x less overhead than the competition. These outcomes demonstrate that not all AI is created the same. And not all AI is Cylance AI.
If you still have questions about Cylance AI or are interested in BlackBerry solutions, please email moc.hcetokab%40yrrebkcalb and our engineers will be happy to help.