A new arXiv preprint from researchers at Palo Alto Networks reports that XGBoost cybersecurity classifiers score a near-perfect 0.98 robustness against one class of adversarial attack — then collapse to 0.36 under a different method [S1]. The gap exposes a dangerous illusion in how security teams test their own defences, and raises a question every SOC analyst should be asking: if your model's prediction survives an attack but its explanation doesn't, can you still trust the alert?

The number that lies

The preprint, published 5 July on arXiv and not yet peer-reviewed, extends the authors' earlier conference work on multi-layer perceptrons to two of the most widely deployed tree-based models in cybersecurity: Random Forest and XGBoost [S1]. The team tested across four standard tabular security datasets — phishing URLs, UNSW-NB15, NF-ToN-IoT, and HIKARI-2021 — and evaluated five adversarial attacks, including three black-box methods designed for models that, unlike neural networks, can't be differentiated [S1].

The headline finding is a stark reversal. When the researchers ran ZOO — a gradient-based black-box attack — against XGBoost, the model appeared almost unbreakable, returning a Robustness Index of approximately 0.98 [S1]. By that metric, the classifier was near-perfectly defended.

But that number is an artefact, not a result. Tree models like XGBoost produce piecewise-constant prediction surfaces — step functions, not smooth curves — so gradient-based attacks that estimate slopes simply can't get traction [S1]. The attack fails not because the model is robust, but because the attack's machinery doesn't fit the terrain.

When the team switched to Square Attack, a score-based method that doesn't rely on gradient estimation, the picture inverted: XGBoost's Robustness Index dropped to approximately 0.36 [S1]. A model that looked bulletproof was, in fact, deeply vulnerable.

The explanation breaks first

The more unsettling finding sits one layer deeper. The authors introduce a new metric called the Explainability Stability Index (ESI), which measures how much a model's SHAP-based feature attributions — the explanations that tell analysts why a prediction was made — drift when the input is adversarially perturbed [S1]. The ESI runs on the same 0-to-1 scale as the Robustness Index, so the two can be read side by side [S1].

Here's where it gets uncomfortable. Even those degenerate ZOO perturbations — the ones that failed to change XGBoost's predictions and produced that misleading 0.98 robustness score — still drove substantial drift in the model's explanations, yielding an ESI of approximately 0.06 to 0.16 [S1]. Random Forest fared slightly better but still showed ESI values of roughly 0.14 to 0.29 under attack [S1].

In plain terms: an attacker who can't fool your model into making a wrong prediction can still scramble the explanation your analyst sees on screen. The alert fires correctly, but the reasons attached to it are unreliable — and in a security operations centre, a wrong explanation can waste hours of triage time chasing phantom causes.

The authors frame this as a dual threat: adversarial attacks on cybersecurity classifiers degrade predictions and destabilise the SHAP-based explanations that security analysts rely on to understand and triage alerts [S1]. Prediction robustness and explanation stability, they argue, are distinct axes that must be measured jointly [S1].

A two-axis map of attack effectiveness

To make sense of why different attacks produce such wildly different results against tree models, the authors propose a two-axis framework based on gradient dependence and query efficiency [S1]. This framework explains the observed attack rankings and offers practical guidance for evaluating tree ensembles — the idea being that security teams should test against attacks that match their model's architecture, not just the ones that are easiest to run.

The paper also includes a step-size ablation that resolves a counterintuitive anomaly with PGD (Projected Gradient Descent — a standard white-box attack that nudges inputs along the model's gradient) on z-score normalised tabular data [S1]. A detail that matters less for the headline but will interest practitioners who've seen PGD behave oddly on tabular features.

What it means

For anyone running tree-based classifiers in a security pipeline, the takeaway is blunt: a high robustness score against one attack class tells you almost nothing about your real exposure. If your red team only runs gradient-based methods against an XGBoost model, they'll report it's nearly invincible — and they'll be wrong. The model's piecewise-constant prediction surface defeats the attack's gradient estimation, not the attack itself.

The ESI metric is the more important contribution. It formalises something practitioners have suspected but couldn't measure: that adversarial perturbations can corrupt the explanation without changing the prediction. For a security analyst using SHAP values to decide whether an alert is a genuine intrusion or a false positive, a destabilised explanation is a silent failure — the model still says "malicious," but the reasons it gives are unreliable. That's the kind of gap an attacker can exploit to slow down response times or misdirect investigation.

What it means for business

For a two-person security firm running XGBoost for phishing URL detection, the practical implications are immediate:

  • Re-evaluate your adversarial testing. If your robustness assessments only use gradient-based attacks, you're likely overestimating your model's resilience. Score-based attacks like Square Attack are the ones that will expose real weaknesses in tree models.
  • Don't trust SHAP explanations under attack. If an adversary is actively perturbing inputs — crafting a phishing URL to evade detection, for instance — the SHAP values your analyst sees may not reflect the true drivers of the model's decision. Build workflows that treat explanations as potentially compromised during an active attack window.
  • Budget for joint measurement. The paper's core argument is that robustness and explainability stability need to be tracked together. For a small team, that means adding an ESI-style check to your model evaluation pipeline — not just "did the prediction hold?" but "did the explanation hold?"

For larger security vendors, the two-axis framework offers a way to prioritise which attacks to include in adversarial test suites based on a model's architecture, rather than running every available method indiscriminately.

What we don't know yet

This is an unreviewed preprint; all findings — the 0.98 and 0.36 robustness figures, the ESI ranges, the two-axis framework — are self-reported by the authors and have not undergone independent peer review [S1]. The paper is cross-listed on arXiv under both cs.CR (cryptography and security) and q-fin.GN (quantitative finance general news), an unusual categorisation that likely reflects an arXiv sorting quirk rather than a substantive finance angle [S1].

Several questions remain open:

  • Generalisation beyond the four datasets. The results cover phishing URLs, UNSW-NB15, NF-ToN-IoT, and HIKARI-2021 — all tabular. Whether the ESI framework transfers to other data types or security domains is untested.
  • Real-world attack feasibility. The study measures robustness under controlled perturbation budgets. How easily an attacker can craft those perturbations in a live system — where inputs are constrained by protocol formats and behavioural patterns — is a separate question.
  • Defensive countermeasures. The paper diagnoses the problem and proposes a measurement framework, but doesn't evaluate whether adversarial training or explanation-stabilising techniques can close the gap.

The next concrete signal to watch for is whether this work survives peer review and whether independent teams reproduce the ESI findings on additional datasets. Until then, the core insight stands on its own logic: if you're only measuring one axis of robustness, you're measuring the wrong thing.

If that gap between what your robustness score says and what your model actually survives keeps you up at night, there's more where this came from — subscribe and we'll keep decoding the research that matters.

Sources

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.