How a Working Exploit Made Me Forget How to Threat Model

We talk a lot in AppSec about “shifting left” and automated discovery, but we rarely talk about the human blind spots those tools create.

Recently, we were testing Rowan, our proprietary SAST platform built specifically for AI/ML security. It does the heavy lifting: inter-procedural call graphs, taint flows, and chain detection for composite vulnerabilities. When it works, it is a beautiful thing. It is the kind of tool that can surface a path-traversal buried in an LLM framework’s prompt loaders, the sort of issue conventional scanners walk right past.

But this post isn’t a victory lap. It’s a post-mortem. It is a story about what happens when your tooling is faster than your threat modeling, and why getting a shell doesn’t automatically mean you’ve found a vulnerability.

The Dopamine Hit and the Red Team Ego

I am a massive advocate for threat modeling. I firmly believe security starts at the whiteboard. Before you look at a single line of code, you have to ask: Who is the adversary? What is the asset? What is the stated trust boundary?

But let me be completely honest: when you see a raw, un-sanitized deserialization sink, build the payload, and catch a reverse shell, the dopamine hits hard. The sheer technical success of the exploit creates a massive confirmation bias. You forget the whiteboard entirely.

During a routine scan of an open-source ML library, Rowan flagged an inter-procedural taint flow from a user-controlled file upload directly to an unsafe deserialization call. We fed this into our automated review pipeline, generated three candidate exploit vectors, and went to work. We spun up a private replica of the target, got our shell execution, confirmed persistence, and confidently submitted a high-severity RCE report.

“This is expected behavior. The utility is built to execute these files.”

My initial reaction was typical red-team ego. I spent hours building a rebuttal package with live exploit harnesses, environment dumps, and undeniable proof of impact.

Then, finally, I stopped and actually read the documentation.

The RTFM Failure

The library’s documentation was precise. It framed its format as a restricted alternative to raw serialization, explicitly stating it wasn’t a complete security guarantee. For the specific utility we exploited, the docs carried a prominent warning: it must load legacy formats to function, and users should only run it on files they absolutely trust.

The triager wasn’t blowing us off. They were exactly right. By ignoring the documented boundaries, I had effectively submitted a bug report complaining that eval() executes code.

The Lens: Dropping the By-Design vs. Keeping the Silent Opt-In

This failure forced me to completely re-evaluate how I triage bug bounty findings, specifically in the data science and ML ecosystem where “unsafe deserialization” is a notoriously noisy flag. I had to build a clear lens to separate a known risk from a broken security claim.

• × Drop: The Documented Risk You have to drop findings where the danger is an acknowledged architectural reality. If a CLI utility warns users to only parse trusted files, that is a user-awareness issue, not a CVE. If a library has an internal data-handling function literally named unpickle_data(), the developer has already conceded the design. Blanketing these with “deserialization is dangerous” reports exhausts engineering teams and burns your credibility.
• ✓ Keep: The Silent Opt-In The findings that survive scrutiny are fundamentally different. The vulnerability isn’t just “this library uses unsafe deserialization.” The true vulnerability is when a library silently opts the user into a dangerous default without consent, visibility, or a documentation warning. Imagine a widely used deep learning framework that provides a simple function to download and load standard tutorial datasets. Under the hood, if that function hardcodes an allow_untrusted=True flag and passes it to a raw deserializer, with no gate, no allowlist, and no warning in the docstring, that is a critical finding. The user calls a harmless-looking load_data() function with zero visibility into the fact that their trust boundary has just been dissolved.

When you frame a finding this way (proving you know how to drop a documented risk, but highlighting a silent opt-in) your argument becomes unassailable.

Vector 1: The Reference Web App (The Actual Vulnerability)

Ironically, while our primary exploit was a documented feature, the blanket dismissal swept up a finding that actually deserved a closer look.

The target’s reference web application routed uploads through a single, fragile trust check:

if filename.endswith(TRUSTED_EXTENSION):
    model = restricted_load(file_bytes, trusted=audit_types(data=file_bytes))
else:
    model = legacy_deserialize(file_bytes)   # ← any other extension lands here

This is textbook fail-open design. The entire trust boundary is a string suffix check. If your file doesn’t have the exact trusted extension, it falls straight into raw, unrestricted execution.

Unlike the local developer utility, this public-facing web app had no documentation warning users that uploads would be executed. The reasonable expectation is that a public upload interface won’t detonate untrusted input.

if not filename.endswith(TRUSTED_EXTENSION):
    raise ValueError("Unsupported format. Only trusted format files accepted.")
model = restricted_load(file_bytes, trusted=audit_types(data=file_bytes))

Vector 2: Path Traversal (Defense-in-Depth)

Another vector Rowan found involved a write function that accepted caller-provided paths without sanitizing ../ sequences. But looking at the adversary context: for this to be exploitable, an attacker has to control the path argument. In standard usage, the developer controls this. It’s only a vulnerability if the implementing application passes raw user input to it. This is a hardening suggestion, not a standalone CVE.

import os
from pathlib import Path

def write_with_boundary_check(model, destination, allowed_dir):
    resolved = Path(os.path.realpath(destination))
    boundary = Path(os.path.realpath(allowed_dir))
    if not str(resolved).startswith(str(boundary) + os.sep):
        raise ValueError(f"Path outside allowed directory: {destination}")
    library_write(model, resolved)

The ML Security Blind Spot

The ML ecosystem currently relies on a universal disclaimer: deserializing untrusted data is your problem, not ours.

When triagers see “unsafe deserialization” in a report, they instinctively close it as a known risk. Our web app finding got caught in the crossfire because we led with the documented utility, and the triager shut the door before looking at the context.

If you want a security report to land, you cannot just say “unsafe code is dangerous.” You have to open with a claim:

“[Library] claims to protect against [specific threat] via [specific mechanism]. This report demonstrates that [specific input] bypasses that protection because [specific reason].”

Tools like Rowan map the flows. They find the sinks. But threat modeling (understanding what a system actually claims to protect) remains a human responsibility. Next time, I’m reading the README before I drop the shell.

Pragmatic Allowlisting for Legacy Formats

If you are stuck supporting legacy formats and cannot drop them entirely, do not use a blocklist. Blocklists for dangerous modules will inevitably fail against bypasses like builtins.eval. Use a strict allowlist that explicitly permits only the modules your application requires.

# Audit and expand this list for your specific model types before production use
ALLOWED_MODULE_ROOTS = {
    "your_ml_framework",
    "numerical_library",
    "dataframe_library",
}

ALLOWED_BUILTINS = {
    "list", "dict", "tuple", "set", "frozenset",
    "str", "int", "float", "bool", "bytes", "bytearray",
    "type", "object", "slice", "range",
}

class RestrictedDeserializer(BaseDeserializer):
    def resolve_class(self, module, name):
        if module == "builtins":
            if name not in ALLOWED_BUILTINS:
                raise DeserializationError(
                    f"Blocked builtin: builtins.{name}"
                )
            return super().resolve_class(module, name)
        root = module.split(".")[0]
        if root not in ALLOWED_MODULE_ROOTS:
            raise DeserializationError(
                f"Blocked module: {module}.{name}"
            )
        return super().resolve_class(module, name)