Presentation overview
This presentation argues that a plausible vulnerability explanation is not security evidence. Instead of treating an LLM as a final judge, it proposes using the model to generate hypotheses while execution, replay, tests, sanitizers, and observable effects determine whether a finding is real.
Topics covered
- Why low vulnerability prevalence can make seemingly small false-positive rates operationally expensive.
- How feedback-guided and stateful protocol fuzzing turn hypotheses into executable experiments.
- How ChatAFL uses an LLM for protocol priors while retaining the target runtime as the source of truth.
- How verification-oriented systems such as FirmAgent, Big Sleep, and Codex Security connect search, tool use, sandboxed execution, and reproducible evidence.
Core takeaway
The research target should be the verification system around the model, not the model’s confidence alone. A useful security agent narrows the search space, chooses the cheapest falsifying experiment, discards findings that cannot be reproduced, and operates under strict sandboxing, provenance, and least-privilege controls.