GitHub - zeroFhacker/promptfuzz: Automated red-team framework for LLMs evolves jailbreaks with genetic…

David S. — Information Security Engineer / Security Operations Consultant

2026-09-22

PromptFuzz v2 has been released and uploaded to my github after months of hands-on LLM red-teaming.

Spent the last few months actually breaking LLM applications in production environments, RAG pipelines leaking documents across users, MCP tool descriptions executing embedded instructions the model was never supposed to follow, multi-turn conversations slowly steering models past boundaries that single-turn scanners never caught.

v1 was a payload generator. That wasn't enough. v2 is built around one question: can you prove a specific attack path was fixed, not just that an attack failed?

Every finding now produces a full causal chain:

entry point → adversarial input → model decision → tool call → policy boundary → observed outcome → fix → replay result

Each finding gets a stable deterministic ID. Run the scan after applying a fix, if the ID reappears, the path is still open. If it doesn't, it's closed. That's the difference between a security test and a regression gate.

What's in v2:

Multi-turn Crescendo engine: 5 escalation strategies with per-turn state, full conversation history, breakthrough-turn tracking. Single-turn classifiers miss this entirely

MCP tool poisoning scanner: static analysis of tool descriptions (11 patterns) plus dynamic model susceptibility tests. The model sees a poisoned tool definition and either follows the embedded instruction or it doesn't.

RAG context injection + document leakage: canary-based. The canary appears in the response or it doesn't. No LLM judge guessing

Agent memory integrity: delayed activation, precedence confusion, cross-user namespace, stale reuse. Records the full causal path from write to action

A2A trust boundary: rogue agent cards, task injection, authority escalation between agents

Autonomous attacker mode: configurable attacker model proposes and revises payloads from feedback, policy controller enforces hard ceilings

Compliance mapping: OWASP LLM Top-10 2025, MITRE ATLAS, EU AI Act (descriptive mapping, not certification)

SARIF output, exit codes, diff command for baseline comparison

Runs fully offline. No API key needed to get started.

pip install -e ".[dev]" then promptfuzz scan -t mock -m mock-model --profile agent

GitHub: https://github.com/zeroFhacker/PromptFuzz

PromptFuzz v2 - open-source LLM red-teaming with an Attack-to-Fix Evidence Graph

Built a Python tool that connects every finding to a full causal chain: entry point → adversarial input → model decision → tool call → boundary crossed → fix → replay result. Each finding gets a stable ID so you can prove a fix actually closed the path.

10 modules: multi-turn Crescendo, MCP tool poisoning scanner, RAG injection, A2A trust boundary, agent memory integrity, autonomous attacker, and OWASP LLM Top-10. Runs fully offline- no API key needed.

GitHub: https://github.com/zeroFhacker/PromptFuzz

Open to contributors working on serious LLM security challenges, Please recommend me for job opportunity.

2 reactions · 1 comments

Read and join the discussion on Cyberr