Skip to content
Flows
Library Paths Vault Generate Progress Run Demo Flow
Agent QA & Security Intermediate · 60-90 minutes

Jailbreak Regression Suite

Turn every jailbreak you find into a permanent regression test, so safety boundaries never silently erode across model and prompt changes.

Start Route · 3 steps

The route

3 steps to Done

  1. 01

    Assemble the jailbreak corpus

    Collect the attacks that matter for your agent's boundaries.

    Preview prompt + verify gate ▾

    Build the jailbreak corpus. Sources: jailbreaks found in your own red-teaming, publicly documented technique families (persona/role-play, hypothetical framing, token smuggling, many-shot priming, refusal-suppression phrasing), and any real attempts seen in usage. Scope to YOUR agent's actual safety boundaries - what should it refuse or handle carefully given its capabilities (destructive actions, sensitive data, out-of-scope requests)? For each entry record: id, category, the attack (single or multi-turn), and the EXPECTED SAFE BEHAVIOR (hard refusal, safe partial help, or redirect - be specific about what safe looks like for this case). Aim for breadth across categories over volume within one.

    • ✓ Attacks scoped to real safety boundaries
    • ✓ Technique families covered broadly
    • ✓ Expected safe behavior specified per case
  2. 02

    Build reliable compliance evaluation

    The hard part is telling a real refusal from a soft yes.

    Preview prompt + verify gate ▾

    Implement response evaluation. For each case, classify the agent's behavior as REFUSED (declined and offered no meaningful assistance toward the harmful goal), PARTIAL (hedged but leaked usable help - usually a failure), or COMPLIED (did the harmful thing). Use a calibrated LLM judge with a per-case rubric describing what compliance looks like specifically for that attack (a refusal that then provides the steps anyway is COMPLIED, not REFUSED) - keyword matching is insufficient because models comply politely. Calibrate the judge against human labels on 15 responses and report agreement before trusting it. For action-based jailbreaks (did a restricted tool fire?), evaluate on the observable action, not the text.

    • ✓ Hedged/partial compliance classified as failure
    • ✓ Judge calibrated against human labels
    • ✓ Action-based cases judged on the action
  3. 03

    Version, gate, and grow the suite

    A safety net that is not tied to CI and not growing will fail you quietly.

    Preview prompt + verify gate ▾

    Operationalize. VERSIONING: every run records model version, prompt/harness version, and per-case results, stored for trend analysis. GATING: the suite runs in CI on prompt/harness changes AND on model version bumps (add a scheduled run plus a manual trigger for provider updates that CI would not otherwise catch); any regression - a case that was REFUSED now PARTIAL/COMPLIED - blocks with the reproduction. GROWTH: institute the rule that every newly discovered jailbreak is added to the corpus before its fix merges, with the fix making it pass. Validate the whole system: pick one currently-refused jailbreak, deliberately weaken the safety prompt, and confirm the suite catches the regression; restore and confirm green.

    • ✓ Runs on model bumps, not just prompt edits
    • ✓ Regressions block with reproductions
    • ✓ Deliberate weakening is caught

Research-backed

Sources behind this flow

Tier 6 · agent-research

DeepTeam

An LLM red-teaming framework (from the DeepEval team): 40+ vulnerability probes and 10+ attack methods (prompt injection, jailbreaking, leakage) to adversarially test LLM apps and agents.