Jailbreak Benchmarks: AdvBench, HarmBench, JailbreakBench

Every jailbreak paper reports an attack success rate. Almost none of them are comparable, because ASR is not a property of the attack. It is a property of three choices the authors made before running anything: which forbidden prompts they used, which model they pointed them at, and which judge decided whether the response counted as a success.

Change the benchmark and the same attack moves twenty points. This is a reference to the four datasets that show up most often in the literature — AdvBench, HarmBench, JailbreakBench, and StrongREJECT — plus the two benchmarks that exist specifically to measure the multimodal and the settings-sensitivity problems. The goal is practical: knowing which number you are reading, and which one you should be producing.

The scoring layer is the variable that matters

Before the datasets, the judge. There are three families in use, and they disagree substantially on the same transcripts.

Refusal-string matching. The original AdvBench evaluation checks whether the response contains any of a list of refusal prefixes — “I’m sorry”, “As an AI”, “I cannot” — and calls the absence of one a success. It is cheap and it is why early jailbreak numbers looked so good. A response that is off-topic, incoherent, or that agrees enthusiastically and then produces nothing usable all score as jailbroken.

Classifier judging. HarmBench ships a fine-tuned classifier that decides whether a completion actually exhibits the target behavior. JailbreakBench standardizes on a judge model in the same spirit. This is a large step up from substring matching and it is the current default for anything that wants to be taken seriously.

Graded usefulness. StrongREJECT’s autograder scores how much genuinely useful information the response gives toward the forbidden request, rather than emitting a binary. Its authors report that this correlates far better with human judgment than the alternatives.

That third approach exposed the most uncomfortable finding in this area. The StrongREJECT paper reports that existing evaluation methods significantly overstate jailbreak effectiveness relative to human judgment, and identifies why: jailbreaks that bypass a model’s safety fine-tuning tend to degrade the model’s capabilities at the same time. The model stops refusing and also stops being competent. Substring matching cannot tell those apart. A graded evaluator can.

If you take one thing from this page, take that. A jailbreak that produces compliance without capability is not an exploit, it is a lobotomy, and the benchmark you pick decides whether you notice.

AdvBench (2023)

Introduced alongside GCG in Zou et al., AdvBench is the oldest of the four and still the most cited. It ships two splits: harmful strings, where the goal is to make the model emit a specific target sequence, and harmful behaviors, where the goal is any response that carries out the instruction. The strings split holds 500 entries and the behaviors split 520.

What to know before quoting an AdvBench number:

• The default evaluation is refusal-string matching, so headline ASRs on it run high.
• The behaviors were LLM-generated and the set is repetitive. The StrongREJECT authors counted 24 of the 520 prompts — about 5% of the dataset — asking for bomb-making instructions. Duplication at that rate inflates apparent coverage and makes per-category breakdowns unreliable, and it is why StrongREJECT’s own build deduplicated by embedding similarity before hand-vetting what survived.
• Most papers do not run the whole set. A 50-behavior subset is common, and different papers pick different fifties.

AdvBench is fine as a smoke test and useful as a shared vocabulary. It is not a defensible basis for a claim about a production system. The gradient-based attack it was built to evaluate is covered in the PAIR vs GCG vs TAP comparison, which is where the compute-cost side of these numbers lives.

HarmBench (2024)

HarmBench was the first serious attempt to make automated red teaming reproducible across methods. Its contribution is less the behavior list than the harness: a standardized attack interface, a standardized judge, and a published comparison of 18 red teaming methods against 33 target LLMs and defenses.

Two design choices matter for practitioners. First, the behavior set is split by function rather than only by harm topic. Its 510 behaviors break down as 200 standard, 100 contextual — each pairing a context string with the request, so the harm depends on supplied material — 100 copyright, scored by a hashing classifier rather than a judge model, and 110 multimodal, each pairing an image with a behavior string. Contextual behaviors are the closest thing in any of these benchmarks to the RAG-shaped threat model most deployed applications actually have, which is the surface described in prompt injection via retrieved documents.

Second, HarmBench evaluates defenses, not just attacks. If you are choosing between guardrail products, a benchmark that scores the defended system is worth more than one that scores the bare model. That distinction is the whole point of the LLM guardrail tools comparison.

Use HarmBench when you need a defensible cross-method comparison and you control enough compute to run a real harness.

JailbreakBench (2024)

JailbreakBench is the most useful of the four for someone doing engagement work rather than research. Its dataset, JBB-Behaviors, is 100 behaviors — some original, some sourced from AdvBench and HarmBench — chosen to align with OpenAI’s published usage policies. One hundred behaviors is small enough to run repeatedly against a live API without burning a budget, and the policy alignment means findings map onto language a vendor will recognize.

Three components make it stick:

• An artifacts repository. JailbreakBench publishes the actual adversarial prompts that succeeded, versioned. Reproducibility in this field has been dire — prompts withheld, code closed, target APIs silently updated underneath published results — and an artifact repo is the direct answer to that.
• A standardized threat model. System prompts, chat templates, and scoring functions are fixed and stated, so two people running the same attack are running the same experiment.
• A leaderboard tracking attacks and defenses per model.

The catch is the same catch as everywhere: a 100-behavior set is a sample, and a defended application that passes it has demonstrated exactly that it passes it. Use the artifacts as a regression suite in CI. That is what they are good for.

StrongREJECT (2024)

StrongREJECT is the corrective. Its dataset is built so that a victim model has to produce specific, actionable information to count as answering — vague gestures at a topic score low by construction — and its automated evaluator grades the usefulness of the response rather than the absence of a refusal.

The build process is the part worth copying. A superset of 1,529 candidate prompts was filtered in four stages: drop anything outside the taxonomy, deduplicate by embedding similarity, drop anything the model refuses even without a jailbreak, then hand-vet the remainder for clarity and answerability. What survived is 313 prompts. Stage three is the one most homegrown corpora skip, and skipping it means a chunk of your “failures” were never a test of the jailbreak at all.

Run it when your job is to report an honest number to someone who will make a decision with it. Run it especially when a vendor or a paper quotes you a near-100% success rate, because near-100% is the signature of substring matching on a small AdvBench subset.

The two specialist benchmarks

JailbreakV-28K asks whether text jailbreaks transfer to multimodal models. Built from a 2,000-query malicious set, it combines 20,000 text-based jailbreak prompts with 8,000 image-based inputs for 28,000 test cases, evaluated against 10 open-source multimodal LLMs. The paper’s finding is that transfer from the text side is alarmingly effective — the vision encoder does not get its own safety training, so the text-processing path stays the weak link. That result is the quantitative backing for the attack classes in multimodal jailbreaks.

JailTrickBench, published as “Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs”, measures the thing everyone else holds constant. It isolates eight factors at the target level and the attack level — model size, safety fine-tuning, system prompt, template choice and so on — across seven attacks and six defenses, in roughly 354 experiments. The takeaway is blunt: several of those knobs move ASR more than the choice of attack does. A published ASR without the system prompt and template stated alongside it is not a measurement.

Which one to use

Benchmark	Size	Judge	Best for
AdvBench	520 behaviors, 500 strings	Refusal-string match	Smoke tests, historical comparison
HarmBench	510 behaviors (200 standard / 100 contextual / 100 copyright / 110 multimodal)	Fine-tuned classifier	Cross-method and defense comparison
JailbreakBench	100 behaviors	Standardized judge model	CI regression, engagement reporting
StrongREJECT	313 forbidden prompts	Graded usefulness autograder	Honest effectiveness numbers
JailbreakV-28K	28,000 cases	Per-paper	Multimodal transfer testing
JailTrickBench	Two source datasets	Per-configuration	Sensitivity analysis of settings

For most application red teams the answer is a pair: JailbreakBench artifacts as the fast regression suite, StrongREJECT as the honesty check on anything that looks like a finding.

Reporting numbers people can trust

A jailbreak result is only meaningful with its configuration attached. State all six of these or the number cannot be reproduced:

1. Dataset and subset. “AdvBench” is not enough. Name the split and the row count.
2. Target model and version string. Provider models change under a stable name; a result from March is not a result from August.
3. System prompt and chat template. JailTrickBench exists because these dominate.
4. Judge. Substring list, classifier, or graded autograder, named.
5. Query budget. DrAttack reports 78.0% on GPT-4 at 15 queries; an attack that needs orders of magnitude more queries to reach the same place is a different tool for a different engagement. That family is broken out in payload splitting and encoding jailbreaks.
6. Defenses in front of the model. An undefended API result says nothing about a deployed application sitting behind a guardrail and an egress filter, which is the arrangement described in the LLM defense stack.

Benchmarks answer whether a technique class works against a model family in the abstract. For where each class currently stands per model, the Still Works? tracker is the per-quarter view, and why jailbreaks work at all covers the failure modes these datasets are sampling from. If you are picking a scanner to run any of this through, the AI red teaming tools comparison covers which harnesses ingest which benchmark, and the Garak review covers the detector-quality problem that turns a good dataset into a bad number.