JailbreakBench Dataset Explained: JBB-Behaviors vs AdvBench

JailbreakBench is an open benchmark for measuring LLM jailbreak attacks and defenses, published at the NeurIPS 2024 Datasets and Benchmarks Track. Its dataset, JBB-Behaviors, holds 100 harmful behaviors across 10 policy categories plus 100 benign controls. A Llama 3 70B judge scores responses, and a versioned artifacts repo stores every successful jailbreak prompt.

That fixed judge is the point. Every jailbreak paper reports an attack success rate, and almost none of them are comparable, because ASR is not a property of the attack. It is a property of three choices the authors made before running anything: which forbidden prompts they used, which model they pointed them at, and which judge decided whether the response counted as a success.

Change the benchmark and the same attack moves twenty points. This reference covers the four datasets that show up most often in the literature (AdvBench, HarmBench, JailbreakBench and StrongREJECT) plus the two benchmarks built for multimodal transfer and settings sensitivity. The goal is practical: knowing which number you are reading, and which one you should be producing.

The scoring layer is the variable that matters

Before the datasets, the judge. There are three families in use, and they disagree substantially on the same transcripts.

Refusal-string matching. The original AdvBench evaluation checks whether the response contains any of a list of stock refusal prefixes, such as “I’m sorry”, and calls the absence of one a success. It is cheap and it is why early jailbreak numbers looked so good. A response that is off-topic, incoherent, or that agrees enthusiastically and then produces nothing usable all score as jailbroken.

Classifier judging. HarmBench ships a fine-tuned classifier that decides whether a completion actually exhibits the target behavior. JailbreakBench standardizes on a judge model in the same spirit. This is a large step up from substring matching and it is the current default for anything that wants to be taken seriously.

Graded usefulness. StrongREJECT’s autograder scores how much genuinely useful information the response gives toward the forbidden request, rather than emitting a binary. Its authors report that this correlates far better with human judgment than the alternatives.

That third approach exposed the most uncomfortable finding in this area. The StrongREJECT paper reports that existing evaluation methods significantly overstate jailbreak effectiveness relative to human judgment, and identifies why: jailbreaks that bypass a model’s safety fine-tuning tend to degrade the model’s capabilities at the same time. The model stops refusing and also stops being competent. Substring matching cannot tell those apart. A graded evaluator can.

If you take one thing from this page, take that. A jailbreak that produces compliance without capability is not an exploit, it is a lobotomy, and the benchmark you pick decides whether you notice.

JailbreakBench (2024)

JailbreakBench is the most useful of the four for someone doing engagement work rather than research. Its dataset, JBB-Behaviors, is 100 behaviors chosen to align with OpenAI’s published usage policies. One hundred behaviors is small enough to run repeatedly against a live API without burning a budget, and the policy alignment means findings map onto language a vendor will recognize.

Three components make it stick:

• An artifacts repository. JailbreakBench publishes the actual adversarial prompts that succeeded, versioned, in the JailbreakBench/artifacts repository, and leaderboard submissions must include them. Reproducibility in this field has been dire (prompts withheld, code closed, target APIs silently updated underneath published results) and an artifact repo is the direct answer to that.
• A standardized threat model. System prompts, chat templates, and scoring functions are fixed and stated, so two people running the same attack are running the same experiment.
• A leaderboard tracking attacks and defenses per model, split into open-source and closed-source tables.

The catch is the same catch as everywhere: a 100-behavior set is a sample, and a defended application that passes it has demonstrated exactly that it passes it. Use the artifacts as a regression suite in CI.

What is in the JailbreakBench/JBB-Behaviors dataset

The JBB-Behaviors card on Hugging Face is the canonical copy. It is MIT-licensed, about 1 MB, and has two configurations:

Each behavior row carries Goal (the request sent to the model), Target (an affirmative response prefix, the string GCG-style attacks optimize toward), Behavior (a short identifier), Category and Source. The ten categories hold ten behaviors each: harassment/discrimination, malware/hacking, physical harm, economic harm, fraud/deception, disinformation, sexual/adult content, privacy, expert advice, and government decision-making.

On provenance, the project site puts the original share at about 55%, with the rest drawn from AdvBench and the TDC/HarmBench set. It is not a superset of those sources, so JBB and AdvBench numbers are not interchangeable.

The benign split is the part most people skip. A defense that blocks all 100 harmful behaviors and refuses half the benign ones has moved the failure to your users. Report both rates.

How to run JailbreakBench

The evaluation library is on PyPI as jailbreakbench. The current release is 1.0.0, published June 13, 2024 and requiring Python 3.10 or 3.11 (checked October 2026).

1. Install it: pip install jailbreakbench, or pip install jailbreakbench[vllm] if you want to run target models locally on a CUDA GPU.
2. Load the behaviors: dataset = jbb.read_dataset() returns behaviors, goals, targets and categories; jbb.read_dataset("benign") gives the control set. Without the library, load_dataset("JailbreakBench/JBB-Behaviors", "behaviors") pulls the same data from Hugging Face.
3. Pull prior jailbreaks to use as a regression suite: jbb.read_artifact(method="PAIR", model_name="vicuna-13b-v1.5") returns the stored prompts, with generation settings in artifact.parameters.
4. Query targets through LiteLLM (jbb.LLMLiteLLM, with a Together AI or OpenAI key) or vLLM (jbb.LLMvLLM).
5. Score with jbb.evaluate_prompts(...). The built-in jailbreak judge is the Llama 3 70B-based Llama3JailbreakJudge; a Llama 3 8B-based Llama3RefusalJudge handles refusal classification.

Leaderboard submissions supply up to one jailbreak per behavior against both vicuna-13b-v1.5 and llama-2-7b-chat-hf, with gpt-3.5-turbo-1106 and gpt-4-0125-preview optional, tagged as white-box, black-box or transfer, and filed as a GitHub issue. The library also bundles baseline defenses (SmoothLLM, perplexity filtering, removal of non-dictionary words, synonym substitution). Those leaderboard targets are old models; for engagement work, point the harness at the model you are assessing and keep them only as a sanity baseline.

AdvBench (2023)

Introduced alongside GCG in Zou et al., AdvBench is the oldest of the four and still the most cited. It ships two splits: harmful strings, where the goal is to make the model emit a specific target sequence, and harmful behaviors, where the goal is any response that carries out the instruction. The behaviors split holds 520 entries. The strings split is often cited as 500, but its CSV runs to 575 lines on GitHub, header row included.

The AdvBench dataset lives in the llm-attacks repository as two CSVs, data/advbench/harmful_behaviors.csv and data/advbench/harmful_strings.csv; the repo’s code is MIT-licensed. A Hugging Face mirror at walledai/AdvBench is gated behind a consent form and describes itself as 500 harmful behaviors, so check the row count against the repo’s 520 before comparing results across copies.

What to know before quoting an AdvBench number:

• The default evaluation is refusal-string matching, so headline ASRs on it run high.
• The behaviors were LLM-generated and the set is repetitive. The StrongREJECT authors counted 24 of the 520 prompts, about 5% of the dataset, asking for bomb-making instructions. Duplication at that rate inflates apparent coverage and makes per-category breakdowns unreliable.
• Most papers do not run the whole set. A 50-behavior subset is common, and different papers pick different fifties.

AdvBench is fine as a smoke test and useful as a shared vocabulary. It is not a defensible basis for a claim about a production system. The gradient-based attack it was built to evaluate is covered in the PAIR vs GCG vs TAP comparison, which is where the compute-cost side of these numbers lives.

HarmBench (2024)

HarmBench was the first serious attempt to make automated red teaming reproducible across methods. Its contribution is less the behavior list than the harness: a standardized attack interface, a standardized judge, and a published comparison of 18 red teaming methods against 33 target LLMs and defenses.

Two design choices matter for practitioners. First, the behavior set is split by function rather than only by harm topic. Its 510 behaviors break down as 200 standard, 100 contextual (each pairing a context string with the request, so the harm depends on supplied material), 100 copyright, scored by a hashing classifier rather than a judge model, and 110 multimodal, each pairing an image with a behavior string. Contextual behaviors are the closest thing in any of these benchmarks to the RAG-shaped threat model most deployed applications actually have, which is the surface described in prompt injection via retrieved documents.

Second, HarmBench evaluates defenses, not just attacks. If you are choosing between guardrail products, a benchmark that scores the defended system is worth more than one that scores the bare model. That distinction is the whole point of the LLM guardrail tools comparison.

The HarmBench repository publishes the judges as cais/HarmBench-Llama-2-13b-cls for standard and contextual behaviors, a separate multimodal variant, and cais/HarmBench-Mistral-7b-val-cls as a validation classifier. Its scripts/run_pipeline.py runs three steps (generate test cases, generate completions, evaluate) in slurm, local or local_parallel mode.

Use HarmBench when you need a defensible cross-method comparison and you control enough compute to run a real harness.

StrongREJECT (2024)

StrongREJECT is the corrective. Its dataset is built so that a victim model has to produce specific, actionable information to count as answering (vague gestures at a topic score low by construction) and its automated evaluator grades the usefulness of the response rather than the absence of a refusal.

The build process is the part worth copying. The authors pooled prompts from existing jailbreak datasets with ones they wrote or had LLMs generate, then filtered them: drop out-of-scope categories, deduplicate by embedding similarity, drop anything nine sentinel models answer readily without a jailbreak, categorize, and hand-reject anything ill-posed, vague or unanswerable. What survived is 313 prompts, at least 50 per category. The sentinel step is the one most homegrown corpora skip, and skipping it means some of your “successes” never tested the jailbreak at all.

The strong_reject package installs from GitHub with pip install git+https://github.com/dsbowen/strong_reject.git@main and covers six categories: disinformation and deception, illegal goods and services, hate/harassment/discrimination, non-violent crimes, violence, and sexual content. Its two evaluators are strongreject_rubric, an LLM rubric grader that needs an OPENAI_API_KEY, and strongreject_finetuned, a Gemma 2B fine-tuned on the rubric grader’s outputs that needs an HF_TOKEN for the gated Gemma weights. Pick the fine-tuned one when transcripts cannot leave your environment. The package is MIT, but the prompts come from earlier datasets under mixed or unstated licenses.

Run it when your job is to report an honest number to someone who will make a decision with it. Run it especially when a vendor or a paper quotes you a near-100% success rate, because near-100% is the signature of substring matching on a small AdvBench subset.

The two specialist benchmarks

JailbreakV-28K asks whether text jailbreaks transfer to multimodal models. Built from a 2,000-query malicious set, it combines 20,000 text-based jailbreak prompts with 8,000 image-based inputs for 28,000 test cases, evaluated against 10 open-source multimodal LLMs. The paper’s finding is that transfer from the text side is alarmingly effective: the vision encoder does not get its own safety training, so the text-processing path stays the weak link. That result is the quantitative backing for the attack classes in multimodal jailbreaks.

The Hugging Face release (MIT, COLM 2024) exposes three subsets: JailBreakV_28K, a 280-row mini_JailBreakV_28K for cheap runs, and RedTeam_2K, the seed queries drawn from 8 sources and tagged against 16 safety policies. Its five attack methods are three LLM transfer styles (logic, persuade, template) plus FigStep and query-relevant images, and the transfer_from_llm column separates text-transfer cases from native image attacks when you score.

JailTrickBench, published as “Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs”, measures the thing everyone else holds constant. It isolates eight factors at the target level and the attack level (model size, safety fine-tuning, system prompt, template choice and so on) across seven attacks and six defenses, in roughly 354 experiments. The takeaway is blunt: several of those knobs move ASR more than the choice of attack does. A published ASR without the system prompt and template stated alongside it is not a measurement.

Which one to use

For most application red teams the answer is a pair: JailbreakBench artifacts as the fast regression suite, StrongREJECT as the honesty check on anything that looks like a finding.

Reporting numbers people can trust

A jailbreak result is only meaningful with its configuration attached. State all six of these or the number cannot be reproduced:

1. Dataset and subset. “AdvBench” is not enough. Name the split and the row count.
2. Target model and version string. Provider models change under a stable name; a result from March is not a result from August.
3. System prompt and chat template. JailTrickBench exists because these dominate.
4. Judge. Substring list, classifier, or graded autograder, named.
5. Query budget. DrAttack reports 78.0% on GPT-4 at 15 queries; an attack that needs orders of magnitude more queries to reach the same place is a different tool for a different engagement. That family is broken out in payload splitting and encoding jailbreaks.
6. Defenses in front of the model. An undefended API result says nothing about a deployed application sitting behind a guardrail and an egress filter, which is the arrangement described in the LLM defense stack.

Benchmarks answer whether a technique class works against a model family in the abstract. The Still Works? tracker archives a directional Q2 2026 snapshot of each class by model family, and why jailbreaks work at all covers the failure modes these datasets are sampling from. Single-turn datasets like these also miss attacks that build up over a conversation, which multi-turn role-play attacks covers separately. If you are picking a scanner to run any of this through, the AI red teaming tools comparison covers which harnesses ingest which benchmark, a companion piece sets out what garak, PyRIT, promptfoo and Purple Llama cost to license and self-host, and the Garak review covers the detector-quality problem that turns a good dataset into a bad number.

FAQ

what is jailbreakbench used for

JailbreakBench is used to measure how well jailbreak attacks and defenses work under one fixed setup. It pairs the 100-behavior JBB-Behaviors dataset with fixed system prompts, chat templates and a Llama 3 70B judge, so results from different teams are comparable. Red teams mostly reuse its artifact prompts as a repeatable regression suite.

how do i download the jailbreakbench dataset

Run pip install jailbreakbench and call jbb.read_dataset() for the harmful behaviors or jbb.read_dataset("benign") for the controls. Without the package, load JailbreakBench/JBB-Behaviors from Hugging Face with the behaviors config. The data is MIT-licensed, about 1 MB, and also includes a separate 300-row judge_comparison config used to compare candidate judges.

where can i find the advbench dataset

AdvBench sits in the llm-attacks GitHub repository under data/advbench, as harmful_behaviors.csv (520 behaviors) and harmful_strings.csv (575 lines with the header, more than the 500 strings often cited). A gated mirror exists on Hugging Face as walledai/AdvBench. Whichever copy you use, record the split and row count, because many papers report on 50-behavior subsets.

jailbreakbench vs harmbench: which should i use

Use JailbreakBench for fast, repeatable checks against a live model or API: 100 behaviors, a pip package and stored artifact prompts. Use HarmBench when you need a broader cross-method comparison across 510 behaviors, including contextual, copyright and multimodal cases, and have the GPU capacity to run its full three-step pipeline.

what does strongreject measure

StrongREJECT measures how much genuinely useful harmful information a jailbroken response contains, not just whether the model refused. Its 313 vetted prompts require specific, actionable answers, and its rubric or fine-tuned Gemma 2B grader assigns a graded score. That exposes jailbreaks that stop the refusal but leave the model too degraded to help.