0Pricing
AI Agents · Lesson

Benchmark Suites: SWE-Bench, GAIA, ToolBench

Public benchmarks for coding agents (SWE-Bench), general assistants (GAIA), and tool use (ToolBench).

Benchmark Suites: SWE-Bench, GAIA, ToolBench is a free AI Agents lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Public Benchmarks

For comparing models and frameworks across teams, public benchmarks are essential. They cover common agent capabilities:

  • SWE-Bench — software engineering tasks
  • GAIA — general assistant tasks
  • ToolBench / BFCL — tool use
  • WebArena — browser navigation

SWE-Bench

SWE-Bench tests whether an agent can solve real GitHub issues:

  • 2294 real issues from popular Python repos
  • The agent must produce a patch that passes all hidden tests
  • Top frontier agents now solve 50-70% (was <5% in 2023)

SWE-Bench Verified

OpenAI released a 500-issue subset with stricter quality checks (SWE-Bench Verified). This is the industry standard.

GAIA

General AI Assistant benchmark (Meta + HF, 2023):

  • 466 questions across three difficulty levels
  • Requires browsing, code execution, multimodal reasoning
  • Designed so a human takes ~30 seconds; top agents reach ~60%

Sample GAIA Question

"How many studio albums of Mercedes Sosa were published between 1972 and 1985? Use the list from her English Wikipedia page."

Requires browsing, table reading, counting — typical of multi-step agent tasks.

Berkeley Function Calling Leaderboard (BFCL)

The standard for tool-calling evaluation:

  • Thousands of (query, tool_def, expected_call) triples
  • Covers simple/parallel/missed-tool scenarios
  • Updated quarterly

ToolBench

Larger but messier: 16k+ APIs from RapidAPI, multi-step tool chains. Less canonical than BFCL but covers more breadth.

WebArena

Open-source benchmark for web-browsing agents. Hosts 4 self-contained websites (e-commerce, forum, dev portal, GitLab); agents must complete realistic tasks.

Running SWE-Bench Locally

git clone https://github.com/princeton-nlp/SWE-bench
cd SWE-bench
pip install -e .

# Configure your agent as a 'model' adapter
python -m swebench.harness.run_evaluation \
    --predictions_path my_agent_preds.json \
    --max_workers 4

Benchmark Limitations

  • Public benchmarks leak into training data (gaming risk)
  • Single-domain — your task may be harder/easier
  • "Solve X%" obscures which X% — long-tail matters

Your Own Benchmark Is More Important

Public benchmarks are useful for comparing approaches. But your own gold set on YOUR data is the only number that matters for YOUR product.

Combining Internal + Public

Healthy team practice:

  • Run public benchmarks quarterly to track frontier capability
  • Run internal evals on every PR
  • Trust internal numbers for decisions; public numbers for reading the field

Benchmark Result Reading

When a paper says "75% on SWE-Bench":

  • Check WHICH SWE-Bench (Lite, Verified, full)
  • Check whether multi-attempts allowed
  • Check whether they used hidden tools / hints

Headline numbers are easy to misread.

Internal vs Public Evals

Which matters more for your product?

Recap

SWE-Bench for code agents, GAIA for general assistants, BFCL for tool use, WebArena for browsers. Use them to read the field; trust your own eval for decisions.

Frequently asked questions

Is the “Benchmark Suites: SWE-Bench, GAIA, ToolBench” lesson free?

Yes — the full text of “Benchmark Suites: SWE-Bench, GAIA, ToolBench” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Benchmark Suites: SWE-Bench, GAIA, ToolBench”?

Public benchmarks for coding agents (SWE-Bench), general assistants (GAIA), and tool use (ToolBench). You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Benchmark Suites: SWE-Bench, GAIA, ToolBench” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Eval-Driven Development for Agents
  2. Building a Golden Test Set
  3. LLM-as-a-Judge Pitfalls
  4. Benchmark Suites: SWE-Bench, GAIA, ToolBench
← Back to AI Agents