Methodology
What SkillsBench measures, and how a run turns into a score.
What SkillsBench measures
Most benchmarks test whether a model can produce the right answer to a question. SkillsBench tests something different: whether an agent can use its skills — tools, multi-step planning, and real APIs — to get from a task description to a verified, working result. Every task has a concrete pass/fail check rather than a subjective grade.
The six categories
Every task belongs to exactly one category. An agent's overall score is the simple average of its six category scores.
How a task is scored
Sandboxed run
The agent runs the task in an isolated environment with only the tools the task explicitly grants.
Automated verification
Each task ships with a checker — unit tests, a diff check, or a rubric grader for open-ended output — that decides pass or fail.
Category score
A category's score is the percentage of that category's tasks the agent passed, scaled 0–100.
Overall score & rank
Overall score is the unweighted average of the six category scores. Ranks are recomputed whenever a new run is submitted.
Difficulty tiers
Tasks are tagged easy, medium, or hard based on the number of steps and how easy it is to fail silently (e.g. producing a plausible-looking but wrong result). Difficulty does not currently weight the score — it's shown so agent builders can see where failures concentrate.
Submitting a run
Anyone can submit a run from the submission page by providing category scores. In this demo, submissions are accepted directly; in a production benchmark, submitted runs would be independently re-verified before appearing on the public leaderboard.