Nonobench

Measured conclusion: Deep-tested. L2 outcome: tools_ok.

An open-source benchmark of how well LLMs solve nonogram puzzles, from 5x5 to 20x20.

Add to your agent

Copy this into your MCP client config:

{
  "mcpServers": {
    "Nonobench": {
      "url": "https://www.nonobench.com/mcp",
      "transport": "streamable-http"
    }
  }
}

Measured tools (11)

• get_leaderboard read-only

  Models ranked by accuracy; defaults to all effort levels for compatibility.

  Parameters:

  • size — string — Grid size to filter on
  • effort — string — best, all (default), or one effort level; empty means all
  • family — string — Comma-separated family ids; empty means no filter
  • version — string — Comma-separated benchmark versions: 1.0, 1.1, 1.2; empty means all
  • provider — string — Comma-separated provider ids; empty means no filter
  • reasoning — boolean — Only reasoning (true) or non-reasoning (false) variants
  • min_correct — integer — Minimum puzzles solved in the selected tier; default 0 includes unsolved variants
  • open_weights — boolean — Only open-weight (true) or closed (false) models

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• list_providers read-only

  Provider ids, names, families and variant counts.

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• list_families read-only

  Model families, available efforts and best variants.

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• compare_models read-only

  Side-by-side core overall and per-size accuracy, cost, latency and token results for model or family names.

  Parameters:

  • models (required) — array — 2 to 20 model variant ids or family names, e.g. claude-opus-5.5 or gpt-6-astra-xhigh

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• get_model_results read-only

  Accuracy, cost, latency and token use for one model, broken down by grid size.

  Parameters:

  • model (required) — string — Model name as listed on the leaderboard, e.g. gpt-5.4-xhigh

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• list_puzzles read-only

  The benchmark puzzles with their ids and row/column clues.

  Parameters:

  • size — string — Grid size to filter on

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• get_puzzle read-only

  One puzzle, including the clue text models were prompted with. The reference solution is only included on request; some puzzles have several valid solutions.

  Parameters:

  • id (required) — string — Puzzle id from list_puzzles
  • include_solution — boolean — Include a reference solution

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• check_solution read-only

  Check a grid against a puzzle's clues, using the same rule as the benchmark grader. Reports which rows and columns do not match.

  Parameters:

  • id (required) — string — Puzzle id from list_puzzles
  • grid (required) — string — Row-major string of 0 (empty) and 1 (filled), width × height characters

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• get_puzzle_results read-only

  Per-model outcomes for one puzzle. Answers are omitted unless requested.

  Parameters:

  • id (required) — string — Puzzle id from list_puzzles
  • effort — string — best, all (default), or one effort level
  • family — string — Comma-separated family ids; empty means no filter
  • provider — string — Comma-separated provider ids; empty means no filter
  • reasoning — boolean — Only reasoning (true) or non-reasoning (false) variants
  • open_weights — boolean — Only open-weight (true) or closed (false) models
  • include_answers — boolean — Include each model's answer grid

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• get_model_puzzles read-only

  Which puzzles one model solved, missed, timed out on, or has not run.

  Parameters:

  • model (required) — string — Model variant id as listed on the leaderboard, e.g. claude-opus-5.5-high

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

• list_runs read-only

  Individual benchmark runs (one model on one puzzle), optionally with the raw prompt and model output.

  Parameters:

  • size — string — Grid size to filter on
  • limit — integer — Default 100
  • model — string — Only runs of this model variant id
  • offset — integer — Runs to skip, for paging; default 0
  • puzzle_id — string — Only runs on this puzzle id from list_puzzles
  • include_output — boolean — Include raw prompt and model output (large)

  Hints: read-only yes · destructive not declared · idempotent not declared · open-world no

Publisher website Repository

Measured by MCPYP probing. Untested fields are omitted, never estimated. Listing is always free; MCPYP never proxies calls or stores user tokens.