OpenMark

Measured conclusion: Deep-tested. L2 outcome: tools_ok.

Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.

Add to your agent

Copy this into your MCP client config:

{
  "mcpServers": {
    "OpenMark": {
      "url": "https://openmark.ai/mcp",
      "transport": "streamable-http"
    }
  }
}

Measured tools (9)

• get_task_format

  How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name.

  Parameters:

  • topic — string — Optional: fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name such as regex_match

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

• validate_task

  Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nothing. Returns the problems to fix and warnings worth a look.

  Parameters:

  • task_yaml (required) — string — The full task YAML (a JSON object with the same fields also works)

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

• list_models

  Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.

  Parameters:

  • search — string — Filter by id, name or provider, e.g. 'claude', 'gemini flash'
  • images_only — boolean — Only models that accept image attachments
  • max_output_price — number — Only models at or below this $ per 1M output tokens

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

• prepare_attachments

  Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark.

  Parameters:

  • file_names — array — Optional: the local file paths to upload, to get a ready-to-run command

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

• estimate_cost

  Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.

  Parameters:

  • task_yaml (required) — string — The full task YAML (a JSON object with the same fields also works)
  • models (required) — array
  • preset — string — Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it
  • fail_fast — boolean — Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures
  • max_tokens — integer — Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value
  • upload_uid — string — From prepare_attachments, so image and document tokens are counted
  • custom_timeout — integer — Seconds per answer, with timeout_profile custom
  • stability_runs — integer — Passes per model; 2+ shows which models are consistent
  • temp_preference — string — With temperature_discovery: which temperature wins a tie
  • timeout_profile — string — How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout
  • temperature_discovery — boolean — Find each model's best temperature for this task. Costs about 6x as much

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

• run_benchmark

  Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast).

  Parameters:

  • task_yaml (required) — string — The full task YAML (a JSON object with the same fields also works)
  • models (required) — array — Model ids from list_models
  • preset — string — Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it
  • fail_fast — boolean — Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures
  • max_tokens — integer — Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value
  • upload_uid — string — From prepare_attachments, when tests use attachments
  • max_credits — integer — Budget: if the expected cost is higher, nothing runs and you get the quote instead
  • custom_timeout — integer — Seconds per answer, with timeout_profile custom
  • stability_runs — integer — Passes per model; 2+ shows which models are consistent
  • temp_preference — string — With temperature_discovery: which temperature wins a tie
  • timeout_profile — string — How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout
  • temperature_discovery — boolean — Find each model's best temperature for this task. Costs about 6x as much

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

• get_results

  Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.

  Parameters:

  • job_id (required) — string
  • tolerance — number — Score points below the top score that still count as good enough
  • show_misses — boolean — List the tests each model got wrong, with the expected answer and the model's reply
  • optimize_for — string — accuracy (same order as the app's results table): best score, then cheapest, then fastest. cost / speed / stability: cheapest / fastest / most consistent model within `tolerance` points of the top score

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

• get_account

  The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits.

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

• watch_task

  Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.

  Parameters:

  • job_id (required) — string
  • model — string — The model to keep as the pick (default: the best-scoring model). Use the model you recommended.

  Hints: read-only not declared · destructive not declared · idempotent not declared · open-world not declared

Publisher website

Measured by MCPYP probing. Untested fields are omitted, never estimated. Listing is always free; MCPYP never proxies calls or stores user tokens.