agentcheck

Measured conclusion: Deep-tested. L2 outcome: tools_ok.

Synthetic checks, nightly regression replay and model-drift alerts for AI agents

Add to your agent

Copy this into your MCP client config:

{
  "mcpServers": {
    "agentcheck": {
      "url": "https://agentwares-agentcheck.vercel.app/api/mcp",
      "transport": "streamable-http"
    }
  }
}

Measured tools (11)

• agentcheck_get_status read-only

  Current status of a public monitored target: overall state, uptime over 24h/7d/30d, last check time, last nightly scores, open incidents and the badge/status URLs. Use the owner (GitHub login) and target slug from the status page URL https://agentwares-agentcheck.vercel.app/<owner>/<slug>. No API key needed.

  Parameters:

  • owner (required) — string — GitHub login of the target's owner, e.g. demo
  • slug (required) — string — target slug, e.g. demo

  Hints: read-only yes · destructive no · idempotent yes · open-world no

• agentcheck_get_pricing read-only

  Machine-readable pricing for agentcheck: tiers with monthly USD price, target limits, check interval and features, plus per-run add-ons. Same data as /pricing.json. No API key needed.

  Hints: read-only yes · destructive no · idempotent yes · open-world no

• agentcheck_model_retirements read-only

  Is this model retiring? Pass model IDs as your code or config names them (gpt-5-mini, claude-sonnet-4-5-20250929, gemini-2.5-flash, openai/o3-mini). For each, returns every announced shutdown on OpenAI's, Anthropic's and Google's own deprecation pages (OpenAI API, Claude API, Gemini API, Vertex AI): status, shutdown date, recommended replacement, the replacement's list-price ratio per token, the API changes the provider documents, the source URL with the date it was read, and a page with the details. retiring: false means no shutdown for that ID is on those pages. Language models only. No API key needed.

  Parameters:

  • models (required) — array — model IDs exactly as your code or config names them, e.g. ["gpt-5-mini", "claude-sonnet-4-5"]; router prefixes such as openai/ or models/ are ignored, and fine-tuned IDs (ft:…) match their base model

  Hints: read-only yes · destructive no · idempotent yes · open-world no

• agentcheck_api_sunsets read-only

  When does the API version this code pins stop working, and what happens then? Pass provider and version pairs as your code pins them (shopify 2025-10, meta v21.0, google-ads v22, hubspot v1, twilio-flex 2.16.2, twilio-notify v1, keap xml-rpc, exchange-online ews, stripe 2024-06-20). For each, returns every dated end the provider's own versioning, changelog or retirement page lists: status, end date (and whether the provider gives only a month), what happens after it (error, silent fall-forward to another version, app delisting, or unsupported), whether the failure is silent, the replacement version, the source URL with the date it was read, and a page with the details. Where the provider's own pages give different dates for the same end, other_dates lists each one with its page and date_note says which the counts run to. A pair with no entry comes back with the reason: older than every listed version, no end date published yet, or a provider that publishes none (Stripe). Covers Shopify, HubSpot, Meta Graph and Marketing APIs, Google Ads and Content API for Shopping, Twilio Flex UI and Notify, Thryv's Keap XML-RPC, and Exchange Web Services (EWS) in Exchange Online. No API key needed.

  Parameters:

  • pins (required) — array — provider and version pairs, e.g. [{"provider": "shopify", "version": "2025-10"}]

  Hints: read-only yes · destructive no · idempotent yes · open-world no

• agentcheck_statuspage_preview read-only

  Read a public Atlassian Statuspage page once: its name, overall status, components (id, name, status, group) and unresolved incidents, from the page's own /api/v2/summary.json. Pass any address on the page: status.example.com, https://status.example.com/history or the JSON URL. Use it before POST /api/v1/statuspage, which turns the components you give a URL into a live agentcheck status page checked by agentcheck itself, with no signup; the answer carries that request's body. Same read as /statuspage/preview: only public addresses, 5-second timeout, 512 KB cap. Private pages cannot be read. No API key needed.

  Parameters:

  • url (required) — string — the status page's address, e.g. https://status.example.com or its summary.json

  Hints: read-only yes · destructive no · idempotent yes · open-world yes

• agentcheck_create_target destructive

  Enroll something to monitor: an http endpoint (JSON or OpenAI-style chat), a remote MCP server (Streamable HTTP url) or an A2A agent (origin with /.well-known/agent-card.json). Pass `checks` to create checks in the same call (POST /api/v1/probe proposes three). Returns the target id, the public status page, the badge SVG URL and a README snippet. The first check runs on the next scheduled tick, within five minutes; call agentcheck_run_now to run immediately. Free tier: 1 target, hourly; Starter+: 5-minute checks. Requires an API key.

  Parameters:

  • kind (required) — string — http (JSON or chat endpoint), mcp (Streamable HTTP url, or npx/uvx package spec), a2a (agent card)
  • url — string — endpoint URL (http/mcp) or the agent's origin (a2a)
  • auth — credential the prober sends to your endpoint: {type:'bearer',token} or {type:'header',name,value}. Stored encrypted and never returned or logged.
  • name — string — display name; defaults to the host
  • slug — string — URL slug for /<owner>/<slug>; defaults to a slug of the name
  • alerts — object — where to send an incident when a check starts failing. Any combination; omit it and incidents are visible only on the status page.
  • checks — array — checks to create right away (the probe proposes three)
  • format — string — http: openai_chat (POST /chat/completions body), json (raw POST), get (plain fetch)
  • headers — object — extra request headers
  • isPublic — boolean — public status page + badge (default true)
  • modelVar — string — the model your agent runs on (e.g. claude-sonnet-5); enables model-drift re-runs
  • corpusUrl — string — RAG targets: public corpus URL for the nightly groundedness scorer (Pro)
  • modelHeader — string — request header your endpoint accepts to override the model (drift re-runs try the new model)
  • packageSpec — string — mcp only: `npx @org/server` / `uvx server` — runs on the mcpcheck runner, not the scheduled checks
  • agentCardUrl — string — a2a: explicit agent card URL when it is not at /.well-known/agent-card.json

  Hints: read-only no · destructive yes · idempotent no · open-world yes

• agentcheck_add_check

  Add a check to one of your targets. A check runs an input (http path/prompt, mcp tool call, a2a message) on a schedule and judges the answer with a golden: exact, contains, regex and json_schema cost nothing; rubric and baseline use the LLM judge (baseline = same outcome as the last known-good answer). Returns the check id. Requires an API key.

  Parameters:

  • name (required) — string — short name shown on the status page
  • kind (required) — string — what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message)
  • golden (required) — object — how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer.
  • targetId (required) — string — id from agentcheck_create_target or GET /api/v1/targets
  • input — object — http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message }
  • intervalSec — integer — seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min)

  Hints: read-only no · destructive no · idempotent no · open-world no

• agentcheck_run_now

  Run every check of one of your targets immediately (outside the schedule) and return pass/fail per check with latency, judge cost and any incident opened or closed. Use it right after enrolling, or to confirm a fix. Requires an API key.

  Parameters:

  • targetId (required) — string — target id

  Hints: read-only no · destructive no · idempotent no · open-world yes

• agentcheck_record

  Record one production interaction with your agent (the prompt or messages, the final answer, the tools it called, optionally the model) as a trace on a target. Call it from your agent or from a proxy in front of it after each task; promote a good trace with agentcheck_promote_trace to replay it nightly and catch regressions. Requires an API key.

  Parameters:

  • targetId (required) — string — id from agentcheck_create_target or GET /api/v1/targets
  • input (required) — the prompt, the messages array, or an object with prompt/messages
  • output (required) — string — the agent's final answer
  • name — string — short label, e.g. 'refund for order A-1029'
  • model — string — the model that produced this answer, so drift re-runs can compare across models
  • toolCalls — array — tool calls in order

  Hints: read-only no · destructive no · idempotent no · open-world no

• agentcheck_promote_trace

  Turn an imported or recorded trace into a replayable check whose golden is the recorded outcome (tool sequence + final-answer rubric). The check runs daily and in the nightly replay (Pro). Returns the check. Requires an API key.

  Parameters:

  • traceId (required) — string — trace id from agentcheck_record or POST /api/v1/targets/{id}/traces

  Hints: read-only no · destructive no · idempotent yes · open-world no

• agentcheck_list_incidents read-only

  Open and recent incidents across your targets (or one target): when they opened/closed, the failing check and the cause. Requires an API key.

  Parameters:

  • targetId — string — narrow to one target; omit for incidents across every target on the account

  Hints: read-only yes · destructive no · idempotent yes · open-world no

Publisher website Repository

Measured by MCPYP probing. Untested fields are omitted, never estimated. Listing is always free; MCPYP never proxies calls or stores user tokens.