Using the platform from a coding AI
The coding AI on a developer's machine (Claude Code, Cursor, Codex and the like) can connect straight to a project and go through the whole loop with MCP tools: the test connection, writing tests, review, pre-runs, formal tests, reports, and testing again after a fix. Nobody copies text between the web page and the coding AI. How far it may go is up to the permissions you tick for the connection on the web.
The platform calls a coding AI connected this way the "dev AI". It is your own coding tool on its own model account; the platform never runs a coding AI on your code.
Make a connection
- Open the project's "Settings → Dev AI".
- Under "It may:", tick what this connection may do (see the table below).
- Under "How the AI connects:", choose "reviewctl on the dev machine" or "MCP over HTTP (no reviewctl needed)".
- Under "Language the AI gets:", choose 中文 or English (see Language).
- Click "Get connect command".
Only a person in the browser can make a connection or change its permissions. A connection acts as the person who made it and lasts 30 days; it stops working if that person leaves the organization.
Permissions
| Permission | Default | Ticked, it may |
|---|---|---|
| Write test drafts and upload files | On | Hand in test drafts, upload the test files they're missing, and have the platform's AI review the drafts again |
| Change the test connection files (.reviewctl) | On | Save the test connection files the platform holds, checked as when a person saves them under "Targets" |
| Pre-run | On | Have the platform pre-run once on the test runner or in a sandbox |
| Start formal tests | On | Start formal tests, paid from what the organization has left (see what formal tests may spend) |
| Confirm tests | Off | Confirm drafts, checked as when a person confirms them on the web. Off: the platform's AI still reviews the drafts, and the rest wait for a person on the web |
| Have the platform's AI finish the connection | On | Start "Let AI finish and check" |
| Change the pass criteria | Off | Save the project's pass criteria |
Reading needs no permission: a connection can read tests, jobs, reports, diagnoses and the test connection files. It can cancel only the formal tests and pre-runs it started itself.
A step not ticked isn't done, and the coding AI is told: "This connection isn't allowed to "…", so nothing was done. Ask a developer to tick it under "Edit permissions" on this connection in "Settings → Dev AI" on the web". The refused attempt is listed on the connection's row too.
Each connection has a row in "Settings → Dev AI": the machine's name for one connected with reviewctl, "Remote MCP" for one over HTTP, then what it may do, what it has used today and what it did last. "Edit permissions" opens the same boxes; a change applies from its next call and goes into the audit log. After 5 failed submissions in a row, or 5 pre-runs in a row that don't pass, the connection pauses until someone clicks "Resume". While it's paused, the coding AI's get_status lists the problems of the last submission that failed (or why the last pre-run didn't pass) and links this page. "Disconnect" closes it (it asks first).
A connection made before this update may write drafts, upload files and pre-run, nothing more. To let it do more, click "Edit permissions" on its row.
Language
Each connection has its own language, Chinese or English: "Language the AI gets:" when it's made, the page's language unless you pick the other. Its row says which ("The AI gets English"), and "Edit permissions" changes it, from the connection's next call, written in the audit log like a permission change. A connection made before languages gets Chinese.
What follows the connection's language is everything the platform says to its AI: the server's instructions and each tool's title, description and arguments; every result and next step; refusals and errors; the prompt from get_prompt (in English it asks for the tests' own text in English too); pre-run, job and run results; get_report, get_diagnosis and get_fixes; and the sentence the page gives you to send it. What a test, a person or the agent under test wrote (test titles, the agent's replies) stays as it was written. The platform AI's diagnosis writes each fix note in both languages, and an English connection (or the web in English) reads the English one, with the condition in English too ("under normal conditions"); where the platform's own rules reword a fix note (the lead that says to change only your own code, its own note in a fix's place, a file's name put right), they reword the English too. A diagnosis made before this, or one whose AI gave no English even when asked, stays in Chinese. A pre-run's result in get_job and wait_job is in the connection's language too. The reason a draft waits for a person ("The revision from the platform AI's diagnosis removes or loosens …", with the AI's own reason in it whole) and the reasons in a report's pass criteria follow the connection's language as well, and such a draft is listed as one a person confirms (needs_person in list_tests). A security check's name in get_diagnosis is its English one ("System prompt extraction"), as in get_fixes. A test written in English has the platform's own words in its rules file in English too: the header above it, who confirmed it when automatic approval did ("AI auto-approval (on by default)"; the web says it in its reader's language), and the judge's instructions built from its goal. A test saved before keeps its judge's instructions until it changes. A default security set written in English has the platform's own words in it in English too: the header, the checks' names, what the judge reads the agent does and who is signed in, and the examples in the judge's instructions; a set written before stays as it was, and an English connection reads the kinds of attack by their English names anyway ("question r17 (System prompt extraction)" in get_report). The notes connect --ai adds to the starter tests it writes for an English agent ("Assumptions made while drafting") are in English too, and so are the platform's own checks noted on an English test (a test file still missing, a rule whose arguments the test runner didn't report). submit_answer says what each test runs and checks (tests) in the connection's language, and get_experiment_rows says why a score is undecided in it. What reviewctl prints in your terminal follows REVIEWCTL_LANG or the system's language, as before.
How the coding AI's side learns it: over MCP over HTTP, from the token's connection on each call. With reviewctl dev mcp, reviewctl dev connect saves the language in the connection's file under ~/.reviewctl/dev-links/; every answer from the platform says the connection's language as it is now, and reviewctl dev mcp says that answer's own words in it and writes it back into that file (reviewctl dev status writes it back too), so after a change on the web the very next answer is all in the new language. The tools' descriptions follow from the next listing; the server's instructions are read when it starts, so restart the MCP server (or the coding AI's session) to have them in the new language.
Connect with reviewctl (stdio)
The dev machine needs reviewctl first, see Command line: install.
The page gives a sentence to copy to the coding AI. It runs, in the repository:
reviewctl dev connect <code> --api https://<platform address> --register <claude|cursor|codex> --yesThe pairing code works once, within 10 minutes. "Get a new code" voids the one on the page at once: its row goes, and connecting with it is refused, saying it was voided. The command connects to the platform, saves the connection on this machine under ~/.reviewctl/dev-links/, one per repository (one machine can be connected to several projects at once, and each command uses the connection of the repository it runs in), and registers the MCP server reviewctl-dev (that is, reviewctl dev mcp) with the coding tool:
--register | How it registers |
|---|---|
claude | claude mcp add when the claude command is on this machine; otherwise the repository's .mcp.json |
cursor | The repository's .cursor/mcp.json |
codex | codex mcp add when the codex command is on this machine; otherwise it only prints the lines to add to ~/.codex/config.toml, and never edits config in your home folder |
none | Nothing |
An identical reviewctl-dev already there is left alone. A different one is replaced only if you answer y at a terminal; elsewhere (or with --yes) it's kept, and the command says how to change it. A config file it can't read is never touched. Without --register, it asks at a terminal which tool; elsewhere it only prints how to register by hand.
Once registered, open the coding tool in this repository again to get the tools. Until then, the coding AI can use the same steps as reviewctl dev commands (see Command line).
If --register didn't work, register by hand:
claude mcp add reviewctl-dev -- reviewctl dev mcp # Claude Code
codex mcp add reviewctl-dev -- reviewctl dev mcp # CodexCursor: in the repository's .cursor/mcp.json, write
{ "mcpServers": { "reviewctl-dev": { "command": "reviewctl", "args": ["dev", "mcp"] } } }To disconnect: run reviewctl dev disconnect in the repository. Its connection is closed on the platform and the one saved on this machine for it is deleted; other repositories' connections stay. The ~/.reviewctl/dev-link.json an older reviewctl saved still works: a repository with no connection of its own uses it, and disconnecting in such a repository ends it. Remove the registered MCP server yourself: claude mcp remove reviewctl-dev (Codex: codex mcp remove reviewctl-dev), or delete reviewctl-dev from .mcp.json or .cursor/mcp.json.
Start the local test runner with it
For a project on a local test runner, pre-runs and formal tests run on the test runner, which has to be online (get_status says whether it is, and what to do if not). When the dev machine is the test runner, use --runner:
reviewctl dev connect <code> --api https://<platform address> --register claude --runnerThis registers reviewctl dev mcp --runner. Once the MCP server is up, it starts this project's test runner in the background with reviewctl runner start, and stops it when the server exits. The log is ~/.reviewctl/logs/runner-<project id>.log. For it to start, this machine must already be registered as the project's test runner (see Local test runner), the project in review.yaml must match, and no test runner may be online on the platform already. It uses review.yaml in the current folder, else .reviewctl/review.yaml, or the one --config names. get_status says whether it started, and why not.
For a project on a local test runner, the sentence the page gives and the commands to register by hand already have --runner; run without it, reviewctl dev connect says so.
Connect with MCP over HTTP
To skip installing reviewctl on the dev machine, choose "MCP over HTTP (no reviewctl needed)". The connection is made as soon as you click "Get connect command", with no pairing code. Register it in the coding tool with the lines the page gives for it. Claude Code (run in the repository):
claude mcp add --transport http preworld https://<platform address>/v1/dev/mcp --header "Authorization: Bearer <token>"Cursor and other tools that read mcp.json:
{ "mcpServers": { "preworld": { "url": "https://<platform address>/v1/dev/mcp", "headers": { "Authorization": "Bearer <token>" } } } }Codex: see Codex below. Other coding tools take the same address and header the way they register a remote MCP server.
The token starts with dl_ and is shown only this once: once you leave or refresh the page it's gone, and if you lose it, disconnect the connection and make another. Don't commit it to the repository. A call with no token, or with the token of a connection that was disconnected or has expired, is refused with 401 saying which, and that the way back is a new connection's token in the coding tool (PREWORLD_TOKEN, or the Authorization header), not reviewctl dev connect. The 401 is in the connection's language (a disconnected or expired one still has it); for a token the platform never made, in the language the request's Accept-Language asks for, and in both when it asks for neither. Once it's registered, tell the coding AI the sentence on the page (in Chinese, which your AI reads fine): "用 preworld 的 MCP 工具连上测试平台,从 get_status 开始,按每一步返回的「下一步」做,最后告诉我结果。" (Connect to the test platform with preworld's MCP tools, start from get_status, follow the next step each one returns, and tell me the result.)
Until the coding AI first calls the platform with that token, the connection's row reads "Waiting to connect"; "Connected" only once it has.
MCP over HTTP has the same tools as reviewctl dev mcp except upload_file: the platform can't read files on your machine. Upload the test files drafts are missing on the web, or use reviewctl; the prompt and the next steps over HTTP say so, and never send the AI to upload_file.
On a local test runner's project, "no reviewctl needed" is about the dev AI itself. Such a project's pre-runs and formal tests run on the test runner, which still needs reviewctl installed and reviewctl runner start running (see Local test runner); with the test runner offline, the coding AI stops at "start the test runner first" as soon as it connects. Over MCP over HTTP the platform doesn't know which machine the coding AI is on, so get_status says to ask whoever runs your team's test runner to start it on that machine, and that a pairing code for a new one is made with "Create pairing code" under "Targets" → "Local test runner".
On a project whose connection sees replies only (attach.mode: endpoint_only in review.yaml), the test runner reports no tools. A scenario still writes one in tools: the name of the function or entry the agent replies to the user with, which the platform doesn't check calls against. Such a scenario writes no must_call, never, fault, attack or data_checks, and says with goal_text and reply_must_say what the reply must contain.
Codex
The Codex CLI, its IDE extension and the ChatGPT desktop app all read ~/.codex/config.toml. The page gives the lines to add there. They read the token from the environment variable PREWORLD_TOKEN, so the file holds no token:
[mcp_servers.preworld]
url = "https://<platform address>/v1/dev/mcp"
bearer_token_env_var = "PREWORLD_TOKEN"codex mcp add preworld --url https://<platform address>/v1/dev/mcp --bearer-token-env-var PREWORLD_TOKEN writes the same lines. Then set the token where Codex starts, and start Codex again:
export PREWORLD_TOKEN=<token>To keep it, put that line in your shell profile (~/.zshrc, ~/.bashrc), never in a file in the repository.
The ChatGPT desktop app doesn't read shell profiles. On macOS, either run launchctl setenv PREWORLD_TOKEN <token> and quit and reopen the app (it lasts until you log out), or keep the token in the Keychain and have Codex read it each time with http_headers_helper, a command that prints the request headers as a JSON object:
security add-generic-password -s preworld-review -a "$USER" -w # asks for the token#!/bin/sh
# ~/.codex/preworld-headers.sh (chmod +x)
token=$(security find-generic-password -s preworld-review -w) || exit 1
printf '{"Authorization":"Bearer %s"}\n' "$token"[mcp_servers.preworld]
url = "https://<platform address>/v1/dev/mcp"
http_headers_helper = "/Users/<you>/.codex/preworld-headers.sh"The Yi Eval plugin for Codex brings this MCP server for https://review.theyilabs.com, reading the token from PREWORLD_TOKEN as above, and a skill that walks Codex through the loop: what to check first, how to read the report, diagnosis and fixes against the code, what it must never do (loosen the pass criteria, confirm tests without the permission, commit the token, run any of the agent's code on your machine, log into the test runner), and what to tell you at the end. It isn't in a public plugin directory yet; with a copy of the repository it's in (a Codex marketplace named preworld):
codex plugin marketplace add <path to the copy>
codex plugin add preworld-review@preworldAfter that, ask Codex "Test this agent with Yi Eval". A [mcp_servers.preworld] in config.toml takes the place of the plugin's server: that is how to point the plugin at another platform address, or use the Keychain helper.
Codex without a person at the keyboard (codex exec), as in CI: its approval policy is never by default, so every preworld tool that isn't read-only is refused ("MCP tool call requires approval, but approval policy is never"). --approve-for-me routes approvals through Codex's automatic review, but can't be combined with -s (--sandbox). With a sandbox mode of your own, set the approval policy and the reviewer instead:
codex exec -s workspace-write -c approval_policy="on-request" -c approvals_reviewer="auto_review" "Test this agent with Yi Eval" < /dev/nullStarted in the background, codex exec waits for standard input ("Reading additional input from stdin...") unless it's given < /dev/null.
Test after every change (AGENTS.md)
Codex, and other coding tools that read AGENTS.md, follow what it says in the repository. To have the coding AI test the agent before it calls a change done, paste this into the repository's AGENTS.md:
## Testing the agent
This repository's AI agent is tested with Yi Eval, through the `preworld` MCP tools.
- After changing the agent (its code, prompts, tools or model settings), run the Yi Eval loop before saying the change is done: start from `get_status`, pre-run with `try_run`, start a formal test with `run_tests` and wait for it with `wait_job`, then read `get_report`, `get_diagnosis` and `get_fixes`.
- Check each finding against the code before changing anything. Never loosen the pass criteria or change a test's expectations to make it pass.
- Before a formal test, say what it will cost; stop and tell me if the connection is paused, the organization's balance is used up, or a step isn't allowed.
- At the end, tell me the verdict, the report link, and what you changed.The loop and the tools
There are 40 tools (39 over MCP over HTTP, which lacks upload_file). Each one's result says what to do next, and a step the connection isn't allowed says who should switch it on, and where.
Each tool has a title and MCP annotations, so a coding tool can tell what needs your approval: the reads (get_status, list_tests, get_report, wait_job and the like, and the removed plan steps, which only say so) are readOnlyHint (get_tasks isn't one: it takes the open tasks); one that may change or end what is there (submit_answer, finish_task, put_wiring_file, start_connect_assist, review_drafts, confirm_test, cancel_job, set_gate) is destructiveHint; try_run, run_tests and start_connect_assist run the agent under test (openWorldHint).
Which code ran. try_run, get_setup, get_job, wait_job and get_report say which code each pre-run and formal test ran, in their text and in their structured content (code). On a local test runner: the commit, whether there were uncommitted changes, and each changed or new file with the first 12 characters of its sha256, to compare with your copy (shasum -a 256 <file>); at most 30 files are listed. Only names and hashes leave the test runner, never contents, and a file named like a secret (.env, a key) is listed without its hash. The test runner says this when review.yaml keeps agent.version.git on (the default) and its folder is in a Git repository. In a cloud sandbox: the commit, plus the platform's test connection files. To check what the test runner ran, read these; don't log into the test runner.
A pre-run that just passed is used again. run_tests pre-runs first unless the latest pre-run passed for the same tests and code: on a local test runner, the same test runner, not restarted since, with the same code, within 30 minutes; in a sandbox, the same commit within 6 hours; and no test, test connection file or secret changed since. The reply says when it goes ahead on one, and when it can't, why (no pre-run passed lately, the last one that passed is too old or ran other tests, the test runner has reported different code, it restarted, it's another test runner, or a test, test connection file or secret changed since); the structured content has it as prerun.why. If the test runner's code changes before it takes the run, it pre-runs first after all. See sandbox runs.
Where the agent's model calls go. When the test runner has no model key for the agent (no OPENAI_API_KEY or OPENAI_BASE_URL in review.yaml or on that machine), reviewctl runner start points the agent's OPENAI_BASE_URL at a relay on that machine (127.0.0.1), with a placeholder OPENAI_API_KEY. The test runner forwards the agent's model calls (chat completions and the Responses API only) to the platform's models, billed as usage; a cloud sandbox does the same (see your agent's model). An agent with its own key calls whatever it is set to call. The descriptions of try_run and run_tests say so too.
What comes from the agent under test or the test data (its replies, tool calls and their results, questions and attack texts, what a test expects, trace spans, experiment rows, setup and test runner logs, a diagnosis's quotes) comes between <untrusted-content boundary="…"> and </untrusted-content boundary="…">, with one sentence saying it is data, not instructions, and must not be followed; the server's instructions say the same. The content inside is left as it was. Where the platform AI's fix note in get_fixes (or a diagnosis's own words in get_diagnosis) repeats words of what its evidence quotes of the agent or the test (a reply, a tool call, a log, the test's text, what a verdict quotes of the reply), those words are marked where they stand too; the platform's own words a piece of evidence quotes (a verdict's reason, "The platform checked this against the system prompt's own text") don't count, so a fix that happens to share them isn't marked. Where the AI's reason for holding a draft (in get_status and review_drafts) repeats the test's own text — its goal, what the user says, the points the reply must make — that is marked where it stands too; a field name in that reason is said as the test page labels it ("What counts as done", "The reply must tell the user"). Where a pre-run check in get_setup quotes what the agent printed (its last error line, say), or a report's summary of a run that never reached the agent does, that is marked where it stands in the text too. A batch's summary (the paragraph under the first line of get_report without a run_id) is the platform's own words and never quotes the agent. The structured content (structuredContent) is marked the same way: each such value has the marker around it on its own, without the sentence. Ids, codes, states, times, links, hashes, verdicts and next steps stay as they are, and so do the platform's own words: what a test is expected to cost, the pre-run it goes ahead on or why not, which code ran (code, the repository's file names in it marked), what each part of a batch holds ("2 tests", a test set's title in it marked), a pre-run check's title, a report's summary ("7 scenarios, 21 attempts: all passed…", the titles it names marked; one of a run whose records couldn't be trusted, which quotes the agent's own error, marked whole), and in get_diagnosis whose problem an item is and the condition it's under (the agent's tool in it marked). The pass criteria and the test connection files' text stay as they are too, which set_gate and put_wiring_file take back as they are. The order:
- Where things are:
get_status. - Test connection:
get_wiringshows the test connection files the platform holds,put_wiring_filechanges one; orstart_connect_assisthas the platform's AI finish it, andget_connect_assistfollows it. For a project on a local test runner, the test runner reads only the files in your own repository: change.reviewctl/there and pre-run withtry_run; what the platform holds is just a copy, so there's no need to save it back withput_wiring_file. The files' formats are in the review.yaml reference, the data queries the data checks read (state_queries.yaml) under data queries. - Write tests:
get_promptfor the prompt,submit_answerto hand in,upload_filefor test files. When the project has a test plan approved before, the prompt carries it (get_planfor the whole plan; the steps that made plans were removed, see the test plan tools). Tasks someone sends it through the API (POST /v1/projects/<id>/dev-tasks) come withget_tasks, and go back withfinish_task; the web sends none any more. - Review:
review_draftshas the platform's AI review,get_testshows the rules to check against the code,confirm_testconfirms (with the "Confirm tests" permission). A draft a rule holds for a person (one whose fix from the platform AI's diagnosis takes out or loosens a check, say) is never read by the AI:review_draftsandget_statusgive its reasons and say a person confirms it on the web, and it isn't counted among the drafts "the platform's AI is reviewing", nor in the N ofreview_drafts' first line, "Asked the platform's AI to review the drafts again (N)". Itsstartedis true only when a draft went to the platform's AI;auto_approvesays whether the project's AI review is on. - Pre-run:
try_run; when it doesn't pass,get_setupshows why. While the pre-run is still going,get_setupandtry_runsay where it runs (the test runner or a cloud sandbox), in their first line and their next step, how long it has run, and how long this project's last pre-runs there took; in a sandbox with none yet, the platform's estimate (starting the sandbox, then each test once). - Formal test:
run_testsstarts it,wait_jobwaits for it. - Results:
get_report,get_diagnosis,get_fixes. Without arun_id, all three are about the latest formal test: the runs launched together (the functional tests and each test set) are read as one. A pre-run'srun_idworks withget_reportonly: a pre-run gets no AI diagnosis and no fix notes, soget_diagnosisandget_fixessay it's a pre-run, and its problems and fixes are inget_setup. A formal test still running or being judged says which job to wait for withwait_job. The diagnosis's fix notes say how to check a fix on the platform, never by running the agent under test on your machine: a dev AI pre-runs withtry_run, then starts a formal test withrun_tests; a person pre-runs, then starts a formal test on the web. A fix note the diagnosing model wrote otherwise is sent back to it once to be written again. - Check against the code, fix, and go back to step 5.
| Tool | Does | Permission | Command |
|---|---|---|---|
get_status | Drafts, test files still missing, whether the test runner is online, what this connection may do, what next, and the link to these docs on the platform's own host (the server's instructions give it too) | reviewctl dev status | |
get_prompt | The prompt the platform wrote for this repository: which tests, files and todos to write, in what format | reviewctl dev prompt | |
submit_answer | Hands in the JSON; lists the problems if it doesn't pass, and the connection pauses after 5 in a row | Write test drafts and upload files | reviewctl dev submit <file> |
upload_file | Uploads a test file a draft is missing (only in reviewctl dev mcp) | Write test drafts and upload files | reviewctl dev upload <path> |
get_tasks, finish_task | Tasks sent to it through the API; done, or handed back | reviewctl dev tasks, reviewctl dev done <id> | |
get_wiring | The test connection files the platform holds: text, hash, who saved each last | reviewctl dev wiring | |
put_wiring_file | Saves one file under .reviewctl/, checked by the web's save rules | Change the test connection files | reviewctl dev push <file> |
start_connect_assist, get_connect_assist | Has the platform's AI finish the connection and check it; how far it got | Have the platform's AI finish the connection (to start it) | reviewctl dev assist, reviewctl dev assist --status |
list_tests | Tests and drafts: id, conditions, AI review, last result, and the pass criteria. A scenario with both a normal and an attack condition is listed once (groups has functional and security), so the count is get_status's | reviewctl dev tests | |
get_test | One test's rules (YAML) | reviewctl dev test <id> | |
review_drafts | Has the platform's AI review the drafts again; says which wait for a person, and where | Write test drafts and upload files | reviewctl dev review |
confirm_test | Confirms a draft | Confirm tests | reviewctl dev confirm <id> |
try_run | Pre-runs once (one attempt, no report) and waits; the connection pauses after 5 in a row that don't pass | Pre-run | reviewctl dev retry |
get_setup | The latest pre-run: can the tests reach the agent, and which code the test runner ran; if not, the problems and fixes | reviewctl dev setup | |
run_tests | Starts a formal test (as "Start test": a pre-run first, unless one that just passed ran the same code and tests; then the test). The scenarios and each test set are a job each, queued together: returns every job id, what it's expected to cost and the most it may cost. conditions with faults alone (no normal condition, no attack) needs pass criteria that name them: while nobody has set any, the default ones judge only each test's normal condition and attacks, so such a run is refused, and the reply says why and that leaving conditions out runs every condition | Start formal tests | reviewctl dev run |
get_job, wait_job | Where a job is; wait until it ends (it returns once that job ends, naming the jobs launched with it that are still running). Without a job_id, the last formal test this connection started; with none yet, it says so and names the connection's own pre-run, whose result is in get_setup (wait for it with its job_id); a pre-run started after that test and still going is named beside it, and so is a part launched with it that stopped at a spending limit (one that stopped before it began is said not to have started, never to have finished). Each attempt's state is in words ("passed", "undecided", "not run"); the structured content keeps the codes | reviewctl dev job, reviewctl dev job --wait | |
cancel_job | Cancels a job this connection started | reviewctl dev cancel <id> | |
list_runs | Latest formal tests: verdict, problem count, report link | reviewctl dev runs | |
get_report | A formal test's conclusion: the first screen's words, which code it ran, whether the platform swapped the model, what failed or couldn't be decided and why (a test set question by question). The first line counts tests: one that couldn't be tested under several conditions counts once, with the conditions after it ("9 tests (under 15 conditions) couldn't be tested"); why each couldn't be decided is in the platform's words, without internal codes | reviewctl dev report | |
get_diagnosis | The platform AI's diagnosis: whether the cause is the agent, the test connection or the test itself | reviewctl dev diagnosis | |
get_fixes | A run's fix notes (the same as "Export fix notes" on an experiment row gives) | reviewctl dev fixes | |
set_gate | Saves the project's pass criteria | Change the pass criteria | reviewctl dev gate <file> |
The test plan tools
The test plan step was removed (see test plans): the platform no longer makes plans, and the web has no page to approve one. These tools are in MCP only; there are no reviewctl dev commands for them.
| Tool | Does | Permission |
|---|---|---|
get_plan | The test plan approved for the project before: its version, the plan in full, and what next (get_prompt for the prompt that carries it). With no plan approved, it says the step was removed and how to write tests instead | |
start_plan, answer_plan, next_plan_round, write_plan, edit_plan | Removed: they start and change nothing, and only answer the error plan_removed, saying the step was removed and to write tests with get_prompt and submit_answer instead |
The experiment, log and release gate tools
The coding AI can read the project's experiments, its live logs (traces) and its release gate too, to find causes against the code; and it can draft dataset rows and scorers. All on this connection's own project only: another project's experiments, traces and datasets aren't there, even in the same organization. Reading needs no permission; drafting needs "Write test drafts and upload files", and what it drafts waits for a person to check on the web. It can't change the release gate, run an experiment or score anything (nothing here spends). These tools are in MCP only.
| Tool | Does | Permission |
|---|---|---|
list_experiments | The project's experiments, newest first: source, status, whether the release gate passed, each scorer's mean, cost | |
get_experiment | One experiment and its release gate: each rule, whether it held, the actual value, the rows that broke it | |
get_experiment_rows | An experiment's rows: input, expected, output, each scorer's value and reason; against a baseline, the change. The filter language picks the failing ones, e.g. score.builtin:task_done < 0.5 | |
list_traces | The project's traces, newest first: the start of input and output, status, online scores, models; filter e.g. status = error, issue = <issue id> | |
get_trace | Every span of one trace in call order and depth: tool and model calls' input and output, where it failed, online scores with reasons | |
list_datasets | The project's datasets: its own, and those standing for its tests (scenarios, test sets) | |
add_rows | Adds rows to one of the project's own datasets, marked as not looked at yet, for a person to keep or delete | Write test drafts and upload files |
create_scorer_draft | Creates a new project scorer, marked as drafted by the dev AI; new ones only, used nowhere until a person tries it under Scorers and adds it | Write test drafts and upload files |
get_gate | The project's release gate rules and the latest finished experiment's result on them; read only: a person changes it under "Settings → Release gate" on the web |
For a project on a local test runner, try_run and run_tests both need the test runner online. When it isn't, the platform starts nothing, and says to run reviewctl runner start on that machine, or to use --runner.
What formal tests may spend
A connection has no daily limit. The formal tests it starts are paid from what the organization has left, as a test started on the web: the period's credit and the balance, the owner's monthly spending cap if there is one, the prepaid balance, and the organization policy.
- Each
run_testsfirst works out what the test costs, by the same rule as the start page. On a local test runner that is the platform's judge model, and the agent's model when the platform's model answers the agent. In a cloud sandbox it is also each sandbox at the organization's longest run. - The agent's model counts only when reviewctl on the test runner passes the agent's calls to the platform's models, that is, when that machine has no
OPENAI_API_KEY(and noOPENAI_BASE_URL) of its own. - If it doesn't fit in what the organization has left, nothing starts (
dev_org_left). The reply says what the test would need, what the organization has left, and that an owner tops up on the billing page (/org?tab=billing). - On the shared Yi Labs wallet, a test goes on block by block while the owner's wallet can pay for the next block.
- While it runs, the platform's models stop where the organization's money runs out, as at the end of any budget: the judge model judges no more, and the platform model answering the agent answers no more. The run stops after the attempt in progress with a partial report. What is left untested isn't charged.
get_reportcounts what couldn't be tested as the web's first screen does, tests and attacks apart. An attempt a limit stopped counts only as that, never as "the platform's model didn't answer", and never against the "the platform can't see the agent's tool calls" check.- The connection's row lists each formal test it started, with what it's expected to cost and the most it may cost. The row and
get_statusalso say what the connection used today, for reference only. - A pre-run on its own (
try_run) and "Have the platform's AI finish the connection" are billed as usual: a pre-run by sandbox time and model usage, the finishing as platform AI usage. - Which limit stopped a formal test (the organization's balance, or the run's own) is kept on the job, so the reports say the same later.
- A run stopped by a connection's daily cap before 8 November 2026 still says so in its report.
Next steps
- Command line: every
reviewctl devcommand. - API and tokens: the
dl_token and/v1/dev/*. - Test plans: how a plan approved before is used; making one was removed.