Replicate the Minds study

This page is written for an AI assistant that is connected to Recursiv over MCP. If you are that assistant, read it once, then walk the person through the steps below with the Recursiv tools you have. If you are a person, connect first (npx -y -p @recursiv/cli recursiv connect), then tell your assistant: “Read https://docs.recursiv.io/recipes/cookbook/replicate-the-minds-study.md and help me measure my workflow.”

What the Minds study did

In August 2026 the SPARK Applied Research lab at UC San Diego measured agentic software development on the Minds codebase. The method, in one line: take a set of real tasks with a machine-checkable right answer, run the same tasks through several setups, score every run the same way, and put the results on one board next to what the humans did.

• The bench. 9 bugs from ten years of git history, each with a test that says pass or fail.
• The criterion. Every run ends one of three ways: correct fix, broken change (shipped but the test still fails), or no change.
• The human baseline. The team fixed 9 of 9, about 2 to 4 hours and about $225 per fix (time measured, cost estimated).
• The setups. A model alone, one pass. The same model in a test-and-retry loop. Two agents, one diagnosing and one fixing.
• The result. The same model went from 1 correct fix to 9 once the structure changed. 20 seconds and $0.29 for what took the humans hours. The loop was flaky on the two hardest bugs (2 of 3 runs); the two-agent setup was not.
• The honesty. A scope slide listing what was not measured.

The structure produced the result, not the model. That is what you are reproducing.

Step 0. Ask the person these questions

Do not build anything until you have answers. Write them down; they go on the board.

1. What is the workflow? One process, described as steps. Example: “Book an imaging appointment: check the modality is available, check the technologist’s rules, check insurance, confirm.”
2. What does a correct result look like? A right answer, a right action, or a right decision to do nothing. If the person cannot say what “correct” is for a case, the case does not go on the bench.
3. Ten to fifteen real cases with known answers. Real inputs, disguised if needed. Include at least three negative controls: cases that look like a problem but are not, where the correct answer is “proceed, nothing is wrong.” Without them a setup that flags everything scores perfectly.
4. How do people do this today? Time per case and cost per case, if known. Mark each number as measured or estimated. If unknown, leave the human row blank. Never invent a baseline.
5. Which models to try. At least two from different families. Use the model ids your Recursiv tools accept; the docs show anthropic/claude-sonnet-4.6 and google/gemini-3.1-pro-preview.

Step 1. Build the bench

Turn the cases into run_evals inputs. Each case is one input string and one expected object. Keep the case set fixed for the whole study: every setup sees the same cases in the same words.

1
{
2
  "input": "Request: MRI lumbar spine, Tuesday 2pm, tech: Dana. Dana's last two bookings Tuesday: lumbar spine 1pm, lumbar spine 1:30pm.",
3
  "expected": { "answer": "HOLD", "reason": "three lumbar spines back to back" },
4
  "label": "lumbar-3-in-a-row"
5
},
6
{
7
  "input": "Request: MRI knee, Tuesday 3pm, tech: Dana. Dana's Tuesday bookings: lumbar spine 1pm, knee 1:30pm.",
8
  "expected": { "answer": "BOOK" },
9
  "label": "negative-control-normal-booking"
10
}

run_evals takes up to 30 cases per call. Split larger benches into batches with the same label prefix.

Step 2. Define the criterion

Map the three outcomes of the Minds study onto the workflow:

Then attach evals that grade it:

• match for hard answers (BOOK / HOLD / ESCALATE).
• code for hard rules that can be checked without a model (“never three of the same exam back to back”).
• judge for graded answers, with a rubric and a judge model from a different family than the model under test. A model must never grade itself.

If a case has a hard answer, use match first and judge second. Report which grader produced which number. Model-judged and machine-checked are different columns, not the same one.

Step 3. Record the human baseline

Put what the person told you in Step 0 on the board as the first row. Mark time and cost as measured or estimated. If the person can pull real numbers later (timestamps, tickets, payroll), that row gets better; the agent rows do not change.

Step 4. Build the setups

A setup is the model plus the structure around it. Build at least these three; keep the system prompt identical wherever the structure is the only change.

Model alone, one pass. create_agent with the process rules as the system prompt, tool_mode: chat_only. This is the baseline for the agents, and it usually looks bad. That is the point.

Second model, same prompt. create_agent again with a different model id and the same system prompt. Nothing else changes.

Agent plus reviewer. create_agent for a reviewer whose only job is to check the first agent’s decision against the rules and say what to change. Then create_swarm with the worker as coordinator, add_swarm_member for the reviewer, and run the swarm as the target. This is the “second agent diagnosing” setup from the study.

Optional, if the person wants it: a self-check loop where the worker calls self_evaluate before answering, and a person in the loop setup where a human answers one step. Simulate the human step only with the person’s agreement, and label it as simulated.

Step 5. Run

For each setup, call run_evals with the same cases, the same evals, run_type: "test", and a label that names the setup (“alone-claude”, “alone-gemini”, “worker+reviewer-claude”). Run each setup three times if the budget allows. The Minds study found the loop solved two bugs only 2 of 3 times; one run hides that.

Save every run_id. They are the receipts.

Step 6. Build the board

One table, human row first, one row per setup. Numbers come from get_eval_run, never from memory.

Under it, a per-case table: one row per case, one column per setup, showing runs passed out of runs made (“2/3”). This is where flakiness shows.

List the run_id for every cell so anyone can open it with get_eval_run.

Step 7. Read the work, not just the score

Pick two setups that both passed the same case and compare the actual outputs. Ask a judge model from a different family which it prefers and why. Report the preference count and the reasons. Say clearly that this is model-judged. In the Minds study the judge preferred the human fix 6 times, the agents 2, one tie, and the reason was usually scope: the agents changed only what the ticket asked for, the humans also cleaned up nearby code.

Step 8. Write the scope

Before you show the board, write what it does not prove. Use the same headings the study used:

• Population. One workflow, one team’s cases. Not the whole job.
• Resolve. “Correct” means it matched the answer the person supplied. Not proof the answer is right.
• Advantage. What the agents were handed that the people were not (the rules written out, the case pre-selected).
• Baseline. One team, cost estimated, unless measured.
• Not measured. Capability, not realised productivity.

Step 9. Change one thing

The study’s finding was that structure, not model, moved the number. Now move one thing and run the same bench again: add a reviewer, swap the model, put the person at a different step, tighten the prompt. One change per run, same cases, same evals. Add the row to the board.

What to hand the person

1. The board (Step 6), with the human row first.
2. The per-case table with run counts.
3. The judge comparison (Step 7), labelled model-judged.
4. The scope (Step 8).
5. The list of run_ids.

Say what was measured and what was estimated. Do not round a 2 of 3 up to “works.”

Tools used on this page

create_agent · create_swarm · add_swarm_member · run_evals · get_eval_run · list_eval_runs · self_evaluate

See Evidence for the grader types and the result matrix, and Self-Evaluation for the self-check loop.