LLMVibeCheck guide

Think of it like hiring. You assemble a small team, for example one model that plans, one that does the work and one that checks it. You give the team a task. LLMVibeCheck records everything the team does, what it cost and how long it took, and then scores the result.

Run a few different teams on the same task and the Compare page tells you which one to use: the one that gets the job right for the least money and time.

1. 1
   Open Build. A three-step squad is already on the canvas.
2. 2
   Pick a task from the Choose a task… list at the top right, or create one with New task in the side panel.
3. 3
   Click Run squad. You'll watch each step light up as it works. Most runs finish in under a minute.
4. 4
   When it's done, click Judge this run. Hidden tests and three AI judges score the result.
5. 5
   Change a model or two, give the squad a new name, and run the same task again. Then open Compare to see which squad wins.

Start from a template

Open Start from a template… in the top bar and pick one:

Add a step

1. 1
   Choose a model in the model list next to Add model. Popular models are at the top; the rest are grouped by company.
2. 2
   Click Add model. A new box appears on the canvas and the view zooms to fit it.
3. 3
   Use Add tool for a voice tool, or Add approval for a point where you approve the work yourself.

Change a step

Click any box. The side panel lets you rename it, choose its role, swap its model (type in the search box, e.g. “gemini”), edit its instructions, tick skills and set an answer length limit. Click Remove to delete the step.

Connect steps

Each box has a dot on its left (work comes in) and on its right (work goes out). Drag from one box's right dot to another box's left dot to pass work along. To remove an arrow, click it and press Delete.

Steps that don't depend on each other run at the same time, which saves time.

Save

Give your squad a name in the box at the top left and click Save. Squads are also saved automatically when you run them. Your saved squads are listed on the home page.

Click the step that should be able to send work back (usually the reviewer). Under Passes work to, set send back up to to a number such as 2. The arrow becomes a dashed orange line underneath the boxes.

When the reviewer is happy it starts its answer with the word APPROVED and the run moves on. Otherwise its comments go back to the earlier step, which tries again. This repeats at most the number of times you set, so a run can never loop forever.

Add an approval step with Add approval and connect it between two steps. When the run reaches it, a yellow banner appears on the run page showing the work so far. You can:

• Approve: pass it on unchanged.
• Approve with my edits: change the text first, then pass it on.
• Stop the run: end it here.

If you don't answer, the step waits up to 30 minutes (you can change this) and then either approves or stops, whichever you chose for If you don't answer.

Text to speech reads the previous step's text aloud and saves an audio file you can play on the run page. Speech to text listens to audio and writes down what it hears.

The “Voice round-trip” template uses both, so a final model can check that nothing was lost in the conversion. Arrows that carry audio are purple.

1. 1
   On the Build page, open New task at the bottom of the side panel.
2. 2
   Give it a short title and describe exactly what you want back.
3. 3
   Choose a type (General, Coding or Audio) and click Create task. It's selected automatically for your next run.

Writing a good task

• Say what the finished result looks like: “a 150-word product description with a headline”, not “write about our product”.
• List the rules: length, tone, format, things to include or avoid.
• For code, name the function and give two or three example inputs and outputs.

After a run, LLMVibeCheck takes the code from the final answer and runs it against the task's hidden tests, in a safe sandbox that can't reach your files or keys. You'll see a result such as 24/24 passed.

• All tests pass: the run is marked Complete.
• Some pass: Partial.
• None pass, or the code doesn't run at all: Failed.

Tests are the strongest signal LLMVibeCheck has, because they check facts, not opinions. When a task has tests, they decide whether the run is complete.

1. 1
   Open Skills, give the skill a name and write the instructions, e.g. “Use type hints and docstrings.”
2. 2
   Click Add skill.
3. 3
   On the Build page, click a step and tick the skill under Skills. That step now follows those instructions.

You can Edit or Delete a skill at any time. A skill that a squad still uses can't be deleted; untick it from those steps first.

Pick a task and click Run squad. You'll land on the run page, where:

• Each box shows Running, Done, Error or Waiting for you.
• A progress bar shows how many steps have finished.
• Under each finished box: its cost, time and tokens.

Final output

The panel under the canvas shows the team's finished work. Code appears in its own box with a Copy button; Copy all copies everything. Drag the line above the panel, or use Expand and Collapse, to resize it.

Trace tab

Click any box to see what that step was asked, what it answered for each attempt, its cost and time, and what the judges thought of it. Under Conversation you can follow every message between steps, in order.

Cost & time tab

Shows the total cost, a projection for 1,000 runs, total time, time spent waiting for you, and a bar for each step showing its share of the cost and time. The biggest bar is usually the first place to try a cheaper model.

Why three judges from three companies?

A model can be biased toward work written in its own style. Using judges from Anthropic, OpenAI and Google, and averaging them, cancels most of that out. Each judge also sees the hidden test results, so it can't call broken code perfect.

The Evaluation tab shows each judge's score and reasoning. If they differ by 3 points or more you'll see Judges disagree, a sign that the result is debatable and worth reading yourself.

The judges also score each step and say whether it did its job. That's how you spot, for example, a reviewer that rejected work which was already correct.

On the Evaluation tab, choose 1 to 5 stars, tick Task is complete if it is, and say what's right or wrong, ideally naming the step that let it down. Click Submit verdict.

An AI reads your comment and notes whether it was useful (specific), vague (“it's fine”) or contradictory (5 stars but “it's wrong”), which steps you blamed, and whether you agree with the judges.

1. 1
   Open a finished run and click the step you want to change.
2. 2
   In Try a different model from this step, type a model name, or leave it empty to rerun with the same model.
3. 3
   Click Fork run. Steps before it are reused at $0; that step and everything after it run again.

It's the cheapest way to answer “would a cheaper executor have been good enough?”. Forks are listed under Forks of this run on the Cost & time tab.

Quality (out of 100)

The average of whatever checks exist for the run: the judge panel's score (×10), the share of hidden tests passed (×100) and your star rating (1 star = 0, 5 stars = 100). Tests and judges count equally; your rating counts half as much by default.

Overall (out of 100)

Overall = Quality, reduced for being pricier, slower or needing more help than the best squad on the same task. With the default settings:

• Costing twice as much as the cheapest squad takes away up to 10% (half of the 20% cost weight).
• Being twice as slow as the fastest squad takes away up to 5% (half of the 10% speed weight).
• Each time you had to edit or stop the work takes away about 3%.

So a squad that fails scores close to 0 however cheap it is, and the best, cheapest and fastest squad scores 100. Change the weights under How the overall score works at the bottom of the Compare page.

Cheapest correct answer

What you'd actually pay, on average, to get one correct answer: average cost per run ÷ share of runs that were correct. A squad that never got it right shows “never”. Use this when correctness is all that matters.

On the Compare page, select a squad and click Run 3× more in its breakdown. It runs the same squad on the same task three more times (five at most per click) and judges each run automatically. Progress shows on the page.

Each squad then shows how many of its judged runs were correct, for example 2/4 correct, and its cost per correct answer updates.

Every model has a price per million tokens read (input) and written (output). Prices are shown next to each model in the model list. LLMVibeCheck adds up the real cost of every step and shows it everywhere: on each box, on the run page, and on Compare.

Per 1,000 runs on the Cost & time tab shows what a squad would cost if you used it at scale. That's often what decides between two squads that are equally good.