Testing AI systems has moved from a quiet engineering concern to the question that decides whether a system ships at all. This report is our yearly read on where that work stands: what holds up in production, what breaks in public, and where the discipline is heading by 2027.

Why this report exists

We write it for three readers. Founder-operators commissioning AI work, who need a way to tell shippable from not. Engineering leads, who want a yardstick for their own practice. And investors reading AI claims during diligence. The four levels and the twelve questions are meant to be useful in all three conversations.

We are not the first to write about this. The Anthropic engineering blog1, OpenAI's published research and engineering posts2, talks from the AI Engineer conferences3 and open-source frameworks such as OpenAI Evals4 shaped much of the vocabulary. Our contribution is the operator's side: what we see when we sit with the teams that run these systems, and where the discipline holds or cracks.

Four levels of maturity

The AI systems we reviewed fall into four levels. The level is set by the engineering artefacts in the repository, not by how good the demo looks, and most systems sit lower than their teams believe.

Figure 1 · Where the systems we reviewed sit

  • Level 0 · VibeAbout half
  • Level 1 · ScriptedAbout a tenth
  • Level 2 · Eval-gatedAbout a quarter
  • Level 3 · Eval-drivenAbout an eighth
Estimates from the systems we reviewed, not a survey. Observation window: 2025 to May 2026. Shares are rounded and do not add up to exactly one.

Level 0Vibe

No fixture set, no rubric, no baseline. The team ships when the demo feels right and rolls back when the operator complains. Often there is a page called “evaluation” holding five hand-typed prompts and an opinion. This was the most common level by a distance: about half of the systems we reviewed.

Level 1Scripted

A few smoke-test prompts, run by hand before each release. No versioned fixtures, no written rubric, no stored baseline. Regressions are found by users rather than by gates. Roughly a tenth of the systems we reviewed sit here.

Level 2Eval-gated

A fixture set in version control, a written rubric, and a script that produces scores on demand. People read the scores before each release; the gate is human judgement. This level is competent and shippable. Roughly a quarter of the systems we reviewed reach it.

Level 3Eval-driven

Level 2 plus an automated regression gate in CI. A pull request that drops below tolerance on any metric fails. Baselines move only when a release ships green, fixtures grow whenever production surprises someone, and the grader model is pinned. Roughly an eighth of the systems we reviewed are here. These teams tend to be the quieter ones, because the scoreboard settles most arguments for them.

Field observation

In the systems we reviewed, the level claimed in a pitch was usually higher than the level in the codebase. The quickest diligence question is still: can you show me the eval script and last week's scores?

Six patterns from the field

Patterns we and the operators we work with saw recur through 2025 and 2026. Each comes from a real incident or a shape that kept repeating.

Pattern 1Grader-model drift

Teams using a hosted model as their grader saw scores shift as the provider updated the model. The harness had not changed and the workflow had not changed, but the numbers moved. The fix is to pin the grader like any other dependency, re-run the whole fixture set when the pin moves, and rule the grader out before reading a score change as a workflow change.

Pattern 2Fixture decay

Fixtures captured at the MVP stop representing production six to twelve months later. Rare edge cases become common, and the harness keeps scoring well on stale examples while customers see worse behaviour. Every operator surprise should become a new fixture; the set should look like a year of real traffic.

Pattern 3The multi-agent gap

Scoring a single model call is now well understood. Scoring a flow, where a planner hands off to retrieval, then to drafting, then to a critic, is not. Per-agent rubrics are necessary but not enough: the flow has its own properties (handoff fidelity, end-to-end coverage, a latency budget) that need their own grading.

Pattern 4The cost of evaluating

LLM-graded rubrics with several dimensions per fixture are not cheap. In some of the systems we reviewed, one full eval run cost more than a day of the workflow it tests. Teams respond with tiers (a small smoke set per pull request, the full set nightly), deterministic checks first, and sampled runs. Teams that have not redesigned for cost tend to run the full suite rarely, and ship blind in between.

Pattern 5Grader and operator disagree

The rubric says green; the operator reads the output and says it is wrong. The rubric has learned the shape of a good answer (sections, length, tone words) instead of what matters: does it move the customer forward, does it sound like us. The fix is weekly human review of five to ten percent of outputs, with every disagreement fed back into the rubric.

Pattern 6Retrieval goes untested

When a retrieval-augmented system fails, the model usually gets the blame. In our experience the cause is more often retrieval: the wrong documents, unhelpful chunking, a query rewrite that dropped context. Few of the teams we reviewed test retrieval on its own, so they are debugging a two-stage system with one instrument.

Three principles taking hold

Among the teams at Level 2 and Level 3, three practices recur.

  1. Fixtures from real traffic. Where fixtures come from is the biggest single decision in a harness. Real inputs catch real failures; imagined personas catch only what the team already expected.
  2. Pin the grader like the model. Fix the grader's version and prompt, and treat a change to either as a reason to re-run everything against a new baseline.
  3. The baseline owns the release. The regression check is automated, the tolerance is set per metric, and a regression past tolerance fails the pull request. The discipline works when the gate cannot be argued with.

The self-audit · twelve questions

Answer each question from the codebase, not from what the team intends. It takes about thirty minutes with the engineering lead and an operator in the room: the lead points to a file, a commit or a CI config for each yes, and the operator watches. The score maps to the four levels above.

Tick each question you can answer yes to

Score 0 of 12. Level 0 · VibeThe system ships when someone says it feels right. Start with question 1: a fixture set is the unlock.

Five predictions for 2027

Forecasts, not certainties. We will check each one in the 2027 edition and say which moved the way we expected.

  1. Retrieval evals become routine. Separate harnesses for retrieval quality (recall, rerank precision, chunking) and for generation. Testing a RAG system without retrieval metrics will look like shipping a backend without integration tests.
  2. Open frameworks for multi-agent flows. Tools for grading handoffs, flow coverage and per-agent accountability, closer to distributed tracing than to unit tests.
  3. Grading sold as a service. Pinned, audited grader models with their own service levels and change logs, so teams stop running their own for common rubric shapes.
  4. The harness becomes an interview question. “Walk me through the eval harness on the last AI system you shipped” becomes as standard for AI engineers as asking a backend engineer about their tests.
  5. The gap shows in public. Level 0 and Level 1 teams account for a visible share of incident reports and forced rollbacks, and procurement language starts to ask for eval evidence.

Method and limits

The report draws on three sources. Our own work: AI workflows we shipped through eval harnesses since 2025, across CRM enrichment, customer replies, document summaries, sales-call recaps and hospitality and marketplace operations. The open literature: posts, talks and frameworks published since 20251,2,3,4. And conversations with founder-operators in hospitality, marketplaces and B2B services who have commissioned AI work or assessed it in an acquisition.

The shares in Figure 1 (about half, a tenth, a quarter and an eighth) are our estimates, not measurements, and they describe the systems we saw, not the market. The patterns are a read of how often things recurred in that same set. The predictions are there to be checked. If your team would like to take part in the 2027 survey, write to hello@morvion.com.

The how-to companion is The Morvion Eval Spec, with an open template on GitHub5.

Sources

  1. [1]Anthropic, Engineering blog. anthropic.com/engineering
  2. [2]OpenAI, News, research and engineering posts. openai.com/news
  3. [3]AI Engineer, conference site and recorded talks. ai.engineer · youtube.com/@aiDotEngineer
  4. [4]OpenAI, Evals, open-source evaluation framework. github.com/openai/evals
  5. [5]Morvion, eval-spec, open template repository (MIT). github.com/aloalads/eval-spec

Cite this report

Morvion. (2026). The State of Eval-Driven AI 2026.
Morvion field reports, v1.1.0.
https://morvion.com/state-of-eval-2026

Excerpts and figures may be reproduced with attribution. The report is republished each May; revisions to this edition bump the minor version.

Questions

What changed between 2025 and 2026?
Three things. Grader drift became a routine problem as hosted models updated under teams that had not pinned versions. Multi-agent eval became the open question, while single-agent eval became expected. And the cost of running rubric suites started to matter, so fixture sets now get designed with a budget.
Who is this report for?
Founder-operators commissioning AI work, engineering leads building it, and investors reading AI claims during diligence. The scorecard is built so the answers come from the codebase, not from the pitch deck.
How often is it updated?
Once a year, in May. The 2026 edition was first published on 18 May 2026 and last reviewed on 10 October 2026.