The Morvion Eval Spec

The Eval Spec defines what an evaluation harness for an AI system must contain, the rules that decide whether a release ships, and four worked examples for workflows we meet in most engagements. Reading this takes about eight minutes. A first working harness takes about four days (see the plan below).

Why this spec exists

AI outputs are not deterministic. Without a harness, a model swap, a prompt edit or a change to retrieval can make the system worse without anyone noticing. The harness is the one artefact that survives those changes, and the only objective answer to “is this better than last week?”

We wrote the spec after projects where we built the eval first, projects where we built it second, and one where we did not build it at all. Others have written about the same discipline, notably on the Anthropic engineering blog² and in OpenAI's open-source Evals framework³. The template that goes with this page is public on GitHub¹.

An AI system without evals is a vibe. A vibe is not a product.

The three layers

Every working harness has three layers. Skip one, and the harness flatters the system instead of judging it.

Layer 1Fixtures

A curated set of real inputs, each labelled with what a good output looks like. Fifty to two hundred examples is enough to start. Take them from real traffic, not from imagined personas. If you cannot produce a fixture set, you do not understand the problem yet, and the system will show it.

{
  "id": "fix-001",
  "input": {
    "company": "Aperitivo Bar Zurich",
    "industry": "hospitality",
    "context": "..."
  },
  "expected": {
    "category": "small-venue",
    "key_signals": ["live-music", "aperitivo", "zurich"],
    "draft_tone": "warm, specific, never generic"
  },
  "labels": { "difficulty": "medium", "edge_case": false },
  "source": "real-traffic-2026-04-12"
}

The set lives in version control. Whenever a real output surprises an operator, a new record goes in. Old records stay unless the workflow itself has changed.

Layer 2Rubric

The written definition of “good” for each input class, versioned next to the prompts. A prompt change that needs a new dimension bumps the rubric version too.

workflow: customer-reply-draft
version: 4
dimensions:
  - name: factual_correctness
    grader: deterministic
    check: cited_sources_match_retrieval
    weight: 0.4
  - name: tone_alignment
    grader: llm
    prompt: |
      Does this draft sound like the brand voice in the
      reference samples? Score 0-3 where 0=robotic,
      3=on-voice.
    weight: 0.3
  - name: actionability
    grader: llm
    prompt: |
      Does the draft propose a clear next step the
      customer can act on? Score 0-2.
    weight: 0.2
  - name: refusal_appropriateness
    grader: deterministic
    check: refused_when_out_of_scope
    weight: 0.1

Layer 3Regression gate

A baseline score per metric, stored with every release. A new release ships only when no metric falls further below its baseline than an agreed tolerance. It is the most valuable layer and the one teams skip first.

Illustrative pseudo-code, not a published package

# .github/workflows/eval-gate.yml (illustrative)
- name: Run eval harness
  run: morvion-evals run --workflow customer-reply --fixtures ./fixtures
- name: Check regression
  run: |
    morvion-evals compare \
      --current ./evals/current.json \
      --baseline ./evals/baseline.json \
      --tolerance 0.02
  # fails if any weighted dimension dropped more than
  # 2% below the stored baseline

There is no morvion-evals package; the commands stand in for whatever runner you use. The gate files in the template repository¹ show the real shape. The baseline moves only when a release ships green, and tolerances are set per workflow: tighter for factual correctness than for tone.

Choosing a grader

Use the simplest grader that answers the question reliably.

1. Deterministic first. If a check can be written in code (does the JSON parse, does the function call match the schema, is the output under budget), write it in code. It is free, fast, and it does not drift.
2. Pin the LLM grader. A reference model scoring against a written rubric is part of the harness, not the product. Pin its version, and re-run the whole fixture set when you change it.
3. People are the audit layer. Sample five to ten percent of outputs each week for human review, and feed disagreements back into the rubric.

Four worked examples

Four workflows we meet in most engagements, with the eval shape that has worked for each. The full examples, fixtures included, are in the repository¹.

A four-day plan

1. Day 1. Put fifty real records into a fixture set, before writing any prompt. If you cannot, the workflow is not ready for AI.
2. Day 2. Write the rubric on one page and have the operator and the engineer both sign it before any code.
3. Day 3. Build the harness: a script that runs the workflow on every fixture and prints the scores. Store the baseline.
4. Day 4. Wire the gate into CI, so a pull request that regresses past tolerance fails.
5. After that. Add a fixture whenever a real output surprises someone. Move the baseline only when a release ships green.

A quick test for any AI vendor: ask to see the eval script and last week's scores. If neither exists, nothing stops the next release from being worse than this one. For where teams stand on this today, see the State of Eval-Driven AI 2026; for the practice that builds these systems, see Intelligent Systems & AI Infrastructure.

Sources

1. [1]Morvion, eval-spec, open template repository (MIT). github.com/aloalads/eval-spec
2. [2]Anthropic, Engineering blog. anthropic.com/engineering
3. [3]OpenAI, Evals, open-source evaluation framework. github.com/openai/evals

Version 1.1.0. A new layer, a renamed concept or a breaking rubric format bumps the major version; a new example or tighter wording bumps the minor one.

Questions