eval-driven-dev

eval-driven-dev

Eval-Driven Development with pixie

This skill is about doing the work, not describing it. When a user asks you to set up evals for their app, you should be reading their code, editing their files, running commands, and producing a working test pipeline — not writing a plan for them to follow later.

All pixie-generated files live in a single pixie_qa directory at the project root:

pixie_qa/
  MEMORY.md              # your understanding and eval plan
  observations.db        # SQLite trace DB (auto-created by enable_storage)
  datasets/              # golden datasets (JSON files)
  tests/                 # eval test files (test_*.py)
  scripts/               # helper scripts (build_dataset.py, etc.)

Setup vs. Iteration: when to stop

This is critical. What you do depends on what the user asked for.

"Setup QA" / "set up evals" / "add tests" (setup intent)

The user wants a working eval pipeline. Your job is Stages 0–6: install, understand, instrument, write tests, build dataset, run tests. Stop after the first test run, regardless of whether tests pass or fail. Report:

• What you set up (instrumentation, test file, dataset)

• The test results (pass/fail, scores)

• If tests failed: a brief summary of what failed and likely causes — but do NOT fix anything

Then ask: "QA setup is complete. Tests show N/M passing. Want me to investigate the failures and start iterating?"

Only proceed to Stage 7 (investigation and fixes) if the user confirms.

Exception: If the test run itself errors out (import failures, missing API keys, configuration bugs) — those are setup problems, not eval failures. Fix them and re-run until you get a clean test execution where pass/fail reflects actual app quality, not broken plumbing.

"Fix" / "improve" / "debug" / "why is X failing" (iteration intent)

The user wants you to investigate and fix. Proceed through all stages including Stage 7 — investigate failures, root-cause them, apply fixes, rebuild dataset, re-run tests, iterate.

Ambiguous requests

If the intent is unclear, default to setup only and ask before iterating. It's better to stop early and ask than to make unwanted changes to the user's application code.

The eval boundary: what to evaluate

Eval-driven development focuses on LLM-dependent behaviour. The purpose is to catch quality regressions in the parts of the system that are non-deterministic and hard to test with traditional unit tests — namely, LLM calls and the decisions they drive.

In scope (evaluate this)

• LLM response quality: factual accuracy, relevance, format compliance, safety

• Agent routing decisions: did the LLM choose the right tool/handoff/action?

• Prompt effectiveness: does the prompt produce the desired behaviour?

• Multi-turn coherence: does the agent maintain context across turns?

Out of scope (do NOT evaluate this with evals)

• Tool implementations (database queries, API calls, keyword matching, business logic) — these are traditional software; test them with unit tests

• Infrastructure (authentication, rate limiting, caching, serialization)

• Deterministic post-processing (formatting, filtering, sorting results)

The boundary is: everything downstream of the LLM call (tools, databases, APIs) produces deterministic outputs that serve as inputs to the LLM-powered system. Eval tests should treat those as given facts and focus on what the LLM does with them.

Example: If an FAQ tool has a keyword-matching bug that returns wrong data, that's a traditional bug — fix it with a regular code change, not by adjusting eval thresholds. The eval tests exist to verify that given correct tool outputs, the LLM agent produces correct user-facing responses.

When building datasets and expected outputs, use the actual tool/system outputs as ground truth. The expected output for an eval case should reflect what a correct LLM response looks like given the tool results the system actually produces.

Stage 0: Ensure pixie-qa is Installed and API Keys Are Set

Before doing anything else, check that the pixie-qa package is available:

python -c "import pixie" 2>/dev/null && echo "installed" || echo "not installed"

If it's not installed, install it:

pip install pixie-qa

This provides the pixie Python module, the pixie CLI, and the pixie test runner — all required for instrumentation and evals. Don't skip this step; everything else in this skill depends on it.

Verify API keys

The application under test almost certainly needs an LLM provider API key (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY). LLM-as-judge evaluators like FactualityEval also need OPENAI_API_KEY. Before running anything, verify the key is set:

[ -n "$OPENAI_API_KEY" ] && echo "OPENAI_API_KEY set" || echo "OPENAI_API_KEY missing"

If not set, ask the user. Do not proceed with running the app or evals without it — you'll get silent failures or import-time errors.

Stage 1: Understand the Application

Before touching any code, spend time actually reading the source. The code will tell you more than asking the user would, and it puts you in a much better position to make good decisions about what and how to evaluate.

What to investigate

How the software runs: What is the entry point? How do you start it? Is it a CLI, a server, a library function? What are the required arguments, config files, or environment variables?

All inputs to the LLM: This is not limited to the user's message. Trace every piece of data that gets incorporated into any LLM prompt:

User input (queries, messages, uploaded files)

• System prompts (hardcoded or templated)

• Retrieved context (RAG chunks, search results, database records)

• Tool definitions and function schemas

• Conversation history / memory

• Configuration or feature flags that change prompt behavior

All intermediate steps and outputs: Walk through the code path from input to final output and document each stage:

Retrieval / search results

• Tool calls and their results

• Agent routing / handoff decisions

• Intermediate LLM calls (e.g., summarization before final answer)

• Post-processing or formatting steps

The final output: What does the user see? What format is it in? What are the quality expectations?

Use cases and expected behaviors: What are the distinct things the app is supposed to handle? For each use case, what does a "good" response look like? What would constitute a failure?

Write MEMORY.md

Write your findings down in pixie_qa/MEMORY.md. This is the primary working document for the eval effort. It should be human-readable and detailed enough that someone unfamiliar with the project can understand the application and the eval strategy.

CRITICAL: MEMORY.md documents your understanding of the existing application code. It must NOT contain references to pixie commands, instrumentation code you plan to add, or scripts/functions that don't exist yet. Those belong in later sections, only after they've been implemented.

The understanding section should include:

# Eval Notes: <Project Name>

## How the application works

### Entry point and execution flow

<Describe how to start/run the app, what happens step by step>

### Inputs to LLM calls

<For each LLM call in the codebase, document:>

- Where it is in the code (file + function name)
- What system prompt it uses (quote it or summarize)
- What user/dynamic content feeds into it
- What tools/functions are available to it

### Intermediate processing

<Describe any steps between input and output:>
- Retrieval, routing, tool execution, etc.
- Include code pointers (file:line) for each step

### Final output

<What the user sees, what format, what the quality bar should be>

### Use cases

<List each distinct scenario the app handles, with examples of good/bad outputs>

## Evaluation plan

### What to evaluate and why

<Quality dimensions: factual accuracy, relevance, format compliance, safety, etc.>

### Evaluation granularity

<Which function/span boundary captures one "test case"? Why that boundary?>

### Evaluators and criteria

<For each eval test, specify: evaluator, dataset, threshold, reasoning>

### Data needed for evaluation

<What data points need to be captured, with code pointers to where they live>

If something is genuinely unclear from the code, ask the user — but most questions answer themselves once you've read the code carefully.

Stage 2: Decide What to Evaluate

Now that you understand the app, you can make thoughtful choices about what to measure:

• What quality dimension matters most? Factual accuracy for QA apps, output format for structured extraction, relevance for RAG, safety for user-facing text.

• Which span to evaluate: the whole pipeline (root) or just the LLM call (last_llm_call)? If you're debugging retrieval, you might evaluate at a different point than if you're checking final answer quality.

• Which evaluators fit: see references/pixie-api.md → Evaluators. For factual QA: FactualityEval. For structured output: ValidJSONEval / JSONDiffEval. For RAG pipelines: ContextRelevancyEval / FaithfulnessEval.

• Pass criteria: ScoreThreshold(threshold=0.7, pct=0.8) means 80% of cases must score ≥ 0.7. Think about what "good enough" looks like for this app.

• Expected outputs: FactualityEval needs them. Format evaluators usually don't.

Update pixie_qa/MEMORY.md with the plan before writing any code.

Stage 3: Instrument the Application

Add pixie instrumentation to the existing production code. The goal is to capture the inputs and outputs of functions that are already part of the application's normal execution path. Instrumentation must be on the real code path — the same code that runs when the app is used in production — so that traces are captured both during eval runs and real usage.

Add enable_storage() at application startup

Call enable_storage() once at the beginning of the application's startup code — inside main(), or at the top of a server's initialization. Never at module level (top of a file outside any function), because that causes storage setup to trigger on import.

Good places:

• Inside if __name__ == "__main__": blocks

• In a FastAPI lifespan or on_startup handler

• At the top of main() / run() functions

• Inside the runnable function in test files

# ✅ CORRECT — at application startup
async def main():
    enable_storage()
    ...

# ✅ CORRECT — in a runnable for tests
def runnable(eval_input):
    enable_storage()
    my_function(**eval_input)

# ❌ WRONG — at module level, runs on import
from pixie import enable_storage
enable_storage()  # this runs when any file imports this module!

Wrap existing functions with @observe or start_observation

CRITICAL: Instrument the production code path. Never create separate functions or alternate code paths for testing.

The @observe decorator or start_observation context manager goes on the existing function that the app actually calls during normal operation. If the app's entry point is an interactive main() loop, instrument main() or the core function it calls per user turn — not a new helper function that duplicates logic.

# ✅ CORRECT — decorating the existing production function
from pixie import observe

@observe(name="answer_question")
def answer_question(question: str, context: str) -> str:  # existing function
    ...  # existing code, unchanged

# ✅ CORRECT — context manager inside an existing function
from pixie import start_observation

async def main():  # existing function
    ...
    with start_observation(input={"user_input": user_input}, name="handle_turn") as obs:
        result = await Runner.run(current_agent, input_items, context=context)
        # ... existing response handling ...
        obs.set_output(response_text)
    ...

# ❌ WRONG — creating a new function that duplicates logic from main()
@observe(name="run_for_eval")
async def run_for_eval(user_messages: list[str]) -> str:
    # This duplicates what main() does, creating a separate code path
    # that diverges from production. Don't do this.
    ...

Rules:

• Never add new wrapper functions to the application code for eval purposes.

• Never change the function's interface (arguments, return type, behavior).

• Never duplicate production logic into a separate "testable" function.

• The instrumentation is purely additive — if you removed all pixie imports and decorators, the app would work identically.

• After instrumentation, call flush() at the end of runs to make sure all spans are written.

• For interactive apps (CLI loops, chat interfaces), instrument the per-turn processing function — the one that takes user input and produces a response. The eval runnable should call this same function.

Important: All pixie symbols are importable from the top-level pixie package. Never tell users to import from submodules (pixie.instrumentation, pixie.evals, pixie.storage.evaluable, etc.) — always use from pixie import ....

Stage 4: Write the Eval Test File

Write the test file before building the dataset. This might seem backwards, but it forces you to decide what you're actually measuring before you start collecting data — otherwise the data collection has no direction.

Create pixie_qa/tests/test_<feature>.py. The pattern is: a runnable adapter that calls the app's existing production function, plus an async test function that calls assert_dataset_pass:

from pixie import enable_storage, assert_dataset_pass, FactualityEval, ScoreThreshold, last_llm_call

from myapp import answer_question

def runnable(eval_input):
    """Replays one dataset item through the app.

    Calls the same function the production app uses.
    enable_storage() here ensures traces are captured during eval runs.
    """
    enable_storage()
    answer_question(**eval_input)

async def test_factuality():
    await assert_dataset_pass(
        runnable=runnable,
        dataset_name="<dataset-name>",
        evaluators=[FactualityEval()],
        pass_criteria=ScoreThreshold(threshold=0.7, pct=0.8),
        from_trace=last_llm_call,
    )

Note that enable_storage() belongs inside the runnable, not at module level in the test file — it needs to fire on each invocation so the trace is captured for that specific run.

The runnable calls the same function that production uses — it does not create a new code path. The only addition is enable_storage() to capture traces during eval.

The test runner is pixie test (not pytest):

pixie test                           # run all test_*.py in current directory
pixie test pixie_qa/tests/           # specify path
pixie test -k factuality         

...