quality-playbook

quality-playbook

Quality Playbook Generator

When this skill starts, display this banner before doing anything else:

Quality Playbook v1.2.0 — by Andrew Stellman
https://github.com/andrewstellman/

Generate a complete quality system tailored to a specific codebase. Unlike test stub generators that work mechanically from source code, this skill explores the project first — understanding its domain, architecture, specifications, and failure history — then produces a quality playbook grounded in what it finds.

Why This Exists

Most software projects have tests, but few have a quality system. Tests check whether code works. A quality system answers harder questions: what does "working correctly" mean for this specific project? What are the ways it could fail that wouldn't be caught by tests? What should every developer (human or AI) know before touching this code?

Without a quality playbook, every new contributor (and every new AI session) starts from scratch — guessing at what matters, writing tests that look good but don't catch real bugs, and rediscovering failure modes that were already found and fixed months ago. A quality playbook makes the bar explicit, persistent, and inherited.

What This Skill Produces

Six files that together form a repeatable quality system:

File Purpose Why It Matters Executes Code?

quality/QUALITY.md Quality constitution — coverage targets, fitness-to-purpose scenarios, theater prevention Every AI session reads this first. It tells them what "good enough" means so they don't guess. No

quality/test_functional.* Automated functional tests derived from specifications The safety net. Tests tied to what the spec says should happen, not just what the code does. Use the project's language: test_functional.py (Python), FunctionalSpec.scala (Scala), functional.test.ts (TypeScript), FunctionalTest.java (Java), etc. Yes

quality/RUN_CODE_REVIEW.md Code review protocol with guardrails that prevent hallucinated findings AI code reviews without guardrails produce confident but wrong findings. The guardrails (line numbers, grep before claiming, read bodies) often improve accuracy. No

quality/RUN_INTEGRATION_TESTS.md Integration test protocol — end-to-end pipeline across all variants Unit tests pass, but does the system actually work end-to-end with real external services? Yes

quality/RUN_SPEC_AUDIT.md Council of Three multi-model spec audit protocol No single AI model catches everything. Three independent models with different blind spots catch defects that any one alone would miss. No

AGENTS.md Bootstrap context for any AI session working on this project The "read this first" file. Without it, AI sessions waste their first hour figuring out what's going on. No

Plus output directories: quality/code_reviews/, quality/spec_audits/, quality/results/.

The critical deliverable is the functional test file (named for the project's language and test framework conventions). The Markdown protocols are documentation for humans and AI agents. The functional tests are the automated safety net.

How to Use

Point this skill at any codebase:

Generate a quality playbook for this project.

Update the functional tests — the quality playbook already exists.

Run the spec audit protocol.

If a quality playbook already exists (quality/QUALITY.md, functional tests, etc.), read the existing files first, then evaluate them against the self-check benchmarks in the verification phase. Don't assume existing files are complete — treat them as a starting point.

Phase 1: Explore the Codebase (Do Not Write Yet)

Spend the first phase understanding the project. The quality playbook must be grounded in this specific codebase — not generic advice.

Why explore first? The most common failure in AI-generated quality playbooks is producing generic content — coverage targets that could apply to any project, scenarios that describe theoretical failures, tests that exercise language builtins instead of project code. Exploration prevents this by forcing every output to reference something real: a specific function, a specific schema, a specific defensive code pattern. If you can't point to where something lives in the code, you're guessing — and guesses produce quality playbooks nobody trusts.

Scaling for large codebases: For projects with more than ~50 source files, don't try to read everything. Focus exploration on the 3–5 core modules (the ones that handle the primary data flow, the most complex logic, and the most failure-prone operations). Read representative tests from each subsystem rather than every test file. The goal is depth on what matters, not breadth across everything.

Step 0: Ask About Development History

Before exploring code, ask the user one question:

"Do you have exported AI chat history from developing this project — Claude exports, Gemini takeouts, ChatGPT exports, Claude Code transcripts, or similar? If so, point me to the folder. The design discussions, incident reports, and quality decisions in those chats will make the generated quality playbook significantly better."

If the user provides a chat history folder:

• Scan for an index file first. Look for files named INDEX*, CONTEXT.md, README.md, or similar navigation aids. If one exists, read it — it will tell you what's there and how to find things.

• Search for quality-relevant conversations. Look for messages mentioning: quality, testing, coverage, bugs, failures, incidents, crashes, validation, retry, recovery, spec, fitness, audit, review. Also search for the project name.

• Extract design decisions and incident history. The most valuable content is: (a) incident reports — what went wrong, how many records affected, how it was detected, (b) design discussions — why a particular approach was chosen, what alternatives were rejected, (c) quality framework discussions — coverage targets, testing philosophy, model review experiences, (d) cross-model feedback — where different AI models disagreed about the code.

• Don't try to read everything. Chat histories can be enormous. Use the index to find the most relevant conversations, then search within those for quality-related content. 10 minutes of targeted searching beats 2 hours of exhaustive reading.

This context is gold. A chat history where the developer discussed "why we chose this concurrency model" or "the time we lost 1,693 records in production" transforms generic scenarios into authoritative ones.

If the user doesn't have chat history, proceed normally — the skill works without it, just with less context.

Step 1: Identify Domain, Stack, and Specifications

Read the README, existing documentation, and build config (pyproject.toml / package.json / Cargo.toml). Answer:

• What does this project do? (One sentence.)

• What language and key dependencies?

• What external systems does it talk to?

• What is the primary output?

Find the specifications. Specs are the source of truth for functional tests. Search in order: AGENTS.md/CLAUDE.md in root, specs/, docs/, spec/, design/, architecture/, adr/, then .md files in root. Record the paths.

If no formal spec documents exist, the skill still works — but you need to assemble requirements from other sources. In order of preference:

• Ask the user — they often know the requirements even if they're not written down.

• README and inline documentation — many projects embed requirements in their README, API docs, or code comments.

• Existing test suite — tests are implicit specifications. If a test asserts process(x) == y, that's a requirement.

• Type signatures and validation rules — schemas, type annotations, and validators define what the system accepts and rejects.

• Infer from code behavior — as a last resort, read the code and infer what it's supposed to do. Mark these as inferred requirements in QUALITY.md and flag them for user confirmation.

When working from non-formal requirements, label each scenario and test with a requirement tag that includes a confidence tier and source:

• [Req: formal — README §3] — written by humans in a spec document. Authoritative.

• [Req: user-confirmed — "must handle empty input"] — stated by the user but not in a formal doc. Treat as authoritative.

• [Req: inferred — from validate_input() behavior] — deduced from code. Flag for user review.

Use this exact tag format in QUALITY.md scenarios, functional test documentation, and spec audit findings. It makes clear which requirements are authoritative and which need validation.

Step 2: Map the Architecture

List source directories and their purposes. Read the main entry point, trace execution flow. Identify:

• The 3–5 major subsystems

• The data flow (Input → Processing → Output)

• The most complex module

• The most fragile module

Step 3: Read Existing Tests

Read the existing test files — all of them for small/medium projects, or a representative sample from each subsystem for large ones. Identify: test count, coverage patterns, gaps, and any coverage theater (tests that look good but don't catch real bugs).

Critical: Record the import pattern. How do existing tests import project modules? Every language has its own conventions (Python sys.path manipulation, Java/Scala package imports, TypeScript relative paths or aliases, Go package/module paths, Rust use crate:: or use myproject::). You must use the exact same pattern in your functional tests — getting this wrong means every test fails with import/resolution errors. See references/functional_tests.md § "Import Pattern" for the full six-language matrix.

Identify integration test runners. Look for scripts or test files that exercise the system end-to-end against real external services (APIs, databases, etc.). Note their patterns — you'll need them for RUN_INTEGRATION_TESTS.md.

Step 4: Read the Specifications

Walk each spec document section by section. For every section, ask: "What testable requirement does this state?" Record spec requirements without corresponding tests — these are the gaps the functional tests must close.

If using inferred requirements (from tests, types, or code behavior), tag each with its confidence tier using the [Req: tier — source] format defined in Step 1. Inferred requirements feed into QUALITY.md scenarios and should be flagged for user review in Phase 4.

Step 4b: Read Function Signatures and Real Data

Before writing any test, you must know exactly how each function is called. For every module you identified in Step 2:

• Read the actual function signatures — parameter names, types, defaults. Don't guess from usage context — read the function definition and any documentation (Python docstrings, Java/Scala Javadoc/ScalaDoc, TypeScript type annotations, Go godoc comments, Rust doc comments and type signatures).

• Read real data files — If the project has items files, fixture files, config files, or sample data (in pipelines/, fixtures/, test_data/, examples/), read them. Your test fixtures must match the real data shape exactly.

• Read existing test fixtures — How do existing tests create test data? Copy their patterns. If they build config dicts with specific keys, use those exact keys.

• Check library versions — Check the project's dependency manifest (requirements.txt, build.sbt, package.json, pom.xml/build.gradle, go.mod, Cargo.toml) to see what's actually available. Don't write tests that depend on library features that aren't installed. If a dependency might be missing, use the test framework's skip mechanism — see references/functional_tests.md § "Library version awareness" for framework-specific examples.

Record a function call map: for each function you plan to test, write down its name, module, parameters, and what it returns. This map prevents the most common test failure: calling functions with wrong arguments.

Step 5: Find the Skeletons

This is the most important step. Search for defensive code patterns — each one is evidence of a past failure or known risk.

Why this matters: Developers don't write try/except blocks, null checks, or retry logic for fun. Every piece of defensive code exists because someone got burned. A try/except around a JSON parse means malformed JSON happened in production. A null check on a field means that field was missing when it shouldn't have been. These patterns are the codebase whispering its history of failures. Each one becomes a fitness-to-purpose scenario and a boundary test.

Read references/defensive_patterns.md for the systematic search approach, grep patterns, and how to convert findings into fitness-to-purpose scenarios and boundary tests.

Minimum bar: at least 2–3 defensive patterns per core source file. If you find fewer, you're skimming — read function bodies, not just signatures.

Step 5a: Trace State Machines

If the project has any kind of state management — status fields, lifecycle phases, workflow stages, mode flags — trace the state machine completely. This catches a category of bugs that defensive pattern analysis alone misses: states that exist but aren't handled.

How to find state machines: Search for status/state fields in models, enums, or constants (e.g., status, state, phase, mode). Search for guards that check status before allowing actions (e.g., if status == "running", match self.state). Search for state transitions (assignments to status fields).

For each state machine you find:

• Enumerate all possible states. Read the enum, the constants, or grep for every value the field is assigned. List them all.

• For each consumer of state (UI handlers, API endpoints, control flow guards), check: does it handle every possible state? A switch/match without a meaningful default, or an if/elif chain that doesn't cover all states, is a gap.

• For each state transition, check: can you reach every state? Are there states you can enter but never leave? Are there states that block operations that should be available?

• Record gaps as findings. A status guard that allows action X for "running" but not for "stuck" is a real bug if the user needs to perform action X on stuck processes. A process that enters a terminal state but never triggers cleanup is a real bug.

Why this matters: State machine gaps produce bugs that are invisible during normal operation but surface under stress or edge conditions — exactly when you need the system to work. A batch processor that can't be killed when it's in "stuck" status, or a watcher that never self-terminates after all work completes, or a UI that refuses to resume a "pending" run, are all symptoms of incomplete state handling. These bugs don't show up in defensive pattern analysis because the code isn't defending against t

...