mantis-structural-index

Structural Code Index Builder

This is an optional first-class stage in the Pass Lifecycle Contract. It runs immediately after the snapshot is pinned (Block D), before the first code-reading analysis stage (summarize/architecture). It only needs CODE_ROOT + SNAPSHOT_ID and must not depend on architecture/KB.

System Goal

Structural Code Index Builder. Builds a content-addressed semantic-unit index from source code using capability-based per-partition backend selection, degrading gracefully to grep. Provides find_callers(symbol), get_function_boundary(file, line), and call-site awareness to improve LLM reasoning quality during discovery — supplementing (never replacing) grep-based call-site discovery with structural data. The index is composed from independent semantic compilation units, persisted as a manifest + SQLite catalog, and exposed through a bounded query helper.

Command Definition

• Command: /mantis-structural-index
• Description: Build a content-addressed semantic-unit index from source code under CODE_ROOT.
• Arguments (optional; supplied by the orchestrator, consumed by Block A): --snapshot_root/--snapshot_id/--state_root. All absent → MODE-OFF/legacy mode (reads source from the current directory, writes index to ./workspace/kb/structural_index/).

Input/Output Contract

• Reads:
  • workspace/.mantis_state.json (to read active_snapshot for provenance checking and snapshot-aware rebuild logic).
  • workspace/kb/structural_index/manifest.json (to check snapshot_id for reuse-on-match idempotency — primary check).
  • workspace/kb/structural_index.jsonl (backward-compat provenance check if manifest absent).
  • workspace/kb/structural_index/units/ (content-addressed cache for incremental unit reuse).
  • workspace/kb/structural_index/native/ and sidecar provenance.json files (prebuilt index attachments and metadata manifests).
  • CODE_ROOT source files (via generated helper script — the script parses all source files under CODE_ROOT).
• Writes:
  • workspace/kb/structural_index/manifest.json (STATE-RELATIVE — atomic commit point, written LAST).
  • workspace/kb/structural_index/catalog.sqlite (STATE-RELATIVE — query-optimized serving store).
  • workspace/kb/structural_index/units/ (STATE-RELATIVE — content-addressed immutable unit outputs).
  • workspace/kb/structural_index/shards/ (STATE-RELATIVE — partitioned serving data for large corpora).
  • workspace/kb/structural_index/native/ (STATE-RELATIVE — prebuilt index attachments: SCIP, Kythe, LSIF).
  • workspace/kb/structural_index/tmp/ (STATE-RELATIVE — temporary objects during build).
  • workspace/kb/structural_index.jsonl (STATE-RELATIVE — compatibility pointer; full export below threshold).
  • workspace/helpers/build_structural_index.py (STATE-RELATIVE — the builder helper script).
  • workspace/helpers/query_structural_index.py (STATE-RELATIVE — the query helper script).
• Preconditions:
  • Source files must exist under CODE_ROOT. If CODE_ROOT is not resolved (MODE-OFF and no readable active_snapshot), build against the current directory with snapshot_id set to "unknown". Do NOT skip — this is the standalone-efficiency case.
• Inert until wired: This skill returns an empty index until a caller invokes it (the harness, mantis-plan, or mantis-researcher). It never fails — it simply returns an empty index if tools are unavailable or source cannot be parsed.
• Idempotency Guarantee:
  • Read-only on CODE_ROOT. Writes only to STATE-RELATIVE paths. Re-running with the same CODE_ROOT and SNAPSHOT_ID reuses the existing index (manifest snapshot_id match) rather than rebuilding — except in MODE-OFF, where it always rebuilds. Individual semantic units are reused across snapshots when their content-addressed cache keys match (incremental rebuild).

Instructions

Step 0: Locator Resolution (run first)

LOCATOR RESOLUTION (before reading ANY target code or artifact):
0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
   you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
   state for provenance/annotation; NEVER stop merely because a code root is unset.
1. Determine CODE_ROOT, in this priority order:
   a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
      It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
      (used when a caller hands you a prepared tree, e.g. a patched shadow).
   b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
   c. Else read state_root/workspace/.mantis_state.json (state_root from
      --state_root if passed, else ./workspace/... relative to the current dir)
      -> active_snapshot.root / .snapshot_id / .snapshot_pinned.
   d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
      treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
   verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
   or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
   deliberately mutated and is sentinel-EXEMPT.)
3. PATH FIELDS:
   - SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
     that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
     containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
     NOT of the form <existing-path>:<integer> is a non-source LOCATOR
     (symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
     line-range and line-existence logic.
   - STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
     kb_references, repro_file_path, reattack_file_path, helper scripts, report
     files, and all state/findings JSON.
4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
   compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
   (mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
   cd into CODE_ROOT.
5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
   run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
   (the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
   lacks .git/.hg/.repo.
6. Every shell command uses ABSOLUTE paths and sets its own working directory on
   that call. Do NOT assume the working directory persists between calls.

This is a CODE-READING stage — it reads target source files under CODE_ROOT via the helper script. Block A step 0's findings-only skip does NOT apply.

Step 1: Idempotency / Freshness Check (MANDATORY FIRST)

MANDATORY FIRST STEP: Before writing any build scripts, probing backends, or extracting symbols, check for MODE-OFF or an existing manifest:

1. MODE-OFF check (FIRST): In MODE-OFF (SNAPSHOT_ID is "unknown" or absent) → always rebuild. Do NOT reuse a previous "unknown" index, because the live tree is mutable and "unknown" is a constant (not a freshness signal). Skip directly to Step 2 (individual units whose content has not changed may still hit the content-addressed cache).
2. If workspace/kb/structural_index/manifest.json exists, read its snapshot_id field.
3. If snapshot_id matches the current SNAPSHOT_ID (and SNAPSHOT_ID is NOT "unknown") → reuse the index immediately and STOP. Do NOT invoke Step 2, probe backends, or write build scripts. This bounds cost across retries/crash-resume.
4. If manifest.json is absent but workspace/kb/structural_index.jsonl exists (backward compat), read its provenance header (_provenance, snapshot_id keys). If snapshot_id matches (and is NOT "unknown") → reuse and STOP. Otherwise proceed to rebuild.
5. If SNAPSHOT_ID differs → proceed to rebuild (Steps 2–5). For incremental reuse: before rebuilding a semantic unit, check units/ for a content-addressed cache hit (see Content-Addressed Cache Key below). A cache hit reuses the unit output without re-extraction.

Step 2: Select Backend per Partition

Partition the codebase into semantic units (see Per-Language Semantic Units below). For each unit, select the most precise backend available in this environment using a capability-based, per-partition decision — not a single global ladder. The following are examples, not an exhaustive enum. The helper probes each tier per partition and selects the highest available:

1. Snapshot-matched SCIP / LSIF / Kythe / clangd-static / remote index (precision: semantic) — if a pre-built index matching the current snapshot_id or root_fingerprint is available. Most precise: full type-aware cross-reference, call hierarchy, and hover/signature data.
2. Compiler / typechecker-backed extractor (precision: typecheck) — if compile_commands.json, build context, or typechecker is available. Type-accurate symbol resolution and call edges.
3. Language-aware AST extraction (precision: ast) — if tree-sitter, ast-grep, or a language-specific parser is available. Full AST parsing: function boundaries, call expressions, signatures.
4. Symbol-only extraction (precision: symbol-only) — if ctags or equivalent is on PATH. Symbol table only (function definitions, locations — no call graph). Call-site extraction uses a lightweight regex pass within known function boundaries.
5. Heuristic fallback (precision: heuristic) — Python stdlib regex pass over source files. Identifies function definitions and call patterns using language-agnostic heuristics. Less precise but zero-dependency.
6. Coverage-only manifest + lexical fallback (precision: coverage-only) — grep; no structural index is written. Manifest records status: "empty". Consumers fall back to grep-based discovery (today's behavior byte-for-byte).

Native Index Probing & Resolution Rules:

• Probe instruction: Before evaluating per-partition backends, probe workspace/kb/structural_index/native/ and subdirectories native/{scip,lsif,kythe}/ for prebuilt index files.
• Snapshot-declaration convention: Because native formats (SCIP, LSIF, Kythe) do not embed snapshot identity directly in their binary payload, prebuilt indexes MUST declare their target snapshot using a sidecar provenance.json manifest located at workspace/kb/structural_index/native/provenance.json or native/<kind>/provenance.json. The manifest contains an array of attachments: [{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}]. A native index is matched if its declared snapshot_id equals SNAPSHOT_ID (when SNAPSHOT_ID != "unknown") or its root_fingerprint matches the workspace's calculated root fingerprint. If the files array is absent or empty, treat the native index as covering no individual files directly (record in manifest.native_indexes but do not update coverage rows; fall through to lower tiers for all files).
• Record-and-Defer Ingestion Rule: Parsing raw binary native indexes (e.g. SCIP protobuf) in pure Python without dependencies is costly and complex. Option A builder scripts MUST detect matching prebuilt native indexes, record their entries in manifest.native_indexes, and set the coverage table backend (e.g., "scip" or "scip-clangd") for all files listed in the provenance manifest. If catalog.sqlite is NOT populated with symbols from the native index (raw binary deferred to harness/MCP readers), set coverage.status = "deferred" and precision = "deferred" (with indexed_files = 0). This prevents the query helper from claiming an un-ingested partition is "authoritative empty" at semantic precision, ensuring consumers run the mandatory grep fallback. When catalog.sqlite IS populated (e.g., via Option B pre-ingestion or scip-to-sqlite), set precision = "semantic" and status = "indexed".

LSP is NOT equivalent to SCIP / LSIF. LSP is an interactive protocol whose workspace state may be partial or mutable. Use it only when the server can demonstrate snapshot identity AND complete workspace coverage. A running language server does not automatically qualify as a semantic backend.

SCIP merging. SCIP explicitly permits merging complementary information from indexers with different precision levels. Its format records indexer version and per-document language metadata. The builder MAY merge results from multiple indexers (e.g., a SCIP index for Go + tree-sitter for Python) within a single catalog.

The determinism lives in a runtime-generated versioned helper (build_structural_index.py, # MANTIS_HELPER_VERSION = 5, grep-and-regenerate on reuse) that probes and selects backends per partition. No shipped binaries; air-gapped-safe.

Deterministic partial coverage. On very large source trees, a full rebuild can dominate stage-0 wall-clock. Replace discovery-order truncation with a deterministic priority queue:

1. Explicit target files and symbols (from plan.json, if available).
2. Changed units and known reverse dependencies (from Block E diff + reverse-dependency edges in the existing catalog).
3. Containing packages/modules and direct imports of tier 1+2 units.
4. Remaining units in normalized path order.

Apply deterministic max_units (default: 10000) or max_source_bytes (default: 500MB) bounds to that queue. Persist deferred units in the manifest so another invocation can resume. Statuses: complete, partial, empty, failed.

A consumer should only interpret "no callers" strongly when the relevant partition is complete, snapshot-matched, covered, and semantically precise. Otherwise the answer is "no indexed callers", followed by the existing lexical fallback. The index is HINT-only, so partial coverage is safe (grep remains authoritative).

Step 3: Write and Run Helper Scripts

Two runtime-generated helpers are used. Both follow the grep-and-regenerate pattern: before reuse, grep the first line for the version marker; if absent or a different integer, REGENERATE.

Builder: build_structural_index.py

1. Write the builder to workspace/helpers/build_structural_index.py. The FIRST LINE MUST be exactly # MANTIS_HELPER_VERSION = 5. Before reusing an existing helper, grep its first lines for MANTIS_HELPER_VERSION = 5; if that marker is absent or a different integer, REGENERATE the helper.
2. The builder partitions the codebase into semantic units (see Per-Language

...