mantis-dedupe

Deduplicator (/mantis-dedupe)

System Goal

Duplicate Finding Merger. Evaluates lists of raw findings to cluster and consolidate identical or highly overlapping issues into singular, descriptive records.

Command Definition

• Command: /mantis-dedupe
• Description: Consolidates raw security findings to eliminate redundant reports.
• Arguments (optional; supplied by the orchestrator, consumed by Block A): --snapshot_root/--snapshot_id/--state_root. All absent -> MODE-OFF/legacy mode (behaves as today; snapshot gating disabled).

Input/Output Contract

• Reads:
  • workspace/findings/ (raw finding JSON files, ignoring .trash/).
  • workspace/archive/findings_pass_*/*.json and workspace/archive/loop*_findings/*.json (to skip findings already evaluated and triaged in previous passes).
  • workspace/.mantis_state.json (to track current loop pass).
• Writes:
  • Moves duplicate findings to workspace/findings/.trash/ after setting "status": "DUPLICATE" and "duplicate_of".
  • Sets "possible_duplicate_of" (soft, non-terminal) on NOT_MATCHED matches and stamps "discovery_commit" on current findings that lack it. Reads active_snapshot/snapshot_pinned from .mantis_state.json.
  • Appends transaction logs to workspace/.tx_log.jsonl.
  • Generates/executes merging script workspace/helpers/merge_findings.py.
  • Updates primary finding workspace/findings/<primary_id>.json (merges fields and history).
• Preconditions:
  • workspace/findings/ must exist and contain finding files.
• Idempotency Guarantee:
  • Cross-references against archived findings in workspace/archive/ to filter out any findings already processed in previous passes of this run. Snapshot-gated: a resolved archived finding on a differing snapshot is flagged as POSSIBLE REGRESSION (never silently filtered). Logs transactions to workspace/.tx_log.jsonl to support tracking and potential rollbacks. Deterministic merging rules implemented in merge_findings.py.

Instructions

Step 0: Locator Resolution (run first)

LOCATOR RESOLUTION (before reading ANY target code or artifact):
0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
   you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
   state for provenance/annotation; NEVER stop merely because a code root is unset.
1. Determine CODE_ROOT, in this priority order:
   a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
      It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
      (used when a caller hands you a prepared tree, e.g. a patched shadow).
   b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
   c. Else read state_root/workspace/.mantis_state.json (state_root from
      --state_root if passed, else ./workspace/... relative to the current dir)
      -> active_snapshot.root / .snapshot_id / .snapshot_pinned.
   d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
      treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
   verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
   or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
   deliberately mutated and is sentinel-EXEMPT.)
3. PATH FIELDS:
   - SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
     that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
     containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
     NOT of the form <existing-path>:<integer> is a non-source LOCATOR
     (symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
     line-range and line-existence logic.
   - STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
     kb_references, repro_file_path, reattack_file_path, helper scripts, report
     files, and all state/findings JSON.
4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
   compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
   (mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
   cd into CODE_ROOT.
5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
   run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
   (the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
   lacks .git/.hg/.repo.
6. Every shell command uses ABSOLUTE paths and sets its own working directory on
   that call. Do NOT assume the working directory persists between calls.

[!NOTE] CURRENT-PASS CHECK (defensive; the binding guarantee is on the harness per mantis-pipeline-adapter Scenario 2): if active_snapshot is present AND active_snapshot.pass != state.pass_number, treat the snapshot as STALE for this pass — STOP "stale active_snapshot: pass mismatch" or degrade as HALT (snapshot_pinned effectively false: no authoritative verdicts, Block B NOT_MATCHED, reproduce not_attempted). This catches a custom harness that preserved active_snapshot across the Stage 15 pass increment without re-pinning. The reference meta-agent re-pins every pass, so this check never fires there. Block B itself cannot detect this (it is snapshot_id-only, not pass-aware).

Notes: workspace/findings/, workspace/archive/, workspace/.tx_log.jsonl, workspace/helpers/ and .mantis_state.json are STATE-RELATIVE (under --state_root). Any code snippet you inspect for a finding is SNAPSHOT-RELATIVE (under CODE_ROOT). Never write under CODE_ROOT.

Review a list of security findings and merge duplicate findings that refer to the exact same security flaw or adjacent code paths.

Execute your task as follows:

1. Load Raw Findings & Archived Findings Queue:

   • List the contents of the directory and read the files in workspace/findings/. If the directory is empty or does not exist, notify the user and exit.
   • Important: Ignore hidden files and directories (such as the .trash/ subdirectory) when listing or processing findings.
   • Locate and load all archived finding JSON files from previous loop passes, if they exist, under workspace/archive/findings_pass_*/*.json and workspace/archive/loop*_findings/*.json. These files represent vulnerabilities that have already been fully evaluated, triaged, and potentially patched in previous passes.
   • Important: Do NOT read or deduplicate against workspace/historical_learnings.jsonl (VCS history), as we want to catch regressions if old bugs were reintroduced.

2. Filter Loop Duplicates (snapshot-gated). First, stamp discovery_commit on any CURRENT finding that lacks it, using active_snapshot.snapshot_id from .mantis_state.json (skip when unpinned). Then, for each current finding that matches an archived finding (by code_paths+title similarity), run:

   Signature-based candidate matching (Phase 3) — TIGHTENS, never replaces: signature may only PROMOTE a pair to "candidate for the pairwise snapshot check"; it may NEVER by itself cause a hard DUPLICATE/trash. A pair is a candidate for the Pairwise Snapshot Match Check below ONLY if it satisfies BOTH:

   • it matches under today's code_paths + title similarity, comparing code_paths entries line-inclusively (WITH their trailing :line); AND
   • (when both findings have a signature) their signature fields are equal. A signature match WITHOUT the code_paths + title agreement is NOT a duplicate — at most a soft possible_duplicate_of (keep the finding ACTIVE), never a trash. Rationale: signature strips the line number and all-but-first code_paths entry, so two DISTINCT bugs in the same file (e.g. parser.c:100 vs parser.c:900) with the same title+CWE share one signature; trashing on signature alone would silently delete a real finding. If EITHER lacks signature, use today's code_paths + title similarity matching unchanged.

   PAIRWISE SNAPSHOT MATCH CHECK (decides MATCHED vs NOT_MATCHED) — compares the CURRENT finding's discovery_commit against the ARCHIVED finding's discovery_commit for this pair (NOT against SNAPSHOT_ID):

   1. If snapshot_pinned is false AND there is NO active_snapshot in state (MODE-OFF) -> NOT_MATCHED. Stop. (In HALT — active_snapshot present but snapshot_pinned=false — do NOT short-circuit here; fall through to the pairwise comparison below, which will be NOT_MATCHED because the current finding's discovery_commit is a live: id that will not equal the archived one.)
   2. Read the CURRENT finding's discovery_commit and the ARCHIVED finding's discovery_commit:
      • If EITHER is missing, empty, or the literal "MIXED" -> NOT_MATCHED.
      • If they are NOT byte-for-byte equal to each other -> NOT_MATCHED.
      • If they ARE byte-for-byte equal to each other (both present, non-MIXED) -> MATCHED. There is no other route to MATCHED; never fuzzy-compare. The global "default the field and proceed" backward-compat rule does NOT apply to discovery_commit: absent = NOT_MATCHED. (There is NO separate "dirty" gate: a dirty tree's SNAPSHOT_ID already embeds the working-tree content hash, so within-pass findings MATCH and cross-pass bare-commit findings do not.) Note: this is a PAIRWISE check (current vs archived), NOT a check against the global SNAPSHOT_ID — dedupe stamps the current finding's discovery_commit to SNAPSHOT_ID in Step 2 above, so a check against SNAPSHOT_ID would always be MATCHED and would trash reintroduced/regression bugs as DUPLICATE.

   Then, using the idempotency rule (Input/Output Contract → Idempotency Guarantee) to avoid double-writes, decide mechanically:

   • MATCHED (both present and equal): soft-delete the current finding as a loop-duplicate exactly as before — set "status": "DUPLICATE" and "duplicate_of": "<archived_uuid>", clear possible_duplicate_of if present, ensure mkdir -p workspace/findings/.trash/, move it there, and log a loop_filter transaction in workspace/.tx_log.jsonl. If the current finding lacks lineage_id but the archived finding has one, inherit the archived finding's lineage_id onto the current finding before moving it (so the lineage chain is preserved across the merge).
   • NOT_MATCHED (differ, or either absent): do NOT set DUPLICATE and do NOT move to trash. Keep the current finding ACTIVE and set "possible_duplicate_of": "<archived_uuid>" (a soft, non-terminal hint). If the current finding lacks lineage_id but the archived finding has one, inherit the archived finding's lineage_id onto the current finding (so the lineage chain is preserved for report folding even when the findings are on different snapshots).

   [!IMPORTANT] STATUS & DUPLICATE INVARIANTS:

   • A finding MUST NOT carry both duplicate_of and possible_duplicate_of pointing to the same target UUID.
   • status = "DUPLICATE" MUST NOT coexist with possible_duplicate_of pointing to the same target UUID.
   • Under NOT_MATCHED (outside the MODE-OFF fallback exception), the finding's status MUST remain active (e.g. VALID, PROVISIONALLY_VALID, NEEDS_RESEARCH), duplicate_of MUST NOT be set, and the finding MUST NOT be moved to .trash/. Setting possible_duplicate_of is a non-terminal hint only.

   • POSSIBLE REGRESSION: if the archived match has a RESOLVED status (patch_status in {VERIFIED_SECURE,MITIGATION_PROPOSED} OR status==FALSE_POSITIVE OR production_viability==NON_VIABLE) AND the pair is NOT MATCHED, keep the current finding ACTIVE, add a history note "POSSIBLE REGRESSION vs <archived_uuid>", and do NOT filter it. (A reverted fix re-discovered on new code must never be trashed.)
   • EXACT-UUID retry exception (unchanged): if the current finding has the EXACT SAME UUID as the archived one, it was intentionally copied back for a retry — do NOT filter it (keep as-is).
   • Permanently-unpinned exception (MODE-OFF only — no active_snapshot): if there is NO active_snapshot in state (MODE-OFF = today's default; a target with no snapshot boundary, e.g. a live endpoint), snapshot_pinned is false at the PASS level (active_snapshot.snapshot_pinned — it is NOT a per-finding field) and Block B is uninformative — fall back to today's dedup by signature if present, else stable_key = normalized_title + first code_paths entry including its trailing :line (line-inclusive, same as Step 2). Rationale: stripping :line would collapse two DISTINCT bugs in the same file (e.g. parser.c:100 vs parser.c:900) with the same title+CWE into one — silently deleting a real finding. Note: signature itself strips :line by design (it is a coarse identity for cross-pass lineage, not a dedup key); this fallback therefore prefers signature only when stable_key's line-inclusive match ALSO agrees, never on signature alone. This preserves dedup for targets that can never MATCH. When active_snapshot IS present but unpinned (HALT mode), this exception does NOT fire: keep the snapshot-gated behavior above (NOT_MATCHED → keep ACTIVE + possible_duplicate_of, never DUPLICATE). POSSIBLE REGRESSION takes precedence over this fallback: a resolved-archived finding paired with a NOT_MATCHED current finding is ALWAYS kept active (never trashed), regardless of mode.

3. 3-Tier Deduplication Ladder (Deterministic & Semantic Matching): When resolving finding lineages and evaluating deduplication candidates, the deduplicator uses a hierarchical 3-tier ladder designed for sub-millisecond fast-path execution with semantic RAG fallback:

   • Tier 1 (Fast-Path Exact Heuristic Anchors — < 1ms, 0 tokens):

     1. Exact content identity signature match: hashlib.sha256(canonical_fp | canonical_cwe | target_symbol) (invariant to line shifts, backticks, and title paraphrasing).
     2. Exact canonical_filepath + normalized CWE + target_symbol match.
     3. Strict line proximity window ($\le 3$ lines) on exact canonical filepath when target symbol is empty.
     • Result: If matched, immediately inherit ancestor lineage_id without LLM or embedding overhead.

   • Tier 2 (RCA Normalization): If Tier 1 heuristi

...