ai-tool-compliance

ai-tool-compliance

ai-tool-compliance - Internal AI Tool Compliance Automation When to use this skill Starting a new AI project: When scaffolding the compliance foundation (RBAC, Gateway, logs, cost tracking) from scratch Pre-deploy P0 full verification: When automatically evaluating all 13 P0 mandatory requirements as pass/fail and computing a compliance score RBAC design and permission matrix generation: When defining the 5 roles (Super Admin/Admin/Manager/Viewer/Guest) + granular access control per game/menu/feature unit Auditing existing code for compliance: When inspecting an existing codebase against the guide and identifying violations Implementing cost transparency: When building a tracking system for model/token/BQ scan volume/cost per action Designing a behavior log schema: When designing a comprehensive behavior log recording system (Firestore/BigQuery) Role-based verification workflow: When configuring the release approval process based on Section 14 (ServiceStability/Engineer/PM/CEO) Building a criteria verification system: When setting up the Rule Registry + Evidence Collector + Verifier Engine + Risk Scorer + Gatekeeper architecture Installation npx skills add https://github.com/supercent-io/skills-template --skill ai-tool-compliance Quick Reference Action Command Description Project initialization /compliance-init Generate RBAC matrix, Gateway boilerplate, log schema, cost tracking interface Quick scan /compliance-scan, /compliance-quick, /quick Quick inspection of P0 key items (code pattern-based) Full verification /compliance-verify, /compliance-full, /full Full verification of 11 P0 rules + compliance score computation Score check /compliance-score Display current compliance score (security/auth/cost/logging) Deploy gate /compliance-gate Green/Yellow/Red verdict + deploy approve/block decision Improvement guide /compliance-improve, /improve Specific fix suggestions per violation + re-verification loop Slash Mode Router Mode slash commands are mapped as follows. /quick, /compliance-quick -> Quick Scan (/compliance-scan) /full, /compliance-full -> Full Verify (/compliance-verify) /improve -> Improve (/compliance-improve) 3 Execution Modes 1. Quick Scan (quick-scan) Statically analyzes the codebase to quickly identify potential P0 violations. How to run: /compliance-scan, /compliance-quick, /quick or trigger keywords compliance scan, quick scan What it does: Grep/Glob-based code pattern search Detect direct external API calls (whether Gateway is bypassed) Detect direct Firestore client access Detect hardcoded sensitive data Check for missing Guest role Output: List of suspected violations (file path + line number + rule ID) Duration: 1–3 minutes 2. Full Verify (full-verify) Fully verifies all 11 P0 rules and computes a quantitative compliance score. How to run: /compliance-verify, /compliance-full, /full or trigger keywords P0 verification, full verify, deploy verification What it does: Collect Evidence and evaluate pass/fail for each of the 11 P0 rules Compute scores per 4 domains (Security 40pts / Auth 25pts / Cost 20pts / Logging 15pts) Calculate total compliance score (out of 100) Determine deploy gate grade (Green/Yellow/Red) Generate role-based approval checklist Output: Compliance report (compliance-report.md) ## Compliance Report - Date: 2026-03-03 - Project: my-ai-tool - Score: 92/100 (Green) ### Rule Results | Rule ID | Rule Name | Result | Evidence | |---------|-----------|--------|----------| | AUTH-P0-001 | Force Guest for New Signups | PASS | signup.ts:45 role='guest' | | AUTH-P0-002 | Block Guest Menu/API Access | PASS | middleware.ts:12 guestBlock | | ... | ... | ... | ... | ### Score Breakdown - Security: 33/40 - Auth: 25/25 - Cost: 17/20 - Logging: 12/15 - Total: 92/100 ### Gate Decision: GREEN - Deploy Approved Duration: 5–15 minutes (varies by project size) 3. Improve (improve) Provides specific fix guides for violations and runs a re-verification loop. How to run: /compliance-improve, /improve or trigger keywords compliance improvement, fix violations What it does: Code-level fix suggestions for each FAIL item (file path + before/after code) Re-verify the rule after applying the fix Track score changes (Before -> After) Guide for gradually introducing P1 recommended requirements after passing P0 Output: Fix proposal + re-verification results Improve Mode Auto-Fix Logic /compliance-improve runs | 1. Load latest verification-run.json | 2. Extract FAIL items (rule_id + evidence) | 3. For each FAIL: | a. Read violation code from evidence file:line b. Derive fix direction from rule.remediation + rule.check_pattern.must_contain c. Generate before/after code diff d. Apply fix via Write (after user confirmation) e. Re-verify only that rule (re-run Grep pattern) f. Confirm transition to PASS | 4. Full re-verification (/compliance-verify) | 5. Output Before/After score comparison | 6. If no remaining FAILs → present guide for introducing P1 recommended requirements Fix application priority: must_not_contain violations (requires immediate removal) → delete the code or replace with server API call must_contain unmet (pattern needs to be added) → insert code per the remediation guide Warning (partially met) → apply supplement only to unmet files P0 Rule Catalog 11 P0 rules based on the internal AI tool mandatory implementation guide v1.1: Rule ID Category Rule Name Description Score AUTH-P0-001 Auth Force Guest for New Signups Automatically assign role=Guest on signup; elevated roles granted only via invitation Auth 8 AUTH-P0-002 Auth Block Guest Menu/API Access Do not expose tool name, model name, internal infrastructure, cost, or structure to Guest. Only allow access to permitted menus/APIs Auth 7 AUTH-P0-003 Auth Server-side Final Auth Check Server-side auth verification middleware required for all API requests. Client-side checks alone are insufficient Auth 10 SEC-P0-004 Security Prohibit Direct Firestore Access Direct read/write to Firestore from client is forbidden. Only via Cloud Functions is allowed Security 12 SEC-P0-005 Security Enforce External API Gateway Direct calls to external AI APIs (Gemini, OpenAI, etc.) are forbidden. Must route through internal Gateway (Cloud Functions) Security 18 SEC-P0-009 Security Server-side Sensitive Text Processing Sensitive raw content (prompts, full responses) is processed server-side only. Only reference values (IDs) are sent to clients Security 10 COST-P0-006 Cost Model Call Cost Log Must record model, inputTokens, outputTokens, estimatedCost for every AI model call Cost 10 COST-P0-007 Cost BQ Scan Cost Log Must record bytesProcessed, estimatedCost when executing BigQuery queries Cost 5 COST-P0-011 Cost Cache-first Lookup Cache lookup required before high-cost API calls. Actual call only on cache miss Cost 5 LOG-P0-008 Logging Mandatory Failed Request Logging Must log all failed requests (4xx, 5xx, timeout). No omissions allowed Logging 10 LOG-P0-010 Logging Auth Change Audit Log Record all auth-related events: role changes, permission grants/revocations, invitation sends Logging 5 Scoring System Domain Max Score Included Rules Security 40 SEC-P0-004, SEC-P0-005, SEC-P0-009 Auth 25 AUTH-P0-001, AUTH-P0-002, AUTH-P0-003 Cost 20 COST-P0-006, COST-P0-007, COST-P0-011 Logging 15 LOG-P0-008, LOG-P0-010 Total 100 11 P0 rules Per-rule Automatic Verification Logic Verification for each rule is performed based on the check_pattern defined in rules/p0-catalog.yaml. The core mechanism is Grep/Glob static analysis. Verdict Algorithm (per rule): 1. Glob(check_targets) → collect target files 2. grep_patterns matching → identify files using that feature - 0 matches → N/A (feature not used, no penalty) 3. must_not_contain check (excluding exclude_paths) - Match found → immediate FAIL + record evidence 4. must_contain check - All satisfied → PASS - Partially satisfied → WARNING - Not satisfied → FAIL Key Grep Patterns per Rule: Rule ID Feature Detection (grep_patterns) Compliance Check (must_contain) Violation Detection (must_not_contain) AUTH-P0-001 signup|register|createUser role.['"]guest['"] role.['"]admin['"] (on signup) AUTH-P0-002 guard|middleware|authorize guest.*block|guest.*deny -- AUTH-P0-003 router.(get|post|put|delete) auth|verify|authenticate -- SEC-P0-004 -- (all targets) -- firebase/firestore|getDocs|setDoc (client paths) SEC-P0-005 -- (all targets) -- fetch(['"]https?://(?!localhost) (client paths) SEC-P0-009 -- (all targets) -- res.json(.*password|console.log(.*token COST-P0-006 openai|vertexai|gemini|anthropic cost|token|usage|billing -- COST-P0-007 bigquery|BigQuery|createQueryJob totalBytesProcessed|bytesProcessed|cost -- COST-P0-011 openai|vertexai|gemini|anthropic cache|Cache|redis|memo -- LOG-P0-008 catch|errorHandler|onError logger|log.error|winston|pino -- LOG-P0-010 updateRole|changeRole|setRole audit|auditLog|eventLog -- Detailed schema: see rules/p0-catalog.yaml and the "Judgment Algorithm" section in REFERENCE.md Verification Scenarios (QA) 5 key verification scenarios run in Full Verify mode (/compliance-verify). Each scenario groups related P0 rules for end-to-end verification. ID Scenario Related Rules Verification Method Pass Criteria SC-001 New Signup -> Guest Isolation AUTH-P0-001, AUTH-P0-002 Verify role=guest assignment in signup code + confirm 403 return pattern when Guest calls protected API PASS when role is guest and access-denied pattern exists for protected API SC-002 AI Call -> Via Gateway + Cost Logged SEC-P0-005, COST-P0-006, COST-P0-011 (1) Confirm absence of direct external API calls (2) Confirm routing via Gateway function (3) Confirm cost log fields (model, tokens, cost) recorded (4) Confirm cache lookup logic exists PASS when Gateway routing + 4 cost log fields recorded + cache layer exists SC-003 Firestore Access -> Functions-Only SEC-P0-004, AUTH-P0-003 (1) Detect direct Firestore SDK import in client code (2) Confirm server-side auth verification middleware exists PASS when 0 direct client access instances + server middleware exists SC-004 Failed Requests -> No Log Gaps LOG-P0-008, LOG-P0-010 (1) Confirm log call in error handler (2) Confirm no log gaps in catch blocks (3) Confirm audit log exists for auth change events PASS when all error handlers call log + auth change audit log exists SC-005 Sensitive Data -> Not Exposed to Client SEC-P0-009, AUTH-P0-002 (1) Confirm API responses do not include raw prompts/responses, only reference IDs (2) Confirm Guest responses do not include model name/cost/infrastructure info PASS when raw content not in response + Guest exposure control confirmed Verification Flow by Scenario SC-001: grep signup/register -> assert role='guest' -> grep guestBlock/guestDeny -> assert exists SC-002: grep fetch(https://) in client -> assert 0 hits -> grep gateway log -> assert cost fields -> assert cache check SC-003: grep firebase/firestore in client/ -> assert 0 hits -> grep authMiddleware in functions/ -> assert exists SC-004: grep catch blocks -> assert logAction in each -> grep roleChange -> assert auditLog SC-005: grep res.json for raw text -> assert 0 hits -> grep guest response -> assert no model/cost info Role-based Go/No-Go Checkpoints After the deploy gate verdict, the role's Go/No-Go checkpoints must be cleared based on the grade. 4 roles × 5 items = 20 checkpoints total. Service Stability (5 items) # Checkpoint Go Condition No-Go Condition 1 SLA Impact Analysis Confirmed no impact on existing service availability/response-time SLA SLA impact unanalyzed or degradation expected 2 Rollback Procedure Rollback procedure documented + tested Rollback procedure not established 3 Performance Test Load/stress test completed + within threshold Performance test not run 4 Incident Alerts Incident detection alert channels (Slack/PagerDuty, etc.) configured Alert channels not configured 5 Monitoring Dashboard Dashboard for key metrics (error rate, response time, AI cost) exists Monitoring absent Engineer (5 items) # Checkpoint Go Condition No-Go Condition 1 FAIL Rule Root Cause Analysis Root cause identified + documented for all FAIL rules Unidentified items exist 2 Fix Code Verification Fixed code accurately reflects the intent of the rule Fix does not match rule intent 3 Re-verification Pass Rule transitions to PASS in re-verification after fix Re-verification not run or still FAIL 4 No Regression Impact Fix confirmed to have no negative impact on other P0 rules Another rule newly FAILs 5 Code Review Done Code review approval completed for fixed code Code review not completed PM (5 items) # Checkpoint Go Condition No-Go Condition 1 User Impact Assessment User impact of non-compliant items is acceptable User impact not assessed 2 Schedule Risk Fix timeline is within release schedule Schedule overrun expected 3 Scope Agreement Stakeholder agreement completed for scope changes Agreement not reached 4 Cost Impact AI usage cost within approved budget Budget overrun expected 5 Communication Changes shared with relevant teams Not shared CEO (5 items) # Checkpoint Go Condition No-Go Condition 1 Cost Cap Monthly AI cost within pre-approved budget Budget cap exceeded 2 Security Risk All security P0 passed or exception reason is reasonable P0 security FAIL + insufficient exception justification 3 Legal/Regulatory Risk Data processing complies with applicable laws (privacy laws, etc.) Legal risks not reviewed 4 Business Continuity Business impact is limited if deployment fails Business disruption risk exists 5 Final Approval Final approval when all 4 above are Go Deferred if even 1 is No-Go Report Format compliance-report.md, generated when /compliance-verify runs, consists of 6 sections. Report Section Structure (6 sections) # Compliance Report ## 1. Summary - Project name, verification date/time, verification mode (quick-scan / full-verify) - Total compliance score / 100 - Deploy gate grade (Green / Yellow / Red) - P0 FAIL count - Verification duration ## 2. Rule Results | Rule ID | Category | Rule Name | Result | Score | Evidence | |---------|----------|-----------|--------|-------|----------| | AUTH-P0-001 | Auth | Force Guest for New Signups | PASS | 10/10 | signup.ts:45 | | SEC-P0-005 | Security | Enforce External API Gateway | FAIL | 0/15 | client/api.ts:23 direct fetch | | ... ## 3. Score Breakdown | Domain | Score | Max | % | |--------|-------|-----|---| | Security | 20 | 40 | 50% | | Auth | 25 | 25 | 100% | | Cost | 17 | 20 | 85% | | Logging | 12 | 15 | 80% | | Total | 79 | 100 | 79% | ## 4. Failures Detail For each FAIL item: - Violation code location (file:line) - Description of the violation - Recommended fix (remediation) - Related verification scenario ID (SC-001–SC-005) ## 5. Gate Decision - Verdict grade + basis for verdict - List of required approval roles - Role-based Go/No-Go checkpoint status (unmet items out of 20 shown) ## 6. Recommendations - Immediate action: Fix P0 FAILs (file path + fix guide) - Short-term improvement: Path from Yellow to G

...