caveman-compress
Compress natural language files into raw human language style, significantly reducing input tokens while retaining readable backups.
npx skills add https://github.com/juliusbrussee/caveman --skill caveman-compressBefore / After Comparison
1 组Long documents like CLAUDE.md and to-do lists consume a large number of input tokens. When the context window is limited, content needs to be manually condensed, which is time-consuming and can lead to the loss of key information.
Automatically compresses lengthy descriptions into concise, natural human language, retaining all core information. Token usage is reduced by 60%, and readable backups are automatically saved.
Caveman Compress
Purpose
Data leaves the machine. Compression sends the file's full text to the model provider that
scripts/compress.pyis configured for: theclaudeCLI, the Anthropic API, or an OpenAI-compatible endpoint. Files whose names look like secrets (.env,credentials,secrets,.netrc, …) are refused before they are read, and a file whose text holds a credential (a private key block, or a cloud, Git host or model API token) is refused before anything is sent. Don't compress a file the provider shouldn't see.
Compress natural language files (CLAUDE.md, todos, preferences) into caveman-speak to reduce input tokens. Compressed version overwrites original. Human-readable backup saved as <filename>.original.md, but NOT beside the source file — it lives in an out-of-tree data dir ($XDG_DATA_HOME/caveman-compress/backups/<parent-dir-name>/, or %LOCALAPPDATA%\caveman-compress\backups\<parent-dir-name>\ on Windows) so skill auto-loaders don't re-ingest it as a live file.
Trigger
/caveman-compress <filepath> or when user asks to compress a memory file.
Requires an explicit filepath. If the user pastes raw text instead of naming a file, do NOT improvise an inline compression — ask them for the filepath to compress, or offer to save the pasted text to a file first. Writing your own reasoning/commentary about the request into the compressed output (e.g. "User passed text directly, not a filepath...") is the exact failure this guards against — never let that leak into a saved file.
Process
-
The compression scripts live in
scripts/(adjacent to this SKILL.md). If the path is not immediately available, search forscripts/__main__.pynext to this SKILL.md. -
From the directory containing this SKILL.md, run:
python3 -m scripts <absolute_filepath>
- The CLI will:
- detect file type (no tokens)
- call Claude to compress
- validate output (no tokens)
- if errors: cherry-pick fix with Claude (targeted fixes only, no recompression)
- retry up to 2 times
- if still failing after 2 retries: report error to user, leave original file untouched
- Return result to user
Compression Rules
Remove
- Articles: a, an, the
- Filler: just, really, basically, actually, simply, essentially, generally
- Pleasantries: "sure", "certainly", "of course", "happy to", "I'd recommend"
- Hedging: "it might be worth", "you could consider", "it would be good to"
- Redundant phrasing: "in order to" → "to", "make sure to" → "ensure", "the reason is because" → "because"
- Connective fluff: "however", "furthermore", "additionally", "in addition"
Preserve EXACTLY (never modify)
- Code blocks (fenced ``` and indented)
- Inline code (
backtick content) - URLs and links (full URLs, markdown links)
- File paths (
/src/components/...,./config.yaml) - Commands (
npm install,git commit,docker build) - Technical terms (library names, API names, protocols, algorithms)
- Proper nouns (project names, people, companies)
- Dates, version numbers, numeric values
- Environment variables (
$HOME,NODE_ENV)
Preserve Structure
- All markdown headings (keep exact heading text, compress body below)
- Bullet point hierarchy (keep nesting level)
- Numbered lists (keep numbering)
- Tables (compress cell text, keep structure)
- Frontmatter/YAML headers in markdown files
Compress
- Use short synonyms: "big" not "extensive", "fix" not "implement a solution for", "use" not "utilize"
- Fragments OK: "Run tests before commit" not "You should always run tests before committing"
- Drop "you should", "make sure to", "remember to" — just state the action
- Merge redundant bullets that say the same thing differently
- Keep one example where multiple examples show the same pattern
CRITICAL RULE:
Anything inside ... must be copied EXACTLY.
Do not:
- remove comments
- remove spacing
- reorder lines
- shorten commands
- simplify anything
Inline code (...) must be preserved EXACTLY.
Do not modify anything inside backticks.
If file contains code blocks:
- Treat code blocks as read-only regions
- Only compress text outside them
- Do not merge sections around code
Pattern
Original:
You should always make sure to run the test suite before pushing any changes to the main branch. This is important because it helps catch bugs early and prevents broken builds from being deployed to production.
Compressed:
Run tests before push to main. Catch bugs early, prevent broken prod deploys.
Original:
The application uses a microservices architecture with the following components. The API gateway handles all incoming requests and routes them to the appropriate service. The authentication service is responsible for managing user sessions and JWT tokens.
Compressed:
Microservices architecture. API gateway route all requests to services. Auth service manage user sessions + JWT tokens.
Boundaries
- ONLY compress natural language files (.md, .txt, .typ, .typst, .tex, extensionless)
- NEVER modify: .py, .js, .ts, .json, .yaml, .yml, .toml, .env, .lock, .css, .html, .xml, .sql, .sh
- If file has mixed content (prose + code), compress ONLY the prose sections
- If unsure whether something is code or prose, leave it unchanged
- Original file is backed up as FILE.original.md before overwriting — in the out-of-tree backup data dir (see Purpose), not beside the source file
- Never compress FILE.original.md (skip it)
A Claude Code skill that compresses project memory files (CLAUDE.md, todos,
preferences) into caveman format, reducing repeated input size.
Claude loads CLAUDE.md on every session start, so large files add repeated
input tokens. Caveman shortens supported natural-language files.
What It Do
/caveman-compress CLAUDE.md
CLAUDE.md ← compressed (Claude reads smaller file each session)
CLAUDE.original.md ← human-readable backup (you edit this)
Original remains in data directory rather than next to live file, so skill
auto-loaders do not read it twice. Path is
$XDG_DATA_HOME/caveman-compress/backups/<parent-dir-name>-<hash>/ on macOS and
Linux, or %LOCALAPPDATA%\caveman-compress\backups\<parent-dir-name>-<hash>\ on
Windows. <hash> keys the file's full parent path, so ~/.claude/CLAUDE.md and
some-repo/.claude/CLAUDE.md keep separate backups.
To edit and re-compress, restore first — the skill refuses to run while a backup exists, so that it never captures already-compressed text as the new "original":
FILE=path/to/the/file/you/compressed
cp "$BACKUP" "$FILE" # restore the uncompressed original
mv "$BACKUP" "$BACKUP.kept" # move it aside (do not just delete it)
# change $FILE, then run the skill again
Benchmarks
The five fixture pairs in tests/caveman-compress/,
counted with tiktoken o200k. Reproduce with
uv run --with tiktoken python skills/caveman-compress/scripts/benchmark.py.
| File | Original | Compressed | Saved |
|---|---|---|---|
claude-md-preferences.md | 827 | 421 | 49.1% |
claude-md-project.md | 1628 | 1117 | 31.4% |
mixed-with-code.md | 1432 | 1106 | 22.8% |
project-notes.md | 1431 | 846 | 40.9% |
todo-list.md | 880 | 648 | 26.4% |
| Total | 6198 | 4138 | 33.2% |
All fixture validations passed: headings, code blocks, URLs, and file paths were preserved exactly.
Before / After
Original (827 tokens)
"I strongly prefer TypeScript with strict mode enabled for all new code. Please don't use
anytype unless there's genuinely no way around it, and if you do, leave a comment explaining the reasoning. I find that taking the time to properly type things catches a lot of bugs before they ever make it to runtime."
Caveman (421 tokens)
"Prefer TypeScript strict mode always. No
anyunless unavoidable; comment why if used. Proper types catch bugs early."
This fixture produced 49.1% fewer counted tokens. Structural validation passed; result does not prove semantic equivalence on other files or models.
Security
caveman-compress is flagged as Snyk High Risk due to subprocess and file I/O
patterns detected by static analysis. See SECURITY.md for why
these operations exist and how paths are constrained.
Install
Compress is built into the caveman plugin. Install caveman once, then use /caveman-compress.
If you need local files, the compress skill lives at:
skills/caveman-compress/
Requires Python 3.10 or newer.
Provider / model
Default path uses Claude: ANTHROPIC_API_KEY + Anthropic SDK when set, else claude --print.
To use your existing Codex CLI login, install Codex CLI and sign in with ChatGPT:
codex login
export CAVEMAN_COMPRESS_PROVIDER=codex
ChatGPT login uses your plan's Codex allowance; a saved API-key login uses API
billing. Neither OPENAI_API_KEY nor ANTHROPIC_API_KEY is required. User CLI
configuration, including custom providers and configured models, is ignored.
Optional: set CAVEMAN_COMPRESS_MODEL to a model available in your Codex account:
export CAVEMAN_COMPRESS_MODEL=your-codex-model
CAVEMAN_MODEL is the fallback; otherwise Codex uses its built-in default.
Requires CLI 0.166.0 or newer, with isolation verified on 0.166.0. Unsupported
controls fail the call without weaker settings or switching providers.
Runs codex exec in an empty temporary directory, including outside Git repos.
The prompt goes through stdin; --output-last-message supplies the answer.
Commands and other tool surfaces are disabled; a custom profile denies all
sandboxed filesystem and network access. Raw failure diagnostics are withheld.
See security details and limits.
To use opencode instead (any model opencode can reach, free ones included):
export CAVEMAN_COMPRESS_PROVIDER=opencode
export CAVEMAN_COMPRESS_MODEL=opencode/big-pickle # any id from `opencode models`
To use a local model (or any OpenAI-compatible server) instead:
export CAVEMAN_COMPRESS_PROVIDER=openai-compat
export CAVEMAN_COMPRESS_MODEL=qwen3:8b # required
export CAVEMAN_COMPRESS_ENDPOINT=http://localhost:11434/v1 # default (Ollama)
# export CAVEMAN_COMPRESS_API_KEY=... # optional, sent as Bearer token
Common endpoints: Ollama http://localhost:11434/v1, LM Studio http://localhost:1234/v1, llama.cpp server http://localhost:8080/v1, vLLM http://localhost:8000/v1. Nothing leaves your machine with a local server. A non-local http:// endpoint sends the file in plaintext; use https:// for remote servers. Small local models get it wrong more often; when they do, validation fails and your file stays untouched.
CAVEMAN_MODEL is the fallback when CAVEMAN_COMPRESS_MODEL is unset. On the Claude path the model also applies to claude --print. The opencode path needs opencode 2.x (it uses opencode run --standalone).
Usage
/caveman-compress <filepath>
Examples:
/caveman-compress CLAUDE.md
/caveman-compress docs/preferences.md
/caveman-compress todos.md
What files work
| Type | Compress? |
|---|---|
.md, .mdc, .txt, .rst, .typ, .typst, .tex | Yes |
| Extensionless natural language | Yes |
.py, .js, .ts, .json, .yaml | ❌ Skip (code/config) |
*.original.md | ❌ Skip (backup files) |
How It Work
/caveman-compress CLAUDE.md
↓
basic checks: file exists, under 500KB, not a sensitive filename
↓
acquire cross-session lock on the file (waits up to 15 min if another run holds it, then errors)
↓
detect file type (no tokens)
↓
refuse if text holds a key or token (no tokens, nothing sent)
↓
configured provider compresses (tokens: one call)
↓
validate output (no tokens)
checks: headings, code blocks, URLs, file paths, bullets
↓
if errors: configured provider fixes cherry-picked issues only (tokens: targeted fix)
does NOT recompress; only patches broken parts
↓
validate again (one fix call at most)
↓
compressed text → same path as input
original → backup dir, as <name>.original.md
Only two things use tokens: initial compression + targeted fix if validation fails. Everything else is local Python.
What Is Preserved
Caveman compress natural language. It never touch:
- Code blocks (
```fenced or indented) - Inline code (
`backtick content`) - URLs and links
- File paths (
/src/components/...) - Commands (
npm install,git commit) - Technical terms, library names, API names
- Headings (exact text preserved)
- Tables (structure preserved, cell text compressed)
- Dates, version numbers, numeric values
- Anything you wrap in
<!-- nocompress -->...<!-- /nocompress --> - HTML comments that start a line (
<!-- ... -->)
File that say generated by on line 1 get skipped. Generator overwrite it anyway.
Want a part left exactly as written (an <example> block, a prompt template)? Wrap it. Each tag go on its own line. Model never change what inside; it come back byte for byte. Model can still read it if first try need fixing, so not hiding place for secrets. Forget the closing tag and compress stop before touching file.
<!-- nocompress -->
<example>
Reply in this exact format.
</example>
<!-- /nocompress -->
Why This Matter
CLAUDE.md loads on every session start. A 1,000-token project memory file adds
1,000 input tokens each time project opens, or 100,000 across 100 sessions.
Caveman reduced counted tokens by 33.2% in total on the five listed fixtures (22.8% to 49.1% per file). Validators confirmed headings, code blocks, URLs, and file paths. They did not establish general semantic or task-quality equivalence.
┌────────────────────────────────────────────┐
│ TOKEN SAVINGS, ALL FIVE ███ 33.2% │
│ FIXTURES IN TABLE 5 │
│ STRUCTURAL VALIDATION passed on all │
│ SETUP TIME █ 1x │
└────────────────────────────────────────────┘
Part of Caveman
This skill is part of the caveman toolkit.
caveman: ask Claude to answer in shorter prosecaveman-compress: shorten supported project-memory files with backups and validation
User Reviews (0)
Write a Review
No reviews yet
Statistics
User Rating
Rate this Skill