indexion-segment
This skill intelligently splits text into contextually meaningful chunks using window, TF-IDF, punctuation, or hybrid strategies. It is primarily designed for preparing data for RAG (Retrieval Augmented Generation) and embedding pipelines, ensuring text content is effectively organized. This enhances AI models' ability to understand and process long documents, improving the quality and relevance of generated responses or semantic search results.
npx skills add https://github.com/trkbt10/indexion-skills --skill indexion-segmentBefore / After Comparison
1 组Before this skill, users had to manually or use simple scripts to segment long texts for RAG or embedding models. This process was time-consuming and often resulted in poor contextual quality, leading to AI models misinterpreting complex documents and affecting the final generation quality.
This skill automatically and intelligently segments text into contextually meaningful chunks, significantly reducing the time and effort required for data preprocessing. By optimizing the quality of text blocks, this skill enhances the retrieval accuracy of RAG systems and AI models' understanding of long documents, thereby improving overall application performance.
indexion segment
Split text into contextual segments using divergence-based, TF-IDF, or punctuation strategies.
When to Use
- User needs to chunk text for RAG or embedding pipelines
- User wants to split a document into meaningful sections
- User asks to segment text for processing
- Preparing text for similarity analysis at sub-document level
Usage
# Default window divergence strategy
indexion segment <input-file> <output-dir>
# TF-IDF based segmentation
indexion segment --strategy=tfidf <input-file> <output-dir>
# Punctuation-based segmentation
indexion segment --strategy=punctuation <input-file> <output-dir>
# Custom segment sizes
indexion segment --min-size=200 --max-size=3000 --target-size=800 document.txt output/
# Custom divergence threshold
indexion segment --threshold=0.5 document.txt output/
# Adaptive threshold mode (default)
indexion segment --adaptive document.txt output/
# Hybrid NCD+TF-IDF mode
indexion segment --hybrid --ncd-weight=0.6 --tfidf-weight=0.4 document.txt output/
# Custom window size
indexion segment --window-size=5 document.txt output/
# Custom output prefix
indexion segment --prefix=chunk document.txt output/
Options
| Option | Default | Description |
|---|---|---|
--strategy=NAME | window | Strategy: window, tfidf, punctuation |
--min-size=INT | 100 | Minimum segment characters |
--max-size=INT | 2000 | Maximum segment characters |
--target-size=INT | 500 | Target segment characters |
--threshold=FLOAT | 0.42 | Divergence threshold |
--window-size=INT | 3 | Window size |
--adaptive | true | Adaptive threshold mode |
--hybrid | false | NCD+TF-IDF hybrid mode |
--ncd-weight=FLOAT | 0.5 | NCD weight in hybrid mode |
--tfidf-weight=FLOAT | 0.5 | TF-IDF weight in hybrid mode |
--prefix=NAME | segment | Output file prefix |
Strategies
| Strategy | Description |
|---|---|
window (default) | Sliding window divergence detection |
tfidf | TF-IDF based topic change detection |
punctuation | Punctuation/sentence boundary based |
Workflow
- Run
indexion segment <input-file> <output-dir>to split text with defaults - Adjust
--thresholdand--target-sizeto tune segmentation granularity - Use
--hybridmode for better accuracy on mixed-content documents
User Reviews (0)
Write a Review
No reviews yet
Statistics
User Rating
Rate this Skill