---
id: gh-indexion-segment
name: "indexion-segment"
url: https://skills.yangsir.net/skill/gh-indexion-segment
author: trkbt10
domain: ai-llm-engineering
tags: ["text-segmentation", "rag", "llm-optimization", "data-preparation", "nlp"]
install_count: 4900
rating: 4.40 (120 reviews)
github: https://github.com/trkbt10/indexion-skills/tree/main/skills/indexion-segment
---

# indexion-segment

> 此技能能将文本智能地分割成具有上下文意义的片段，支持窗口、TF-IDF、标点或混合策略。它主要用于为 RAG（检索增强生成）和嵌入管道准备数据，确保文本内容被有效组织，从而提升 AI 模型理解和处理长文档的能力。

**Stats**: 4,900 installs · 4.4/5 (120 reviews)

## Before / After 对比

### RAG 数据预处理效率

**Before**:

在没有此技能之前，用户需要手动或通过简单的脚本来分割长文本，以适应 RAG 或嵌入模型的输入要求。这个过程不仅耗时，而且难以保证分割的上下文质量，经常导致 AI 模型在理解复杂文档时出现偏差，影响最终的生成效果。

**After**:

此技能能够自动且智能地将文本分割成具有上下文意义的片段，显著减少了数据预处理所需的时间和精力。通过优化文本块的质量，该 Skill 提升了 RAG 系统的检索准确性和 AI 模型对长文档的理解能力，从而提高了整体应用性能。

| Metric | Before | After | Change |
|---|---|---|---|
| 数据预处理时间 | 120分钟 | 10分钟 | -91% |

## Readme

# indexion segment

Split text into contextual segments using divergence-based, TF-IDF, or punctuation strategies.

## When to Use

- User needs to chunk text for RAG or embedding pipelines
- User wants to split a document into meaningful sections
- User asks to segment text for processing
- Preparing text for similarity analysis at sub-document level

## Usage

```bash
# Default window divergence strategy
indexion segment <input-file> <output-dir>

# TF-IDF based segmentation
indexion segment --strategy=tfidf <input-file> <output-dir>

# Punctuation-based segmentation
indexion segment --strategy=punctuation <input-file> <output-dir>

# Custom segment sizes
indexion segment --min-size=200 --max-size=3000 --target-size=800 document.txt output/

# Custom divergence threshold
indexion segment --threshold=0.5 document.txt output/

# Adaptive threshold mode (default)
indexion segment --adaptive document.txt output/

# Hybrid NCD+TF-IDF mode
indexion segment --hybrid --ncd-weight=0.6 --tfidf-weight=0.4 document.txt output/

# Custom window size
indexion segment --window-size=5 document.txt output/

# Custom output prefix
indexion segment --prefix=chunk document.txt output/
```

## Options

| Option | Default | Description |
|--------|---------|-------------|
| `--strategy=NAME` | window | Strategy: window, tfidf, punctuation |
| `--min-size=INT` | 100 | Minimum segment characters |
| `--max-size=INT` | 2000 | Maximum segment characters |
| `--target-size=INT` | 500 | Target segment characters |
| `--threshold=FLOAT` | 0.42 | Divergence threshold |
| `--window-size=INT` | 3 | Window size |
| `--adaptive` | true | Adaptive threshold mode |
| `--hybrid` | false | NCD+TF-IDF hybrid mode |
| `--ncd-weight=FLOAT` | 0.5 | NCD weight in hybrid mode |
| `--tfidf-weight=FLOAT` | 0.5 | TF-IDF weight in hybrid mode |
| `--prefix=NAME` | segment | Output file prefix |

## Strategies

| Strategy | Description |
|----------|-------------|
| `window` (default) | Sliding window divergence detection |
| `tfidf` | TF-IDF based topic change detection |
| `punctuation` | Punctuation/sentence boundary based |

## Workflow

1. Run `indexion segment <input-file> <output-dir>` to split text with defaults
2. Adjust `--threshold` and `--target-size` to tune segmentation granularity
3. Use `--hybrid` mode for better accuracy on mixed-content documents


---
*Source: https://skills.yangsir.net/skill/gh-indexion-segment*
*Markdown mirror: https://skills.yangsir.net/api/skill/gh-indexion-segment/markdown*