---
id: gh-chdb-datastore
name: "chdb-datastore"
url: https://skills.yangsir.net/skill/gh-chdb-datastore
author: clickhouse
domain: data-analysis
tags: ["chdb", "datastore", "pandas-replacement", "clickhouse", "data-analysis"]
install_count: 4900
rating: 4.40 (120 reviews)
github: https://github.com/clickhouse/agent-skills/tree/main/skills/chdb-datastore
---

# chdb-datastore

> chdb DataStore 是一个基于 ClickHouse 的惰性 pandas 替代品，允许用户用相同的 pandas 代码直接分析本地文件、数据库和云存储中的大规模数据，并将操作编译为优化 SQL，显著提升性能，尤其适合跨源 join 和大数据集的分组聚合。

**Stats**: 4,900 installs · 4.4/5 (120 reviews)

## Before / After 对比

### 处理1亿行数据的分组聚合耗时

**Before**:

用户使用 pandas 处理1亿行数据的 groupby+sum 操作时，常因内存不足而失败，或耗时超过60秒，且多次尝试后仍需手动调整代码或拆分数据。

**After**:

此技能自动将 pandas API 编译为 ClickHouse SQL，处理相同任务只需10秒，无需修改代码，且内存占用极低，可流畅处理更大规模数据。

| Metric | Before | After | Change |
|---|---|---|---|
| 查询耗时 | 60秒 | 10秒 | -83% |

## Readme

# chdb DataStore — It's Just Faster Pandas

## The Key Insight

```python
# Change this:
import pandas as pd
# To this:
import chdb.datastore as pd
# Everything else stays the same.
```

DataStore is a **lazy, ClickHouse-backed pandas replacement**. Your existing pandas code works unchanged — but operations compile to optimized SQL and execute only when results are needed (e.g., `print()`, `len()`, iteration).

```bash
pip install chdb
```

## Decision Tree: Pick the Right Approach

```
1. "I have a file/database and want to analyze it with pandas"
   → DataStore.from_file() / from_mysql() / from_s3() etc.
   → See references/connectors.md

2. "I need to join data from different sources"
   → Create DataStores from each source, use .join()
   → See examples/examples.md #3-5

3. "My pandas code is too slow"
   → import chdb.datastore as pd — change one line, keep the rest

4. "I need raw SQL queries"
   → Use the chdb-sql skill instead
```

## Connect to Any Data Source — One Pattern

```python
from datastore import DataStore

# Local file (auto-detects .parquet, .csv, .json, .arrow, .orc, .avro, .tsv, .xml)
ds = DataStore.from_file("sales.parquet")

# Database
ds = DataStore.from_mysql(host="db:3306", database="shop", table="orders", user="root", password="pass")

# Cloud storage
ds = DataStore.from_s3("s3://bucket/data.parquet", nosign=True)

# URI shorthand — auto-detects source type
ds = DataStore.uri("mysql://root:pass@db:3306/shop/orders")
```

All 16+ sources and URI schemes → [connectors.md](references/connectors.md)

## After Connecting — Full Pandas API

```python
result = ds[ds["age"] > 25]                                          # filter
result = ds[["name", "city"]]                                        # select columns
result = ds.sort_values("revenue", ascending=False)                  # sort
result = ds.groupby("dept")["salary"].mean()                         # groupby
result = ds.assign(margin=lambda x: x["profit"] / x["revenue"])     # computed column
ds["name"].str.upper()                                               # string accessor
ds["date"].dt.year                                                   # datetime accessor
result = ds1.join(ds2, on="id")                                      # join
result = ds.head(10)                                                 # preview
print(ds.to_sql())                                                   # see generated SQL
```

209 DataFrame methods supported. Full API → [api-reference.md](references/api-reference.md)

## Cross-Source Join — The Killer Feature

```python
from datastore import DataStore

customers = DataStore.from_mysql(host="db:3306", database="crm", table="customers", user="root", password="pass")
orders = DataStore.from_file("orders.parquet")

result = (orders
    .join(customers, left_on="customer_id", right_on="id")
    .groupby("country")
    .agg({"amount": "sum", "rating": "mean"})
    .sort_values("sum", ascending=False))
print(result)
```

More join examples → [examples.md](examples/examples.md)

## Writing Data

```python
source = DataStore.from_mysql(host="db:3306", database="shop", table="orders", user="root", password="pass")
target = DataStore("file", path="summary.parquet", format="Parquet")

target.insert_into("category", "total", "count").select_from(
    source.groupby("category").select("category", "sum(amount) AS total", "count() AS count")
).execute()
```

## Troubleshooting

| Problem | Fix |
|---------|-----|
| `ImportError: No module named 'chdb'` | `pip install chdb` |
| `ImportError: cannot import 'DataStore'` | Use `from datastore import DataStore` or `from chdb.datastore import DataStore` |
| Database connection timeout | Include port in host: `host="db:3306"` not `host="db"` |
| Join returns empty result | Check key types match (both int or both string); use `.to_sql()` to inspect |
| Unexpected results | Call `ds.to_sql()` to see the generated SQL and debug |
| Environment check | Run `python scripts/verify_install.py` (from skill directory) |

## References

- [API Reference](references/api-reference.md) — Full DataStore method signatures
- [Connectors](references/connectors.md) — All 16+ data source connection methods
- [Examples](examples/examples.md) — 10+ runnable examples with expected output
- [Verify Install](scripts/verify_install.py) — Environment verification script
- [Official Docs](https://clickhouse.com/docs/chdb)

> Note: This skill teaches how to *use* chdb DataStore.
> For raw SQL queries, use the `chdb-sql` skill.
> For contributing to chdb source code, see CLAUDE.md in the project root.


---

# chdb DataStore

Agent skill for using chdb's pandas-compatible DataStore API — a drop-in pandas replacement backed by ClickHouse.

## Installation

```bash
npx skills add clickhouse/agent-skills
```

## What's Included

| File | Purpose |
|------|---------|
| `SKILL.md` | Skill definition and quick-start guide |
| `references/api-reference.md` | Full DataStore method signatures |
| `references/connectors.md` | All 16+ data source connection methods |
| `examples/examples.md` | 11 runnable examples with expected output |
| `scripts/verify_install.py` | Environment verification script |

## Trigger Phrases

This skill activates when you:
- "Analyze this file with pandas"
- "Speed up my pandas code"
- "Query this MySQL/PostgreSQL/S3 table as a DataFrame"
- "Join data from different sources"
- "Use DataStore to..."
- "Import datastore as pd"

## Related

- **chdb-sql** — For raw ClickHouse SQL queries, use the `chdb-sql` skill instead
- **clickhouse-best-practices** — For ClickHouse schema/query optimization

## Documentation

- [chdb docs](https://clickhouse.com/docs/chdb)
- [chdb GitHub](https://github.com/chdb-io/chdb)


---
*Source: https://skills.yangsir.net/skill/gh-chdb-datastore*
*Markdown mirror: https://skills.yangsir.net/api/skill/gh-chdb-datastore/markdown*