---
id: daily-scrape
name: "scrape"
url: https://skills.yangsir.net/skill/daily-scrape
author: brightdata
domain: ai-data-management-analysis
tags: ["web-scraping", "data-extraction", "markdown"]
install_count: 10300
rating: 4.50 (4 reviews)
github: https://github.com/brightdata/skills
---

# scrape

> 使用 Bright Data Web Unlocker API 自动绕过反爬虫检测和验证码，将任意网页转为干净的 Markdown 格式内容

**Stats**: 10,300 installs · 4.5/5 (4 reviews)

## Before / After 对比

### 网页抓取成功率

**Before**:

使用传统爬虫工具频繁触发 403 错误和验证码，需要不断更换代理和请求头，成功率仅 40%，大量时间浪费在反爬虫对抗上

**After**:

Web Unlocker 自动处理 IP 轮换、浏览器指纹和验证码绕过，一次 API 调用即可获取完整页面内容，成功率提升至 95% 以上

| Metric | Before | After | Change |
|---|---|---|---|
| 抓取成功率 | 40% | 95% | +138% |
| 平均抓取时间 | 30秒 | 3秒 | -90% |

## Readme

# scrape

# Bright Data - Web Scraper

Scrape any webpage and get clean markdown content using Bright Data's Web Unlocker API. Automatically bypasses bot detection and CAPTCHA.

## Setup

**1. Get your API Key:**
Get a key from [Bright Data Dashboard](https://brightdata.com/cp).

**2. Create a Web Unlocker zone:**
Create a zone at brightdata.com/cp by clicking "Add" (top-right), selecting "Unlocker zone".

**3. Set environment variables:**

```
export BRIGHTDATA_API_KEY="your-api-key"
export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name"

```

## Usage

```
bash scripts/scrape.sh "url"

```

**Parameters:**

- `url` (required): The webpage URL to scrape

**Examples:**

```
# Scrape a news article
bash scripts/scrape.sh "https://example.com/article"

# Scrape a product page
bash scripts/scrape.sh "https://shop.example.com/product/123"

```

## Output Format

Returns clean markdown content extracted from the webpage:

```
# Page Title

Main content of the page converted to markdown format...

## Section Heading

More content...

```

## Features

- **Bot Detection Bypass**: Automatically handles anti-bot measures

- **CAPTCHA Solving**: Bypasses CAPTCHA challenges

- **Clean Markdown**: Returns well-formatted markdown content

- **JavaScript Rendering**: Handles JavaScript-heavy pages

## Dependencies

- `curl` - For API requests

Weekly Installs1.1KRepository[brightdata/skills](https://github.com/brightdata/skills)GitHub Stars61First SeenFeb 10, 2026Security Audits[Gen Agent Trust HubPass](/brightdata/skills/scrape/security/agent-trust-hub)[SocketPass](/brightdata/skills/scrape/security/socket)[SnykWarn](/brightdata/skills/scrape/security/snyk)Installed onclaude-code1.0Kcodex60opencode60gemini-cli59amp59github-copilot59

---
*Source: https://skills.yangsir.net/skill/daily-scrape*
*Markdown mirror: https://skills.yangsir.net/api/skill/daily-scrape/markdown*