scrape
使用 Bright Data Web Unlocker API 自动绕过反爬虫检测和验证码，将任意网页转为干净的 Markdown 格式内容
npx skills add brightdata/skills --skill scrapeBefore / After 效果对比
1 组使用传统爬虫工具频繁触发 403 错误和验证码，需要不断更换代理和请求头，成功率仅 40%，大量时间浪费在反爬虫对抗上
Web Unlocker 自动处理 IP 轮换、浏览器指纹和验证码绕过，一次 API 调用即可获取完整页面内容，成功率提升至 95% 以上
scrape
Bright Data - Web Scraper
Scrape any webpage and get clean markdown content using Bright Data's Web Unlocker API. Automatically bypasses bot detection and CAPTCHA.
Setup
1. Get your API Key: Get a key from Bright Data Dashboard.
2. Create a Web Unlocker zone: Create a zone at brightdata.com/cp by clicking "Add" (top-right), selecting "Unlocker zone".
3. Set environment variables:
export BRIGHTDATA_API_KEY="your-api-key"
export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name"
Usage
bash scripts/scrape.sh "url"
Parameters:
url(required): The webpage URL to scrape
Examples:
# Scrape a news article
bash scripts/scrape.sh "https://example.com/article"
# Scrape a product page
bash scripts/scrape.sh "https://shop.example.com/product/123"
Output Format
Returns clean markdown content extracted from the webpage:
# Page Title
Main content of the page converted to markdown format...
## Section Heading
More content...
Features
-
Bot Detection Bypass: Automatically handles anti-bot measures
-
CAPTCHA Solving: Bypasses CAPTCHA challenges
-
Clean Markdown: Returns well-formatted markdown content
-
JavaScript Rendering: Handles JavaScript-heavy pages
Dependencies
curl- For API requests
Weekly Installs1.1KRepositorybrightdata/skillsGitHub Stars61First SeenFeb 10, 2026Security AuditsGen Agent Trust HubPassSocketPassSnykWarnInstalled onclaude-code1.0Kcodex60opencode60gemini-cli59amp59github-copilot59
用户评价 (0)
发表评价
暂无评价
统计数据
用户评分
为此 Skill 评分