OpenClaw 新手教程

openclaw 中文openclaw 2026-04-09 1

什么是 OpenClaw？

OpenClaw 是一个开源的网络爬虫框架，专为数据采集和分析设计，它提供了简单易用的API，支持分布式爬取、反爬虫绕过、数据清洗等功能。

OpenClaw 新手教程-第1张图片-OpenClaw下载中文-AI中文智能体

系统要求

Python 3.7+
内存：至少 2GB RAM
网络连接

安装步骤

pip安装

pip install openclaw

源码安装

git clone https://github.com/openclaw/openclaw.git
cd openclaw
pip install -r requirements.txt
python setup.py install

基础使用

1 创建第一个爬虫

from openclaw import Spider, Request
class MySpider(Spider):
    name = "my_first_spider"
    def start_requests(self):
        # 起始URL
        yield Request("https://example.com")
    def parse(self, response):
        # 提取数据
        title = response.css('h1::text').get()
        yield {
            'title': title,
            'url': response.url
        }
# 运行爬虫
if __name__ == "__main__":
    spider = MySpider()
    spider.run()

2 配置爬虫

class ConfigSpider(Spider):
    name = "config_demo"
    # 基础配置
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/page1", "https://example.com/page2"]
    # 爬取间隔（秒）
    download_delay = 2
    # 并发数
    concurrent_requests = 5
    def parse(self, response):
        # 提取链接并跟进
        for link in response.css('a::attr(href)').getall():
            yield Request(response.urljoin(link), callback=self.parse_page)
    def parse_page(self, response):
        # 页面解析逻辑
        pass

进阶功能

1 数据处理管道

from openclaw import Pipeline
class CleanDataPipeline(Pipeline):
    def process_item(self, item, spider):
        # 清洗数据
        if 'title' in item:
            item['title'] = item['title'].strip()
        return item
class SaveToJSONPipeline(Pipeline):
    def __init__(self):
        self.items = []
    def process_item(self, item, spider):
        self.items.append(item)
        return item
    def close_spider(self, spider):
        import json
        with open('output.json', 'w') as f:
            json.dump(self.items, f, indent=2)

2 中间件使用

from openclaw import Middleware
class CustomMiddleware(Middleware):
    def process_request(self, request, spider):
        # 添加自定义请求头
        request.headers['User-Agent'] = 'MyCustomAgent/1.0'
        return request
    def process_response(self, response, spider):
        # 处理响应
        if response.status == 403:
            spider.logger.warning(f"被封禁: {response.url}")
        return response

实战示例：爬取新闻网站

from openclaw import Spider, Request
import re
class NewsSpider(Spider):
    name = "news_crawler"
    def start_requests(self):
        urls = [
            'https://news.example.com/tech',
            'https://news.example.com/business'
        ]
        for url in urls:
            yield Request(url, callback=self.parse_category)
    def parse_category(self, response):
        # 提取文章链接
        article_links = response.css('.article-list a::attr(href)').getall()
        for link in article_links:
            yield Request(
                response.urljoin(link),
                callback=self.parse_article,
                meta={'category': response.url}
            )
    def parse_article(self, response):
        yield {
            'title': response.css('h1.article-title::text').get(),
            'content': ' '.join(response.css('.article-content p::text').getall()),
            'author': response.css('.author-name::text').get(),
            'publish_date': response.css('.publish-date::text').get(),
            'category': response.meta['category'],
            'url': response.url
        }
    def process_exception(self, request, exception, spider):
        # 异常处理
        self.logger.error(f"请求失败: {request.url}, 错误: {exception}")

配置文件

创建 config.yaml：

spider:
  name: "my_spider"
  download_delay: 1
  concurrent_requests: 3
  user_agent: "OpenClaw/1.0"
database:
  enabled: true
  type: "sqlite"
  path: "./data.db"
middlewares:
  - "openclaw.middlewares.RetryMiddleware"
  - "myproject.middlewares.CustomMiddleware"
pipelines:
  - "openclaw.pipelines.ValidationPipeline"
  - "myproject.pipelines.CleanDataPipeline"

常用命令

# 运行爬虫
openclaw run myspider.py
# 带配置运行
openclaw run myspider.py -c config.yaml
# 查看爬虫状态
openclaw status
# 导出数据
openclaw export -f json -o output.json
# 分布式运行
openclaw run --distributed --workers 4

调试技巧

1 使用Shell调试

from openclaw.shell import inspect_response
# 在parse方法中调试
def parse(self, response):
    inspect_response(response)  # 进入交互式shell
    # 继续编写解析代码

2 日志配置

import logging
# 设置日志级别
logging.getLogger('openclaw').setLevel(logging.DEBUG)
# 自定义日志格式
logging.basicConfig(
    format='%(asctime)s [%(name)s] %(levelname)s: %(message)s',
    level=logging.INFO
)