unclecode/crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper

Summary

Crawl4AI is an open-source Python web crawler and scraper that turns any website into clean, LLM-ready Markdown. It supports both self-hosted usage (library, Docker, CLI) and a hosted cloud API with MCP integration for AI agents. Licensed under Apache-2.0, it builds on Playwright with Chromium, Firefox, and WebKit support.

What it is useful for

Crawl4AI fits scenarios where you need to feed web content into LLMs or AI agents. Its Markdown output preserves headings, tables, code blocks, and citations, and content filters (PruningContentFilterLXML, BM25ContentFilter, LLMContentFilter) trim menus and footers to keep prompts lean.

For structured data, you can extract records without an LLM using CSS or XPath schemas (JsonCssExtractionStrategy, RegexExtractionStrategy), or call any LiteLLM-supported model for typed JSON extraction (LLMExtractionStrategy). Long pages can be chunked by topic or sentence, and cosine similarity helps pick relevant chunks for a query.

The browser layer offers persistent profiles for saved logins, proxy rotation, headers/cookies control, and a stealth mode for sites that detect automation. For discovery, AsyncUrlSeeder pulls from sitemaps and Common Crawl, prefetch=True speeds up URL finding, and AdaptiveCrawler stops once it has enough to answer your query. Deep crawl supports BFS, DFS, and best-first strategies with crash recovery via resume_state. JS-heavy pages can be handled with scan_full_page, wait_for, and custom JS execution, plus screenshots and PDF output.

Cloud features (pay-as-you-go) add a search API, an answers endpoint (experimental), batch jobs for up to 10,000 URLs, and MCP endpoints so tools like Claude Code, Codex, Cursor, and OpenCode can call the scraper directly. A documented limitation: web search and answer endpoints are cloud-only, and the answers endpoint is labeled experimental.

How engineers can use it

Local pip install with browser setup:

pip install -U crawl4ai
crawl4ai-setup

A minimal scrape:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown)

asyncio.run(main())

The CLI (crwl) can fetch a page, deep-crawl a site, or run an LLM extraction:

crwl https://news.ycombinator.com -o markdown
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
crwl https://www.example.com/products -q "Extract all product prices"

For self-hosting as a server, the Docker image exposes /md, /html, /crawl, /screenshot, /pdf, /execute_js, and an MCP endpoint, all guarded by a CRAWL4AI_API_TOKEN. A dashboard and playground are included.

For deeper configuration options (proxies, sessions, content filters, extraction strategies, hooks) the README points to docs.crawl4ai.com.