apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Summary

Crawlee is a Node.js web scraping and browser automation library written in TypeScript and published under the Apache 2.0 license. It provides a single interface for both HTTP-based scraping and headless browser crawling, and it integrates with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP.

What it is useful for

Engineers reach for Crawlee when they need to extract data from websites, whether for AI or LLM datasets, archiving, or pulling external content into an application. The library handles recurring pain points of building scrapers from scratch: a persistent URL queue that supports breadth- and depth-first traversal, retries, configurable routing, error handling, and pluggable storage for both tabular data and files. It works against static HTML, JSON APIs, and JavaScript-rendered pages, with headless and headful browser modes, browser-like TLS fingerprints, and HTTP/2 support including for proxies. The project ships Dockerfiles for deployment, and since it is open source it runs anywhere Node.js does.

How engineers can use it

The README documents two ways to start. The fastest is the Crawlee CLI, which scaffolds a runnable example:

npx crawlee create my-crawler
cd my-crawler
npm start

For an existing project, install the package and a browser driver. Playwright is not bundled (the README notes this is deliberate to keep install size small), so it is added separately:

npm install crawlee playwright

A minimal PlaywrightCrawler logs page titles, saves results to a Dataset, and follows links on the page:

import { PlaywrightCrawler, Dataset } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks, log }) {
        const title = await page.title();
        log.info(`Title of ${request.loadedUrl} is '${title}'`);
        await Dataset.pushData({ title, url: request.loadedUrl });
        await enqueueLinks();
    },
});

await crawler.run(['https://crawlee.dev']);

By default, data and requests are written to ./storage in the current working directory. Crawlee requires Node.js 22.13 or higher, which is the only documented runtime requirement. For pre-release features, the README documents installing crawlee@next. It also notes that if you combine Crawlee with the Apify SDK you should add dependency overrides in package.json so a single Crawlee version is installed; configuration, request storage, and result storage options are covered in the project's documentation.