Request queue, retries and storage for Node.js crawlers
Crawlee is a Node.js library, written in TypeScript, for building crawlers. You hand it URLs and a function that handles each page; it takes care of the request queue, retries, concurrency and saving results. HTTP crawlers fetch the HTML and parse it with Cheerio or JSDOM without running the page's JavaScript. Browser crawlers drive Playwright or Puppeteer, headless or headful. Both kinds sit behind the same interface.
Apify, which runs a cloud scraping platform, develops Crawlee under the Apache-2.0 license. It also maintains Crawlee for Python, a port that lives in its own repository.
The open source web scraping tools comparison sets Crawlee against anti-detect browsers and proxy rotators, and spells out what each one assumes.
Fingerprints by default, concurrency sized to the host
- One API for HTTP and browsers:
CheerioCrawler,JSDOMCrawler,PlaywrightCrawlerandPuppeteerCrawlershare the router, the request queue, error handling and storage. - A crawler that picks its engine:
AdaptivePlaywrightCrawlerdrops to plain HTTP on pages where it detects the browser isn't needed. The class is flagged experimental. - Fingerprints by default: the Playwright and Puppeteer crawlers generate browser fingerprints out of the box with two Apify packages,
fingerprint-generatorandfingerprint-injector. Over HTTP, the default client, got-scraping, sends browser-like headers and replicates browser TLS fingerprints. - Concurrency sized to the host: the number of pages processed in parallel follows available CPU and memory.
- Proxies tied to sessions:
ProxyConfigurationrotates through a proxy list, and a session keeps the same proxy from one request to the next. - Tooling to get started:
npx crawlee createscaffolds a project from templates, and Docker base images (apify/actor-node-playwright-chromeand its siblings) ship Node.js with a browser.
Bring your own proxies, or pay for Apify's platform
No proxies included. Crawlee rotates whatever proxies you give it, yet its own anti-blocking guide names the IP address as the first cause of blocking. Apify sells its own: the docs describe Apify Proxy as a subscription, and the pricing page bills residential proxies per gigabyte.
The Apify platform is the paid product. The deployment docs list what it covers: compute instances (Actors), request and result storage, proxies, scheduling and webhooks. Moving a crawler there takes Actor.init() from the Apify SDK, then apify push. The Crawlee home page sums it up: "Crawlee, by Apify, works anywhere, but Apify offers the best experience." A free account gets $5 of usage a month, and paid plans charge usage on top of the subscription. The docs also walk through deployments on AWS Lambda and Google Cloud Run.
Storage is local out of the box. The request queue and results land in ./storage; hosted storage you can query over an API is Apify's.
Built for JavaScript developers. Crawlee has no UI, and scheduling is an Apify service. To get an alert when a page changes without writing any code, changedetection.io is a ready-made self-hosted app.
Cloudflare challenges need more. The docs admit the built-in protections are not enough there and point to Camoufox, a modified Firefox. Camoufox alone may not clear the challenge; with the handleCloudflareChallenge helper it "should be able to". Apify maintains camoufox-js, a JavaScript port its repository calls experimental.
v3 is stable, v4 is a release candidate. 3.18.2 shipped on September 29, 2026, after 3.16.0 in February, 3.17.0 in June and 3.18.0 in August. v4.0.0-rc.1 followed on October 6, 2026, and its upgrade guide warns that there are many breaking changes. v3 runs on Node.js 16 or later; v4 needs Node.js 22.13 or later and ships as native ESM. Playwright is not bundled, to keep the install small, so it goes in as a separate dependency.