Crawlee for Python

Typed asyncio crawlers over HTTP or Playwright

Crawlee for Python is the Python edition of Crawlee, the crawling library Apify first wrote for Node.js. A crawler takes a list of URLs, puts them in a persistent request queue, calls a handler for each page and writes what it extracts to a dataset. The whole thing is a plain Python script, built on asyncio and fully typed.

BeautifulSoupCrawler and ParselCrawler fetch pages over HTTP with no browser involved. PlaywrightCrawler drives Chromium, Firefox, WebKit or Chrome for sites that build their content in JavaScript, and AdaptivePlaywrightCrawler learns, as it goes, which pages can skip the browser.

The comparison of open source web scraping tools files Crawlee for Python with the crawling frameworks, next to anti-detect browsers and proxy rotators.

Proxy tiers from cheapest to most reliable

• One API for HTTP and browser crawling: same router, same enqueue_links, same push_data. Moving from ParselCrawler to PlaywrightCrawler changes the class and how you read the page; the structure of the code stays put.
• An HTTP client that passes for a browser: the default is impit, a Rust library from Apify. httpx2 or curl-cffi can replace it through one parameter.
• Fingerprints on by default: PlaywrightCrawler gives each browser context a fingerprint with matching HTTP headers. For better-protected sites, the docs show how to plug Camoufox or CloakBrowser into the same crawler.
• Tiered proxies: tiered_proxy_urls orders proxies from cheapest to most reliable, and the crawler stays on the lowest tier that doesn't get blocked.
• LLM extraction: PydanticAiCrawler fills a Pydantic model from the page, with any provider Pydantic AI supports. StagehandCrawler drives the browser from natural-language instructions.
• A Scrapy translation table: the migration guide maps each Scrapy concept (spiders, item pipelines, AutoThrottle, scrapy-playwright) to its Crawlee counterpart.

SQL and Redis storage Node lacks, both experimental

The Node.js version is the template, minus Puppeteer. Class names come straight from Crawlee for Node.js (BasicCrawler, RequestQueue, SessionPool, ProxyConfiguration), and the request queue was aligned with its "Request Queue V2" implementation. Playwright is the only browser driver, plus Stagehand, which supports Chromium only; there is no PuppeteerCrawler. Fingerprints come from the browserforge library, whereas the Node version uses Apify's own fingerprint-suite.

Python has storage backends Node lacks. Crawlee 3.18 for Node keeps its state in memory and mirrors it to disk. The Python version adds SQL (SQLite, PostgreSQL, MySQL, MariaDB) and Redis 8, its two built-in backends that let several processes share the same data. Both are flagged experimental. The default file system backend is not safe once two processes touch it, and purge_on_start wipes it at every launch: resuming an interrupted crawl means setting it to False.

The two versions part ways with Node's v4. Crawlee for Node has had a v4 release candidate out since 13 August 2026. It replaces proxy tiers with per-session rotation and requires Node 22.13. The Python version, at 1.10, keeps tiered_proxy_urls.

Crawlee brings no IP addresses. ProxyConfiguration rotates whatever proxy URLs you hand it. A self-hosted pool such as Rota shows up as a single URL, and Rota then picks the exit IP. Apify's platform (compute, storage, proxies, scheduling) needs an account; Crawlee runs without one, under the Apache-2.0 license.

Python 3.10 and browsers to download. The crawlee package installs with the extras you need (parsel, playwright or all), then playwright install fetches the browsers. By default, autoscaling caps itself at 25% of available memory.

A release almost every week. The repository dates from January 2024. The 1.x line has 35 releases, from 1.0.0 on 29 September 2025 to 1.10.4 on 6 October 2026.