Product Hunt logo dark
  • Launches
    Coming soon
    Upcoming launches to watch
    Launch archive
    Most-loved launches by the community
    Launch Guide
    Checklists and pro tips for launching
  • Products
  • News
    Newsletter
    The best of Product Hunt, every day
    Stories
    Tech news, interviews, and tips from makers
    Changelog
    New Product Hunt features and releases
  • Forums
    Forums
    Ask questions, find support, and connect
    Streaks
    The most active community members
    Events
    Meet others online and in-person
  • Advertise
Subscribe
Sign in
Subscribe
Sign in
Crawlee

Crawlee

Crawlee helps you build reliable crawlers. Fast.

4.4•8 reviews•

520 followers

Crawlee helps you build reliable crawlers. Fast.

4.4•8 reviews•

520 followers

Visit website
Engineering & Development
•
Automation tools
•
Databases and backend frameworks
Crawlee is an intuitive, customizable open-source library for web scraping and browser automation. Quickly scrape data, store it, and avoid getting blocked with auto-generated human-like fingerprints, headless browsers, and smart proxy rotation.
  • Overview
  • Launches2
  • Reviews8
  • Alternatives
  • Team
  • Awards
  • More
Company Info
crawlee.devGitHub
Crawlee Info
Launched in 2022View 2 launches
Forum
p/crawlee
  • Blog
  • •
  • Newsletter
  • •
  • Questions
  • •
  • Forums
  • •
  • Product Categories
  • •
  • Apps
  • •
  • About
  • •
  • FAQ
  • •
  • Terms
  • •
  • Privacy and Cookies
  • •
  • X.com
  • •
  • Facebook
  • •
  • Instagram
  • •
  • LinkedIn
  • •
  • YouTube
  • •
  • Advertise
© 2025 Product Hunt
SocialFacebookInstagramX

Similar Products

AnyPicker
AnyPicker
Scrape web data without any code, just click what you see.
3.5(2 reviews)
No-code platformsData analysis tools
Sitebulb
Sitebulb
The website auditing tool for SEO consultants and agencies
Seomator
Seomator
Seomator analyzes site
JetOctopus
JetOctopus
SEO crawler and log analyzer
Ninfex
Experimental people-powered search engine
Appwrite
Appwrite — The open-source Vercel alternative
The open-source Vercel alternative
Promoted

Crawlee Launches

Crawlee for Python
Crawlee for Python Build reliable scrapers in Python

Launched on July 9th, 2024

Do you use Crawlee?

This is the 2nd launch from Crawlee. View more
Crawlee for Python

Crawlee for Python

Build reliable scrapers in Python
We are launching Crawlee for Python, an open-source library for web scraping and browser automation. Quickly scrape data, store it, and avoid getting blocked, headless browsers, and smart proxy rotation.
Crawlee for Python gallery image
Crawlee for Python gallery image
Crawlee for Python gallery image
Free
Launch tags:
Open Source•Growth Hacking•Developer Tools
Launch Team
Saurav JainMartin AdámekVlada Dusek

What do you think? …

Saurav Jain
Saurav Jain
Crawlee

Crawlee

Maker
📌
Hello Hunters and Makers, I am Saurav, Developer Community Manager of Apify, the company building Crawlee. I am happy to hunt Crawlee for Python today. We launched (Crawlee) in August 2022 and received an amazing response from the community, as well as continuous demand for building it in Python. Finally, after a lot of hard work from our team, we are launching Crawlee for Python today. It has all of these features: - Unified interface for HTTP & headless browser crawling. - Automatic parallel crawling based on available system resources. - Written in Python with type hints - enhances DX (IDE autocompletion) and reduces bugs (static type checking). - Automatic retries on errors or when you’re getting blocked. - Integrated proxy rotation and session management. - Configurable request routing - direct URLs to the appropriate handlers. - Persistent queue for URLs to crawl. - Pluggable storage of both tabular data and files. - Robust error handling. Why use Crawlee rather than Scrapy? - Crawlee has out-of-the-box support for headless browser crawling (Playwright). - Crawlee has a minimalistic & elegant interface - Set up your scraper with fewer than 10 lines of code. - Complete type hint coverage. - Based on standard Asyncio. Please pass on your feedback and thoughts in the comments below!
Report
1yr ago
Csaba Kissi
Csaba Kissi
Invoice generator

Invoice generator

@sauain, I wish I could see the version of this for PHP. Anyway... Great product!
Report
1yr ago
Zeiki Yu
Zeiki Yu
Tidyread

Tidyread

@sauain This is exactly what I needed. Thanks for building Crawlee!
Report
1yr ago
Saurav Jain
Saurav Jain
Crawlee

Crawlee

Maker
@csaba_kissi thanks for the support, well you never know ;)
Report
1yr ago
Khyati Agarwal
Khyati Agarwal
Congratulations on the launch🎉 Amazing work👏 Scraping in headless browser had so many gaps!
Report
1yr ago
Saurav Jain
Saurav Jain
Crawlee

Crawlee

Maker
@khyati_tmw thanks for the support, Khyati!
Report
1yr ago
Kyrylo Silin
Kyrylo Silin
Telebugs

Telebugs

This looks like a powerful tool for web scraping and browser automation. How does Crawlee's proxy rotation and session management compare to other tools on the market? Any plans to add more integrations? Congrats on the launch, Saurav!
Report
1yr ago
Saurav Jain
Saurav Jain
Crawlee

Crawlee

Maker
hey @kyrylosilin! we use our [Session Pool](https://crawlee.dev/python/api/c...) system to rotate the sessions, and similar to Crawlee TS/JS we are going to use [Tiered Proxies](https://crawlee.dev/blog/proxy-m...) in Crawlee for Python as well.
Report
1yr ago
4.4
Based on 8 reviews
Review Crawlee?

Crawlee is praised for its ease of use and comprehensive features, making it accessible even for those with limited programming experience. Users appreciate its efficient setup process and the ability to extend functionality with minimal effort. The library's browser fingerprinting and proxy rotation capabilities are highlighted as standout features. However, some users note that onboarding could be improved, citing weak examples and limited support. Overall, Crawlee is seen as a powerful tool for web scraping and automation, offering a balance of simplicity and advanced functionality.

Robert-Andrei DamianAlexey UdovydchenkoLuisa Calixto
+5
Summarized with AI
Pros
Cons
Reviews
Helpful

You might also like

AnyPicker
AnyPicker
Scrape web data without any code, just click what you see.
Sitebulb
Sitebulb
The website auditing tool for SEO consultants and agencies
Seomator
Seomator
Seomator analyzes site
JetOctopus
JetOctopus
SEO crawler and log analyzer
Ninfex
Experimental people-powered search engine
View more
Pros
easy to use (6)
browser fingerprinting (2)
quick setup (2)
Matthias Stephens
Matthias Stephens
•1 review
Compared to competing libraries, Crawlee is easier to use, yet also more feature rich. In just like 20 lines of code, you can have a crawler set up and ready to go that uses proxy rotation, automatic session management, header generation, fingerprint generation, and request management. When building a crawler with Crawlee, you have more time to focus on the job at hand rather than reinventing the wheel and creating custom implementations for proxy-management or request-management etc.
Report
3yr ago
Antonis Blackjim
Antonis Blackjim
•1 review
not that great onboarding. Weak examples and poor support in the discord channel. Almost no support I'd say.
Report
3yr ago
Steven Smith
Steven Smith
•1 review
A convenient tool. It allows you to quickly set up and start gathering information, even if you don't have much programming experience.
Report
11mo ago