Crawl4AI
A crawler and scraper that turns websites into Markdown for retrieval systems and agents. It can run in your own Python process, a Docker server, or a hosted service.
Utility: Browser automation; kind: infrastructure.
Stars: 86k+; Starred in September: 4k+.
Metrics updated: 31 Oct 2026. Profile reviewed: 31 Oct 2026.
Utilities
- Build web-to-Markdown ingestion pipelines.
- Extract structured content using configured crawling and extraction strategies.
Who it is for
- Developers who want control over web collection infrastructure.
Requirements
- Install the Python package and browser setup, or deploy the Docker server.
- Configure model credentials only for the extraction features that need them.
Limitations
- Crawling is infrastructure, not an autonomous task planner.
- Site access rules, dynamic content, and extraction settings affect coverage.
Cost and privacy notes
- Local crawling uses your compute and bandwidth; optional models and cloud services can incur charges.
- Review collected content, browser state, and model routing; self-hosting does not remove website privacy obligations.
Illustrative example (untested)
Illustrative, not tested: crawl an allowlisted documentation section and inspect its Markdown before using it in retrieval.
License
Apache-2.0; reviewed LICENSE and README attribution guidance.