TOPTRACING

Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper

Jan 6, 2025

0
https://crawl4ai.com/https://github.com/unclecode/crawl4ai

Crawl4AI is an open-source Python library designed to simplify web crawling and extract useful information from web pages.

Crawl4AI has one clear task: to make crawling and data extraction from web pages easy and efficient, especially for large language models (LLMs) and AI applications. Whether you are using it as a REST API or a Python library, Crawl4AI offers a robust and flexible solution with full asynchronous support.

It delivers blazing-fast, AI-ready web crawling tailored for LLMs, AI agents, and data pipelines. Open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.

Why Crawl4AI?

  1. Built for LLMs: Creates smart, concise Markdown optimized for RAG and fine-tuning applications.
  2. Lightning Fast: Delivers results 6x faster with real-time, cost-efficient performance.
  3. Flexible Browser Control: Offers session management, proxies, and custom hooks for seamless data access.
  4. Heuristic Intelligence: Uses advanced algorithms for efficient extraction, reducing reliance on costly models.
  5. Open Source & Deployable: Fully open-source with no API keys—ready for Docker and cloud integration.
  6. Thriving Community: Actively maintained by a vibrant community and the #1 trending GitHub repository.

Quick Start

Install Crawl4AI:

# Install the package
pip install -U crawl4ai

# Run post-installation setup
crawl4ai-setup

# Verify your installation
crawl4ai-doctor

If you encounter any browser-related issues, you can install them manually:

python -m playwright install --with-deps chromium

Run a simple web crawl:

import asyncio
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

Features

Markdown Generation

  • Clean Markdown: Generates clean, structured Markdown with accurate formatting.
  • Fit Markdown: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
  • Citations and References: Converts page links into a numbered reference list with clean citations.
  • Custom Strategies: Users can create their own Markdown generation strategies tailored to specific needs.
  • BM25 Algorithm: Employs BM25-based filtering for extracting core information and removing irrelevant content.

Structured Data Extraction

  • LLM-Driven Extraction: Supports all LLMs (open-source and proprietary) for structured data extraction.
  • Chunking Strategies: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
  • Cosine Similarity: Find relevant content chunks based on user queries for semantic extraction.
  • CSS-Based Extraction: Fast schema-based data extraction using XPath and CSS selectors.
  • Schema Definition: Define custom schemas for extracting structured JSON from repetitive patterns.

Browser Integration

  • Managed Browser: Use user-owned browsers with full control, avoiding bot detection.
  • Remote Browser Control: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
  • Session Management: Preserve browser states and reuse them for multi-step crawling.
  • Proxy Support: Seamlessly connect to proxies with authentication for secure access.
  • Full Browser Control: Modify headers, cookies, user agents, and more for tailored crawling setups.
  • Multi-Browser Support: Compatible with Chromium, Firefox, and WebKit.
  • Dynamic Viewport Adjustment: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.

Crawling & Scraping

  • Media Support: Extract images, audio, videos, and responsive image formats like srcset and picture.
  • Dynamic Crawling: Execute JS and wait for async or sync for dynamic content extraction.
  • Screenshots: Capture page screenshots during crawling for debugging or analysis.
  • Raw Data Crawling: Directly process raw HTML (raw:) or local files (file://).
  • Comprehensive Link Extraction: Extracts internal, external links, and embedded iframe content.
  • Customizable Hooks: Define hooks at every step to customize crawling behavior.
  • Caching: Cache data for improved speed and to avoid redundant fetches.
  • Metadata Extraction: Retrieve structured metadata from web pages.
  • IFrame Content Extraction: Seamless extraction from embedded iframe content.
  • Lazy Load Handling: Waits for images to fully load, ensuring no content is missed due to lazy loading.
  • Full-Page Scanning: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.

Deployment

  • Dockerized Setup: Optimized Docker image with API server for easy deployment.
  • API Gateway: One-click deployment with secure token authentication for API-based workflows.
  • Scalable Architecture: Designed for mass-scale production and optimized server performance.
  • DigitalOcean Deployment: Ready-to-deploy configurations for DigitalOcean and similar platforms.

Additional Features

  • Stealth Mode: Avoid bot detection by mimicking real users.
  • Tag-Based Content Extraction: Refine crawling based on custom tags, headers, or metadata.
  • Link Analysis: Extract and analyze all links for detailed data exploration.
  • Error Handling: Robust error management for seamless execution.
  • CORS & Static Serving: Supports filesystem-based caching and cross-origin requests.
  • Clear Documentation: Simplified and updated guides for onboarding and advanced usage.
  • Community Recognition: Acknowledges contributors and pull requests for transparency.

Installation

Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.

Choose the installation option that best fits your needs:

Basic Installation

For basic web crawling and scraping tasks:

pip install crawl4ai
crawl4ai-setup # Setup the browser

By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.

Note: When you install Crawl4AI, the crawl4ai-setup should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:

Through the command line:

playwright install

If the above doesn't work, try this more specific command:

python -m playwright install chromium

This second method has proven to be more reliable in some cases.

Installation with Synchronous Version

The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:

pip install crawl4ai[sync]

Development Installation

For contributors who plan to modify the source code:

git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e .                    # Basic installation in editable mode

Install optional features:

pip install -e ".[torch]"           # With PyTorch features
pip install -e ".[transformer]"     # With Transformer features
pip install -e ".[cosine]"          # With cosine similarity features
pip install -e ".[sync]"            # With synchronous crawling (Selenium)
pip install -e ".[all]"             # Install all optional features

Current Docker Support

The existing Docker implementation is being deprecated and will be replaced soon. If you still need to use Docker with the current version:

  • Deprecated Docker Setup - Instructions for the current Docker implementation
  • Note: This setup will be replaced in the next major release

What's Coming Next?

Our new Docker implementation will bring:

  • Improved performance and resource efficiency
  • Streamlined deployment process
  • Better integration with Crawl4AI features
  • Enhanced scalability options

Quick Test

Run a quick test (works for both Docker options):

import requests

# Submit a crawl job
response = requests.post(
    "http://localhost:11235/crawl",
    json={"urls": "https://example.com", "priority": 10}
)
task_id = response.json()["task_id"]

# Continue polling until the task is complete (status="completed")
result = requests.get(f"http://localhost:11235/task/{task_id}")

For more examples, see our Docker Examples. For advanced configuration, environment variables, and usage examples, see our Docker Deployment Guide.

Advanced Usage Examples

You can check the project structure in the directory https://github.com/unclecode/crawl4ai/docs/examples. Over there, you can find a variety of examples.

Mission

Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.

We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.

Key Opportunities

  • Data Capitalization: Transform digital footprints into measurable, valuable assets.
  • Authentic AI Data: Provide AI systems with real human insights.
  • Shared Economy: Create a fair data marketplace that benefits data creators.

Development Pathway

  1. Open-Source Tools: Community-driven platforms for transparent data extraction.
  2. Digital Asset Structuring: Tools to organize and value digital knowledge.
  3. Ethical Data Marketplace: A secure, fair platform for exchanging structured data.

For more details, see our full mission statement.