Firecrawl
The API to search, scrape, and crawl the web for AI — clean Markdown out of any site, LLM-powered extraction, and a first-class MCP server.
Firecrawl turns the messy web into agent-ready data: /scrape renders any page to clean Markdown, /crawl walks whole sites, /map discovers URLs, /search queries the web, and /extract pulls structured data with an LLM. Open-source core (AGPL-3.0) with a hosted API, and an MIT MCP server installable into Claude Code as a hosted remote or local npx server.
Firecrawl is the ingestion workhorse of the agent stack: give it a URL and get back clean Markdown; give it a domain and get back the whole site, crawled and converted. At ~179k GitHub stars (as of September 2026) it has become the default answer to "how do I get web content into my LLM pipeline without writing a scraper per site."
Highlights
/scrape— any page to clean Markdown or JSON, JavaScript rendering included./crawl+/map— walk entire sites with depth/limit controls, or just discover the URL tree fast./search— web search with optional content scraping of the results in one call./extract— LLM-powered structured extraction: define a schema, get validated objects from messy pages (Firecrawl now calls/agentits successor).- Agent-grade MCP server — about 25 tools spanning scrape/map/search/crawl, parse, research, monitoring, and newer agent/browser-session tools (the former Extract tool is deprecated); hosted or local.
- Open core — AGPL-3.0, self-hostable; the hosted cloud adds managed scale and the proprietary Fire-Engine.
In an AI-assisted workflow
claude mcp add --transport http firecrawl https://mcp.firecrawl.dev/v2/mcp-oauth
# OAuth sign-in; the keyless /v2/mcp endpoint covers only search, scrape, and parse
# then:
# > Crawl docs.example.com, extract every API endpoint and its auth requirements
# > into a table, and flag the ones our client doesn't implement yetFor RAG ingestion, Firecrawl is the step before chunking: site → clean Markdown → chunks → embeddings, without the per-site parser zoo.
WARNING
Two operational cautions: keep your API key in an environment variable or secret store, never in the hosted MCP URL (Firecrawl's docs are explicit on this), and scraped content is untrusted input to your model (the classic indirect prompt-injection vector). Respect target sites' policies; Firecrawl's own terms put that responsibility on you.
Good to know
Plans, as of September 2026 from firecrawl.dev/pricing: Free is 1,000 credits a month with no card, then Hobby at $16/month for 5,000 credits, Standard at $83/month for 100,000, Growth at $333/month for 500,000, and Scale at $599/month for 1,000,000 — all quoted on annual billing, with monthly billing costing more. Credits don't roll over, except on Scale (one month) and Enterprise (a custom amount). The company raised a $14.5M Series A (Nexus, with Y Combinator) alongside the v2 API in August 2025, and the GitHub org renamed from mendableai to firecrawl. Pair with Exa — search to find pages, Firecrawl to extract them — for the full web-data layer under an agent.
Frequently asked questions
- What does Firecrawl do that plain fetching doesn't?
- It handles the web's hostile parts — JavaScript rendering, anti-bot friction, pagination, layout noise — and returns clean Markdown or structured JSON ready for an LLM. One endpoint scrapes a page; /crawl does entire sites with depth and limit controls; /extract turns 'get every product's name and price' into a schema-validated result.
- How do I add Firecrawl to Claude Code?
- Hosted remote, as Firecrawl documents it: claude mcp add --transport http firecrawl https://mcp.firecrawl.dev/v2/mcp works keyless, or send your API key as an Authorization: Bearer header, never in the URL; an OAuth variant lives at /v2/mcp-oauth. Local: run npx -y firecrawl-mcp with FIRECRAWL_API_KEY set in the environment.
- Is Firecrawl open source?
- The core is AGPL-3.0 and self-hostable (SDKs and some components are MIT, as is the MCP server). The hosted cloud adds proprietary niceties like Fire-Engine. AGPL matters if you modify and operate it as a service — most teams just use the hosted API, whose free tier is 1,000 credits a month as of September 2026.
Filed under
scraping · crawling · web-data · mcp · api · agents
Related
- ExaThe search engine built for AIs — semantic web search, page contents, Websets, and research APIs, plus a hosted MCP server.
- The Best MCP Servers in 2026The MCP servers actually worth connecting in 2026 — Context7, GitHub, Chrome DevTools, Playwright, Serena, Exa, Firecrawl, and official vendor servers.
- Adding MCP Servers to Claude Code: Local, Remote, and Project-ScopedThe complete claude mcp add reference — stdio vs HTTP transports, local/project/user scopes, .mcp.json with env expansion, OAuth via /mcp, and the gotchas.
- Chunking Strategy OptimizerFind the chunking strategy and size that maximizes retrieval quality for a specific corpus, by sweeping configurations against a fixed eval set instead of guessing. Use when RAG answers miss obvious content, when standing up a new corpus, or when picking chunk size/overlap.
- Data EngineerUse this agent to build and maintain data pipelines — ingestion, ELT/ETL, warehouse modeling, orchestration, and data-quality tests. Examples — building an idempotent ingestion job, modeling a fact/dimension table in dbt, writing a safe backfill for a changed schema.
- How RAG Actually Works: Ingestion, Chunking, Retrieval & RerankingA clear, practical walkthrough of the retrieval-augmented generation pipeline — what each stage does, where it fails, and how the pieces fit together.
- TavilyThe web-access layer for agents — Search, Extract, Crawl, Map, and Research APIs purpose-built for LLMs, behind one key, with a hosted MCP server.
- Jina ReaderPrepend r.jina.ai/ to any URL and get LLM-ready markdown — JS rendering, PDFs and Office docs, image captioning, and s.jina.ai for read-the-results search.