The shortlist
Agents are only as good as what they read. This job — crawl a site, strip the noise, hand back clean markdown — has one dominant tool, a stealthier alternative, and an MCP bridge that wires it straight into your coding agent.
171k stars · Open source · TypeScript
The standard for agent-ready web data — crawl, scrape, and convert to clean markdown. One of the most adopted projects in the entire index and moving fast.
— You need a full browser-automation agent rather than a data API.
76k stars · Open source · Python
Adaptive scraping built to survive site changes and bot detection — the strongest pick when the target sites actively resist automation.
— You need a non-Python stack — Scrapling requires Python 3.10 or higher.
7.3k stars · Open source · JavaScript
Firecrawl exposed as an MCP server — give Claude Code or Cursor web-scraping ability with a single config entry instead of writing integration code.
Scraping has rules: respect robots.txt and site terms, rate-limit, and don't collect personal data. If the data feeds a product, check the source's licensing before it becomes a dependency you can't remove.
