The shortlist
Agents are only as good as what they read. This job — crawl a site, strip the noise, hand back clean markdown — can use a hosted API, a Python toolkit, or an MCP connector. These require different setup and processing.
These are documentation and catalogue-based suggestions, not hands-on benchmarks. Check the linked product sources, supported languages, regional availability, permissions and current costs. Use the same evaluation worksheet for each candidate.
189k stars · Open source · TypeScript
Web-content extraction for an application: evaluate page coverage, output formats and downstream processing against your actual sources. The hosted service and self-hosted software can have different requirements.
Capability source ↗— You need a full browser-automation agent rather than a data API.
86k stars · Open source · Python
A Python fetching, parsing and crawling toolkit. Adaptive extraction and browser behavior depend on configuration; test your pages instead of assuming a bot-detection or resilience advantage.
Capability source ↗— You need a non-Python stack — Scrapling requires Python 3.10 or higher.
7.6k stars · Open source · JavaScript
Firecrawl exposed as an MCP server — give Claude Code or Cursor web-scraping ability with a single config entry instead of writing integration code.
— You need crawl, map or agent without signing up; those tools require an API key or OAuth.
Scraping has rules: respect robots.txt and site terms, rate-limit, and don't collect personal data. If the data feeds a product, check the source's licensing before it becomes a dependency you can't remove.
