How to build a browser-automation agent: perception is the hard part, not clicking

The core decision: adopt vs build, managed vs self-hosted

Two decisions sit above every tool choice, and only some of what follows is a project.

Adopt a ready agent, or build your own loop. browser-use and skyvern are complete agents: hand them a goal and credentials and they perceive, plan, and act. The alternative is your own loop on a raw driver — Playwright or Puppeteer, external libraries — where you own perception, planning, and recovery. Choose adopt if the task is mainstream web work (log in, fill a form, extract, click through a flow), you want something running this week, and you can live with the framework’s perception choices. Choose build if you have tight latency or cost budgets, unusual perception needs, or you’re embedding automation inside a larger system where the framework’s abstractions fight you. Building is real work, and you will re-implement things these agents already solved — do it deliberately, not by default.

A third path, if you’re wiring browser control into a coding agent rather than shipping a standalone app: MCP servers like chrome-devtools-mcp, browser-tools-mcp, Browser MCP, and glance expose browser control to IDE agents such as Cursor or Claude Code.

Managed browser cloud, or self-hosted Chrome. Someone has to run the actual browser. Go managed — Browserbase, Steel’s hosted cloud, or Browser Use Cloud (all commercial) — if you need many concurrent sessions, built-in proxies and stealth, and you’d rather not operate browser infrastructure. Self-host headless Chrome, optionally via the open-source steel-browser, if volume is low, cost matters at scale, or data-handling rules keep the browser inside your own network. Most teams start managed to skip the infra work, then pull hot paths in-house once volume makes the per-session price sting.

The six layers

Six layers of a browser agent, roughly bottom to top. For each: what it is, how to decide, and the one failure mode that quietly costs you.

When the goal is data, not actions

If you only need structured content off a page — prices, listings, article text — an agent is overkill and a liability. A perceive-plan-act loop is slow, costly, and non-deterministic; a scraper is fast and repeatable. firecrawl turns URLs into clean, LLM-ready markdown and handles crawling and JS rendering. llm-scraper extracts data to a schema you define, running on top of Playwright. Reach for an agent only when the data sits behind actions — a login, a search you must perform, a multi-step flow — not merely behind a URL.

The mistakes that sink browser agents

• Assuming the click is the hard part. Chrome will fire a click at any coordinate you name. Knowing which element to hit, that it’s the real one and not a decoy, and that the page finished changing first — that’s the work. Budget your effort for perception, not actuation.
• Trusting the DOM on a canvas or vision-heavy app. Maps, editors, charts, cross-origin iframes, and shadow DOM are invisible or opaque to DOM extraction. If your targets render in pixels, a DOM-only agent is blind exactly where it matters — plan for vision before you’re surprised.
• Running unattended with no progress detection. Without a way to tell “still loading” from “stuck,” the loop retries the same dead end forever, burning tokens until the bill tells you. Cap steps and wall-clock time, and compare state between steps.
• Shipping on a single happy-path demo. A run that works once is not a job that runs a thousand times. The captcha, the cookie banner, the slow response, the shifted layout — the case you didn’t script is the one that sets your real success rate.
• Ignoring bot detection until you’re blocked. A clean-room browser on a datacenter IP gets flagged before it clicks anything. Stealth, fingerprinting, and proxies aren’t a scaling problem you defer — they decide whether the agent works at all.
• Reaching for an agent when a scraper would do. If the data sits behind a URL, not an action, a perceive-plan-act loop is slow, costly, and non-deterministic for no reason. Use a scraper; save the agent for tasks that require a login, a search, or a click-through.

Which should you pick?

• Scrape structured data at scale → firecrawl for crawl-and-convert, or llm-scraper when you need a strict schema. Add a managed browser only if the site is JS-heavy or actively blocks you. No agent loop.
• Automate a known, repeating workflow → prefer determinism: a Playwright script, a skyvern workflow, or record-and-replay with Ui.Vision RPA or OpenAdapt, keeping an LLM only as the fallback for when the layout shifts.
• General open-ended web agent → browser-use if the target sites have honest DOMs; midscene if they’re canvas- or vision-heavy. Run it on a managed cloud or self-host with steel-browser.
• Needs the full desktop, not just a browser → cua or open-computer-use for the sandbox, driven by a general computer-use model or a local VLA like UI-TARS-desktop; reach for Agent-S if you want a planning framework that spans applications.

Frequently asked

DOM or vision — which should I start with?

Start with DOM/structured extraction. It’s faster, cheaper, and more precise, and most of the web still has a readable DOM — browser-use is the common on-ramp. Switch to vision (Midscene, or a VLA model) when the DOM fails you: canvas apps, heavy iframes, shadow DOM, or sites where what renders and what’s in the markup diverge. Most production systems end up hybrid — DOM by default, vision to disambiguate the hard cases.

Do I need a specialized VLA model, or is a general model fine?

Start general. A frontier multimodal computer-use model (Anthropic or OpenAI, commercial) gets you further faster and is easier to prompt. Reach for a specialized open vision-language-action model — UI-TARS-desktop, ShowUI, Fara — when cost per run, latency, data residency, or on-device/offline requirements make per-token API calls untenable. The open models are smaller and cheaper to serve, but you inherit the harness and the ops work.

Managed browser cloud or self-host?

Managed (Browserbase, Steel’s hosted cloud, Browser Use Cloud — all commercial) when you need concurrency, proxies, and stealth immediately and don’t want to run browser fleets. Self-host (steel-browser or plain headless Chrome) when volume is low, cost dominates at scale, or compliance keeps browsers inside your own network. Prototyping managed and later moving hot paths in-house is a normal progression.

How do I stop it looping forever?

Three guards. Cap steps and wall-clock time per task. Add real progress detection — compare state between steps so the agent can distinguish motion from stalling. And define terminal conditions the agent must recognize (success, hard failure, human-needed) rather than hoping it infers them. Skyvern and Nanobrowser give you some scaffolding, but the budgets and stop conditions are ultimately yours to set.

Can I run this unattended in production?

Only behind guardrails. Unattended runs need retries, alerting, an evaluation suite you score continuously, and hard stops before irreversible or sensitive actions — payments, deletions, outbound messages. Some models build this in: Microsoft’s Fara pauses at critical points like checkout and authentication for explicit approval. Assume a human in the loop for anything that spends money or can’t be undone, until your measured success rate earns the longer leash.

Browser agent vs just Playwright scripts?

If the task is fixed and the site is stable, a Playwright script is more reliable, faster, and cheaper — don’t bolt an LLM onto a solved problem. An agent earns its cost when the task is open-ended, the target sites vary, or layouts change often enough that maintaining selectors costs more than tolerating a probabilistic agent. Plenty of mature systems are exactly this: a deterministic Playwright script with an agent fallback for the cases the script can’t handle.

More stack guides