browser-agent case study

The problem

A browser agent that runs on a local model and works in the user's real, logged-in browser, with no cloud API. The first use case was typing into comment boxes that live inside a cross-site iframe, behind Shadow DOM, in a ProseMirror editor, which ordinary page scripting can't reach. Later the problem became safety: a local model reading arbitrary web pages is easy to steer with planted instructions.

Constraints

Apple Silicon Mac; the README recommends 32 GB or more. The model is served locally through MLX by the server from claude-code-local.
Model time dominates. The README puts model reasoning at about 99% of each step.
Local models return unreliable JSON. Gemma 4 sometimes answers in its own native tool-call format.
Small context budget: history, tool results and page snapshots are all capped in agent.py.

What he built

Everything is in agent.py (about 1,090 lines), talking to the browser over raw Chrome DevTools Protocol with the websockets package.

class CDP builds page snapshots from Accessibility.getFullAXTree, keeping actionable roles and text that is either long or contains a digit. click() resolves the node, scrolls it into view, reads its box model and dispatches real mouse events. type_into() sends one Input.insertText plus an arrow key nudge so React inputs register the change.
post_comment() finds the comment widget's iframe target, opens a second websocket directly to it, searches with pierce: true for the ProseMirror node, focuses it and inserts text. It never clicks send.
Batching. Since model calls cost seconds and browser calls cost milliseconds, parse_multi() accepts up to 5 tool calls per model turn. The system prompt forbids batching a click on a page the model hasn't seen yet.
Prompt injection guard in code (guard_check(), around line 458). The comment says the checks "live in code... the model cannot switch them off." Text invisible to a human (tiny, transparent, off-screen, or colored like the background) is dropped from snapshots by _HIDDEN_TEXT_JS. After the agent reads a page, shell and file tools require the original task to ask for them; commands built from page text, JavaScript with network calls, clicks on risky buttons, secret fields, banking hosts, and data flowing between sites are blocked unless the task names them.

Upstream credit: MLX and mlx-lm, Chrome DevTools Protocol and Brave, the Gemma 4 and Qwen 3.8 models. The second round of the injection guard was modeled on the attack categories in Anthropic's published browser-use research.

Hard bugs

"Brain-dead" Gemma (9daa713). Gemma 4 answered in its native tool-call format, which the MLX server turned into Anthropic tool_use blocks with empty text. The agent saw no tool call and stalled for 12 steps in a bake-off. ask_model() now folds tool_use blocks back into the agent's JSON format.
Planted instructions (2eb1b8d, a1740f2). Trap pages got the agent to open a link carrying the task text, report a planted answer, and run a shell command from a fake system message. Moving enforcement from the prompt into code fixed it.
Lost tab on reconnect (3ee821f). Behind browser-broker, reconnecting got a throwaway tab, so every new task started on a blank page. One lease per process now, renewed in the background and released on SIGTERM, since atexit doesn't run on SIGTERM.

Results (from the repo)

Typing: 117x faster on a 300-character field, measured 2026-07-01 in an isolated tab against real Brave, Input.insertText at 0.9 ms against 107 ms char by char (commit 7c392a1).
Four-tab test (2026-09-16): four jobs with script-checked answers, run one at a time and then in four tabs at once. Gemma 4 31B got 4/4 in each mode (123 s, then 110 s) and Qwen 3.8 27B got 4/4 in each mode (175 s, then 176 s), so 8/8 each. The 4/4 in commits 2eb1b8d and a1740f2 is a single four-at-once rerun after the injection guard went in (62 s). Source: the bake-off NOTES.md and results.jsonl on his Mac, not checked in.
No automated tests in the repo.

What it shows

He gets a small local model doing real work by putting the deterministic parts in code: parsing, batching, context limits, and safety rules the model can't argue its way past. He tests against adversarial pages instead of only happy paths.