The hidden token cost of giving an agent context — and how pointing cuts it

The part of a fix you never see in the diff

Ask a coding agent to "fix the spacing on this button" or "this dropdown is cut off," and the model doesn't start editing. It starts looking for the thing you meant. That looking — the discovery phase — is where most of the tokens on a UI task go, and it's invisible because it happens inside the agent loop, not in the diff you review at the end.

Discovery is a loop: screenshot, reason, maybe screenshot again, dump the accessibility tree or the DOM, guess a selector or a component, try it, screenshot to verify. Every step is a tool call, and every tool result lands in the context window and stays there for the rest of the turn.

The math

Order-of-magnitude, not fake decimals.

Screenshots

Anthropic prices images at roughly (width × height) / 750 tokens. A non-retina window grab around 1456×816 is about 1,600 tokens; a fuller retina capture is 2,000–3,000. One is cheap. But discovery is never one — an establishing shot, a zoom, a post-action shot to confirm — and each image stays in context and gets re-sent on every subsequent turn until something prunes it. Three screenshots across eight reasoning turns isn't 3 × 2k, it's closer to 3 × 2k × 8 in cumulative context pressure.

Accessibility / DOM dumps

This is the expensive one. To work out which control you meant, an agent serializes the accessibility tree or the DOM. A moderately complex web app's a11y snapshot runs 5,000–50,000 tokens. And modern apps make it worse: Chromium and Electron build their accessibility tree lazily, so until something wakes it, an Electron app — Claude Desktop, VS Code, Slack, Notion, Discord — exposes mostly generic unlabeled AXGroup nodes. The agent dumps a huge tree, finds it full of anonymous groups, and still doesn't know which one is your button. It paid the tokens and learned nothing.

Reasoning between the calls

Every screenshot and every tree dump is followed by the model reasoning over it — output tokens, at the higher output rate. A discovery phase of four or five round trips is four or five rounds of "look at what I have, decide what to click next."

Add it up and a single "which element did you mean" resolution can run tens of thousands of tokens before the model writes a line of the actual fix. At a few dollars per million input tokens, no one fix breaks the bank — but the cost is real on the two axes that matter more than the dollar: context-window budget (every discovery token is one not available for the codebase, the diff, or the conversation) and round-trip latency (each tool call is a full model turn you wait through).

What pointing hands the agent instead

PinVari resolves the element on your machine, once, at the moment you point — not repeatedly inside the agent's context. You hold a hotkey, circle the thing, and speak. The macOS app hit-tests the exact pixel you drew around with AXUIElementCopyElementAtPosition and walks up the parent chain to a named element path.

Getting a correct name is the hard part, and it's most of what the app does:

• The naive hit-test returns the topmost accessibility window — which is PinVari's own transparent overlay, so everything resolves to the overlay. The app finds the real window underneath you from the on-screen window list (chainExcludingSelf → targetPID) and hit-tests that process only.
• When the point lands on a bare, unlabeled AXGroup — the Electron cold-tree problem above — it sets AXManualAccessibility to wake the Chromium tree, then descends with labeledDescendant to the deepest child whose frame actually contains your point and that carries a real label, DOM id, placeholder, or class. That's the difference between the agent receiving AXGroup and receiving the "Publish" button in the composer toolbar.
• Because resolution can be wrong, each element carries a confidence score and provenance: circled means you deliberately drew around it (trust it); dwelled means the pointer merely passed over it. The agent knows how much to trust what it got.

When your agent pulls the next instruction over the local MCP server, it receives a compact text payload: the resolved element path, your spoken transcript, the region you circled, the provenance and confidence, deixis bindings (which element the pointer was on the instant you said "this" or "that"), the window's text, and — for surfaces accessibility can't see — on-device OCR. Optionally one small cropped thumbnail of the region. That's a few hundred to low-thousands of tokens, delivered once. The agent skips the discovery loop and goes straight to the change.

The ROI

State it as a ratio, because that's the honest way. Discovery for one UI element: order 10k–100k tokens across several round trips. The resolved handoff: order 1k tokens in one call. Roughly a 10× to 50× reduction on the discovery portion — and discovery is usually the majority of the tokens on a "fix this on screen" task, since the edit itself is a small, targeted diff.

In dollars, saving ~30k tokens is a few cents per fix. Multiply by the point-at-something fixes in a real session and it adds up, but the numbers worth optimizing for are the other two:

• Context headroom. Tokens not spent staring at screenshots are tokens left for your codebase, the surrounding conversation, and the model's reasoning about the change. Discovery bloat is the quiet reason long agent sessions degrade.
• Round trips. Collapsing a four-or-five-turn discovery loop into one handoff is four or five fewer full model turns you wait through, per fix.

None of this replaces the agent's judgment — it removes the guessing about what you were pointing at, which is exactly the part vision-plus-tree-walking is expensive and unreliable at. The MCP server, the accessibility tree, and the screenshots aren't the differentiator; every capable agent stack has those. The difference is you get back the named element with a confidence score and circled-vs-dwelled provenance, resolved once on-device, instead of making the model rediscover it inside its context window every task.

Trying it

PinVari runs entirely on-device — transcription, OCR, and element resolution all local, no API keys, nothing uploaded. You bring your own agent; it talks to PinVari over a local MCP server on 127.0.0.1. Wiring it into Claude Code is one line:

claude mcp add pinvari

Launch price is a one-time $39 for the first 500 seats, then $59. If a chunk of your agent's token budget currently goes to figuring out which button you meant, this is the part it stops paying for.