AX tree vs screenshots: why your agent keeps editing the wrong button

with a click handler.

So with three "Delete" buttons, the screenshot gives no way to tell them apart. The agent falls back to the most likely one — usually the first match in the source — and edits the wrong thing. The failure is silent. Nothing errors. The agent was confident. It was grounded in pixels instead of in the element.

The named element is a different kind of fact

macOS already maintains a structured description of every on-screen control: the accessibility (AX) tree. Each node has a role (AXButton, AXTextField, AXLink), a title, a value, an exact frame, and — for Chromium and Electron apps — the underlying AXDOMIdentifier and class list. That is not a guess about pixels. It is the OS stating: this is a button, labeled "Delete", at this frame, DOM id danger-delete-2.

PinVari resolves that named element under the point you circle. The core call is AXUIElementCopyElementAtPosition; it then walks up kAXParentAttribute to build the ancestry — the button, the card holding it, the section containing that. The agent receives the named element, not a rectangle to interpret. Three "Delete" buttons stop being ambiguous the moment each carries its own role, frame, and DOM identity.

Two problems that make this hard

If naming the element were trivial, every tool would do it. Two things get in the way, and both took real work.

1. Everything resolves to your own overlay

The instant you draw a circle, you put a transparent window on top of everything. But AXUIElementCopyElementAtPosition on the system-wide element returns the topmost AX window at that point — now your own overlay. Naively, every capture resolves to "PinVari's transparent window."

The fix is to skip yourself explicitly. Before hit-testing, PinVari walks CGWindowListCopyWindowInfo front-to-back, ignores any window owned by its own PID, filters to normal app windows (kCGWindowLayer == 0), and finds the first real window whose bounds contain your point. It builds an AX element for that app specifically and hit-tests inside it. The mark binds to the app underneath, never to the tool doing the capture. That is what chainExcludingSelf does — the difference between a resolver that works and one that resolves to itself.

2. The Electron cold tree

Half the apps you care about — Claude, VS Code, Slack, Discord, Notion — are Chromium, and Chromium builds its accessibility tree lazily. Until something asks, the app exposes a shallow tree of generic, untitled AXGroups. You circle a button; the tree says "a group." No label, no identity.

PinVari sets AXManualAccessibility = true on the target app to wake that tree — the light, Chromium-specific flag, deliberately not the sticky AXEnhancedUserInterface VoiceOver uses, which can trigger a full-tree rebuild and a CPU spike. Even then Chromium builds asynchronously, so the first hit can still come back generic; the resolver settles briefly and retries, bounded, off the main thread, only while nothing labeled has been found.

And when the point lands on a bare wrapper group, labeledDescendant walks a few levels down to the deepest child whose frame still contains your point and carries real identity — a title, a DOM id, a help string, a placeholder, or a class list. A label-less AXGroup in an Electron app resolves to the button you pointed at, not the container three levels up.

Provenance, and being honest about it

A resolved element is a strong signal, not certainty. Sometimes you deliberately draw a circle around a control. Sometimes the cursor merely passes over it while you talk. Treating those as the same is how you get a tool that is confidently wrong.

So every element PinVari hands the agent carries its provenance and a confidence score. In the payload, a target is tagged circled — you drew around it, trust it — or dwelled — the pointer passed over it while you spoke. When you say "move this into the sidebar," the deictic word is bound to the exact element the pointer was on the instant that word was spoken, pulled from an ordered trail of what you pointed at.

The insight is boring on purpose: tell the agent how sure you are, and why. A circled element with high confidence is something it can act on directly. A dwelled element with lower confidence is something it should confirm first. Provenance turns "here is the element" into "here is the element, and here is how much to trust it" — a judgment a screenshot cannot offer, because it has no idea whether you meant to point at anything at all.

What the agent ends up holding

One capture hands the agent the named element with its role and frame, its DOM identity when the app has one, the provenance and confidence of how you indicated it, the deictic bindings for words like "this" and "that," the full text of the window (not just the visible crop), and on-device OCR for surfaces AX cannot see. It arrives over a local MCP server on 127.0.0.1 — nothing uploaded, no API keys, your own agent.

The screenshot, the AX tree, the MCP transport — competitors have those; they are table stakes. What stops your agent editing the wrong button is the named element plus the honesty about how you named it.

PinVari is a one-time $39 for the first 500 (then $59), on-device, bring your own agent. To try it: claude mcp add pinvari.