Give your AI coding agent eyes: hand Claude Code the exact element

Your coding agent can refactor a function across forty files, but it can't see the button you're pointing at. So you narrate: "the settings icon, top right, next to the avatar — no, the other one." You paste a screenshot and hope the model reads the right pixels. You describe a layout bug in three paragraphs because the agent has no way to look at your screen and know what "this" means.

"Give Claude Code eyes" usually gets read as "let me paste a screenshot faster." That helps, and it misses what actually breaks the loop.

What a screenshot gives an agent — and what it doesn't

A screenshot is a flat grid of pixels. Paste one into Claude Code and the model runs vision, guesses at labels, and infers which rectangle you probably meant. It can read visible text and describe the layout. It cannot know which element you care about, what that element is named in the accessibility tree, or what lives off-screen in the scrollback above the crop.

"The third button" in a screenshot isn't an address — it's a description the model re-derives from pixels every time, and re-pastes at full image-token cost on every follow-up. You pay tokens for the model to re-guess what it guessed a message ago, and you still have no guarantee it landed on the element you meant.

The missing piece isn't a picture. It's an identity: the specific named control you were pointing at, with enough provenance for the agent to trust it.

"This button" has no address

macOS already knows the identity of nearly everything on screen. Every native control has an accessibility (AX) element with a role, a title, a position, and a frame — AXButton "Publish", an AXTextField with a placeholder, an AXWebArea carrying the tab's real URL. The data the agent needs exists. Nobody hands it over.

That's PinVari's whole job. You hold Option+Space, circle the thing on screen, and speak. It screenshots, transcribes on-device, and resolves the exact named AX element you circled — with a confidence score and a note on how it got there — then sends it to Claude Code over a local MCP server on 127.0.0.1, or files it into a tracker.

How point-and-speak resolves the named element

Resolving the element under your circle is where the real work is, and the honest details are the differentiator.

The overlay resolves to itself. The obvious approach — AXUIElementCopyElementAtPosition on the system-wide element — makes every mark resolve to PinVari's own transparent overlay, because that overlay is the topmost thing at that pixel. So chainExcludingSelf walks the on-screen window list front-to-back, skips its own process, finds the real application window under your circle, and hit-tests that app only. Every mark binds to the app you're looking at, never to the tool drawing the circle.

Electron and Chromium apps expose nothing by default. Point at a button in Claude Desktop, VS Code, Slack, Discord, or Notion and the raw AX tree hands back a stack of untitled AXGroups — Chromium builds its accessibility tree lazily. PinVari sets AXManualAccessibility on the target process to wake it, then, because the tree builds asynchronously, retries the hit-test (settling ~150ms, up to two tries) until a labeled element appears. When the point still lands on a bare group, labeledDescendant descends into the container to the deepest labeled child whose frame contains your point — the real thing, not the wrapper. It also reads Chromium's identity attributes (AXDOMIdentifier, AXDOMClassList, help text, subrole, placeholder) so a title-less group still carries a usable name.

Confidence and provenance. Each element carries a confidence score and a provenance tag: circled means you deliberately drew around it — trust it — and dwelled means the pointer merely passed over it. The agent weights a deliberate gesture over an incidental one instead of treating every hit the same.

Deixis. When you say "move this into the sidebar," the word "this" points at wherever your pointer was at the instant you said it. PinVari records a timestamped pointer trail and aligns it against the on-device transcript, so each deictic word binds to the element under the cursor at that moment. The agent receives "this" already resolved to a named control, not a pronoun to guess at.

What the agent receives over MCP

Run claude mcp add pinvari and Claude Code gets a handful of tools. The one that matters, pinvari_next_instruction, returns an executable task: the resolved element path, your spoken instruction, the region circled, and a screenshot cropped to that region as ground truth. Alongside it comes the spatial context — the ordered list of controls the pointer moved over (each tagged circled or dwelled, with confidence), the exact deictic bindings, the window's full text (the entire scrollback, not just the visible crop), the active tab URL read straight from the AXWebArea, and any text you had selected.

The screenshot is grabbed the instant your first mark goes down, not when you finish speaking — so it matches what you circled even if a chat or feed scrolled underneath while you talked. The overlay is excluded from the capture, so none of your ink lands in the image the agent sees.

Where AX runs out

AX isn't universal. A canvas, a game, a poorly-built custom UI can expose a genuinely empty tree, and no amount of tree-waking conjures labels that were never published. For those, PinVari falls back to on-device Vision OCR — of the circled crop, and optionally the whole screen, including non-accessibility apps — so the agent at least gets the visible text. That's a labeled fallback, not a resolved element. When the named element is there you get it; when it isn't, you get the honest next-best thing instead of a confident wrong answer.

Everything runs on your Mac. Transcription and OCR are on-device, nothing is uploaded, there are no API keys, and you bring your own agent — PinVari resolves what you pointed at and hands it over the local MCP server.

One-time $39 for the first 500, then $59. If you already run Claude Code: claude mcp add pinvari, hold Option+Space, and circle the thing you've been describing in paragraphs.