Element Grounding: Named Elements, Not Pixels

Element grounding is the step that turns a human gesture at a screen — a circle, a point, a "this button" — into the exact named UI element: its role, its label, its bounds, with a confidence score attached. It exists so an AI agent edits the thing you meant instead of inferring it from pixels.

Most tooling in this space treats the picture as the interface contract. The picture is the last place the answer lives, because by the time a frame is encoded the operating system has already thrown away the one thing worth keeping: which object owned that pixel.

What is element grounding, and why name it?

Grounding is a borrowed word. In linguistics, grounding is how a reference in speech gets attached to a thing in the world; in robotics, symbol grounding is how the token "cup" attaches to the object on the table. Element grounding is the same problem aimed at a user interface: the referent is a control on a screen, and the resolution has to be exact enough to edit code against.

It is worth naming because the industry keeps describing the pieces without describing the step. "Screen understanding," "visual grounding," "UI parsing" all describe reading a screen.

Element grounding describes resolving one referent on it, which is a narrower and much more testable job.

A grounded element has four parts:

• Role — what kind of control it is (AXButton, AXTextField, AXLink).
• Label — its accessible name, the string a screen reader would speak.
• Bounds — its frame in screen coordinates, plus its parent chain up to the window.
• Confidence + provenance — how sure the resolver is, and how the human indicated it.

Drop any one and you are back to guessing. Role without label is "a button" among nine; bounds without confidence looks authoritative and is sometimes wrong.

Why do pixel coordinates fail as a way to point?

Because a coordinate describes a moment, not a thing. The pair (842, 517) is only meaningful under a frozen set of assumptions, and every one of those assumptions breaks in normal use.

The window moves. Coordinates captured in screen space point at whatever is now under them. Drag the window an inch and your reference is aimed at the sidebar.

Display scaling changes. A Retina panel reports points, not pixels. Mix screenshot pixel-space with AX point-space and every coordinate is off by a scale factor.

The content scrolls. Scroll offset is not encoded in a screen coordinate at all. The element you circled is now 400 points higher, and something entirely different occupies the old rectangle.

Contrast that with AXButton labeled "Checkout", frame {x: 842, y: 517, w: 96, h: 32}, parent chain AXWindow → AXGroup → AXToolbar. Move the window and the role and label are unchanged; only the frame updates, and it updates itself, because you can re-query the element. A coordinate is a snapshot; a named element is a handle.

That difference is exactly why agents keep touching the neighbor of what you meant, which I unpacked in why Claude Code fixes the wrong element.

How does element grounding actually work on macOS?

macOS is unusually generous here, because the same infrastructure that powers VoiceOver exposes every control as a queryable object. The entry point is one call:

AXUIElementCopyElementAtPosition(systemWide, x, y, &element)

Give it a screen point and it hands back the AXUIElement under that point. From there you read kAXRoleAttribute, kAXTitleAttribute, kAXValueAttribute, kAXPositionAttribute, kAXSizeAttribute, and walk kAXParentAttribute up to the window.

That parent chain is the provenance trail — it is what lets you say "the Checkout button inside the cart toolbar" rather than "a button." The macOS Accessibility API is the whole substrate, and the tree it exposes is the same accessibility tree browsers build from the DOM.

The naive version of this works for about ten minutes. Then you hit the four failure modes that separate a demo from a resolver.

Overlay wins the hit test

A marking layer is the topmost window, so a naive call returns your overlay. Walk the on-screen window list, skip yourself, and hit-test the app beneath (chainExcludingSelf).

Electron and Chromium build the tree lazily

Query too early and you get an empty shell. Set AXManualAccessibility and retry (~150ms) until a labeled element appears.

The point lands on a bare AXGroup

Descend to the deepest labeled child whose frame still contains the point (labeledDescendant).

The node has no title

Chromium identity attrs (AXDOMIdentifier, AXDOMClassList) give a greppable name to a title-less node.

What is the element grounding ladder?

Every screen-to-agent approach sits on one of four rungs. The rungs differ in what the agent receives, and therefore in what the agent has to invent.

1. Raw screenshot

What the agent receives: An image, sometimes with a red circle drawn on it

What the agent must do: Vision-infer which control is meant, then map it to code by guessing

Failure mode: Edits a visually similar sibling; burns image tokens every turn

2. OCR text

What the agent receives: Strings plus word boxes

What the agent must do: Match the spoken phrase to a string, hope it is unique

Failure mode: "Save" appears four times; no role, no hierarchy

3. Full AX tree dump

What the agent receives: Hundreds of nodes for the window

What the agent must do: Search the tree and pick a node

Failure mode: Ambiguous and enormous; the agent picks plausibly, not correctly

4. Grounded named element

What the agent receives: One element: role, label, bounds, parent chain, confidence, provenance

What the agent must do: Act

Failure mode: Resolver reports low confidence and asks

The jump that matters is rung 3 to rung 4. Dumping the whole tree feels rigorous and is really just relocating the guess — you have replaced "find the button in this image" with "find the button in these 400 nodes," and the model still has no idea which one your finger was over.

Rung 4 is the only rung where the ambiguity is resolved before the model is involved, by the layer that actually observed the human gesture. That is the entire argument for element grounding as a distinct step rather than a model capability.

How do confidence and provenance keep grounding honest?

A resolver that always returns an answer is a resolver that lies sometimes. Real UI produces genuinely ambiguous hits: overlapping frames, a tooltip mid-fade, a control whose label is an icon glyph, a hit landing between two rows.

So the output carries a confidence score, and the contract is simple: below threshold, ask; never silently guess. PinVari's line is 0.8.

Under it, the agent is told the resolution is uncertain and what the candidates were, and the right behavior is a clarifying question — "the Checkout button, or the cart total above it?" — not a confident edit to the wrong file.

Provenance is the second honesty signal, and it captures how the human indicated the element. A circled mark is deliberate: someone drew a closed shape around a specific region.

A dwelled mark is weaker: the pointer rested near something while the person spoke. Same element, different human certainty, and the agent should weight them differently.

How does grounding bind the word "this" to a thing?

Deixis is the linguistic term for words whose meaning depends entirely on context: this, that, here, it. They are the most natural way to talk about UI and the most useless thing to hand a model alone.

"Make this blue" contains zero information about what is blue.

Grounding fixes that with a timestamped pointer trail alongside the transcript. Every word from on-device transcription carries a timestamp; every pointer sample carries one too.

Bind them and "this" resolves to the element under the pointer at the instant that syllable was spoken, not at the start or end of the sentence.

Per-mark word buckets slice the transcript so each grounded element carries only the words spoken while its mark was active. That is the mechanism underneath spatial context: one problem is which element, the other is what about it.

For the binding walkthrough, see how AI agents know which UI element you mean.

What happens on surfaces with no accessibility data?

Grounding degrades, and the honest move is to say so rather than pretend.

Some surfaces genuinely have nothing to read. A <canvas> element is one opaque node no matter how rich the drawing inside it.

Games rendering through Metal expose a window and little else. Certain custom-drawn native controls never implemented the protocol.

On those, the fallback is on-device Vision OCR with word boxes: recognized text plus a rectangle for each word, run locally on the Mac with no upload. You lose role, hierarchy, and the parent chain.

You keep a string and a location, which lands you on rung 2 of the ladder — worse than a grounded element, meaningfully better than a bare image.

Where does element grounding land in a real workflow?

Element grounding is one step, and it is upstream of everything else. If it is missing, the information was destroyed at capture time — the moment your gesture became a JPEG.

If it is present, the agent receives a named element, the words spoken about it, a confidence score, and a crop. It edits the right file, or it asks.

PinVari does this step on macOS: hold ⌥⌘A, circle anything on screen, say what is wrong, and your own agent receives the resolved element over a local MCP server on 127.0.0.1.

Connect it in one click from the app, or from the terminal:

claude mcp add --scope user pinvari -- "$HOME/.pinvari/mcp/pinvari-mcp"

It is a one-time purchase, priced here.

FAQ

What is element grounding in AI?

Element grounding is the step that resolves a human's gesture at a screen — a circle, a point, a spoken "this" — into one specific named UI element with its role, accessible label, screen bounds, and a confidence score.

How do AI agents know which button I mean?

They only know if something upstream resolved it. On macOS, AXUIElementCopyElementAtPosition returns the accessibility element under any screen point, giving the agent AXButton labeled "Checkout" at a known frame.

Why do coding agents click or edit the wrong element?

Usually because the input was a screenshot and the agent had to guess which of several similar controls you meant. Pixel coordinates make it worse, since they go stale the moment a window moves, a display scale changes, or the page scrolls.

A named element with a parent chain does not have that failure mode.

Is element grounding the same as OCR?

No. OCR gives you text and word boxes with no role, no hierarchy, and no interactivity information — "Save" as a string, four times over.

Grounding gives you the object: its role, its accessible name, its frame, its parents. OCR is the fallback used when a surface exposes no accessibility data at all.

Does element grounding work on Electron apps like VS Code and Slack?

Yes, with one extra step. Chromium builds its accessibility tree lazily, so a resolver must set AXManualAccessibility and retry until labeled elements appear.