Element Grounding: Named Elements, Not Pixels

Element grounding is the step that turns a human gesture at a screen — a circle, a point, a "this button" — into the exact named UI element: its role, its label, its bounds, with a confidence score attached. It exists so an AI agent edits the thing you meant instead of inferring it from pixels, and it is the piece almost every screenshot-to-agent workflow skips.

Most tooling in this space treats the picture as the interface contract. The picture is the last place the answer lives, because by the time a frame is encoded the operating system has already thrown away the one thing worth keeping: which object owned that pixel.

What is element grounding, and why name it?

Grounding is a borrowed word. In linguistics, grounding is how a reference in speech gets attached to a thing in the world; in robotics, symbol grounding is how the token "cup" attaches to the object on the table. Element grounding is the same problem aimed at a user interface: the referent is a control on a screen, and the resolution has to be exact enough to edit code against.

It is worth naming because the industry keeps describing the pieces without describing the step. "Screen understanding," "visual grounding," "UI parsing" all describe reading a screen. Element grounding describes resolving one referent on it, which is a narrower and much more testable job.

A grounded element has four parts:

• Role — what kind of control it is (AXButton, AXTextField, AXLink).
• Label — its accessible name, the string a screen reader would speak.
• Bounds — its frame in screen coordinates, plus its parent chain up to the window.
• Confidence + provenance — how sure the resolver is, and how the human indicated it.

Drop any one and you are back to guessing. Role without label gives you "a button" among nine. Label without bounds gives you a string you cannot verify. Bounds without confidence gives you a number that looks authoritative and is sometimes wrong, which is worse than one that admits doubt.

Why do pixel coordinates fail as a way to point?

Because a coordinate describes a moment, not a thing. The pair (842, 517) is only meaningful under a frozen set of assumptions, and every one of those assumptions breaks in normal use.

The window moves. Coordinates captured in screen space point at whatever is now under them. Drag the window an inch and your reference is aimed at the sidebar.

Display scaling changes. A Retina panel reports points, not pixels, at a 2x backing scale. Screenshot pipelines commonly emit pixel-space images while the AX layer and event system speak point space. Mix the two and every coordinate is off by exactly a factor of two — a bug that looks like "the agent clicked the wrong region" and is really a unit error.

The content scrolls. Scroll offset is not encoded in a screen coordinate at all. The element you circled is now 400 points higher, and something entirely different occupies the old rectangle.

Themes and states shift layout. Dark mode changes nothing about geometry, but a longer localized string, a wrapped label, a collapsed sidebar, or a visible scrollbar all reflow the row. So does an A/B variant that ships a different padding scale to half your users.

Contrast that with AXButton labeled "Checkout", frame {x: 842, y: 517, w: 96, h: 32}, parent chain AXWindow → AXGroup → AXToolbar. Move the window and the role and label are unchanged; only the frame updates, and it updates itself, because you can re-query the element. A coordinate is a snapshot; a named element is a handle.

That difference is exactly why agents keep touching the neighbor of what you meant, which I unpacked in why Claude Code fixes the wrong element.

How does element grounding actually work on macOS?

macOS is unusually generous here, because the same infrastructure that powers VoiceOver exposes every control as a queryable object. The entry point is one call:

AXUIElementCopyElementAtPosition(systemWide, x, y, &element)

Give it a screen point and it hands back the AXUIElement under that point. From there you read kAXRoleAttribute, kAXTitleAttribute, kAXValueAttribute, kAXPositionAttribute, kAXSizeAttribute, and walk kAXParentAttribute up to the window. That parent chain is the provenance trail — it is what lets you say "the Checkout button inside the cart toolbar" rather than "a button." The macOS Accessibility API is the whole substrate, and the tree it exposes is the same accessibility tree browsers build from the DOM.

The naive version of this works for about ten minutes. Then you hit the four failure modes that separate a demo from a resolver.

Your own overlay wins the hit test. Any tool that draws a marking layer over the screen is, by definition, the topmost window at the point being tested. So AXUIElementCopyElementAtPosition dutifully returns your overlay. The fix is to walk the on-screen window list yourself, skip your own windows, and hit-test the real application beneath. PinVari calls this chainExcludingSelf. It is unglamorous and it is load-bearing — without it, every single resolution returns the wrong app.

Electron and Chromium build their tree lazily. Chromium-based apps do not construct an accessibility tree until something asks for one, because building it is expensive. Query too early and you get an empty or nameless shell. The remedy is to set AXManualAccessibility on the application element and retry — roughly 150ms in practice — until a labeled element appears. Skip this and half the apps developers live in (VS Code, Slack, Discord, Figma desktop) look accessibility-blind when they are not.

The point lands on a bare AXGroup. Web content is full of nested wrappers with no name. Hit-testing a padded container returns that container, which tells you nothing. So you descend: from the hit element, walk children looking for the deepest descendant whose frame still contains the point and which carries a usable label. PinVari calls this labeledDescendant. The gesture said "this"; the resolver's job is to find the most specific nameable thing under it.

The node has no title at all. Chromium exposes identity attributes most consumers ignore: AXDOMIdentifier and AXDOMClassList. A <div id="checkout-cta" class="btn btn--primary"> with no text content still yields checkout-cta and a class list. That is a name — often a better name for a coding agent than the visible label, because it maps to something greppable in the repo.

What is the element grounding ladder?

Every screen-to-agent approach sits on one of four rungs. The rungs differ in what the agent receives, and therefore in what the agent has to invent.

The jump that matters is rung 3 to rung 4. Dumping the whole tree feels rigorous and is really just relocating the guess — you have replaced "find the button in this image" with "find the button in these 400 nodes," and the model still has no idea which one your finger was over.

Rung 4 is the only rung where the ambiguity is resolved before the model is involved, by the layer that actually observed the human gesture. That is the entire argument for element grounding as a distinct step rather than a model capability.

How does this compare to grounding in computer-use agents?

Autonomous computer-use agents face the same problem from the other direction: no human is pointing, so the agent must ground its own intent onto a screen. The published techniques are worth understanding, because they show how hard the pixel route is even when it is done well.

Set-of-Mark prompting overlays numbered bounding boxes on a segmented screenshot, so the model can say "click 14" instead of producing x-y coordinates it is bad at. The Set-of-Mark paper exists precisely because large multimodal models struggle to emit exact button coordinates from an image — the marks are a workaround for that weakness.

OmniParser goes further, combining a finetuned interactable-icon detector, an icon description model, and OCR to produce a DOM-like structured representation plus an annotated screenshot. Its paper notes that models using set-of-mark prompting are prone to misassigning label IDs to boxes, which is the reason for adding local semantics.

Both approaches are reconstructing, from pixels, a structure the operating system already maintains natively. The detector is inferring "this is interactable" about a thing the AX tree already flags as AXButton with AXPress available.

That reconstruction is necessary when you have only a screenshot — a remote VM, a recording, a cross-platform agent. It is unnecessary when you are running locally and a human is pointing. The gesture plus a hit test collapses the search space to one element in a single call, with no detector, no segmentation, and no ID-assignment failure mode.

Neither replaces the other. They answer different questions: one asks "what is on this screen," the other asks "what did this person mean."

How do confidence and provenance keep grounding honest?

A resolver that always returns an answer is a resolver that lies sometimes. Real UI produces genuinely ambiguous hits: overlapping frames, a tooltip mid-fade, a control whose label is an icon glyph, a hit landing between two rows.

So the output carries a confidence score, and the contract is simple: below threshold, ask; never silently guess. PinVari's line is 0.8. Under it, the agent is told the resolution is uncertain and what the candidates were, and the right behavior is a clarifying question — "the Checkout button, or the cart total above it?" — not a confident edit to the wrong file.

Provenance is the second honesty signal, and it captures how the human indicated the element. A circled mark is deliberate: someone drew a closed shape around a specific region. A dwelled mark is weaker: the pointer rested near something while the person spoke. Same element, different human certainty, and the agent should weight them differently.

How does grounding bind the word "this" to a thing?

Deixis is the linguistic term for words whose meaning depends entirely on context: this, that, here, it. They are the most natural way to talk about UI and the most useless thing to hand a model alone. "Make this blue" contains zero information about what is blue.

Grounding fixes that with a timestamped pointer trail alongside the transcript. Every word from on-device transcription carries a timestamp; every pointer sample carries one too. Bind them and "this" resolves to the element under the pointer at the instant that syllable was spoken, not at the start or end of the sentence.

Timing precision matters most when you circle several things in one breath. "This padding is off, this label is wrong, and this whole card should be narrower" is three references in eight seconds. Per-mark word buckets slice the transcript so each grounded element carries only the words spoken while its mark was active — three instructions bound to three elements, not one paragraph attached to a screenshot. That is the mechanism underneath spatial context, a different problem from element identity: one is which, the other is what about it.

For the deeper walkthrough of the binding itself, see how AI agents know which UI element you mean.

What happens on surfaces with no accessibility data?

Grounding degrades, and the honest move is to say so rather than pretend.

Some surfaces genuinely have nothing to read. A <canvas> element is one opaque node no matter how rich the drawing inside it. Games rendering through Metal expose a window and little else. Certain custom-drawn native controls never implemented the protocol.

On those, the fallback is on-device Vision OCR with word boxes: recognized text plus a rectangle for each word, run locally on the Mac with no upload. You lose role, hierarchy, and the parent chain. You keep a string and a location, which lands you on rung 2 of the ladder — worse than a grounded element, meaningfully better than a bare image.

The important part is that the output says which rung it came from. An agent that knows it received an OCR fallback rather than a resolved AX element can adjust: ask more questions, verify before editing, avoid claiming certainty it does not have.

Where this lands

Element grounding is one step, and it is upstream of everything else. If it is missing, no amount of model capability recovers it, because the information was destroyed at capture time — the moment your gesture became a JPEG.

If it is present, the rest of the stack simplifies. The agent receives a named element, the words spoken about it, a confidence score, and a crop of the region. It edits the right file, or it asks. Both are acceptable. Guessing is not.

PinVari does this step on macOS: hold ⌥⌘A, circle anything on screen, say what is wrong, and your own agent — Claude Code, Cursor, Codex, Zed — receives the resolved element over a local MCP server on 127.0.0.1. On-device transcription, no API keys, nothing uploaded by default. Connect it in one click from the app, or from the terminal:

claude mcp add --scope user pinvari -- "$HOME/.pinvari/mcp/pinvari-mcp"

It is a one-time purchase, priced here, because a grounding layer is infrastructure, not a metered service.

FAQ

What is element grounding in AI?

Element grounding is the step that resolves a human's gesture at a screen — a circle, a point, a spoken "this" — into one specific named UI element with its role, accessible label, screen bounds, and a confidence score. It happens before the AI model runs, so the model receives an identified element rather than an image it has to interpret.

How do AI agents know which button I mean?

They only know if something upstream resolved it. On macOS, AXUIElementCopyElementAtPosition returns the accessibility element under any screen point, giving the agent AXButton labeled "Checkout" at a known frame. Without that step, the agent is doing vision inference on a screenshot and picking the most plausible candidate.

Why do coding agents click or edit the wrong element?

Usually because the input was a screenshot and the agent had to guess which of several similar controls you meant. Pixel coordinates make it worse, since they go stale the moment a window moves, a display scale changes, or the page scrolls. A named element with a parent chain does not have that failure mode.

Is element grounding the same as OCR?

No. OCR gives you text and word boxes with no role, no hierarchy, and no interactivity information — "Save" as a string, four times over. Grounding gives you the object: its role, its accessible name, its frame, its parents. OCR is the fallback used when a surface exposes no accessibility data at all.

Does element grounding work on Electron apps like VS Code and Slack?

Yes, with one extra step. Chromium builds its accessibility tree lazily, so a resolver must set AXManualAccessibility on the app and retry for roughly 150ms until labeled elements appear. Chromium also exposes AXDOMIdentifier and AXDOMClassList, which give a name to nodes that have no visible title.