What Is Spatial Context for AI Coding Agents?

Spatial context is everything on your screen that your repository cannot tell an AI coding agent: the screenshot, the exact element you are pointing at, the words you say about it, and the full text of the window it lives in — delivered together, so the agent acts on the right thing. It is the difference between "the toggle is broken" and AXSwitch "Email notifications" inside AXWindow "Settings", at 0.94 confidence, with your sentence attached.

Most discussion of agent context stops at the repo — files, symbols, tests, a CLAUDE.md. That is the half the agent can already read on its own; spatial context is the half only you can see, and handing it over badly is why agents confidently patch the wrong element.

What is spatial context in plain terms?

Think about what actually happens when you find a bug in a running app.

You are looking at a screen. You know exactly which control is wrong, because your eyes are on it. You know what should happen instead. And you have a repository open in another window that contains, somewhere, the code behind that control.

The agent has the repository. It does not have the screen, your gaze, or your finger. Everything on that side of the gap is spatial context.

It breaks into four things the agent is missing:

1. What the screen looks like — the pixels.
2. Which thing on it you mean — the region and the gesture.
3. What you want done to it — your spoken or typed instruction.
4. What surrounds it — the labels, the error text, the state of the window.

Give it one of those and it guesses. Give it all four, fused into a single named target, and it acts.

Why is spatial context different from repository context?

Repository context is symbolic, stable, and already machine-readable. A function has a name, a file path, and a line number. Two agents reading the same repo see the same thing.

Spatial context is none of that. It is transient, visual, and indexed by where your pointer happens to be at one moment. Scroll the page and it changes. Resize the window and it changes. Switch to dark mode and half the pixels change while the meaning does not.

That instability is why the naive approaches fail. Pixel coordinates — "the element at (842, 391)" — are the most obvious encoding and the most fragile one. They break on scroll, on a different display scale, on a window a few points wider than yours, on any layout the agent did not see with its own eyes.

The stable encoding is a name. AXButton titled "Publish", inside AXGroup "Composer toolbar", inside AXWindow "Draft". That survives scrolling, resizing, and theming, because it describes identity rather than position. I go deeper on the structure itself in what the accessibility tree is.

Where does the idea of spatial context come from?

It is not new, and knowing the history stops you from reinventing a solved problem badly.

In 1980, Richard A. Bolt of the MIT Architecture Machine Group published "Put-That-There: Voice and Gesture at the Graphics Interface" at SIGGRAPH. In the Media Room demo, a user sat in front of a wall-sized display, pointed at a shape, and said "put that there." Speech supplied the verb. The pointing supplied the nouns. Neither channel could have carried the command alone.

That is the founding experiment for spatial context, and its lesson is still the whole thing: the words and the pointing are one signal, not two. Forty-five years later, most tooling still treats them as two — a screenshot in one field, a text prompt in another, and a model left to align them.

The linguistics term for the phenomenon is deixis: words whose meaning depends entirely on the situation they are uttered in. "This", "that", "here", "there", "now". A deictic word carries no content of its own — it is a pointer, and it resolves only against the context of the utterance.

When you say "make this blue" while your cursor is over a button, the sentence is complete for a human standing beside you and useless for an agent reading it an hour later in a log. The missing piece is not vocabulary. It is the pointing, timestamped to the word.

What are the channels of spatial context?

Four channels carry it in, and a fifth step decides whether any of it is usable.

Channel 1: The screenshot (pixels)

The screenshot is what the screen looked like. It is the channel everyone reaches for first because it is the easiest to produce and the easiest for a multimodal model to accept.

It is also the weakest on its own. A flat image has no element names, no roles, no frames, no scroll state. The model sees shapes and infers meaning, which is exactly the guessing you are trying to eliminate.

And it is expensive. A full-resolution capture is a large block of image tokens that carries no structure in return — the arithmetic is in why screenshots waste Claude Code tokens. A screenshot cropped to the region you circled costs a fraction of a full-screen one and, crucially, has already had the disambiguation done for it.

Channel 2: Pointing (region and gesture)

Pointing is the channel that turns "somewhere on this screen" into "here." A circle drawn around a control, a hover that lingers, a freehand loop around three related fields.

The gesture carries more than location. It carries intent strength: a deliberate circle around an element means something different from a pointer that merely passed over it on the way somewhere else. A system that records which happened can weight them differently instead of treating every coordinate as equally meaningful.

Pointing also scopes the problem. Circle one row in a table and the agent gets that row, not the table, not the page.

Channel 3: Voice (deictic speech)

Voice is how the instruction arrives, and it is the channel most tools throw away by making you type instead.

Speaking is faster than typing, but that is the small reason. The real reason is that speech is where deixis lives. Nobody types "make the element currently under my cursor blue." They say "make this blue" — and they say it while pointing.

The requirement, then, is not just transcription. It is transcription with timing: knowing which word was spoken at which millisecond, so each word can be matched to where the pointer was at that instant.

Channel 4: The accessibility tree (named elements and window text)

This is the channel that converts everything above from a guess into a fact.

On macOS, every on-screen control is exposed through the Accessibility API. Ask the system for the element at a screen point — AXUIElementCopyElementAtPosition — and you get back a real node: role, title, value, frame, and the chain of parents it sits inside. That is not a description of a button. It is the button.

The same tree carries the surrounding text: the labels, the error message under the field, the value in the input, the heading of the section. That surrounding text is often the part that tells the agent why something is wrong, and it is invisible in a cropped screenshot.

I unpack the resolver mechanics — including the awkward parts, like an overlay window hit-testing to itself, and Chromium apps that build their accessibility tree lazily — in how AI agents know which UI element you mean.

What is element grounding, and why do most tools miss it?

The four channels are inputs. On their own they are four separate blobs handed to a model that then has to align them itself — which is the guessing problem moved one step later, not solved.

The fifth channel is resolution: collapsing all four into a single named element with a confidence score. Call it element grounding — the same move as grounding a pronoun in linguistics, or grounding a bounding box to an object in vision. You take an ambiguous reference and you bind it to one specific thing.

Element grounding is what makes the output actionable. Instead of handing the agent:

• an image,
• a coordinate pair,
• the string "make this blue",
• and a wall of window text,

you hand it one card: this element, this role, this label, this parent chain, this instruction, this confidence, this provenance. One target, already disambiguated, with an honest number attached saying how sure the resolver is.

The confidence score is the part that separates a tool from a toy. A grounded element without a confidence score is still a guess, just a well-dressed one. If the resolver lands on a bare unlabeled group, or the tree came back thin, the number should drop and the agent should ask rather than silently pick something.

Provenance matters for the same reason. An element you deliberately circled deserves more trust than one your pointer happened to dwell on. Tagging which is which lets the agent weight them instead of flattening both into "the user meant this, probably."

How does PinVari capture spatial context on macOS?

PinVari is a native macOS app built around exactly this pipeline. You hold ⌥⌘A, circle or point at anything on screen, and speak. It captures all four channels and performs the element grounding step before your agent ever sees anything.

Concretely, here is what it collects and hands over:

• The named element. The resolved accessibility node under your mark — role, label, frame, parent chain — with a confidence score and provenance marked as circled or dwelled.
• Full window text, up to 40,000 characters. Not just what is visible: the text of the focused window including content scrolled out of view. The error message above the fold that you scrolled past is still in the payload.
• Whole-screen OCR, optionally. On-device, for surfaces where the accessibility tree is empty.
• Per-word voice-to-mark alignment. Circle three things in one breath and each mark gets its own words. "Make this blue, this bigger, and delete that" splits correctly across three targets instead of arriving as one ambiguous sentence.
• Deictic binding. Each spoken word is bound to the element that was under the pointer at the instant you said it, so "this" resolves to what you were actually pointing at when the word left your mouth.
• Dwell detection. You do not have to circle. Hovering for 0.2 seconds or more is enough to resolve an element, which matters when you are talking through a screen quickly.
• Multi-display capture. Each mark remembers which display it was drawn on, so a three-monitor setup does not collapse into one confused coordinate space.

Transcription runs on-device — Apple's system speech engine, with a bundled whisper.cpp as the floor when the system engine is unavailable. It runs on Apple Silicon and Intel Macs alike.

The delivery path is a local MCP server on 127.0.0.1, so the grounded capture goes to your agent — Claude Code, Cursor, Codex, Zed — and not through anyone's cloud. You add it once:

claude mcp add --scope user pinvari -- "$HOME/.pinvari/mcp/pinvari-mcp"

There is a reverse direction too, which is the part people underestimate. The agent can request spatial context mid-task with pinvari_request_capture: it hits a point where it genuinely does not know which element you meant, and instead of guessing, it asks you to point. The notch island lights up, you circle the thing, and it continues. That turns spatial context from a one-shot handoff into a channel that stays open for the length of the task.

How is spatial context different from computer use?

They sound similar and solve opposite problems.

Computer use means the agent drives your machine: it captures frames, moves your cursor, clicks, and types. The human is the supervisor. It is genuinely powerful for automating a flow end to end.

Spatial context means the human drives and the agent receives. You point, you speak, the agent gets a grounded target and goes to work in the codebase. Nothing takes over your mouse.

For the everyday loop of reviewing a build and pointing out what is broken, spatial context is the lighter and more accurate tool. You already know which element is wrong. The job is transmitting that knowledge losslessly, not asking a model to rediscover it from a video feed.

Do you actually need spatial context, or is pasting a screenshot fine?

For one bug a week, paste the screenshot. It works, and adding tooling to a rare task is not worth it.

The break-even arrives fast, though, once you are doing UI review as a loop. Ten captures a day means ten chances to describe the wrong element, ten rounds of the agent asking "which button?", and ten full-resolution images in a context window that has better uses. The cost is not the screenshot itself. It is the wrong fixes and the clarifying round-trips.

The other break-even is surface coverage. Browser-only capture tools are excellent inside Chromium and blind everywhere else — your IDE, your native Mac app, your Electron client. Spatial context resolved through the system accessibility layer works on all of them, because the operating system exposes the tree for every app, not just web pages.

And if you are still deciding whether your agent can see anything at all, start with the honest baseline in can Claude Code see my screen. The short version is no, not by itself — which is precisely the gap spatial context fills.

FAQ

What is spatial context in AI?

Spatial context is the on-screen information an AI agent cannot get from your code: the screenshot, the specific element you are pointing at, the words you say about it, and the surrounding window text. It is delivered together so the agent acts on one unambiguous target rather than inferring one from an image.

How is spatial context different from computer use?

Computer use puts the agent at the controls — it captures live frames and drives your mouse and keyboard. Spatial context keeps you at the controls: you point and speak, and the agent receives a single grounded element plus your instruction. One automates a flow; the other transmits which thing you mean.

Do AI coding agents need screen access?

Not raw screen access, no. What they need is the resolved identity of the element you care about, which can be produced on-device and sent as a small text payload. Handing an agent a stream of pixels is a heavier and less precise way to answer the same question.

What is deixis and why does it matter for AI agents?

Deixis is the linguistic term for words like "this", "that", "here", and "there" that only mean something in the situation they are spoken in. Agents fail on deictic instructions because the pointing that resolved the word was never captured. Binding each spoken word to where the pointer was at that instant fixes it.

What is element grounding?

Element grounding is the step that collapses the screenshot, the gesture, the speech, and the accessibility tree into one named element with a confidence score and a note about how it was selected. It is the difference between four raw signals and one actionable target, and it is the step most screen-capture tools skip.

Does spatial context work outside the browser?

Yes, when it is built on the operating system's accessibility layer rather than on a browser extension. macOS exposes the element under any point for every application, so native apps, IDEs, and Electron windows are all covered — with on-device OCR as the fallback for canvas-rendered surfaces that expose no tree.