The PinVari MCP server, documented
PinVari runs a local MCP server on your Mac (127.0.0.1 only) that gives any MCP agent — Claude Code, Cursor, VS Code, Codex, Zed — eyes on what you point at. Seven tools, one capture format, zero cloud. This page is the full reference for agents and the people who script them.
Connect
One click inside the app (PinVari → Connect), or from the terminal for Claude Code:
PinVari must be installed and running — the connector talks to the app on 127.0.0.1:3402. Setup for Cursor, VS Code, Codex, Zed and Claude Desktop: pinvari.com/connect.
The seven tools
pinvari_next_instructionread
The workhorse. Returns the oldest capture your agent hasn't handled yet, framed as an executable task: the resolved UI element (role, label, frame, confidence), the spoken instruction, the circled region and a screenshot cropped to it. Vague instructions arrive flagged UNSPECIFIC with an ask-never-guess rule. Poll it to act on what the user points at, hands-free. No parameters.
pinvari_mark_donewrite
Marks a capture-instruction as handled so pinvari_next_instruction won't return it again. Parameters: id (required).
pinvari_list_capturesread
Lists recent captures — app, URL, transcript, resolved element, classification. Parameters: limit (default 10), query (free-text filter over title, transcript, app, URL, element), type (bug | feature | question | note).
pinvari_get_captureread
Full detail of one capture: the cropped screenshot (or the full annotated shot with full_screenshot: true), every mark with its own words, the window's full text, dwell targets and deictic bindings. Parameters: id (required), full_screenshot.
pinvari_request_captureasks the user
The bidirectional tool: mid-task, your agent can ask the human to point. Tell the user in chat what to show, then call this — PinVari's notch island lights up with a point-at-it prompt, and the tool waits until the user makes a capture (⌥⌘A, circle, speak), then returns it with the screenshot. Parameters: timeout_seconds (default 120, max 600). If the user doesn't capture in time it returns instructions, never a stale answer.
pinvari_healthread
Service status: capture count, on-device resolution rate, queue depth, licence state. Use it to confirm the app is running before asking the user to point.
pinvari_get_framerefuses by design
Deliberately does not take a screenshot. PinVari never captures the screen unless the human holds the capture hotkey — an agent cannot silently look at your display. Agents that need to see something are redirected to pinvari_request_capture.
What a capture contains
One ⌥⌘A capture bundles everything an agent needs to act on the right thing — this is the spatial context payload:
| Field | What it is |
|---|---|
element | The resolved named UI element you circled: role, label, frame, parent path, a confidence score, and provenance — circled (you drew around it) or dwelled (you hovered ~0.2s). |
marks[] | Every region you circled, numbered, each with its own bounding box, its own slice of your words, and — if you scrolled between marks — its own screenshot frame. |
transcript | On-device transcription with per-word timing. Circle three things in one breath and each mark gets its own words. |
deixis[] | Each spoken word bound to the element under the pointer at the instant it was said — "make this blue" knows what this was. |
window.fullText | The full text of the focused window, up to 40,000 characters, including text scrolled out of view — the whole stack trace, not the three lines that fit in the shot. |
window.elements[] | An interactive-element map of the window: role, title, value and global frame for up to 160 elements. |
browser | The real page URL and tab title (Chrome, Safari, Arc, Brave, Edge and friends). |
screen_ocr[] | Optional whole-screen on-device OCR with word boxes — the fallback for canvas and other accessibility-blind surfaces. |
screenshot | An annotated, agent-sized JPEG plus the original retina PNG on disk. Multi-display aware: each mark's shot comes from the monitor it was drawn on. |
selected_text, notes | Whatever you had selected when you started, and anything you typed on the capture. |
Raw files live in ~/PinvariCaptures/<id>/ — shot.png, per-mark frames, and context.json with everything above. Any tool can read them; MCP is the convenient path, not the only one.
Security model
- Local only. The service binds 127.0.0.1 — nothing on your network can reach it, and nothing is uploaded anywhere.
- Consent-gated. The screen is captured only while the human holds the capture hotkey. There is no tool that lets an agent take a screenshot on its own;
pinvari_get_frameexists specifically to refuse. - Bring your own agent. PinVari ships no cloud LLM and needs no API keys or account. Transcription and OCR run on the Mac.