Computer Use AI: What It Is & How It Works

Computer use AI means an AI agent can see your screen, read UI element names and positions, and execute clicks or keystrokes, not by parsing text logs, but by treating your interface like a human would. Anthropic shipped the first public beta in October 2024 (their Claude 3.5 Sonnet API), and local implementations followed immediately via open-source MCP servers.

Most takes conflate "screen recording" with "understanding", computer use AI is the layer that identifies the button under a point and returns its name, not a pixel guess.

I'll walk through what computer use AI does, how Anthropic's hosted version compares to local MCP server setups on macOS, and the accessibility-tree trick that makes the whole system deterministic. If you've tried to get an AI coding agent to edit the right component and it clicks the wrong element, this is the missing piece.

What is computer use AI and why does it exist?

Computer use AI solves the grounding problem: an agent reads a text instruction like "change the label on the Submit button to Send" and must resolve which on-screen pixel rectangle is the Submit button. Before computer use, agents guessed element selectors from DOM snapshots or asked you to paste a screenshot, slow, brittle, and wrong half the time.

With computer use, the agent sends a point (x, y) and receives back the named element at that location, role (AXButton), title (Submit), frame ({427, 389, 72, 32}), plus the full window's text and a cropped screenshot. The moat is the Accessibility API on macOS (or UIA on Windows): AXUIElementCopyElementAtPosition hands you the actual UI object under any coordinate, built into the OS.

No pixel OCR, no guessing.

Anthropic's computer-use beta runs in a Docker container with a VNC desktop; Claude screenshots the VM, parses the screen, and clicks.

It's a proof-of-concept for hosted agents.

Local MCP servers like PinVari's pinvari-mcp give the same capability to Claude Code and Cursor on your Mac, reading your real desktop's accessibility tree, with no Docker overhead and no API upload.

How does Anthropic's computer use AI work?

Anthropic's computer-use implementation (announced October 2024, currently beta) runs Claude in a headless Ubuntu Docker container with a virtual desktop. You POST a prompt to the Messages API with type: computer_20241022 in the tools array;

Claude responds with tool-use blocks requesting screenshots, mouse moves, or clicks.

The API returns:

• A base64 screenshot of the full desktop (1024×768 or your chosen resolution). - A coordinate tuple [x, y] for the next click.

• Optionally, a text string to type.

Your client (the script wrapping the API) executes the action in the Docker environment, screenshots again, and sends the new state back. Anthropic's reference implementation on GitHub packages this loop in Python, Docker Compose spins up the container, a FastAPI server accepts API responses, and xdotool injects the mouse/keyboard events.

Limits: it's a VM desktop, no access to your real Mac's windows, your IDE, or your browser's real DOM. You're teaching Claude to navigate a simulated Linux environment, not the tools you already have open.

Latency is one round-trip per action (POST → response → execute → screenshot → POST again). For agentic coding workflows where the agent edits files in VS Code or Cursor, this detour is a non-starter.

What is a local MCP server and how does it replace the hosted API?

An MCP server (Model Context Protocol) is a small program your AI editor (Claude Code, Cursor, Codex, Zed) connects to over a local socket. The editor sends tool-call requests; the server responds with structured data, screenshots, file contents, UI element metadata.

No API keys, no cloud upload, no Docker. The server runs at 127.0.0.1 on a high port and talks to your editor's agent process in real time.

For computer use, a local MCP server:

1. Listens for a tool call like pinvari_next_instruction.

1. Reads the macOS Accessibility tree at the point you marked (via AXUIElementCopyElementAtPosition).

1. Returns the element's name, role, frame, spoken instruction, and a cropped screenshot of the region you circled.

The editor's agent (Claude 3.5 Sonnet, GPT-4o, Gemini 2.0) receives a resolved instruction, not "click somewhere near the middle of the screen," but "change the AXButton titled 'Submit' at frame {427, 389, 72, 32} in Safari's toolbar." The agent writes the code/clicks the element, marks the capture done with pinvari_mark_done, and moves to the next task.

PinVari's connector installs at ~/.pinvari/mcp/pinvari-mcp. Connect it in one click inside the app (PinVari → Connect → Claude Code / Cursor / VS Code / Codex) or via CLI:

claude mcp add --scope user pinvari -- "$HOME/.pinvari/mcp/pinvari-mcp"

The app must be running (the connector talks to it on port 3402). Never write the bare claude mcp add pinvari, it errors because the tool requires the full path.

How does the macOS Accessibility API enable deterministic computer use?

macOS exposes every on-screen UI element via the Accessibility API, a C framework (ApplicationServices/HIServices.h) that lets any process query the UI tree. Call AXUIElementCopyElementAtPosition(systemWide, x, y, &element) and the OS walks the window list, hit-tests the topmost non-obscured window at (x, y), and returns the AXUIElement at that point.

Each element has attributes:

• AXRole (button, text field, group, web area, etc.)
• AXTitle or AXDescription (the label)
• AXValue (the current text/state)
• AXFrame (the {x, y, w, h} rectangle)
• AXParent (the containing element)

This is the same tree VoiceOver uses. The API is always available, no special build flags, no opt-in, just an entitlement and user permission (granted once in System Settings → Privacy & Security → Accessibility).

PinVari queries this tree every time you circle or point at something. The overlay is topmost, so a naive hit-test resolves to PinVari's own window, chainExcludingSelf walks the on-screen window list (CGWindowListCopyWindowInfo) and hit-tests the real app.

Electron/Chromium build their AX tree lazily; PinVari sets AXManualAccessibility on the target app and retries (~150ms) until a labeled element appears.

When a point lands on a bare AXGroup, labeledDescendant descends to the deepest labeled child.

Chromium identity attributes (AXDOMIdentifier, AXDOMClassList) give a title-less node a name.

You can inspect this tree yourself with the Accessibility Inspector (Xcode → Open Developer Tool → Accessibility Inspector). Hover over any UI element and you'll see the same role/title/frame PinVari reads.

Why this matters: a pixel-based tool sees 0xE8F7A3 at (430, 400) and guesses "maybe a button"; the Accessibility API returns AXButton title:"Submit" frame:{427,389,72,32}. One is a guess, the other is a named, executable reference the agent can act on immediately.

Computer use AI: Anthropic API vs. local MCP servers

The Anthropic API proves the concept works; local MCP servers prove it's faster and cheaper to run on your own machine. If you're coding in Cursor or Claude Code, connecting a local server means the agent sees your real IDE, browser, and terminal, not a simulated desktop.

FAQ

#

What is computer use AI?

Computer use AI is an AI agent's ability to see your screen, identify UI elements by name and position, and execute clicks or keystrokes, treating your interface like a human would. It replaces pixel-guessing with deterministic element resolution via the macOS Accessibility API or Windows UIA.

#

How does Anthropic's computer use API work?

Anthropic's computer-use beta runs Claude in a Docker Ubuntu container with a VNC desktop. You send a prompt with type: computer_20241022 in the tools array;

Claude screenshots the VM, returns [x, y] coordinates for the next click, and your client executes the action.

The reference implementation is on GitHub.

#

Can I use computer use AI with local models instead of Claude?

Yes, any MCP server (like PinVari's) is model-agnostic. If your editor (Cursor, Continue) supports local LLMs and the model handles function calling (Llama 3.1 70B, Qwen 2.5 Coder), it can call the same pinvari_next_instruction tool.

Performance lags frontier models for complex UI reasoning, but simple tasks work.

#

What is an MCP server and how does it differ from Anthropic's API?

An MCP server is a local program your AI editor connects to over a socket (usually 127.0.0.1). It exposes tools the agent can call (screenshots, UI element queries, file reads).

Unlike Anthropic's hosted API, an MCP server runs on your machine, sees your real desktop, and uploads nothing. Setup is one CLI command or one in-app click.

#

Does computer use AI work on Windows or Linux?

Anthropic's Docker demo runs on any platform that supports Docker. For native computer use, Windows has the UI Automation API (similar to macOS Accessibility), but as of August 2026 no production MCP server ships for Windows.

Linux has AT-SPI, theoretically queryable, but accessibility support varies by desktop environment. macOS has the most mature, universal Accessibility API.

#

How much does computer use AI cost?

Anthropic's API charges per token; screenshots are large (a 1024×768 PNG base64-encoded ~1-2MB → ~10K tokens), so expect $3-15 per thousand actions at current rates. Local MCP servers like PinVari cost a one-time $39 app purchase with no per-action fee, unlimited captures, on-device processing, no upload.

Start giving your agent named UI elements, not pixel guesses

Computer use AI is the difference between "click near the middle" and "click the AXButton titled 'Submit' at frame {427, 389, 72, 32}." Anthropic proved the concept with a hosted API; local MCP servers bring the same capability to your real Mac desktop, your real IDE, and your real browser, no Docker, no upload, sub-50ms latency.

If you're running Claude Code, Cursor, or another AI coding agent, the missing piece is a server that reads the Accessibility tree and hands back resolved, named elements. PinVari does that in one ⌥⌘A press-to-mark interaction: circle or point at anything, speak the instruction, and the agent receives the exact element path plus a cropped screenshot.

Connect it with one click and stop watching your agent click the wrong button.