Computer Use AI GitHub: Open Source Projects 2024

Computer use AI GitHub repositories are MCP servers and agent frameworks that let coding assistants control your desktop—take screenshots, read UI trees, click buttons, type text—over a local connection. The main projects are Anthropic's official anthropic/anthropic-quickstarts (VNC/Docker reference), ModelContextProtocol's modelcontextprotocol/servers collection (filesystem, Git, Postgres connectors), and macOS-native tools like PinVari's MCP server that read the Accessibility tree and resolve the exact UI element you point at. Most developers search "computer use AI GitHub" expecting a drop-in library; what they find is a toolkit pattern—you run a local server exposing desktop primitives (screenshot, element query, window list), and your agent (Claude Code, Cursor, Codex) calls those primitives to execute tasks.

What is computer use AI and why GitHub?

Computer use AI means an LLM-powered agent controlling a graphical desktop instead of just writing code in a terminal. GitHub hosts the open-source MCP servers (the connectors between the agent and your OS) and the agent frameworks that orchestrate them. The Model Context Protocol (MCP) is Anthropic's standard: a server exposes tools and resources over stdio or SSE, the agent calls tools (screenshot, click, read_element), and the server returns structured data. MCP servers run locally—nothing uploads unless you explicitly connect a cloud LLM.

Why this matters: traditional AI coding agents see only your editor and filesystem. Computer use AI lets them read the UI of your running app, see error dialogs, verify a button landed where expected, and file a bug report with the exact element that failed. The bottleneck is UI resolution—knowing that "this button" the user circled is role=AXButton label="Save" frame={x:420,y:180,w:64,h:28} in the app's Accessibility tree, not a pixel guess.

Check Claude Code MCP for a full walkthrough of installing and configuring MCP servers inside Claude Code.

How do computer use AI open source projects differ?

Most GitHub repos fall into three buckets:

Anthropic's official computer-use-demo (in anthropic-quickstarts) is a VNC-based proof-of-concept. It spins up a Linux desktop in Docker, streams the framebuffer, and the agent clicks pixel coordinates. Great for demos; impractical for a Mac developer running Xcode, Figma, or Chromium—it doesn't see what is under the click, just that it clicked (420, 180). When the UI shifts by 10 pixels, the agent misses. No element names, no confidence scores, no recovery.

MCP server collections (modelcontextprotocol/servers) provide filesystem, database, and web-search primitives. The filesystem server reads/writes files; git wraps git commands; postgres runs queries. These are data connectors, not UI automation. They work on macOS but don't touch the Accessibility tree—your agent can't "click the Save button" because there's no button primitive.

macOS-native Accessibility tools (PinVari) solve the element-resolution problem. macOS exposes every on-screen UI element via AXUIElement (role, label, value, frame, parent chain). PinVari's MCP server wraps AXUIElementCopyElementAtPosition, resolves the element under a point, and returns the named path (window > toolbar > AXButton "Save") plus a confidence score. The agent receives an executable instruction like "rename AXButton 'Save' label to 'Export'" tied to the real element, not a pixel guess. Connect with one click (PinVari → Connect → Claude Code / Cursor) or CLI: claude mcp add --scope user pinvari -- "$HOME/.pinvari/mcp/pinvari-mcp".

The difference: Anthropic's demo shows what's possible; MCP servers show what's composable; macOS-native tools show what's reliable—an agent fixing the wrong element is worse than no automation.

What does computer use AI Anthropic mean on GitHub?

Computer use AI Anthropic refers to repositories built on Anthropic's computer use API (the Messages API tool type computer_20241022) and the MCP standard. Anthropic's anthropic-quickstarts/computer-use-demo is the canonical example: a Python script that streams a desktop over VNC, feeds screenshots to Claude 3.5 Sonnet, and executes the model's tool calls (screenshot, mouse_move, click, type). It's open-source (MIT) but Docker-dependent and coordinate-based.

Key repos:

• anthropic/anthropic-quickstarts — the official computer-use demo (VNC + Docker + Claude API). Run docker compose up and open the Streamlit UI; the agent sees a Linux desktop and can browse the web, open apps, click buttons. Limitation: pixel coordinates break when the UI reflows; no named elements.
• anthropic/anthropic-sdk-python — the Python client with computer tool support. If you're writing your own MCP server, you'll wrap this SDK and expose your own tools.
• modelcontextprotocol/python-sdk — the MCP SDK for building servers. Defines the stdio/SSE transport, tool/resource schemas, and prompts protocol. Not specific to computer use, but every macOS MCP server uses it.

Anthropic doesn't maintain a "computer use AI SDK" repo—the SDK is the MCP standard itself. Third-party developers build servers (PinVari, others) that implement MCP and expose OS-specific primitives (Accessibility tree, window management, keystrokes). The agent (Claude Code, Cursor, Zed) connects to those servers and calls their tools.

For a practical comparison of agents that support MCP, see AI coding agents.

What are computer use agent local AI models and how do they run on macOS?

Computer use agent local AI models are LLMs running entirely on your Mac (no API calls) that can invoke MCP tools. The workflow: the model (e.g., Llama 3.3 70B via Ollama, Qwen2.5-Coder via LM Studio) runs inference on Apple Silicon, the agent framework (Aider, Continue, or a custom script) feeds the model's tool calls to your MCP servers (PinVari, filesystem, Git), and the servers execute the actions locally. Zero telemetry, no rate limits, full control.

Practical stack on macOS:

1. Model runtime: Ollama (brew install ollama), LM Studio, or MLX. Ollama's llama3.3:70b fits in 48GB unified memory and supports tool calling.
2. Agent framework: Aider (pip install aider-chat), Continue (VS Code extension), or a custom Python script using anthropic-sdk-python pointed at Ollama's OpenAI-compatible endpoint (http://localhost:11434/v1).
3. MCP servers: PinVari MCP for UI automation, filesystem and git from modelcontextprotocol/servers for code edits.
4. Connection: Aider auto-discovers MCP servers in ~/.config/aider/mcp/ (symlink ~/.pinvari/mcp/pinvari-mcp there). Continue reads ~/.continue/config.json → mcpServers.

The local-model bottleneck is tool-call reliability. Llama 3.3 and Qwen2.5 both support function calling, but they hallucinate tool schemas more than Claude 3.7 Sonnet. You'll get pinvari_next_instruction(region="top-left quadrant") when the schema expects {mark_id: "abc123"}. The workaround: constrained decoding (Ollama's grammar parameter or Outlines library) or a validation layer that retries malformed calls. For production work, most teams run Claude/GPT-4 via API and save local models for draft commits and low-stakes edits.

For a deep dive on running local models with coding agents, see best local AI for Mac.

Step-by-step: connecting PinVari's MCP server to Claude Code on macOS

Here's the concrete example most "computer use AI GitHub" searches want—a working localhost MCP setup in under 5 minutes:

1. Install PinVari (macOS 14+, Apple Silicon or Intel). Download the DMG from pinvari.com, drag to Applications, open, grant Accessibility permission (System Settings → Privacy & Security → Accessibility → toggle PinVari).

1. Install Claude Code (brew install --cask claude-code or download from Anthropic). Sign in, verify the CLI is in PATH: claude --version.

1. Connect the MCP server (GUI method): Open PinVari → menu bar icon → Connect → Claude Code. PinVari writes the config to ~/Library/Application Support/Claude/claude_desktop_config.json and restarts Claude. Alternatively, CLI method:

   claude mcp add --scope user pinvari -- "$HOME/.pinvari/mcp/pinvari-mcp"
   claude mcp refresh

Verify: claude mcp list shows pinvari with status running.

1. Trigger a capture: Press ⌥⌘A (Option-Command-A), circle a UI element (a button, a text field, a toolbar item), and speak an instruction like "change this button's label to Export". PinVari transcribes on-device, resolves the AXButton under your circle (role, label, frame, confidence score), and files the instruction in the notch island (the lime HUD at the top of your screen).

1. Agent picks it up: In Claude Code, type /tools → select pinvari_next_instruction. The agent receives:

   {
     "instruction": "change this button's label to Export",
     "element_path": "window > toolbar > AXButton 'Save'",
     "confidence": 0.94,
     "screenshot_base64": "<cropped image of the button>",
     "provenance": "circled"
   }

The agent reads the screenshot, identifies the button in your SwiftUI/React code, and edits the label. Reply with pinvari_mark_done to close the task.

1. Test with a second mark: Press ⌥⌘A again, circle a different element (e.g., a slider), speak "move this to 75%". PinVari queues both marks. The agent calls pinvari_next_instruction twice—once per mark—and handles them sequentially or in parallel (your prompt controls the order).

The failure mode you'll hit: circling a blank canvas or an image (no AX tree) returns confidence: 0.0. PinVari falls back to on-device Vision OCR and returns the OCR text + bounding boxes, but the agent can't click a coordinate—it can only describe what it sees. For clickable automation, you need AX-aware UIs (SwiftUI, AppKit, Electron with AXManualAccessibility).

For element-debugging tips, see Accessibility tree and Accessibility Inspector Mac.

How do you compare computer use AI open source projects?

When evaluating a GitHub repo, check:

• Transport: stdio (launches a subprocess, agent controls lifetime) or SSE (long-running server, agent connects over HTTP). Stdio is simpler; SSE supports multiple agents sharing one server.
• Element resolution: pixel coordinates (fragile), OCR bounding boxes (better, no click), or Accessibility tree (named elements, click/type/query). macOS AX is AXUIElement; Windows is UIA; Linux is AT-SPI.
• On-device vs cloud: does it upload screenshots/audio? PinVari transcribes and OCRs locally (Apple Speech + Vision frameworks); Anthropic's demo uploads to Claude API.
• Agent compatibility: does it publish an MCP config schema? PinVari auto-configures Claude Code, Cursor, VS Code (Codex), and Zed. Others require manual JSON edits.
• Provenance: does it track how an instruction was captured (circled vs dwelled vs typed)? Low-confidence marks need human review; high-confidence marks can auto-execute.

The test: can it fix a typo in a running app's UI without restarting the agent? Coordinate-based tools break when a window resizes. Named-element tools (AXButton "Submit" → role, frame, parent) stay valid through scroll, reflow, and light/dark mode.

What should you build next with computer use AI?

If you're starting a computer-use project on macOS:

1. Pick one task—e.g., "file a Linear issue from a screenshot + voice note" or "rename all buttons in a Figma file with consistent casing." Don't build a general-purpose agent; build a one-shot workflow.
2. Use an existing MCP server (PinVari for UI, filesystem for code edits, git for commits). Write a prompt template that chains the tools: pinvari_next_instruction → extract button label → filesystem_write_file → git_commit.
3. Run it 10 times and log failures. Most errors are schema mismatches (the agent invents a parameter) or confidence gaps (circled a logo, got 0.3 confidence). Fix the prompt, not the server.
4. Extract the pattern. Once your 10-task batch succeeds, turn it into a Cursor rule or an Aider command (/pinvari-to-linear, /fix-ui-typo). Share the rule in your team's cursor-rules repo.

The real win isn't "an agent that does everything"—it's a library of reliable, one-click workflows for the 5–10 tasks your team does daily. Computer use AI is the plumbing; your workflow prompt is the product.

For workflow templates, see AI agent workflow and AI agent workflow diagram.

If you want PinVari's point-and-speak MCP server without writing any setup code, grab it at pinvari.com/#pricing—$39 one-time, connects to Claude Code / Cursor / Codex / Zed in one click, no subscription.

FAQ

What is the difference between computer use AI and MCP servers?

Computer use AI is the capability (an agent controlling a desktop). MCP servers are the implementation (the localhost tools that expose desktop primitives). Think of MCP as the "driver" and computer use as the "car"—you need both, but MCP is the part you install and configure.

Can I use computer use AI GitHub repos with Cursor or only Claude Code?

Most MCP servers (including PinVari) work with any agent that implements MCP—Claude Code, Cursor (via ~/.cursor/config.json), VS Code with Codex, and Zed. The server doesn't care which client connects. Anthropic's official demo is Claude-API-only, but the modelcontextprotocol/servers collection and PinVari support all MCP-compatible agents.

Do computer use agent local AI models need an API key?

No. If you're running Ollama or LM Studio locally, the model runs entirely on your Mac. The agent framework (Aider, Continue) points its base_url at http://localhost:11434/v1 (Ollama) or http://localhost:1234/v1 (LM Studio), and tool calls stay on 127.0.0.1. Zero OpenAI/Anthropic charges. The tradeoff: local models hallucinate tool schemas more than cloud models, so you'll retry calls manually.

What happens if PinVari's confidence score is below 0.8?

PinVari returns the mark with confidence: 0.xx and needs_review: true. The agent receives the screenshot and OCR text but won't auto-execute unless your prompt explicitly allows low-confidence actions. Typical causes: circled a canvas element (no AX tree), circled mid-animation, or the spoken instruction was ambiguous ("this thing"). Review the screenshot, re-circle with a tighter selection, or add a voice clarification ("the Save button, not the icon").

Can I run computer use AI on Windows or Linux?

The MCP protocol is cross-platform, but most GitHub repos target macOS (Accessibility API) or Linux (AT-SPI). Anthropic's official demo runs in Docker (Ubuntu + VNC) and works on any host OS. For Windows-native automation, you'd build an MCP server wrapping UIA (UI Automation) instead of AX—no popular open-source server exists yet (as of Aug 2026). PinVari is macOS-only (14+, Silicon or Intel).

How do I debug an MCP server that won't connect?

Run claude mcp list (or cursor mcp status if Cursor exposes it). If the server shows error, check: (1) Is the connector path valid? (ls -l "$HOME/.pinvari/mcp/pinvari-mcp" should exist and be executable.) (2) Is the parent app running? (PinVari must be open; the connector talks to it on 127.0.0.1:3402.) (3) Are stdio/SSE transports configured correctly? (stdio needs args, env; sse needs url.) The MCP inspector (npx @modelcontextprotocol/inspector stdio "$HOME/.pinvari/mcp/pinvari-mcp") will show JSON-RPC errors in real time.