Computer Use

CLI-JAW supports browser DOM automation through CDP and desktop interaction through Computer Use. Computer Use is an MCP tool, not an employee.

Two Approaches

Approach	Method	Best For
CDP (Chrome DevTools Protocol)	Browser automation via `src/browser/`	Web pages, precise DOM interaction
Computer Use (CU)	Desktop automation via jaw-computer-use MCP or the Codex native plugin	Native apps, desktop UI interaction

CDP Browser Automation

CLI-JAW launches and controls Chrome through CDP:

# Start browser
jaw browser start

# Take a screenshot
jaw browser screenshot

# Take a DOM snapshot with ref attributes
jaw browser snapshot

# Click an element by ref
jaw browser click ref123

# Type text into an element
jaw browser type ref456 "Hello World" --submit

# Navigate to a URL
jaw browser navigate https://example.com

# Get page text
jaw browser text

Vision Click Pipeline

When neither DOM ref attributes nor direct coordinates are suitable, the vision click pipeline uses AI vision to locate and click UI elements:

jaw browser vision-click "Submit button" --provider codex

The pipeline: screenshot → AI vision extracts coordinates → DPR correction → click → verify.

An explicit $computer-use request uses the exposed Computer Use tool. Never substitute CDP. Jaw registers the jaw-computer-use server (tool js; mcp__jaw-computer-use__js in Claude) for MCP-aware CLIs except Codex. Codex uses its native plugin. Read the first tool call's documentation for cua.getApp(name), app.getAXState(), app.click(index|[x,y]), and other supported actions.

macOS TCC Permissions

Install and open the ChatGPT desktop app with Computer Use and restart jaw to register the MCP server. On macOS, the responsible jaw service process or binary may need permission to access data from other apps (TCC App Data for the ChatGPT Computer Use container). The exact Privacy & Security pane name varies by macOS version. A Terminal-launched jaw without this grant can block on the first app call. Check other TCC permissions with:

# Check TCC permissions
jaw doctor --tcc

Jaw answers app-approval prompts from the run permission policy: auto approves; safe declines and reports not approved. If no tool is exposed, report precondition failed: no Computer Use surface. If the first app call blocks, report precondition failed: computer-use app access blocked. On Windows, the same mechanism requires the Codex desktop plugin cache. Linux, WSL, and Docker have no host Computer Use surface; CDP remains available for DOM work.

Safari Tip

For Safari automation, enable "Allow Remote Automation" in Safari's Develop menu. However, Chrome via CDP is the recommended and best-supported path.

Browser API Endpoints

Method	Path	Description
POST	`/api/browser/start`	Start browser
POST	`/api/browser/stop`	Stop browser
GET	`/api/browser/status`	Browser status
GET	`/api/browser/snapshot`	DOM snapshot with refs
POST	`/api/browser/screenshot`	Take screenshot
POST	`/api/browser/act`	Click, type, press, hover
POST	`/api/browser/vision-click`	Vision-based click
POST	`/api/browser/navigate`	Navigate to URL

Runtime Diagnostics

# Check browser runtime health
jaw browser status

# Doctor check for orphan processes
GET /api/browser/doctor

# Cleanup orphan runtimes
POST /api/browser/cleanup-runtimes

Try it:

설정 앱 열어줘
이 버튼 클릭해줘
$computer-use 사파리 열어줘

Discord Electron Desktop