Capabilities
Computer Use
Also known as: computer-using agent, GUI agent
An AI that can operate a computer the way a person does, looking at the screen and clicking, typing and navigating apps to complete tasks.
Computer use is AI that can actually drive a computer, seeing the screen and moving the mouse, clicking buttons and typing, just like you would. Instead of connecting to each app through special code, it uses the same buttons and menus a human uses. That makes it very flexible (it can use almost any software), but also slower and more error-prone than purpose-built connections.
Computer use lets an AI agent operate software through its graphical interface: it takes a screenshot, "sees" the screen (using multimodal vision), and issues actions like move-cursor, click and type. This is a general alternative to connecting via APIs or tools.
Why it is powerful, and tricky
- Flexible: it can use apps and websites that offer no API, doing whatever a person could.
- General: one capability, many applications.
- But: it is slower, can misread the screen, and, because it can click anything, needs careful guardrails and human oversight to avoid mistakes or being tricked (prompt injection via on-screen content).
It is an early but fast-moving capability, pointing toward agents that can handle general digital work.
Computer use equips an agent with a perception-action loop over a graphical UI: multimodal perception of screenshots (and accessibility trees) grounded to coordinates, and low-level actions (click, type, scroll, keypress), enabling operation of arbitrary software without bespoke integrations (Anthropic 2024). It complements API/tool-use by covering the long tail of apps lacking programmatic interfaces.
The hard problems are reliability and safety. Visual grounding and precise UI element localisation are error-prone; trajectories are long and errors compound; latency is high (screenshot-reason-act per step). Security is acute: on-screen or web content can carry prompt-injection payloads, and the agent's broad action space enables real harm, so sandboxing, least-privilege, action confirmation and human-in-the-loop checkpoints are essential. Evaluation uses interactive benchmarks over real applications; current systems are capable but not yet robust, making bounded, supervised deployment the norm.