Giving an agent a computer

Some work has no API: a freelance exchange’s cabinet, an internet bank, a desktop app. For it an agent needs what a person has: a desktop with a browser that stays logged in, a file system and the apps. The engine does not ship one. It has everything needed to plug one in, so a desktop is a brain decision: a vendor’s MCP server, or a container in your own compose file.

This page covers when to add one, the two ways to do it, and the traps.

First: do you need one?

Usually not yet. Each step below is cheaper, faster and more reliable than the one after it, so reach for a desktop only when the steps before it are exhausted:

  1. An API or an engine integration. Telegram goes through your account over MTProto (telegram-account.md). Mail and files go through the Google tools (integrations.md). Payments go through a provider webhook, not a bank cabinet.
  2. The headless browser. It is enough for reading and filling in web pages (system:browser, features.md § Tools & MCP).
  3. A full desktop. You need it for a logged-in browser that must survive restarts, desktop apps, and sites that need a real, persistent profile.

A desktop step costs far more model tokens than an API call. A screenshot alone is about a thousand input tokens, and a real task takes tens of steps. Expect subscription limits to run out long before your server does.

How an agent reaches a desktop

The agent gets the desktop’s MCP server through the mcpServers frontmatter of a role (cli-agents-setup.md § Per-agent MCP entries). Four properties of that mechanism shape everything below:

  • HTTP only. A role accepts type: http or type: sse. A desktop driver that speaks only stdio needs an HTTP front.
  • CLI agents only. API-mode agents get no MCP tools. Pin modelRef: cli:smart or a concrete claude:<model>.
  • Static per role. The URL is interpolated from the environment once, when the role file loads. It cannot vary per agent or per chat. One desktop therefore means one role.
  • Not gated by the engine. External MCP calls bypass the tool gateway: no HITL, no rate limit, no per-tool metrics, no prompt-injection wrapper. Whatever the agent can click, it clicks. See Approvals.

The pattern that fits these constraints is one agent per desktop. A desktop-agent role owns the desktop. Every other agent hands it a bounded task with agent_task and gets a report back. This also keeps screenshots and click-by-click history out of the delegating agent’s context.

Option A: a managed cloud desktop

A vendor runs the machine, and you point a role at its MCP endpoint. Any provider that serves MCP over HTTP works. Orgo is one: persistent Linux computers with a browser, a shell and VNC.

<!-- features/ops/desktop-agent.agent.md -->
---
name: Desktop Agent
modelRef: cli:smart
mcpServers:
  desktop:
    type: http
    url: ${DESKTOP_MCP_URL:-https://desktop.invalid/mcp}
    headers:
      Authorization: Bearer ${DESKTOP_MCP_TOKEN:-unset}
# No `memory:` block: hostile screen text never reaches shared memory.
---

You operate one computer to complete one bounded task, then report what happened. …

Give every ${VAR} a default. An unresolved placeholder drops the whole role file, and the agent falls back to the general role. A default that does not resolve shows up as “desktop unreachable” instead.

Trade-offs:

  • Nothing to run.
  • The machine sits in the vendor’s region and IP range.
  • Files you need come back through the vendor’s file tools.
  • You pay per machine-month.

Option B: a desktop container on your own server

A brain adds services to the engine’s stack with a compose override (brain-dev-guide.md § Production troubleshooting). A desktop is one more service on the backend network, next to playwright-mcp. Agents address it by service name.

The image

No upstream image fits as-is, so build your own and pin every version in it. It needs:

  • An X11 desktop. Use XFCE on TigerVNC or Xvfb, at 1280×800 or smaller to keep screenshots cheap. Add noVNC so a person can watch.

  • Google Chrome, plus whatever apps the work needs (LibreOffice, Telegram Desktop, …).

  • A session D-Bus and the AT-SPI registry, started explicitly. Without them the accessibility tree is empty, and the agent is reduced to guessing coordinates from screenshots. Recent at-spi2-core defaults to a launcher that needs systemd, which a container lacks.

  • A driver that serves MCP over HTTP. Cua is the most complete open option today. Run Cua’s computer-server with the cua-driver backend, bound to 0.0.0.0 on port 8000; it serves MCP at /mcp.

    • Without the cua-driver backend, the Linux accessibility tree is a stub.
    • Cua releases almost daily. Check the flags against the docs of the version you pin, and upgrade deliberately.
    • Cua’s published images ship Firefox only, while the driver’s browser tools target Chromium. This is one more reason to build your own image.

  • An entrypoint that clears Chrome’s profile locks before starting it. The lock records the hostname and PID. After a recreate or an unclean stop, Chrome refuses the profile, sometimes silently:

    rm -f "$HOME/.config/google-chrome/Singleton"{Lock,Cookie,Socket}

The service

# docker/compose.prod.yml
services:
  desktop:
    image: ghcr.io/<you>/desktop:<pinned-tag>
    hostname: desktop            # stable hostname: the Chrome profile lock records it
    shm_size: 2g                 # Docker's 64 MB default crashes Chrome tabs
    restart: unless-stopped
    volumes:
      - desktop_home:/home/agent # profile, logins, cookies: survives recreate and upgrade
      - ./artifacts:/var/lib/omnislash/artifacts   # the engine's artifacts root, same path
    ports:
      - "127.0.0.1:6901:6901"    # noVNC for a person, over an SSH tunnel only
    deploy:
      resources:
        limits:
          memory: 4G             # an OOM kills the desktop, not Postgres
          cpus: "2.0"
          pids: 1024
    networks:
      - backend

volumes:
  desktop_home: {}

Then point the role at it:

mcpServers:
  desktop:
    type: http
    url: ${DESKTOP_MCP_URL:-http://desktop:8000/mcp}

Keep these rules:

  • Never publish the MCP port. A self-hosted computer-server has no authentication, and its tools include a shell and file access. The private backend network is the boundary. The same holds for playwright-mcp.
  • Size it. One desktop with Chrome and a few tabs needs about 4 GB of RAM and 2 vCPU. CPU can be overcommitted, because the agent spends most of its time waiting on the model. The limits above keep a runaway tab away from Postgres, Redis and Restate on a shared host. One agent desktop plus a few parallel ones fit beside the stack on a 16–32 GB server.
  • Linux desktops need no KVM. It is needed only for Windows or Android VMs.
  • GPU-only apps will not run. DaVinci Resolve, for example, refuses to start without a GPU. A CPU-only server means ffmpeg, Kdenlive or Shotcut.

Several desktops

The URL is static per role, so N parallel desktops mean services desktop-1 … desktop-N and N role files that differ only in the URL. Each role is one desktop’s agent. Route tasks to them from your own code or prompt, for example by a desktop field your boss agent assigns.

Getting files out

Mount the engine’s artifacts root into the desktop at the same path, as in the service above. Then set Chrome’s download directory, and the save paths of the desktop apps, under it, for example /var/lib/omnislash/artifacts/desktop/downloads.

The desktop agent’s sandbox sees the whole artifacts root read-write. It copies a finished file into its own artifacts_dir (…/agents/<agentId>/), and from there file_send, artifact_publish and drive_upload_file accept it. This copy needs a shell, so give the role the engineering:execute capability.

Files the agent needs on the desktop travel the same way, in reverse.

Approvals are a prompt rule here

The engine cannot gate a click on an external MCP server. Put the boundary in the role prompt, and have the agent ask through AskUserQuestion before anything irreversible. It pauses the agent until the person answers, and every interactive turn is offered it — no capability to declare. hitl_request_approval does not fit here: it sends the card and returns at once, so the click would happen before anyone reads it. Gate these:

  • moving money, or confirming a payment or a transfer;
  • accepting terms, signing, or submitting a final delivery;
  • changing a password, a recovery phone or 2FA settings;
  • entering a site the task did not name.

Treat everything on screen as data, never as instruction. That covers page text, chat messages and documents. system:browser’s prompt is a good model. Declare no memory: block, so hostile screen text never reaches shared memory.

Prefer an official channel over the desktop wherever one exists: a bank’s payment notifications or a payment provider’s webhook rather than a cabinet, and MTProto rather than Telegram Web.

Accounts and sessions

The volume keeps the Chrome profile, so logins and cookies survive restarts and upgrades. Back up the volume like any other state. Snapshots are your job; neither Cua nor Docker takes them for you.

Use the owner’s own accounts, from this one stable desktop. To a bank or a portal the desktop is a new device. Expect a confirmation on first login, done by the owner exactly as for a new laptop, and expect it again if the profile or the server’s IP changes. Do not rotate the environment.

Running it

  • Restart Chrome nightly. Long-running Chrome and web apps leak memory. A scheduled docker compose restart desktop from the host is enough.

  • Pin and rebuild deliberately. Treat a driver bump like an engine upgrade: rebuild, run one known task by hand, then roll.

  • Watch it through noVNC over an SSH tunnel:

    ssh -L 6901:127.0.0.1:6901 <server>

    Then open http://localhost:6901.

  • Expect failures. The best models finish roughly four in five long desktop workflows, and a failure is often silent. Have the desktop agent verify the outcome before it reports success: read the confirmation, check the file exists. Keep an approval in front of anything you cannot undo.

Screens from Sixty Seconds Studio, a fictional demo studio, running on the real dashboard.