AgentOS

AgentOS is a platform for agent-native control of Linux desktop environments.

At the center of AgentOS is the Agent Control Plane (ACP), a core system component that gives agents a structured, semantic, and permissioned interface to desktop applications. ACP extracts UI state from accessibility APIs, maintains a canonical UI graph, streams UI events, and exposes secure action APIs for agent runtimes.

• reads desktop state through accessibility and desktop metadata
• maintains a live, versioned UI graph
• lets agents query windows, elements, and subtrees deterministically
• executes typed actions like focus, invoke, and set_value
• streams structured events such as focus changes, dialogs, and state updates
• enforces permissions before every action
• records audit logs for replay, debugging, and governance

Vision-driven desktop automation is useful as a fallback, but it is not an ideal primary interface for agent control:

• it is slower
• it is less deterministic
• it is harder to secure
• it makes action confirmation ambiguous

AgentOS is built around a control-plane model:

query graph -> choose action -> execute with version check -> confirm via event

That makes the desktop feel more like an API surface than a pixel surface.

AgentOS is organized around three major layers:

1. Agent Runtime The planner, memory system, and LLM choose actions based on ACP queries and event streams.

2. ACP Daemon A Rust daemon that connects to AT-SPI2 over DBus, normalizes UI state, maintains the canonical graph, serves APIs, enforces policy, and writes audit logs.

3. Linux Desktop Environment GTK, Qt, Electron, browser, and terminal applications that ACP observes and controls through accessibility and app-specific adapters.

• Rust for the ACP daemon
• Tokio for concurrency
• AT-SPI2 over DBus for accessibility integration
• zbus and atspi crates for desktop integration
• gRPC for typed query and action APIs
• gRPC streaming or WebSocket for events
• SQLite for audit and policy persistence
• in-memory graph for active UI state

ACP maintains a canonical in-memory graph of active desktop elements. Every mutation increments a graph version. Agent actions must include the version they were planned against, so ACP can reject stale actions safely.

Agents act on elements such as windows, buttons, editors, and dialogs. The default control path is semantic:

• focus
• invoke
• set_value
• select
• expand

Raw coordinate injection is not the default path.

ACP streams events like:

• WINDOW_OPENED
• WINDOW_CLOSED
• ELEMENT_FOCUSED
• TEXT_CHANGED
• DIALOG_APPEARED
• MODAL_BLOCKING

This lets agents react to the live desktop without polling blindly.

Every action is evaluated against policy before execution. Every attempt is auditable. This is essential for secure agent control in developer, enterprise, or sandboxed environments.

Not every Linux app exposes equally good accessibility data.

• GTK and Qt apps are the strongest baseline targets.
• Browsers are workable but require robust diff handling.
• Electron apps like VSCode often need richer app-specific adapters.
• Terminals need shell-aware adapters for meaningful semantic control.
• Inaccessible or custom-drawn apps may require an explicitly gated fallback path.

ACP is designed to handle this with capability tiers and plugins, not by pretending every app is equally structured.

The implementation plan focuses on:

1. Rust ACP daemon scaffold
2. canonical schema and versioned graph engine
3. AT-SPI2 / DBus integration
4. gRPC query API
5. event streaming
6. action executor with version checks
7. policy engine and audit logging
8. app-specific adapters for Terminal, Browser, and VSCode

• ARCHITECTURE.md
• IMPLEMENTATION_PLAN.md

The next implementation step is to scaffold the Rust daemon and build the first live accessibility-backed UI graph.