Clippy for macOS (on steroids)

The paperclip grew up. It can see your screen now — and click things.

Ask it to do something on your Mac and it takes a screenshot, reads the accessibility tree of whatever's in front of you, and does it — instead of describing the steps back to you and leaving you to click.

_{Clippy, live on the desktop, working out on making itself}

Most "AI desktop assistant" demos stop at chat: you ask, it answers, and you still do the clicking. Clippy closes that last gap. It's the difference between:

"To turn on Dark Mode, open System Settings, click Appearance in the sidebar, then choose Dark."

and Clippy just doing it while you watch the checklist tick over.

It lives as a small floating paperclip on your desktop: short answers come back in the speech balloon, and the full chat window opens underneath it for longer conversations, spoken replies, dictation, and file attachments. The classic assistant experience, minus the 90s bugs.

1. Download the latest DMG — one universal build, Apple Silicon and Intel.
2. Open it, drag Clippy to Applications.
3. First launch: right-click ▸ Open (the build is ad-hoc signed, so Gatekeeper warns once).
4. Grant Accessibility, Screen Recording, Microphone, and Input Monitoring when asked. Without them Clippy still chats, but it cannot see the screen, insert text, or hear global push-to-talk shortcuts.

Then pick how it thinks, in Settings: your existing Claude Code or Codex login (no extra key needed), your own OpenAI / Anthropic API key (stored in the macOS Keychain), or a Local Model — any OpenAI-compatible server on your Mac, like Ollama or LM Studio. Nothing leaves your machine on that path.

Every screen request runs the same loop, one step at a time:

 look  ──▶  plan one step  ──▶  do it  ──▶  look again  ──▶  done?
   ▲                                                          │
   └──────────────  no: what changed? re-plan  ◀──────────────┘

It never builds a ten-step plan up front and runs it blind — a real screen rarely matches a prediction, and looking again after every action is how Clippy notices.

The paperclip is a nod to Microsoft's old Office Assistant. Everything else — the screen reasoning, the safety checks, the retry logic — is original.

It's a solo side project, rebuilt in public commits as bugs turned up from using it day to day. Most of the interesting work is in getting the automation to fail safely: a step that can't find its target should stop and ask, not click something at random.

• Every step is validated before it runs: Send, Submit, publish, buy, pay, delete, accept or agree, sign out, and password or passcode controls are always refused. This check is unconditional and independent of everything else below.
• Return is never sent by Clippy, with exactly one exception: submitting a browser's own address or search field (navigating to a URL or running a search). Every other field gets typed into but never submitted; you press Enter yourself.
• Secure and password fields are refused outright, regardless of what's asked.
• Scrolling is the one action that runs without confirmation: it only changes what's visible, commits nothing, and destroys nothing. It still refuses to scroll a restricted app, since that's still synthetic input aimed at one.
• By default, this build's terminal and agent-CLI restriction is off (gui-restricted-app, enabled: false): Clippy can click and type into Terminal, Warp, iTerm, and similar apps, on the reasoning that everything runs on your own machine and Return still can't be sent there. Flip that one flag to restore it.
• Nothing runs without both Accessibility and Screen & System Audio Recording permission granted in System Settings. Clippy prompts for both the first time it needs them.

Every refusal above comes from Resources/prismor-policy.yaml — a Prismor Warden policy file. AutomationSafety.swift holds no rule text any more; it asks the policy and reports the answer. So changing what Clippy will and won't touch is a diff to a YAML file, reviewable on its own, with no rebuild of anyone's mental model of the Swift.

- id: gui-final-action
  severity: CRITICAL
  category: final_action
  title: Control performs a consequential, non-reversible action
  event_types: [ui_action]
  fields: [control_label]
  mode: enforce
  patterns: ['\bsend\b', '\bsubmit\b', '\bbuy\b', '\bdelete\b', ...]

After editing, recompile the form the app loads (CI fails the build if you forget):

./scripts/compile-policy.sh

It fails closed. A policy that's missing, corrupt, or contains a regex that won't compile does not degrade into "allow" — Clippy treats every control as a final action and stops touching the screen entirely. A guardrail that quietly stops guarding is worse than one that visibly stops working.

Prefer not to trust a stranger's DMG? Fair. Requirements: macOS 14+, Xcode command-line tools, and Swift 5.10+.

chmod +x scripts/build-dmg.sh
./scripts/build-dmg.sh

The output is written to build/Clippy-macOS-universal.dmg. The included app runs natively on both Apple Silicon and Intel Macs.

Local builds automatically use the first Apple Development code-signing identity in the current keychain. This keeps macOS Accessibility permission stable across rebuilds. If no development identity is available, the script falls back to ad-hoc signing.

For public distribution without Gatekeeper warnings, set a Developer ID identity before building:

CODESIGN_IDENTITY="Developer ID Application: Your Name (TEAMID)" ./scripts/build-dmg.sh

Then notarize the resulting DMG with your Apple Developer credentials and staple the ticket.

Published releases are built the same way by .github/workflows/release.yml, which runs the tests and the offline safety eval first and refuses to ship a DMG that lost either architecture slice.

Run the checks yourself with:

swift test                    # 340 unit tests
swift run clippy-eval --offline   # replays recorded screens; safety regression gate

Install Claude Code and authenticate once in Terminal. Clippy finds common Homebrew, npm, and user-local install locations and runs claude -p.

Install Codex and authenticate once in Terminal. Clippy runs codex exec with network and file mutation disabled by the read-only sandbox.

Select OpenAI API or Anthropic API in Settings, enter a model and API key, then click Save key. Keys never enter app preferences; they are stored in the user's macOS Keychain.

• Slash commands. /clear clears the chat and starts a new session, /help lists everything. Save prompts you use often as your own commands in Settings ▸ Custom prompts — anything typed after the command is appended, so /review this function works as well as a bare /review. Typing / shows the matching commands above the composer.

• Custom voice actions. Give any custom prompt its own push-to-talk shortcut. Hold it in any editable field, speak the request, and Clippy uses a lightweight, history-free model call to transform the transcript and insert the result back into the same field. It never presses Return, so generated shell commands remain reviewable before they run. A /command terminal action is included by default on ⌥⌃Space.

• An editable balloon menu. The floating paperclip's options can be renamed, reordered, or hidden in Settings ▸ Balloon menu, and any custom prompt can be surfaced there as a one-click row. Each built-in row keeps its behaviour — only the wording is yours.

• Follows your active display. On a multi-monitor setup Clippy moves to whichever screen holds the app you just switched to, keeping its relative position, and recovers itself if the display it was on is unplugged.

• Spoken replies and microphone dictation, with three engines to choose between — Apple's built-in recogniser (default, nothing to download, streams as you speak), local Whisper, or local Parakeet v3, which detects the language itself and is the fastest of the three on the Neural Engine. Both local engines transcribe entirely on this Mac. Offered once at first launch and switchable any time in Settings.

  The local engines record the whole hold and transcribe once you let go — the pipeline shape Handy uses — so nothing gets clipped off the end of a sentence, and a Silero VAD drops silence before the model ever sees it, which is what stops Whisper hallucinating a "Thank you." into a pause. Local models come from FluidAudio and WhisperKit.

• Push-to-talk dictation — hold a chord to speak, release to stop. The shortcuts are customizable in Settings, with two built-in destinations:

  • Hold ⌘⌥ to type into whatever editable field you had focused. If there's nowhere safe to put it — no editable target, or a password field, which is always refused — it lands on your clipboard instead and Clippy says so. Held while Clippy itself is frontmost, it fills Clippy's composer rather than another app.
  • Hold ⌥⇧ to send what you said to Clippy as a chat message.

  Both work from any app (needs Accessibility; otherwise only while Clippy is frontmost). Pressing any key mid-hold cancels the recording, so the chords stay usable as ordinary typing modifiers. Edit ▸ Toggle Dictation is the hands-free alternative.

• Classic Office-style balloon above the floating character, with an expandable full chat window

• Live provider-aware work stages, elapsed time, and accessible status announcements

• Compact action center for opening trusted apps and common folders

• Local Codex and Claude session discovery with explicit resume-in-Terminal actions

• Contextual answer actions for copying results and opening cited links

• True request cancellation for API calls and local Claude Code/Codex processes

• Compact-balloon attachments for images, PDFs, and text/code files: paste images or file URLs directly with Command-V

• Persistent conversation history

• Optional always-on-top window

Voice actions turn one custom prompt into a hold-to-talk tool that works from another app. Clippy remembers the editable field that was focused when the shortcut began, transcribes the recording after release, runs a small history-free model request, and inserts the finished text back into that same field. Changing windows while it processes does not redirect the result.

Clippy includes this action on first launch and adds it once when upgrading an existing installation:

Try it by focusing a Terminal prompt, holding ⌥⌃Space for half a second, saying “pull the latest git changes,” and releasing the keys. The expected insertion is git pull. Common speech-recognition variants such as “get pull” and “good pull” are corrected in terminal context.

To create another action:

1. Open Settings ▸ Custom prompts and select Add prompt.
2. Give it a slash-command name and the transformation prompt—for example, reply and “Rewrite the spoken text as a concise, friendly reply.”
3. Select Record, hold the desired key or combination for half a second, then release it. Clippy refuses combinations already assigned elsewhere.
4. Focus an editable field in any app, hold the new shortcut, speak, and release to insert the transformed result.

The two built-in dictation routes are customizable in Settings ▸ Dictation:

• ⌘⌥ transcribes directly into the focused app.
• ⌥⇧ sends the transcript to Clippy as a chat message.

All three shortcut types work globally when Input Monitoring is enabled and fall back to Clippy-only handling without it. A shortcut must be held for half a second before recording begins, preventing ordinary modifier combinations from opening the microphone accidentally.

Generated terminal commands are treated more strictly than general text: Clippy requests command-only output, extracts commands from fenced or verbose model responses, and inserts nothing if it cannot identify a usable command. It never presses Return, never types into a secure field, and never executes the generated command for you.

PRs welcome — see CONTRIBUTING.md for setup, the two checks CI runs, and what changing a guardrail involves. If you're using Claude Code or a similar agent, it can read this whole codebase in one pass; just point it at what you want changed.

Found a way to make Clippy operate a control it should refuse? That's the bug class this project cares most about — report it privately via SECURITY.md.

Release notes are in CHANGELOG.md.