Computer Use lets you describe a task in an NVDA dialog and have Computer Use tool operate the foreground Windows application.
Use at your own risk. This add-on uses an AI model to control your computer's mouse and keyboard. The author is not responsible for any irreversible actions, data loss, or other damages this add-on may perform or cause. Please use this tool responsibly and always monitor the add-on while it is running.
- Open NVDA Settings.
- Select the Computer Use category.
- Enter URL, API key, and model to use via OpenAI compatible Chat Completions API.
- Press
NVDA+Shift+Control+Uto open the task dialog. Enter a task, choose Perform.
The add-on captures the active window, sends the screenshot to the configured model, and follows the model's mouse and keyboard actions until the task is complete, cancelled, or an error occurs.
The same shortcut is used to abort while a task is running.
To build the add-on using uv:
- Install uv.
- Run the build command:
uv run scons
The resulting .nvda-addon file is created in the project root.
The OpenAI Python package and its dependencies are bundled in the add-on under addon/globalPlugins/ComputerUse/lib.
A standalone UIChallenge is provided in the for develop purpose only. This allows testing "Computer Use" logic.
uv run UIChallenge.py