A powerful tool that leverages Vision Language Models (VLMs) to enable AI-driven control of your Mac through visual understanding and contextual reasoning.

Automate your workflow with natural language commands and visual intelligence

The project now supports both Level 1 (GUI Agent) and Level 2 (Autonomous GUI Agent) capabilities:

• Level 1 (GUI Agent): Basic visual understanding and action capabilities
• Level 2 (Autonomous GUI Agent): Enhanced memory, planning, reasoning, and human-in-the-loop functionality

Community help is more than welcome! We're looking for contributors to help us enhance these capabilities further. Join us in building the future of computer automation.

Computer Use with MLX-VLM transforms how you interact with your Mac by combining the power of:

• MLX - Apple's machine learning framework optimized for Apple Silicon
• Vision Language Models (VLMs) - AI models that understand both visual and textual information
• Automation - Seamless execution of tasks across your Mac's interface

By processing screenshots and visual information from your screen, the system understands the current state of applications and executes appropriate actions to accomplish tasks you specify in natural language.

• Mac-Native Performance: Optimized for Apple Silicon with MLX for efficient, local processing
• Visual Understanding: Interprets screen content, UI elements, and application states
• Contextual Reasoning: Makes intelligent decisions based on visual context
• Cross-Application Automation: Works across multiple applications and system interfaces
• Natural Language Control: Simple, human-like instructions to control your computer
• Privacy-Focused: All processing happens locally on your device
• Customizable: Adapt to your specific workflow and preferences
• Autonomous Operation: Level 2 agent can plan and execute multi-step tasks with minimal supervision
• Voice Control: Hands-free operation with voice commands using local speech recognition

• macOS running on Apple Silicon (M series)
• Python 3.8+
• pip (Python package manager)

1. Install MLX-VLM package:

   pip install mlx-vlm

2. Clone the repository:

   git clone https://github.com/Blaizzy/mlx-vlm.git

3. Navigate to computer control directory:

   cd computer_use

4. Install dependencies:

   pip install -r requirements.txt

Launch the standard application with:

python main.py

For enhanced autonomous operation with planning capabilities:

python autonomous_gui_agent.py

This launches the Level 2 autonomous agent that can:

• Plan and execute multi-step tasks
• Maintain context across actions
• Make decisions based on visual feedback
• Request human assistance when needed

For hands-free operation, you can use the voice-enabled autonomous agent:

python autonomous_gui_agent_voice.py

This launches a voice-controlled version that:

• Listens for your voice commands using your Mac's microphone
• Converts speech to text using local speech recognition using mlx-whisper
• Processes your commands and executes them visually
• Provides audio feedback on actions taken

Voice commands work just like text commands, so you can say things like:

Control your Mac with natural language instructions like:

"Open Safari and navigate to apple.com"
"Open the notifications tab and click on the first notification"
"Open the email app and reply to the most recent email"

1. Screen Capture: The system takes screenshots of your Mac display
2. Visual Analysis: MLX-VLM processes the visual information to understand:
   • UI elements and their states
   • Text content on screen
   • Application context
   • System status
3. Instruction Processing: Your natural language commands are interpreted
4. Action Planning: The system determines the sequence of actions needed
5. Execution: Actions are performed through macOS APIs or simulated inputs (click, scroll, etc)

• Local Processing: All AI inference happens on your Mac using MLX
• No Cloud Dependency: Your screenshots and data never leave your device
• Permission Control: Fine-grained control over what the system can access
• Transparent Operation: Clear visibility into actions being performed

• Permission Errors: Make sure to grant screen recording permissions in System Preferences > Security & Privacy > Privacy
• Performance Issues: Try reducing the screenshot resolution in config.json
• Application Compatibility: Some applications with non-standard UI elements may have limited support

• Check the Issues page
• Join our Discord community

We welcome contributions! Here's how to get started:

1. Fork the repository
2. Create your feature branch:

   git checkout -b feature/amazing-feature

3. Make your changes
4. Run tests:

   python -m pytest

5. Commit your changes:

   git commit -m 'Add some amazing feature'

6. Push to the branch:

   git push origin feature/amazing-feature

7. Open a Pull Request

Please read CONTRIBUTING.md for detailed guidelines.

This project is licensed under the MIT License - see the LICENSE file for details.

• The MLX team at Apple for creating the MLX framework
• Our community of testers and contributors who help improve the project

Made with ❤️ for Mac users who love automation and AI