An Autonomous General-Intelligence Agent That Controls Your Android Phone
Speak or type a goal β OmniClaw reasons, plans, and executes it on your phone β hands-free.
OmniClaw is an autonomous AI agent that takes a natural-language goal β spoken or typed β and executes it on a connected Android device. It combines LLM reasoning (Llama 3.1 70B via NVIDIA NIM), Android Debug Bridge control, voice input (Whisper), and a real-time web dashboard into a single system that can:
- π§ Draft and send emails with real-time web research
- π Make phone calls and send WhatsApp/SMS messages
- β° Set alarms, timers, and calendar events
- π Open URLs and search the web
- π± Navigate any Android app through UI automation
- π Find, install, and launch apps dynamically
- π Locate local files and push them to the phone
Think Siri + Jarvis β but open-source, running locally, and powered by a 70B-parameter LLM.
| Voice Command | What Happens |
|---|---|
| "Email alex@gmail.com about the latest SpaceX launch" | Searches the web β drafts email with real data β opens Gmail with fields pre-filled |
| "Set an alarm for 7:30 AM" | Fast-path: instant intent dispatch, no LLM needed (< 1 second) |
| "Send 100rs to Yeswanth on FamPay" | Launches FamPay β navigates UI β finds contact β enters amount |
| "Open the calculator and compute 500 Γ 2" | Dynamically finds calculator package β launches β taps buttons β reads result |
| Tier | Speed | When |
|---|---|---|
| Fast-Path | < 1s | Alarms, calls, SMS, browser, calendar β bypasses LLM entirely |
| Intent Dispatch | ~2s | Gmail, WhatsApp, dialer, timer β fires raw Android intents via ADB |
| LLM + UI Automation | 10β60s | Complex tasks requiring multi-step reasoning and screen interaction |
- Fast-Path Router β intercepts known patterns before the LLM even runs
- Intent Interceptor β code-level guardrail that corrects the LLM if it tries to launch apps that have dedicated intents
- Action Blocker β blocks the inner LLM from executing forbidden actions (launch, call, open_url)
- State-Hash Anti-Loop β detects unchanged screens and prevents infinite loops
- Action Deduplication β prevents repeated identical actions
- Voice β local Whisper transcription (base.en model, CPU, int8)
- Text β CLI or web UI
graph TB
subgraph Input
V["ποΈ Voice Engine<br/><small>Whisper STT</small>"]
T["β¨οΈ Text Input<br/><small>CLI / Web UI</small>"]
end
subgraph Core["π§ Core Intelligence"]
FP["β‘ Fast-Path Router<br/><small>Regex pattern matching</small>"]
O["π¦ Orchestrator<br/><small>ReAct Loop</small>"]
LLM["π€ Llama 3.1 70B<br/><small>NVIDIA NIM API</small>"]
INT["π‘οΈ Intent Interceptor<br/><small>Guardrail Layer</small>"]
end
subgraph Tools["π§ Tool Registry"]
AID["π± Intent Dispatcher<br/><small>Gmail, WhatsApp, SMS,<br/>Alarm, Timer, Calendar</small>"]
UI["π±οΈ UI Automation<br/><small>Tap, Type, Navigate</small>"]
APP["π¦ App Launcher<br/><small>Dynamic Package Search</small>"]
WEB["π Web Search<br/><small>DuckDuckGo</small>"]
FS["π File Tools<br/><small>Search, Push, Install</small>"]
HW["β¨οΈ Hardware Keys<br/><small>Back, Home, Enter</small>"]
end
subgraph Device["π² Android Device"]
ADB["ADB Bridge"]
PHONE["Phone Screen"]
end
V --> FP
T --> FP
FP -->|"Known pattern"| AID
FP -->|"Complex task"| O
O <-->|"Reason β Act"| LLM
O --> INT
INT --> AID
INT --> UI
INT --> APP
INT --> WEB
INT --> FS
INT --> HW
AID --> ADB
UI --> ADB
APP --> ADB
HW --> ADB
ADB --> PHONE
style FP fill:#ff6b35,color:#fff
style O fill:#6c5ce7,color:#fff
style LLM fill:#0984e3,color:#fff
style INT fill:#d63031,color:#fff
The orchestrator follows a Reason + Act cycle until the goal is achieved or the iteration limit is reached:
flowchart LR
A["π― User Goal"] --> B{"β‘ Fast-Path?"}
B -->|Yes| C["π Intent Dispatch"]
B -->|No| D["π§ LLM Thinks"]
D --> E["π JSON Action"]
E --> F{"π‘οΈ Interceptor"}
F -->|Corrected| G["π§ Execute Tool"]
F -->|Blocked| D
G --> H["π€ Tool Result"]
H --> I{"β
DONE?"}
I -->|No| D
I -->|Yes| J["π Complete"]
C --> J
Each LLM step outputs a structured JSON action:
{
"thought": "I need SpaceX info. Rule 5: search_web first.",
"tool": "search_web",
"arguments": {"query": "latest SpaceX launch news 2026"}
}OmniClaw/
βββ main.py # π CLI entry point (voice + text modes)
βββ server.py # π Flask web server with SSE streaming
βββ orchestrator.py # π§ ReAct loop, fast-path router, intent interceptor
βββ llm_router.py # π€ LLM integration (Llama 3.1 70B via NVIDIA NIM)
βββ tools.py # π§ 9 tools: intents, UI automation, file ops, web search
βββ adb_utils.py # π± ADB command wrappers (tap, type, dump UI, keys)
βββ voice_engine.py # ποΈ Whisper-based speech-to-text
βββ web_utils.py # π DuckDuckGo search + page scraping
βββ index.html # π¨ Web dashboard (single-file, real-time SSE)
βββ requirements.txt # π¦ Python dependencies
βββ .env # π API keys (not committed)
| Requirement | Details |
|---|---|
| Python | 3.10+ |
| ADB | Platform Tools installed and in PATH |
| Android Device | Connected via USB with USB Debugging enabled |
| NVIDIA NIM API Key | Get one here (free tier available) |
git clone https://github.com/ASIKKANI/OmniClaw.git
cd OmniClaw
pip install -r requirements.txtCreate a .env file in the project root:
NVIDIA_API_KEY=nvapi-your-key-here
# Optional overrides
LLAMA_MODEL=meta/llama-3.1-70b-instruct
LLAMA_BASE_URL=https://integrate.api.nvidia.com/v1adb devices # Verify your device appearsEnable USB Debugging in Developer Options on your Android device.
Web UI (recommended):
python server.py
# Open http://localhost:5000CLI β Voice Mode:
python main.py
# Speak your command, silence stops recordingCLI β Text Mode:
python main.py --text
# Type your goal and press Enter| # | Tool | Description | Speed |
|---|---|---|---|
| 1 | android_intent_dispatcher |
Fire Android intents (Gmail, WhatsApp, browser, alarm, timer, calendar, SMS, call) | β‘ Instant |
| 2 | find_and_launch_app |
Dynamically search device packages and launch by common name | π Fast |
| 3 | press_hardware_key |
Press BACK, HOME, ENTER, TAB, RECENT_APPS | β‘ Instant |
| 4 | execute_android_ui_task |
LLM-driven UI automation with anti-loop protection | π’ Slow |
| 5 | search_web |
Search DuckDuckGo for real-time information | π Fast |
| 6 | search_local_file |
Find files on the local PC (Desktop, Documents, Downloads) | π Fast |
| 7 | adb_push_file |
Push a local file to the Android device | π Fast |
| 8 | adb_check_app |
Check if a package is installed on the device | β‘ Instant |
| 9 | adb_install_app |
Install an APK on the device | π’ Slow |
graph LR
subgraph Layer1["Layer 1: Fast-Path"]
FP["Regex Pattern Match<br/><small>alarm, call, sms, browser,<br/>timer, calendar, WhatsApp</small>"]
end
subgraph Layer2["Layer 2: Orchestrator LLM"]
ORC["Llama 3.1 70B<br/><small>ReAct reasoning loop<br/>Tool selection & arguments</small>"]
end
subgraph Layer3["Layer 3: Inner UI LLM"]
INNER["Llama 3.1 70B<br/><small>Screen understanding<br/>Tap/Type decisions</small>"]
end
subgraph Layer4["Layer 4: Evaluator"]
EVAL["Progress Evaluator<br/><small>On-track assessment<br/>Course correction</small>"]
end
FP -->|"Miss"| ORC
ORC -->|"UI task"| INNER
ORC -->|"Check progress"| EVAL
EVAL -->|"Correction"| ORC
style FP fill:#ff6b35,color:#fff
style ORC fill:#6c5ce7,color:#fff
style INNER fill:#0984e3,color:#fff
style EVAL fill:#00b894,color:#fff
| Layer | Role | Model |
|---|---|---|
| Fast-Path | Instant intent dispatch for known patterns | None (regex) |
| Orchestrator | Strategic reasoning, tool selection | Llama 3.1 70B |
| UI Agent | Screen reading, tap/type decisions | Llama 3.1 70B |
| Evaluator | Progress assessment, course correction | Llama 3.1 70B |
OmniClaw has three independent layers preventing the LLM from going off-track:
flowchart TD
LLM["π€ LLM Output"] --> G1{"π‘οΈ Guard 1:<br/>Intent Redirect"}
G1 -->|"LLM tries UI for alarm"| FIX1["β Redirect to<br/>intent_dispatcher(alarm)"]
G1 -->|"Pass"| G2{"π‘οΈ Guard 2:<br/>Package Block"}
G2 -->|"LLM uses com.samsung.*"| FIX2["β BLOCKED<br/>Use find_and_launch_app"]
G2 -->|"Pass"| G3{"π‘οΈ Guard 3:<br/>Action Block"}
G3 -->|"Inner LLM tries launch/call"| FIX3["β BLOCKED<br/>Return DONE"]
G3 -->|"Pass"| EXEC["β
Execute Action"]
style G1 fill:#e17055,color:#fff
style G2 fill:#d63031,color:#fff
style G3 fill:#c0392b,color:#fff
style EXEC fill:#00b894,color:#fff
The web UI provides a real-time view of the agent's thinking process via Server-Sent Events:
- π― Goal display with input bar
- π§ Live thought stream (see each ReAct step)
- π§ Tool execution with arguments
- π€ Results from each tool call
- βΉοΈ Skip / Stop controls
Start the dashboard:
python server.py
# Navigate to http://localhost:5000| Variable | Required | Default | Description |
|---|---|---|---|
NVIDIA_API_KEY |
β | β | NVIDIA NIM API key for Llama 3.1 |
LLAMA_API_KEY |
β | Alternative to NVIDIA_API_KEY | |
LLAMA_MODEL |
β | meta/llama-3.1-70b-instruct |
Model identifier |
LLAMA_BASE_URL |
β | https://integrate.api.nvidia.com/v1 |
API base URL |
sequenceDiagram
actor User
participant Voice as ποΈ Voice Engine
participant FP as β‘ Fast-Path
participant Orch as π§ Orchestrator
participant LLM as π€ Llama 3.1
participant Tools as π§ Tools
participant ADB as π± ADB
participant Phone as π² Phone
User->>Voice: "Email alex about SpaceX"
Voice->>FP: Transcribed text
FP->>Orch: No fast-path match
Orch->>LLM: GOAL + System Prompt
LLM->>Orch: {"tool": "search_web", ...}
Orch->>Tools: search_web("SpaceX launch")
Tools-->>Orch: "Starship Flight 10..."
Orch->>LLM: Tool result + context
LLM->>Orch: {"tool": "android_intent_dispatcher", ...}
Orch->>Tools: intent_dispatcher(gmail, ...)
Tools->>ADB: am start -a SEND ...
ADB->>Phone: Opens Gmail compose
Phone-->>Orch: Success
Orch->>LLM: Intent result
LLM->>Orch: {"tool": "DONE", ...}
Orch-->>User: β
"Email sent to alex@gmail.com"
| Component | Technology |
|---|---|
| LLM | Meta Llama 3.1 70B Instruct |
| LLM API | NVIDIA NIM (OpenAI-compatible) |
| Voice | faster-whisper (CTranslate2 backend) |
| Device Control | Android Debug Bridge (ADB) |
| Web Server | Flask + Server-Sent Events |
| Web Research | DuckDuckGo + BeautifulSoup4 |
| Language | Python 3.10+ |
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is open source and available under the MIT License.
Built with π¦ by ASIKKANI
OmniClaw β One goal. Every action. Fully autonomous.