Skip to content

Latest commit

Β 

History

6 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

OmniClaw

OmniClaw

An Autonomous General-Intelligence Agent That Controls Your Android Phone

Speak or type a goal β†’ OmniClaw reasons, plans, and executes it on your phone β€” hands-free.

Tools Architecture License Python Llama Android Whisper Flask


🎯 What Is OmniClaw?

OmniClaw is an autonomous AI agent that takes a natural-language goal β€” spoken or typed β€” and executes it on a connected Android device. It combines LLM reasoning (Llama 3.1 70B via NVIDIA NIM), Android Debug Bridge control, voice input (Whisper), and a real-time web dashboard into a single system that can:

  • πŸ“§ Draft and send emails with real-time web research
  • πŸ“ž Make phone calls and send WhatsApp/SMS messages
  • ⏰ Set alarms, timers, and calendar events
  • 🌐 Open URLs and search the web
  • πŸ“± Navigate any Android app through UI automation
  • πŸ” Find, install, and launch apps dynamically
  • πŸ“Ž Locate local files and push them to the phone

Think Siri + Jarvis β€” but open-source, running locally, and powered by a 70B-parameter LLM.


πŸ§ͺ Demo

Voice Command What Happens
"Email alex@gmail.com about the latest SpaceX launch" Searches the web β†’ drafts email with real data β†’ opens Gmail with fields pre-filled
"Set an alarm for 7:30 AM" Fast-path: instant intent dispatch, no LLM needed (< 1 second)
"Send 100rs to Yeswanth on FamPay" Launches FamPay β†’ navigates UI β†’ finds contact β†’ enters amount
"Open the calculator and compute 500 Γ— 2" Dynamically finds calculator package β†’ launches β†’ taps buttons β†’ reads result

✨ Features

⚑ Three Execution Tiers

Tier Speed When
Fast-Path < 1s Alarms, calls, SMS, browser, calendar β€” bypasses LLM entirely
Intent Dispatch ~2s Gmail, WhatsApp, dialer, timer β€” fires raw Android intents via ADB
LLM + UI Automation 10–60s Complex tasks requiring multi-step reasoning and screen interaction

πŸ›‘οΈ Multi-Layer Safety

  • Fast-Path Router β€” intercepts known patterns before the LLM even runs
  • Intent Interceptor β€” code-level guardrail that corrects the LLM if it tries to launch apps that have dedicated intents
  • Action Blocker β€” blocks the inner LLM from executing forbidden actions (launch, call, open_url)
  • State-Hash Anti-Loop β€” detects unchanged screens and prevents infinite loops
  • Action Deduplication β€” prevents repeated identical actions

πŸŽ™οΈ Dual Input Modes

  • Voice β€” local Whisper transcription (base.en model, CPU, int8)
  • Text β€” CLI or web UI

πŸ—οΈ Architecture

graph TB
    subgraph Input
        V["πŸŽ™οΈ Voice Engine<br/><small>Whisper STT</small>"]
        T["⌨️ Text Input<br/><small>CLI / Web UI</small>"]
    end

    subgraph Core["🧠 Core Intelligence"]
        FP["⚑ Fast-Path Router<br/><small>Regex pattern matching</small>"]
        O["πŸ¦€ Orchestrator<br/><small>ReAct Loop</small>"]
        LLM["πŸ€– Llama 3.1 70B<br/><small>NVIDIA NIM API</small>"]
        INT["πŸ›‘οΈ Intent Interceptor<br/><small>Guardrail Layer</small>"]
    end

    subgraph Tools["πŸ”§ Tool Registry"]
        AID["πŸ“± Intent Dispatcher<br/><small>Gmail, WhatsApp, SMS,<br/>Alarm, Timer, Calendar</small>"]
        UI["πŸ–±οΈ UI Automation<br/><small>Tap, Type, Navigate</small>"]
        APP["πŸ“¦ App Launcher<br/><small>Dynamic Package Search</small>"]
        WEB["🌐 Web Search<br/><small>DuckDuckGo</small>"]
        FS["πŸ“‚ File Tools<br/><small>Search, Push, Install</small>"]
        HW["⌨️ Hardware Keys<br/><small>Back, Home, Enter</small>"]
    end

    subgraph Device["πŸ“² Android Device"]
        ADB["ADB Bridge"]
        PHONE["Phone Screen"]
    end

    V --> FP
    T --> FP
    FP -->|"Known pattern"| AID
    FP -->|"Complex task"| O
    O <-->|"Reason ↔ Act"| LLM
    O --> INT
    INT --> AID
    INT --> UI
    INT --> APP
    INT --> WEB
    INT --> FS
    INT --> HW
    AID --> ADB
    UI --> ADB
    APP --> ADB
    HW --> ADB
    ADB --> PHONE

    style FP fill:#ff6b35,color:#fff
    style O fill:#6c5ce7,color:#fff
    style LLM fill:#0984e3,color:#fff
    style INT fill:#d63031,color:#fff
Loading

πŸ”„ ReAct Loop

The orchestrator follows a Reason + Act cycle until the goal is achieved or the iteration limit is reached:

flowchart LR
    A["🎯 User Goal"] --> B{"⚑ Fast-Path?"}
    B -->|Yes| C["πŸš€ Intent Dispatch"]
    B -->|No| D["🧠 LLM Thinks"]
    D --> E["πŸ“‹ JSON Action"]
    E --> F{"πŸ›‘οΈ Interceptor"}
    F -->|Corrected| G["πŸ”§ Execute Tool"]
    F -->|Blocked| D
    G --> H["πŸ“€ Tool Result"]
    H --> I{"βœ… DONE?"}
    I -->|No| D
    I -->|Yes| J["🏁 Complete"]
    C --> J
Loading

Each LLM step outputs a structured JSON action:

{
  "thought": "I need SpaceX info. Rule 5: search_web first.",
  "tool": "search_web",
  "arguments": {"query": "latest SpaceX launch news 2026"}
}

πŸ“ Project Structure

OmniClaw/
β”œβ”€β”€ main.py              # πŸš€ CLI entry point (voice + text modes)
β”œβ”€β”€ server.py            # 🌐 Flask web server with SSE streaming
β”œβ”€β”€ orchestrator.py      # 🧠 ReAct loop, fast-path router, intent interceptor
β”œβ”€β”€ llm_router.py        # πŸ€– LLM integration (Llama 3.1 70B via NVIDIA NIM)
β”œβ”€β”€ tools.py             # πŸ”§ 9 tools: intents, UI automation, file ops, web search
β”œβ”€β”€ adb_utils.py         # πŸ“± ADB command wrappers (tap, type, dump UI, keys)
β”œβ”€β”€ voice_engine.py      # πŸŽ™οΈ Whisper-based speech-to-text
β”œβ”€β”€ web_utils.py         # 🌐 DuckDuckGo search + page scraping
β”œβ”€β”€ index.html           # 🎨 Web dashboard (single-file, real-time SSE)
β”œβ”€β”€ requirements.txt     # πŸ“¦ Python dependencies
└── .env                 # πŸ”‘ API keys (not committed)

πŸš€ Quick Start

Prerequisites

Requirement Details
Python 3.10+
ADB Platform Tools installed and in PATH
Android Device Connected via USB with USB Debugging enabled
NVIDIA NIM API Key Get one here (free tier available)

1. Clone & Install

git clone https://github.com/ASIKKANI/OmniClaw.git
cd OmniClaw
pip install -r requirements.txt

2. Configure

Create a .env file in the project root:

NVIDIA_API_KEY=nvapi-your-key-here

# Optional overrides
LLAMA_MODEL=meta/llama-3.1-70b-instruct
LLAMA_BASE_URL=https://integrate.api.nvidia.com/v1

3. Connect Your Phone

adb devices   # Verify your device appears

Enable USB Debugging in Developer Options on your Android device.

4. Run

Web UI (recommended):

python server.py
# Open http://localhost:5000

CLI β€” Voice Mode:

python main.py
# Speak your command, silence stops recording

CLI β€” Text Mode:

python main.py --text
# Type your goal and press Enter

πŸ”§ Tools Reference

# Tool Description Speed
1 android_intent_dispatcher Fire Android intents (Gmail, WhatsApp, browser, alarm, timer, calendar, SMS, call) ⚑ Instant
2 find_and_launch_app Dynamically search device packages and launch by common name πŸƒ Fast
3 press_hardware_key Press BACK, HOME, ENTER, TAB, RECENT_APPS ⚑ Instant
4 execute_android_ui_task LLM-driven UI automation with anti-loop protection 🐒 Slow
5 search_web Search DuckDuckGo for real-time information πŸƒ Fast
6 search_local_file Find files on the local PC (Desktop, Documents, Downloads) πŸƒ Fast
7 adb_push_file Push a local file to the Android device πŸƒ Fast
8 adb_check_app Check if a package is installed on the device ⚑ Instant
9 adb_install_app Install an APK on the device 🐒 Slow

🧠 Intelligence Layers

graph LR
    subgraph Layer1["Layer 1: Fast-Path"]
        FP["Regex Pattern Match<br/><small>alarm, call, sms, browser,<br/>timer, calendar, WhatsApp</small>"]
    end

    subgraph Layer2["Layer 2: Orchestrator LLM"]
        ORC["Llama 3.1 70B<br/><small>ReAct reasoning loop<br/>Tool selection & arguments</small>"]
    end

    subgraph Layer3["Layer 3: Inner UI LLM"]
        INNER["Llama 3.1 70B<br/><small>Screen understanding<br/>Tap/Type decisions</small>"]
    end

    subgraph Layer4["Layer 4: Evaluator"]
        EVAL["Progress Evaluator<br/><small>On-track assessment<br/>Course correction</small>"]
    end

    FP -->|"Miss"| ORC
    ORC -->|"UI task"| INNER
    ORC -->|"Check progress"| EVAL
    EVAL -->|"Correction"| ORC

    style FP fill:#ff6b35,color:#fff
    style ORC fill:#6c5ce7,color:#fff
    style INNER fill:#0984e3,color:#fff
    style EVAL fill:#00b894,color:#fff
Loading
Layer Role Model
Fast-Path Instant intent dispatch for known patterns None (regex)
Orchestrator Strategic reasoning, tool selection Llama 3.1 70B
UI Agent Screen reading, tap/type decisions Llama 3.1 70B
Evaluator Progress assessment, course correction Llama 3.1 70B

πŸ›‘οΈ Guardrail System

OmniClaw has three independent layers preventing the LLM from going off-track:

flowchart TD
    LLM["πŸ€– LLM Output"] --> G1{"πŸ›‘οΈ Guard 1:<br/>Intent Redirect"}
    G1 -->|"LLM tries UI for alarm"| FIX1["β†’ Redirect to<br/>intent_dispatcher(alarm)"]
    G1 -->|"Pass"| G2{"πŸ›‘οΈ Guard 2:<br/>Package Block"}
    G2 -->|"LLM uses com.samsung.*"| FIX2["β†’ BLOCKED<br/>Use find_and_launch_app"]
    G2 -->|"Pass"| G3{"πŸ›‘οΈ Guard 3:<br/>Action Block"}
    G3 -->|"Inner LLM tries launch/call"| FIX3["β†’ BLOCKED<br/>Return DONE"]
    G3 -->|"Pass"| EXEC["βœ… Execute Action"]

    style G1 fill:#e17055,color:#fff
    style G2 fill:#d63031,color:#fff
    style G3 fill:#c0392b,color:#fff
    style EXEC fill:#00b894,color:#fff
Loading

🌐 Web Dashboard

The web UI provides a real-time view of the agent's thinking process via Server-Sent Events:

  • 🎯 Goal display with input bar
  • 🧠 Live thought stream (see each ReAct step)
  • πŸ”§ Tool execution with arguments
  • πŸ“€ Results from each tool call
  • ⏹️ Skip / Stop controls

Start the dashboard:

python server.py
# Navigate to http://localhost:5000

πŸ”‘ Environment Variables

Variable Required Default Description
NVIDIA_API_KEY βœ… β€” NVIDIA NIM API key for Llama 3.1
LLAMA_API_KEY ⚠️ β€” Alternative to NVIDIA_API_KEY
LLAMA_MODEL ❌ meta/llama-3.1-70b-instruct Model identifier
LLAMA_BASE_URL ❌ https://integrate.api.nvidia.com/v1 API base URL

🧩 How It Works β€” End to End

sequenceDiagram
    actor User
    participant Voice as πŸŽ™οΈ Voice Engine
    participant FP as ⚑ Fast-Path
    participant Orch as 🧠 Orchestrator
    participant LLM as πŸ€– Llama 3.1
    participant Tools as πŸ”§ Tools
    participant ADB as πŸ“± ADB
    participant Phone as πŸ“² Phone

    User->>Voice: "Email alex about SpaceX"
    Voice->>FP: Transcribed text
    FP->>Orch: No fast-path match
    Orch->>LLM: GOAL + System Prompt
    LLM->>Orch: {"tool": "search_web", ...}
    Orch->>Tools: search_web("SpaceX launch")
    Tools-->>Orch: "Starship Flight 10..."
    Orch->>LLM: Tool result + context
    LLM->>Orch: {"tool": "android_intent_dispatcher", ...}
    Orch->>Tools: intent_dispatcher(gmail, ...)
    Tools->>ADB: am start -a SEND ...
    ADB->>Phone: Opens Gmail compose
    Phone-->>Orch: Success
    Orch->>LLM: Intent result
    LLM->>Orch: {"tool": "DONE", ...}
    Orch-->>User: βœ… "Email sent to alex@gmail.com"
Loading

πŸ“Š Tech Stack

Component Technology
LLM Meta Llama 3.1 70B Instruct
LLM API NVIDIA NIM (OpenAI-compatible)
Voice faster-whisper (CTranslate2 backend)
Device Control Android Debug Bridge (ADB)
Web Server Flask + Server-Sent Events
Web Research DuckDuckGo + BeautifulSoup4
Language Python 3.10+

🀝 Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

πŸ“œ License

This project is open source and available under the MIT License.


Built with πŸ¦€ by ASIKKANI
OmniClaw β€” One goal. Every action. Fully autonomous.

About

An autonomous, reasoning-first Android orchestrator. Bridges PC storage and mobile UI using Llama-3.1-70B to execute complex, cross-device workflows via ADB and Intent Injection. πŸ₯ˆ 2nd Place Winner - Hack-N-Android hacakthon.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages