Watch Julie in action:

Welcome to Julie, an open-source, state-of-the-art framework designed to natively interact with your computer.

Julie was built on top of the incredible Agent-S3 project. We extend our deepest thanks to the researchers and teams behind the Agent-S lineage. Julie inherits the core Agent-Computer Interface (ACI) principles pioneered by Agent-S, but introduces a massive suite of improvements regarding OS accessibility, safety, stability, and speed.

• Hybrid Fast Grounding: Sub-pixel grounding can be incredibly slow. Julie automatically intercepts OS elements and extracts native desktop UIAutomation (UIA) metadata to achieve near-instant coordinate synthesis, only falling back to heavy visual models if the DOM fails.
• Advanced Action Primitives: Replaced brittle single-turn behaviors with sophisticated macros (scroll, select_text, show_desktop).
• Built-in Security Profiles: Agent-S could execute destructive OS operations without warning. Julie intercepts these actions and explicitly requires native GUI user-confirmation loops before proceeding.
• Enhanced Context Strategies: Completely overhauled the memory manager to bypass context limits, and removed external dependencies like Tesseract OCR in favor of native OS text extraction.
• Wider Model Support: Julie adds support for --system-in-message prompt layouts, enabling seamless use of localized models like Gemma.

Getting Julie up and running on your local machine is simple.

Note: A dedicated PyPI package for Julie is coming soon. For now, please install locally from source.

Clone the repository and install it in editable mode:

git clone https://github.com/infinit-X/Julie-AI.git
cd Julie
pip install -e .

Dependencies:

• Python 3.9+
• Tesseract OCR (brew install tesseract on Mac, or download the Windows installer)

Julie requires API keys to fuel her vision and reasoning. Create a .env file in your workspace or export these variables directly in your terminal:

# Main Reasoning Model (e.g. GPT-4o, Claude 3.5 Sonnet, etc.)
OPENAI_API_KEY=your_openai_api_key

# Grounding / Vision Coordinate Model
HF_TOKEN=your_huggingface_token_for_ui_tars

You can launch Julie directly from your terminal! The most optimized configuration utilizes GPT-4o for high-level reasoning and UI-TARS for sub-pixel grounding:

julie \
    --provider openai \
    --model gpt-4o \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --system-in-message (Optional)

Once initialized, simply type your query at the prompt:

Query: open Chrome, go to youtube, and play a lofi hip hop radio stream

Julie also ships with a desktop launcher that uses the same shared runtime as the CLI.

pip install -e .
julie-gui

The GUI currently provides:

• reasoning and grounding configuration panels
• persisted workspace settings in .env
• saved non-secret profiles in .julie-gui/state.json
• live timeline, diagnostics, and preview rendering during a run
• Windows active-control overlay with animated glow, top-center control pill, Esc safe-boundary pause, manual resume, and synchronized pause/resume/stop controls in both the overlay and main window

The GUI implementation is Windows-first. Secrets stay in .env; saved profiles intentionally contain only non-secret configuration.

Julie isn't just a CLI tool—she offers a robust Python SDK for custom integrations.

import pyautogui
import io
from gui_agents.s3.agents.julie import JulieAgent
from gui_agents.s3.agents.grounding import OSWorldACI

# 1. Provide your LLM credentials
engine_params = {
  "engine_type": "openai",
  "model": "gpt-4o"
}

# 2. Provide your Coordinate Grounding credentials 
grounding_params = {
  "engine_type": "huggingface",
  "model": "ui-tars-1.5-7b",
  "grounding_width": 1920,
  "grounding_height": 1080,
}

# 3. Initialize the Agents
grounding_agent = OSWorldACI(
    platform="windows",
    engine_params_for_generation=engine_params,
    engine_params_for_grounding=grounding_params
)

julie = JulieAgent(engine_params, grounding_agent, platform="windows")

# 4. Supply context and execute!
screenshot = pyautogui.screenshot()
buffered = io.BytesIO() 
screenshot.save(buffered, format="PNG")

obs = {"screenshot": buffered.getvalue()}
info, action = julie.predict(instruction="Create a new folder called workspace", observation=obs)

exec(action[0])

Note: Julie has not yet undergone independent benchmarks. The incredibly strong results below belong to the base Agent-S3 architecture that Julie is built upon.

• 72.60% on OSWorld: The Agent-S lineage was the first framework to surpass human-level performance (~72%) on the OSWorld benchmark through Behavior Best-of-N rollouts.
• Zero-Shot Generalization: Demonstrated unmatched success curves on WindowsAgentArena (56.6%) and AndroidWorld (71.6%).

For deployment details specifically for the OSWorld framework, check out the OSWorld Deployment setup.

This project represents the continuous evolution of GUI agent frameworks, made possible by the pioneers behind Agent-S. If you utilize Julie or the underlying Agent-S research in your work, please ensure you cite the original papers:

@misc{Julie,
      title={The Unreasonable Effectiveness of Scaling Agents for Computer Use}, 
      author={Gonzalo Gonzalez-Pumariega and Vincent Tu and Chih-Lun Lee and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2510.02250},
      url={https://arxiv.org/abs/2510.02250}, 
}

@misc{Julie,
      title={Julie2: A Compositional Generalist-Specialist Framework for Computer Use Agents}, 
      author={Saaket Agashe and Kyle Wong and Vincent Tu and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2504.00906},
}

@inproceedings{Julie,
    title={{Julie: An Open Agentic Framework that Uses Computers Like a Human}},
    author={Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang},
    booktitle={ICLR},
    year={2025},
}