iOS automation SDK and playground for Midscene.js

• midscene-ios - Core iOS automation SDK
• midscene-ios-playground - Interactive playground for iOS automation

pnpm install

This project requires Python 3 and several Python packages for iOS device automation:

# Install required Python packages
pip3 install flask pyautogui pillow requests

Or run the automated setup script:

cd packages/ios
./setup.sh

For automation to work, you need to grant accessibility permissions:

1. Open System Preferences → Security & Privacy → Privacy
2. Select Accessibility from the left sidebar
3. Add your Terminal application to the list
4. If using VS Code or other IDEs, add them as well

For screenshot functionality, you also need screen recording permissions:

1. In the same Privacy panel, select Screen Recording
2. Add your Terminal application and any IDEs you're using
3. Restart the applications after granting permissions

pnpm run build

Create an .env file.

OPENAI_BASE_URL="xxxx"
OPENAI_API_KEY="xxxx"
MIDSCENE_MODEL_NAME="xxxx"

Run with npx.

npx midscene-ios-playground

• Node.js 18+
• Python 3 with pip
• macOS (required for iOS device mirroring)
• iOS device with screen mirroring support
• System permissions: Accessibility and Screen Recording permissions for your terminal/IDE
• Midscene AI model configuration

pnpm run dev

pnpm test

import { agentFromPyAutoGUI } from 'midscene-ios';
import 'dotenv/config';

const main = async () => {
    // Create agent
    const agent = await agentFromPyAutoGUI();

    // Use AI-powered automation
    await agent.aiAction('tap the home button');

    // Extract data
    const apps = await agent.aiQuery('string[], get all app names on screen');
    console.log('Apps:', apps);
};

main();

The iOS Playground provides a web interface for interactive testing:

npx midscene-ios-playground

This will:

1. Start a local server
2. Open your browser to the playground interface
3. Allow you to control iOS devices through a web UI

If you encounter permission errors:

1. Verify Accessibility Permissions:

   • Check that your terminal/IDE is listed in System Preferences → Security & Privacy → Privacy → Accessibility
   • Try removing and re-adding the application
   • Restart the application after granting permissions

2. Verify Screen Recording Permissions:

   • Check that your terminal/IDE is listed in System Preferences → Security & Privacy → Privacy → Screen Recording
   • Screenshots and screen mirroring require these permissions

If Python package installation fails:

# Update pip first
pip3 install --upgrade pip

# Install packages with user flag if needed
pip3 install --user flask pyautogui pillow requests

# On some systems, you might need to use python instead of python3
python -m pip install flask pyautogui pillow requests

If you see security warnings about Python or terminal access:

1. Click "Open Anyway" in System Preferences → Security & Privacy → General
2. Or temporarily disable Gatekeeper: sudo spctl --master-disable (remember to re-enable it later)

If the default port is occupied:

// Use a different port
const agent = await agentFromPyAutoGUI({ serverPort: 1413 });

MIT