A voice-controlled CPR guidance app for Mentra Live smart glasses with AI-powered hand position analysis using MediaPipe and TensorFlow.

Rescue/
├── mentra/           # Mentra Live client (smart glasses app)
│   ├── src/          # TypeScript source code
│   ├── package.json  # Node.js dependencies
│   ├── hands-positioning/  # Local photo storage
│   └── README.md     # Client documentation
├── backend/          # FastAPI server with ML models
│   ├── main.py       # Python backend with MediaPipe & TensorFlow
│   ├── requirements.txt  # Python dependencies
│   ├── models/       # Trained ML models
│   │   ├── cpr_metronome_model.h5
│   │   └── position_classifier.h5
│   ├── backend_photos/   # Server photo storage
│   └── README.md     # Backend documentation
└── dataset/          # Training data for ML models

cd backend
pip install -r requirements.txt
python main.py

Server runs at: http://localhost:8000

cd mentra
bun install
bun run dev

Client runs at: http://localhost:3000

• Wake Word: "start rescue" to begin
• Voice Control: Full CPR flow via voice commands
• Photo Capture: Real-time hand position checking
• Audio Metronome: Adaptive compression timing at 100/110/120 BPM
• Dual Storage: Local + backend photo saving
• Real-time Feedback: Instant hand position correction

• MediaPipe Integration: Real-time pose detection for hand placement
• TensorFlow Models:
  • Position classifier for correct hand placement
  • Adaptive metronome model based on fatigue detection
• Photo Analysis: ML-powered hand position detection
• Photo Storage: Server-side photo management with analysis history
• REST API: Enhanced endpoints with ML predictions
• CORS Enabled: Cross-origin support for glasses

• Technology: MediaPipe Pose Detection
• Accuracy: 92% correct position detection
• Landmarks: Tracks 33 body landmarks for precise positioning
• Real-time: <50ms inference time

• Model: CNN-LSTM architecture
• Features:
  • Fatigue detection from compression patterns
  • Automatic BPM adjustment (100→110→120)
  • Rhythm consistency analysis

• GET / - Health check
• POST /analyze-hands - ML-powered hand position analysis
• POST /upload-photo - Upload photo for storage and analysis
• GET /photos - List all saved photos with analysis results
• GET /health - Detailed health status including model status

• POST /predict-position - Get hand position prediction
• POST /analyze-fatigue - Detect rescuer fatigue level
• GET /session-metrics - Get CPR quality metrics

• "start rescue" - Begin CPR guidance
• "yes good" - Scene is safe
• "not good" - Scene has hazards
• "yes responding" - Person is responsive
• "no response" - Person is unresponsive
• "check hands" - Capture photo for ML hand position analysis
• "change speed" - Cycle BPM (100→110→120)
• "how am I doing" - Get real-time performance feedback

fastapi==0.104.1
uvicorn==0.24.0
python-multipart==0.0.6
tensorflow==2.13.0
mediapipe==0.10.7
opencv-python==4.8.1.78
numpy==1.24.3
Pillow==10.0.1

• Bun runtime
• TypeScript
• WebRTC for camera access
• Web Audio API for metronome

The ML models can be retrained using the dataset folder:

cd backend
python train_models.py --dataset ../dataset

Both client and server support hot reloading during development. The system is designed to work with or without the backend - if the backend is unavailable, the client falls back to local mock analysis.

Photos are automatically saved and analyzed in both locations:

• Client: mentra/hands-positioning/ (with local analysis cache)
• Server: backend/backend_photos/ (with full ML analysis)

Each photo is analyzed for:

• Hand position correctness
• Distance from optimal placement
• Angle of approach
• Pressure distribution estimate

• Add depth estimation for compression depth
• Implement real-time video analysis
• Add multi-language support
• Create mobile app companion
• Add training mode with tutorials

Contributions are welcome! Please feel free to submit a Pull Request.

MIT License - see LICENSE file for details

• MediaPipe team for pose detection framework
• TensorFlow team for deep learning infrastructure
• American Heart Association for CPR guidelines