VERA is an advanced multi-modal AI pipeline designed to analyze communication performance. It processes video input to evaluate Face (micro-expressions, gaze), Body (posture, gestures), and Audio (tonality, pacing) metrics. By leveraging computer vision and audio signal processing, VERA provides objective, data-driven feedback to help users improve their communication skills.

Public speaking and communication are critical soft skills for Founders, Students, and Executives. Yet, feedback is often subjective and scarce.

VERA solves this by providing:

• Scientific Objectivity: Moving beyond "you look nervous" to "your gaze stability dropped by 20%".
• Holistic Feedback: Analyzing not just what you say, but how you say it (Face, Body, Voice).
• Automated Coaching: offering actionable tips based on your specific communication persona.

• Multi-Modal Analysis: Parallel processing of visual signals (using MediaPipe) and auditory signals (using Librosa).
• 14 Core Metrics: Comprehensive tracking of engagement indicators (e.g., Gaze Stability, Gesture Magnitude, Vocal Punch).
• Persona K-means Clustering: Unsupervised ML that automatically assigns a "Communication Persona" based on high-dimensional metric clusters.
• Rich Visualization: Generates debug videos with skeletal overlays to visualize exactly what the AI sees.

We provide sample outputs in the assets/examples/ directory to demonstrate VERA's capabilities:

(Note: To view these, navigate to the assets/examples/ folder)

VERA/
├── data/
│   ├── clustering_dataset/      # ML dataset for persona clustering
│   ├── processed/               # Pipeline output for CLI runs
│   └── raw/                     # Input videos
├── front-end/
│   ├── app.py                   # Streamlit Dashboard application
│   └── processed/               # Pipeline output for UI runs
├── notebooks/                   # Jupyter notebooks for experimentation & viz
├── reports/                     # Calibration manifests and analysis reports
├── src/
│   ├── analysis/                # Data enrichment, scoring, and prediction logic
│   ├── audio/                   # Audio extraction & processing (Librosa)
│   ├── body/                    # Body pose extraction (MediaPipe Pose)
│   ├── face/                    # Facial landmark extraction (MediaPipe FaceMesh)
│   ├── presentation/            # Result formatting and enrichment
│   ├── schemas/                 # JSON schemas (single source of truth for text)
│   ├── utils/                   # Shared utilities (IO, logging)
│   └── main.py                  # CLI Entry point & Orchestrator
└── README.md

• Python 3.10+
• ffmpeg (Required for audio processing. Install via brew install ffmpeg on Mac or apt install ffmpeg on Linux).

# Clone the repository
git clone https://github.com/WiwiC/VERA.git
cd VERA

# Create a virtual environment
python -m venv VERA-env
source VERA-env/bin/activate  # or VERA-env\Scripts\activate on Windows

# Install dependencies
pip install -r requirements.txt

To analyze a single video file directly from the terminal:

python src/main.py <path/to/video.mp4>

• Output: data/processed/<video_filename>/

To analyze videos and visualize results in the interactive dashboard:

streamlit run front-end/app.py

• Output: front-end/processed/<video_filename>/

• File Types: .mp4 or .MOV.
• Duration: Recommended 1 to 2 minutes (e.g., elevator pitch).
  • Note: This is a prototype. A 1 min video takes ~2 min to process.
• Camera Setup:
  • Front-facing camera.
  • Camera at eye level.
  • Framing: Ensure body is visible from hips to head to capture hand gestures.

All text definitions, coaching tips, and interpretation labels are centralized in:

src/schemas/metrics_spec.json

Important: This file is the "Source of Truth". It should be kept in synchronization with the numerical thresholds defined in src/*/config.py.

To ensure scientific rigor, VERA uses a calibration system to fine-tune scoring thresholds.

1. Labeled Data: We maintain a reports/calibration_manifest.csv containing videos labeled by human experts (scores 0-100).
2. Analysis Scripts: Scripts in src/analysis/ (e.g., generate_calibration_report.py) compare VERA's raw outputs against these human labels.
3. Threshold Tuning: We use final_calibration_search.py to find the optimal parabolic thresholds that maximize correlation with human judgment.

• Add Labeled Videos: Update calibration_manifest.csv with new video paths and manual scores.
• Run Analysis: Execute python src/analysis/generate_calibration_report.py to see current performance.
• Update Thresholds: Adjust BASELINE_* constants in src/*/config.py based on the report.
• Validate: Re-run the report to confirm Spearman correlation improvements.

VERA uses K-means clustering to identify "Communication Personas" without explicit labels.

• Dataset: data/clustering_dataset/master_vector_data_set.csv. This file accumulates feature vectors from all processed videos.
• Mechanism:
  1. Each analysis extracts a high-dimensional feature vector (all 14 metrics).
  2. This vector is added to the master dataset.
  3. We run K-Means to find natural groupings (e.g., "The Orator", "The Jittery Speaker").
  4. New users are assigned to the nearest cluster center to predict their persona.

The pipeline follows a linear data flow:

graph LR
    A[Video Input] --> B(Extraction)
    B --> C{Multimodal Processing}
    C -->|Face| D[FaceMesh + Gaze]
    C -->|Body| E[Pose Landmarks]
    C -->|Audio| F[Librosa Spectral]
    D --> G[Raw 1s Timeline]
    E --> G
    F --> G
    G --> H[5s Rolling Window]
    H --> I[Scoring Engine]
    I --> J[Enrichment & Coaching]
    J --> K[JSON & Dashboard Output]

We prioritize metrics that show strong correlation with communication effectiveness. Based on our latest calibration:

• Strong Signals: speech_rate, gaze_stability, gesture_magnitude show high Spearman correlation (>0.7) with human expertise.
• Noisy Signals: micro_expression_energy was removed due to low reliability in uncontrolled lighting.
• Calibration: The tiered parabolic scoring helps normalize differences between webcam hardware.

• Local Processing: All analysis happens locally on your machine.
• No Cloud Storage: Videos are not uploaded to any external server.
• User Control: You retain full ownership of your data and bio-metric information.
• No Tracking: No third-party analytics or tracking pixels are used.

MIT License Copyright (c) 2025 WiwiC

• Author: William Chalons
• Core Libraries:
  • MediaPipe (Vision)
  • Librosa (Audio)
  • Streamlit (UI)
  • Pandas/Numpy (Data)