VERA is an advanced multi-modal AI pipeline designed to analyze communication performance. It processes video input to evaluate Face (micro-expressions, gaze...), Body (posture, gestures...), and Audio (tonality, pacing...) metrics. By leveraging computer vision and audio signal processing, VERA provides objective, data-driven feedback to help users improve their communication skills.

Disclaimer:

• VERA is a prototype and we recommend to not upload videos longer than 1 to 2 minutes (e.g. elevator pitch).
• Some metrics are under fine-tuning and may not be as accurate as others.

Public speaking and communication are critical soft skills for Founders, Students, Executives.... Yet, feedback is often subjective and scarce.

VERA solves this by providing:

• Scientific Objectivity: Moving beyond "you look nervous" to "your gaze stability dropped by 20%".
• Holistic Feedback: Analyzing not just what you say, but how you say it through your non-verbal communication (Face, Body, Voice).
• Automated Coaching: offering actionable tips.

• Multi-Modal Analysis: Parallel processing of visual signals (using MediaPipe) and auditory signals (using Whisper and Librosa).
• 14 Core Metrics: Comprehensive tracking of engagement indicators (e.g., Gaze Stability, Gesture Magnitude, Vocal dynamic).
• Persona K-means Clustering: Unsupervised ML that automatically assigns a "Communication Persona" based on high-dimensional metric clusters.
• Rich Visualization: Generates debug videos with skeletal overlays to visualize exactly what the AI sees.

VERA sees what you can't. Our computer vision pipelines track micro-expressions and skeletal dynamics in real-time.

Left: Mediapipe FaceMesh - 13 facial landmarks out of the 468 for Face analysis

Right: MediapipeHolistic - 6 body landmarks out of the 33for Gesture Analysis

A minimal, high-impact dashboard gives you instant feedback on your communication skills.

VERA Dashboard: Video upload

VERA Dashboard: Video analysis

VERA Dashboard: Real-time analysis with global scoring

Deep Dive: Granular metrics and actionable coaching tips

History Page to revisit your past coaching sessions

Unsupervised ML that automatically assigns a "Communication Persona" based on raw high-dimensional extracted metrics. e.g:

3D PCA projection (Explained variance: 80.4%)
7 features – K=4 – n_init = 50 – run 30 times

VERA/
├── assets/                 # Project assets (logo, example outputs)
│   └── examples/           # Sample videos, GIFs, and screenshots
├── data/                   # Data storage for analysis
│   ├── clustering_dataset/ # Master dataset (master_vector_data_set.csv)
│   ├── clustering_results/ # Generated K-Means plots
│   ├── models/             # Serialized ML models (scaler.pkl, kmeans_model.pkl)
│   ├── processed/          # Intermediate CSV outputs (CLI)
│   └── raw/                # Source videos (CLI)
├── docs/                   # Documentation resources
│   └── metrics_reference_guide.md # Detailed metrics guide
├── front-end/              # Streamlit Web Application
│   ├── app.py              # Main dashboard application
│   ├── processed/          # Session-specific analysis outputs
│   └── uploaded/           # Temporary user uploads
├── notebooks/              # Jupyter notebooks for exploration
├── reports/                # Calibration and audit reports
├── src/                    # Source Code
│   ├── analysis/           # Analytics, Calibration, Clustering logic
│   ├── api/                # API Logic
│   ├── audio/              # Audio module (Whisper/Librosa)
│   ├── body/               # Body module (MediaPipe Pose)
│   ├── face/               # Face module (MediaPipe FaceMesh)
│   ├── presentation/       # Presentation logic
│   ├── schemas/            # Config and Metric Specs (JSON)
│   ├── utils/              # Shared utilities
│   └── main.py             # CLI Entry point
├── tests/                  # Unit tests
├── .gitignore              # Git configuration
├── Docker_README           # Docker documentation
├── Dockerfile              # Container configuration
├── requirements.txt        # Python dependencies
└── README.md               # Main documentation

• Python 3.10+
• ffmpeg (Required for audio processing. Install via brew install ffmpeg on Mac or apt install ffmpeg on Linux).

# Clone the repository
git clone https://github.com/WiwiC/VERA.git
cd VERA

# Create a virtual environment
python -m venv VERA-env
source VERA-env/bin/activate  # or VERA-env\Scripts\activate on Windows

# Install dependencies
pip install -r requirements.txt

To analyze a single video file directly from the terminal:

python src/main.py <path/to/video.mp4>

• Output: data/processed/<video_filename>/

To analyze videos and visualize results in the interactive dashboard:

streamlit run front-end/app.py

• Output: front-end/processed/<video_filename>/

• File Types: .mp4 or .MOV.
• Duration: Recommended 1 to 2 minutes (e.g., elevator pitch).
  • Note: This is a prototype. A 1 min video takes ~2 min to process.
• Camera Setup:
  • Front-facing camera.
  • Camera at eye level.
  • Framing: Ensure body is visible from hips to head to capture hand gestures.

All text definitions, coaching tips, and interpretation labels are centralized in:

src/schemas/metrics_spec.json

Important: This file is the "Source of Truth". It should be kept in synchronization with the numerical thresholds defined in src/*/config.py.

A more readable reference file constructed from src/schemas/metrics_spec.json list the metrics guide (both should be keep in synch as much as possible):

docs/metrics_reference_guide.md

To ensure scientific rigor, VERA uses a calibration system to fine-tune scoring thresholds.

1. Labeled Data: We maintained a reports/calibration_manifest.csv containing IDs of videos (contained in data/raw folder) labeled by human (scores 0-100).
2. Analysis Scripts: Scripts in src/analysis/ (e.g., generate_calibration_report.py) compare VERA's outputs results against these human labels.
3. Threshold Tuning: We use final_calibration_search.py to find the optimal parabolic thresholds that maximize correlation with human judgment.

• Add Labeled Videos: Update calibration_manifest.csv with new video IDs and manual labels. (For labels list check: src/schemas/metrics_spec.json)
• Run Analysis: Execute python src/analysis/generate_calibration_report.py to see current performance.
• Update Thresholds: Adjust BASELINE_* constants in src/*/config.py based on the report.
• Validate: Re-run the report to confirm Spearman correlation improvements.

VERA uses K-means clustering to identify "Communication Personas" without explicit labels.

• Dataset: data/clustering_dataset/master_vector_data_set.csv. This file accumulates feature vectors from all processed videos.
• Mechanism:
  1. Each analysis extracts a rawhigh-dimensional feature vector (all 14 metrics).
  2. This vector is added to the master dataset.
  3. We run K-Means to find natural groupings.
  4. New users are assigned to the nearest cluster center to predict their persona.

The pipeline follows a linear data flow:

graph LR
    A[Video Input] --> B(Extraction)
    B --> C{Multimodal Processing}
    C -->|Face| D[FaceMesh + Gaze]
    C -->|Body| E[Pose Landmarks]
    C -->|Audio| F[Librosa Spectral]
    D --> G[Raw 1s Timeline]
    E --> G
    F --> G
    G --> H[5s Rolling Window]
    H --> I[Scoring Engine]
    I --> J[Enrichment & Coaching]
    J --> K[JSON & Dashboard Output]

We prioritize metrics that show strong correlation with communication effectiveness. Based on our latest calibration:

• Strong Signals: speech_rate, gaze_stability, gesture_magnitude show high Spearman correlation (>0.7) with human expertise.
• Calibration: The tiered parabolic scoring helps normalize differences between webcam hardware.

Note: Based on our last calibration report:

1. Global accuracy:

(Calibration Report 2025-12-11).

2. Accuracy per metric:

• Strong, reliable metrics (high exact AND high within-1 step):

• gesture_activity
• speech_rate
• posture_openness
• gesture_magnitude
• body_sway
• gesture_stability

• Moderately noisy but still directionally useful:

• head_stability
• gaze_stability
• pause_ratio
• pitch_dynamic
• volume_dynamic

• Noisiest / Further calibration needed:

• smile_activation
• vocal_punch
• head_down_ratio

• Local Processing: All analysis happens locally on your machine.
• No Cloud Storage: Videos are not uploaded to any external server.
• User Control: You retain full ownership of your data and bio-metric information.
• No Tracking: No third-party analytics or tracking pixels are used.

MIT License Copyright (c) 2025 WiwiC Commercial use prohibited without explicit permission

• Author: William Chalons
• Core Libraries:
  • MediaPipe (Vision)
  • Librosa (Audio)
  • Streamlit (UI)
  • Pandas/Numpy (Data)
  • Scikit-learn (Clustering)