AISMS — AI Interview Monitoring System

AISMS (AI Interview Monitoring System) is a full-stack platform designed to assist in monitoring online interview sessions using computer vision, audio monitoring, and risk analysis.

The system captures webcam and microphone signals during an interview, analyzes them in real time, identifies unusual events, calculates a risk level, and provides session summaries and feedback through a web dashboard.

Online interviews can be difficult to monitor consistently. This system helps interviewers by automatically detecting events such as:

• Face absence
• Multiple faces
• Unusual eye or gaze behavior
• Background voice or noise activity

These signals are combined to generate a session risk score and help with reviewing the interview session.

• Eye and gaze tracking
• Face presence detection
• Multiple-face detection
• Background voice/noise monitoring
• Real-time risk calculation

The system combines detected events into a composite risk score and classifies the session into:

• LOW
• MEDIUM
• HIGH

• Session history
• Risk score tracking
• Detected-event summaries
• Candidate feedback
• Admin session review

• Candidate registration and login
• Candidate dashboard
• Live interview monitoring
• Session feedback
• Admin dashboard

Candidate
    │
    ▼
Webcam + Microphone
    │
    ▼
Next.js Frontend
    │
    ▼
Flask Backend API
    │
    ├── Eye/Gaze Tracking
    ├── Face Detection
    └── Voice Monitoring
             │
             ▼
       Risk Analysis Engine
             │
             ▼
        Session Data
          │       │
          ▼       ▼
       SQLite   Summary
          │       │
          └───┬───┘
              ▼
      Feedback & Analytics
              │
              ▼
      Candidate / Admin Dashboard

• Next.js
• React
• TypeScript
• Tailwind CSS
• Framer Motion
• Recharts

• Python
• Flask
• OpenCV
• MediaPipe
• NumPy
• Pandas

• SQLite
• bcrypt

• sounddevice

AI-interview-monitoring-system/
│
├── Frontend/
│   ├── app/
│   ├── components/
│   ├── contexts/
│   ├── lib/
│   └── package.json
│
├── Backend/
│   ├── app.py
│   ├── database.py
│   ├── feedback_engine.py
│   ├── main_system.py
│   ├── eye_tracking/
│   ├── face_tracking/
│   ├── risk_module/
│   └── voice_module/
│
├── face_landmarker.task
└── README.md

1. The candidate creates an account and logs in.
2. The candidate starts an online interview session.
3. The frontend captures webcam and microphone input.
4. The backend processes the monitoring signals.
5. Eye, face, and voice events are detected.
6. The risk engine combines the detected events into a risk score.
7. Session information is stored for later analysis.
8. The dashboard displays session results and feedback.

Uses computer-vision techniques and facial landmarks to analyze eye and gaze behavior during the interview.

Detects:

• Face presence
• Face absence
• Multiple faces
• Changes in face presence during the session

Monitors microphone input for background voice or noise activity when audio support is available.

Detected events are converted into a composite risk score.

Eye Events
     │
Face Events ─────► Risk Engine ─────► Risk Level
     │
Voice Events

After the session, detected events and session patterns are used to generate feedback for the candidate.

The system provides:

• Candidate registration
• Login
• Password hashing using bcrypt
• Authentication state management
• Protected dashboard and interview pages

Candidate and session information is stored using SQLite.

The application provides dashboards for reviewing:

• Session information
• Risk levels
• Detected events
• Session history
• Candidate feedback

The Flask backend provides APIs for the monitoring workflow.

• Python 3.10+
• Node.js 18+
• Webcam
• Microphone

cd Backend
python -m venv .venv

.venv\Scripts\activate

source .venv/bin/activate

Install dependencies:

pip install -r requirements.txt

Start the backend:

python app.py

Backend:

http://localhost:5000

Open another terminal:

cd Frontend
npm install
npm run dev

Frontend:

http://localhost:3000

If required, configure:

NEXT_PUBLIC_API_URL=http://localhost:5000

• More robust gaze and behavioral analysis
• Improved audio classification
• Machine-learning-based risk prediction
• Cloud database integration
• Real-time notifications
• Better model evaluation and performance metrics
• Production deployment

This project was developed as a modular full-stack system with separate components for:

• Frontend development
• Backend/API development
• Computer vision
• Audio monitoring
• Risk analysis
• Dashboard and analytics

The individual components were developed separately during the project and integrated into the final AISMS system.

AISMS combines computer vision, audio monitoring, risk analysis, and a full-stack web application to assist with monitoring online interview sessions.

Technologies: Python · Flask · OpenCV · MediaPipe · Next.js · React · TypeScript · SQLite