An AI-powered Video Intelligence Assistant that can transcribe, summarize, and chat with YouTube videos or local audio/video files.

The system supports both English and Hindi/Hinglish content and uses Retrieval-Augmented Generation (RAG) to enable intelligent question-answering directly from video transcripts.

Built using Whisper, Sarvam AI, Mistral AI, LangChain, ChromaDB, and Streamlit.

• 🎬 Analyze YouTube videos or local media files
• 🇺🇸 English transcription using local OpenAI Whisper
• 🇮🇳 Hindi/Hinglish transcription using Sarvam AI
• ✂️ Automatic audio chunking for long recordings
• 📝 AI-generated summaries
• 🏷️ Automatic title generation
• 📋 Extract key insights from videos
• 🧠 RAG-based conversational question answering
• 📚 ChromaDB vector database for semantic retrieval
• 🤖 Mistral AI powered summarization and chat
• 🌐 Interactive Streamlit web interface
• 💻 CLI version included
• ⚡ Fast environment management using uv

YouTube URL / Local File
            │
            ▼
      Audio Extraction
            │
            ▼
      Audio Chunking
            │
            ▼
      Speech-to-Text
   ┌────────────────────┐
   │ Whisper (English)  │
   │ Sarvam (Hindi)     │
   └────────────────────┘
            │
            ▼
        Transcript
            │
            ▼
      Text Chunking
            │
            ▼
 Sentence Transformers
            │
            ▼
         ChromaDB
            │
            ▼
        Mistral AI
            │
            ▼
 Summary + RAG Chat

This project uses RAG to allow users to ask questions directly from video content.

1. Video/audio is transcribed into text.
2. Transcript is split into chunks.
3. Chunks are converted into embeddings using Sentence Transformers.
4. Embeddings are stored in ChromaDB.
5. User asks a question.
6. Relevant transcript chunks are retrieved.
7. Retrieved context is sent to Mistral AI.
8. Mistral generates a grounded response.

• Reduces hallucinations
• Improves factual accuracy
• Enables semantic search
• Provides context-aware responses

• OpenAI Whisper (Local)

Default model:

WHISPER_MODEL=small

Runs locally without external API calls.

• Sarvam AI Speech-to-Text API

Sarvam transcribes Hindi/Hinglish audio and translates it into English for downstream processing.

• Mistral AI

Used for:

• Title generation
• Summary generation
• Question answering
• RAG responses

• LangChain
• ChromaDB
• Sentence Transformers

• Streamlit

• yt-dlp
• FFmpeg
• pydub

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

curl -LsSf https://astral.sh/uv/install.sh | sh

Verify installation:

uv --version

git clone <your-repository-url>
cd AI-Video-Assistant

uv venv

Activate:

venv\Scripts\activate

source venv/bin/activate

uv pip install -r requirements.txt

FFmpeg is required for audio extraction and processing.

1. Download FFmpeg from:

https://ffmpeg.org/download.html

2. Extract the archive.

3. Add the bin folder to your PATH.

4. Verify installation:

ffmpeg -version

Create a .env file in the project root:

MISTRAL_API_KEY=your_mistral_api_key

SARVAM_API_KEY=your_sarvam_api_key

Launch the web application:

streamlit run app.py

Open:

http://localhost:8501

Run:

python main.py

You will be prompted:

Enter YouTube URL or local file path:
Language (english/hinglish):

Example:

Enter YouTube URL or local file path:
https://youtu.be/example

Language:
english

Paste a YouTube URL or provide a local file path.

Choose:

• english → Whisper
• hinglish → Sarvam AI

Click Analyse.

The application performs:

• Audio Extraction
• Audio Chunking
• Speech-to-Text
• Transcript Generation
• Embedding Creation
• Vector Store Creation
• Summary Generation
• RAG Initialization

After processing, the system generates:

• Session Title
• Transcript Summary
• Key Insights

Ask questions directly from the transcript.

Example:

What are the key points discussed in this video?

The system retrieves relevant transcript chunks and generates grounded answers using Mistral AI.

AI-Video-Assistant/
│
├── app.py
├── main.py
├── requirements.txt
├── .env
│
├── core/
│   ├── transcriber.py
│   ├── summarizer.py
│   ├── rag_engine.py
│   └── vector_store.py
│
├── utils/
│   └── audio_processor.py
│
├── vector_db/
│
└── screenshots/
    ├── home.png
    ├── pipeline-running.png
    ├── summary.png
    └── rag-chat.png

1. User enters a YouTube URL or local file path.
2. Audio is extracted.
3. Audio is chunked.
4. Transcription is performed.
5. Transcript is generated.
6. Transcript is embedded.
7. Embeddings are stored in ChromaDB.
8. Mistral AI generates a title and summary.
9. RAG pipeline is created.
10. User can chat with the transcript.

Ensure:

SARVAM_API_KEY=your_key_here

exists in the .env file.

Verify:

ffmpeg -version

returns a valid version.

Install Node.js:

node -v

This improves YouTube extraction compatibility.

Whisper runs locally.

For faster inference:

WHISPER_MODEL=tiny

• Speaker diarization
• PDF export
• Meeting minutes generation
• Timestamp search
• Multi-language support
• Real-time meeting transcription
• Cloud deployment
• Authentication
• Chat history