Advanced music analysis platform with AI-powered chord recognition, beat detection, and synchronized lyrics.

Clean, intuitive interface for YouTube search, URL input, and recent video access.

Chord progression visualization with synchronized beat detection and grid layout with add-on features: Roman Numeral Analysis, Key Modulation Signals, Simplified Chord Notation, and Enhanced Chord Correction.

Interactive guitar chord diagrams with accurate fingering patterns from the official @tombatossals/chords-db database, featuring multiple chord positions and synchronized beat grid integration.

Synchronized lyrics transcription with AI chatbot for contextual music analysis and translation support.

• Node.js 18+ and npm
• Python 3.9+ (for backend)
• Firebase account (free tier)

1. Clone and install Clone with submodules in one command (for fresh clones)

   git clone --recursive https://github.com/ptnghia-j/ChordMiniApp.git
   cd ChordMiniApp
   npm install

   Verify that submodules are populated

   ls -la python_backend/models/Beat-Transformer/
   ls -la python_backend/models/Chord-CNN-LSTM/
   ls -la python_backend/models/ChordMini/
   

   If chord recognition encounters issue with fluidsynth:

   Install FluidSynth for MIDI synthesis

      
   # --- Windows ---
   choco install fluidsynth
   
   # --- macOS ---
   brew install fluidsynth
   
   # --- Linux (Debian/Ubuntu-based) ---
   sudo apt update
   sudo apt install fluidsynth
   

2. Environment setup

   cp .env.example .env.local

   Edit .env.local:

   NEXT_PUBLIC_PYTHON_API_URL=http://localhost:5001
   NEXT_PUBLIC_FIREBASE_API_KEY=your_firebase_api_key
   NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN=your_project.firebaseapp.com
   NEXT_PUBLIC_FIREBASE_PROJECT_ID=your_project_id
   NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET=your_project.appspot.com
   NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID=your_sender_id
   NEXT_PUBLIC_FIREBASE_APP_ID=your_app_id

3. Start Python backend (Terminal 1)

   cd python_backend
   python -m venv myenv
   source myenv/bin/activate  # On Windows: myenv\Scripts\activate
   pip install -r requirements.txt
   python app.py

4. Start frontend (Terminal 2)

   npm run dev

5. Open application

   Visit http://localhost:3000

• Docker and Docker Compose installed (Get Docker)
• Firebase account with API keys configured

1. Download configuration files

   curl -O https://raw.githubusercontent.com/ptnghia-j/ChordMiniApp/main/docker-compose.prod.yml
   curl -O https://raw.githubusercontent.com/ptnghia-j/ChordMiniApp/main/.env.docker.example

2. Configure environment

   cp .env.docker.example .env.docker
   # Edit .env.docker with your API keys (see API Keys Setup section below)

3. Start the application

   docker compose -f docker-compose.prod.yml --env-file .env.docker up -d

4. Access the application

   Visit http://localhost:3000

5. Stop the application

   docker compose -f docker-compose.prod.yml down

Note: If you have Docker Compose V1 installed, use docker-compose (with hyphen) instead of docker compose (with space).

If you prefer using Docker Desktop GUI:

1. Open Docker Desktop
2. Go to "Images" tab and search for ptnghia/chordminiapp-frontend and ptnghia/chordminiapp-backend
3. Pull both images
4. Use the "Containers" tab to manage running containers

Edit .env.docker with these required values:

• NEXT_PUBLIC_FIREBASE_API_KEY - Firebase API key
• NEXT_PUBLIC_FIREBASE_PROJECT_ID - Firebase project ID
• NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET - Firebase storage bucket
• NEXT_PUBLIC_YOUTUBE_API_KEY - YouTube Data API v3 key
• MUSIC_AI_API_KEY - Music.AI API key
• GEMINI_API_KEY - Google Gemini API key
• GENIUS_API_KEY - Genius API key

See the API Keys Setup section below for detailed instructions on obtaining these keys.

1. Create Firebase project

   • Visit Firebase Console
   • Click "Create a project"
   • Follow the setup wizard

2. Enable Firestore Database

   • Go to "Firestore Database" in the sidebar
   • Click "Create database"
   • Choose "Start in test mode" for development

3. Get Firebase configuration

   • Go to Project Settings (gear icon)
   • Scroll down to "Your apps"
   • Click "Add app" → Web app
   • Copy the configuration values to your .env.local

4. Create Firestore collections

   The app uses the following Firestore collections. They are created automatically on first write (no manual creation required):

   • transcriptions — Beat and chord analysis results (docId: ${videoId}_${beatModel}_${chordModel})
   • translations — Lyrics translation cache (docId: cacheKey based on content hash)
   • lyrics — Music.ai transcription results (docId: videoId)
   • keyDetections — Musical key analysis cache (docId: cacheKey)
   • audioFiles — Audio file metadata and URLs (docId: videoId)

5. Enable Anonymous Authentication

   • In Firebase Console: Authentication → Sign-in method → enable Anonymous

6. Configure Firebase Storage

   • Set environment variable: NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET=your_project_id.appspot.com
   • Folder structure:
     • audio/ for audio files
     • video/ for optional video files
   • Filename pattern requirement: filenames must include the 11-character YouTube video ID in brackets, e.g. audio_[VIDEOID]_timestamp.mp3 (enforced by Storage rules)
   • File size limits (enforced by Storage rules):
     • Audio: up to 50MB
     • Video: up to 100MB

# 1. Sign up at music.ai
# 2. Get API key from dashboard
# 3. Add to .env.local
NEXT_PUBLIC_MUSIC_AI_API_KEY=your_key_here

# 1. Visit Google AI Studio
# 2. Generate API key
# 3. Add to .env.local
NEXT_PUBLIC_GEMINI_API_KEY=your_key_here

To be added in next version...

ChordMiniApp uses a hybrid backend architecture:

For local development, you must run the Python backend on localhost:5001:

• URL: http://localhost:5001
• Port Note: Uses port 5001 to avoid conflict with macOS AirPlay/AirTunes service on port 5000

Production deployments is configured based on your VPS and url should be set in the NEXT_PUBLIC_PYTHON_API_URL environment variable.

• Python 3.9+ (Python 3.9-3.11 recommended)
• Virtual environment (venv or conda)
• Git for cloning dependencies
• System dependencies (varies by OS)

1. Navigate to backend directory

   cd python_backend

2. Create virtual environment

   python -m venv myenv
   
   # Activate virtual environment
   # On macOS/Linux:
   source myenv/bin/activate
   
   # On Windows:
   myenv\Scripts\activate

3. Install dependencies

   pip install --no-cache-dir Cython>=0.29.0 numpy==1.22.4
   pip install --no-cache-dir madmom>=0.16.1
   pip install --no-cache-dir -r requirements.txt

4. Start local backend on port 5001

   python app.py

   The backend will start on http://localhost:5001 and should display:

   Starting Flask app on port 5001
   App is ready to serve requests
   Note: Using port 5001 to avoid conflict with macOS AirPlay/AirTunes on port 5000
   

5. Verify backend is running

   Open a new terminal and test the backend:

   curl http://localhost:5001/health
   # Should return: {"status": "healthy"}

6. Start frontend development server

   # In the main project directory (new terminal)
   npm run dev

   The frontend will automatically connect to http://localhost:5001 based on your .env.local configuration.

• Beat Detection: Beat-Transformer and madmom models
• Chord Recognition: Chord-CNN-LSTM, BTC-SL, BTC-PL models
• Lyrics Processing: Genius.com integration
• Rate Limiting: IP-based rate limiting with Flask-Limiter
• Audio Processing: Support for MP3, WAV, FLAC formats

Create a .env file in the python_backend directory:

# Optional: Redis URL for distributed rate limiting
REDIS_URL=redis://localhost:6379

# Optional: Genius API for lyrics
GENIUS_ACCESS_TOKEN=your_genius_token

# Flask configuration
FLASK_MAX_CONTENT_LENGTH_MB=150
CORS_ORIGINS=http://localhost:3000,http://127.0.0.1:3000

Backend connectivity issues:

# 1. Verify backend is running
curl http://localhost:5001/health
# Expected: {"status": "healthy"}

# 2. Check if port 5001 is in use
lsof -i :5001  # macOS/Linux
netstat -ano | findstr :5001  # Windows

# 3. Verify environment configuration
cat .env.local | grep PYTHON_API_URL
# Expected: NEXT_PUBLIC_PYTHON_API_URL=http://localhost:5001

# 4. Check for macOS AirTunes conflict (if using port 5000)
curl -I http://localhost:5000/health
# If you see "Server: AirTunes", that's the conflict we're avoiding

Frontend connection errors:

# Check browser console for errors like:
# "Failed to fetch" or "Network Error"
# This usually means the backend is not running on port 5001

# Restart both frontend and backend:
# Terminal 1 (Backend):
cd python_backend && python app.py

# Terminal 2 (Frontend):
npm run dev

Import errors:

# Ensure virtual environment is activated
source myenv/bin/activate  # macOS/Linux
myenv\Scripts\activate     # Windows

# Reinstall dependencies
pip install -r requirements.txt

ChordMini provides a comprehensive music analysis workflow from user input to visualization. This diagram shows the complete user journey and data processing pipeline:

graph TD
    %% User Entry Points
    A[User Input] --> B{Input Type}
    B -->|YouTube URL/Search| C[YouTube Search Interface]
    B -->|Direct Audio File| D[File Upload Interface]

    %% YouTube Workflow
    C --> E[YouTube Search API]
    E --> F[Video Selection]
    F --> G[Navigate to /analyze/videoId]

    %% Direct Upload Workflow
    D --> H[Vercel Blob Upload]
    H --> I[Navigate to /analyze with blob URL]

    %% Cache Check Phase
    G --> J{Firebase Cache Check}
    J -->|Cache Hit| K[Load Cached Results]
    J -->|Cache Miss| L[Audio Extraction Pipeline]

    %% Audio Extraction (Environment-Aware)
    L --> M{Environment Detection}
    M -->|Development| N[yt-dlp Service]
    M -->|Production| O[yt-mp3-go Service]
    N --> P[Audio URL Generation]
    O --> P

    %% ML Analysis Pipeline
    P --> Q[Parallel ML Processing]
    I --> Q
    Q --> R[Beat Detection]
    Q --> S[Chord Recognition]
    Q --> T[Key Detection]

    %% ML Backend Processing
    R --> U[Beat-Transformer/madmom Models]
    S --> V[Chord-CNN-LSTM/BTC Models]
    T --> W[Gemini AI Key Analysis]

    %% Results Processing
    U --> X[Beat Data]
    V --> Y[Chord Progression Data]
    W --> Z[Musical Key & Corrections]

    %% Synchronization & Caching
    X --> AA[Chord-Beat Synchronization]
    Y --> AA
    Z --> AA
    AA --> BB[Firebase Result Caching]
    AA --> CC[Real-time Visualization]

    %% Visualization Tabs
    CC --> DD[Beat & Chord Map Tab]
    CC --> EE[Guitar Chords Tab - Beta]
    CC --> FF[Lyrics & Chords Tab - Beta]

    %% Beat & Chord Map Features
    DD --> GG[Interactive Chord Grid]
    DD --> HH[Synchronized Audio Player]
    DD --> II[Real-time Beat Highlighting]

    %% Guitar Chords Features
    EE --> JJ[Animated Chord Progression]
    EE --> KK[Guitar Chord Diagrams]
    EE --> LL[Responsive Design 7/5/3/2/1]

    %% Lyrics Features
    FF --> MM[Lyrics Transcription]
    FF --> NN[Multi-language Translation]
    FF --> OO[Word-level Synchronization]

    %% External API Integration
    MM --> PP[Music.ai Transcription]
    NN --> QQ[Gemini AI Translation]

    %% Cache Integration
    K --> CC
    BB --> RR[Enhanced Metadata Storage]

    %% Error Handling & Fallbacks
    L --> SS{Extraction Failed?}
    SS -->|Yes| TT[Fallback Service]
    SS -->|No| P
    TT --> P

    %% Styling
    style A fill:#e1f5fe,stroke:#1976d2,stroke-width:3px
    style CC fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
    style EE fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style FF fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style Q fill:#ffebee,stroke:#d32f2f,stroke-width:2px
    style AA fill:#e8f5e8,stroke:#4caf50,stroke-width:2px

    %% Environment Indicators
    style N fill:#e3f2fd,stroke:#1976d2
    style O fill:#e8f5e8,stroke:#4caf50

• YouTube Integration: URL/search → video selection → analysis
• Direct Upload: Audio file → blob storage → immediate analysis

• Development: localhost:5001 Python backend with yt-dlp (avoiding macOS AirTunes port conflict)
• Production: Google Cloud Run backend with yt-mp3-go

• Firebase Cache: Analysis results with enhanced metadata
• Cache Hit: Instant loading of previous analyses
• Cache Miss: Full ML processing pipeline

• Parallel Processing: Beat detection + chord recognition + key analysis
• Multiple Models: Beat-Transformer/madmom, Chord-CNN-LSTM/BTC variants
• AI Integration: Gemini AI for key detection and enharmonic corrections

• Beat & Chord Map: Interactive grid with synchronized playback
• Guitar Chords [Beta]: Responsive chord diagrams (7/5/3/2/1 layouts)
• Lyrics & Chords [Beta]: Multi-language transcription with word-level sync

• Next.js 15.3.1 - React framework with App Router
• React 19 - UI library with latest features
• TypeScript - Type-safe development
• Tailwind CSS - Utility-first styling

• Framer Motion - Smooth animations and transitions
• Chart.js - Data visualization for audio analysis
• @tombatossals/react-chords - Guitar chord diagram visualization with official chord database
• React Player - Video playback integration

• Python Flask - Lightweight backend framework
• Google Cloud Run - Serverless container deployment
• Custom ML Models - Chord recognition and beat detection

• Firebase Firestore - NoSQL database for caching
• Vercel Blob - File storage for audio processing

• YouTube Search API - github.com/damonwonghv/youtube-search-api
• yt-dlp - github.com/yt-dlp/yt-dlp - YouTube audio extraction
• yt-mp3-go - github.com/vukan322/yt-mp3-go - Alternative audio extraction
• LRClib - github.com/tranxuanthang/lrclib - Lyrics synchronization
• Music.ai SDK - AI-powered music transcription
• Google Gemini API - AI language model for translations

• Beat Detection - Automatic tempo and beat tracking
• Chord Recognition - AI-powered chord progression analysis
• Key Detection - Musical key identification with Gemini AI

• Accurate Chord Database Integration - Official @tombatossals/chords-db with verified chord fingering patterns
• Enhanced Chord Recognition - Support for both ML model colon notation (C:minor) and standard notation (Cm)
• Interactive Chord Diagrams - Visual guitar fingering patterns with correct fret positions and finger placements
• Responsive Design - Adaptive chord count (7/5/3/2/1 for xl/lg/md/sm/xs)
• Smooth Animations - transitions with optimized scaling
• Unicode Notation - Proper musical symbols (♯, ♭) with enharmonic equivalents

• Synchronized Lyrics - Time-aligned lyrics display
• Multi-language Support - Translation with Gemini AI
• Word-level Timing - Precise synchronization with Music.ai

• Dark/Light Mode - Automatic theme switching
• Responsive Design - Mobile-first approach
• Performance Optimized - Lazy loading and code splitting
• Offline Support - Service worker for caching

ChordMiniApp provides production-ready Docker images for easy deployment:

# Production deployment with Docker Compose
curl -O https://raw.githubusercontent.com/ptnghia-j/ChordMiniApp/main/docker-compose.prod.yml
docker compose -f docker-compose.prod.yml up -d

Available on multiple registries:

• Docker Hub: ptnghia/chordmini-frontend:latest, ptnghia/chordmini-backend:latest
• GitHub Container Registry: ghcr.io/ptnghia-j/chordminiapp/frontend:latest

Deployment targets:

• Cloud platforms: AWS, GCP, Azure, DigitalOcean
• Container orchestration: Kubernetes, Docker Swarm
• Edge computing: Raspberry Pi 4+ (ARM64 support)

For custom deployments, see the Local Setup section above.

We welcome contributions! Please see our Contributing Guidelines for details.