Skip to content
forked from m-bain/whisperX

WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)

License

Notifications You must be signed in to change notification settings

pepinu/whisperX

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

WhisperX

GitHub stars GitHub issues GitHub license Twitter

What is it β€’ Setup β€’ Usage β€’ Multilingual β€’ Contribute β€’ More examples β€’ Paper

whisperx-arch

Whisper-Based Automatic Speech Recognition (ASR) with improved timestamp accuracy + quality via forced phoneme alignment and speech-activity batching.

What is it πŸ”Ž

This repository refines the timestamps of openAI's Whisper model via forced aligment with phoneme-based ASR models (e.g. wav2vec2.0) and VAD preprocesssing, multilingual use-case.

Whisper is an ASR model developed by OpenAI, trained on a large dataset of diverse audio. Whilst it does produces highly accurate transcriptions, the corresponding timestamps are at the utterance-level, not per word, and can be inaccurate by several seconds.

Phoneme-Based ASR A suite of models finetuned to recognise the smallest unit of speech distinguishing one word from another, e.g. the element p in "tap". A popular example model is wav2vec2.0.

Forced Alignment refers to the process by which orthographic transcriptions are aligned to audio recordings to automatically generate phone level segmentation.

Voice Activity Detection (VAD) is the detection of the presence or absence of human speech.

New🚨

  • v3 released, 70x speed-up open-sourced. Using batched whisper with faster-whisper backend!
  • v2 released, code cleanup, imports whisper library, batched inference from paper not included (contact for licensing / batched model API). VAD filtering is now turned on by default, as in the paper.
  • Paper dropπŸŽ“πŸ‘¨β€πŸ«! Please see our ArxiV preprint for benchmarking and details of WhisperX. We also introduce more efficient batch inference resulting in large-v2 with *60-70x REAL TIME speed (not provided in this repo).
  • VAD filtering: Voice Activity Detection (VAD) from Pyannote.audio is used as a preprocessing step to remove reliance on whisper timestamps and only transcribe audio segments containing speech. add --vad_filter True flag, increases timestamp accuracy and robustness (requires more GPU mem due to 30s inputs in wav2vec2)
  • Character level timestamps (see *.char.ass file output)
  • Diarization (still in beta, add --diarize)

Setup βš™οΈ

Tested for PyTorch 0.11, Python 3.8 (use other versions at your own risk!)

GPU execution requires the NVIDIA libraries cuBLAS 11.x and cuDNN 8.x to be installed on the system. Please refer to the CTranslate2 documentation.

1. Create Python3.8 environment

conda create --name whisperx python=3.8

conda activate whisperx

2. Install PyTorch 0.11.0, e.g. for Linux and Windows:

conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch

See other methods here.

3. Install this repo

pip install git+https://github.com/m-bain/whisperx.git@v3

If already installed, update package to most recent commit

pip install git+https://github.com/m-bain/whisperx.git --upgrade

If wishing to modify this package, clone and install in editable mode:

$ git clone https://github.com/m-bain/whisperX.git
$ cd whisperX
$ pip install -e .

You may also need to install ffmpeg, rust etc. Follow openAI instructions here https://github.com/openai/whisper#setup.

Speaker Diarization

To enable Speaker. Diarization, include your Hugging Face access token that you can generate from Here after the --hf_token argument and accept the user agreement for the following models: Segmentation , Voice Activity Detection (VAD) , and Speaker Diarization

Usage πŸ’¬ (command line)

English

Run whisper on example segment (using default params)

whisperx examples/sample01.wav

For increased timestamp accuracy, at the cost of higher gpu mem, use bigger models (bigger alignment model not found to be that helpful, see paper) e.g.

whisperx examples/sample01.wav --model large-v2 --align_model WAV2VEC2_ASR_LARGE_LV60K_960H

Result using WhisperX with forced alignment to wav2vec2.0 large:

sample01.mp4

Compare this to original whisper out the box, where many transcriptions are out of sync:

sample_whisper_og.mov

Other languages

The phoneme ASR alignment model is language-specific, for tested languages these models are automatically picked from torchaudio pipelines or huggingface. Just pass in the --language code, and use the whisper --model large.

Currently default models provided for {en, fr, de, es, it, ja, zh, nl, uk, pt}. If the detected language is not in this list, you need to find a phoneme-based ASR model from huggingface model hub and test it on your data.

E.g. German

whisperx --model large --language de examples/sample_de_01.wav
sample_de_01_vis.mov

See more examples in other languages here.

Python usage 🐍

import whisperx

device = "cuda" 
audio_file = "audio.mp3"

# transcribe with original whisper
model = whisperx.load_model("large-v2", device)

audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=8)

print(result["segments"]) # before alignment

# load alignment model and metadata
model_a, metadata = whisperx.load_align_model(language_code=result["language"], device=device)

# align whisper output
result_aligned = whisperx.align(result["segments"], model_a, metadata, audio, device)

print(result_aligned["segments"]) # after alignment
print(result_aligned["word_segments"]) # after alignment

Whisper Modifications

In addition to forced alignment, the following two modifications have been made to the whisper transcription method:

  1. --condition_on_prev_text is set to False by default (reduces hallucination)

Limitations ⚠️

  • Whisper normalises spoken numbers e.g. "fifty seven" to arabic numerals "57". Need to perform this normalization after alignment, so the phonemes can be aligned. Currently just ignores numbers.
  • If setting --vad_filter False, then whisperx assumes the initial whisper timestamps are accurate to some degree (within margin of 2 seconds, adjust if needed -- bigger margins more prone to alignment errors)
  • Overlapping speech is not handled particularly well by whisper nor whisperx
  • Diariazation is far from perfect.

Contribute πŸ§‘β€πŸ«

If you are multilingual, a major way you can contribute to this project is to find phoneme models on huggingface (or train your own) and test them on speech for the target language. If the results look good send a merge request and some examples showing its success.

The next major upgrade we are working on is whisper with speaker diarization, so if you have any experience on this please share.

Coming Soon πŸ—“

  • Multilingual init

  • Subtitle .ass output

  • Automatic align model selection based on language detection

  • Python usage

  • Character level timestamps

  • Incorporating speaker diarization

  • Model flush, for low gpu mem resources

  • Faster-whisper backend

  • Add benchmarking code (TEDLIUM for spd/WER & word segmentation)

  • Allow silero-vad as alternative VAD option

  • Add max-line etc. see (openai's whisper utils.py)

  • Improve diarization (word level). Harder than first thought...

Contact/Support πŸ“‡

Contact maxhbain@gmail.com for queries and licensing / early access to a model API with batched inference (transcribe 1hr audio in under 1min).

Buy Me A Coffee

Acknowledgements πŸ™

This work, and my PhD, is supported by the VGG (Visual Geometry Group) and the University of Oxford.

Of course, this is builds on openAI's whisper. And borrows important alignment code from PyTorch tutorial on forced alignment

Valuable VAD & Diarization Models from (pyannote.audio)[https://github.com/pyannote/pyannote-audio]

Great backend from (faster-whisper)[https://github.com/guillaumekln/faster-whisper] and (CTranslate2)[https://github.com/OpenNMT/CTranslate2]

Citation

If you use this in your research, please cite the paper:
@article{bain2022whisperx,
  title={WhisperX: Time-Accurate Speech Transcription of Long-Form Audio},
  author={Bain, Max and Huh, Jaesung and Han, Tengda and Zisserman, Andrew},
  journal={arXiv preprint, arXiv:2303.00747},
  year={2023}
}

About

WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%