LLM4S Speech Module

A comprehensive speech recognition and text-to-speech synthesis module for the LLM4S project, built with Scala and functional programming principles.

Features

Speech Recognition (STT)

  • Vosk: Lightweight, offline speech recognition engine
  • Whisper: High-accuracy transcription via CLI integration
  • Audio Preprocessing: Resampling, channel conversion, silence trimming
  • Multiple Input Formats: File, bytes, and stream audio support
  • Cloud: OpenAISTTClient (Whisper API) and AzureSTTClient (Azure AI Speech), see Cloud providers

Text-to-Speech (TTS)

  • Tacotron2: Neural speech synthesis via CLI integration
  • Cloud: OpenAITTSClient, ElevenLabsTTSClient and AzureTTSClient, see Cloud providers
  • Voice Customization: Language, speaking rate, pitch, volume control
  • Output Formats: WAV and raw PCM16 audio support
  • Cross-platform: Works on Windows, Linux, and macOS

Architecture

The module follows functional programming principles with:

  • Result Types: Either[LLMError, T] for error handling
  • Pure Functions: Immutable audio transformations
  • ADTs: Algebraic Data Types for type-safe modeling
  • Composition: Functional composition for audio processing pipelines

Quick Start

Basic Usage 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import org.llm4s.speech._
import org.llm4s.speech.stt.{WhisperSpeechToText, STTOptions}
import org.llm4s.speech.tts.{Tacotron2TextToSpeech, TTSOptions}
import org.llm4s.speech.util.PlatformCommands

// Speech Recognition
val stt = new WhisperSpeechToText(PlatformCommands.mockSuccess)
val audioInput = AudioInput.FileAudio(Paths.get("audio.wav"))
val options = STTOptions(language = Some("en"))
val result = stt.transcribe(audioInput, options)

// Text-to-Speech
val tts = new Tacotron2TextToSpeech(PlatformCommands.echo)
val ttsOptions = TTSOptions(
  voice = Some("en-female"),
  language = Some("en"),
  speakingRate = Some(1.2)
)
val audio = tts.synthesize("Hello, world!", ttsOptions)

Audio Preprocessing 

1
2
3
4
5
6
7
8
9
10
11

import org.llm4s.speech.processing.AudioPreprocessing

val audioBytes = // ... your audio data
val audioMeta = AudioMeta(sampleRate = 44100, numChannels = 2, bitDepth = 16)

// Convert to mono, resample to 16kHz for STT
val processed = AudioPreprocessing.standardizeForSTT(
  audioBytes,
  audioMeta,
  targetRate = 16000
)

AudioPreprocessing.resamplePcm16 (the resampling step of standardizeForSTT) checks its arguments before it converts anything. The target rate and the source rate must be between 1 and 768000 Hz, the channel count between 1 and 64, and the bit depth a multiple of 8 from 8 to 32; anything else is a Left(ValidationError) naming the field (targetRate, source.sampleRate, source.numChannels or source.bitDepth). An output above 256 MiB (about 46 minutes of 48 kHz mono) is a ValidationError on targetRate too, checked before anything is allocated: resample a longer recording in pieces. The output has exactly round(frames * targetRate / sourceRate) frames, so empty input gives empty output and a trailing partial frame is ignored; a converter that delivers noticeably fewer frames than that is a ProcessingError, not a silently padded success. Java Sound’s converter delays the signal by a fraction of a millisecond, and the length fit drops the matching tail, so the source’s final fraction of a millisecond is not in the output.

Configuration

Vosk Configuration 

1
2
3
4
5
6
7

import org.llm4s.speech.stt.VoskSpeechToText

// Use default English model
val stt = new VoskSpeechToText()

// Use custom model path
val stt = new VoskSpeechToText(modelPath = Some("/path/to/vosk-model"))

Environment Variables 

1
2

# Vosk Model Path (optional)
VOSK_MODEL_PATH=/path/to/vosk-model

File Structure 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

src/main/scala/org/llm4s/speech/
├── Audio.scala                    # Core audio data structures
├── stt/                          # Speech-to-Text implementations
│   ├── SpeechToText.scala        # STT trait interface
│   ├── VoskSpeechToText.scala    # Vosk integration
│   └── WhisperSpeechToText.scala # Whisper CLI integration
├── tts/                          # Text-to-Speech implementations
│   ├── TextToSpeech.scala        # TTS trait interface
│   └── Tacotron2TextToSpeech.scala # Tacotron2 CLI integration
├── processing/                    # Audio preprocessing utilities
│   └── AudioPreprocessing.scala  # Audio transformation functions
├── io/                           # Audio I/O operations
│   └── AudioIO.scala             # File saving utilities
└── util/                         # Cross-platform utilities
    └── PlatformCommands.scala    # OS-agnostic command helpers

Cross-Platform Support

The PlatformCommands utility automatically provides the right commands:

Platform Echo File Reader Directory Listing
Windows cmd /c echo cmd /c type cmd /c dir
POSIX echo cat ls

External Tools

Whisper

  • Installation: pip install openai-whisper
  • Usage: The module integrates with Whisper CLI for transcription
  • Models: Supports various model sizes (tiny, base, small, medium, large)

Tacotron2

  • Installation: Requires Tacotron2 CLI tool
  • Usage: The module integrates with Tacotron2 CLI for synthesis
  • Features: Voice customization, language support, audio output

Cloud providers

Three cloud TTS clients and two cloud STT clients live in org.llm4s.speech.tts.provider and org.llm4s.speech.stt.provider. They are selected like chat models, with a provider/model string, and built through SpeechProviderSelector:

Setting Values
SPEECH_TTS_MODEL openai/tts-1, openai/tts-1-hd, elevenlabs/<voice-id>, azure/<voice-name> (e.g. azure/en-US-JennyNeural)
SPEECH_TTS_VOICE optional voice override (OpenAI default alloy)
SPEECH_STT_MODEL openai/whisper-1, azure/<default-language> (e.g. azure/en-US)
OPENAI_API_KEY OpenAI TTS and STT
ELEVENLABS_API_KEY ElevenLabs TTS (optional ELEVENLABS_MODEL_ID, default eleven_multilingual_v2)
AZURE_SPEECH_KEY, AZURE_SPEECH_REGION Azure TTS and STT

Only the selected provider’s credentials are needed. The same keys exist as HOCON under llm4s.speech.* (see this module’s reference.conf); OPENAI_SPEECH_BASE_URL, ELEVENLABS_BASE_URL, AZURE_SPEECH_TTS_BASE_URL and AZURE_SPEECH_STT_BASE_URL override the endpoints.

1
2
3
4
5
6
7
8
9
10
11

import org.llm4s.speech.SpeechProviderSelector
import org.llm4s.speech.tts.TTSOptions
import org.llm4s.speech.stt.STTOptions
import org.llm4s.speech.AudioInput

for {
  tts   <- SpeechProviderSelector.tts()   // reads llm4s.speech.tts / SPEECH_TTS_MODEL
  audio <- tts.synthesize("Hello from LLM4S")
  stt   <- SpeechProviderSelector.stt()   // reads llm4s.speech.stt / SPEECH_STT_MODEL
  text  <- stt.transcribe(AudioInput.FileAudio(java.nio.file.Paths.get("hello.wav")), STTOptions())
} yield text.text

To pick the model in code instead of SPEECH_*_MODEL, use SpeechConfigLoader.tts("openai/tts-1") / .stt("openai/whisper-1") (credentials still come from config) and SpeechProviderSelector.getTTSClient / getSTTClient.

Behaviour worth knowing:

  • Audio is raw PCM. The TTS clients request raw 24 kHz, 16-bit, mono PCM from each service, so GeneratedAudio.data is headerless PCM with an accurate AudioMeta, the same shape Tacotron2 produces. Write a playable file with WavFileGenerator.saveAsWav(audio, path). The default GeneratedAudio.format is labelled AudioFormat.WavPcm16 but, as above, the bytes carry no WAV header.
  • MP3 is opt-in. TTSOptions(outputFormat = AudioFormat.Mp3) makes the OpenAI, ElevenLabs and Azure clients request the service’s MP3 (response_format=mp3, output_format=mp3_44100_128, audio-24khz-48kbitrate-mono-mp3) and return its bytes untouched: no decoding, no resampling. GeneratedAudio.format is AudioFormat.Mp3, so the PCM helpers (WavFileGenerator.saveAsWav, AudioIO.saveWav / saveRawPcm16, AudioPreprocessing.standardizeForSTT(audio, rate)) refuse it with a ValidationError; save it with AudioIO.saveMp3(audio, path). AudioMeta carries the requested sample rate and channel count as nominal labels (not verified against the service) and bitDepth = 0: MP3 has no sample width, so do not compute a duration from the byte count. Tacotron2 produces PCM only and refuses MP3. The default stays PCM.
  • OpenAI TTS limits are checked locally: text over 4096 characters and a speakingRate outside 0.25 to 4.0 are a ValidationError before any request is sent. Split long text yourself; the clients do not chunk it.
  • STT input is WAV. BytesAudio and StreamAudio are treated as WAV data, as for Whisper and Vosk. Azure’s REST endpoint recognises about 60 seconds of audio per request.
  • Errors map like the chat providers: HTTP 401/403 is AuthenticationError, 429 is RateLimitError (with Retry-After), 400 is ValidationError, other statuses are ServiceError, and a timeout or refused connection is TimeoutError / NetworkError.
  • The clients take an Llm4sHttpClient, so tests inject a stub and never touch the network.

Error Handling

The module uses Result[T] (alias for Either[LLMError, T]) for robust error handling:

1
2
3
4
5
6
7
8

val result: Result[Transcription] = stt.transcribe(audioInput, options)

result match {
  case Right(transcription) =>
    println(s"Transcript: ${transcription.text}")
  case Left(error) =>
    println(s"Error: ${error.formatted}")
}

Testing

The module includes comprehensive tests that work cross-platform:

1
2
3
4
5
6
7
8

# Run all tests
sbt test

# Run specific test suites
sbt "testOnly org.llm4s.speech.*"

# Run the current supported Scala build
sbt test

The cloud clients are covered without a network by speech/test (stubbed HTTP layer, including CloudSpeechProviderIntegrationSpec). Live-API smoke suites, one per provider, are in modules/it (org.llm4s.speech.*, tier @Cloud): run them with sbt testSmoke and OPENAI_API_KEY, ELEVENLABS_API_KEY, AZURE_SPEECH_KEY / AZURE_SPEECH_REGION set. They make billed calls; a suite whose key is missing is skipped, or fails under LLM4S_IT_STRICT=true.

See Also