MI Session Processing Pipeline

This repository converts recorded BMI/BAI and SFAS counseling sessions into clean, session-level CSV files. It contains four processing steps:

audio
  -> Step 1: transcript
  -> Step 2: therapist/patient roles
  -> Step 3: MI-coded segments
  -> Step 4: session components and final clean CSV

No participant data, audio, transcripts, generated CSVs, model files, tokens, or cluster-specific private information should be committed to this repository.

Each session produces:

cleaned-final/<session_id>/<session_id>.cleaned_step4.csv

The final columns are:

segment_id, role, behavior, motivation, time, text, session_component

• role: therapist or patient
• behavior: th, tb, alt, oth, or nr
• motivation: - (avoid), = (neutral), or + (approach)
• time: P (past), C (current), or F (future)
• session_component: a predefined BMI/BAI or SFAS intervention component

• Python 3
• WhisperX and PyTorch for Step 1
• An OpenAI-compatible language-model endpoint for Steps 2-4
• Slurm if using the included .sh job scripts

Install the Python packages required by the scripts in your own environments. Do not store API tokens or private model paths in this repository.

Steps 2-4 can use a locally deployed model through vLLM's OpenAI-compatible API. Recommended model families for this pipeline are:

• Qwen3.5
• Gemma 4

Start with a smaller instruction-tuned checkpoint and evaluate it on a few representative sessions. Use a larger checkpoint only if the smaller model does not label speaker roles, MI codes, or session-component boundaries reliably. For transcript-only processing, Qwen3.5 can be served in language-model-only mode to avoid loading unused multimodal components.

Example single-node server:

vllm serve "$MODEL_PATH" \
  --host 0.0.0.0 \
  --port 8000 \
  --language-model-only

Then configure the pipeline:

export MODEL_PATH="/path/to/instruction-model"
export OPENAI_API_BASE="http://localhost:8000/v1"

Purpose: transcribe each recording and optionally identify speakers.

Input:

audio files

Default output:

step1_output/<dataset>/<session_id>/<session_id>.txt

Run:

export AUDIO_DIR="/path/to/audio"
export HF_TOKEN="your-hugging-face-token"
export DATASET_NAME="sessions"
sbatch step1.sh

Step 1 writes only transcript text by default. Use --save_debug_outputs when JSON and segment-level debug CSVs are needed.

Purpose: assign every transcript line to either therapist or patient. Unreliable diarization labels such as SPEAKER_00 are not preserved.

Input:

step1_output/**/*.txt

Output:

speaker_corrected_txt/<session_id>/<session_id>.speaker_corrected.csv

Run:

sbatch step2.sh

Purpose: code patient speech using the MI Quick Key and create analytically consistent segments.

• Consecutive therapist speech is merged.
• Patient speech is split whenever behavior, motivation, or time changes.
• Only patient segments coded tb, alt, or oth receive motivation and time ratings.

Input:

speaker_corrected_txt/**/*.speaker_corrected.csv

Output:

speaker_merged_turns/<session_id>/<session_id>.coded_segments.csv

Run:

export SEGMENTATION_UNIT="punctuation"
sbatch step3.sh

SEGMENTATION_UNIT=none labels the existing units directly. SEGMENTATION_UNIT=punctuation splits by sentence-ending punctuation, labels the pieces, and merges adjacent pieces with identical codes.

Purpose: identify predefined intervention components and propagate each component name from its starting segment_id to the next component boundary. This is protocol-component segmentation, not open-ended topic generation.

BMI/BAI and SFAS use separate prompts containing:

• the allowed session components
• full component definitions
• canonical opening patterns
• strict boundary rules

Input:

speaker_merged_turns/**/*.coded_segments.csv

Default output:

cleaned-final/<session_id>/<session_id>.cleaned_step4.csv

Run:

sbatch step4.sh

The optional --write_full_output flag also writes debugging metadata under component_segments/. It is not needed for the normal final output.

The shell scripts use environment variables instead of private hardcoded paths. Important variables include:

AUDIO_DIR
HF_TOKEN
MODEL_PATH
OPENAI_API_BASE
STEP1_OUT_DIR
STEP2_OUT_DIR
STEP3_OUT_DIR
FINAL_OUT_DIR

Review each .sh file before submission and adjust Slurm resources, environments, model paths, and endpoint settings for the local system.