Perplexity Calculator with GPT-2

A Python script that calculates text perplexity using OpenAI's GPT-2 language model. This project measures how well the model predicts text, comparing perplexity scores between clean and noisy text samples.

Perplexity is a measure of how well a language model predicts a sequence of words. Lower perplexity indicates that the model assigns higher probability to the actual text, suggesting the text is more predictable or closer to the model's training data. This script:

• Loads a pre-trained GPT-2 model and tokenizer
• Calculates perplexity for clean, well-formatted text
• Calculates perplexity for corrupted/noisy text
• Compares the results to demonstrate how text quality affects model predictions
• Tracks execution time with millisecond precision

• Python 3.7 or higher
• torch
• transformers

pip install torch transformers

# Create virtual environment
python -m venv venv

# Activate virtual environment
# On Windows:
.\venv\Scripts\Activate.ps1
# On macOS/Linux:
source venv/bin/activate

# Install dependencies
pip install torch transformers

Run the script from the command line:

python main.py

The script will output:

• Timestamps for each calculation step (Track 1, 2, 3)
• Perplexity score for the clean text
• Perplexity score for the noisy text

Example output:

Track 1: 2026-05-18 14:23:45.123
Track 2: 2026-05-18 14:23:52.456
Track 3: 2026-05-18 14:23:59.789
Clean text perplexity: 45.32
Noisy text perplexity: 82.14

1. Model Loading: The script loads the pre-trained GPT-2 model and tokenizer from Hugging Face
2. Tokenization: Input text is converted to tokens that the model understands
3. Perplexity Calculation:
   • The model generates predictions for the entire text
   • Cross-entropy loss is computed
   • Perplexity is calculated as e^loss
4. Comparison: By comparing clean vs. noisy text, you can see how text quality affects model predictions

• First run will download the GPT-2 model (~500 MB) - may take a few minutes
• Subsequent runs will use the cached model
• The model requires sufficient system memory (GPU recommended for faster processing)

This project demonstrates the use of OpenAI's GPT-2 model via the Hugging Face transformers library.