AI vs. Human Text Detection

A text classifier that decides whether a passage was written by a person or generated by AI. It trains and compares six models (three traditional machine learning, three deep learning) and serves them through a Streamlit web app that accepts a PDF, Word document, or pasted text and returns a prediction with a calibrated confidence score and an explanation of why.

Built for Intro to Large Language Models, Summer I 2026.

Held-out test set (1,226 passages the models never saw during training or tuning):

The app uses the SVM by default: it is within a fraction of a percent of the best accuracy, returns calibrated probabilities, trains in seconds, and its linear weights make per-word explanations possible.

Paste text or upload a PDF or Word file, pick any of the six models, and get a verdict with calibrated confidence. Beyond a single number, the app shows a sentence-level heatmap of which parts read as AI, a word-level explanation, text statistics, and a side-by-side comparison of all six models.

Sentence-level highlighting (red leans AI, green leans human):

Side-by-side model comparison, with an agreement summary:

cd ai_human_detection_project
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m nltk.downloader punkt punkt_tab stopwords wordnet

Run the web app (the trained models are included, so this works immediately):

streamlit run app.py

Open the notebook to see the full analysis. To retrain the models, first place the labeled dataset at data/training_data/train_data_with_labels.xlsx, then:

jupyter lab notebooks/project1_notebook.ipynb

A rendered, read-only copy of the notebook is at docs/project1_notebook.html. Open it in any browser to see every chart and result without running anything.

ai_human_detection_project/
├── app.py                      # Streamlit web application
├── requirements.txt
├── README.md
├── models/                     # trained models, vectorizer, sequence vocabulary
├── data/
│   └── training_data/          # place the labeled dataset here to retrain
├── notebooks/
│   └── project1_notebook.ipynb # data exploration, features, training, evaluation
├── src/                        # shared code used by both notebook and app
│   ├── preprocessing.py        # loading and text cleaning
│   ├── features.py             # linguistic features
│   ├── inference.py            # load models, predict
│   ├── explain.py              # prediction explanations
│   ├── extract.py              # PDF/Word/text extraction
│   └── report.py               # downloadable report
├── docs/                       # rendered notebook
└── reports/                    # generated metrics and reports

The notebook and the app import the same code from src/, so the model logic they use is guaranteed identical.

The labeled dataset (train_data_with_labels.xlsx) is not bundled here. It is 8,176 passages with columns text and label (0 human, 1 AI); after removing duplicates and repairing character-encoding damage, 8,169 remain, balanced almost exactly 50/50. The app runs on the included trained models without it; place the file in data/training_data/ only if you want to retrain.

Stopwords and punctuation are kept, not removed. Distinguishing human from AI writing is a stylometry problem (telling writing styles apart), not topic classification. Function words and punctuation habits are among the strongest, most topic-independent signals, so removing them would discard the evidence. This is the same principle behind classic authorship attribution.

Encoding is repaired at load time. The raw data is full of double-encoded smart quotes (for example don’t for don't). Left alone these become garbage tokens and an artifact the classifier could exploit, so ftfy repairs them before anything else.

Length is treated carefully. AI passages are capped near 400 words while human passages run longer. To confirm the models learned style rather than this artifact, the notebook shows that a length-only baseline scores near chance (about 52%) while the best model stays at 98.8% accuracy on length-matched text.

Confidence is honest. The SVM is calibrated so its probabilities are meaningful; the app says "borderline" near 50% and surfaces model disagreement rather than presenting a single number as proof.

Three feature representations are compared. TF-IDF, Word2Vec embeddings, and handcrafted linguistic features. TF-IDF separated the classes best (about 98% vs. 88% for Word2Vec and 71% for the linguistic features with a simple classifier), which is why the traditional models are trained on it.

Traditional models were tuned with GridSearchCV (5-fold cross-validation) on TF-IDF features. Deep models used fixed seeds, dropout, and early stopping.

The deep models did not beat the traditional ones at this data scale, which is expected: LSTMs and CNNs learn word meaning and order from scratch and need far more than ~5,700 training passages to do so well. The notebook reports this rather than hiding it.

Automated AI-text detection is probabilistic and can be wrong. Published work shows detectors are biased against non-native English writers, are easily evaded by paraphrasing, and degrade on text from models they were not trained on. These classifiers separate this dataset well; they are a signal, not proof.

Python 3.12. All dependencies are pinned in requirements.txt (scikit-learn, TensorFlow/Keras, gensim, NLTK, Streamlit, and supporting libraries).