| Grade | |
|---|---|
🇬🇧 ENGLISH VERSION
This projet is dedicated to building a complete Data Science pipeline without using any ready-made machine learning libraries.
Project goals:
analyzethe Hogwarts students dataset,visualizethe data,- manually implement an equivalent of
describe, - build
logistic regressionfrom scratch, predicta student’s house.
- Custom
describe— manual computation of mean, std, quartiles, etc. - Data cleaning & preprocessing (per-house NaN filling, normalization)
- Visualization tools:
- Histograms
- Scatter plots
- Pair plots
- Fully manual logistic regression
- One-vs-All classification
- Batch GD (Batch Gradient Descent)
- SGD (Stochastic Gradient Descent)
- Mini-batch GD (Mini-batch Gradient Descent)
- Model serialization to JSON
- Prediction pipeline generating
houses.csvfor evaluation
Short description
Logistic Regression is a classification algorithm.
The model computes the probability of a student belonging to each house.
The house with the highest probability is selected as the prediction.
Since there are multiple houses, the task is divided into several binary classifiers.
- Gryffindor vs all
- Slytherin vs all
- Ravenclaw vs all
- Hufflepuff vs all
Each classifier is trained separately.
The algorithm updates the theta weights in order to reduce the model error.
new_theta = theta - learning_rate * gradient
- An Epoch is one full pass through the entire dataset.
- Batch GD: 1 iteration = the whole dataset
- SGD and Mini-batch: many theta updates happen within one epoch.
Batch Gradient Descent
The gradient is computed using the entire dataset.
- stable learning
- smooth loss curve
- slow
SGD (Stochastic Gradient Descent)
Theta is updated after each sample.
- faster
- less memory usage
Drawbacks: - noisy loss
- less stable convergence
Mini-batch Gradient Descent
A compromise between Batch GD and SGD.
- The dataset is split into small batches. Ex: batch_size = 32
- Preprocessing
- Normalization
- All features are scaled to the same range.
Formula:
x_norm = (x - mean) / std
- mean and std -> are saved after training and reused during prediction.
- NaN handling -> Missing values can be filled using the dataset mean or the house mean.
- Visualization (Histogram) -> Helps visualize the distribution of grades across houses.
- Scatter plot -> Helps visualize the relationship between two features.
- Pair plot -> Shows all relationships between numerical features.
make install
source ~/.bashrc
make dataThese commands will create a virtual environment (.venv),
install all dependencies from requirements.txt, and prepare the dataset.
make startmake describe
make hist
make scatter
make pairplot
make train
make predictmake describe-bonus
make train-sgd
make train-sgd-custom
make train-minibatch
make bonus-metricsnorminette_python --versionmake lint-check or
norminette_pythonUsage entering (.venv):
source .venv/bin/activate# Describe dataset
srcs/describe.py datasets/dataset_train.csv
# Histograms
srcs/histogram.py datasets/dataset_train.csv
# Scatter plots
srcs/scatter_plot.py datasets/dataset_train.csv
# Pairplot
srcs/pair_plot.py datasets/dataset_train.csv
# Train model (batch GD)
srcs/logreg_train.py datasets/dataset_train.csv
# Predict houses
srcs/logreg_predict.py outputs/models/weights.json datasets/dataset_test.csv
# Bonus
# Describe dataset bonus
srcs/describe.py datasets/dataset_train.csv --extra
# Train model (sgd)
srcs/logreg_train.py datasets/dataset_train.csv --optimizer sgd --epochs 50 --learning-rate 0.01
# Train model (minibatch gd)
srcs/logreg_train.py datasets/dataset_train.csv --optimizer minibatch --epochs 50 --batch-size 32 --learning-rate 0.05
# Bonus metrics
make bonus-metricsdeactivateUsage with python3 (srcs/*):
# Describe dataset
python3 srcs/describe.py datasets/dataset_train.csv
# Histograms
python3 srcs/histogram.py datasets/dataset_train.csv
# Scatter plots
python3 srcs/scatter_plot.py datasets/dataset_train.csv
# Pairplot
python3 srcs/pair_plot.py datasets/dataset_train.csv
# Train model
python3 srcs/logreg_train.py datasets/dataset_train.csv
# Predict houses
python3 srcs/logreg_predict.py outputs/models/weights.json datasets/dataset_test.csv
# Bonus
# Describe dataset bonus
python3 srcs/describe.py datasets/dataset_train.csv --extra
# Train model (sgd)
python3 srcs/logreg_train.py datasets/dataset_train.csv --optimizer sgd --epochs 50 --learning-rate 0.01
# Train model (minibatch gd)
python3 srcs/logreg_train.py datasets/dataset_train.csv --optimizer minibatch --epochs 50 --batch-size 32 --learning-rate 0.05
# Bonus metrics
make bonus-metricsnumpy,matplotlib– image & math utilitiespandas– CSV loading and DataFrame container only; no pandas statistical helpers are used for mandatory describe/logistic regression.seaborn– data visualizationstreamlit– interactive interface for readme
🇫🇷 VERSION FRANÇAISE
Ce projet est consacré à la création d’un pipeline complet de Data Science sans utiliser de bibliothèques de machine learning prêtes à l’emploi.
Objectifs du projet :
analyserle dataset des étudiants de Poudlard,visualiserles données,- implémenter manuellement un équivalent de
describe, - développer une
régression logistiquefrom scratch, prédirela maison d’un étudiant.
describepersonnalisé — calcul manuel de la moyenne, de l’écart-type, des quartiles, etc.- Nettoyage et prétraitement des données (remplissage des NaN par maison, normalisation)
- Outils de visualisation :
- Histogrammes
- Diagrammes de dispersion
- Pair plots
- Régression logistique entièrement implémentée manuellement
- Classification One-vs-All
- Batch GD (Batch Gradient Descent)
- SGD (Stochastic Gradient Descent)
- Mini-batch GD (Mini-batch Gradient Descent)
- Sérialisation du modèle en JSON
- Pipeline de prédiction générant un fichier
houses.csvpour l’évaluation
Brève description
La régression logistique est un algorithme de classification.
Le modèle calcule la probabilité qu’un étudiant appartienne à chaque maison.
La maison ayant la probabilité la plus élevée est choisie comme prediction.
Comme il existe plusieurs maisons, le problème est divisé en plusieurs classificateurs binaires.
- Gryffondor vs tous
- Serpentard vs tous
- Serdaigle vs tous
- Poufsouffle vs tous
Chaque classificateur est entraîné séparément.
L’algorithme met à jour les poids theta afin de réduire l’erreur du modèle.
new_theta = theta - learning_rate * gradient
- Une Epoch correspond à un passage complet sur tout le dataset.
- Batch GD : 1 itération = tout le dataset
- SGD et Mini-batch : plusieurs mises à jour de theta sont effectuées pendant une epoch.
Batch Gradient Descent
Le gradient est calculé sur l’ensemble du dataset.
- apprentissage stable
- courbe de loss fluide
- lent
SGD (Stochastic Gradient Descent)
Theta est mis à jour après chaque sample.
- plus rapide
- moins de mémoire
Inconvénients : - loss bruitée
- convergence moins stable
Mini-batch Gradient Descent
Un compromis entre Batch GD et SGD.
- Le dataset est divisé en petits batchs. Exemple : batch_size = 32
- Prétraitement
- Normalisation
- Toutes les features sont ramenées à la même échelle.
Formule :
x_norm = (x - mean) / std
- mean et std -> sont sauvegardés après l’entraînement et réutilisés pendant la prédiction.
- Gestion des NaN -> Les valeurs manquantes peuvent être remplacées par la moyenne du dataset ou la moyenne de la maison.
- Visualisation (Histogramme) -> Permet de voir la distribution des notes selon les maisons.
- Scatter plot -> Permet d’observer la relation entre deux features.
- Pair plot -> Affiche toutes les relations entre les variables numériques.
make install
source ~/.bashrc
make dataCes commandes créeront un environnement virtuel (.venv), installeront toutes les dépendances depuis requirements.txt et prépareront le dataset.
make startmake describe
make hist
make scatter
make pairplot
make train
make predictmake describe-bonus
make train-sgd
make train-sgd-custom
make train-minibatch
make bonus-metricsnorminette_python --versionmake lint-check or
norminette_pythonPour exécuter depuis (.venv) :
source .venv/bin/activate# Describe dataset
srcs/describe.py datasets/dataset_train.csv
# Histograms
srcs/histogram.py datasets/dataset_train.csv
# Scatter plots
srcs/scatter_plot.py datasets/dataset_train.csv
# Pairplot
srcs/pair_plot.py datasets/dataset_train.csv
# Train model (batch GD)
srcs/logreg_train.py datasets/dataset_train.csv
# Predict houses
srcs/logreg_predict.py outputs/models/weights.json datasets/dataset_test.csv
# Bonus
# Describe dataset bonus
srcs/describe.py datasets/dataset_train.csv --extra
# Train model (sgd)
srcs/logreg_train.py datasets/dataset_train.csv --optimizer sgd --epochs 50 --learning-rate 0.01
# Train model (minibatch gd)
srcs/logreg_train.py datasets/dataset_train.csv --optimizer minibatch --epochs 50 --batch-size 32 --learning-rate 0.05
# Bonus metrics
make bonus-metricsdeactivateUtilisation avec python3 (srcs/*):
# Describe dataset
python3 srcs/describe.py datasets/dataset_train.csv
# Histograms
python3 srcs/histogram.py datasets/dataset_train.csv
# Scatter plots
python3 srcs/scatter_plot.py datasets/dataset_train.csv
# Pairplot
python3 srcs/pair_plot.py datasets/dataset_train.csv
# Train model
python3 srcs/logreg_train.py datasets/dataset_train.csv
# Predict houses
python3 srcs/logreg_predict.py outputs/models/weights.json datasets/dataset_test.csv
# Bonus
# Describe dataset bonus
python3 srcs/describe.py datasets/dataset_train.csv --extra
# Train model (sgd)
python3 srcs/logreg_train.py datasets/dataset_train.csv --optimizer sgd --epochs 50 --learning-rate 0.01
# Train model (minibatch gd)
python3 srcs/logreg_train.py datasets/dataset_train.csv --optimizer minibatch --epochs 50 --batch-size 32 --learning-rate 0.05
# Bonus metrics
make bonus-metricsnumpy,matplotlib– utilitaires pour les images et les calculs mathématiquespandas– uniquement pour le chargement des CSV et comme conteneur DataFrame ; aucun helper statistique de pandas n’est utilisé pour le describe obligatoire ni pour la régression logistique.seaborn– visualisation des donnéesstreamlit– interface interactive pour README
🇷🇺 RUSSIAN VERSION
Этот проект посвящён созданию полного Data Science pipeline без использования готовых ML-библиотек.
Цель проекта:
проанализироватьдатасет студентов Хогвартса,визуализироватьданные,- вручную реализовать аналог
describe, - написать
логистическую регрессиюс нуля, предсказыватьфакультет студента.
- Персональный
describe— ручное вычисление среднего значения, стандартного отклонения, квартилей и т.д. - Очистка и предобработка данных (заполнение NaN по факультетам, нормализация)
- Инструменты визуализации:
- Гистограммы
- Диаграммы рассеяния
- Pair plots
- Полностью ручная реализация логистической регрессии
- Классификация One-vs-All
- Batch GD (Batch Gradient Descent)
- SGD (Stochastic Gradient Descent)
- Mini-batch GD (Mini-batch Gradient Descent)
- Сериализация модели в JSON
- Pipeline предсказания, генерирующий
houses.csvдля оценки результатов
Краткое описание
Логистическая регрессия — это алгоритм классификации.
Модель вычисляет вероятность принадлежности студента к каждому факультету.
Факультет с максимальной вероятностью выбирается как prediction.
Так как факультетов несколько, задача разбивается на несколько бинарных классификаторов.
- Gryffindor vs all
- Slytherin vs all
- Ravenclaw vs all
- Hufflepuff vs all
Каждый классификатор обучается отдельно.
Алгоритм обновляет веса theta так, чтобы уменьшать ошибку модели.
new_theta = theta - learning_rate * gradient
- Epoch — это один полный проход по всему dataset.
- Batch GD: 1 iteration = весь dataset
- SGD и Mini-batch - внутри одной epoch происходит много обновлений theta.
Batch Gradient Descent
Градиент считается по всему dataset.
- стабильный learning
- плавная loss curve
- медленный
SGD (Stochastic Gradient Descent)
Обновление theta происходит после каждого sample.
- быстрее
- меньше памяти
Минусы: - noisy loss
- менее стабильная сходимость
Mini-batch Gradient Descent
Компромисс между Batch GD и SGD.
- Dataset разбивается на маленькие batch. Ex: batch_size = 32
- Preprocessing
- Нормализация
- Все признаки приводятся к одному масштабу.
Формула:
x_norm = (x - mean) / std
- mean и std -> сохраняются после train и используются при predict.
- Работа с NaN -> Пропущенные значения могут заполняться: средним по dataset, или средним по факультету.
- Визуализация (Histogram) -> Позволяет увидеть распределение оценок по факультетам.
- Scatter plot -> Позволяет увидеть зависимость между двумя признаками.
- Pair plot -> Показывает все взаимосвязи между числовыми признаками.
make install
source ~/.bashrc
make dataЭти команды создадут виртуальное окружение (.venv),
установят все зависимости из requirements.txt и подготовят датасет.
make startmake describe
make hist
make scatter
make pairplot
make train
make predictmake describe-bonus
make train-sgd
make train-sgd-custom
make train-minibatch
make bonus-metricsnorminette_python --versionmake lint-check или
norminette_pythonЗапуск через окружение (.venv):
source .venv/bin/activate# Describe dataset
srcs/describe.py datasets/dataset_train.csv
# Histograms
srcs/histogram.py datasets/dataset_train.csv
# Scatter plots
srcs/scatter_plot.py datasets/dataset_train.csv
# Pairplot
srcs/pair_plot.py datasets/dataset_train.csv
# Train model (batch GD)
srcs/logreg_train.py datasets/dataset_train.csv
# Predict houses
srcs/logreg_predict.py outputs/models/weights.json datasets/dataset_test.csv
# Bonus
# Describe dataset bonus
srcs/describe.py datasets/dataset_train.csv --extra
# Train model (sgd)
srcs/logreg_train.py datasets/dataset_train.csv --optimizer sgd --epochs 50 --learning-rate 0.01
# Train model (minibatch gd)
srcs/logreg_train.py datasets/dataset_train.csv --optimizer minibatch --epochs 50 --batch-size 32 --learning-rate 0.05
# Bonus metrics
make bonus-metricsdeactivateЗапуск через python3 (srcs/*):
# Describe dataset
python3 srcs/describe.py datasets/dataset_train.csv
# Histograms
python3 srcs/histogram.py datasets/dataset_train.csv
# Scatter plots
python3 srcs/scatter_plot.py datasets/dataset_train.csv
# Pairplot
python3 srcs/pair_plot.py datasets/dataset_train.csv
# Train model
python3 srcs/logreg_train.py datasets/dataset_train.csv
# Predict houses
python3 srcs/logreg_predict.py outputs/models/weights.json datasets/dataset_test.csv
# Bonus
# Describe dataset bonus
python3 srcs/describe.py datasets/dataset_train.csv --extra
# Train model (sgd)
python3 srcs/logreg_train.py datasets/dataset_train.csv --optimizer sgd --epochs 50 --learning-rate 0.01
# Train model (minibatch gd)
python3 srcs/logreg_train.py datasets/dataset_train.csv --optimizer minibatch --epochs 50 --batch-size 32 --learning-rate 0.05
# Bonus metrics
make bonus-metricsnumpy,matplotlib— утилиты для работы с изображениями и математическими вычислениямиpandas— используется только для загрузки CSV и как контейнер DataFrame; статистические функции pandas не используются для обязательной реализации describe и логистической регрессии.seaborn— визуализация данныхstreamlit— интерактивный интерфейс для README
Data analysis • Visualization • Custom logistic regression (One-vs-All)
Project Architecture
srcs/
├── describe.py # statistical describe
├── histogram.py # histogram visualization
├── scatter_plot.py # scatter plots
├── pair_plot.py # pair plot visualization
├── plot_training.py # optimizer/loss comparison
├── logreg_train.py # training pipeline
├── logreg_predict.py # prediction pipeline
├── logreg/
│ ├── model.py # One-vs-Rest logistic regression
│ ├── optimizers.py # batch / SGD / mini-batch GD
│ └── features.py # sigmoid & feature utilities
├── data/
│ ├── loading.py # CSV loading
│ ├── hogwarts_clean.py # NaN handling & preprocessing
│ └── common_data.py # shared dataset utilities
├── utils/
│ ├── normalization.py # feature scaling
│ ├── encoding.py # label encoding
│ └── stats/
│ ├── basic.py # mandatory statistics
│ ├── extra.py # bonus statistics
│ └── stats.py # statistical helpers
└── visualization/
└── plots.py # plotting utilities
Full pipeline design, model training, and evaluation, etc.
Made with 🤍 by:
This project was developed as a team collaboration at Ecole 42 in May 2026.