Build_DeepSeek_Step_by_Step

Build_DeepSeek_Step_by_Step is an implementation-oriented project that breaks down the key modules behind modern LLMs from the ground up. It does not try to reproduce a full industrial training system. Instead, it decomposes a typical DeepSeek-like technical path into readable notebooks and teaching-oriented helper code, so readers can trace the full chain:

• how text becomes tokens
• how tokens become embeddings
• how attention works
• what modern structures such as GQA, MLA, and MoE solve in engineering terms
• how training progresses from pretraining to SFT, reward models, PPO, and GRPO
• how data moves from collection, cleaning, and deduplication to trainable corpora

The core goal of this project is not to stack terminology, but to connect modules, formulas, code, figures, and engineering meaning into one coherent learning path.

The project currently covers five main tracks:

1. Foundational Math and Input Representation

   • vectors, matrices, softmax, masks
   • tokenizers, BPE, embeddings

2. Core Transformer Structure

   • self-attention
   • multi-head attention
   • RoPE
   • RMSNorm, residual connections, FFN, SwiGLU
   • basic decoder-only block

3. Inference Efficiency and Capacity Scaling

   • KV cache
   • MQA / GQA
   • MLA
   • MoE

4. Training and Alignment Pipeline

   • pretraining
   • continued training
   • SFT
   • reward model
   • RLHF
   • PPO / DPO / GRPO

5. Data Engineering Pipeline

   • data collection / crawling
   • main-content extraction
   • cleaning / filtering
   • exact deduplication / near-duplicate detection
   • quality bucketing

An additional section covers:

6. Position Encoding Visualization
   • learned absolute position embeddings
   • the original sinusoidal encoding from Attention Is All You Need
   • RoPE
   • relative bias
   • ALiBi
   • RoPE scaling intuition
   • 2D positional encoding

• Each notebook focuses on one core question instead of mixing topics.
• Explanations take priority over showing off technical complexity, and the code keeps intermediate results and tensor shapes visible where useful.
• Important concepts are implemented minimally whenever possible instead of being left as definitions only.
• Helper functions in utils/ are also written in a teaching style, with detailed docstrings and comments.
• The training and data sections are part of the main learning path, not side material.
• File names consistently use numbers and underscores, with no spaces.

Build_DeepSeek_Step_by_Step/
├─ README.md
├─ PRD.md
├─ requirements.txt
├─ notebooks/
│  ├─ 01_python_and_matrix_foundations.ipynb
│  ├─ 02_tokenization_and_bpe.ipynb
│  ├─ 03_embeddings_and_language_model_inputs.ipynb
│  ├─ 04_self_attention_from_scratch.ipynb
│  ├─ 05_multi_head_attention.ipynb
│  ├─ 06_rope_and_position_encoding.ipynb
│  ├─ 07_rmsnorm_and_residual_connections.ipynb
│  ├─ 08_ffn_and_swiglu.ipynb
│  ├─ 09_build_a_basic_transformer_block.ipynb
│  ├─ 10_kv_cache_and_inference.ipynb
│  ├─ 11_mqa_and_gqa.ipynb
│  ├─ 12_mla_from_intuition_to_implementation.ipynb
│  ├─ 13_moe_routing_and_experts.ipynb
│  ├─ 14_pretraining_data_and_objective.ipynb
│  ├─ 15_sft_and_alignment_basics.ipynb
│  ├─ 16_reward_model_and_rl_intro.ipynb
│  ├─ 17_putting_everything_together.ipynb
│  ├─ 18_multi_stage_training_rlhf_ppo_grpo.ipynb
│  ├─ 19_data_collection_and_crawling.ipynb
│  ├─ 20_data_cleaning_filtering_and_dedup.ipynb
│  └─ 21_visualizing_position_encodings.ipynb
├─ utils/
│  ├─ tokenizer_utils.py
│  ├─ attention_utils.py
│  ├─ visualization_utils.py
│  └─ training_utils.py
└─ assets/
   ├─ figures/
   │  └─ README.md
   └─ sample_texts/
      ├─ tiny_corpus.txt
      ├─ instruction_examples.txt
      └─ raw_web_page_mock.html

• 01_python_and_matrix_foundations.ipynb
• 02_tokenization_and_bpe.ipynb
• 03_embeddings_and_language_model_inputs.ipynb
• 04_self_attention_from_scratch.ipynb
• 05_multi_head_attention.ipynb

• 06_rope_and_position_encoding.ipynb
• 07_rmsnorm_and_residual_connections.ipynb
• 08_ffn_and_swiglu.ipynb
• 09_build_a_basic_transformer_block.ipynb
• 10_kv_cache_and_inference.ipynb

• 11_mqa_and_gqa.ipynb
• 12_mla_from_intuition_to_implementation.ipynb
• 13_moe_routing_and_experts.ipynb

• 14_pretraining_data_and_objective.ipynb
• 15_sft_and_alignment_basics.ipynb
• 16_reward_model_and_rl_intro.ipynb
• 17_putting_everything_together.ipynb
• 18_multi_stage_training_rlhf_ppo_grpo.ipynb

• 19_data_collection_and_crawling.ipynb
• 20_data_cleaning_filtering_and_dedup.ipynb

• 21_visualizing_position_encodings.ipynb

utils/ is not a placeholder directory. It is a teaching-oriented helper layer reused by the notebooks:

• tokenizer_utils.py
  • tiny BPE, pair counting, merge replay, encode / decode
• attention_utils.py
  • softmax, causal masks, single-head attention, head reshaping, KV cache intuition helpers
• visualization_utils.py
  • heatmaps, line traces, similarity matrices, figure saving
• training_utils.py
  • cross entropy, masked SFT loss, pairwise reward loss, PPO, and GRPO toy helpers

These files intentionally use a teaching style, so their comments are more detailed than a normal utility library.

assets/ stores reusable static resources for the project.

• assets/figures/README.md
  • explains how image assets should be archived
• assets/sample_texts/tiny_corpus.txt
  • suitable for BPE and basic tokenization notebooks
• assets/sample_texts/instruction_examples.txt
  • suitable for SFT / instruction-format notebooks
• assets/sample_texts/raw_web_page_mock.html
  • suitable for data collection / cleaning notebooks

Python 3.11 is recommended. The first version of the notebooks mainly depends on:

• numpy
• matplotlib
• torch
• jupyter

These dependencies are enough to support the project's minimal implementations, visualizations, and tensor experiments.

The current version still does not aim to provide:

• a full industrial-grade reproduction of the DeepSeek training stack
• a distributed training system implementation
• CUDA / Triton / FlashAttention kernel optimization
• a complete inference-serving framework

This project is closer to a runnable, explainable, and extensible technical dissection notebook. The priority is to explain the full path from modules, to architecture, to training, to data, rather than copying repositories or summarizing concepts.