MATHIS/index
← Back to the index

N°003 · 14 November 2026

GPT-2 from scratch

Re-implemented GPT-2, loaded the official 124M weights into it, and fine-tuned it into an SMS spam classifier. Also coded a tokenizer by hand.

Difficulty
★★★ Deep dive
Status
Done
Topics
#pytorch #llm #fine-tuning

What I built

Following Build a Large Language Model (From Scratch), I implemented a GPT-2 style model and got it running end to end.

  • GPT architecture: multi-head attention, LayerNorm, GELU, feed-forward, transformer blocks
  • Loaded the official GPT-2 (124M) weights into my own implementation
  • Text generation with temperature and top-k sampling
  • Fine-tuned the model as a spam classifier on the UCI SMS Spam Collection
  • A practice notebook where I coded a BPE-style tokenizer by hand