N°003 · 14 November 2026
GPT-2 from scratch
Re-implemented GPT-2, loaded the official 124M weights into it, and fine-tuned it into an SMS spam classifier. Also coded a tokenizer by hand.
What I built
Following Build a Large Language Model (From Scratch), I implemented a GPT-2 style model and got it running end to end.
- GPT architecture: multi-head attention, LayerNorm, GELU, feed-forward, transformer blocks
- Loaded the official GPT-2 (124M) weights into my own implementation
- Text generation with temperature and top-k sampling
- Fine-tuned the model as a spam classifier on the UCI SMS Spam Collection
- A practice notebook where I coded a BPE-style tokenizer by hand