N°004 · 18 November 2026
English → French Transformer, from scratch
A full encoder–decoder Transformer in PyTorch for EN→FR translation, built twice: once like the 2017 paper, once with the modern recipe (RMSNorm, SwiGLU, RoPE).
View the code on GitHub ↗What I built
A complete encoder–decoder Transformer in PyTorch, written from scratch and trained for English → French translation.
Version 1 — the original paper
- Custom
LayerNormand position-wise feed-forward network - Multi-head self-attention, masked self-attention for the decoder, and cross-attention
- Sinusoidal positional encodings
- Stacked encoder/decoder blocks assembled into a full
Transformer
Version 2 — the modern recipe
- RMSNorm in place of LayerNorm
- SwiGLU feed-forward layers
- Rotary positional embeddings (RoPE) in place of sinusoidal encodings
What I learned
Building both versions side by side made it concrete why modern LLMs moved to RMSNorm, SwiGLU and RoPE.