from-scratch 1.20B. built at 18. no shortcuts.
transformer · tokenizer · optimizer · all hand-written · training live
ARCHITECTURE · MODEL EXPLORER
real shapes · real param math · source lines · click any node
thick underline = sliding-2048 local · plain = full global · click to open block
QK-norm stabilizes late training (2.24 vs 2.42 in arch bench) · sliding halves attention on even layers free · diff heads subtract common-mode noise (2.28 fixed-zoo 2nd) · tied head saves 134M.
TERRY
matrices → grafted + orthogonalized momentum · tables → factored adaptive · still one state
7M · 1000 steps · same init, batches, seeds
terry adds factored adaptive tables, grafted matrices and table nesterov, swept across 8 waves of configs. sign-graft ablation flopped at 5.02.
live states · same init, batches, seeds
140M · same weights
113M bf16 · loss and live states, one setup — states in each opt's native dtype
35 archs · 118M · 1000 steps · one harness. click any row for the recipe.