TinyTrain

Last year I built my first autograd engine, ToyGrad (ToyGrad). I experimented with vectorization, but it lacked GPU support.

Since then, I’ve implemented ideas from 14 seminal AI papers - Transformers, FlashAttention, ViTs, RAG, LoRA, & much more -Modern AI Canon.

Now I’m taking it further: building TinyTrain, a minimal ML framework to train a small language model from scratch.

This is the foundation.