NanoGPT speedrun

Work

by Keller Jordan, Andrej Karpathy · 2024

The NanoGPT speedrun is a collaborative and competitive benchmark challenge focused on finding the fastest algorithm to train a 124-million parameter GPT-2 model to a target validation loss on an 8xH100 GPU node. Originating from Andrej Karpathy's nanoGPT and llm.c repositories, the project is primarily maintained and advanced by Keller Jordan through the modded-nanogpt repository.