Next-token prediction

Topic

Next-token prediction is a core training objective and generation mechanism in autoregressive language models, where a model is trained to predict the most likely subsequent token (such as a word or subword) given a sequence of preceding tokens. This sequential, probabilistic approach serves as the foundation for modern generative pre-trained transformers (GPTs) and large language models, enabling them to generate coherent, human-like text.