Next-Concept Modeling: from next tokens to parallel codes
Most familiar language models generate the next token, then repeat. NCM changes what gets predicted and in what order. Here is the mechanism, from tokenizers and KV caches to parallel code levels.
On this page
TL;DR
- In ordinary autoregressive decoding, each selected token becomes context for the next prediction. KV caching saves repeated attention work but preserves that dependency.
- NCM learns discrete code levels and generates them from coarse to fine: sequential between levels, parallel within each level. Its generator is still a transformer.
- The authors report 31x faster XSum generation than AURORA-LM under specific conditions. Public research checkpoints do not yet establish general-purpose agent performance.
Imagine a predictive keyboard that must commit one small piece of a message before it can choose the next. It can consider many possible continuations, but the piece it selects changes what comes afterward. That is a useful starting point for understanding an autoregressive language model. Next-Concept Modeling (NCM) explores a different schedule: generate a hierarchy of learned codes, with many codes predicted together, then decode that representation into text. To see why this matters, start with the work an ordinary language model does between two visible words.
First, a tokenizer turns text into token IDs
A typical autoregressive transformer does not receive a sentence as a row of words. Its tokenizer splits the text into units and maps them to numerical IDs. A token might represent a whole word, part of a word, punctuation or another piece of text. Different tokenizers divide the same sentence differently.
Those IDs are mapped to vectors, called embeddings, that the network can process. When it generates an answer, the model produces token IDs that are decoded back into readable text. This is the input-and-output path shown in Hugging Face’s generation documentation.
