# Next-Concept Modeling: from next tokens to parallel codes

Source: https://www.humanstoknow.com/blog/next-concept-modeling-ncm-explained

Most familiar language models generate the next token, then repeat. NCM changes what gets predicted and in what order. Here is the mechanism, from tokenizers and KV caches to parallel code levels.

Author: Akshay Apsingi
Published: 2026-10-15T07:48:49.945Z
Updated: 2026-10-15T08:12:56.217Z

## TL;DR

- In ordinary autoregressive decoding, each selected token becomes context for the next prediction. KV caching saves repeated attention work but preserves that dependency.
- NCM learns discrete code levels and generates them from coarse to fine: sequential between levels, parallel within each level. Its generator is still a transformer.
- The authors report 31x faster XSum generation than AURORA-LM under specific conditions. Public research checkpoints do not yet establish general-purpose agent performance.

![NCM training stages: learn a multiscale tokenizer, then train a transformer to generate coarse-to-fine code levels.](https://www.humanstoknow.com/api/media/file/sha256-fada122ef347dfe61e05b189393a0d8969ae137095c06c027d3092265aacb3a2.webp)

Explanatory artwork generated with Higgsfield from the NCM authors’ two-stage diagram. The code levels are schematic; training stages and generation levels are different things.

Imagine a predictive keyboard that must commit one small piece of a message before it can choose the next. It can consider many possible continuations, but the piece it selects changes what comes afterward. That is a useful starting point for understanding an autoregressive language model. Next-Concept Modeling (NCM) explores a different schedule: generate a hierarchy of learned codes, with many codes predicted together, then decode that representation into text. To see why this matters, start with the work an ordinary language model does between two visible words.

## First, a tokenizer turns text into token IDs

A typical autoregressive transformer does not receive a sentence as a row of words. Its **tokenizer** splits the text into units and maps them to numerical IDs. A token might represent a whole word, part of a word, punctuation or another piece of text. Different tokenizers divide the same sentence differently.

Those IDs are mapped to vectors, called embeddings, that the network can process. When it generates an answer, the model produces token IDs that are decoded back into readable text. This is the input-and-output path shown in [Hugging Face’s generation documentation](https://huggingface.co/docs/transformers/main/en/llm_tutorial).

So “one word at a time” is an approximation. The ordinary decoding loop advances one **token** at a time, which may be less or more than a word.

## How does the model choose the next token?

The first phase is **prefill**: the model processes the prompt and computes the states needed to predict what follows. It can process the prompt’s positions in parallel under a causal attention mask, which prevents a position from using later text.

The model produces scores, called **logits**, for possible next tokens. A softmax operation can turn those scores into a probability distribution. A decoding rule then selects a token: it might take the highest-probability option, or sample from a controlled distribution to allow variation.

For a toy prompt such as “The cup is on the”, imagine probabilities assigned to continuations such as “table” or “shelf”. These are illustrative options, not measurements from a model. Whichever token is selected joins the context. The model then predicts again using that extended context.

That repeated prediction phase is **decode**. It continues until a stopping condition, such as an end token or a length limit. [NVIDIA’s inference explanation](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization) distinguishes prefill from decode because their computational demands differ.

## What does a KV cache save?

Repeating prediction need not mean recomputing every earlier token from scratch. In a causal transformer, attention computes **queries, keys and values**: vectors used to decide which earlier information is relevant and combine it for the current position.

A **KV cache** retains the key and value vectors for tokens already processed. During a later decoding step, the model computes the new token’s states and reuses those cached vectors when attending to prior context. It avoids recomputing the old keys and values at every step. [Hugging Face’s cache documentation](https://huggingface.co/docs/transformers/main/en/cache_explanation) explains this reuse directly.

The cache stores intermediate attention states. It is not a library of finished answers, and it does not update the model’s weights. It also consumes memory as the context grows; the new position still needs to use prior context.

## Why is autoregressive generation still sequential?

Caching makes a step cheaper. It does not remove the dependency between steps in the ordinary loop:

**Context → next-token probabilities → selected token → extended context → next prediction.**

Return to the predictive-keyboard analogy: remembering what you already typed saves effort, but you still need to commit the next piece before conditioning the following prediction on it. This dependency is what *autoregressive* means here.

There are serving techniques, including speculative decoding, that accelerate this baseline. And not every language model uses the same architecture or generation method. The question NCM addresses is more specific: can we change the units being generated so that fewer decisions have to wait on one another?

## Where do diffusion language models fit?

Diffusion language models challenge the one-token-at-a-time schedule. In a masked-text version, generation can begin with missing tokens and progressively resolve many positions together. Other variants refine continuous representations instead. Parallel prediction creates an opportunity for speed, but successive refinement passes still depend on previous passes.

The [NCM paper](https://arxiv.org/html/2610.14162v1#S2) identifies a bottleneck in full-sequence diffusion approaches: they repeatedly process a representation spanning the output length. Reducing the number of passes can also reduce quality. Block-based diffusion makes a different tradeoff, generating blocks sequentially while refining positions inside them together.

This is the bridge to NCM. **NCM is an alternative to those diffusion-generation procedures, not itself a standard diffusion model.** It changes both the representation being generated and the order of generation.

## What is Next-Concept Modeling?

NCM is a framework from researchers at Stanford and Together AI that represents text as a hierarchy of discrete codes. Short, coarse levels carry broad information; longer, finer levels represent additional detail. A transformer predicts one level after another, generating the codes within each level in parallel.

The paper uses “concept” for information carried by the coarse levels. A code is not guaranteed to correspond to a readable idea, sentence or human thought. There is no explicit English outline hidden between the stages.

The approach builds on [next-scale prediction for image generation](https://arxiv.org/abs/2404.02905). Applying it to text requires more than shrinking and enlarging a sequence: changing a name, digit or negation can change the meaning. The model must learn a useful hierarchy while preserving text accurately.

## Stage 1: learn a coarse-to-fine representation

The first training stage creates a **multiscale residual VQ-VAE**, a learned tokenizer with an encoder, codebooks and a decoder. This is an additional learned representation, distinct from the ordinary text-token IDs introduced earlier.

The encoder maps training text into vectors. Vector quantization replaces vectors with entries from learned codebooks, producing discrete codes. The first level represents the text coarsely. Each later level represents information the earlier levels have left unresolved: the **residual**.

The decoder learns to recover the original text from the combined representation. During training, the decoder is taught to reconstruct text from a randomly retained prefix of code levels. This encourages early levels to contain useful information rather than leaving everything to the final level. The [authors’ technical explanation](https://hazyresearch.stanford.edu/blog/2026-10-12-ncm) describes this as nested dropout.

This stage learns a representation that the generator can work with. It is not yet the process that generates a new answer.

## Stage 2: train the transformer to generate the hierarchy

Next, the tokenizer is frozen and a separate transformer learns to predict its codes. At a given level, the transformer conditions on the **cumulative reconstruction from earlier levels**. It predicts every code in the current level together, rather than selecting one code and waiting before predicting the next code in that same level. Parallel does not mean unconditioned: those predictions share the reconstruction from earlier levels as context.

During training, the system can predict all levels in one forward pass using the appropriate attention structure and training inputs. During generation, however, those levels must be produced in order. This distinction prevents a common misunderstanding: two training stages do not mean that an answer takes only two generation steps.

![Screenshot of the authors’ NCM diagram showing Stage 1 tokenizer training and Stage 2 transformer training.](https://www.humanstoknow.com/api/media/file/sha256-ce2db6a2d5ce1047f5b46a31540ceedcb0db46d3473095a78c8f5ce4cad11cb6.jpg)

*Source: cropped screenshot of [Hermann Kumbong’s architecture post on X](https://x.com/KumbongHermann/status/2110437874927849972/photo/1), captured 15 October 2026. Diagram by the NCM authors.*

## What happens when NCM generates new text?

The generator starts with the coarsest level, using a task input where applicable, then builds progressively finer code levels. Each new level uses the accumulated reconstruction from the levels already generated. After the hierarchy is complete, the tokenizer’s decoder predicts text-token distributions at all output positions. The paper selects the highest-scoring token at each position, so this final step does not restart ordinary autoregressive decoding.

The sequential unit is now a **level**, rather than an individual output token. In the paper’s configurations, XSum uses eight levels and WMT14 translation uses eleven. Those are chosen schedules, not a universal constant for every output length or task. Early levels contain fewer positions, making them cheaper to process than full-length ones.

The authors’ translation example makes the behavior visible. Decoding intermediate levels produces incomplete, repetitive wording; later levels change and resolve that wording. The early output is not simply a polished summary to which the model appends details.

![Screenshot showing a German–English translation decoded after NCM levels 1, 3, 6, 8 and 11.](https://www.humanstoknow.com/api/media/file/sha256-329f3438d1c5495a7e6136cccfd536b73d8979bf0bef67a81ce5e98b79ebadb8.jpg)

*Source: cropped screenshot of [the authors’ coarse-to-fine generation post](https://x.com/KumbongHermann/status/2110437877075292642/photo/1). This is one qualitative example, not an aggregate quality measurement. The paper’s full trajectory also shows that intermediate quality need not improve at every level.*

## NCM vs autoregressive and diffusion generation

| Question | Ordinary autoregressive decoding | Full-sequence diffusion | NCM |
| --- | --- | --- | --- |
| What gets predicted? | The next text token | Many text positions or their representations | A level of learned discrete codes |
| What can happen together? | Prompt processing; ordinary output decoding remains sequential | Many positions within a refinement pass | All codes within the current level |
| What must wait? | The next prediction depends on the selected token | The next refinement pass depends on the previous pass | The next level depends on earlier reconstructions |
| Where is repeated work reduced? | KV caching reuses prior keys and values | Depends on the model and sampler | Short coarse levels reduce early per-pass work |

A transformer can be used inside these different generation strategies. NCM’s distinction is its learned hierarchy and prediction schedule; it has not discarded transformers.

Its efficiency argument therefore has two parts: **fewer sequential passes, and cheaper early passes**. Neither is a guarantee of better answers. They explain where a speed advantage could come from, subject to the quality achieved and the implementation used.

## What does the 31x speed claim actually measure?

The authors report **31× faster XSum summarization than AURORA-LM at comparable ROUGE**, a reference-summary overlap metric. Their timing experiment generated 1,024 sequences at batch size 16 on one NVIDIA H100 SXM 80GB GPU. NCM used 16× fewer generator evaluations in that comparison. These are [the authors’ measurements](https://arxiv.org/html/2610.14162v1#S4.SS6); we have not independently reproduced them.

The comparison does not establish a 31× advantage over a frontier chat assistant or over autoregressive serving stacks generally. Nor does a higher ROUGE score guarantee that a summary preserves every fact.

Parameter accounting also matters. The XSum system has an approximately **88M-parameter generator plus a 61M-parameter tokenizer**. The paper’s generator-size comparisons do not count the complete system as just 88M parameters. Its [architecture tables](https://arxiv.org/html/2610.14162v1#A3) list both components. Reported training-token savings count token exposure, including repeated passes, and are not the same as measured savings in FLOPs or money.

## What tradeoff does the learned tokenizer introduce?

A learned code hierarchy adds another place where information can be lost. Before asking whether the generator predicts useful codes, researchers must check whether the tokenizer can reconstruct text accurately at all.

The paper reports **98.79% token reconstruction accuracy on held-out WMT14 text** using the full hierarchy. That measures recovery of the tokenizer’s input; it is not a translation-accuracy score. The [tokenizer evaluation](https://arxiv.org/html/2610.14162v1#S4.SS1) notes that reconstruction errors can substitute related words or entities. A high average recovery rate can therefore coexist with consequential errors in a particular sentence.

This makes representation quality part of the engineering problem. Saving generation steps is useful only when the resulting text retains the information the task needs.

## What can you use today, and what remains open?

The [official MIT-licensed repository](https://github.com/HazyResearch/ncm) contains model implementations, configurations and evaluation code. Public [tokenizer and generator checkpoints](https://huggingface.co/hazyresearch/ncm-checkpoints) cover task-specific experiments including summarization, German–English translation, Sudoku and unconditional text generation. The paper appeared on 9 October 2026, followed by the lab explanation dated 12 October.

The main experiments use generators around 100M parameters, with a scaling study reaching 725M on LM1B. The authors identify larger-scale training, jointly learning representations and generation, and input-dependent computation as [future directions](https://arxiv.org/html/2610.14162v1#A9). A fixed hierarchy means even a short continuation traverses its configured levels.

For agent builders, the interesting possibility is a smaller serial generation burden. Whether that produces faster, dependable agents needs evidence on instruction following, tool calls and complete workflows. Sudoku performance and task-specific checkpoints do not settle those questions.

NCM makes that possibility concrete enough to inspect: the representation, generation schedule and research artifacts are public. The next useful evidence is how well that design scales while preserving the accuracy and flexibility a real task needs.

If you want to make your agent smarter with specialized playbooks or skills, [explore our playbooks](https://www.humanstoknow.com/skills).