Author(s): Ravishanker Mnit

Originally published on Towards AI.

A Language Model, Built One Token at a Time

The first surprise about a large language model is how small its training instruction sounds: given the tokens so far, predict the next token.

Consider the sentence:

The little robot opened the ___

The model does not receive the answer “door” as a special label attached to the whole sentence. During training, it sees the prefix and learns a probability distribution for the next token. The desired continuation might be door, but other continuations can also be plausible. The model’s job is to put more probability on the training continuation over many examples.

This simple objective became unusually powerful when combined with the Transformer, large text collections, and enough compute.

A short route from sequence models to GPT

Before Transformers, a common pattern for translation and other sequence tasks used an encoder to read the source and a decoder to produce the answer. Recurrent networks processed tokens in order. That ordering was natural, but it limited parallel work across a long sentence: position 20 depended on the hidden state from position 19.

In 2017, Attention Is All You Need introduced the Transformer: attention layers and position-wise feed-forward layers, with residual connections and normalization. The original system was an encoder–decoder model for translation, not the decoder-only GPT architecture. Its key practical advantage was that the training computation for many positions could be organized in parallel. (Attention Is All You Need)

The Transformer idea later appeared in three broad shapes. Encoder-only models such as BERT build contextual representations for understanding tasks. Encoder–decoder models such as T5 read an input sequence and generate a transformed output, which fits translation and summarization naturally. Decoder-only models such as GPT predict the next token and can continue any prompt. These are useful patterns, not rigid boundaries: modern systems can wrap a decoder in multimodal encoders, routers, tools, or other modules. (BERT; T5)

The GPT idea took the Transformer decoder and used it as a generative pre-training model. Train on a broad text corpus with a causal objective, then adapt the resulting representations to tasks. GPT-2 demonstrated how a larger text-trained model could perform several tasks in a zero-shot way; GPT-3 showed strong few-shot behavior when examples were included in the prompt. These are milestones in one research line, not a claim that every modern assistant has the same architecture or training recipe. (GPT-1 paper; GPT-2 announcement; GPT-3 paper)

The progression is roughly:

What does “causal” mean here?

In ordinary self-attention, every position could read every other position. A language model that predicts the next token must not peek at the future answer. A causal mask blocks that information.

For four token positions, a simplified visibility mask is:

Figure 2. Each position can read its own token and preceding tokens; future tokens are masked.

The model can train all four predictions in one forward pass because the mask prevents leakage. Position 1 predicts token 2, position 2 predicts token 3, and so on. At generation time, it starts with a prompt and repeatedly samples or selects one next token.

Text is converted to token IDs first

Neural networks operate on numbers, so a tokenizer turns text into integer IDs. A token can be a whole word, a word fragment, a punctuation mark, or a byte-level unit. For example, a tokenizer might represent unbelievable as two or three pieces rather than one whole-word entry.

The tiktoken package is an implementation of byte-pair-encoding tokenizers used with OpenAI model families. It can encode text and decode token IDs; it does not train a new tokenizer from your corpus. If you are truly creating a new model family, tokenizer design and vocabulary training are a separate step. For a compact experiment, reusing a public tokenizer is easier and lets us focus on the model. (OpenAI tiktoken repository)

import tiktoken

enc = tiktoken.get_encoding("gpt2")
text = "The little robot opened the door."
ids = enc.encode(text)
print(ids)
print(enc.decode(ids))

The returned IDs are not word numbers with human meaning. They are indices into an embedding table. The model learns a vector for each ID. If the vocabulary has about 50,000 entries and the embedding width is 128, the token table alone has about 50,000 × 128 = 6.4 million learned values.

The training target is shifted by one

If token IDs are [A, B, C, D], training pairs are:

input: [A, B, C]
target: [B, C, D]

t each position, the model produces scores (logits) over the entire vocabulary. A softmax converts scores into probabilities. Cross-entropy compares those probabilities with the correct next-token IDs. If the correct token has probability 0.30, its negative log probability is approximately -ln(0.30) = 1.20 nats. Training tries to lower this loss across many positions and batches.

The objective is not “write a perfect answer.” It is “predict the data well.” That difference explains why a base model can continue text fluently but still fail to follow a user’s request reliably. Assistant behavior is built in later stages.

The corpus is part of the model

Before tokenization, a training pipeline usually collects documents, removes malformed or duplicate material, applies quality and language filters, handles personal or sensitive information under the project’s data policy, and checks rights and provenance. It then tokenizes and packs examples into training sequences. Evaluation data should be held out by document or source before windows are sampled; randomly splitting adjacent token windows can put near-identical text on both sides of the test.

These choices affect what the model learns. More text is not automatically better if it contains duplicates, low-quality templates, contamination from evaluation answers, or material the project should not use.

What to remember

• GPT is a family name for generative pre-trained Transformer systems; it does not mean every current model is one plain decoder stack.
• A causal mask lets a model learn next-token prediction without reading future tokens.
• Tokenization is a learned or designed mapping from text fragments to IDs; token IDs then map to vectors.
• Pre-training gives broad statistical capability. It does not, by itself, guarantee helpfulness, factuality, or safe tool use.

Next: inside one decoder block, including why position encoding, normalization, and residual paths matter.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI