Project notes · July 2026 · active

nanoGPT: from characters to a GPT-2 reproduction

nanoGPT is where I am rebuilding the decoder-only Transformer in layers. The current code trains a 10.79M-parameter character model on Shakespeare. The next target is a faithful GPT-2 124M replication with its tokenizer, architecture details, and pretraining pipeline intact.

The current model

The repository contains two model implementations: a bigram baseline and a GPT-style Transformer. Both encode the same 1,115,394-character corpus, reserve the final 10 percent for validation, train on randomly sampled next-character windows, and generate one character at a time.

• Width: 384 channels with six 64-dimensional attention heads.
• Depth: six pre-norm Transformer blocks.
• Context: 256 characters per training example.
• Regularization: 0.2 dropout in attention and feed-forward paths.
• Objective: next-token cross entropy over 65 characters.

Causal self-attention

Every position produces a query, key, and value. Query-key dot products become affinities, scaling by the inverse square root of the head width keeps their magnitude controlled, and a lower-triangular mask removes every future position before softmax. The model can read the prefix, but never the answer to the prediction it is making.

scores  = (query @ key.transpose(-2, -1)) * head_size**-0.5
scores  = scores.masked_fill(causal_mask == 0, -inf)
weights = softmax(scores, dim=-1)
output  = weights @ value

Inside one Transformer block

Each block has two jobs. Attention moves information between time positions; the feed-forward network transforms each position independently. Residual paths keep the original representation available around both operations, and LayerNorm runs before each sublayer.

Where the parameters live

Almost all parameters live inside the repeated blocks. The embeddings are small because a character vocabulary has only 65 entries. Within a block, the 384 to 1,536 to 384 feed-forward path is the largest piece, not attention.

The training loop

A batch contains 64 independently sampled windows. AdamW updates the model at a fixed learning rate of 3e-4 for 5,000 iterations. Every 500 steps, evaluation averages 200 fresh batches from both splits with dropout disabled. Generation then crops the running prefix to the last 256 characters and samples the next token from the final-position distribution.

The GPT-2 124M replication

The current model proves the decoder path end to end, but scaling width and depth is only part of a replication. GPT-2 changes the token unit, context length, activation, parameter sharing, initialization, data pipeline, and optimization regime. I am treating those as explicit milestones rather than hiding them behind a larger layer count.

1. Tokenizer: reproduce byte-level BPE encoding and the 50,257-token vocabulary, with round-trip and reference-token tests.
2. Architecture parity: move to 12 layers, 12 heads, width 768, context 1,024, GELU, weight tying, and GPT-2-compatible parameter names and shapes.
3. Weight verification: load the released GPT-2 124M checkpoint and compare logits for the same token sequence before training anything.
4. Pretraining: add streaming batches, gradient accumulation, learning-rate warmup and decay, mixed precision, checkpoint resume, and deterministic evaluation.
5. Evidence: record validation loss, tokens per second, hardware, wall time, and samples at fixed prompts and seeds.

The architecture starts with Vaswani et al.'s Attention Is All You Need and the replication target is documented in OpenAI's GPT-2 report.

Follow the implementation at github.com/xenonnn4w/nanogpt.