Project notes · July 2026

makemore: names, one character at a time

makemore is my small language-model laboratory. Every model gets the same job: read a prefix of a name, predict the next character, and repeat until it emits the end token. Keeping the task fixed makes every architectural change visible.

32,033names

27tokens

4model stages

2.3286best recorded test NLL

One task, four models

I started with a literal table of transition counts. Then I replaced the table with learned embeddings, made the network deep enough for initialization to matter, and finally expanded its receptive field from three characters to eight through a hierarchy.

Select a stage to follow the growing receptive field.

The bigram baseline

The first model contains no trained tensors. It counts every pair in the corpus, including the . start and stop token, adds one to every cell for smoothing, normalizes each row, and samples from the result. Its average negative log-likelihood is 2.4546. That number became the line every learned model had to cross.

A random 27-way guess has NLL ln(27) = 3.2958. The count model is already much better because spelling has local structure.

Held-out test NLL for the checked-in full MLP run. Lower is better.

Learning an embedding

The MLP follows the setup from Bengio et al.'s neural probabilistic language model: three character IDs become learned two-dimensional vectors, those six values feed a 300-unit tanh layer, and 27 logits score the next character. The checked-in 100,000-step run has 10,281 trainable parameters and reaches 2.3286 test NLL.

context [e, m, m]
      ↓ embedding lookup
shape   [3, 2] → flatten [6]
      ↓ Linear(6, 300) + tanh
hidden  [300]
      ↓ Linear(300, 27)
logits  [27] → cross entropy

The two-dimensional table is also drawable, which makes the learned alphabet inspectable. Vowels and commonly interchangeable consonants can move near one another because the objective rewards similar next-character behavior, not linguistic labels.

Watching gradients

With five tanh layers, a loss value alone is not enough. I added hooks for activation histograms, output gradients, weight gradients, and the log update-to-data ratio for every parameter tensor. The model uses fan-in scaling, a 5/3 tanh gain, and a deliberately quiet final layer so its initial predictions are not overconfident.

Fraction of activations with absolute value above 0.97 after the recorded 1,000-step diagnostic run.

The first tanh layer is the most saturated in the recorded diagnostic run. Later layers sit close to 12 to 14 percent, while their gradient standard deviations remain in the same order of magnitude. That is the practical check: signals should neither collapse nor explode as they cross the stack.

The checked-in deep-network report is a 1,000-step debug run. Its 2.3682 test NLL is useful for validating instrumentation, not for ranking the architecture against the fully trained MLP.

A hierarchical context

The last stage uses an eight-character input and a customFlattenConsecutive(2) layer. Each application pairs adjacent time steps, concatenates their channels, and halves the time dimension. Three applications turn eight positions into one representation without flattening the entire context in a single jump.

This borrows WaveNet's expanding receptive-field idea, but it is not an audio model. It is still a character-level name generator. The useful part is the hierarchy: nearby characters combine first, then those local features combine over progressively longer spans.

What the project taught me

A baseline is most useful when every later model keeps the same data split and metric.
Embeddings turn discrete symbols into geometry that gradient descent can organize.
Initialization becomes observable, not theoretical, once activations and gradients are plotted by layer.
A larger context helps only when the architecture has a sensible way to combine it.

Source and generated reports live in github.com/xenonnn4w/makemore.