micrograd: backprop from scratch in ~100 lines

11th July 2026

I built a scalar-valued automatic-differentiation engine and a small neural-network library on top of it, following Andrej Karpathy's micrograd lecture and then extending it with my own graph-drawing and training-log tooling. The whole autograd core is about a hundred lines of Python with no NumPy, no PyTorch, nothing. And yet it trains a real classifier. This post walks through exactly how the pieces fit together, with the actual graphs and numbers my code produces. Repo: github.com/xenonnn4w/micrograd.

The one idea: a Value

Everything hangs off a single class. A Value wraps one number, but it also remembers where it came from: which other Values produced it and by what operation. That memory is what lets us run the graph backward later.

class Value:
    def __init__(self, data, _children=(), _op="", label=""):
        self.data = data          # the forward number
        self.grad = 0.0           # d(loss)/d(self), filled in later
        self._backward = lambda: None   # how to push grad to children
        self._prev = set(_children)     # who produced me
        self._op = _op                  # the op that produced me

Two fields do the real work. _prev is the set of parent nodes, so a network of Values forms a directed acyclic graph. And _backward is a closure: each operation, when it builds its output, also stashes a little function that knows how to send gradient from the output back to its inputs. Nothing runs yet, it just gets recorded.

One operation, forward and back

Look at multiplication. The forward part is obvious: multiply the two numbers. The interesting half is the closure. For out = self * other, calculus says d(out)/d(self) = other and d(out)/d(other) = self. We multiply each by the gradient already sitting on out (the chain rule) and accumulate it onto the inputs.

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), "*")

    def _backward():
        self.grad  += other.data * out.grad   # chain rule
        other.grad += self.data  * out.grad
    out._backward = _backward
    return out

The += matters. If a Value feeds two places downstream, its gradient is the sum of both contributions, so we add rather than overwrite. Every other op follows the same template: + passes gradient straight through, tanh scales it by 1 - tanh², **k scales it by k·xᵏ⁻¹. That is the entire engine, one local derivative per operation.

A whole neuron as a graph

Wire a few of these together and you get a neuron: o = tanh(x1·w1 + x2·w2 + b). Because every intermediate is a Value, the expression is a computation graph. Here is the classic worked example (x1=2, w1=-3, x2=0, w2=1, with b chosen so the output lands on a clean 0.7071), rendered by my own draw_dot:

Read it left to right and you have the forward pass, each box's data. Read it right to left and you have the backward pass, each box's grad. The output seeds its own gradient at 1.0, then tanh turns that into 0.5 (because 1 - 0.7071² = 0.5), and from there the two + nodes copy 0.5 straight back to everything feeding them. At the multiply nodes the gradient crosses over: x1 gets w1·0.5 = -1.5, while w1 gets x1·0.5 = 1.0. That crossover is the __mul__ closure from above, firing.

Reverse mode over the DAG

The per-op closures only know how to move gradient one step. To differentiate the whole graph we have to call them in the right order: every node must run after everything that depends on it. That is a reverse topological sort, and backward() is the entire "training framework":

def backward(self):
    topo, visited = [], set()
    def build_topo(node):
        if node not in visited:
            visited.add(node)
            for child in node._prev:
                build_topo(child)
            topo.append(node)          # post-order = child before parent
    build_topo(self)

    self.grad = 1.0                    # seed d(self)/d(self)
    for node in reversed(topo):        # parents before children
        node._backward()               # fire each local rule once

Seed the root at 1.0, walk the topological order in reverse, and let each node push gradient to its parents exactly once. This is reverse-mode autodiff, the same algorithm PyTorch and JAX run, minus the tensors and the C++.

From neuron to MLP

With one neuron working, the network layers are almost trivial. A Neuron holds a weight per input plus a bias; a Layer is a list of neurons; an MLP chains layers. The only method any of them needs beyond __call__ is parameters(), which gathers every Value the optimizer is allowed to nudge.

class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1, 1)) for _ in range(nin)]
        self.b = Value(random.uniform(-1, 1))
    def __call__(self, x):
        act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
        return act.tanh()

class MLP:
    def __init__(self, nin, nouts):
        sz = [nin] + nouts
        self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]
    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

Training a tiny MLP

The training loop is the honest version of what every framework hides behind .fit(): forward to a loss, zero the old gradients, backward(), then step every parameter a little way down its gradient. I trained MLP(3, [4, 4, 1]) to memorise four labelled examples.

n  = MLP(3, [4, 4, 1])
xs = [[2.0, 3.0, -1.0], [3.0, -1.0, 0.5], [0.5, 0.5, 1.0], [1.0, 1.0, -1.0]]
ys = [1.0, -1.0, -1.0, 1.0]          # targets

for k in range(20):
    ypred = [n(x) for x in xs]                                  # forward
    loss  = sum((yo - yt)**2 for yt, yo in zip(ys, ypred))      # MSE
    for p in n.parameters():
        p.grad = 0.0                                            # zero grads
    loss.backward()                                             # backward
    for p in n.parameters():
        p.data += -0.05 * p.grad                                # SGD step

Twenty steps take the loss from 3.16 down to 0.03, and the four predictions slide onto their targets of +1, -1, -1, +1. This is the exact loss column my TrainingLog wrote out:

A real classifier: make_moons

The four-point demo is memorisation. The real test is generalisation, so I pointed a bigger net, MLP(2, [10, 10, 1]) with 151 parameters, at scikit-learn's make_moons: two interleaving half-circle clusters that no straight line can separate. The loss is an SVM-style max-margin loss with a small L2 regularisation term, and the learning rate decays over the run.

X, y = make_moons(n_samples=100, noise=0.1)
model = MLP(2, [10, 10, 1])          # 151 parameters

for k in range(100):
    total_loss, acc = loss()                     # max-margin + L2 reg
    for p in model.parameters():
        p.grad = 0
    total_loss.backward()
    lr = 1.0 - 0.9 * k / 100                      # decaying step size
    for p in model.parameters():
        p.data -= lr * p.grad

Over 100 steps the loss falls from 0.53 to 0.12 while accuracy climbs from 62% to a stable 91%. Because both quantities sit naturally on the same 0-to-1 scale, I can plot them together without a second axis:

Feeding a grid of points through the trained net and colouring by the sign of the output draws the decision boundary, a curved band that hugs the two moons. The same scalar engine that differentiated one neuron above scaled, unchanged, to a 151-parameter network solving a nonlinear problem.

What I took from it

• Backprop is not magic. It is one local derivative per operation, plus a topological sort to call them in order. Once the Value class clicked, the "neural network" part was almost an afterthought.
• The graph is the model. There is no separate forward and backward description, building the forward expression out of Values is building the backward graph, for free.
• Tooling paid off. Writing draw_dot, draw_mlp, and the training loggers myself is what turned abstract gradients into the pictures above, and made the bugs obvious.

The full engine, the neural-net library, and all four worked examples (single neuron, tanh neuron, the trained MLP, and the moons classifier) are on GitHub: github.com/xenonnn4w/micrograd.

Tags