This chapter links to the course’s interactive demos and to original papers on the open web, and it uses MathJax for its formulas. Read it through the course website (or download the file and open it in a browser); a Canvas inline preview will reach neither the demos nor the math. All chapters.
Neural Networks and Embeddings
Frank Rosenblatt demonstrated the perceptron in 1958; the Mark I Perceptron's camera was a 20 × 20 grid of photocells, a 400-pixel image. Rosenblatt, Psychological Review, 1958; Cornell Aeronautical Laboratory, Mark I Perceptron, 1960
Three ideas underlie every model in the chapters that follow: a neural network is a function whose weights are set by examples; an embedding is a learned vector that stands for a concept, placed so that similar concepts are near; and an encoder and a decoder move between things and their embeddings, with two encoders trained side by side making words and pictures one coordinate system. This chapter states each idea, runs the smallest possible example of it by hand, and shows the same thing computed at full size from the course’s own trained network. Between the ideas it supplies what a working model needs: the activation functions and their derivatives, backpropagation through more than one hidden layer, the optimizers that replace plain gradient descent, the overfitting that a held-out set exposes and weight decay tames, and the two layer types, convolution and attention, that image and text models are built from.
- A network is a function
- Training is gradient descent
- Backpropagation through two hidden layers
- Optimizers: momentum and Adam
- What a trained network is good for, and overfitting
- Two more layers: convolution and attention
- Embeddings
- Encoders, decoders, and one shared space
- Coordinate networks
- The papers
- Exercises
1 · A network is a function
There is no mystery in evaluating one. The point of the next example is that you can do it with a pocket calculator, and that every larger network is the same arithmetic with more rows.
| pre-activations | \(z_1 = 1.0\times0.4 + 0.5 = 0.9\), \(z_2 = -1.0\times0.4 + 0.5 = 0.1\) |
| activations | \(a_1 = \tanh 0.9 = 0.7163\), \(a_2 = \tanh 0.1 = 0.0997\) |
| output | \(f = 0.1 + 0.8\times0.7163 - 0.6\times0.0997 = 0.1 + 0.5730 - 0.0598 = 0.6132\) |
1.1 The nonlinearity
Without the squashing step a network of any depth is a single linear map, because a composition of linear maps is linear; the nonlinearity is what makes depth do anything. Which one is a choice, and the choices in use differ in what their derivative does, which Section 2 makes matter.
| name | \(\sigma(z)\) | \(\sigma'(z)\) | value at \(z = 0.9\) | slope at \(0.9\) | where it is used |
|---|---|---|---|---|---|
| sigmoid | \(1/(1 + e^{-z})\) | \(\sigma(1 - \sigma)\) | 0.711 | 0.206 | probabilities; gates |
| tanh | \((e^{z} - e^{-z})/(e^{z} + e^{-z})\) | \(1 - \tanh^2 z\) | 0.716 | 0.487 | the perceptron above; small networks |
| ReLU | \(\max(0, z)\) | \(0\) or \(1\) | 0.900 | 1 | most deep networks since 2012 |
| SiLU | \(z\,\sigma(z)\) | \(\sigma(z)\,(1 + z(1 - \sigma(z)))\) | 0.640 | 0.896 | the course denoiser; diffusion U-Nets |
| GELU | \(z\,\Phi(z)\), \(\Phi\) the normal CDF | numerical | 0.734 | near 0.9 | transformers |
The saturating functions were the original choice and are still right for a small network like the perceptron; in a deep one their vanishing derivative multiplies through the layers and the early weights stop learning (the vanishing gradient). The rectifiers keep the gradient alive on their active side, and their smooth variants keep it alive everywhere.
2 · Training is gradient descent
Training is not programming. Given examples \((x_i, y_i)\), define a loss such as the mean squared error \(\mathcal{L}(\theta) = \frac{1}{N}\sum_i (f_\theta(x_i) - y_i)^2\), and repeat two steps: compute the gradient \(\nabla_\theta \mathcal{L}\), the direction in weight space that increases the loss fastest, and move every weight a little the other way,
\[ \theta \;\leftarrow\; \theta - \eta\, \nabla_\theta \mathcal{L}(\theta), \]with a small learning rate \(\eta\). The gradient is computed by the chain rule, organized so that every weight’s derivative is obtained in one backward sweep through the diagram; that organization is backpropagation (Rumelhart, Hinton & Williams 1986). For the perceptron the four derivatives are short enough to write down:
\[ \begin{gathered} \frac{\partial \mathcal{L}}{\partial b_2} = \delta, \qquad \frac{\partial \mathcal{L}}{\partial w^{(2)}_h} = \delta\, a_h, \qquad \delta = 2\,(f - y), \\[4pt] \frac{\partial \mathcal{L}}{\partial w^{(1)}_h} = \delta\, w^{(2)}_h (1 - a_h^2)\, x, \qquad \frac{\partial \mathcal{L}}{\partial b^{(1)}_h} = \delta\, w^{(2)}_h (1 - a_h^2), \end{gathered} \]for a single example (average over the batch for more), using \(\tanh' = 1 - \tanh^2\). Notice how the backward sweep reuses the forward one: the \(a_h\) were already computed, and \(\delta\) is one number passed back through the red edges, then through the green boxes, to reach the blue edges.
| \(\partial/\partial b_2\) | \(0.6265\) |
| \(\partial/\partial w^{(2)}\) | \(0.6265\times(0.7163,\; 0.0997) = (0.4487,\; 0.0624)\) |
| \(1 - a_h^2\) | \((1 - 0.5131,\; 1 - 0.0099) = (0.4869,\; 0.9901)\) |
| \(\partial/\partial b^{(1)}\) | \(0.6265\times(0.8\times0.4869,\; -0.6\times0.9901) = (0.2440,\; -0.3722)\) |
| \(\partial/\partial w^{(1)}\) | the previous row times \(x = 0.4\): \((0.0976,\; -0.1489)\) |
epochs from 0 and watch the curve bend a few steps per
frame while the loss plot falls; try 2 hidden units (it cannot bend
enough) and 40 (it fits a step with a smooth ramp).
3 · Backpropagation through two hidden layers
The perceptron has one hidden layer, and its gradient has a special form. The general form is one rule applied once per layer: the error signal \(\boldsymbol\delta^{(k)} = \partial\mathcal{L}/\partial\mathbf{z}^{(k)}\) at layer \(k\) is the error signal of the layer above, multiplied by that layer’s weights transposed, times the derivative of the activation, \[ \boldsymbol\delta^{(k)} \;=\; \big(W^{(k+1)\top}\,\boldsymbol\delta^{(k+1)}\big)\odot \sigma'(\mathbf{z}^{(k)}), \qquad \frac{\partial\mathcal{L}}{\partial W^{(k)}} = \boldsymbol\delta^{(k)}\,\mathbf{a}^{(k-1)\top}, \quad \frac{\partial\mathcal{L}}{\partial \mathbf{b}^{(k)}} = \boldsymbol\delta^{(k)}, \] with \(\odot\) the elementwise product and \(\mathbf{a}^{(k-1)}\) the activations feeding the layer. Each weight’s gradient is the error arriving at its output times the activation at its input. The forward pass stores every \(\mathbf{a}\); the backward pass is one matrix multiply per layer.
| forward, layer 1 | \(\mathbf{z}^{(1)} = (0.9, 0.1)\), \(\mathbf{a}^{(1)} = (0.7163, 0.0997)\), as before |
| forward, layer 2 | \(z^{(2)}_1 = 0.6(0.7163) - 0.4(0.0997) + 0 = 0.3899\), \(z^{(2)}_2 = 0.3(0.7163) + 0.9(0.0997) - 0.2 = 0.1046\); \(\mathbf{a}^{(2)} = (0.3713, 0.1042)\) |
| forward, output | \(f = 0.1 + 0.8(0.3713) - 0.6(0.1042) = 0.3345\); loss \((0.0345)^2 = 0.0012\) |
| \(\delta^{(3)}\) | \(2(f - y) = 0.0690\); gradients \(\partial/\partial b^{(3)} = 0.0690\), \(\partial/\partial\mathbf{w}^{(3)} = 0.0690\,\mathbf{a}^{(2)} = (0.0256, 0.0072)\) |
| \(\boldsymbol\delta^{(2)}\) | \(\delta^{(3)}\,\mathbf{w}^{(3)}\odot(1 - \mathbf{a}^{(2)2}) = 0.0690\,(0.8, -0.6)\odot(0.8621, 0.9891) = (0.0476, -0.0410)\) |
| \(\partial/\partial W^{(2)}\) | \(\boldsymbol\delta^{(2)}\mathbf{a}^{(1)\top} = \begin{pmatrix} 0.0341 & 0.0047 \\ -0.0293 & -0.0041 \end{pmatrix}\); \(\partial/\partial\mathbf{b}^{(2)} = (0.0476, -0.0410)\) |
| \(\boldsymbol\delta^{(1)}\) | \(W^{(2)\top}\boldsymbol\delta^{(2)} = (0.6(0.0476) + 0.3(-0.0410),\; -0.4(0.0476) + 0.9(-0.0410)) = (0.0163, -0.0559)\), times \((1 - \mathbf{a}^{(1)2}) = (0.4869, 0.9901)\): \((0.0079, -0.0553)\) |
| \(\partial/\partial W^{(1)}\) | \(\boldsymbol\delta^{(1)}\,x = (0.0032, -0.0221)\); \(\partial/\partial\mathbf{b}^{(1)} = (0.0079, -0.0553)\) |
| after the step | \(f = 0.3126\), loss \(0.0002\): down by a factor of six |
Automatic differentiation (PyTorch’s backward()) does
exactly this sweep for any composition of operations, recording the
forward pass as a graph and walking it in reverse; nobody writes the
gradient of a modern network by hand. The point of writing this one is
that there is nothing else in the box.
4 · Optimizers: momentum and Adam
Plain gradient descent has two faults that the loss curve of Section 2 shows: it takes tiny steps where the loss surface is flat, and it oscillates where the surface is a narrow valley, because the gradient points across the valley more than along it. Two amendments fix both, and one of them is the default of every training script.
On a large network trained on minibatches, whose gradients are noisy estimates, Adam’s per-weight normalization is what makes one learning rate work across layers whose gradient scales differ by orders of magnitude, and the AdamW variant (weight decay applied directly, Section 5) is the default optimizer of the diffusion models and transformers of the later chapters. The course’s denoiser was trained with it.
5 · What a trained network is good for, and overfitting
Two facts about a trained network carry the rest of the chapter.
Capacity. A network with enough hidden units can approximate any continuous function on a bounded domain to any accuracy (Cybenko 1989, Hornik 1991). Each \(\tanh\) unit is a smooth step placed at \(x = -b^{(1)}_h/w^{(1)}_h\) with steepness \(w^{(1)}_h\); a sum of enough steps can trace any curve. The theorem says nothing about how many units or how much training; the left panel below shows the practical version.
Smoothness between the examples. More important for what follows: a trained network interpolates. Between two training points it draws a smooth curve, not a jump to the nearer one. A lookup table has the first property trivially (store every example) and the second not at all. The right panel is the whole argument of the diffusion chapter in one picture: the exact denoiser it derives is a lookup table over the training set, and replacing it by a network is what lets the model produce images that are not in the set.
5.1 Overfitting, and the held-out set
Capacity cuts both ways. A network large enough to trace any curve can trace the noise in its examples as well as the signal, and a training loss that keeps falling then says nothing about how the network will do on data it has not seen. The only honest measure is a held-out set: examples the optimizer never touches, evaluated as training proceeds.
Other regularizers do the same job by other means: dropout zeroes a random subset of activations at each training step so that no unit can rely on a specific partner; data augmentation (flipping, cropping, recoloring the examples) enlarges the training set with transformations the network should be invariant to; and more data is the regularizer that beats all of them, which is why the diffusion chapter’s largest models are trained on billions of images.
6 · Two more layers: convolution and attention
A fully connected layer on a \(512\times512\) image would have \(786{,}432\) inputs per unit and learn nothing about the fact that a cat in the corner is the same cat as one in the middle. Two layer types encode such structure, and between them they cover every model in the later chapters.
6.1 Convolution
Stacking convolutions with downsampling between them (pooling, or strided convolution) gives each deeper unit a larger receptive field: edges, then corners and textures, then parts, then objects. Running the stack in reverse with upsampling gives a decoder, and a stack with skip connections from each encoder level to the matching decoder level is the U-Net, the denoiser of the first diffusion models (the diffusion chapter). The filters of the image-space chapter are convolutions with kernels chosen by hand; a convolutional layer is the same operation with kernels chosen by the loss.
6.2 Attention
A transformer is attention layers alternating with small fully connected layers, applied to a sequence of tokens with a positional encoding added so that order is not lost (the same sines and cosines as the NeRF encoding). It is the architecture of CLIP’s text encoder (Section 8), of the language models, and since 2023 of the largest diffusion models, which cut the image into patches and treat them as tokens. The cross-attention by which a caption steers an image model is the same formula with the queries from the image tokens and the keys and values from the caption’s.
7 · Embeddings
Nobody places the points. A network puts them where its task goes better; the famous 2013 result was that word vectors trained only to predict neighboring words end up supporting arithmetic, with “king” minus “man” plus “woman” landing near “queen” (Mikolov et al.). The lecture’s embedding exhibit is a smaller and more honest version that you can check to the last digit: the digit-drawing network of the diffusion chapter carries a table of eleven 64-dimensional class vectors (one per digit, one for “no class”) that it learned only from denoising. Nobody told it that a 4 looks like a 9.
css551/ directory with
node -e "…".
- Train the perceptron of Section 2 and evaluate it at a point that is not a training point:
import('./lib/core/mlp-fit.js').then(m=>{const r=m.fit({target:'wave',hidden:12,epochs:3000}); console.log('loss',r.loss.toFixed(5),'f(0.25)=',m.predict(r.model,0.25).toFixed(4), 'target',m.TARGETS.wave(0.25).toFixed(4));});Expected:loss 0.00079 f(0.25)= 0.4096 target 0.4181. The point \(x = 0.25\) lies between two training points; the network is within 0.01 of the hidden curve there. Changehiddento 2 and the loss stays above 0.05. - Read two class vectors from the trained denoiser and compare them:
import('./lib/core/pca2.js').then(({cosine})=>{const fs=require('fs'); const w=new Float32Array(fs.readFileSync('lib/assets/mnist/mlp-denoiser.bin').buffer.slice(0)); const e=(c)=>Array.from(w.subarray(c*64,(c+1)*64)); console.log('cos(4,9)=',cosine(e(4),e(9)).toFixed(3),'cos(1,0)=',cosine(e(1),e(0)).toFixed(3));});Expected:cos(4,9)= 0.537 cos(1,0)= 0.141, the two extremes of the matrix above. The class table is the first \(11\times64\) floats of the weight file. - In the learning demo, set 40 hidden units and the step target and watch the ramp: it is the \(H = 40\) curve of Section 5, and it overshoots at the step for the same reason.
8 · Encoders, decoders, and one shared space
An encoder is a network that maps a thing to its embedding, \(E : \text{thing} \to \mathbb{R}^d\); a decoder maps back, \(D : \mathbb{R}^d \to \text{thing}\). Trained together to reproduce their input through a narrow middle, \(\min_{E,D} \sum_i \|D(E(x_i)) - x_i\|^2\), they form an autoencoder, and the middle is forced to keep only what matters (Hinton & Salakhutdinov 2006). Stable Diffusion’s autoencoder is exactly this, with a variational twist that keeps the latent well behaved (the diffusion chapter says which): a \(512\times512\times3\) image becomes a \(64\times64\times4\) latent, 48 times fewer numbers.
Two encoders, one space. Train an image encoder \(E_I\) and a text encoder \(E_T\) side by side on hundreds of millions of (image, caption) pairs with a contrastive objective: for each batch, a matching pair must have a higher cosine than every mismatched pair,
\[ \mathcal{L} \;=\; -\sum_{i} \log \frac{\exp\big(\cos(E_I(x_i), E_T(t_i))/\tau\big)} {\sum_{j} \exp\big(\cos(E_I(x_i), E_T(t_j))/\tau\big)} , \]with a temperature \(\tau\). After training, a sentence and a photograph are vectors in the same space, so “aim the picture at this caption” becomes geometry. This is CLIP (Radford et al. 2021), and every concept you can say in words is now a coordinate.
Two consequences of the shared space matter for graphics. A CLIP embedding of a caption is the condition every text-to-image model reads, and its cosine to the embedding of a rendered image is a score that can be optimized: that is the signal DreamFusion follows, through a differentiable renderer, to shape a 3D scene from a sentence. And an embedding is many-to-one, which is the subject of the image-space chapter’s last section.
9 · Coordinate networks
Every network so far was trained on many examples to answer for new ones. A coordinate network is trained on one signal to reproduce that signal: its input is a position, its output the value there. It isolates the function-approximation nature of a network with nothing else in the way, and it is the building block of the radiance fields of the learned-scenes chapter.
lectures/L14-neural-nets-embeddings/analysis/coord-net-figures.mjs with lib/core/coord-net.js: ReLU on raw \((x, y)\), 11.82 dB; ReLU on Fourier features with \(\sigma = 6\), 21.21 dB; SIREN with \(\omega_0 = 30\), 18.89 dB.9.1 Spectral bias
A network trained by gradient descent fits the low frequencies of its target first and the high frequencies late or never (Rahaman et al. 2019). A ReLU network is continuous and piecewise linear in its input, so a 4-pixel checkerboard asks for a fold every 4 pixels in both directions; in the neural tangent kernel analysis of Tancik et al. (2020) the error’s frequency components shrink at rates that fall with frequency. The same bias is what made the networks of Section 5 smooth between their examples. On a picture with only low frequencies (three soft Gaussian blobs) the plain network reaches 41.54 dB in 2,000 steps: the problem is the signal’s frequency content, not ReLU as such.
9.2 Fourier features and positional encoding
| \(\sigma\) | 1 | 3 | 6 | 12 | 24 | 48 |
|---|---|---|---|---|---|---|
| PSNR on the pixels | 14.79 | 20.04 | 21.21 | 18.72 | 15.55 | 15.11 |
| PSNR between the pixels | 18.06 | 22.58 | 21.03 | 16.34 | 9.12 | 9.02 |
9.3 SIREN: periodic activations
9.4 Multiresolution hash encoding
Instant NGP (Müller et al. 2022) trains the encoding itself. At each of \(L\) grid resolutions \(N_l = \lfloor N_{\min} b^l\rfloor\) the point’s cell is found, the feature vectors (\(F\) numbers each) of its corners are read from a table of \(T\) entries and interpolated, and the \(L \cdot F\) results are concatenated and passed to a small MLP. Coarse grids have one entry per vertex; fine grids are hashed, \(h(\mathbf v) = (v_1 \cdot 1 \oplus v_2 \cdot 2654435761) \bmod T\), and colliding vertices share an entry. With \(T = 2^{14}\), the corners of one cell, \((37, 12), (38, 12), (37, 13), (38, 13)\), hash to 13417, 13418, 11736, 11739. With \(L = 16\), \(F = 2\), \(T = 2^{19}\) the tables hold at most \(16\cdot2^{19}\cdot2 = 16{,}777{,}216\) parameters, yet each point reads only 4 entries per level in 2D, so a training step is cheap; in 2D with \(N_{\min} = 16\), \(N_{\max} = 2048\) (\(b = 1.3819\)) the first 12 levels fit densely and the last 4 are hashed. The paper reports training “in a matter of seconds”. The same recipe, encoding plus MLP plus loss, represents video \((x, y, t)\), shapes as signed distances \((x, y, z)\), and radiance fields \((x, y, z, \text{direction}) \mapsto (\text{density}, \text{colour})\), whose loss compares rendered pixels with photographs (the learned-scenes chapter).
css551/ directory with node -e "…".
- Train the three networks of this section for 1,000 steps:
import('./lib/core/coord-net.js').then(C=>{const img=C.makeImage('card'); for(const k of ['relu','fourier','siren']){const t=C.makeTrainer(k,img); for(let s=0;s<1000;s++)t.step(); console.log(k,C.paramCount(t.net),C.psnr(C.mse(C.render(t.net),img)).toFixed(2));}});Expected:relu 2643 11.67,fourier 4083 19.24,siren 2643 18.14. - Encode pixel \((10, 20)\) with the Fourier features:
import('./lib/core/coord-net.js').then(C=>{const net=C.makeNet('fourier',{sigma:6}); const o=new Float64Array(32);C.encode(net,-0.671875,-0.359375,o); console.log(o[0].toFixed(4),o[16].toFixed(4),o[1].toFixed(4),o[17].toFixed(4));});Expected:-0.1222 0.9925 -0.7661 0.6427, the two \((\cos, \sin)\) pairs of the worked example. - In the demo, play to step 2,000, then set \(\sigma = 48\) and play again: the Fourier panel turns to speckle. Switch the picture to blobs and compare the plain network’s PSNR with its test-card value.
10 · The papers
| Year | Paper | What it added |
|---|---|---|
| 1986 | Rumelhart, Hinton & Williams | backpropagation |
| 1989 / 1991 | Cybenko; Hornik | universal approximation |
| 1998 | LeCun, Bottou, Bengio & Haffner | convolutional networks trained end to end (on the digits of this text) |
| 2006 | Hinton & Salakhutdinov | deep autoencoders |
| 2013 | Mikolov et al., word2vec | embeddings that support arithmetic |
| 2015 | Kingma & Ba, Adam | the default optimizer |
| 2016 | He et al., ResNet | skip connections; depth without vanishing gradients |
| 2017 | Vaswani et al., Transformer | attention as the whole architecture |
| 2020 | Ho, Jain & Abbeel, DDPM | predict the noise; it works at scale |
| 2021 | Radford et al., CLIP | text and images, one space |
| 2019 | Rahaman et al. | spectral bias: networks learn low frequencies first |
| 2020 | Mildenhall et al., NeRF | positional encoding of coordinates; radiance fields |
| 2020 | Tancik et al. | Fourier features; the scale as a bandwidth |
| 2020 | Sitzmann et al., SIREN | sine activations and their initialization |
| 2022 | Müller et al., Instant NGP | multiresolution hash encoding |
11 · Exercises
Solution
\(z = (-0.4 + 0.5,\; 0.4 + 0.5) = (0.1, 0.9)\), \(a = (0.0997, 0.7163)\), \(f = 0.1 + 0.8\times0.0997 - 0.6\times0.7163 = 0.1 + 0.0798 - 0.4298 = -0.2500\). The two units have swapped roles: the network is not symmetric in \(x\) because \(w^{(2)}\) is not.Solution
After the first step \(f = 0.4814\), so \(\delta = 2(0.4814 - 0.3) = 0.3628\) and \(\partial\mathcal{L}/\partial b_2 = 0.3628\). Moving only \(b_2\): \(b_2 = 0.0374 - 0.0363 = 0.0011\), \(f = 0.4451\), loss \(0.0211\), down from \(0.0329\). The full step (all seven weights) would fall faster, but every single weight’s step is in a descending direction on its own; that is what a gradient is.Solution
\(z = (0.9, 0.1)\), both positive, so \(a = z = (0.9, 0.1)\) and \(f = 0.1 + 0.72 - 0.06 = 0.76\). \(\delta = 0.92\); the ReLU derivative is 1 for both units, so \(\partial/\partial w^{(1)}_2 = \delta\, w^{(2)}_2 \cdot 1 \cdot x = 0.92\times(-0.6)\times0.4 = -0.221\), against \(-0.149\) with \(\tanh\), whose derivative 0.99 at \(z_2 = 0.1\) barely damped it and whose derivative 0.49 at \(z_1 = 0.9\) halved the other unit’s gradient. Had \(z_2\) been negative, the ReLU gradient through it would be exactly zero: a dead unit.Solution
Gradient \(0.0341\), so the predicted change is \(-0.01\times0.0341 = -0.00034\), from 0.00119 to 0.00085. Recomputing: \(z^{(2)}_1 = 0.59(0.7163) - 0.4(0.0997) = 0.3827\), \(a = 0.3650\), \(f = 0.1 + 0.2920 - 0.0625 = 0.3295\), loss \(0.00087\). The linear estimate is good to two digits at this step size, which is the assumption gradient descent rests on.Solution
\(m_2 = 0.9(0.0627) + 0.1(0.6265) = 0.1190\), \(\hat m = 0.1190/(1 - 0.81) = 0.6265\); \(v_2 = 0.999(0.000392) + 0.001(0.3925) = 0.000784\), \(\hat v = 0.000784/(1 - 0.998) = 0.3925\); update \(= 0.01\times0.6265/0.6265 = 0.01\). A consistent gradient keeps the step at exactly \(\eta\); the bias correction is what makes the early steps come out this cleanly.Solution
Twelve points are 12 constraints; \(H = 4\) already has 13 weights, so in principle enough freedom exists. Freedom is not reachability: gradient descent finds a nearby smooth solution, the target points carry noise (\(\pm0.03\)), and a smooth curve does not pass through noisy points exactly. The residual is the noise, not a shortage of weights; the \(H = 40\) run stops at a similar loss.lib/core/mlp-fit.js, train \(H = 40\) on makeSamples('wave', 12, 3, 0.15) for 5000 epochs and measure the mean squared error against TARGETS.wave on 150 evenly spaced points every 250 epochs. Where is the minimum?
Solution
The generator’s run finds the held-out minimum near epoch 950 (0.0099) with the error rising to 0.0108 by 5000; a run with the library’strainStep and the same seed reproduces the curve to the precision of the figure. A different seed moves the minimum by a few hundred epochs but not its existence: with 12 noisy points and 121 weights, late training always fits noise.
Solution
Mirror: \(+3.96\), a right-facing edge, and the ReLU passes it. Transposed kernel on the original: the rows are identical, so the top row contributes \(-(0.99 + 2\cdot0.29 + 0)\) and the bottom row \(+(0.99 + 2\cdot0.29 + 0)\): exactly 0. A horizontal-edge detector sees nothing in a vertical edge; the two kernels are orthogonal detectors, and a trained first layer contains both.Solution
By symmetry with token 1, scores \((0, 0.707, 0.707)\), weights \((0.198, 0.401, 0.401)\), \(\mathbf{y}_2 = (1.102, 1.000)\). Doubling the tokens quadruples every dot product (both query and key double), so the scores become \((0, 2.83, 2.83)\) and the weights sharpen to \((0.029, 0.486, 0.486)\): attention’s temperature is the scale of the vectors, which is why the \(\sqrt d\) is there and why the inputs are normalized before each layer.Solution
The 1: its nearest neighbor (3) has cosine only 0.31, the lowest nearest-neighbor value of any class. A 1 is a single stroke and shares no loop or crossbar with anything; a denoiser that has decided the image is a 1 does different work than for any other class, so the class vector ended up alone.Solution
Weights \(= 2H + H + H^2 + H + 3H + 3 = H^2 + 7H + 3\); setting this to 12,288 gives \(H \approx 107.5\), so \(H = 108\) (12,423 weights). At \(H = 48\) the network has 2,643 weights (4,083 with 16 Fourier frequencies), so it cannot store the pixels one by one: it compresses them into a function. The test card is mostly smooth regions and periodic patterns, which a sum of waves describes with few numbers; the residual error (21.21 dB, MSE 0.0076) is what that compression loses.lib/core/coord-net.js (or the demo’s \(\sigma\) slider and its pause at step 2,000), train the Fourier-feature network at \(\sigma = 3\) and \(\sigma = 24\), and evaluate each at the corners shared by four pixels against the four pixels’ mean. Which \(\sigma\) would you choose for a network that will be rendered at twice the resolution, and why does the PSNR on the pixels alone mislead?