This chapter links to the course’s interactive demos and to original papers on the open web, and it uses MathJax for its formulas. Read it through the course website (or download the file and open it in a browser); a Canvas inline preview will reach neither the demos nor the math. All chapters.

Learned Scenes

A 3D Gaussian with degree-3 spherical harmonics has 59 numbers: 3 for position, 4 for rotation, 3 for scale, 1 for opacity, and 48 for view-dependent color. computed; Kerbl et al., SIGGRAPH 2023

Rather than learn the distribution of all images, a learned scene learns one place from its photographs and renders it from viewpoints the camera never saw. This chapter states the neural radiance field and its volume-rendering integral, derives that integral from the physics of absorption so that its discrete form is not a recipe, runs it by hand along one ray, shows what goes wrong when the ray is sampled too coarsely and how a second, importance-sampled pass fixes it, and explains why Gaussian splatting replaced the network with explicit primitives and got real-time rates. For one splat it builds the covariance from a scale and a rotation, projects it to the screen through the Jacobian of the perspective divide, evaluates its view-dependent color from spherical harmonics, and blends it with two others on one pixel; it then says how a splat scene is trained and grown. A short section returns from fields to the meshes of the rest of the text by marching squares, and the chapter closes by setting both representations against meshes with the numbers their papers report, and against diffusion models, which they meet in score distillation and ControlNet.

Contents
  1. A scene as a function
  2. Rendering a field: the volume integral
  3. Sampling the ray: coarse to fine
  4. Training through the renderer
  5. 3D Gaussian splatting
  6. One splat: shape, projection, color
  7. Training a splat scene
  8. From a field back to a mesh
  9. Learned scenes against meshes, and against learned images
  10. The papers
  11. Exercises

1 · A scene as a function

The other half of the neural era learns one scene from its own photographs and renders it from viewpoints the camera never saw. A neural radiance field (NeRF, Mildenhall et al. 2020) represents the scene as a function, a small network \(F_\theta : (\mathbf{p}, \mathbf{d}) \mapsto (\mathbf{c}, \sigma)\) from a 3D point and a viewing direction to a color and a density, with no mesh anywhere. Density is how much a ray passing through \(\mathbf{p}\) is attenuated per unit length: zero in empty air, large inside a leaf. This is the function view of the image-space chapter applied to a volume rather than a picture plane, and the network is exactly the approximator of the networks chapter: eight layers of 256 units, a few hundred thousand weights, one scene per network.

A camera on the left shoots a ray through a translucent orange blob; seven sample points sit along the ray, three of them inside the blob drawn larger and red; a box on the right labels the network F of position and direction giving color and density, and a second box lists the compositing formulas.
One pixel of a NeRF render. The ray from the camera through the pixel (the ray-tracing chapter builds it) is sampled at points \(s_1 \dots s_7\); the network is evaluated at each, returning a color and a density; the results are composited front to back with the formulas at the right. The density is large only inside the object, so only the red samples contribute.

One detail makes the function view work in practice. Fed raw coordinates, a small network produces blurry fields, because a perceptron is biased toward smooth functions and a scene has sharp edges. NeRF first maps each coordinate through fixed sines and cosines of increasing frequency,

\[ \gamma(p) \;=\; \big(\sin(\pi p), \cos(\pi p), \sin(2\pi p), \cos(2\pi p), \dots, \sin(2^{L-1}\pi p), \cos(2^{L-1}\pi p)\big), \]

and feeds the network \(\gamma\) of every coordinate instead. This is the cosine basis of the image-space chapter again: the network now combines basis functions of many frequencies and can be sharp.

Worked example (positional encoding of one coordinate) With \(L = 4\) and \(p = 0.3\):
\(k\)frequency \(2^k\)\(\sin(2^k \pi p)\)\(\cos(2^k \pi p)\)
010.8090.588
120.951−0.309
24−0.588−0.809
380.9510.309
One number became eight, and two points that differ by \(0.01\) now differ by a lot in the high-frequency entries (\(\cos(8\pi\cdot0.31) = 0.063\) against \(\cos(8\pi\cdot0.30) = 0.309\)), which is what lets the network place an edge between them. NeRF uses \(L = 10\) for position (60 numbers for \(\mathbf{p}\)) and \(L = 4\) for direction. The same encoding gives a transformer its sense of position (the networks chapter); the later hash-grid encodings (Instant-NGP, Müller and others, 2022) replace the sines by learned features looked up in a multi-resolution grid, and cut training from hours to seconds.

2 · Rendering a field: the volume integral

A pixel is made by marching a camera ray \(\mathbf{r}(s) = \mathbf{o} + s\,\mathbf{d}\) through the field and compositing front to back. The formula comes from one physical fact.

Absorption, and the transmittance Light traveling a short distance \(ds\) through a medium of density \(\sigma\) is absorbed in proportion, \(dT = -\sigma\,T\,ds\). Solving, the fraction that survives from the ray’s start to depth \(s\) is \[ T(s) = \exp\!\Big(-\int_{s_n}^{s} \sigma(\mathbf{r}(u))\,du\Big), \] the Beer–Lambert law (the light-transport chapter). The light that reaches the eye from depth \(s\) is what is emitted there, \(\sigma\,\mathbf{c}\,ds\) (the medium glows in proportion to how much of it there is), times what survives the trip back, \(T(s)\).
Definition (volume rendering) \[ C(\mathbf{r}) \;=\; \int_{s_n}^{s_f} T(s)\, \sigma(\mathbf{r}(s))\, \mathbf{c}(\mathbf{r}(s), \mathbf{d})\, ds . \] In practice the ray is sampled at points \(s_1 < \dots < s_N\) with spacings \(\delta_i\), the density taken constant on each slab, and the integral becomes a sum: \[ \alpha_i = 1 - e^{-\sigma_i \delta_i}, \qquad T_i = \prod_{j < i} (1 - \alpha_j), \qquad C = \sum_i T_i\, \alpha_i\, \mathbf{c}_i . \]

The discrete form is exact for a piecewise-constant density: on a slab of constant \(\sigma_i\) the transmittance falls by exactly \(e^{-\sigma_i\delta_i}\), and the light emitted along the slab that reaches its front is \(\int_0^{\delta_i} \sigma_i e^{-\sigma_i u}\,du\;\mathbf{c}_i = (1 - e^{-\sigma_i\delta_i})\,\mathbf{c}_i = \alpha_i\mathbf{c}_i\). So \(\alpha_i\) is the opacity of the \(i\)-th slab, \(T_i\) is how much of the ray is still unblocked when it reaches the slab, and \(T_i\alpha_i\) is the slab’s weight in the final color: it is the alpha compositing of the texturing chapter, run front to back. The weights sum to \(1 - T_{N+1}\); whatever is left is the background.

Worked example (one ray, five samples, unrolled) Spacing \(\delta = 0.2\) everywhere. The network returned:
\(i\)\(\sigma_i\)\(\mathbf{c}_i\)\(\alpha_i = 1 - e^{-0.2\sigma_i}\)\(T_i\)\(w_i = T_i\alpha_i\)
10(air)\(1 - e^{0} = 0\)10
20.5gray (0.5, 0.5, 0.5)\(1 - e^{-0.1} = 0.095\)10.095
34red (0.9, 0.2, 0.2)\(1 - e^{-0.8} = 0.551\)\(1\times0.905 = 0.905\)0.498
48red (0.9, 0.2, 0.2)\(1 - e^{-1.6} = 0.798\)\(0.905\times0.449 = 0.407\)0.325
51blue (0.2, 0.3, 0.9)\(1 - e^{-0.2} = 0.181\)\(0.407\times0.202 = 0.082\)0.015
\(C = 0.095\,(0.5,0.5,0.5) + 0.498\,(0.9,0.2,0.2) + 0.325\,(0.9,0.2,0.2) + 0.015\,(0.2,0.3,0.9) = (0.791,\; 0.217,\; 0.226)\), a red, and \(T_6 = 0.082\times0.819 = 0.067\) of the ray reaches the background. The blue sample behind the red ones barely counts (weight 0.015) because the red slabs absorbed 93 % of the ray before it got there: occlusion, with no depth buffer and no sorting of surfaces, just a product of transmittances.
Left: the transmittance T stepping down from 1 to 0.07 along the five samples, with the per-sample weights drawn as bars peaking at samples 3 and 4. Right: a red swatch labeled with the resulting pixel color (0.791, 0.217, 0.226).
The example as a picture. Transmittance (blue) is the fraction of the ray still alive after each sample; the weights (red bars) are what each sample contributes to the pixel. The two dense red samples take 82 % of the weight between them.
Worked example (three checks on the same ray)

3 · Sampling the ray: coarse to fine

The discrete sum is only as good as the sample spacing, and a real scene is mostly empty space with thin surfaces in it. A fixed spacing that is fine enough to hit every leaf wastes hundreds of network evaluations per ray on air; a coarse one misses the leaf.

Pitfall A surface of density 50 that is 0.02 units thick, at \(s = 0.61\), has an exact opacity of \(1 - e^{-50\cdot0.02} = 0.632\). Sample the ray at \(\delta = 0.2\) starting from \(s = 0\): the samples fall at 0.6 and 0.8, both outside the slab, the density reads zero everywhere, and the rendered opacity is 0: the surface is invisible. Start the same sampling at \(s = 0.02\): the sample at 0.62 lands inside the slab, the slab is assumed to fill the whole 0.2 step, and the opacity comes out \(1 - e^{-50\cdot0.2} = 1.000\), fully opaque. The truth, 0.632, is recovered only when \(\delta\) is comparable to the surface’s thickness (\(\delta = 0.02\) gives 0.632 exactly). Coarse sampling does not blur a thin surface; it either misses it or overstates it, and which one depends on where the samples happen to fall, so the error flickers from ray to ray and frame to frame.
Transmittance along a ray crossing a thin dense slab at 0.61: the exact curve drops from 1 to 0.37 in a narrow step; the fine sampling follows it; the coarse sampling from 0.0 stays at 1 throughout and the coarse sampling from 0.1 also stays at 1, both having missed the slab.
The thin surface under three samplings. Both coarse samplings shown here miss the slab entirely; the offset that hits it (0.02) overshoots to full opacity instead.

NeRF’s answer is hierarchical sampling: march a coarse set of samples, treat their weights \(w_i\) as a distribution along the ray, and draw a second, fine set of samples from that distribution by inverting its cumulative sum, so that the fine samples cluster where the coarse pass found density. Two networks (coarse and fine) are trained together, and the final color uses both sets. It is the importance sampling of the ray-tracing chapter, with the coarse pass supplying the importance.

Worked example (fine samples from the five coarse weights) Normalize the weights of Section 2: \((0, 0.102, 0.534, 0.348, 0.016)\), cumulative \((0, 0.102, 0.636, 0.984, 1)\). A uniform random number \(u\) is mapped to the depth at which the cumulative curve reaches it, interpolating within the slab:
\(u\)0.100.250.500.750.90
slab23344
fine sample \(s\)0.3960.4550.5490.6650.752
Four of the five fine samples land between \(s = 0.4\) and \(0.8\), the two red slabs, and none in the empty first slab; the thin surface of the pitfall, once a coarse sample has brushed it, gets a dozen fine samples across its thickness in the second pass. NeRF uses 64 coarse and 128 fine samples per ray; the hash-grid methods that followed keep an explicit occupancy grid instead and skip empty cells outright.
The five coarse weights as red bars along the ray, their cumulative distribution as a blue staircase rising from 0 to 1 between 0.2 and 1.0, dashed horizontal lines at five uniform values, and five green dots on the axis where those values invert to, clustered under the tallest bars.
Inverting the cumulative distribution of the coarse weights. Where the weights are large the cumulative curve is steep and a uniform interval of \(u\) maps to a short interval of \(s\): more samples per unit length.

4 · Training through the renderer

Every operation above is differentiable: the exponentials, the products, the sum, and the network itself. So the whole march is a function from the weights \(\theta\) to a pixel color, and training is the march run backwards: gradient descent on \(\theta\) until rendered rays match the training photographs,

\[ \min_\theta \sum_{\text{photos}}\;\sum_{\text{pixels}} \big\| C_\theta(\mathbf{r}) - C_{\text{photo}}(\mathbf{r}) \big\|^2 . \]

The camera poses of the photographs are known (recovered beforehand by structure from motion; see 3D scanning and reconstruction in the history chapter), so each pixel of each photograph defines a ray with a known answer: the unprojection of the interaction chapter, once per pixel per photograph. Nothing tells the network where the surfaces are; it discovers that putting density there makes many rays from many viewpoints agree with their photographs, and that is the only way to make them agree. It is slow: hundreds of network evaluations per ray, millions of rays, and the original paper reports one to two days on a single GPU per scene and about thirty seconds per \(800\times800\) frame to render. A batch of 4,096 rays per step, drawn at random from all photographs, is what the Adam optimizer of the networks chapter sees.

What training cannot see, it cannot fix. A region no photograph looked at is unconstrained and fills with whatever density the network finds convenient, which is the source of the floaters, wisps of density hanging in the air of a reconstructed room; and a scene that changes between photographs (a person walking through, the sun moving) is inconsistent with itself and comes out as a smear. Later work adds per-image appearance codes for the lighting, transient-object handling for the people, and priors on density to suppress the floaters.

5 · 3D Gaussian splatting

3D Gaussian splatting (Kerbl et al. 2023) keeps the learn-from-photos idea and replaces the network with millions of explicit primitives: anisotropic Gaussians, each with a position \(\boldsymbol\mu\), a \(3\times3\) covariance \(\Sigma\) (its size and orientation), an opacity, and a view-dependent color. Rendering projects each Gaussian onto the screen, where it becomes a 2D Gaussian with covariance \(\Sigma' = J W \Sigma W^\top J^\top\) (\(W\) the view rotation, \(J\) the Jacobian of the projection), sorts by depth, and blends front to back with the same \(C = \sum_i T_i\alpha_i\mathbf{c}_i\) as above, with \(\alpha_i\) now the Gaussian’s opacity times its falloff at the pixel. That is a rasterizer (the rasterization chapter), so it runs in real time, and it needs the course’s view and projection matrices to do it.

Left: three nested ellipses, the 1, 2 and 3 sigma contours of a 2D Gaussian with covariance 0.5, 0.3, 0.3, 0.2, with its two principal axes drawn. Right: three colored splats on one pixel row, red in front, blue, then yellow, and their front-to-back composite, with a dashed line marking the pixel worked in the text.
Left: a projected splat is an ellipse whose axes are the eigenvectors of \(\Sigma'\) and whose radii are the square roots of its eigenvalues (worked below). Right: three splats seen along one row of pixels, front to back, and their composite. The composite is the same front-to-back formula as the NeRF march with three samples per pixel instead of hundreds.
Worked example (the footprint of one splat) A splat projects to the screen with covariance \(\Sigma' = \begin{pmatrix} 0.5 & 0.3 \\ 0.3 & 0.2 \end{pmatrix}\) (pixel units squared). Trace \(0.7\), determinant \(0.10 - 0.09 = 0.01\); the eigenvalues solve \(\lambda^2 - 0.7\lambda + 0.01 = 0\): \(\lambda = 0.35 \pm \sqrt{0.1225 - 0.01} = 0.35 \pm 0.335\), so \(\lambda_1 = 0.685\), \(\lambda_2 = 0.015\). The ellipse’s radii are \(\sqrt{\lambda_1} = 0.83\) and \(\sqrt{\lambda_2} = 0.12\), a 7-to-1 sliver, tilted by \(\tfrac12\arctan\big(2\cdot0.3/(0.5 - 0.2)\big) = 31.7^\circ\). The rasterizer touches only pixels within about three radii, so this splat costs a few pixels; a splat facing the camera edge-on costs even fewer.
Worked example (three splats on one pixel) Along the row in the figure, at \(u = 0.45\), the three splats (front to back) have opacities \(0.8, 0.7, 0.9\), centers \(0.35, 0.55, 0.75\) and widths \(0.10, 0.14, 0.08\). Each contributes \(\alpha_i = \text{opacity}_i \cdot \exp\!\big(-(u - \mu_i)^2/2\sigma_i^2\big)\):
splatfalloff at \(u\)\(\alpha_i\)\(T_i\)\(w_i\)color
1 (red, front)\(e^{-0.5} = 0.607\)0.48510.485(0.85, 0.2, 0.2)
2 (blue)\(e^{-0.255} = 0.775\)0.5420.5150.279(0.2, 0.45, 0.9)
3 (yellow, back)\(e^{-7.03} = 0.001\)0.0010.2360.000(0.95, 0.75, 0.15)
\(C = 0.485\,(0.85,0.2,0.2) + 0.279\,(0.2,0.45,0.9) + 0.000\,(\dots) = (0.469,\; 0.223,\; 0.348)\), and \(T = 0.235\) of the pixel still shows the background (white in the figure, giving the drawn color \((0.70, 0.46, 0.58)\), a muted purple). The yellow splat is at this pixel invisible not because it is occluded (24 % of the ray reaches it) but because its own footprint does not extend here; one row over it dominates.
A rendered view of the bonsai scene from the Mip-NeRF 360 dataset: a small tree in a pot on a table, reconstructed from photographs as a cloud of Gaussian splats and drawn in the browser.
The “bonsai” capture of the Mip-NeRF 360 dataset (Barron et al. 2022), reconstructed as about 240,000 Gaussians by an independent training run and rendered live by the course’s splat demo, screenshotted headless. Every leaf is a few dozen overlapping ellipses; there is no mesh, no texture, and no light source, only colors that vary with the viewing direction as the photographs did. The small three-colored cross is the demo’s own marker at the camera’s look-at point, not part of the scene. Attribution in media/gsplat/ATTRIBUTION.md.
See it live A real splat scene: the same bonsai, about 240,000 Gaussians rendered live in the browser; fly around it and change the field of view.

6 · One splat: shape, projection, color

The previous section took the screen-space covariance as given. Here is where it comes from, for one splat, with every matrix written out.

6.1 Shape: a scale and a rotation

A \(3\times3\) covariance must be symmetric and positive definite, which gradient descent on its nine entries does not preserve. The splatting paper instead stores a scale vector \(\mathbf{s}\) and a rotation quaternion \(\mathbf{q}\) (the rotation chapter) and builds \[ \Sigma = R\,S\,S^\top R^\top, \qquad S = \mathrm{diag}(\mathbf{s}),\; R = R(\mathbf{q}), \] which is positive definite for any \(\mathbf{s}\) and \(\mathbf{q}\): the TRS factorization of the affine chapter without the translation, applied to the unit sphere.

Worked example (a flat splat) Scale \(\mathbf{s} = (0.16, 0.04, 0.04)\), a disk-like blob four times longer than it is thick, rotated \(30^\circ\) about \(z\), so \(R = \begin{pmatrix} 0.866 & -0.5 & 0 \\ 0.5 & 0.866 & 0 \\ 0 & 0 & 1 \end{pmatrix}\). \(RS\) has columns \((0.1386, 0.08, 0)\), \((-0.02, 0.0346, 0)\), \((0, 0, 0.04)\), and \[ \Sigma = RS(RS)^\top = \begin{pmatrix} 0.0196 & 0.0104 & 0 \\ 0.0104 & 0.0076 & 0 \\ 0 & 0 & 0.0016 \end{pmatrix}. \] Its eigenvalues are \(0.16^2, 0.04^2, 0.04^2\) with the first eigenvector \((0.866, 0.5, 0)\), the rotated long axis: the covariance remembers the scale and the rotation exactly, and the network of gradients can move either without ever producing an invalid shape.

6.2 Projection: the Jacobian of the perspective divide

The projection of the viewing chapter is not affine, so a Gaussian does not project to a Gaussian exactly. Splatting linearizes it at the splat’s center. With the splat at view-space position \(\mathbf{t} = (t_x, t_y, t_z)\) and focal length \(f\) in pixels, the screen position is \((f t_x/t_z, f t_y/t_z)\) and the Jacobian of that map is \[ J = \begin{pmatrix} f/t_z & 0 & -f t_x/t_z^2 \\ 0 & f/t_z & -f t_y/t_z^2 \end{pmatrix}, \] so that \(\Sigma' = J\,W\Sigma W^\top J^\top\), with \(W\) the view rotation (the identity if \(\Sigma\) is already in view space).

Worked example (the splat above, at depth 2 and at depth 4) \(\mathbf{t} = (0.5, 0.3, 2.0)\), \(f = 500\) pixels: \(J = \begin{pmatrix} 250 & 0 & -62.5 \\ 0 & 250 & -37.5 \end{pmatrix}\), the screen center is at \((125, 75)\) pixels, and \[ \Sigma' = J\Sigma J^\top = \begin{pmatrix} 1231 & 653 \\ 653 & 477 \end{pmatrix}. \] From the unrounded entries (1231.25, 653.27, 477.25): trace 1708.5, determinant 160,854; eigenvalues \(1608.5\) and \(100.0\); radii \(40.1\) and \(10.0\) pixels, at \(30.0^\circ\): the long axis of 0.16 units at depth 2 spans \(500\cdot0.16/2 = 40\) pixels, and the rotation survives the projection because the splat lies nearly in the image plane. Move the splat to depth 4: the radii halve to 20.0 and 5.0 pixels and the ellipse covers a quarter of the pixels. The third column of \(J\) (the depth terms) reaches \(\Sigma'\) only through \(\Sigma_{zz} = 0.0016\), because \(\Sigma\) has no \(z\) covariance with \(x\) or \(y\): it adds \(62.5^2\cdot0.0016 = 6.25\), \(62.5\cdot37.5\cdot0.0016 = 3.75\) and \(37.5^2\cdot0.0016 = 2.25\) px\(^2\) to the \(xx\), \(xy\) and \(yy\) entries, small because the splat is thin and near the view axis. Tilt the splat toward the camera and its \(xz\) and \(yz\) covariances bring the depth column in at first order, shearing the ellipse.
Screen-space ellipses of the worked splat: three nested solid ellipses at 1, 2 and 3 sigma centered at (125, 75) pixels with radii 40 and 10 tilted 30 degrees, and three dashed ellipses half the size centered at (62, 37) for the same splat at twice the depth.
The worked splat on the screen at depth 2 (solid) and depth 4 (dashed), contours at one, two and three standard deviations. The rasterizer bins each splat into the \(16\times16\)-pixel tiles its three-sigma ellipse touches, sorts the splats in each tile by depth on the GPU, and blends each tile’s list front to back per pixel.

6.3 Color: spherical harmonics

A leaf looks different from different sides, and a splat’s color must too. Instead of a network, each splat stores the coefficients of its color in the spherical harmonic basis, the Fourier basis on the sphere of directions: \(\mathbf{c}(\mathbf{d}) = \sum_\ell\sum_m \mathbf{k}_{\ell m}\,Y_{\ell m}(\mathbf{d})\). Degree 0 is a constant; degree 1 adds three functions linear in the direction, \(0.4886\,(y, z, x)\); the splatting paper uses degree 3, 16 functions, 48 coefficients per splat for RGB.

Worked example (a degree-1 color from two directions) Coefficients for red \((2.5, 0, 0.6, -0.4)\), green \((1.2, 0, 0.1, 0.1)\), blue \((0.8, 0, -0.3, 0.2)\) on the basis \((Y_{00}, Y_{1,-1}, Y_{10}, Y_{11}) = (0.2821, 0.4886\,y, 0.4886\,z, 0.4886\,x)\). Seen from \(\mathbf{d} = (0.6, 0, 0.8)\) the basis values are \((0.2821, 0, 0.3909, 0.2932)\), and \(r = 2.5(0.2821) + 0.6(0.3909) - 0.4(0.2932) = 0.823\), \(g = 0.407\), \(b = 0.167\): a warm orange. Seen from the mirrored direction \((-0.6, 0, 0.8)\), \(r = 1.057\), \(g = 0.348\), \(b = 0.050\): redder and brighter (the renderer clamps \(r\) to 1). The \(x\) coefficient of red, \(-0.4\), is what makes the two sides differ; a splat with only degree-0 coefficients is a matte point. This is how the bonsai’s leaves keep their highlights as you orbit them without a light source in the scene: the view dependence was captured, not simulated.

7 · Training a splat scene

Training is unchanged in spirit from NeRF: the rasterizer is differentiable with respect to every Gaussian’s position, scale, rotation, opacity and color, so gradient descent moves, stretches and recolors the Gaussians until the renders match the photographs. What is new is that the number of Gaussians changes during training.

ingredientwhat the paper does
initializationone Gaussian per point of the structure-from-motion point cloud that came with the camera poses
loss\((1 - \lambda)\,L_1 + \lambda\,(1 - \text{SSIM})\) between the render and the photograph, \(\lambda = 0.2\) (SSIM as in the image-space chapter)
densificationevery 100 iterations, from iteration 500 to 15,000: a Gaussian whose screen-space position gradient is large (above \(2\times10^{-4}\)) is cloned if it is small (under-reconstruction) or split into two smaller ones if it is large (over-reconstruction)
pruningGaussians whose opacity falls below a threshold are deleted; every 3,000 iterations all opacities are reset near zero so that floaters must re-earn their place
schedule30,000 iterations; spherical-harmonic degree raised one step at a time so that view dependence is learned last

The bonsai above came out of such a run with about 240,000 Gaussians from an initial cloud of a few tens of thousands. The result is a list of primitives an engine can hold beside its meshes, sort with its own depth, and light only by what the photographs already contained.

8 · From a field back to a mesh

The rest of this text renders triangles, and a learned scene is often wanted as one. A density field defines a surface implicitly, as the set where \(\sigma\) crosses a threshold, and the algorithm that turns an implicit surface into triangles is marching cubes (Lorensen and Cline, 1987). Its two-dimensional version, marching squares, shows the whole mechanism.

Definition (marching squares) Sample the field on a grid. For each cell, compare its four corners to the threshold, giving a 4-bit code (16 cases); a table says which of the cell’s edges the contour crosses for that code; the crossing point on each edge is placed by linear interpolation between the two corner values. The segments from all cells join into the contour. Marching cubes is the same with eight corners, 256 cases, and triangles for output.
A 12 by 12 grid of gray cells shading a density field with two blobs, blue dots at samples below 0.5 and red dots above, and a green contour of connected straight segments enclosing the two blobs.
Marching squares on a two-blob density field at threshold 0.5: 22 of the 144 cells are crossed, each contributing one segment.
Worked example (one cell) Cell \((9, 5)\) has corner values \((0.630, 0.247, 0.180, 0.460)\), listed from its lower left counter-clockwise; only the first is above 0.5, so its code is \(0001 = 1\) and the table says the contour cuts the left edge and the bottom edge. On the bottom edge, between 0.630 and 0.247, the crossing is at \((0.5 - 0.630)/(0.247 - 0.630) = 0.34\) of the way from the left corner; on the left edge, between 0.630 and 0.460, at \(0.76\) of the way up. One segment joins those two points. A NeRF’s density is sampled on a \(256^3\) or \(512^3\) grid and marched the same way; the mesh is then simplified by the meshes chapter’s edge collapses and its color baked into a texture by rendering the field from each triangle’s normal.

The extracted mesh inherits the field’s faults: floaters become small disconnected shells, thin structures marched at a coarse grid vanish exactly as in Section 3’s pitfall, and the density threshold is a choice with no right answer for a fuzzy scene. Methods that train a signed-distance field instead of a density (NeuS, VolSDF) give cleaner surfaces at the cost of the volumetric look, and splat scenes have their own extraction routes. Extraction is where the two halves of this text meet: a scene captured as a function, delivered as the vertex and index arrays of the meshes chapter.

Hands-on
  1. Composite the five samples of Section 2 in eight lines, then change the fourth density to 80 and see the blue sample’s weight vanish: 
    const sig=[0,0.5,4,8,1],d=0.2;let T=1,w=[];
for(const s of sig){const a=1-Math.exp(-s*d);w.push(T*a);T*=1-a;}
console.log('weights',w.map(x=>x.toFixed(3)),'T_end',T.toFixed(3),'sum',(w.reduce((a,b)=>a+b,0)).toFixed(3));
    Expected: weights [ '0.000', '0.095', '0.498', '0.324', '0.015' ] T_end 0.067 sum 0.933.
  2. Decompose the projected covariance of Section 6 into its ellipse: 
    const S=[[1231.25,653.27],[653.27,477.25]];const tr=S[0][0]+S[1][1],det=S[0][0]*S[1][1]-S[0][1]*S[1][0];
const l1=tr/2+Math.sqrt(tr*tr/4-det),l2=tr/2-Math.sqrt(tr*tr/4-det);
console.log('radii px',Math.sqrt(l1).toFixed(2),Math.sqrt(l2).toFixed(2),
  'angle',(0.5*Math.atan2(2*S[0][1],S[0][0]-S[1][1])*180/Math.PI).toFixed(1));
    Expected: radii px 40.11 10.00 angle 30.0.
  3. In the splat demo, orbit slowly around a leaf of the bonsai and watch its highlight move with you: that is Section 6.3’s spherical-harmonic color. Then widen the field of view: every ellipse shrinks by the Jacobian of Section 6.2, since \(f\) fell.

9 · Learned scenes against meshes, and against learned images

mesh (the rest of this text)NeRF3D Gaussian splatting
the scene isvertices, triangles, textures, materialsa network \(F_\theta\)a list of Gaussians
authored bya modeler, or a scanner plus cleanupphotographs and a day of optimizationphotographs and half an hour of optimization
rendered byrasterization, in a millisecondray marching, hundreds of network calls per pixelrasterization of sorted splats
lightingcomputed from lights and materials; relightablebaked into the view-dependent color; not relightable without further work
editableyes, vertex by vertexwith difficultysplat by splat, coarsely
Mip-NeRF 360 benchmark, as reported by Kerbl et al. 2023 (Table 1)not in the benchmarkMip-NeRF 360: PSNR 27.69, about 48 h to train, 0.06 frames per second; Instant-NGP (base configuration): PSNR 25.30, about 6 min, 11.7 fps, 13 MBPSNR 27.21, 41 min, 134 fps, 734 MB per scene

The table’s last row is the reason splatting spread in a year: the image quality within half a decibel of the best field, at about 2,200 times its frame rate (134 against 0.06 frames per second), from a representation an engine can already draw. Its memory cost (hundreds of megabytes per room, against a few for a textured mesh) and its baked lighting are the open problems.

Learned scene (NeRF, 3DGS)Learned image (diffusion)
one scene, from its own photosall images, from an archive
a camera you can moveno camera, no scene
renders new views of the same thingsamples new things
trained through a differentiable renderer (your \(V\) and \(P\) inside it)trained by predicting noise; renders nothing itself
a function of 3D position (a field)a function of a noisy image (a denoiser)
meets diffusion in score distillationmeets the pipeline in ControlNet on your depth buffer

The two meet in the middle. Score distillation (DreamFusion) trains a field so that its renders please a diffusion model: the renderer of this chapter supplies the gradient path from a pixel back to the scene, and the denoiser of the diffusion chapter supplies the direction. ControlNet runs the other way: a depth buffer, which a NeRF (Section 2’s expected depth) or a splat scene or an ordinary mesh renderer produces for free, fixes the layout of a diffusion sample. In both, the graphics pipeline provides what the generative model lacks: a scene, a camera, and a place where geometry is a number you control.

10 · The papers

YearPaperWhat it added
1984Kajiya & Von Herzen, “Ray Tracing Volume Densities,” SIGGRAPH 1984the volume rendering integral in graphics
1987Lorensen & Cline, “Marching Cubes,” SIGGRAPH 1987from a scalar field to triangles
2020Mildenhall et al., NeRFa scene as a learned field, rendered by the volume integral
2022Müller et al., Instant-NGPhash-grid encoding; seconds to train
2022Barron et al., Mip-NeRF 360unbounded real scenes; the dataset the bonsai comes from
2022Poole et al., DreamFusiontext-to-3D by score distillation
2023Kerbl et al., 3D Gaussian splattinglearned scenes at real-time rates

11 · Exercises

Exercise 1 (weights sum to one) Show that in the discrete volume formula \(\sum_i T_i\alpha_i = 1 - T_{N+1}\), and check it on the five-sample example.
Solution \(T_i \alpha_i = T_i - T_i(1-\alpha_i) = T_i - T_{i+1}\), so the sum telescopes to \(T_1 - T_{N+1} = 1 - T_{N+1}\). In the example \(0 + 0.095 + 0.498 + 0.325 + 0.015 = 0.933 = 1 - 0.067\).
Exercise 2 (a thinner slab) Halve the spacing to \(\delta = 0.1\) but keep the same five densities. What happens to \(\alpha_3\) and to the final transmittance, and why is that the right behavior?
Solution \(\alpha_3 = 1 - e^{-0.4} = 0.330\) (was 0.551); every \(\alpha\) falls, and \(T_6 = e^{-0.1(0 + 0.5 + 4 + 8 + 1)} = e^{-1.35} = 0.259\) (was 0.067). The ray now traverses half as much material, so more of it survives. Density is per unit length; halving the length halves the absorption exponent, which is exactly what \(\alpha = 1 - e^{-\sigma\delta}\) encodes. A formula with \(\alpha_i = \sigma_i\) would not have this property.
Exercise 3 (the thin surface) With \(\delta = 0.2\), at what starting offsets does the sampling of Section 3’s pitfall hit the slab at all, and what is the rendered opacity when it does?
Solution The samples are at \(\text{offset} + 0.2k\); one lands in \([0.61, 0.63)\) exactly when the offset is in \([0.01, 0.03)\), a tenth of the possible offsets. When it does, that sample is assumed to fill a slab of length 0.2 at density 50, opacity \(1 - e^{-10} = 1.000\). So the surface is rendered as absent nine times in ten and as a solid wall once, with the true answer, 0.632, never. Jittering the offset per ray, which NeRF does, turns this into flicker; hierarchical sampling turns it into the right answer.
Exercise 4 (hierarchical sampling) Where does \(u = 0.64\) land under the cumulative distribution of Section 3, and what fraction of fine samples fall in slab 3?
Solution The cumulative reaches 0.636 at the end of slab 3 and 0.984 at the end of slab 4, so 0.64 is \((0.64 - 0.636)/(0.984 - 0.636) = 0.011\) into slab 4: \(s = 0.6 + 0.011\times0.2 = 0.602\). Slab 3 receives every \(u\) in \([0.102, 0.636)\), 53 % of the fine samples, in proportion to its weight 0.534.
Exercise 5 (a round splat) What covariance gives a circular footprint of radius 2 pixels, and what does the \(2\times2\) covariance of a splat seen exactly edge-on look like?
Solution \(\Sigma' = \begin{pmatrix} 4 & 0 \\ 0 & 4 \end{pmatrix}\): equal eigenvalues, radii \(\sqrt4 = 2\), no preferred direction. A splat seen edge-on has one eigenvalue near zero, for example \(\begin{pmatrix} 4 & 0 \\ 0 & 0.01 \end{pmatrix}\) after rotation: a line 2 pixels long and a tenth of a pixel wide. Splats that thin are what let a Gaussian scene represent a sharp edge or a leaf.
Exercise 6 (the Jacobian) Move the worked splat of Section 6 sideways to \(\mathbf{t} = (1.5, 0.3, 2.0)\). What changes in \(J\), and what changes in \(\Sigma'\)?
Solution Only the third column: \(-f t_x/t_z^2 = -500\cdot1.5/4 = -187.5\) in the first row (it was \(-62.5\)). The worked \(\Sigma\) has zero \(xz\) and \(yz\) entries, so the third column of \(J\) reaches \(\Sigma' = J\Sigma J^\top\) only through \(\Sigma_{zz} = 0.0016\): \(\Sigma'_{xx}\) gains \(187.5^2\times0.0016 = 56.25\) px\(^2\) over the depth-free 1225 (against 6.25 at \(t_x = 0.5\)), becoming 1281.25; \(\Sigma'_{xy}\) gains \(187.5\cdot37.5\cdot0.0016 = 11.25\), becoming 660.77; \(\Sigma'_{yy}\) stays 477.25. The radii grow to 40.7 and 10.3 pixels and the angle drops to \(29.3^\circ\). A thicker splat, \(\Sigma_{zz} = 0.0256\), would gain \(187.5^2\times0.0256 = 900\) px\(^2\) in \(\Sigma'_{xx}\): a splat off to the side of the view, extended in depth, smears sideways on screen, the perspective foreshortening of its far end.
Exercise 7 (spherical harmonics) Evaluate the red channel of Section 6.3’s splat from straight ahead, \(\mathbf{d} = (0, 0, 1)\), and from the side, \((1, 0, 0)\). Which coefficient could be zeroed without changing the frontal color?
Solution From \((0, 0, 1)\): basis \((0.2821, 0, 0.4886, 0)\), \(r = 0.705 + 0.293 = 0.998\). From \((1, 0, 0)\): basis \((0.2821, 0, 0, 0.4886)\), \(r = 0.705 - 0.195 = 0.510\). The \(x\) coefficient (\(-0.4\)) does not enter the frontal color and could be zeroed for it; it is exactly what darkens the side view. The \(y\) coefficient is already zero, so the splat looks the same from above and below.
Exercise 8 (marching squares) A cell has corner values \((0.7, 0.2, 0.8, 0.3)\) (lower left, lower right, upper right, upper left) at threshold 0.5. What is its code, and why is this case ambiguous?
Solution Bits \((1, 0, 1, 0)\), code \(0101 = 5\): two opposite corners above. The contour must cut all four edges, and the four crossings can be joined two ways, separating the two high corners or connecting them through the middle. The value at the cell’s center decides (here \(0.5\), which is on the threshold, so either); marching cubes has the same ambiguity on faces and resolves it with the same test. The generator’s table joins the lower-left corner’s edges together.
Exercise 9 (what each can and cannot do) You have a splat scene of a room and a diffusion model. Which one do you use to (a) render the room from the doorway, (b) put a different painting on its wall in one still image, (c) make a video walking through it with a painting changed? Say what breaks in (c).
Solution (a) The splat scene: it has a camera. (b) The diffusion model, with the splat render as a depth or edge condition (ControlNet) and a mask on the painting: it has appearance. (c) Neither alone: the splat scene cannot change the painting, and the diffusion model has no scene, so a painting changed frame by frame flickers, because nothing ties frame sixty’s painting to frame one’s. Editing the splat scene itself (repainting the Gaussians on the wall) is the current research answer, and it is a graphics problem.