This chapter links to the course’s interactive demos and to original papers on the open web, and it uses MathJax for its formulas. Read it through the course website (or download the file and open it in a browser); a Canvas inline preview will reach neither the demos nor the math. All chapters.

Inverse and Differentiable Rendering

“All models are wrong, but some are useful.” George Box, Robustness in the Strategy of Scientific Model Building, 1979

A renderer maps a scene’s parameters (shapes, materials, lights, camera) to an image. Inverse rendering asks for the parameters given the image. The modern answer is to make the renderer differentiable and descend the gradient of an image loss: render, compare with the photograph, push the error back through every pixel to every parameter, and repeat. Shading and lighting differentiate by the chain rule. Visibility does not: a pixel is inside an object or outside it, and the derivative of that step is zero almost everywhere, so a naive differentiable renderer cannot see an edge move. The four families of fixes (edge sampling, reparameterization, path-space boundary integrals, and soft rasterization) are the core of the field, and the systems built on them (Mitsuba 3 and Dr.Jit) recover materials, lights, geometry and volumes from photographs. Every worked number below comes from the course’s own 64 × 64 differentiable renderer, the engine of the inverse-render demo.

1 · Rendering as a function

Every renderer in this text, the rasterizer of the rasterization chapter and the path tracer of the ray-tracing chapter alike, computes a function from numbers that describe a scene to numbers that describe an image.

The course renderer has \(m = 8\): an RGB albedo \(\rho\), a light direction given by azimuth and elevation, \(\mathbf{l} = (\cos\phi_e \sin\phi_a,\ \sin\phi_e,\ \cos\phi_e\cos\phi_a)\), and a sphere’s radius \(r\) and center \((c_x, c_y)\), seen by an orthographic camera on a \(64 \times 64\) grid over \([-1,1]^2\). A pixel center at distance \(d\) from the center is covered when \(d < r\); there the normal is \(\mathbf{n} = (d_x, d_y, \sqrt{r^2 - d^2})/r\), and the color is \[ P = a\,S + (1-a)\,B, \qquad S = \rho\,\big(k_a + E\max(0, \mathbf{n}\cdot\mathbf{l})\big), \] with coverage \(a \in \{0, 1\}\), background \(B = (0.10, 0.12, 0.16)\), ambient \(k_a = 0.08\) and light intensity \(E = 1\): the Lambert model of the illumination chapter on the ray–sphere geometry of the ray-tracing chapter. The output has \(3 \cdot 64^2 = 12{,}288\) numbers.

2 · The inverse problem, and why it is ill-posed

Hadamard called a problem well-posed when a solution exists, is unique, and depends continuously on the data. Inverse rendering fails the second test routinely and the third often.

The albedo–intensity product is the simplest of many such families. Shape and shading trade off (a darker patch is either darker paint or a surface turned away from the light); the bas-relief ambiguity lets a flattened shape under a moved light produce the same shaded image; a mirror reflects an environment that cannot be told apart from a texture painted on the mirror. Counting makes the point too: a 64 × 64 photograph gives 12,288 numbers, while a per-pixel albedo texture alone has 12,288 unknowns, a per-pixel depth another 4,096, and the light 3 more. Ramamoorthi and Hanrahan (2001) made the lighting case exact: for a convex Lambertian object, distant lighting can be recovered only up to its low spherical-harmonic frequencies, because the cosine kernel filters out the rest (the illumination chapter uses the same nine coefficients forward).

3 · Analysis by synthesis

Analysis by synthesis means explaining an observation by generating candidates and adjusting them until they match. Blanz and Vetter (1999) fitted a morphable face model to a photograph this way, with hand-derived gradients of a rasterized face; Loper and Black (2014, OpenDR) made a general differentiable rasterizer by approximating the derivative of each pixel with image-space finite differences. The loop is the same in every system since:

   θ  ──render──▶  R(θ)  ──compare with I*──▶  L(θ)
   ▲                                             │
   └────────── θ ← θ − η · (preconditioned ∇L) ◀─┘   (backward pass)

The update is the gradient descent of the networks chapter, usually with Adam (same chapter) because parameters with different units (an albedo in [0, 1], an angle in radians, a radius) need different step sizes and Adam normalizes each by its own gradient history.

4 · Automatic differentiation: forward and reverse

A renderer is a long composition of elementary operations. Automatic differentiation (AD) applies the chain rule to that composition exactly, operation by operation; it is neither symbolic differentiation (which builds an expression) nor finite differences (which perturb inputs and suffer truncation and rounding error). Wengert (1964) described the forward mode; Linnainmaa (1976) the reverse mode, which reached machine learning as backpropagation (Rumelhart, Hinton and Williams 1986). Baydin et al. (2018) survey both, and Griewank and Walther’s Evaluating Derivatives is the reference.

The course renderer writes its reverse pass by hand: for each pixel it evaluates the color, then applies the chain rule backward through that pixel’s coverage, normal, Lambert and Blinn–Phong terms, accumulating eight adjoints. The function lossAndGrad in lib/core/inverse-render.js is about sixty lines; the test file checks every component against central finite differences (next section).

5 · Differentiating shading and lighting

Away from edges, a pixel’s color is a smooth function of the parameters, and its derivatives follow from the shading model of the illumination chapter:

The \(\max(0, \cdot)\) is already a mild problem: its derivative jumps from 0 to 1 at the terminator, and a point in shadow passes no gradient to the light at all. A pixel that should be lit but is in the current shadow cannot pull the light toward itself; only lit pixels vote. Global effects add more of the same: a shadow’s edge, a reflection’s boundary and a refraction all move with the parameters, and each is a discontinuity in the integrand.

6 · The visibility discontinuity

A pixel’s true value is an integral of the scene’s radiance over the pixel’s area (the antialiasing chapter). When an edge crosses the pixel, the integrand is discontinuous, and the discontinuity moves with the parameters.

Sampling a pixel at fixed points and differentiating each sample captures the interior term only. A sample is either on one side of the edge or the other; an infinitesimal change of \(\theta\) moves no sample across, so its derivative is the interior one, and the boundary term is lost. This is not a numerical accident: Monte Carlo estimators of the pixel integral differentiated sample by sample are biased estimators of its derivative.

7 · Four answers to the edge problem

Each family restores the boundary term in a different way.

Rasterization-based systems in production combine these ideas: nvdiffrast (Laine et al. 2020) keeps a hard, fast rasterizer and adds an analytic antialiasing pass whose derivative supplies the silhouette gradient, and PyTorch3D (Ravi et al. 2020) implements a soft rasterizer for meshes and point clouds.

8 · Differentiable path tracers

A naive reverse-mode differentiation of a path tracer must store every intermediate value of every path for the backward pass. Assume 50 stored floats per path vertex: a 512 × 512 image at 64 samples per pixel and 6 bounces records \(512^2 \cdot 64 \cdot 6 \cdot 50 \cdot 4\) bytes, 20.1 GB, for one gradient. The systems below exist to avoid that.

Monte Carlo adds a statistical problem. A gradient estimated from the same random samples as the forward value it multiplies is biased (the product of two correlated estimates is not the estimate of the product), so these systems decorrelate, using independent samples for the two factors. Zeltner et al. (2021) analyze the estimators and their variance; the practical rule is that a differentiable path tracer needs far fewer samples per iteration than a final render, because an optimizer averages noisy gradients over steps, but its estimators must be unbiased or the average converges to the wrong answer.

9 · What can be recovered

Geometry is the hardest case for a reason visible in the course renderer: raw vertex gradients are large near silhouettes and noisy, and a step that follows them moves vertices independently and tangles the mesh. Nicolet et al. precondition the step by \((I + \lambda L)^{-1}\) with \(L\) the mesh Laplacian (the meshes chapter), which spreads each vertex’s update over its neighbors and allows large, smooth steps.

10 · NeRF and splatting as inverse rendering

The learned scenes of the learned-scenes chapter are inverse rendering with a particular choice of representation and renderer, chosen so that the visibility problem does not arise.

• NeRF renders by the volume integral, in which occlusion is a product of transmittances \(T_i = \prod_{j<i}(1-\alpha_j)\) and every quantity is a smooth function of the densities. There is no edge to sample: a surface is a steep but differentiable rise in density.
• 3D Gaussian splatting renders by sorted alpha blending of Gaussians, each with a smooth footprint. Coverage is the Gaussian’s value, a soft edge by construction; only the sort order is discrete, and it changes rarely enough that its gradient is ignored.

Both invert for radiance, not for materials and lights: the scene they recover reproduces the photographs from new viewpoints but bakes in the capture’s lighting, which is why they cannot be relit without a further decomposition (the relighting question of the generative-3D lecture). Physically based inverse rendering recovers the factors at the cost of the ambiguities of Section 2; learned scenes avoid the ambiguities by not factoring.

11 · Regularization and priors

Barron and Malik (2015) recovered shape, illumination and reflectance from a single image by combining many such hand-built priors (smooth shape, sparse reflectance changes, natural illumination). Learned priors replace the hand-built ones: a network trained on many scenes predicts parameters directly or initializes the optimization, and a diffusion model can act as the prior itself, scoring how plausible a rendered view looks. Score distillation (DreamFusion, Poole et al. 2022; the diffusion chapter) is this idea taken to its limit: no photograph at all, only the prior, pushed through a differentiable renderer.

12 · Open problems

• Global-illumination gradients. Boundary terms from shadows, reflections and caustics deep in a path are expensive to find and high in variance; unbiased path-space estimators exist, but their cost per useful gradient is still far above that of a forward render.
• Noise. Gradient variance slows optimization and correlated estimators bias it; how many samples to spend on the forward value against the gradient, and how to share them across iterations, is open.
• Local minima. The loss of an image is not convex in geometry or light: a shape that must move further than its silhouette’s width sees no gradient pulling it there, and the explaining-away of Section 6 is a minimum the optimizer is content with. Coarse-to-fine schedules, wide soft edges at first, and learned initializations are the current remedies.
• Real capture. A photograph is not a render: the camera has an unknown response curve, white balance, lens distortion, vignetting, motion blur and noise, and the scene has interreflections and lights outside the frame. Every one must be modeled or calibrated, or the optimizer absorbs it into the parameters.

13 · The papers

14 · Exercises

import { TRUE_PARAMS, render, mse } from './lib/core/inverse-render.js';
const o = { n: 64, edge: 'hard', shading: 'flat', spp: 16 };
const T = render(TRUE_PARAMS, o);
const at = (r) => { const p = TRUE_PARAMS.slice(); p[5] = r; return mse(render(p, o), T); };
for (const h of [1e-3, 1e-6]) console.log(h, ((at(0.5 + h) - at(0.5 - h)) / (2 * h)).toFixed(4));