HW8 · Diffusion and learned scenes

Stable Diffusion denoises a 64 × 64 × 4 latent for a 512 × 512 RGB image: 48 times fewer numbers than the pixels.computed: 512 × 512 × 3 / (64 × 64 × 4); Rombach et al., arXiv:2112.10752

Out Thursday December 3 · due Wednesday December 9, 11:59 PM on Canvas · 25 points · JavaScript or Python · hub · schedule · syllabus · all homework

Read before this homework: Diffusion Models §2, the exact denoiser · §4, sampling · Learned Scenes §2, the volume integral · §6, one splat

What you build

Three mechanisms of the neural weeks, each small enough to write out: the exact denoiser for a point cloud and a deterministic DDIM sampler that walks noise onto the spiral; one ray through a volume, composited front to back; and one Gaussian splat, projected to a pixel ellipse, then three splats composited on a pixel row.

JavaScript	Python	What it computes
`denoise`	`denoise`	E[x₀ \| x_t] = Σ w_i data_i, softmax weights with logits −\|x − √ᾱ data_i\|² / (2(1 − ᾱ)).
`ddimStep`	`ddim_step`	ε = (x_t − √ᾱ₁ x̂₀) / √(1 − ᾱ₁); x_next = √ᾱ₂ x̂₀ + √(1 − ᾱ₂) ε.
`sampleDDIM`	`sample_ddim`	Every start through the steps of the given timeline.
`renderRay`	`render_ray`	α_i = 1 − exp(−σ_i δ), weights T_i α_i; the color and the transmittance left.
`projectSplat`	`project_splat`	Σ = M Mᵀ, M = R_z S; Σ′ = J Σ Jᵀ with the pinhole Jacobian; its eigen-radii, angle, and center.
`compositeSplats`	`composite_splats`	Front to back over three 1D splats at u: C = Σ T_k α_k c_k.

alphaBar (the cosine schedule) and timeline are given.
The spiral (400 points) and the 300 seeded normal starts are in data.json and lib/skeletons/data-hw08.js; the sampler check uses the first 60 starts and 50 steps.
The splat: scales (0.16, 0.04, 0.04), 30° about z, center (0.5, 0.3, 2) in camera space, f = 500 px.

The numbers a correct build reproduces

Each row is one check, run on your functions with the published inputs below. The same table appears under the scene in the runner and in the notebook's last cell.

#	Check	Expected	Tolerance	Points
1	denoise: x̂0 for the point x_t at step 15 of 50	(0.03671, 0.149)	0.0001	12
2	ddimStep: the next point, t from step 15 to step 16	(0.85343, 1.04423)	0.0001	14
3	sampleDDIM, 50 steps, first 60 starts: mean distance to the nearest data point	0.00902	0.0001	14
4	renderRay through the five samples: color C	(0.79103, 0.2166, 0.22552)	0.0001	12
5	renderRay: transmittance left after the ray	0.06721	0.0001	8
6	projectSplat: the pixel covariance entries (xx, xy, yy)	(1231.25, 653.269, 477.25)	0.05	14
7	projectSplat: ellipse radii in pixels, angle, center	((40.1061, 10.0001), 30.0054, (125, 75))	0.002	12
8	compositeSplats at u = 0.45: color C and transmittance T	((0.46846, 0.22283, 0.34836), 0.23538)	0.0001	14

The published inputs

{
 "step": {
  "xt": [ 0.866218995142179, 1.05619399937019 ],
  "tFrom": 0.6993,
  "tTo": 0.67932
 },
 "sampler": {
  "steps": 50,
  "count": 60
 },
 "volume": {
  "sigmas": [ 0, 0.5, 4, 8, 1 ],
  "colors": [
   [ 0, 0, 0 ],
   [ 0.5, 0.5, 0.5 ],
   [ 0.9, 0.2, 0.2 ],
   [ 0.9, 0.2, 0.2 ],
   [ 0.2, 0.3, 0.9 ] ],
  "delta": 0.2
 },
 "splat": {
  "s": [ 0.16, 0.04, 0.04 ],
  "rotZ": 30,
  "t": [ 0.5, 0.3, 2 ],
  "f": 500
 },
 "row": {
  "u": 0.45,
  "splats": [
   {
    "mu": 0.35,
    "sd": 0.1,
    "a": 0.8,
    "c": [ 0.85, 0.2, 0.2 ]
   },
   {
    "mu": 0.55,
    "sd": 0.14,
    "a": 0.7,
    "c": [ 0.2, 0.45, 0.9 ]
   },
   {
    "mu": 0.75,
    "sd": 0.08,
    "a": 0.9,
    "c": [ 0.95, 0.75, 0.15 ]
   } ]
 }
}

Implement and replace

Build each function from primitives (arithmetic, exp, sqrt, atan2; in Python, NumPy.) and use the engine's own helper, where one exists, only to check your result. These are off limits in your submission:

Track	Off limits
WebGL (JavaScript)	Any diffusion, rendering or splatting library; the course library's `diffusion.js` sampler and denoiser.
Python	Any diffusion, rendering or splatting library. NumPy is allowed.

Getting started

WebGL track.

Get the homework starter repository (about 2 MB): git clone https://github.com/mgav-uwb/UWB-CSS-551-AU26-homework.git, or Code, then Download ZIP. It gains each homework the day it goes out; git pull fetches it.
In its root folder run python3 -m http.server 8000 and open http://localhost:8000/homework/run.html?hw=hw08.
Edit lib/skeletons/hw08.js, the only file you change, and reload: the Checks table under the scene compares your numbers with the ones below.

The unfilled skeleton runs on this site too, so you can see the page before you start. The setup, the blank page a file opened from disk gives, and how to read the Checks table are in WebGL and three.js for This Course.

Python track.

From the homework starter repository, open homework/hw08/hw08.ipynb in Jupyter or VS Code, with data.json and expected.json in the same folder.
Fill in the functions marked TODO and run all cells: the last cell prints each check with your value and the expected one.

Submitting and grading

WebGL: the one file hw08.js.
Python: the notebook hw08.ipynb, run, with the check cell's output visible.

Grading runs your functions twice: on the published inputs above and on a second set of inputs that is not published. A check earns its points only when your value is within its tolerance on both, so the numbers have to come from the math. The rubric score is the sum of the points earned, out of 100; it counts as 25 course points (the rubric score divided by 4). A submission that does not load or compile earns no check points. Name any collaborators and tools in the submission comment.

Late tokens. You have two for the quarter. Declare one in the submission comment to extend this homework by 48 hours without penalty. Without a token, late work is not accepted, and nothing is accepted after Friday, December 11.