Week 14: Generative AI Capstone

Purpose of this lecture#

Thirteen weeks assembled a complete generative modeling curriculum: probabilistic foundations, VAEs, GANs, EBMs, normalizing flows, DDPM, flow matching, conditioning, latent diffusion, evaluation, representation learning, world models, and safety. Each lecture addressed a specific component of the generative modeling pipeline.

This final lecture addresses the harder question: what does it mean to build and evaluate a generative system with scientific rigor? A compelling demo is not evidence of capability; a single FID number is not evidence of general quality; an ablation that was not pre-registered is not evidence of necessity. This lecture develops the experimental discipline that separates published research from exploratory tinkering, and that determines whether a deployed system will perform as reported when exposed to the full distribution of real-world inputs.

The generative pipeline as a system#

The failure-mode table below is the reason this section exists as a system rather than a checklist. Change the latent compression (pixel-space vs. LDM) and you change the compute budget available for the noise schedule; change the noise schedule and the optimal guidance scale moves; push guidance to $s = 20$ and diversity collapses even though FID on the standard split barely moves. Each decision below names the component it interacts with, because the ablations in the capstone tracks only mean something when run against the full coupled system — knocking out CLIP conditioning ($+8.1$ FID in this week's ablation exercise) dwarfs the noise-schedule choice ($+0.6$), and no single-component wisdom survives contact with the joint distribution.

Data pipeline decisions#

Data pipeline decisions determine what distribution the model will learn. Deduplication rates affect memorization; image resolution affects spatial fidelity; caption quality for text-to-image models determines semantic grounding. A model cannot learn structure that is absent from or inconsistent in the training data, no matter the architecture.

Architecture decisions#

Architecture decisions determine the model's capacity and inductive biases. A diffusion transformer (DiT) operating on image patches has no inherent spatial locality bias, so it must learn spatial structure from data. A convolutional U-Net has strong locality bias built in, which accelerates learning but may limit long-range coherence. The right architecture depends on the dataset scale and the downstream use case.

Training decisions#

Training decisions determine whether the architecture's capacity is used. Noise schedule, learning rate schedule, batch size, EMA coefficient, and gradient clipping all affect the resulting model. The interaction between noise schedule and model capacity is particularly important: a model undertrained at high noise levels will produce globally incoherent samples even if reconstruction at low noise levels is sharp.

Inference decisions#

Inference decisions determine sample quality for the deployed model. Guidance scale, number of sampling steps, sampler choice (DDIM, DPM-Solver, flow ODE), and classifier-free guidance dropout rate all affect the fidelity-diversity tradeoff. The optimal inference configuration is task-dependent.

Experimental methodology#

The scientific method for generative model research requires treating experimental design as carefully as model design.

Controlled comparisons#

Controlled comparisons: changing more than one variable between compared systems prevents attribution of performance differences to any single cause. The gold standard is an ablation study: start from the full system, remove one component at a time, and measure the performance impact of each removal. A well-designed ablation answers: which components are necessary (large performance drop when removed), which are redundant (small performance drop), and which interact (removing A and B together costs more than removing A or B alone).

Reproducibility requirements#

Reproducibility requirements: fixed random seeds for all stochastic evaluations; exact specification of the evaluation protocol (number of samples, FID implementation, guidance scale); checkpoint identification (training step, effective batch size, hyperparameters). Results without these specifications cannot be reproduced, compared across papers, or validated.

Baseline selection#

Baseline selection: a result is only meaningful relative to what was already possible. The appropriate baselines are: (1) a simple baseline that requires minimal implementation (a VAEVariational Autoencoder or a DDPM with standard settings); (2) the best published method on the same benchmark; (3) the ablated version of the proposed system that isolates the novel contribution. A method that outperforms only weak baselines provides weak evidence.

Avoiding cherry-picking#

Cherry-picking, selecting the best samples for visual presentation, is the most common source of misleading results in generative model papers. Quantitative metrics (FID, precision/recall) computed on random samples rather than curated selections provide protection against cherry-picking, but even quantitative metrics can be optimized specifically during development while hiding failures in other evaluation dimensions.

Scaling laws for generative models#

Performance improvements in generative models follow predictable power-law relationships that guide resource allocation and enable informed architectural decisions.

Neural scaling laws#

Neural scaling laws (Hoffmann et al., 2022; Chen et al., 2023 for diffusion) describe how performance metrics improve predictably with model scale, data scale, and compute. For language models, the Chinchilla scaling law provides: $L(N, D) = E + A/N^\alpha + B/D^\beta$ where $N$ is the number of parameters, $D$ is the number of training tokens, and $E$ is the irreducible loss. This law reveals that performance depends on both model and data scale in a balanced way, not dominated by either alone.

Diffusion model scaling#

Diffusion model scaling exhibits similar power-law relationships. FID scales approximately as $N^{-0.5}$ with model parameters (holding data and compute fixed), and as $D^{-0.3}$ with training data. These exponents are empirical observations from large-scale training studies that guide resource allocation: doubling parameters and doubling data have different returns depending on the current operating point. A model operating at the parameter-limited regime (trained to convergence on fixed data) benefits more from additional parameters, while a data-limited regime (parameters far from saturation) benefits more from additional training examples.

Compute-optimal frontier#

Compute-optimal frontier (Kaplan scaling law; Chinchilla refinement): for a fixed FLOPs budget $C$, the optimal parameter count $N^*$ and dataset size $D^*$ satisfy $N^* \propto C^{0.5}$ and $D^* \propto C^{0.5}$. This implies equal allocation to model and data: spending all compute on a very large model trained for few steps (overparameterized regime), or a small model trained for many steps (undertrained regime), both underperform the balanced allocation. For diffusion models, this principle is particularly important for text-to-image systems where the cost of data is training compute, not acquisition.

Practical implications#

Practical implication for generative model research: before proposing a new architecture, establish the scaling baseline. The critical question is: does the proposed method improve FID at the same parameter count as the baseline, or does it achieve the same FID with fewer parameters? Single-point comparisons at fixed parameter counts are uninformative if the scaling curves differ. Scaling curves are more informative than single-point comparisons and enable proper architectural attribution.

Saturation and emergence#

Saturation and capability emergence: at certain scale thresholds, qualitatively new capabilities emerge that are absent at smaller scales. Text-to-image models show emergent prompt binding, the ability to correctly associate multiple attributes with multiple objects (e.g., "a red sphere and a blue cube"), only above approximately 1B parameters. Video diffusion models show temporal coherence emergence around 10B parameters, where the model transitions from generating frame-by-frame incoherent motion to smooth, continuous trajectories. These emergence points are not predicted by smooth scaling laws and require empirical verification at scale, making early-stage scaling studies essential for long-term research planning.

Ablation design#

A minimal ablation set for a text-to-image latent diffusion system should answer:

Latent compression#

Is the latent compression necessary? Compare pixel-space diffusion at the same compute budget. This quantifies the efficiency gain of LDM compression.

Conditioning mechanism#

Is the conditioning mechanism necessary? Compare CLIP text conditioning vs. no conditioning vs. a simple class label. This quantifies the semantic grounding provided by the conditioning.

Classifier-free guidance#

Is CFG necessary at the chosen guidance scale? Compare $s = 1$ (no guidance), $s = 5$ (standard), and $s = 15$ (high guidance). This quantifies the fidelity-diversity tradeoff and identifies the guidance scale where the improvement saturates.

Noise schedule#

Is the noise schedule optimal? Compare linear vs. cosine schedules. This quantifies the effect of schedule design on sample quality at different generation budgets.

Inference steps#

Where is the knee of the steps-vs-quality curve? Measure FID at 5, 10, 20, 50, and 250 sampling steps on the fixed eval split. The curve has the shape $1/\text{steps}$ plus a capacity floor — past the knee (often 20–50 steps for DDIM/DDPM, fewer for DPM-Solver) extra steps buy latency, not quality. Report the knee location, not just the best FID: the capstone deploy gate prices every step in P95 milliseconds.

Capstone project tracks#

The capstone tracks below are where the experimental discipline of this lecture gets exercised on a system of your choice — pick the track whose failure modes interest you, and hold yourself to its reporting bar (fixed seeds, confidence intervals, failure examples). A rigorous Track A on CIFAR-10 with 50k-sample FID curves teaches more than an undocumented large-scale demo, because only the former produces numbers a second team can reproduce.

Track A — Architecture and Training Ablation.

Goal: rigorously evaluate a proposed modification to the DDPM or flow matching training pipeline.

Requirements: (1) Implement a baseline DDPM or flow matching model on a standard dataset (CIFAR-10, CelebA-HQ 64×64, or LSUN bedroom). (2) Propose and implement one architectural or training change (e.g., modified noise schedule, alternative conditioning architecture, latent compression ratio). (3) Run a controlled ablation: full system vs. system without the proposed change vs. simple baseline. (4) Report FID with 95% confidence intervals at 3 sample sizes ($N = 5000, 20000, 50000$). (5) Generate qualitative failure examples that illustrate where each model breaks down.

Deliverables: training code, evaluation script with fixed seeds, FID curves with confidence bands, and 10–15 generated samples per configuration.

Track B — Conditioning and Control.

Goal: implement and evaluate a conditioning mechanism on a specialized domain.

Requirements: (1) Collect or curate a small dataset with conditioning signal (sketch-to-photo, pose-to-person, or audio-to-spectrogram; 500–2000 samples is sufficient). (2) Implement either ControlNet-style structural conditioning or classifier-free guidance on a pretrained base model. (3) Evaluate three metrics: image quality (FID), conditioning adherence (task-specific metric such as SSIM for structural correspondence or pose distance for pose-to-person), and failure rate at high guidance scales (e.g., at $s = 20$ where diversity breaks down). (4) Document the guidance scale tradeoff curve — generate samples at $s \in \{1, 3, 5, 10, 15, 20\}$ and plot adherence vs. diversity.

Deliverables: conditioning implementation, evaluation dataset with ground truth conditioning signals, guidance curve plots, and qualitative examples across the guidance spectrum.

Track C — Generative Models for Physical AI.

Goal: apply a generative model to a physical AI task (robotics, autonomous driving, or sequential decision making).

Requirements: (1) Train a diffusion policy or world model on a simulated environment (D4RL locomotion task, Isaac Gym manipulation, or a custom 2D gridworld environment). (2) Evaluate on task success rate with 95% confidence intervals over 10+ random seeds. (3) Ablate the generative model component — compare to a non-generative baseline (behavioral cloning, PPOProximal Policy Optimisation without world model) to demonstrate the value of generative modeling. (4) Analyze failure cases: categorize when the generative model fails (e.g., distribution shift, long horizon, sparse reward regime) and estimate failure rates for each category.

Deliverables: training script, evaluation protocol with fixed seeds, success rate curves with confidence intervals, ablation comparison, failure analysis with 5–10 visualized failure trajectories.

For all tracks: use fixed random seeds for all stochastic components, report all hyperparameters (learning rate, batch size, number of steps, network architecture), provide all generated samples in an appendix or supplementary materials. A well-executed Track A project on CIFAR-10 with 50k evaluation samples is more scientifically valuable than an under-documented Track C project with impressive but unvalidated results at large scale.

Multi-dimensional evaluation#

One number never clears the deploy gate in the Browser lab — here is why, axis by axis. Report FID, precision/recall, CLIP Score, a compositional benchmark, and the safety red-team rate jointly, because each is gameable in isolation: FID rewards memorization of the eval split, precision hides mode collapse behind realism, CLIP Score is bag-of-concepts blind to spatial arrangement, and none of them sees harmful content. A system that clears all five has survived five independent adversaries; a system clearing one has survived none.

For text-to-image systems:

FID on a standard benchmark (COCO-30K or MS-COCO) with 50,000 samples measures the overall fidelity and diversity of the joint distribution. Report with 95% confidence intervals from bootstrap sampling.

Precision and recall decompose FID into quality (precision) and coverage (recall). A system with high precision but low recall is a mode-collapsed model that produces realistic but narrow outputs.

CLIP Score measures prompt adherence. A system can achieve low FID and high CLIP Score by memorizing the evaluation set, so both should be verified against a training-set-disjoint evaluation split.

DrawBench / PartiPrompts evaluation uses a curated prompt set covering challenging semantic compositions (counting, spatial relationships, attribute binding) and reports human preference win rates against baselines. This captures capabilities that FID and CLIP Score miss.

Safety evaluation reports the rate of harmful content generation under a standardized red-teaming prompt set, with and without safety filters. Safety evaluation must be on-distribution for the deployment context.

Systematic failure analysis#

Every generative system fails on some inputs — the table above proves it with three concrete categories and their trigger conditions. A failure-mode section in a capstone report is therefore not an apology but the highest-information part of the document: it states the operating envelope (which prompts, which counts, which relations), gives each mode a measured rate, and says whether FID or CLIP Score would have caught it (usually not). Reviewers trust a report that prices its own failures; practitioners need the envelope to decide whether the system survives their inputs.

Attribute binding failures: "a red cube on a blue sphere" generates the right objects with wrong color assignments. Frequency on standard benchmarks.

Counting failures: "three dogs" generates two or four dogs. Rate as a function of object count.

Spatial relationship failures: "object A to the left of object B" generates the wrong spatial arrangement. Rate by relationship type.

Style-content entanglement: requesting a photorealistic image in a specific artistic style produces an image that is neither photorealistic nor artistically consistent.

For each failure mode, document: the frequency in standard evaluation, the input conditions that trigger it, and whether it would be detected by standard automatic metrics (FID, CLIP Score).

Deployment engineering: quantization, distillation, and production inference#

Training optimizes quality per FLOP; deployment optimizes quality per millisecond and per megabyte. The Browser lab's deploy gate makes the split explicit: the quality/diversity/faithfulness bars come from Weeks 1–11, while the safety and P95-latency bars come from Week 13 and this section. A model that trains beautifully and serves at 3.5s per image in eager mode fails the gate until TensorRT compilation (4.4× on SD-1.5/A10G), INT8 quantization (under 1 FID point cost), or consistency distillation (4–8 steps) moves it under the bar. Training-time and deployment-time are different optimization problems sharing one weights file.

Quantization#

Quantization reduces model weights from 32-bit floating point (float32) to lower precision representations to reduce memory footprint and increase inference throughput. INT8 quantization maps weights to 8-bit integers using per-channel or per-layer scaling: $w_\text{int8} = \text{round}(w / s)$ where $s = \max(|w|) / 127$; dequantize for computation: $w \approx w_\text{int8} \cdot s$. INT8 quantization reduces memory by 4× and increases throughput by 2–3× on modern hardware with GEMM INT8 acceleration (e.g., TensorRT on NVIDIA GPUs). Quality degradation is typically <1 FID point for diffusion models, making INT8 the practical choice for deployment where memory is constrained.

4-bit quantization#

4-bit quantization (GGUF format for LLMs, GPTQ for weights) performs aggressive quantization to 4 bits per weight, reducing memory by 8× but requiring careful per-channel or per-group scaling to limit accuracy degradation. 4-bit quantization is primarily used for large language models but is increasingly applied to diffusion transformer models where memory becomes a bottleneck (e.g., DiT-XL models on edge devices). Calibration on a representative sample of training data is essential to maintain quality.

Knowledge distillation#

Knowledge distillation for diffusion trains a small student diffusion model to match the output distribution of a large teacher using feature-matching loss: $\mathcal{L}_\text{distill} = \mathbb{E}_t[\|\phi_l^\text{student}(x_t) - \phi_l^\text{teacher}(x_t)\|^2]$ where $\phi_l$ are intermediate features at layer $l$. The student model (e.g., 100M parameters) achieves comparable FID to the teacher (1B parameters) on standard benchmarks through distillation, reducing inference cost by 10×. Distillation is especially effective when combined with consistency distillation, where the student is trained to match single-step or multi-step teacher outputs, enabling very fast generation (4–8 steps for high-quality output).

ONNX and TensorRT export#

ONNX and TensorRT export: for production deployment to diverse hardware, export the trained model to ONNX (Open Neural Network Exchange) for hardware-agnostic representation. Compile with TensorRT for GPU inference acceleration, which applies graph fusion, operator optimization, and precision calibration automatically. A production text-to-image system (Stable Diffusion 1.5 on an A10G GPU) generates 512×512 images in 0.8s with DDIM-25 sampling steps after TensorRT compilation, compared to 3.5s with PyTorch eager mode — a 4.4× speedup from compilation alone.

Dynamic batching and streaming#

Dynamic batching and streaming: production serving systems (NVIDIA Triton, vLLM, Replicate, BentoML) batch multiple user requests to maximize GPU utilization and reduce per-request latency. For diffusion models, batching across requests is straightforward since each inference is independent and stateless. Monitoring P95 latency and error rates under realistic load is essential before public deployment. A typical target is P95 latency < 5s for a 512×512 image generation on a single GPU, achievable with batch size 4–8 and DDIM-25 steps.

GenAI context#

This course on generative modeling for physical AI is part of a broader four-course curriculum. The table below maps Generative Models topics to foundational concepts from Reinforcement Learning and Robot Learning, and to applications in Physical AI (vision–language models).

Synthesis#

Synthesis: Every generative model is a world model. The generative modeling curriculum reveals a fundamental insight for physical AI: the distinction between "generative models" (this course) and "robot learning" (the Robot Learning course) is architectural, not conceptual. A VAEVariational Autoencoder that compresses robot observations to latent states is a perceptual world model. A diffusion policy that generates action sequences conditioned on observations is a generative inverse dynamics model. A flow matching model trained on robot trajectories is both a motion generator and a trajectory prior for planning. The same mathematical frameworks (ELBO, score matching, optimal transport) operate seamlessly at the intersection of both domains. This unification is what makes foundation models for physical AI (the curriculum's central theme) a coherent research program rather than an assembly of disconnected methods. The next frontier is scaling these unified models to the complexity of real-world physical interaction while maintaining safety, interpretability, and human alignment.

Course conclusion#

The generative modeling curriculum has traced a complete arc from the first principles of variational inference through the full modern stack: VAEs, GANs, energy-based models, normalizing flows, DDPM, flow matching, conditioning, latent diffusion, evaluation, representation learning, world models, safety, and alignment.

Every topic connects to the themes established at the start. The ELBO from Week 1 reappears in the DDPM ELBO, the RSSM training objective, and the VLMVision-Language Model alignment objectives. The score function from Week 4 is the denoising target of DDPM and the vector field of flow matching. The KL divergence that regularizes the VAEVariational Autoencoder latent space is the same KL that constrains policy updates in RLHFReinforcement Learning from Human Feedback. Flow matching's optimal transport objective is the Schrödinger bridge of Week 5 in the limit of no diffusion. The representation learning of Week 11 is the foundation for the VLA models of Robot Learning.

The problems below are open in the specific sense that no method in Weeks 1–13 closes them: attribute binding still fails on "red cube on a blue sphere" at rates the compositional benchmarks measure; certified safety radii cover $\ell_2$ balls, not semantic jailbreaks; low-data regimes still defeat likelihood-based training; and KL-constrained alignment still degrades under distribution shift. Each is a research program with a metric attached — GenEval accuracy, bypass rate, sample efficiency curves, shifted-distribution preference win rates — which is what makes them thesis topics rather than wishes.

Browser lab: capstone tradeoffs — scaling, steps, and multi-metric gates#

Capstone numerics: (1) power-law quality vs compute, (2) DDIM-style steps vs error, (3) a multi-metric deploy gate (quality / diversity / safety / latency).

import numpy as np
import matplotlib.pyplot as plt

# 1) Scaling law toy: loss = a * C^(-b) + c_inf
C = np.array([1e17, 3e17, 1e18, 3e18, 1e19])  # FLOPs
a, b, c_inf = 2.5, 0.05, 1.8
loss = a * C ** (-b) + c_inf
print("Compute-optimal style curve (toy):")
print(f"{'FLOPs':>10}  {'loss':>8}")
for c, L in zip(C, loss):
    print(f"{c:10.1e}  {L:8.3f}")

# 2) Inference steps: error ~ 1/steps for smooth ODE (proxy)
print("\nInference steps vs integration error proxy:")
print(f"{'steps':>6}  {'err':>8}  {'latency ms':>12}")
for steps in [1, 4, 10, 25, 50, 100]:
    err = 0.8 / steps + 0.02  # floor from model capacity
    lat = 8.0 * steps  # ms, single GPU toy
    print(f"{steps:6d}  {err:8.3f}  {lat:12.1f}")

# 3) Multi-metric gate for ship / no-ship
metrics = {
    "FID↓": 12.0,       # lower better — store as score later
    "Recall↑": 0.55,
    "CLIP-score↑": 0.28,
    "Safety pass↑": 0.97,
    "P95 lat ms↓": 4200,
}
# Normalize to [0,1] higher-better
def norm_metrics(m):
    return {
        "quality": max(0, 1 - m["FID↓"] / 40.0),
        "diversity": m["Recall↑"],
        "faithfulness": min(1.0, m["CLIP-score↑"] / 0.35),
        "safety": m["Safety pass↑"],
        "latency": max(0, 1 - m["P95 lat ms↓"] / 8000.0),
    }

scores = norm_metrics(metrics)
bars = 0.7  # deploy bar
print("\nDeploy radar (higher better; bar=0.7):")
fail = []
for k, v in scores.items():
    mark = "OK" if v >= bars else "FAIL"
    if v < bars:
        fail.append(k)
    print(f"  {k:14s} {v:5.2f}  {mark}")
print("SHIP" if not fail else f"NO-SHIP — fix: {', '.join(fail)}")

fig, ax = plt.subplots(1, 2, figsize=(8, 3))
ax[0].loglog(C, loss, "o-", color="#2f55f0")
ax[0].set_xlabel("FLOPs"); ax[0].set_ylabel("loss")
ax[0].set_title("Scaling (toy)")
names, vals = list(scores.keys()), list(scores.values())
colors = ["#2f55f0" if v >= bars else "#f5a623" for v in vals]
ax[1].barh(names, vals, color=colors)
ax[1].axvline(bars, color="gray", ls="--", lw=1)
ax[1].set_xlim(0, 1)
ax[1].set_title("Deploy metrics")
plt.tight_layout()
plt.show()
print("Notice: more compute helps until c_inf; fewer steps buy latency; one FID number never clears a multi-gate.")

What to try: drop FID↓ to 8 and P95 lat ms↓ to 2000 — does the gate flip to SHIP? Raise the bar to 0.85.

Key takeaways#

• A generative system is a coupled pipeline, not a list of independent choices. Data, architecture, training, and inference decisions interact, and a change in one propagates to the others in ways that are often nonlinear.
• Scientific rigor is experimental design: controlled comparisons, pre-registered ablations, and explicit baselines. A compelling demo is not evidence of capability, and a single FID number is not evidence of general quality.
• An ablation answers which components are necessary (large drop when removed), which are redundant (small drop), and which interact (removing both costs more than removing either alone).
• Scaling laws are the baseline a new architecture must beat. Compare scaling curves, not single points, and ask whether the gain is better FID at equal parameters or equal FID with fewer.
• Compute-optimal allocation splits a fixed budget roughly equally between parameters and data ($N^* \propto C^{0.5}$, $D^* \propto C^{0.5}$); the parameter-limited and data-limited regimes call for different spending.
• Evaluate on multiple axes and report the failure-mode analysis, not one leaderboard number — cherry-picked samples and single metrics are the most common source of misleading generative results.
• Deployment engineering (quantization, distillation, dynamic batching) and reproducibility (seeds, evaluation protocol, checkpoint identity) are part of the result, not an afterthought.

Knowledge Check#

Check generative systems engineering and the capstone lens.

Conceptual questions#

1. An ablation study for a latent diffusion model shows: full system → FID 6.2; no latent compression (pixel-space) → FID 8.5; no CFG (guidance $s = 1$) → FID 7.1; no CLIP conditioning → FID 14.3; linear noise schedule (vs. cosine) → FID 6.8. Interpret these results: rank the components by importance, identify which result is most surprising relative to prior expectations, and propose two follow-up experiments that would deepen understanding of the most important component.

2. A paper reports FID = 4.1 on MS-COCO with 30,000 samples and no confidence interval. A follow-up evaluation with 50,000 samples measures FID = 5.3 for the same model. Explain the statistical mechanism that causes FID to decrease with fewer samples. Derive how the bias scales with sample size, and propose a minimum sample size for reporting FID that limits the bias to less than 0.5 FID points.

3. A generative model achieves CLIP Score = 32 and FID = 8 on standard evaluations but shows a 40% failure rate on spatial relationship prompts ("A is to the left of B"). The CLIP Score for spatial relationship prompts is 27 (lower than average). Design a comprehensive evaluation protocol that would detect this spatial failure mode using automatic metrics, and explain why standard CLIP Score evaluation fails to flag it.

4. Trace the role of the ELBO through the generative modeling curriculum: show how the VAEVariational Autoencoder ELBO (Week 2), the DDPM ELBO (Week 6), and the RSSM ELBO (Week 12) are all instances of the same general bound $\log p_\theta(x) \geq \mathbb{E}_{q_\phi}[\log p_\theta(x \mid z)] - D_\text{KL}(q_\phi(z \mid x) \| p(z))$. For each model, identify what plays the role of the observation $x$, the latent $z$, the prior $p(z)$, and the approximate posterior $q_\phi(z \mid x)$.

5. Synthesize the full generative modeling curriculum by tracing the design decisions for a language-conditioned video generation system that is safe for public deployment. For each of the following components, identify: (a) the technical choice required, (b) the evaluation metric for that component, and (c) the failure mode if the component is absent: (i) video compression (VAEVariational Autoencoder), (ii) generation model (diffusion vs. flow matching), (iii) text conditioning mechanism, (iv) safety filter, (v) content watermarking.

Further reading#

• Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv. (Empirical power laws that organize capacity–data–compute tradeoffs).
• Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla). NeurIPS. (Compute-optimal allocation of parameters vs. tokens).
• Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS. (Canonical DDPM formulation for the generation backbone).
• Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR. (Latent diffusion / Stable Diffusion system design).
• Heusel, M., et al. (2017). GANs Trained by a Two Time-Scale Update Rule… (FID). NeurIPS. (Standard image-quality metric and its evaluation protocol).