Week 12: Vision-Language Models for Robotics

Purpose of this lecture#

A VLM becomes an embodied system the moment its outputs move mass: tokens become torque, and every failure mode from Weeks 1–11 — hallucinated objects, mis-grounded referents, latency spikes — becomes a physical event with a workspace, a force limit, and a clock. Robotics is therefore not one more application but the strictest test in the course: unstructured environments (no curated image distribution), abstract instructions (compositional language, not class labels), and continuous control under hard real-time deadlines, partial observability, and safety constraints that cannot be patched after the fact.

This lecture bridges the multimodal architectures developed in this course with the physical control and safety theories established in Robot Learning. We examine how VLMs are integrated into hierarchical control pipelines, the mathematics of discrete action tokenization (RT-2), the "System 1 vs. System 2" architecture (GR00T), and the integration of 3D spatial representations (NeRFs/3DGS) into the VLMVision-Language Model control loop.

Hierarchical robot control architecture#

Modern robot control operates simultaneously across vastly different timescales. Attempting to run a 70B parameter VLMVision-Language Model at the 1000Hz frequency required to calculate joint torques is both mathematically unnecessary and computationally impossible. Instead, embodied VLMs are integrated into a hierarchical control stack:

1. Semantic Task Level (System 2): Operating at roughly 0.5 to 2 Hz, a massive VLMVision-Language Model (e.g., LLaVA, PaLM-E) sits in the cloud or on a heavy local server. It receives a natural language instruction ("Make a cup of tea") and a streaming visual observation (RGB video or 3D point cloud). It performs deep reasoning and outputs a high-level sequence of semantic sub-goals ("locate mug", "grasp handle", "navigate to kettle").

2. Kinematic Skill Level (System 1): Operating at 10 to 50 Hz directly on the robot's edge-compute board (e.g., Nvidia Jetson). This layer receives the semantic sub-goal from the VLMVision-Language Model and executes a parameterized, continuous control policy. As explored in Course 2, this is frequently a Diffusion Policy or an Action Chunking Transformer (ACTAction Chunking with Transformers) that maps the VLMVision-Language Model's instruction to specific end-effector velocities $(v_x, v_y, v_z, \omega_x, \omega_y, \omega_z)$.

3. Reactive Control Level: Operating at 1000+ Hz. Low-level operational-space controllers (like PD controllers) track the desired velocities from the skill layer, reject physical disturbances (e.g., the mug being heavier than expected), and enforce strict joint safety limits.

This hierarchy decouples deep language understanding (System 2) from real-time physical stabilization (System 1).

Language-conditioned spatial grounding#

To pass instructions down the hierarchy, the VLMVision-Language Model must translate abstract nouns into explicit physical coordinates.

2D Grounding to 3D Targets: If the VLMVision-Language Model commands "grasp the red screwdriver," it must first identify it. Using grounding architectures (Week 4), the VLMVision-Language Model predicts a 2D bounding box. If the robot is equipped with an RGB-D camera, the 3D position of the target is mathematically estimated by projecting the 2D center $(u, v)$ through the depth map $d(u,v)$ using the camera's intrinsic matrix $K$:

This $(X, Y, Z)$ coordinate is transformed into the robot's base frame using standard extrinsic calibration matrices, providing the exact spatial target for the System 1 reaching policy.

Integrating 3DGS and NeRFs: Relying purely on 2D cameras creates occlusion problems (the robot's arm blocks the view of the screwdriver). Advanced systems increasingly query 3D Gaussian Splatting (3DGS) or Neural Radiance Fields (NeRFs) instead of flat 2D images. If the scene is represented as a Language-Embedded Radiance Field (LERF), the VLMVision-Language Model can query the 3D coordinate space directly using the CLIP text embedding of "red screwdriver," bypassing the 2D-to-3D projection step entirely and inherently solving occlusion via multi-view geometry.

RT-2: the VLMVision-Language Model as an end-to-end policy#

RT-2 (Robotics Transformer 2; Brohan et al., 2023) challenges the hierarchical approach by asking: What if we fine-tune the entire massive VLMVision-Language Model to directly output robot actions?

RT-2 takes a frozen, pre-trained VLMVision-Language Model (like PaLI-X) and unfreezes it for co-training on robotic trajectory data. The continuous physical actions (e.g., end-effector position and rotation commands) are mapped directly into the VLMVision-Language Model's discrete text vocabulary.

The mathematics of discrete action tokenization#

RT-2 must predict continuous variables $a \in \mathbb{R}^n$. To do this autoregressively alongside text, it discretizes the continuous space into $B$ uniform bins (typically $B=256$).

For an action dimension $x$ strictly bounded between $[x_\text{min}, x_\text{max}]$, the discrete token index $k \in \{0, 1, \dots, 255\}$ is computed as:

The action sequence for a 6-DOF arm plus gripper is represented as 7 discrete tokens appended to the text generation sequence. The VLMVision-Language Model is trained using standard cross-entropy loss to predict these action tokens identically to how it predicts language.

Quantization Error (position): If the arm spans a 2.0 meter workspace, quantizing the position into 256 bins results in a spatial resolution error of $\frac{200 \text{ cm}}{256} \approx 0.78 \text{ cm}$. This is a position figure, not a velocity bin — the action-binning lab below works in velocity units (m/s) and must be read on its own scale. While 0.78 cm is acceptable for gross reaching movements, this mathematical quantization error makes delicate, high-precision tasks (like threading a needle) impossible for a pure RT-2 model without a continuous low-level correction controller.

GR00T: formalizing System 1 and System 2#

Nvidia's Project GR00T and models like $\pi_0$ (Physical Intelligence) represent the modern synthesis of these approaches, explicitly formalizing the handoff between the VLMVision-Language Model and the continuous controller.

In these architectures, the VLMVision-Language Model (System 2) processes the complex multimodal prompt and outputs a dense Goal Embedding Vector $e_\text{goal}$, rather than discrete text tokens or quantized actions.

This embedding is passed down to a high-frequency Diffusion Policy or Flow Matching Policy (System 1, from Course 2). The continuous control policy $\pi(a \mid s, e_\text{goal})$ treats the VLMVision-Language Model's output as a strict geometric conditioning signal.

Because System 1 predicts continuous actions in $\mathbb{R}^n$ rather than quantized tokens, it completely bypasses the quantization error of RT-2. Furthermore, because System 1 only needs to evaluate a lightweight diffusion UNet (and not a 70B parameter LLMLarge Language Model), it easily achieves the 50Hz latency requirement for dynamic physical tasks like catching moving objects.

VLMVision-Language Model safety layers and control barrier functions#

A VLMVision-Language Model outputting actions in a chatbot window is safe; a VLMVision-Language Model outputting actions that swing a 20kg titanium robot arm is dangerous. Because VLMs are known to hallucinate, they cannot be trusted with raw physical access. They must be wrapped in mathematical safety layers.

Semantic Safety Filters: The VLMVision-Language Model's proposed plan is checked against a safety policy. If the VLMVision-Language Model suggests "move the knife over the human's hand," an auxiliary model flags the semantic violation.

Control Barrier Functions (CBFs): As covered extensively in Course 2 (Week 12), the VLMVision-Language Model's output must pass through a strict mathematical filter. Let the safe physical workspace of the robot be defined as a set $\mathcal{C} = \{x \in \mathbb{R}^n \mid h(x) \geq 0\}$.

If the VLMVision-Language Model commands a velocity $u_\text{VLM}$ that would drive the robot arm into a table (violating $h(x) \geq 0$), the CBF intercepts the command and solves a rapid Quadratic Program (QP) to project the VLMVision-Language Model's command to the nearest safe velocity $u^*$ that satisfies the barrier derivative:

This ensures that no matter how severely the VLMVision-Language Model hallucinates, the physical hardware mathematically cannot violate its collision constraints.

Browser lab: RT-2 action binning vs continuous System 1#

Discretize a 1-D end-effector command into $B$ bins (RT-2 style) and measure quantization error. Compare to a continuous command (System 1 diffusion / flow). Also sketch a CBF clamp that projects unsafe velocity.

import numpy as np

x_min, x_max = -1.0, 1.0  # m/s
B = 256
workspace_m = x_max - x_min

def quantize(x, B=B, lo=x_min, hi=x_max):
    k = np.round((x - lo) / (hi - lo) * (B - 1)).astype(int)
    k = np.clip(k, 0, B - 1)
    x_hat = lo + k / (B - 1) * (hi - lo)
    return k, x_hat

# Uniform targets across the range
xs = np.linspace(x_min, x_max, 501)
ks, x_hat = quantize(xs)
err = np.abs(xs - x_hat)
bin_width = workspace_m / (B - 1)
print(f"B={B} bins over [{x_min}, {x_max}] m/s → velocity step ≈ {bin_width*1000:.2f} mm/s")
print(f"Max quantization |error|: {err.max()*1000:.3f} mm/s")
print(f"Mean quantization |error|: {err.mean()*1000:.3f} mm/s")

# Order-of-magnitude motivation: 1 mm clearance is a *position* tolerance, not a
# velocity bin — the comparison is scale, not a like-for-like conversion.
need_mm = 1.0
print(f"\nOrder-of-magnitude: {need_mm} mm position clearance vs {bin_width*1000:.2f} mm/s velocity step → "
      f"{'a discretized policy must fall back on closed-loop feedback/compliance' if bin_width*1000 > need_mm else 'OK'}")

# Continuous System 1: same targets, no binning
print(f"Continuous System 1 max error: 0.000 mm/s (by construction)")

# CBF-style projection: keep state x away from wall at x_wall with velocity u
x_wall = 0.8
gamma = 2.0
h = lambda x: x_wall - x  # safe if h >= 0 (left of wall)

def cbf_filter(x, u_vlm, gamma=gamma):
    # Require ḣ = -u >= -γ h  ⇒  u <= γ h  (for 1D, moving right is +u)
    u_max = gamma * h(x)
    u_star = min(u_vlm, u_max)
    return u_star

x = 0.7
for u in [0.05, 0.5, 2.0]:
    u_safe = cbf_filter(x, u)
    print(f"x={x}, u_VLM={u:.2f} → u*={u_safe:.3f} (h={h(x):.2f})")

print("Notice: 255 intervals ≈ 7.8 mm/s per velocity step (worst-case error ≈ 3.9 mm/s) on a 2 m/s span — fine for reach, far too coarse for the sub-mm/s modulation an insert needs, so a discretized policy depends on closed-loop feedback/compliance; note the 1.2 mm clearance is a position tolerance, not a velocity bin (order-of-magnitude motivation only); CBF clamps unsafe u.")

What to try: set B = 64 or B = 1024, or move x closer to x_wall and command large u_vlm.

Key takeaways#

Embodied AI requires integrating VLMs into hierarchical control stacks to balance deep semantic reasoning with high-frequency physical stabilization. VLMs bridge language to physics by predicting explicit 3D spatial coordinates, increasingly using NeRFs and 3DGS to bypass 2D occlusion. Architectures like RT-2 attempt to run end-to-end by tokenizing continuous actions into the LLMLarge Language Model vocabulary, suffering from unavoidable quantization error. Modern architectures (like GR00T/$\pi_0$) resolve this by using the VLMVision-Language Model as a "System 2" semantic planner that outputs continuous embeddings to condition a "System 1" high-frequency Flow Matching or Diffusion policy. Finally, because VLMs are prone to hallucination, their outputs must be mathematically filtered through hard constraints like Control Barrier Functions (CBFs) to guarantee physical safety.

Conceptual questions#

Question 1: RT-2 quantization#

1. RT-2 Quantization Mathematics: An RT-2 style model is controlling a robot arm. The Z-axis velocity is bounded between $[-1.0, 1.0]$ m/s and is quantized into $B=256$ uniform bins. The VLMVision-Language Model predicts bin $k=128$, which corresponds to $0.0$ m/s. However, due to the softmax distribution over the tokens, the VLMVision-Language Model is slightly uncertain, and the actual physical target requires a velocity of $0.005$ m/s. Calculate the exact mathematical quantization error range. Explain why this specific error prevents an RT-2 model from successfully inserting a 1mm peg into a 1.2mm hole without relying on passive mechanical compliance.

Question 2: System 1 vs System 2#

2. System 1 vs System 2 Handoff: In the GR00T architecture, the VLMVision-Language Model (System 2) outputs a goal embedding $e_\text{goal}$ which conditions a Diffusion Policy (System 1). Assume the VLMVision-Language Model operates at 1 Hz and the Diffusion Policy operates at 50 Hz. Describe the exact temporal flow of data during a 2-second maneuver where a human suddenly pushes the target object 10 cm to the right at $t=0.5$ seconds. How does the Diffusion Policy react instantly (before the VLMVision-Language Model updates $e_\text{goal}$ at $t=1.0$), and what specific physical state variables $s_t$ must the Diffusion Policy have access to for this to work?

Question 3: Sim2Real and NeRF#

3. Sim2Real and NeRF Integration: A robotic manipulation policy trained purely in IsaacSim struggles to grasp objects in the real world because the real-world camera feed contains complex lighting, shadows, and occlusions not present in the simulator. You integrate a real-time 3D Gaussian Splatting (3DGS) pipeline. How does explicitly passing a 3D semantic point cloud (rather than a 2D RGB image) into the VLMVision-Language Model reduce the visual Covariate Shift (Week 9) between the simulated training data and the real-world deployment?

Question 4: CBF safety#

4. CBF Safety Interception: A VLMVision-Language Model commands a robot arm to move linearly across a table to grab an apple. The VLMVision-Language Model hallucinates the depth of the apple and outputs a velocity vector $u_\text{VLM}$ that points directly through a glass vase. A Control Barrier Function (CBF) is monitoring the system. Mathematically describe how the CBF's Quadratic Program modifies $u_\text{VLM}$. Does the robot stop completely, or does it dynamically curve around the vase? What is the tradeoff between setting a very high $\gamma$ value (aggressive approach) versus a low $\gamma$ value (conservative approach) in the CBF constraint $\dot{h}(x, u) \geq -\gamma h(x)$?

Question 5: Language relabeling#

5. Language Relabeling Bias: To acquire more robotics data, a research team uses an off-the-shelf VLMVision-Language Model to watch 10,000 hours of teleoperated robot video and automatically generate text instructions (Language Relabeling). The VLMVision-Language Model has a known spurious correlation (from Week 9): it labels any video containing a sponge as "cleaning the table," even if the robot is just moving the sponge. Analyze the downstream effect on the RT-2 policy trained on this relabeled data. If a human operator later commands the robot to "Move the sponge to the bin," what unintended physical action is the robot mathematically highly likely to execute, and why?

Knowledge Check#

Check VLMs for robotics.

Looking ahead#

Hierarchical control (System 2 planner → System 1 policy), language grounding to manipulation targets, and closed-loop VLM correction are now in place — the robot acts on language in the physical world. What none of those mechanisms addresses is whose world the training data describes: LAION-scale corpora over-represent some demographics, scenes, and cultural contexts and under-represent others, and the policy inherits the skew silently. Week 13 makes the skew measurable — representation auditing, stereotype-amplification metrics, multimodal RLHF/RLAIF — before the deployment in Week 14 has to answer for it.

Week 13: Bias, Fairness, and Safety in VLMs. We examine representation bias in web-scale datasets (like LAION), stereotype amplification in generative models, the mathematical mechanisms of RLHFReinforcement Learning from Human Feedback and RLAIF alignment for multimodal systems, and the content moderation pipelines necessary for responsible real-world deployment.

Further reading#

• Brohan, A., et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. CoRL. (End-to-end action tokenization).
• Driess, D., et al. (2023). PaLM-E: An Embodied Multimodal Language Model. ICML. (Continuous state injection).
• Kerr, J., et al. (2023). LERF: Language Embedded Radiance Fields. ICCV. (Grounding VLMs in 3D NeRF space).
• Physical Intelligence. (2024). $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control. (System 1 / System 2 formalization with Flow Matching).