Week 14: Sim2Real Capstone

Purpose of this lecture

The preceding thirteen weeks assembled a complete robot learning curriculum: perception and kinematics, classical control, teleoperation and data collection, imitation learning, RLReinforcement Learning for robotics, simulation infrastructure, foundation model architectures, flow matching and diffusion policies, VLA models, fine-tuning, safety, and multi-task learning. Each lecture addressed a specific component of the robot learning pipeline.

This final lecture puts all components together. Rather than adding new algorithms, it asks: what does it mean to build and evaluate a complete system? The emphasis is on the engineering discipline and scientific rigor that separate research-grade systems from demonstrations — reproducibility, ablation methodology, failure analysis, and the honest accounting of where systems succeed and fail. These skills are what distinguish practitioners who can build and evaluate real robotic learning systems from those who can only reproduce benchmark results.

The sim2real pipeline as a system

A sim2real pipeline is not a sequence of independent steps. It is a system where decisions made at each stage propagate downstream and interact with decisions at other stages in ways that are often non-obvious at design time. Understanding the pipeline as a system — with feedback loops, emergent interactions, and compounding failure modes — is the prerequisite for building systems that reliably transfer from simulation to hardware.

The pipeline stages are:

Task Specification

Task specification defines what the robot must do, what constitutes success, what constraints must be respected, and how both will be measured. Ambiguous task specifications produce ambiguous results: a policy that "performs well" on a vaguely specified task may perform very differently from what was intended when evaluated rigorously.

Simulation Environment

Simulation environment construction instantiates the physical model — robot URDF or USD assets with accurate mass, inertia, and actuator models; object assets with physics properties calibrated via system identification; sensor models with realistic noise and field of view. The fidelity of this instantiation is the primary determinant of how large the sim2real gap will be.

Policy Training

Policy training applies the learning algorithm of choice — SACSoft Actor-Critic, PPOProximal Policy Optimisation, behavior cloning, ACTAction Chunking with Transformers, diffusion, or a VLA — with the action space, observation design, reward function, and curriculum schedule specified for the task. The training decisions interact with the simulation environment: a policy trained with zero contact friction will behave differently than one trained with friction even if the reward function is identical.

Domain Randomization

Domain randomization is applied during training to produce a policy robust to parameter variation. The randomization ranges, which parameters are randomized, and when during training the ranges are widened all affect the final policy's robustness profile.

Evaluation in Simulation

Evaluation in simulation establishes a quantitative baseline before hardware contact. Evaluation must test not only nominal performance but robustness to the specific parameters that will be different on hardware — actuator delays, contact friction, sensor noise levels — to identify the likely failure modes before physical deployment.

Hardware Deployment

Hardware deployment (or high-fidelity simulation stress testing) applies the trained policy to conditions the simulator did not exactly represent. This stage is informative regardless of outcome: success validates the pipeline design; failure reveals specific gaps in the simulation model or domain randomization coverage.

Failure Analysis

Failure analysis closes the loop. Every hardware failure, anomalous behavior, or unexpected result is data about the gap between simulation and reality. Systematic analysis of failures drives targeted improvements to simulation fidelity, domain randomization coverage, or policy architecture.

Task specification: rigor as a prerequisite

A well-specified task has four components: a success criterion that is measurable, unambiguous, and equivalent to the real-world objective; a failure criterion that identifies when an episode has failed irrecoverably; initial condition distributions that define the range of starting configurations the policy will be tested from; and safety constraints that bound the space of allowable actions during evaluation.

The success criterion is particularly important to specify carefully. "The object reaches the target" is ambiguous: within what position tolerance? With what orientation? Must the robot be stationary at success? Does the object have to remain at the target, or just pass through it? Each ambiguity in the success criterion corresponds to a potential discrepancy between simulation evaluation and hardware evaluation, which makes it impossible to predict hardware performance from simulation results.

Initial condition distributions are the primary lever for controlling evaluation difficulty. Testing only from a single fixed initial condition produces an artificially narrow assessment: a policy that happens to have memorized the nominal trajectory will appear to succeed while generalizing poorly to any variation. Testing from a distribution with significant variance in initial object placement, lighting, and robot configuration produces an assessment that is much more predictive of deployment performance.

Simulation environment: the fidelity-efficiency tradeoff

Building a high-fidelity simulation environment that accurately represents the physical system is the most labor-intensive part of the sim2real pipeline. The key accuracy targets are:

Actuator models: the relationship between commanded control input (position, velocity, or torque setpoint) and actual joint motion must be modeled accurately. Physical actuators exhibit position-dependent friction, current limits, thermal derating, and communication delays. Omitting these effects produces a policy that assumes perfect torque control authority — authority that the physical actuator cannot deliver in contact-rich operation. The standard approach is to identify actuator parameters from system identification experiments (applying known commands and measuring the resulting motion) and implement the identified model in the simulator.

Contact geometry: manipulation tasks require accurate contact modeling. Inaccurate collision geometry (convex hull approximations that differ from the true object shape) produces grasps that succeed in simulation on virtual contacts that don't exist on the physical object. Using actual mesh geometry (from 3D scanning or precise CAD) for collision, and calibrating contact material parameters (friction, stiffness, damping) from physical experiments on representative surfaces, reduces this source of gap.

Sensor models: camera images, depth maps, and force-torque readings in the simulation must resemble what the physical sensors produce. Visual domain randomization is the primary tool for camera images; force-torque sensor models must capture the sensor's noise characteristics, bandwidth, and mounting compliance.

The fidelity-efficiency tradeoff is real: higher-fidelity models require more computational resources to simulate, reducing the parallelism that enables rapid data collection. The practical approach is to use simplified models during early-stage training (to enable high parallelism and fast iteration) and switch to higher-fidelity models for final training and evaluation.

Policy training decisions and their consequences

The choice of learning algorithm, action space, reward function, and curriculum are major determinants of final performance and should be treated as design decisions that require justification, not as default settings to be filled in.

Action space: as analyzed in Week 6, Cartesian end-effector delta commands are generally the best starting point for manipulation tasks. The effective choice depends on the task's force control requirements: tasks requiring precise contact forces (assembly, deformable object manipulation) may need direct torque control or admittance control interfaces, whereas tasks requiring precise positioning (pick-and-place) are well-served by position control with Cartesian deltas.

Reward function: the shaped reward function must be designed to avoid the Goodhart's Law failure modes discussed in Week 6. Every shaped term should be traceable to a mechanism by which it accelerates learning without introducing incentives for unintended behaviors. Ablating the reward function — running the policy with each shaped term removed in turn — is the standard way to verify that the shaping is contributing to task success rather than producing reward-hacking behaviors that bypass it.

Curriculum: the curriculum structure should be validated empirically by comparing curriculum training against training from the final curriculum stage directly. If the curriculum provides significant benefit, the performance gap between curricula will be large; if it does not, the experiment without curriculum provides a simpler baseline.

Domain randomization: scope and schedule

Domain randomization is the primary mechanism for bridging the sim2real gap, and its design requires explicit justification of which parameters are randomized, over what ranges, and with what schedule.

Parameter selection should be motivated by uncertainty: parameters that are known precisely (robot joint masses obtained from manufacturer specifications) require narrow randomization, while parameters that are unknown or variable (table surface friction, object mass variation between production units) require wide randomization. Applying wide randomization to well-known parameters wastes training capacity; applying narrow randomization to uncertain parameters fails to cover the real-world distribution.

Schedule design should follow the analysis from Week 7: narrow randomization early in training (to enable rapid acquisition of task behavior) followed by progressive widening (to build robustness). The transition timing can be automated using adaptive domain randomization — widening the ranges when performance is high and narrowing when it drops.

Identifying missed parameters requires physical evaluation: deploy the policy on hardware (or high-fidelity simulation), observe the failure mode, and reverse-engineer which simulation parameter, if it had been randomized, would have covered the physical condition that caused the failure. Common missed parameters include: communication delay jitter (not just mean delay), object surface adhesion under load, and background visual clutter.

Evaluation methodology: rigor is not optional

Evaluation design is as important as training design. A rigorous evaluation answers: what is the policy's success rate, over what distribution of initial conditions, and how does this compare to relevant baselines?

| Reproducibility | Ablation Studies | Coverage | | --- | --- | --- | | Fixed seeds, standardized initial distributions, and explicit success criteria are mandatory for valid performance claims. | Systematically removing components (e.g., randomization, filters) to isolate their actual contribution to policy success. | Evaluating across the full operational range, not just nominal cases, to characterize failure modes and boundary conditions. |

Reproducibility Requirements

Reproducibility requires fixed random seeds for the evaluation environment, a fixed evaluation protocol (number of episodes, initial condition distribution, success criterion), and exact checkpoint identification (training step, hyperparameters, domain randomization settings). Without these, results cannot be reproduced, compared, or validated by others.

Ablation Study Design

Ablation studies isolate the contribution of each component of the system. A minimal set of ablations for a sim2real pipeline should include: no domain randomization, no curriculum, no safety filter, and a simpler action space. Each ablation reveals which components are contributing to performance and which are irrelevant overhead. The result of a well-designed ablation is not a claim that the full system is best — it is an understanding of why the full system performs as it does.

Statistical Validity

Statistical validity of success rate comparisons requires explicit confidence intervals. A policy that succeeds on 40 of 50 trials has a measured success rate of 80%, but the true rate could plausibly be anywhere from 66% to 90% (95% Wilson score confidence interval). Reporting only the point estimate is misleading. For a single policy with $k$ successes in $n$ trials, the Wilson score interval with $z_\alpha = 1.96$ is:

For comparing policy A ($k_A$ of $n_A$) to policy B ($k_B$ of $n_B$), Fisher's exact test computes the exact p-value for the $2 \times 2$ contingency table $\begin{pmatrix} k_A & n_A - k_A \\ k_B & n_B - k_B \end{pmatrix}$ under the null hypothesis that both policies have equal success rates. With $n = 50$ trials per condition, a difference of 15–20 percentage points is detectable at $\alpha = 0.05$; detecting a 5-point difference requires approximately $n = 400$ trials. These sample size requirements should inform how many hardware trials are planned — arbitrary choices like "we ran 10 trials" are typically insufficient for reliable comparison.

Failure Mode Analysis

Failure mode analysis is a positive contribution. For every evaluation condition where the policy fails, documenting the failure mode and its frequency provides actionable information for system improvement. A structured failure taxonomy prevents failures from being lumped together under vague labels. A useful first-level taxonomy for manipulation systems separates:

• Perception failures: object mislocalization $>$ 1 cm, wrong object selected, failure to detect contact
• Planning/policy failures: suboptimal mode selection, incorrect task sequencing, reward-hacking behavior
• Execution failures: joint limit violation, trajectory tracking error $>$ 5 mm, grasp slip during transport
• Safety-induced failures: CBF-QP modification that blocks the intended motion, fallback controller engagement that prevents task completion
• Domain shift failures: success rate drop attributable to a specific parameter mismatch (lighting change, surface friction change, object mass change)

Each category points to a different fix: perception failures to data collection or visual backbone fine-tuning; planning failures to reward redesign or policy architecture; execution failures to action space, controller gains, or sim2real; safety-induced failures to CBF parameter tuning; domain shift failures to domain randomization coverage. Concealing failures is the cardinal sin of evaluation design.

Hardware deployment: the moment of truth

Hardware deployment is both the culmination of the pipeline and its most informative phase. The first hardware deployment should be treated as a data collection experiment, not as a demonstration: the goal is to observe what fails, not to achieve a high success rate.

| Safety First | Logging & Video | Iterative Testing | | --- | --- | --- | | Conservative speed limits, active safety filters, and immediate fallback mechanisms are mandatory for the first deployment. | Synchronized recording of all sensor streams, policy actions, and external video for post-hoc failure analysis. | Starting from nominal cases and systematically introducing perturbations to identify the limits of sim2real transfer. |

Conservative initial settings: begin with reduced speed limits and safety threshold defaults set to conservative values. The safety mechanisms from Week 12 should be fully active. The first priority is observing behavior safely; optimization for performance comes after the failure modes are understood.

Comprehensive logging: record all sensor streams, all policy actions, all safety filter activations, and video of the physical execution. Hardware experiments are expensive and difficult to repeat; every run should yield maximum information.

Systematic perturbation: once nominal performance is established, systematically test the conditions that differ from the simulation training: different object placement distributions, different lighting, different operator-provided perturbations. Each condition that causes failure is a gap in the simulation model or domain randomization coverage.

If hardware is unavailable, the same structure applies to a high-fidelity simulation stress test: deploy the policy in a simulation environment with higher fidelity than the training environment (e.g., full mesh collision geometry instead of convex hulls, recorded sensor noise profiles instead of Gaussian approximations), systematically varying the parameters that are most likely to differ from hardware.

Case study: bimanual cloth folding

Cloth folding serves as a canonical case study for the full sim2real pipeline because it exercises every challenging aspect of the curriculum simultaneously: deformable object modeling (Week 3 contact and friction), high-dimensional visual observation (Week 2), bimanual teleoperation and data quality (Week 4), behavior cloning with multimodal grasping strategies (Week 5/8), sim2real for deformable objects (Week 7), and safety constraints on bimanual workspaces (Week 12). Tracing a single pipeline through all stages makes the inter-stage dependencies concrete.

Task Specification (Case Study)

Task specification: the robot must fold a rectangular cloth (30 × 40 cm) in half along its longer axis, starting from a randomly placed flat configuration on the table. Success is defined as the folded cloth landing within 3 cm of the table center, with folded-over crease within 10° of horizontal, evaluated by a depth camera. Initial conditions randomize cloth position (±10 cm), orientation (±30°), and initial drape configuration (some cloth may hang off the table edge). Failure is declared if the cloth falls off the table or if either arm exceeds 40 N end-effector force.

Simulation Setup

Simulation setup: deformable cloth is modeled as a mass-spring system (or, in Isaac Sim, with PhysX cloth simulation). The critical parameters are cloth stiffness (bending and stretching moduli), damping, and table friction — all of which must be identified from physical experiments because no standard values transfer reliably across different fabrics. System identification uses a calibration protocol: drop a known cloth from a fixed height and fit simulation parameters to match the settling trajectory and contact shape. This narrows the domain randomization distribution for cloth stiffness from a broad prior $[0.1, 10.0]$ N/m to an identified range $[0.8, 2.5]$ N/m centered on the physical value.

Policy Training (Case Study)

Policy training: behavior cloning with ACTAction Chunking with Transformers (Week 8) is the natural choice for this task because the bimanual coordination requires predicting joint action chunks for both arms simultaneously, and the multimodal grasping strategies (fold from left edge vs. fold from right edge) benefit from CVAE mode selection. The observation space includes: both arm proprioceptive states ($q_t, \dot{q}_t$ for each arm), wrist force-torque readings ($F_t \in \mathbb{R}^6$ for each arm, included for slip detection), and two wrist-mounted RGB cameras providing close-up views of the grasped cloth during folding. The BC training objective is the ACTAction Chunking with Transformers CVAE-ELBOEvidence Lower Bound over demonstration chunks from 150 bimanual demonstrations collected via leader-follower teleoperation.

Domain Randomization (Case Study)

Domain randomization: in addition to the identified cloth parameters, randomize lighting (intensity ±40%, direction ±20°), table texture (from a library of 50 textile surface photos), arm joint friction (±15% of identified values), and communication delay (1–5 ms per arm, independently). Visual domain randomization with aggressive texture randomization is applied to the background and table surface, but not to the cloth itself — randomizing cloth texture too aggressively prevents the policy from tracking cloth edges visually.

Evaluation (Case Study)

Evaluation: 100 trials per condition across three evaluation conditions: nominal (cloth as during training), cold cloth (stiffer: +30% bending modulus), and worn cloth (lower friction: -25%). Wilson score intervals are computed for each condition. Fisher's exact test compares nominal to off-nominal conditions. Expected result: ACTAction Chunking with Transformers with CVAE achieves 70–80% nominal success; the fold-from-left and fold-from-right modes are cleanly separated in the CVAE latent space; performance drops to 50–60% on worn cloth (lower friction means grasps slip during lift), revealing that the randomization range for cloth-table friction was insufficient.

Hardware deployment failures and their diagnoses: on first hardware deployment, two failure modes appear. First, grasps succeed in simulation but slip during lift in 30% of trials on the physical cloth — a domain shift failure traceable to insufficient friction randomization for the specific fabric. Fix: widen friction randomization range and add more demonstrations with deliberate grip tightening. Second, the fold crease is systematically off-center by 5 cm — a perception failure traceable to camera calibration error (the wrist camera had shifted 2° from its calibrated pose). Fix: re-calibrate and add camera pose randomization (±3°) to the domain randomization schedule. After these fixes, the re-trained policy achieves 78% success on the physical robot, matching the simulation evaluation performance.

Course conclusion

The Modern Robot Learning curriculum has traced a complete arc from the first principles of robot kinematics and dynamics through the full modern stack: imitation learning, RLReinforcement Learning with safety constraints, sim2real infrastructure, foundation model architectures, diffusion and flow matching, VLAs, adaptation, and multi-task generalization.

Every topic has a precise relationship to the themes established at the start of the course. Classical control (PID, LQR, MPC) provides the baseline against which learned controllers are measured and the safety guarantees that learning-based systems must respect. Imitation learning initializes policies from human demonstrations — the same supervised fine-tuning paradigm that initializes language models before RLHFReinforcement Learning from Human Feedback. Domain randomization is the physical counterpart of data augmentation and distributional robustness in vision and language. Flow matching and diffusion policies are the generative modeling paradigm applied to the action distribution, analogous to their application to image and text generation. VLA models are the embodied analog of large language models, instantiating the same pretraining-plus-fine-tuning paradigm in physical systems with physical consequences.

The open problems that remain are the same problems the field has always faced, now at larger scale and higher stakes: sample efficiency, distributional robustness, long-horizon credit assignment, and safe exploration. The tools have improved dramatically; the fundamental challenges have not. Understanding both what has been solved and what remains open is what defines technical depth in this field.

Conceptual questions

1. Design a complete task specification for a sim2real experiment involving a robot arm that must sort three objects (cube, cylinder, sphere) into color-coded bins while avoiding a dynamic obstacle (a second arm moving on a fixed trajectory). Write the formal success criterion, failure criterion, initial condition distribution, and safety constraints. Identify the three most likely sources of sim2real gap for this task, the simulation parameters you would prioritize for system identification, and the domain randomization ranges you would apply to each.

2. A policy trained in IsaacLab with domain randomization over friction coefficient $\mu \in [0.3, 0.8]$ achieves 90% success in simulation but 45% success on physical hardware. Physical measurement shows $\mu = 0.35$ — within the randomization range. Design a systematic failure analysis: which parameters, other than friction, should be investigated as root causes? Describe the specific experiment for each parameter that would confirm or rule it out as a contributing factor. Which failure mode would you investigate first, and why?

3. A team presents a robot manipulation result with 80% success rate in simulation, using domain randomization, but no hardware results. As a peer reviewer, design a set of additional experiments (either simulation-based or hardware) that would substantially increase or decrease your confidence in the system's real-world viability. Specifically identify: (a) what information the current results provide, (b) what critical information is missing, and (c) what the minimum additional evidence would be to claim the system is ready for hardware deployment.

4. An ablation study for a sim2real pipeline shows: full system → 85% success; no domain randomization → 60% success; no curriculum → 75% success; no safety filter → 84% success; simpler action space → 70% success. Interpret these results: which components are most critical? Which component appears redundant? Propose two follow-up experiments motivated by the ablation results that would deepen understanding of the system's behavior — specifically targeting one surprising result and one result that confirms a prior hypothesis.

5. Synthesize the full Course 2 curriculum by tracing the role of each major topic in a single pipeline for training a language-conditioned bimanual manipulation policy (folding a garment). Specifically, for each of the following topics, identify: what specific engineering decision it informs, what failure mode occurs if it is ignored, and what metric would reveal that failure in evaluation: (a) teleoperation and data collection, (b) imitation learning vs RLReinforcement Learning, (c) sim2real and domain randomization, (d) VLA architecture, (e) safety constraints, and (f) multi-task learning.

Further reading

• OpenAI et al. (2019). Solving Rubik's Cube with a Robot Hand. arXiv. (The ultimate case study of Sim2Real, Domain Randomization, and scale).
• Margolis, Ro., et al. (2022). Rapid Locomotion via Reinforcement Learning. RSS. (Excellent end-to-end pipeline example for quadrupedal robots).