Overview
How the flow goes into Cosmos
- Cosmos only takes in and predicts latent video frames, each a fixed block of 16 × 32 × 56 = 28,672 numbers. The flow is 896 numbers.
- To predict flow without changing the model, we write the flow into one extra latent frame, repeated 32 times to fill it, and put it after the video frames (the Cosmos Policy trick).
- Cosmos treats it like any other frame: it is noised, cut into tokens and denoised with the video, so the hand's motion and the video are predicted together. We read the flow back by averaging the 32 copies.
- The noise is the same for every copy. With different noise per copy, the model could clean the flow by averaging its copies without looking at the image, and would learn to predict the average flow.
- The flow state (v2) uses the same trick: one row of 56 numbers, the hand and mug now, fills its own frame.
End-to-end evaluation
Results
- Two flow-conditioned policies from the Policy page (one table each), with the same eval: closed loop in the sim, 10 actions per prediction. Within a table only the flow differs: ground truth, or the WAM's plan from the first real frame. The same WAM plans are used for both policies.
- Test (val + test, 20 episodes): 5 WAM plans per episode (plan seeds) × 5 policy seeds = 25 runs. Train and failed: 1 plan × 5 policy seeds. Success rate (%), mean ± std over the runs. The baseline and ground truth flow rows are the Policy page's evals (5 seeds × 5 attempts).
- Flow state: v1 has none; v2 predicts it in the first chunk, or is given the ground truth flow row at frame 0 (ablation).
- Pauses: the WAM's plans rarely hold still, so the replay's pauses are inserted after planning: the flow holds 1.0 s at the grasp and 0.75 s at the release, found from the plan's thumb and index tips.
E2E Policy: full_wrist_state_no_img_pos (wrist camera + flow + robot state)
| flow | flow state | pauses | train (80) | test (20) | failed (46) |
|---|---|---|---|---|---|
| no flow: baseline, both cameras | – | – | 84.4 ± 1.6 | 77.8 ± 1.7 | 42.1 ± 1.6 |
| no flow: baseline, wrist camera | – | – | 59.1 ± 1.2 | 36.6 ± 2.9 | 24.9 ± 1.7 |
| flow-conditioned, ground truth flow (E2E Policy) | – | – | 93.5 ± 0.4 | 73.0 ± 1.8 | 6.8 ± 0.4 |
| WAM v1 | none | – | 48.5 ± 2.8 | 25.6 ± 7.3 | 3.5 ± 1.7 |
| WAM v1 | none | ✓ | 58.2 ± 2.9 | 27.8 ± 8.0 | 4.8 ± 0.9 |
| WAM v2 | predicted at frame 0 | – | 36.5 ± 3.4 | 18.0 ± 4.5 | 3.9 ± 0.9 |
| WAM v2 | predicted at frame 0 | ✓ | 40.8 ± 3.8 | 21.6 ± 7.2 | 2.2 ± 1.4 |
| WAM v2 | ground truth at frame 0 | – | 43.2 ± 3.3 | 22.4 ± 7.1 | 3.9 ± 2.5 |
| WAM v2 | ground truth at frame 0 | ✓ | 51.0 ± 1.5 | 25.4 ± 5.3 | 3.9 ± 1.6 |
E2E Policy: full_state (flow + robot state, no camera)
| flow | flow state | pauses | train (80) | test (20) | failed (46) |
|---|---|---|---|---|---|
| no flow: baseline, both cameras | – | – | 84.4 ± 1.6 | 77.8 ± 1.7 | 42.1 ± 1.6 |
| no flow: baseline, wrist camera | – | – | 59.1 ± 1.2 | 36.6 ± 2.9 | 24.9 ± 1.7 |
| flow-conditioned, ground truth flow (E2E Policy) | – | – | 88.3 ± 1.0 | 62.2 ± 1.9 | 3.8 ± 0.4 |
| WAM v1 | none | – | 43.2 ± 3.2 | 23.6 ± 8.4 | 2.6 ± 0.9 |
| WAM v1 | none | ✓ | 59.2 ± 1.9 | 28.8 ± 5.7 | 2.6 ± 2.1 |
| WAM v2 | predicted at frame 0 | – | 32.8 ± 1.7 | 21.0 ± 7.6 | 4.8 ± 2.1 |
| WAM v2 | predicted at frame 0 | ✓ | 36.2 ± 2.5 | 20.4 ± 5.1 | 5.2 ± 1.7 |
| WAM v2 | ground truth at frame 0 | – | 39.0 ± 2.7 | 23.6 ± 7.8 | 2.2 ± 1.4 |
| WAM v2 | ground truth at frame 0 | ✓ | 39.5 ± 4.3 | 22.2 ± 6.6 | 0.0 ± 0.0 |
Which episodes succeed: wrist-camera baseline vs WAM v1 + pauses
- An episode counts as a success for a run when more than half of its trials succeed. Baseline: 25 trials per episode. WAM v1 + pauses: 25 on test (5 plans × 5 policy seeds), 5 on train and failed (1 plan × 5 policy seeds).
- Each dot is an episode at its cup start position (top view of the table, as in the Data page's splits chart).
| episodes | train | test | failed |
|---|---|---|---|
| both succeed | 38 | 2 | 0 |
| baseline only | 15 | 4 | 8 |
| WAM only | 10 | 2 | 2 |
| neither | 17 | 12 | 36 |
Visualizations
- WAM v1 + pauses with the flow-conditioned policy (full_wrist_state_no_img_pos): 5 val and test episodes it solves and 5 it fails. Each is a real eval run (policy seed 0; plan seed 0, or plan seed 1 for 111418), re-rendered with the same outcome.
- Panels: my recording with the ground truth flow | the WAM's generated video with its planned flow | the robot's wrist camera | the robot from a third-person view. The plan is made from the first real frame only.
success
failure
Third-party dependencies
| tool | used for |
|---|---|
| Cosmos-Predict2-2B Video2World | pretrained video model (2B DiT), fine-tuned with LoRA to predict video + flow |
| Wan2.1 VAE | encodes frames to latents and decodes predicted latents to video (frozen; the VAE Cosmos-Predict2 uses) |
| T5-11B text encoder | embeds the prompt once (frozen; shipped with Cosmos-Predict2) |
| Cosmos Policy (arXiv 2601.16163) | the method for writing small vectors into a latent frame, and its noise schedule |
Limitations
Content to come.