Overview

The policy is trained for 20k steps with a batch size of 64 on one RTX 5090 (29 min). Everything is trained from scratch except the DINOv2 image encoder, which starts from its pretrained weights and is fine-tuned. The policy is conditioned on the wrist camera, the action flow and the robot state, and generates the next 32 action steps (1.6 s at 20 Hz).

• Robot state (9 values): gripper position x, y, z in the sim world; gripper orientation as a quaternion (4, near constant, gripper pointing down); finger positions (2), how far each of the two fingers has slid out from the centre, 0 (closed) to 4 cm (fully open).
• Action (4 values per step): gripper move dx, dy, dz in [−1, 1], where 1 asks the arm controller for 5 cm in one 20 Hz step; gripper command −1 (open) or +1 (close). The orientation is never changed.

Results

• All policies are trained for 20k steps (batch 64).
• At evaluation the policy predicts 32 actions, the first 10 are run, then it predicts again.
• Each episode is run 5 times for each of 5 seeds (25 runs). Tables show the success rate (%), mean ± std over the 5 seeds.
• Flow-conditioned evals use the ground truth action flow from each episode's data, aligned to the sim's clock.

Key comparison

• Baselines are standard visuomotor policies: the same model with no action flow conditioning.
• The baseline with only the wrist camera scores poorly: 59 % train / 37 % test, against 84 % / 78 % with both cameras.
• Adding the action flow to the wrist camera raises it to 92 % / 77 %: close to the two-camera baseline on test, and better on train.
• So the flow-conditioned policy's performance comes from the action flow conditioning, not from the image observations.
• Interestingly, the baselines do surprisingly well on the episodes that failed in our data collection (42 %), while the flow-conditioned policies fail on them (7 %): they follow the flow, and the flow is of a failed replay.

All runs

Ablations

• No image pos. emb.: the learned position embedding is added to the state and flow tokens only, not the image patches.
• Flow dropout 0.2: during training, the whole flow is replaced by a learned mask token in 20 % of samples.
• World-frame flow: flow points are given in the sim world frame instead of relative to the gripper.

Visualizations

• 10 val and test episodes, chosen where, in the same eval run (same seed and attempt for all three), the baseline with the wrist camera fails while the baseline with both cameras and the flow-conditioned policy succeed. Of the 20 val and test episodes, 17 have such a run; these are the 10 where it happens most often.
• Each column: the sim's third-person camera on top, the wrist camera below. Success is LIBERO's cup-on-plate check after 1 s.

Third-party dependencies

Limitations

Content to come.