Agile-WAM Speeds Up Tactile World Models for Robot Manipulation

A new robot control architecture called Agile-WAM aims to solve a familiar tradeoff in robotics: models that understand rich, contact-heavy physical interactions tend to be too slow for real-time control. As reported in a paper posted on arXiv, the system is designed to predict both future world states and robot actions jointly, a class of methods known as World Action Models (WAMs), while staying fast enough for high-frequency manipulation tasks.

According to the paper, previous tactile WAMs typically depend on large-scale pretrained generative backbones to model complex contact dynamics — a design choice that limits inference speed and makes the models harder to deploy flexibly. Agile-WAM instead encodes visual and tactile observations into a single shared latent space, which feeds directly into a vision-tactile-to-action flow-matching process. This process simultaneously generates latent representations for chunks of robot actions and for predicted future visual and tactile states, without requiring a heavyweight generative model as an intermediary.

A central insight behind the design is that vision and touch don't change at the same pace. Consecutive camera frames tend to look very similar from one moment to the next, but tactile signals can shift abruptly the instant a robot's fingers make or break contact with an object. To account for this mismatch, the researchers built in what they call multi-horizon multimodal prediction: the model is trained to predict visual latents further ahead in time, while tactile latents are predicted just one frame ahead, capturing the fine-grained, fast-changing nature of contact events.

The team evaluated Agile-WAM across nine simulated manipulation tasks and five real-world contact-rich tasks. Across this test suite, the model outperformed the strongest baseline in task success rate while keeping inference latency low. The real-world results were the most striking: a 29.4% relative gain in overall success rate compared to the best baseline, paired with an inference latency of just 11.9 milliseconds — fast enough to support tight, high-frequency control loops needed for delicate manipulation.

The paper frames these results as evidence that multimodal world action models don't need massive generative backbones to work well. A more compact, purpose-built architecture, the authors argue, can deliver both the physical understanding needed for contact-rich tasks and the responsiveness needed for real-time robot control. The work was submitted by Hanchu Zhou and collaborators, with additional details promised on a project page linked from the paper, though that page's URL was not specified in the abstract text available at publication.

While the paper doesn't specify exactly which robot platforms or objects were used in the real-world trials, the reported gains suggest a meaningful step toward tactile-aware robots that can react quickly to unexpected contact — a capability relevant to tasks like insertion, assembly, and delicate object handling where visual feedback alone often falls short.