The policy that is used for trajectory generation here is a Multitask Diffusion Transformer Policy [1] [2].

And it’s worth noting that this isn’t a VLA2 in the sense that it doesn’t use a pretrained VLM backbone, instead using much smaller vision and text encoders, but still shows extremely impressive performance. I don’t like to throw around the phrase “SOTA” in manipulation, because the benchmarking and evaluation process is quite nascent/hand-wavy in the field right now3 for anything beyond straight-forward tasks (LIBERO, etc), so without recreating on your own setup it’s impossible to know. With that being said, the results are pretty majestic.

But with no code publicly available, it’s not possible to compare this against other approaches.

This motivated me to work out an implementation from what has been released publicly, literature from TRI, additional digging and intuition. I’ve built a version of this policy, break down the implementation in this post, and have released an open-source implementation for the community.

What made it stand out to me is how far you can push performance without a massive compute budget. I’ve found you can achieve amazing results on single-task to few-task datasets with ~100-200 examples and ~10 hours of training on a single H200 (so for tens of dollars you can iterate on policies).

I’ve open-sourced the implementation, and made it available in two places:

• A standalone repo with a simple training pipeline and example inference loop which can be found here

  • this doesn’t directly integrate with any downstream hardware and users will need to do that themselves

  • there are a few extension pieces added here, like experimental vision encoders, etc, along with modularity to add experiment with future observation and training objective strategies

• Directly integrated it into HuggingFace’s LeRobot project, as I believe this was the best way to make the work the accessible and easily deployable on hardware

  • this is implementation is as close to the vanilla policy implementation as I could conceive; it should be simple and easy to understand, and give users a solid baseline to work from

Policy Architecture and Details:

The policy follows a diffusion transformer (DiT) adapted for 1D sequence modeling instead of 2D image generation, and is conditioned on all of the relevant observation information during generation [3].

This is policy is trained with classic behavior cloning, gathering data through tele-operation. TRI presented pretty in-depth analysis for the performance you can expect with this model when training on single task and leveraging large-scale pretraining, so refer to that to calibrate expectations [1]. Additionally, I’ve trained and evaluated the model against the LIBERO benchmark as a sanity check, and saw the following performance which may be helpful as well:

I’ve aimed to make this implementation relatively modular without introducing a ton of complexity, making it easier to extend or swap components for future experiments and ablations.

Submodules:

Vision Encoder

The vision encoder is a CLIP ViT (specifically ViT-B/16) that only uses the CLS token encoding to represent information for each image [4]. The majority of research on Diffusion Policy models still seem to use ResNet vision encoders, but when ablating between CLIP and ResNet there was a significant performance boost from CLIP, which isn’t surprising.

The idea of only using the CLS token and none of the patch embeddings is to keep the representation compact while retaining rich semantic grounding from the pretrained backbone.

The encoder is co-trained end-to-end, and follows the setup described in prior work, it uses a learning rate 1/10th that of the rest of the model parameters to maintain the stability of pretrained features. This balance allows the vision module to slightly adapt to the your environment/task without losing the rich generalization capabilities learned from large-scale pretraining.

Additional Note on Vision Encoders:

In the released implementation, I’ve also added a separate vision encoder module that can be used to extract visual features, DINOv3 [6]. The usage is identical to CLIP in that only the CLS token is used. I did not see better performance with DINOv3, but did not experiment very heavily (and actually saw degraded performance).

There’s many reasons this could be: the objectives of the models are inherently very different, there’s no semantic grounding in the DINOv3 features which may impede performance.

The rich spatial features in DINOv3 make me think it was possible that there could be an interesting performance boost in there somewhere (whether for singular task performance or generalizability), so I’ve left it in so that it can be experimented with. In addition, I’ve made it straightforward to add and swap in other vision encoders in the future (such as SigLIP, etc).

Text Encoder

For language conditioning, the policy uses the CLIP text tokenizer and embedding layers, but these are kept frozen throughout training. The only learnable component is a single projection layer that maps the CLIP text embedding into the features that are concatenated with the rest of the conditioning vector.

This keeps compute requirements minimal and stabilizes training, but it also leaves the language steering relatively weak (I talk about this more below), as the conditioning is a shallow text embedding rather than a deeply aligned joint representation. Still, this approach strikes a good balance for multitask learning without requiring full-scale language models and the downstream compute implications.

Proprioceptive Observations:

The proprioceptive observations (joint angles, or otherwise) are notably not processed or embedded prior to being concatenated into the conditioning vector. In practice this seems to work well, but it is interesting that the amount of proprioceptive features are in the conditioning vector are an order of magnitude or more smaller than the embedded exteroceptive and language features.

My intuition is this could be interesting to explore other rich feature encoding strategies as has been seen in many other papers, but the likely result is only marginal improvement.

Diffusion Transformer Prediction Model

The core prediction model is a Diffusion Transformer adapted from the original formulation from Meta, for image generation, to 1D sequence modeling for joint trajectory generation [3].

The network models action sequences as temporally correlated trajectories, where each denoising step uses the model to predict the noise, and refines the predicted actions, iteratively. This noise prediction is conditioned on an observation vector that concatenates the encoded images, encoded task description, joint angles, and embedded diffusion time step.

The actions are of shape (N, H), where N is the size of your action space, and H is the horizon length. They are initially sampled from a normal distribution and iteratively denoised to the final action trajectory.

In the implementation, I’ve added both standard vanilla positional encodings and rotary positional embeddings (RoPE) within the transformer blocks. In practice, RoPE consistently shows clear improvements in performance and robustness, especially in diffusion transformers, by better capturing relative positional structure across long horizons. With this, RoPE is used as the default positional encoding throughout the model.

Bringing All the Conditioning Together

When being passed into the DiT model, the visual, task-instruction, proprioceptive feedback, and diffusion time-step features are all concatenated into a single feature vector that the DiT conditions on during the denoising process.

This conditioning vector this the singular point that all of the observation information is used to generate the output trajectory.

Model Objectives

To make the policy flexible, I’ve implemented support for multiple generative objectives:

• Classic Diffusion Objective:

  This follows the standard denoising diffusion formulation, where Gaussian noise is iteratively added to the action sequence, and the model learns to predict and remove it step-by-step. The objective minimizes the expected L2 loss between the predicted and true noise, learning a score function over action trajectories.
  
  The implementation supports DDPM and DDIM sampling strategies as is common among most diffusion models.
  

• Flow Matching Objective:

  This replaces the discrete diffusion process with flow-matching that models the trajectory evolution as an ordinary differential equation. The model learns the instantaneous velocity field that transports noisy samples toward the data distribution, providing smoother training dynamics and often faster convergence.
  
  This objective can be configured as an alternative loss mode in the policy configuration. Additionally, I was interested in the beta distribution time step sampling that Physical Intelligence described in their π0 model, and implemented that as a time step sampling option as well [5].
  
  Note that their use of a flow-matching objective was for an action expert that’s integrated into a different architecture than described in this post. But, in practice, I found that this beta distribution did result in faster training and higher quality samples being generated at inference time.

Language Steerability Limitations

One of the trade-offs with this architecture is limited language steerability. Because the model only uses a small set of embeddings from the CLIP language encoder, the conditioning signal is relatively weak.

In practice, this means that if you try to compose instructions, for example, asking the policy to perform a sequence of skills it has seen individually, you’ll often get inconsistent or ignored responses. The model tends to “latch” onto one dominant skill from training rather than synthesizing or sequencing behaviors based on nuanced phrasing.

Whether this matters depends on what you’re trying to do. For focused, narrow-task applications, it’s not a deal breaker. In fact, I’d argue we’re at the point where these lightweight LBMs can deliver real value in production environments with well-scoped manipulation tasks, and the constrained language conditioning actually simplifies evaluation. If your goal is open-world generalization or robust instruction following though, this approach may hit its ceiling early. Still, it’s a strong baseline for benchmarking and a pragmatic step toward scalable multitask manipulation.

Common Failure Modes and How to Debug Them

These models are still early and they can exhibit finicky, hard-to-interpret failures. Below is a distilled guide to the most common, reproducible failure modes I’ve encountered while training this model, along with practical debugging strategies try.

1. Idling / Output Collapse (No Motion)

At inference time, the policy produces near-zero or static actions. The collapse often happens immediately at the trajectory start, but can also appear mid-task.

Potential Causes:
When the observations do not sufficiently disambiguate between multiple possible behaviors, the generation process seems to gravitate toward an “average” trajectory where the output can stall out. Two common causes I’ve seen for this

1. Insufficient training data for a given task

   • With only ~20–50 demonstrations, the policy often may not be able resolve action modes and collapses.

   • Once you’ve exceed ~300 examples for a single task, this collapse seems less likely to occur; if it persists, the task likely has hidden complexity or unobservable state information that you be using when gathering demonstrations.

2. Multiple similar tasks in the dataset

   • Example: moving two different objects that share similar geometry or visual appearance.

   • The CLIP text conditioning is shallow in this architecture, and it may not be strong enough to sharply separate tasks or environments with semantically similar observations and instructions.

How to Debug:

• Increase dataset size: double example count until you exceed ~300 demos per task.

• Train longer: don’t stop when loss plateaus; the model often needs ~50k–100k steps to converge.

• Strengthen or diversify task instructions: language steerability is limited; more explicit phrasing helps, but don’t make the description too long.

2. Performing the Wrong Task

As described above, at inference time, the robot may confidently perform a behavior from your dataset… but not the one you instructed.

Potential Causes:

• Ambiguous or overly similar text prompts: with the shallow CLIP text encoder frozen during training, small description differences often fail to meaningfully steer the model.

• Insufficient per-task data: a task with only a handful of examples is easily overshadowed by more popular tasks in the dataset.

• High overlap between tasks: when two tasks share visual structure, dynamics, or geometry, the model might to “default” to whichever appears more frequently.

How to Debug:

• Diversify task description (same as above)

• Rebalance task distribution: up-weight the underperforming task distribution when sampling during training

• Perform task-specific fine-tuning: A few thousand steps on only the failing task can dramatically improve instruction adherence

“How Can I Use This?”

I’ve released this in two places, and provide initial suggestions for dataset size and training parameters:

1. In a standalone repo that is stripped down to the bare-bones of the implementation for those interested in understanding the internals to dig into and build off of. This can be found here.

2. By integrating it into HuggingFace’s LeRobot project would make it the most accessible for others to actually getting it running on hardware4, which can be found here.

Note: I have not trained and released any pre-trained base-model weights; If there’s interest in this, please reach out to me!

I’ve defaulted to using the LeRobotDataset format in both places, and if you have an existing dataset in that format, training on this policy should work out of the box.

I won’t go into detail about LeRobot specifics, and defer to their own documentation within the repo, but within the policy README, I note high level training and inference suggestions.

So far, I’ve only trained and tested policies for simple single- and bi-manual manipulation setups, but I expect it would work well for whole-body manipulation on humanoids (as BD has shown).

Please share any trained policies demos you create — I’d love to see them!

What’s Next?

The majority of my current work is focused on RL fine-tuning, representation learning, and training encoders for new sensing modalities, all areas that are rich for robot learning research right now!

If this is an area you’re interested in or working on, I’d love to chat with you, so please reach out!

Acknowledgements:

A special thanks is given to the HuggingFace LeRobot team: Khalil Meftah, Steven Palma, Thomas Wolf, Michel Aractingi, as well as Aditya Bhat, and others at South Park Commons for their help reviewing this work.

Cite This Work

@misc{jones2025multitaskditpolicy,
  author = {Bryson Jones},
  title = {Dissecting and Open-Sourcing Multitask Diffusion Transformer Policy},
  year = {2026},
  url = {https://brysonkjones.substack.com/p/dissecting-and-open-sourcing-multitask-diffusion-transformer-policy},
  note = {Blog post}
}

@misc{jones2025multitaskditpolicy,
  author = {Bryson Jones},
  title = {Dissecting and Open-Sourcing Multitask Diffusion Transformer Policy},
  year = {2026},
  url = {https://brysonkjones.substack.com/p/dissecting-and-open-sourcing-multitask-diffusion-transformer-policy},
  note = {Blog post}
}

If you liked this post, please subscribe and share with others! I’m also always happy to chat about work like this, so please reach of if you’re interested.

Share

References

1. A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation - TRI LBM Team

2. Large Behavior Models and Atlas Find New Footing - Eric Cousineau, Scott Kuindersma, Lucas Manuelli, Pat Marion

3. Scalable Diffusion Models with Transformers - William Peebles, Saining Xie

4. Learning Transferable Visual Models From Natural Language Supervision - Radford et al

5. π0: A Vision-Language-Action Flow Model for General Robot Control - Black et al.

6. DINOv3 - Siméoni et al.

7. LeRobot

8. Flow Matching vs Diffusion - Blog Post - Harsh Maheshwari

9. FastUMI - Ding et al.

“Is vision really enough?”

“Tactile sensing is necessary to solve manipulation!”

“What about depth and LiDAR?”

If you’ve been working on robot manipulation over the past few years, you’ve no doubt either said this, or heard someone say it to you:

“It makes sense that using richer sensing modalities should produce a better foundation model, so why do the best methods not use them? Why not just build an advanced gripper with rich tactile data? Why are we ignoring all these amazing sensing modalities? Are people just stupid?”

Yes, there has been tons of great research over the past decade on this area, but the leading manipulation foundation models don’t incorporate any of these modalities besides vision and joint angles1. Why?

Alas, you’re not playing the game you think you are.

It’s not about which modality has more information or a richer representation. Those are important, but they aren’t the high-order bit2.

It’s about scale.

It’s all about data scale and diversity. That sounds obvious and non-helpful, I know, but it really is worth stating plainly. “Everyone knows data is the important part of training ML models!”

But do you really internalize the implications of that?

RGB cameras are dominating for a few reasons here:

• Cameras are dirt cheap

• There’s effectively zero integration lift: camera drivers and software to capture and process the data already exist, and beyond that, are highly optimized

• Large amounts of existing internet-scale data for this modality → enables large, general, pre-trained encoders (CLIP, SigLIP, etc) to be built, which produce rich representations that robotics models can leverage “for free”

Even though cameras alone aren’t able to provide any rich tactile feedback for grasping and dexterous manipulation tasks, you get to ride the coattails of all of these advantages. It’s just inherently very easy to use, deploy, and scale; so it wins by default.

This is also logically consistent with why the “new-hotness” in robotics foundation models is video-model backbones.

I seriously think the last point on availability of pre-trained modality-specific encoders is vastly underappreciated when considering sensing strategies. It’s an easy trap to fall into to over-extrapolate from previous paradigms of automation to try and inform your strategy, but thinking by analogy can get you into serious trouble.

Previous wisdom was to start with the most expensive sensor stack you can buy to make your classical, deterministic, optimal control stack work, and once it’s reliable, apply pressure to bring the sensing package down the cost curve. But doing that here just makes you end up with a sensing stack that’s too expensive and specialized to gather a meaningful enough amount of data to train a useful model, so your “high-precision-ultra-super-mega-sensor” is likely a hindrance, not a stepping stone.

So are new sensing modalities useless?

No! I think working on new approaches to bring in new sensors is a great place to focus on right now. However, you should attack it from the right “angle".

What modalities and sensing technologies are inherently compatible with the needs to drastically scale?

• Are they cheap enough to deploy and replace? As in, tens of dollars

• Are they easy to integrate into HW and SW platforms? Connectors, drivers, communication protocols, SDKs, etc

• What are different ways to build standalone pre-trained encoders for this modality that can be used downstream in various models?

• Bonus: is there the possibility of some way to extract data of this modality without collecting it all yourself? Yes, the siren song of “we’ll just push until the Fairytale Never-Ending Data Flywheel Deployment happens and all will work out” can be strong, but I encourage you to be intellectually honest about this.

A $10k tactile sensing gripper is unlikely to be the silver bullet here3.

I’ve got some work in this exact area going on right now, so if this is something that interests you, please reach out!

It made me want to make a quick blog post about it, so here you go!

Here is a look at the four conditioning paradigms that define the current landscape of diffusion modeling. There’s obviously more work going on than just these four, but this is what I’ve seen prominently and work effectively. If you have other methods you think are critical to know, mention them below!

1. Early Days: Simple Cross-Attention

The first widely successful approach to conditioning diffusion models came from using cross-attention to merge latent information from different signals, and this was before transformers were the common backbone of diffusion models.

In early latent diffusion systems, the problem was framed very simply: how do you inject a text description into an image denoising process to augment the generation process?

The initial approach that saw success was popularized by High-Resolution Image Synthesis with Latent Diffusion Models (Rombach et al., 2022) from Stable Diffusion 1.5 and beyond. This was to keep the two streams largely separate and connect them through cross-attention.

Text is encoded once by a frozen model such as CLIP, producing a set of keys (K) and values (V). The image latent, evolving through a U-Net backbone during denoising, produces the queries (Q). At each cross-attention layer, the model combines them through a classic attention block:

The cross-attention effectively “injects” conditioning by projecting text embeddings into the image denoiser, where image features act as queries over text-derived keys and values. This mechanism biases the denoising updates toward prompt-consistent semantics while preserving a separation between text representation and spatial image processing.

This design choice was extremely pragmatic and simple. For straightforward text-to-image synthesis, this ended up working remarkably well and defined the baseline for an entire generation of models.

However, the same separation that gives cross-attention its stability also limits its expressiveness. Because text and image representations never fully merge, the model struggles with fine-grained spatial reasoning, compositional logic, and especially typography (remember seeing letters scrambled like they were written schizophrenically?). The conditioning signal is directional and coarse: the text can guide what appears, but not reliably where or how precisely.

2. AdaLN (Adaptive Layernorm): Global Conditioning for Diffusion Transformers

When the paper Scalable Diffusion Models with Transformers (Peebles & Xie, 2022) came out, the field shifted from U-Nets to Diffusion Transformers (DiTs) as the backbones of diffusion models. In this same paper, AdaLN conditioning was introduced, and has really become a preferred method for injecting global context extremely effectively.

The idea is that, rather than adding extra tokens or specialized attention layers, AdaLN modulates the layer normalization blocks. The model learns to predict the scale (γ) and shift (β) parameters of the Layer Normalization based on a condition vector c.

• The operation is defined as:

This ends up being a really efficient way of applying global conditioning to the diffusion transformer backbone, like adding a time or style conditioning

AdaLN is nice because it’s computationally cheap, O(D) complexity versus the quadratic cost of attention. It influences the entire generation uniformly, making it excellent for setting global style or coherence.

One of the challenges though is is that lacks precision, and it’s edits can be coarse or blunt. It can’t easily tell the model to place a specific object at specific coordinates, or adjust the orientation of that object, etc.

Still, it is one of the most common and powerful ways to apply conditioning to one of these DiT models, and it’s implementation simplicity is quite elegant, making it easy to understand.

Side Note:

This is one of the most effective ways we’ve seen to condition diffusion and flow matching action chunking policies and -heads for robotics. You take all of your observations, camera view encodings, joint angles, task description, etc, and concatenate them into an AdaLN conditioning vector.

You can see this in action in a repo I’ve open-sourced for Multitask Diffusion Policy, here: https://github.com/brysonjones/multitask_dit_policy/tree/main

3. In-Context Learning (ICL): Few-Shot Adaptation

In-context learning is an approach to apply conditioning in the inputs of the diffusion transformer rather than an architectural modification to inject conditioning during processing.

One of the first papers we saw this with was In-Context Learning Unlocked for Diffusion Models (Wang et al., 2023), where the idea mirrors almost exactly few-shot prompting in LLMs. The model is given one or more example pairs, like an input image and its desired outputs (segmenting, edge detection, style transfer, etc) followed by a new query input.

We end up effectively treating these image tokens as "visual prompts" alongside or instead of text. Just as you might show a language model a few examples of English-to-French translation to teach it the pattern, in diffusion ICL, you feed the model a "context pair" consisting of a source image and a transformed version.

Practically, this is implemented either by concatenating images into a single tensor (for example: stacking an edge map and its target photo next to a new edge map), or by using attention masking so that query tokens can attend to example tokens. The diffusion process implicitly learns the transformation by pattern matching within its context window.

This approach ends up working quite well, and multitask demonstrations end up teaching the model how to combine and generalize these editing concepts. It allows a single pretrained model to perform edge-to-image, colorization, style transfer, and other conditional tasks without specialized adapters.

But this flexibility and expressivity comes at a cost. Because the model is not explicitly optimized for the task, output fidelity usually lags behind fine-tuned or adapter-based methods (this starts to change as you scale, as most models do). Additionally, the approach is fundamentally bounded by context length (quadratic cost on computation) and GPU memory, making it difficult to scale to complex or high-resolution demonstrations.

4. Joint Attention for Multiple Modalities (MM-DiT)

Joint Attention with MM-DiTs represents the one of the most recent big leap in diffusion conditioning. This approach basically creates two separate streams of data that run in parallel through the model layers:

• Image Stream: Processes the visual latent patches (the noisy image being denoised).

• Text Stream: Processes the text tokens (from the prompt).

Importantly, each stream has its own set of weights. This means the model learns separate parameters to process visual data and textual data, acknowledging that pixels and words behave very differently and have different patterns to learn, etc.

Then within the MM-DiT blocks, image and text tokens are concatenated into a single sequence, processed with attention, and then split back into their distinct streams.

This iterative merging and separation helps ensure the model retains exact character sequences and resolves ambiguity, leading to superior typography (no more misspellings, mangled letters, etc) and complex prompt adherence.

This idea was introduced in Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Esser et al., 2024) for Stable Diffusion 3. The architecture evolves the simple cross-attention concept into an model that more fully merges the multi-modal signals and you can see the diagram below:

There’s quite a few trade-offs though:

• Just like ICL, because we are increasing our input context for these signals, we incur quadratic compute cost increases for increasing context/conditioning length

• As you can see from the diagram above, this architecture is quite complicated with lots of sub-details of implementation, making the implementation and hyperparameter tuning cumbersome

There’s a lot of exciting work going on with diffusion/flow-matching generative models right now, and it’s easy to get lost in the sauce of all the different methods. I found trying to group and categorize approaches like this helped me grasp when and where to try and apply the different strategies.

I’m working on diffusion models and other architectures for multi-modal generation, representation learning research, and beyond.

Most of my work is focused on robot manipulation, world-modeling, and decision-making. Reach out if you are interested in chatting (https://x.com/brysonkjones), and share this post with others!

Share Bryson Jones

Subscribe now

However, the frontier challenges we face are very much in manipulation, or the ability to handle, assemble, and interact with the physical world with our robotic systems.

We don’t think of the ability to pick up an egg without cracking it or folding a blanket as the most impressive tasks humans are capable of, but they are some of the hardest problems to build intelligence for.

This is a prime example of Moravec’s Paradox, effectively: the hard things are easy, and the easy things are hard.

What makes up robot autonomy?

The problems within robotics can largely be broken up into:

• Navigation

• Locomotion

• Manipulation

Jitendra Malik gave a keynote talk over the past year (I unfortunately forget which conference this was1) that went as far as spicily saying something along the lines of:

“Navigation is solved, locomotion is basically solved, and while we are making progress on manipulation, we really don’t have what the recipe is yet to solve it”

I’m a huge fan of Jitendra, and while I know part of his goal with this slide was to be provocative, I generally agree (although I’m more bullish on the track we’re on in manipulation than he may be).

Three ways to train manipulation policies

Researchers are attacking manipulation from three main directions:

1. Behavior cloning from tele-operation data – teaching robots by having humans control them and mimicking the demonstrations.

2. Large-scale unsupervised learning – using internet-scale video data to learn broad priors about how actors in a scene interact with objects and the environment they’re in

3. Reinforcement learning in simulation – training policies in massive parallelized environments where billions of interactions can be run safely and cheaply.

Each approach has made progress, but none has cracked the full reliability and generalization needed for everyday manipulation tasks.

I don’t think it’s controversial to say that #1 (BC through tele-op) is the leading approach in terms of performance, but there’s a lot of work going on behind the scenes at places like SkildAI, Tesla, etc that we aren’t privy to and things are moving so quickly this could be wrong by the time you read this.

Why locomotion is simpler

“Making this huge hunk of metal run looks pretty hard, man.”

Locomotion benefits from properties that make it extremely well-suited for massively parallel reinforcement learning in simulation:

• Clear reward functions:

  The reward function for locomotion is actually pretty easy to formulate: move forward, have smooth motions, minimize energy usage. These are mathematically compact objectives, and almost feel tailor-made for reinforcement learning.

• Minimal sensing needs:

  Proprioceptive feedback (joint angles, velocities, foot contacts) often suffices. Many locomotion policies succeed even without cameras, relying only on physical feedback.

• Representation efficiency:

  High-performing locomotion policies can evidently be squeezed into ~10 million parameters (you could argue more for multiple embodiments, but I’ll make the generalization)

• Structural regularity:

  Locomotion has strong symmetries where gaits are often periodic and physics is uniform across terrains. This allows policies to generalize well, with intelligent behaviors like efficient gait cycles or arm-swinging for energy minimization emerging naturally.

All of these aspects lead to the ability to leverage large-scale simulation + RL to train super-human (super-dog for quadrupeds??) locomotion policies.

What makes manipulation different (and harder)?

“How hard can folding these f&%$ing clothes be?”

Manipulation poses a more sobering challenge:

• No compact reward functions

  The “cost function” for inserting a screw or cooking an omelet is hard to specify. Rewards often need careful shaping or human priors, leading to brittleness and unexpected outcomes from reward-hacking.

• Rich sensing requirements
  Unlike locomotion, manipulation usually requires vision and tactile feedback to estimate object shape, pose, affordances, and contacts. Tactile sensing hardware is still immature. The example I like to give people is: “imagine dipping your hand in anesthetic and trying to pick your phone up… it’s basically impossible”

• Discrete modes and long-horizons introduce unique complexity

  Manipulation often involves discrete modes for task completion: grasping, lifting, pushing, where each object can behave differently. Contact dynamics are messy, occlusions are common, and the precision bar is much higher than for locomotion.

• Few emergent behaviors

  Unlike locomotion, manipulation doesn’t “discover” elegant solutions on its own (so far). Without demonstrations or heavy engineering, policies struggle to converge on useful strategies.

This is why we haven’t been able to just “zero-shot” transfer all of the methods that accelerated locomotion progress in the past 5 years into manipulation.

Where does this leave us?

This field is moving so fast that some of these challenges in manipulation could be solved in the next month, or it may take a few years; I don’t want to paint too bleak of a picture, but really just wanted to highlight that progress in locomotion often gets conflated with how to tackle manipulation, and I don’t believe that’s fair to do.

I’m personally much more optimistic than many in the field are about how quickly behavior cloning with tele-operation data collection will result in useful2 manipulation policies that can be deployed into real-world settings and kick off a data flywheel of self-improvement (this is the bet we3 are making).

On the topic of reward functions, there’s a lot of amazing work happening leveraging large pre-trained VLMs (and other foundation models) to bootstrap reward models that can rapidly be adapted to new tasks.

I could talk for hours about all of the exciting work going on in the field right now, but will save that for a future post.

If you liked this post, feel free to subscribe, share with others that might enjoy, or reach out to chat!

Subscribe now

“The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.” - George Bernard Shaw