MicroDuck

How to train Microduck: reinforcement learning from sim to robot

Train new moves for Microduck with reinforcement learning: set up the mjlab environments, shape a reward, run training, then move the policy onto the robot. What you need, what to expect, and the traps to avoid.
Oct 13, 2026

A Microduck robot walking past a monitor showing training curves

Every one of Microduck's seven moves - standing up, sitting and standing, walking, turning, kicking, driving and grabbing - is a reinforcement-learning policy that was trained in simulation and then transferred to the hardware. Nothing about that stack is secret: the training environments are public, and this guide walks the same path the shipped policies took.

If you searched for "microduck how to train" or "microduck rl", start here. And if you do not own a robot yet: you can do every step up to "deploy" without one. That is the point of the simulator.

What you need

  • A computer. Linux x86_64 is the recommended environment for training. Python 3.12 for the mjlab side.

  • A GPU, ideally. An NVIDIA GPU (8 GB or more) makes training runs practical. Small experiments and driving around run on CPU while you learn the loop.

  • The repositories. The robot repo and the RL environments repo, both open source.

  • Optional: the robot. Everything below can be done in simulation first. The Dev pack - spare motors, motor cables and batteries - is the sensible add-on once you start deploying your own policies, because experimental policies are the ones that find the weak spots.

Step 0: meet the robot in simulation

Before training anything, run the shipped policies. The simulator guide covers both the browser Space and the local install. Watch what the seven trained moves look like - you are going to be asking a neural network to reproduce and extend that behaviour, so it helps to know the target.

Step 1: set up the training environment

  1. Clone the RL environments: https://github.com/pollen-robotics/microduck_rl - the mjlab-based training environments for Microduck.

  2. Clone the robot repository as well: https://github.com/pollen-robotics/microduck - the robot software you will deploy onto.

  3. Set up Python 3.12 and the project dependencies. The repositories document the exact steps; the documentation explains how the pieces fit together.

  4. Run a stock environment once. You want to see the robot in simulation controlled by a training loop before you change anything.

Step 2: pick the behaviour, then shape the reward

This is the real work. A policy learns whatever your reward function actually rewards, so the first skill is stating the goal numerically:

  • A faster walk? Reward forward velocity, penalise falling and energy use.

  • A tighter turn? Reward angular velocity in the direction you want.

  • A recovery? Reward getting back upright quickly after a push, and punish the time spent down.

  • A trick? Break it into stages and reward progress through them.

Start from an environment that already trains something close - the walking environment is the reference implementation - and change one thing at a time. Reward changes that arrive in bundles are the hardest bugs to find, because every one of them works until it does not.

Step 3: train

Run the training, watch the curves, and be patient. The shipped moves are the output of exactly this process, so expect it to take real compute: your first custom policy is a days-to-weeks project depending on the behaviour and your hardware. Shorter runs are still useful - a policy that walks badly in simulation teaches you what to change before the long run.

Step 4: randomise everything you can

A policy that only walks in the one exact simulation you trained it in will fall over on a real floor. Domain randomisation is the fix: randomise friction, masses, sensor noise, and pushes - so the policy learns the behaviour, not the scene. The environments expose these knobs; turn them up before your final run, not after.

Step 5: move the policy onto the robot

When a policy survives simulation and the randomisation, deploy it to the Microduck and expect to tune: real motors are not simulated motors. Sim-to-real is not the last step, it is its own phase of work - usually the phase where you learn the most about your own reward function. Give yourself a safe space: a clear floor, and the game controller within reach for when the duck gets confident.

What to train first

  • Turn in place - the classic first custom policy: simple to score, easy to see improvement.

  • Faster walk - velocity tracking, measurable in both sim and reality.

  • Recovery - train the duck to get back up after a shove. The most impressive and the most useful.

  • A dance - a chain of poses with timing rewards. Optimisation for style points.

Training advice that saves weeks

  • One change at a time. Reward, randomisation, network size - change one, rerun, compare.

  • Watch replays, not just numbers. A curve that goes up while the robot moves like a spider means your reward is wrong, not the training.

  • Expect to throw the first policy away. Everyone's first reward function rewards something unintended.

  • Keep the shipped policies safe. Your own policies are experiments; the seven trained moves are your baseline to compare against.

  • Ask the community. The Pollen Robotics Discord is full of people training the same robot, and the repos take issues and pull requests.

How to train Microduck: FAQ

Do I need to know Python? Yes to be productive: the SDK, the environments and the training scripts are Python. You can start by editing configs rather than writing code.

Do I need a GPU? Strongly recommended for real training runs. Small experiments and evaluation work on CPU.

Can I train without owning a robot? Yes - the entire pipeline up to deployment runs in simulation.

How long does training take? A first custom policy is typically a days-to-weeks project: setup is an afternoon, the first honest attempt takes longer, and success depends on behaviour complexity and hardware.

Can a bad policy damage the robot? A bad policy mostly embarrasses itself - falls, flailing, wrong directions. Mitigate with test space and prepared spares (the Dev pack exists for exactly this).

Where are the training environments? In the microduck_rl repository on GitHub, built on mjlab. Start from the documentation.

Where to go next

How to train Microduck: reinforcement learning from sim to robot