Learning Expressive Humanoid Locomotion from Monocular Runway Videos for Robot Fashion Shows

Kyrylo Kolesnichenko; Irvin Steve Cardenas; Jong-Hoon Kim

Summary

Locomotion controllers are tuned for stability and speed, not for how the walk looks. This project asks whether a humanoid can learn a recognisable runway walk from nothing more than an ordinary video of a person doing it. The pipeline takes a single monocular clip, recovers the 3D human motion, cleans it up, retargets it to the K1, and produces a motion-tracking task that trains a policy in Isaac Lab. The policy is checked in MuJoCo before it goes on the robot.

Pipeline

Stage	Tool	Notes
Frames	ffmpeg	resample to 30 fps
3D human motion	GVHMR	SMPL-X in world space, static-camera mode
Clean-up	this repo	foot contacts, floor levelling, root-drift removal
Retargeting	GMR	human skeleton → K1 joints (UMR in later variants)
Policy	BeyondMimic, Isaac Lab	motion-tracking RL
Check	MuJoCo	sim-to-sim before hardware
Robot	Booster K1	physical trials

Results

On the physical K1, over a 5 m runway:

20 of 20 trials completed without a fall, about 23 steps each.
Step width of −0.8 to 1.8 cm for the learned gait, against 5.1 to 11.4 cm for the stock K1 walk. A negative width means the feet cross the midline: the policy produces occasional crossover steps, which is what a runway walk looks like.

The paper is a Late Breaking Results poster at IROS 2026.¹

Release

Notes from packaging the code: what it takes to get a fresh machine to the point where process.py runs at all.

Two environments, on purpose. GVHMR (video → 3D human motion) and GMR (human → robot retargeting) pin incompatible dependencies, so the install script creates two:

Environment	Used for
`expressive-motion`	retargeting, CSV conversion, task generation. The one you activate.
`expressive-motion-gvhmr`	GVHMR inference on the GPU. Called by `process.py`; you never activate it.

The cost is about 25 GB of disk. The benefit is that neither tool’s install can break the other’s.

Linux x86_64 only. The pinned pytorch3d wheel exists only for linux_x86_64. GVHMR needs an NVIDIA GPU with CUDA 12.1+; nothing else in the pipeline does, so bash install.sh main skips it on machines without a GPU.

The weights are the real blocker. About 5.2 GB, none of it downloadable by a script: SMPL and SMPL-X need separate registrations with signed licences, and the GVHMR checkpoints come from the upstream Google Drive folder. The README has a table of every file, its size and what needs it.

Static camera, so no DPVO. Runway videos are shot from a fixed camera, so GVHMR runs in static-camera mode and the visual-odometry model is skipped.

Isaac Lab itself is never installed by the script. bash install.sh isaac <env> only adds the Booster packages to an Isaac Lab environment you already have.

An LBR poster, not a full conference paper. The arXiv version is listed as an “IROS 2026 Poster Paper”. ↩

Media

Runway video, recovered 3D human motion, retargeted K1 in simulation, and the physical robot — The four stages side by side.

The IROS 2026 poster — IROS 2026, Pittsburgh: the poster.

Kyrylo at his poster at IROS 2026 — At my poster.

At the Booster Robotics booth at IROS 2026 with two Booster robots — At the Booster booth.

Paper

Learning Expressive Humanoid Locomotion from Monocular Runway Videos for Robot Fashion ShowsKyrylo Kolesnichenko, Irvin Steve Cardenas, Jong-Hoon Kim.IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026), Late Breaking Results (LBR), 2026 · presented, posterarXiv pdf code video

BibTeX

@inproceedings{kolesnichenko2026learning,
  title         = {{Learning Expressive Humanoid Locomotion from Monocular Runway Videos for Robot Fashion Shows}},
  author        = {Kolesnichenko, Kyrylo and Cardenas, Irvin Steve and Kim, Jong-Hoon},
  booktitle     = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026), Late Breaking Results (LBR)},
  year          = {2026},
  eprint        = {2609.27003},
  archivePrefix = {arXiv}
}