ReContact

Reconstruct and Retarget Human Contacts from Video for Contact‑Rich Humanoid Manipulation

Zhongkai GuYang LiuWei Zhu*Mitsuhiro Hayashibe*

Tohoku University

*Corresponding authors

Real robot · 1.5×
A Unitree G1 moves a 1.9 kg box from a side clamp through its thigh to a two-palm hold, and back.

Abstract

Contact transitions allow humanoid robots to redistribute object support across the whole body, enabling manipulation beyond hand-only grasps. Human videos provide a scalable source of such transitions for robots to learn from. However, body occlusion disrupts object pose tracking and corrupts retargeted motion. Moreover, accurately reproducing the object trajectory does not ensure that the robot applies stabilizing forces at the intended contacts, risking slips or drops in later phases. To address these challenges, we propose ReContact, which reconstructs and retargets human-object interactions through an object-centric contact plan. The plan specifies contact phases, contact pairs, anchors, pressing directions, and sticking or sliding modes. Contact priors stabilize object pose reconstruction through occlusion. A residual policy over a frozen whole-body tracker is rewarded for making contact at planned anchors with forces aligned to the specified directions. In simulation, on a thigh-supported regrasp task, ReContact achieves the planned two-palm bottom hold in 83.5 ± 5.2% of episodes over three training seeds, versus 0.16% for the best adapted baseline. One policy trained on 247 references re-solved for new placements raises in-workspace tube-transfer success from 20.4% (source reference only) to 97.8%. On a real Unitree G1, policies execute seven contact-rich manipulation and two loco-manipulation tasks.

Method

Plan the contacts. Transfer the support.

One human video becomes an object-centric contact plan. The plan shapes the whole-body reference, rewards contact during training, and generates references at new object placements.

ReContact overview: (a) object motion from video, (b) review and contact-constrained retargeting, (c) the object-centric contact plan, (d) residual control with contact rewards, (e) a task family from one reference
Supplementary video

From human video to a real humanoid

Unitree G1

Real robot and simulation

Regrasp, carry, regrasp, place on the floor

2×

Side-clamp pickup, thigh-supported regrasp, turn and loaded walk, reverse regrasp and floor placement.

Forward and reverse thigh-supported regrasps

Box B · 0.7 kg · 1.5×
Box A · 0.88 kg · 1.5×
Box B · 1.0 kg · 1.5×
Box B · 1.4 kg · 1.5×
Box B · 1.5 kg · 1.5×
Box B · 1.9 kg · 1.5×

Six boxes from 0.7 to 1.9 kg.

Walk-then-grasp

Simulation
Real robot · 1.25×

Under-arm transfers and irregular-object grasps

Under-arm box carry
Tissue-pack transfer
Spray-bottle grasp
Banana grasp
From first-person video

Hand-held objects, whole-body references

Hand-held grasps filmed from the first-person view become whole-body G1 references.

First-person video
RH56F1 hand-only retarget
Whole-body reference

The hand is retargeted to a floating RH56F1 hand, rendered from the calibrated source camera in sync with the video; a planned whole-body approach brings the 23-DoF G1 to the grasp.

From third-person video

Same inputs, four retargeters

Every method receives the same body and object track and runs on G1 29-DoF.

Human video
ReContact
OmniRetarget
UMR + contact map
HOI-Retarget
Task families

One tube transfer, 247 references

Kinematic referencesKinematic reference
Nine of the 247 references.The reference for the position selected on the map. Drag to turn the view.
Tube-transfer success over the generated workspace and the exterior band, with coverage maps comparing single-reference and family policies
Transfer success over the generated workspace: ReContact family policy (top) and single-reference versus family policies (bottom). White dots: the 247 references; click one to play it. Star: the source position.

Real robot

Trial 1
Trial 2
Trial 3
Trial 4
Trial 5
Trial 6
Trial 7
Trial 8

One family policy. The tube position comes from the head camera's RGB image, and the nearest family reference runs.