ACE ROBOTICS, in collaboration with the Chinese University of Hong Kong’s Multimedia Laboratory and Nanyang Technological University, has open-sourced ACE-Ego-Hand, a 3D dual-hand shape reconstruction model designed for first-person video, generating approximately 5,000 hours of training data that could serve as a foundational dataset for embodied AI model training.
The model extracts structured 3D bimanual manipulation trajectories from massive egocentric video footage, translating human grasping, assembly and hand-coordination actions into machine-learnable data. It marks the first time a video generation model has been repurposed as a DreamHand encoder for end-to-end training, outputting hand pose, shape and 3D position in a single forward pass without frame-by-frame detection or temporal regression.
Efficiency gains over traditional approaches
Unlike previous hand models reliant on per-frame detection, window-based temporal regression or multi-step diffusion sampling, ACE-Ego-Hand converts the video diffusion model into a deterministic geometric encoder, recovering continuous 3D hand trajectories in one feed-forward pass. This reduces inference latency and error accumulation while enabling the model to leverage contextual information from full video clips to restore occluded or temporarily out-of-view hand trajectories.
On a single A100 GPU, ACE-Ego-Hand achieves approximately 63.1 frames per second – roughly 33 times faster than the high-precision configuration of ViDiHand, according to the research team.
Benchmark dominance
The model was evaluated on five first-person hand benchmarks – ARCTIC, HOT3D, HOI4D, H2O and OakInk2 – winning 45 out of 48 reported metrics. On the heavily occluded ARCTIC dataset, ACE-Ego-Hand reduced MPJPE-p by approximately 30% compared with ViDiHand. On the complex first-person and wide-angle HOT3D dataset, MPJPE-p dropped by about 40%, with performance advantages widening to 45.9%–61.1% when evaluating fully out-of-view hands.
On HOT3D, ACE-Ego-Hand recorded MPJPE-p of 12.888 mm against ViDiHand’s 21.514 mm. On OakInk2, the model achieved MPJPE-p of 8.988 mm, outperforming the previous best (WiLoR at 26.520 mm) by roughly 66%.
Occlusion recovery and metric 3D positioning
A key innovation lies in bidirectional spatiotemporal reasoning. The model analyses entire video clips without causal masking, using information from both before and after hand occlusion to infer missing intermediate segments – maintaining identity and trajectory continuity. A clip-level shape prior prevents scale flickering and identity mismatch upon reappearance.
The model also recovers metric 3D hand positions in camera coordinates without explicit camera intrinsic input, supporting native fisheye ultra-wide-angle input. It estimates observation rays from distorted images to complete 3D positioning, effectively extending from 2D localisation to metric 3D motion recovery. A “K-free” configuration allows the model to solve hand spatial positions using its own predicted line-of-sight field, reducing reliance on fixed camera calibration.
Industrial deployment pathway
ACE-Ego-Hand enables a complete pipeline from egocentric video to dexterous hand motion transfer, mapping finger flexion, joint motion and bimanual coordination to robot hands. This lowers the barrier for collecting robotic manipulation data: companies can record real production-line operations with head-mounted cameras and batch-convert footage into 3D action data for robot teaching and imitation learning.
The open-source release provides a scalable data foundation for future embodied large models, transforming massive real-world human manipulation footage into structured bimanual motion data.
