Chinese researchers said they have developed MEgoVista, a framework that converts first-person, or ego, video into 4D motion data for training embodied AI models, addressing a gap between the growing supply of ego video and the precise motion data robots need.
The paper was produced by Maniformer Technology, Northwestern Polytechnical University and Xi’an Jiaotong University.
Ego video has become an important source of pretraining data because it can be captured in homes, workplaces and other real environments rather than specialized laboratories. But recording an action does not automatically produce usable 4D data, meaning three-dimensional motion over time. Systems must identify the wearer’s hands, reconstruct finger poses and measure how far the hands move in space.
MEgoVista is designed as an offline reconstruction pipeline. It first identifies and tracks the wearer’s hands, distinguishing left from right and separating the wearer’s hands from those of other people in the frame. In a manicure scenario, the researchers said, the system correctly distinguished the manicurist’s hands from the customer’s, while two comparison methods confused them.
The system uses a MEgoView head-mounted device equipped with three fisheye cameras shooting at 60 frames per second. The central camera is paired with the left and right cameras to create two stereo vision setups. By comparing the same hand from different viewpoints and using calibrated camera spacing and relative positions, the system estimates the distance between the hand and the cameras.
To address fisheye lens distortion and camera mounting angles, MEgoVista corrects the images, uses Foundation Stereo to estimate depth, combines the two stereo results and prioritizes higher-confidence information.
The framework then optimizes the entire motion sequence. Frame-by-frame hand poses can appear plausible individually but produce jumps or unnatural finger bending when viewed continuously. MEgoVista filters reliable observations and uses information from adjacent frames to make movement more coherent and finger poses more anatomically plausible.
It also reduces the effect of head movement. Using multi-camera images, it reconstructs the head trajectory, establishes real-world scale early in reconstruction and uses inertial information to determine up-down orientation, separating hand movement from camera movement. Hand motion and head trajectory are then placed in the same real-scale coordinate system.
For evaluation, the team used an independent optical motion capture system with 26 cameras as reference. Missed hand detections were penalized to prevent algorithms from gaining better scores by skipping difficult frames.
After aligning position, orientation and overall scale, MEgoVista recorded a finger pose error of 4.29 mm, compared with 16.30 mm for HaWoR and 49.18 mm for Dyn-HaMR. The metric reflects finger pose accuracy and does not represent the absolute position error of the whole hand in space.
MEgoVista had no missed or false hand detections in the test data, and its motion jitter was lower than the two comparison methods at 1.11 mm/frame². In two head-trajectory tests, average position errors for full clips were 2.87 mm and 0.75 mm.
The team also compared performance in three real-world scenes. In milk-tea preparation, MEgoVista and Dyn-HaMR tracked hand skeletons closely during rapid hand movement, while HaWoR showed clear misalignment in the left-hand skeleton. In manicure, only MEgoVista correctly distinguished the wearer from the customer. In pet bathing, where hands were soiled and moving quickly, MEgoVista and Dyn-HaMR detected the hands, while HaWoR showed offset in the left hand and wrist.
The paper said MEgoVista restores first-person video operations into real-scale 4D motion data and validates reconstruction accuracy through independent measurement, providing a more reliable data source for robot learning.
Maniformer Technology is pushing forward with mass production of its MEgo series devices and with capabilities for data collection, governance and evaluation. The company said the MEgoVista research provides technical support for that effort by expanding data sources while improving 3D motion reconstruction accuracy.
The paper is available at: https://arxiv.org/abs/2609.16684
the 2026 Bund Summit