Astribot, a Chinese robotics startup, has unveiled SmoothRL, an online reinforcement learning framework designed to train policies during asynchronous execution – a growing reality for embodied AI deployments where inference and action run at different frequencies. The work, led by the company’s foundation model team, addresses a fundamental mismatch: when a robot receives new action chunks before previous ones finish, conventional RL that updates on entire chunks ends up crediting or blaming actions that were never actually executed.
The framework partitions each action chunk into three regions – committed (already fixed), execution (actually performed), and discarded (overwritten by later inference) – and backpropagates gradients only through the execution region. This ensures that optimisation mirrors the robot’s real experience. SmoothRL also applies the same asynchronous timing during training rollouts, a principle the team calls “Reinforce in Deployment”. In practice, the system runs a 30 Hz control loop with 5 Hz inference, generating 32‑frame chunks where only 6 frames fall into the execution window per cycle.
On Astribot’s cable‑driven S1 humanoid, the team tested three real‑world tasks. Dynamic throwing success rose from 39% to 94% after 250 rollout episodes; pen‑capping improved from 8% to 83%; and parcel‑opening – requiring a 1‑mm blade to slice a 2‑3 mm tape seam – climbed from 30% to 90%, despite a non‑monotonic learning curve that dipped to 20% midway. Beyond raw success, failures converged closer to the correct pose, with blade deviation shrinking to 1‑2 mm. Smoothness also improved: root‑mean‑square acceleration fell 52% and jerk dropped 47% in a throwing rollout.
The team acknowledges limitations: the current residual policy relies on a frozen base policy (fine‑tuned π0.5) and cannot salvage grossly incorrect behaviours, while the approach assumes predictable latency budgets. Future work will explore broader policy updates and end‑to‑end optimisation with generative models. SmoothRL continues Astribot’s long‑standing philosophy that action should be a “first‑class modality” in robot intelligence, rather than an afterthought of vision or language. The company is building a full stack – from agent and foundation models to RL post‑training – alongside its proprietary cable‑driven hardware and embodied OS, aiming to accelerate Physical AI deployment in real‑world settings.
