Self-verifiable reinforcement learning

WorldCycle

Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

Turn reversible action cycles into dense, annotation-free supervision for world models that remain consistent over long horizons.

Bohai Gu1,* Yueyang Yuan2,* Taiyi Wu3 Dazhao Du1 Jian Liu1 Xiaoyi Pang1 Jie Zhang1 Xiaocheng Lu1 Haobin Zhong3 Xiaotong Zhao3 Alan Zhao3 Song Guo1,†
* Equal contribution† Corresponding author
1The Hong Kong University of Science and Technology 2Wuhan University 3AI Technology Center, Tencent Video, Tencent
closed loop state0 returns verified
Wforward Dturn Sinverse Aclose t0 tk
44%lower state-return
drift, at best
4×composite-action
accuracy vs. base
4CycleBench
task families
0ground-truth future
videos required
01 / Overview

A verification signal hidden inside the action.

Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Reinforcement learning can improve these models, but it faces a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift.

WorldCycle exploits a structural exception. A reversible action sequence composed with its inverse must analytically return to the initial state. This turns closed action cycles into annotation-free supervision on long-horizon correctness.

The framework constructs closed cycles and repeated executions from ordinary action sequences, then optimizes a spatial closure reward between mirrored forward and reverse states and a temporal consistency reward across repeated cycles.

Comparison of WorldPlay, WorldCompass, and WorldCycle across two 253-frame action cycles. WorldCycle returns more closely to the same visual state.
Figure 1 State-returning failure in interactive video world models. Spatial closure and temporal consistency errors accumulate as autoregressive rollouts extend.
02 / Method

Close the loop.
Then learn from it.

WorldCycle turns analytically known action structure into trajectory-level rewards, making long-horizon post-training verifiable without future video labels.

WorldCycle method overview showing cycle action inputs, spatial and temporal cycle rewards, and DiffusionNFT optimization.
Method overview Mirrored spatial states and phase-aligned temporal states provide complementary, trajectory-level rewards.
Rspatial

Spatial closure

Every partial forward trajectory has a corresponding reverse state that should coincide with it. Dense mirrored comparisons localize drift instead of relying on one sparse endpoint.

Rtemporal

Temporal consistency

Repeated cycles should revisit the same phases. Comparing later executions to the first cycle penalizes position-dependent action drift over extended rollouts.

03 / CycleBench

Evaluate the model
as a simulator.

CycleBench diagnoses how transition errors accumulate when actions are reversible, repeated, composed, or cascaded.

T1

Reversible-Cycle

Follow an action sequence, execute its inverse, and measure forward–reverse symmetry.

T2

Closed-Cycle

Trace a non-retracing loop and evaluate whether the endpoint returns to the initial state.

T3

Repeated-Cycle

Repeat one closed program and track phase-aligned drift across executions.

T4

Cascaded-Cycle

Compose different closed cycles and test whether residuals propagate into later actions.

47

action trajectories

380

initial frames

4

scenario settings

ESC↓RPS↓RCS↓

state-consistency metrics

CycleBench overview with four benchmark tasks, metrics, and short-, mid-, long-term and composite-action settings.
CycleBench Four task families probe endpoint closure, reverse-path symmetry, repeated-cycle stability, and cross-cycle error propagation.
04 / Comparisons

Watch consistency
survive the horizon.

Each synchronized group compares the WorldPlay base model, WorldCompass post-training, and WorldCycle under the same action program.

Videos are synchronized to the same cycle.

W + →then inverseS + ←
WorldPlayBase model
WorldCompassPost-trained
WorldCycleOurs
W · A · S · Drepeated across 5 cycles
WorldPlayBase model
WorldCompassPost-trained
WorldCycleOurs
W · A · S · Dsingle closed cycle
WorldPlayBase model
WorldCompassPost-trained
WorldCycleOurs
05 / Results

Lower drift.
Stronger control.

WorldCycle improves state-returning consistency across horizons and generalizes to composite actions without sacrificing visual quality.

Short-term ESC−32%vs. WorldCompass
Short-term RPS−44%vs. WorldCompass
Long-term RCS−34%vs. WorldCompass
Composite accuracy0.5534× the base model
WorldCycle on CycleBench — weighted average over four benchmark tasks
SettingESC ↓RPS ↓RCS ↓Accuracy ↑HPSv3 ↑
Short-term 125 frames0.0260.0190.0200.83311.24
Mid-term 253 frames0.0380.0280.0180.87810.81
Composite 125 frames0.0570.0420.0260.55310.42
Long-term 381 frames0.1630.0870.0490.09510.31
Qualitative 253-frame comparison showing that WorldCycle better preserves scene colors, geometry, and structural details after returning.
Qualitative comparison After 253 frames, WorldCycle recovers states that remain closer to the initial scene under both composite and simple actions.
Training curves showing improved HPSv3 quality and decreasing RPS, ESC, and RCS scores.
Training dynamics Cycle consistency improves while visual quality rises above the pre-trained baseline.
Consistency without collapse

Better state dynamics do not trade away visual quality.

The full model reaches an HPSv3 score of 10.42 on composite actions, a 16% gain over the 8.98 base-model score reported in the ablation.

Base8.98 WorldCycle10.42
06 / Citation

Build on the
closed loop.

Cite the arXiv preprint (arXiv:2608.04964).

BibTeX
@misc{gu2026worldcycleselfverifiablereinforcementlearning,
      title={WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models},
      author={Bohai Gu and Yueyang Yuan and Taiyi Wu and Dazhao Du and Jian Liu and Xiaoyi Pang and Jie Zhang and Xiaocheng Lu and Haobin Zhong and Xiaotong Zhao and Alan Zhao and Song Guo},
      year={2026},
      eprint={2608.04964},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.04964},
}