Pros4D. Modeling Evolving 4D Worlds in Latent Space forDynamic Spatial Reasoning

Rongxin Gao1,*Dongyuan Liu1Chu Li2Chengyang Ji2Jialing Huang1Runyu Chen1Kairun Wen3Chenxin Li3
Jingrui Ye4Haosong Zhen5Xinghao Ding1,†Yue Huang1Yuzhi Huang4,†Xiaotong Tu1Jingyan Jiang4
1XMU2CUMT3CUHK4THU5HIT
* Project Lead† Corresponding Authors

A shared internal state connects visible motion to scene evolution beyond the observation boundary.

The world keeps moving.
Reasoning should, too.

Video understanding tells us what happened. Acting in a dynamic world also requires anticipating what happens next.

Pros4D brings both capabilities into one vision-language model. It compresses observed dynamics into latent observed states, then autoregressively evolves them into latent future states—preserving spatial structure while reasoning across time.

At inference, both tasks use only the same observed video prefix. No external geometric tool calls are needed during question answering.

Read the full abstract

Dynamic spatial reasoning from monocular video enables vision-language models (VLMs) to understand how 3D environments evolve over time, supporting physical world understanding and embodied intelligence. Existing efforts primarily address observed-dynamics understanding, whereas agents in dynamic environments also require future-dynamics prediction beyond the observation boundary. Moreover, external geometric processing and textual descriptions can increase inference complexity or lose information when representing continuous dynamics. These limitations motivate a shared internal state that preserves observed dynamics and remains predictive beyond the input video. Accordingly, we present Pros4D, a framework for modeling evolving 4D worlds in latent space that unifies retrospective understanding of observed-dynamics with prospective reasoning for future-dynamics prediction. Specifically, Pros4D-Engine derives geometry-grounded supervision and constructs paired examples for both tasks, while Pros4D-Bench evaluates them across eight spatial task categories spanning dynamic cameras, objects, and scenes. At the model level, Pros4D compresses observed dynamics into latent observed states and autoregressively rolls them forward into latent future states. 4D Latent Alignment (4DLA) grounds these states through complementary visual-semantic and geometric supervision, while joint language supervision connects them to task reasoning. Experiments across benchmarks show consistent improvements over strong baselines in both capabilities.

40Ktraining examples
3,000benchmark questions
8spatial task categories
2temporal perspectives

A latent world.
Two ways to reason.

Textual reasoning and continuous latent states work together, connecting visual evidence, spatial structure, and temporal evolution.

01

Encode the observed world

Compress entity-specific visual and motion evidence into ordered latent observed states.

Observed dynamics
02

Evolve beyond observation

Autoregressively roll observed states forward to model the future continuation in latent space.

Future dynamics
03

Ground the reasoning

Align states with visual-semantic and geometric targets, alongside joint language supervision.

Training-time alignment
FIG. 014D Latent Alignment. Complementary visual-semantic and geometric supervision ground the evolving states. Alignment is active only during training; both encoders remain frozen.

Geometry at training. Latent reasoning at inference.
The model answers from the observed video and question, without access to future frames or an external geometry pipeline.

From video to
geometry-grounded supervision.

Pros4D-Engine constructs paired examples for understanding and prediction. Geometry determines the answer before the language model verbalizes it.

  1. 01 Ground & track entities
  2. 02 Recover 3D geometry
  3. 03 Derive temporal labels
  4. 04 Synthesize state traces
FIG. 02Pros4D-Engine. Entity tracking and estimated 3D geometry provide spatial supervision. Verified labels and ordered state targets form the Pros4D-40K training set.

Same observation.
A broader test of understanding.

Eight spatial categories span cameras, objects, and scenes. Both temporal tasks receive an identical observed prefix, with videos separated from training by video identifier.

3,000questions · 2,496 videos
1,891 observed 1,109 future
CAMERA / 01

Camera Motion Direction

How does the observer move?

606 questions
CAMERA / 02

Camera–Object Distance

Is the camera getting closer?

335 questions
OBJECT / 03

Object Motion Direction

Which way does the target move?

469 questions
OBJECT / 04

Inter-Object Distance

How does their separation change?

155 questions
OBJECT / 05

Object Speed Change

Does the target speed up or slow down?

463 questions
OBJECT / 06

Object Travel Distance

What is the target’s endpoint displacement?

472 questions
SCENE / 07

Scene Spatial Relationship

How does an object move relative to a landmark?

248 questions
SCENE / 08

Scene Object Trajectory

Which direction does the main object travel?

252 questions
Explore the benchmark distribution
Paper figure showing the distribution of Pros4D-Bench questions across eight spatial categories
Questions by spatial and temporal task
Spatial taskObservedFuture
Camera Motion Direction503103
Camera–Object Distance200135
Object Motion Direction279190
Inter-Object Distance10550
Object Speed Change275188
Object Travel Distance279193
Scene Spatial Relationship122126
Scene Object Trajectory128124
Total1,8911,109

Better understanding.
Stronger anticipation.

Consistent gains in both temporal tasks, with transfer to dynamic spatial benchmarks, robot manipulation, and driving scenes.

Across the observation boundary

Pros4D-Bench · comparison with the corresponding backbone

Observed-dynamics understanding

+15.2 pts
Qwen3-VL-8B
49.7
Pros4D-8B
64.9

Future-dynamics prediction

+18.8 pts
Qwen3-VL-8B
40.7
Pros4D-8B
59.5
Overall Pros4D-Bench accuracy62.9%Same observed prefix for both tasks

Selected comparisons on Pros4D-Bench

Reported in the paper · accuracy (%)
ModelOverallObservedFuture
MLLM-4D-8B51.759.338.7
DynTrace54.657.549.6
Qwen2.5-VL-3B backbone45.450.936.0
Pros4D-3B62.063.958.8
Qwen3-VL-8B backbone46.449.740.7
Pros4D-8B62.964.959.5
DYN-BENCH · 8B65.6%

+4.6 pts over the backbone

VLM4D · 8B68.2%

+9.2 pts over the backbone

OPENX-VQA · 3B51.39%

+20.91 pts over the backbone

All values are from the accompanying manuscript. Gains are percentage points over the corresponding Qwen backbone. See the paper for the complete comparisons, evaluation protocols, and ablations.

Reasoning with
a world that evolves.

Qualitative examples connect the queried entity to its surroundings and to the order of events.

Retrospective understanding

How does the distance between the camera and the person sitting on the bench change from 7.5 s to 11.2 s?

FIG. 03Camera–Object Distance on Pros4D-Bench. The appendix comparison shows how Pros4D reasons about the decreasing distance between the camera and the seated person.

BibTeX

@misc{gao2026pros4d,
  title   = {Pros4D: Modeling Evolving 4D Worlds in Latent Space
             for Dynamic Spatial Reasoning},
  author  = {Rongxin Gao and Dongyuan Liu and Chu Li and
             Chengyang Ji and Jialing Huang and Runyu Chen and
             Kairun Wen and Chenxin Li and Jingrui Ye and
             Haosong Zhen and Xinghao Ding and Yue Huang and
             Yuzhi Huang and Xiaotong Tu and Jingyan Jiang},
  year    = {2026},
  note    = {Research manuscript},
  url     = {https://pros4d.github.io/}
}
Download citation (.bib)