Holistic 4D Reconstruction as
Geometric Control for Video Diffusion
NeurIPS 2026
One source video. Persistent 3D tracks. New viewpoints.
Overview
A 4D scaffold to tame video generation.
Holo4D re-films a video from a new camera path. A feed-forward 4D reconstructor turns the source video into persistent 3D tracks: points that keep their identity as they move. We project these tracks into the requested camera and give them to a video diffusion model, together with the source video and the target camera. Where the source saw the scene, the tracks hold the result to the reconstructed geometry and motion; where the new camera sees what the source never showed, the diffusion prior fills it in. The same trained model works with different 4D reconstructors without retraining, so it improves as reconstruction improves. On DAVIS, with the same 4D reconstruction, Holo4D reduces camera trajectory error (ATE) by 54% compared with Vista4D. Below: humanoid and arm robots, dynamic everyday scenes, long videos generated window by window, and side-by-side comparisons.
01 / Robotics
Robotics from new viewpoints
One source scene, two camera paths. Watch the result, then reveal the tracks.
Humanoid robots
Robot manipulation
About the tracks and video loading
Holo4D receives source video, camera rays and dense persistent 3D-track features carrying position, visibility, source-query RGB and target-camera depth—not rendered target RGB-D. The source-condition panel visualizes the input; it is not target ground-truth RGB. Post-generation overlays, where labelled, are display tracks estimated on generated RGB, not the model input. Use the view controls to inspect clean video or tracks.
02 / Video reshooting
The same moment. A different perspective.
One source scene, two camera paths. Selected DAVIS and Vista4D examples share this gallery; each preview interleaves clean video and 3D-track overlays. Switch views or scrub through time to inspect the result.
Geometry and visualization details
All short examples use the step-19,500 model. Native-trajectory examples use VGGT-Omega + CoWTracker geometry; DAVIS entries retain their labelled 4RC input. Desktop pairs share the exact source clip and show two different camera paths. Presentation-only jitter rejection removes unreliable displayed identities; it does not change generated RGB or model conditions. Only visible examples load MP4s; the rest show lightweight first-frame posters. Playback waits for both visible videos to buffer. The short previews use a forward/reverse presentation loop, not a longer generated sequence.
Media refresh in progress: the players below still use the previous exports until the filtered single-video batch completes. Original inference videos are unchanged.
03 / Long videos
Beyond a single video chunk
Selected humanoid and manipulation demonstrations, continued beyond 49 frames: one source scene, two camera paths. Source and generated videos play together; a quiet chunk label identifies each continuation.
How are the continuations generated?
A T2V seed is continued by the 14B I2V model, conditioned on the previous generated last frame. Overlaps are deduplicated and padded tails removed. These are full-duration qualitative demonstrations, not a quantitative long-horizon motion-preservation claim.
04 / Baselines
Comparison with baselines
Two methods, the same requested native camera trajectory. Browse representative cases—including baseline failures—and inspect synchronized outputs.
Comparison protocol and measured speed
Holo4D uses VGGT-Omega + CoWTracker source geometry; Vista4D uses its official pipeline. Source geometry backends differ, and these examples do not imply an appearance win on every case. Camera diagnostics below are not appearance scores.
Separate matched H200 timing: Holo4D 7.39 s/denoising step versus Vista4D 9.89–9.90 s (about 25% less denoising time). Two native cases, 49 frames, 384×672, 50 steps. Excludes reconstruction/rendering and model loading; not an end-to-end or training speed claim.
How it works
Persistent tracks, generative completion.
Source reconstruction supplies persistent 3D tracks. Their features are projected into the requested camera and injected into video diffusion, alongside source video and camera-ray features.
Behind the scenes
Five stages, from the input video to the output. Select a stage to read what happens there and to outline it on the figure above.
Input video. One monocular source video and the requested target camera path. Nothing else about the scene is given.
4D reconstruction (4RC, or VGGT-Ω + CoWTracker). A feed-forward 4D reconstructor recovers persistent 3D trajectories, for static and moving points alike, with colors sampled from the source. Other backends (V-DPM, Open-D4RT) plug in without retraining the generator.
Track condition. The trajectories are projected into the target cameras and encoded as track features T: position, visibility, source-query RGB and target-camera depth—not a rendered target RGB-D image.
Video diffusion (Wan, 14B). Source and noisy target latents are concatenated along width. Each DiT block concatenates T with its hidden features along channels and adds camera features C from Plücker rays, so the attention sequence length is unchanged.
Output. The video from the new camera path: observed content follows the 4D scaffold, newly visible regions come from the generative prior. Track overlays on outputs are for visualization only.
Takeaway
Reconstruct what was seen; generate what was not.
A 4D scaffold of persistent 3D trajectories grounds video generation in what the source video observed, and leaves only the unseen to the generative prior.
1Source geometry keeps the camera on track. On DAVIS, Holo4D has the lowest trajectory and translation errors of all methods; with the same 4D reconstructor it reduces ATE by 54% and RPE-T by 33% compared with Vista4D. With the two-stage backend it obtains the best PSNR and LPIPS on iPhone.
2Generated views keep the scene's geometry. Holo4D has the lowest AbsRel and highest δ1 under both VGGT-Ω and 4RC as output reconstructors; under VGGT-Ω, AbsRel falls from Vista4D's 0.628 to 0.423.
3Swap the reconstructor, keep the generator. The same generator accepts 4RC, V-DPM and VGGT-Ω paired with Open-D4RT or CoWTracker, without retraining. The backends trade off (CoWTracker: best ATE, PSNR and SSIM; V-DPM: best rotation error; 4RC: best LPIPS), so Holo4D benefits directly from better reconstructors.
4Channel-wise conditioning is lighter. Track features join the hidden features along channels, so the attention sequence length is unchanged: denoising is 1.34× faster than Vista4D at similar peak memory (end-to-end time per video is similar).
Limits. Holo4D remains sensitive to geometric error, incomplete source coverage and difficult camera trajectories.
Acknowledgements
We thank Runjia Li, Zeqi Xiao, Yiming Chen, Wenqi Ouyang, Zhouxia Wang and Minghao Chen for valuable feedback and discussions. This research was supported by the Sony Research Award Programme 2024, ERC CoG UNION (101001212) and EPSRC EP/Z001811/1 SYN3D. We thank the Isambard-AI and Dawn AIRR supercomputers for supporting this project.
Concurrent works
- Vista4D conditions on rendered point clouds with persistent static pixels; Holo4D makes moving points persistent too and injects them along channels, without an extra token stream.
- Lyra [1, 2] distills video diffusion into 3D Gaussians; Holo4D goes the other way, using 4D reconstruction to guide video generation.
- Atlas is a multimodal world model for views, video and 3D; Holo4D focuses on reshooting one monocular video and swaps reconstructors without retraining.
References
- Bai et al. ReCamMaster: Camera-Controlled Generative Rendering from a Single Video. ICCV, 2025.
- Hu et al. EX-4D: Extreme Viewpoint 4D Video Synthesis via Depth Watertight Mesh. arXiv, 2025.
- Khazatsky et al. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset. RSS, 2024.
- Lai et al. CoWTracker: Tracking by Warping instead of Correlation. CVPR, 2026.
- Lin et al. Vista4D: Video Reshooting with 4D Point Clouds. CVPR, 2026.
- Luo et al. 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere. ICML, 2026.
- Namekata et al. Go-with-the-Track: Video Compositing and Motion Control with Point Tracking. SIGGRAPH, 2026.
- Pont-Tuset et al. The 2017 DAVIS Challenge on Video Object Segmentation. arXiv, 2017.
- Ren et al. GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control. CVPR, 2025.
- Sucar et al. V-DPM: Video Reconstruction with Dynamic Point Maps. CVPR, 2026.
- Wan Team. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv, 2025.
- Wang et al. VGGT-Ω. CVPR, 2026.
- Yu et al. TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models. ICCV, 2025.
- Zhang et al. Efficiently Reconstructing Dynamic Scenes One D4RT at a Time. CVPR, 2026.
BibTeX
@inproceedings{lan2026holo4d,
title = {{Holo4D}: Holistic 4D Reconstruction as Geometric
Control for Video Diffusion},
author = {Lan, Yushi and Jiang, Zeren and Namekata, Koichi and
Pan, Xingang and Zheng, Chuanxia and Vedaldi, Andrea},
booktitle = {The Fortieth Annual Conference on Neural Information
Processing Systems},
year = {2026},
url = {https://openreview.net/forum?id=Cu9FVR16z6}
}