Holistic 4D Reconstruction as
Geometric Control for Video Diffusion
NeurIPS 2026
One source video. Persistent 3D tracks. New viewpoints.
Abstract
A 4D scaffold to tame video generation.
Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content. We introduce Holo4D, a video reshooting framework that uses reconstructed 3D trajectories as a holistic source-view 4D scaffold. We project these trajectories into the requested views and encode their geometry and source appearance to guide a pretrained video diffusion model. Together with the source video and target cameras, the scaffold grounds generation in reconstructed observations while leaving unsupported regions to the generative prior. Experiments demonstrate the value of trajectory-based conditioning for camera control and novel-view synthesis. The framework accommodates different 4D reconstructors without retraining the video generator, allowing it to benefit from future advances in source reconstruction.
01 / Robotics
Robotics from new viewpoints
One source scene, two camera paths. Watch the result, then reveal the tracks.
Humanoid robots
Robot manipulation
About the tracks and video loading
Holo4D receives source video, camera rays and dense persistent 3D-track features carrying position, visibility, source-query RGB and target-camera depth—not rendered target RGB-D. The source-condition panel visualizes the input; it is not target ground-truth RGB. Post-generation overlays, where labelled, are display tracks estimated on generated RGB, not the model input. Use the view controls to inspect clean video or tracks.
02 / Video reshooting
The same moment. A different perspective.
One source scene, two camera paths. Selected DAVIS and Vista4D examples share this gallery; each preview interleaves clean video and 3D-track overlays. Switch views or scrub through time to inspect the result.
Geometry and visualization details
All short examples use the step-19,500 model. Native-trajectory examples use VGGT-Omega + CoWTracker geometry; DAVIS entries retain their labelled 4RC input. Desktop pairs share the exact source clip and show two different camera paths. Presentation-only jitter rejection removes unreliable displayed identities; it does not change generated RGB or model conditions. Only visible examples load MP4s; the rest show lightweight first-frame posters. Playback waits for both visible videos to buffer. The short previews use a forward/reverse presentation loop, not a longer generated sequence.
Media refresh in progress: the players below still use the previous exports until the filtered single-video batch completes. Original inference videos are unchanged.
03 / Long videos
Beyond a single video chunk
Selected humanoid and manipulation demonstrations, continued beyond 49 frames: one source scene, two camera paths. Source and generated videos play together; a quiet chunk label identifies each continuation.
How are the continuations generated?
A T2V seed is continued by the 14B I2V model, conditioned on the previous generated last frame. Overlaps are deduplicated and padded tails removed. These are full-duration qualitative demonstrations, not a quantitative long-horizon motion-preservation claim.
04 / Baselines
Comparison with baselines
Two methods, the same requested native camera trajectory. Browse representative cases—including baseline failures—and inspect synchronized outputs.
Comparison protocol and measured speed
Holo4D uses VGGT-Omega + CoWTracker source geometry; Vista4D uses its official pipeline. Source geometry backends differ, and these examples do not imply an appearance win on every case. Camera diagnostics below are not appearance scores.
Separate matched H200 timing: Holo4D 7.39 s/denoising step versus Vista4D 9.89–9.90 s (about 25% less denoising time). Two native cases, 49 frames, 384×672, 50 steps. Excludes reconstruction/rendering and model loading; not an end-to-end or training speed claim.
How it works
Persistent tracks, generative completion.
Source reconstruction supplies persistent 3D tracks. Their features are projected into the requested camera and injected into video diffusion, alongside source video and camera-ray features.
Takeaway
Reconstruct what was seen; generate what was not.
A 4D scaffold of persistent 3D trajectories grounds video generation in what the source video observed, and leaves only the unseen to the generative prior.
Source geometry keeps the camera on track. On DAVIS, Holo4D has the lowest trajectory and translation errors of all methods; with the same 4RC reconstruction it reduces ATE by 54% and RPE-T by 33% compared with Vista4D. With the two-stage backend it obtains the best PSNR and LPIPS on iPhone.
Generated views keep the scene's geometry. Holo4D has the lowest AbsRel and highest δ1 under both VGGT-Ω and 4RC as output reconstructors; under VGGT-Ω, AbsRel falls from Vista4D's 0.628 to 0.423.
Swap the reconstructor, keep the generator. The same generator accepts 4RC, V-DPM and VGGT-Ω paired with Open-D4RT or CoWTracker, without retraining. The backends trade off (CoWTracker: best ATE, PSNR and SSIM; V-DPM: best rotation error; 4RC: best LPIPS), so Holo4D benefits directly from better reconstructors.
Channel-wise conditioning is lighter. Track features join the hidden features along channels, so the attention sequence length is unchanged: denoising is 1.34× faster than Vista4D at similar peak memory (end-to-end time per video is similar).
It transfers to robots. Trained without robot videos, Holo4D-14B reshoots DROID manipulation episodes and humanoid-robot clips without fine-tuning.
Limits. Holo4D remains sensitive to geometric error, incomplete source coverage and difficult camera trajectories.
Acknowledgements
We thank Runjia Li, Zeqi Xiao, Yiming Chen, Wenqi Ouyang, Zhouxia Wang and Minghao Chen for valuable feedback and discussions. This research was supported by the Sony Research Award Programme 2024, ERC CoG UNION (101001212) and EPSRC EP/Z001811/1 SYN3D. We thank the Isambard-AI and Dawn AIRR supercomputers for supporting this project.
Concurrent works
- Vista4D conditions on rendered point clouds with persistent static pixels; Holo4D makes moving points persistent too and injects them along channels, without an extra token stream.
- Lyra [1, 2] distills video diffusion into 3D Gaussians; Holo4D goes the other way, using 4D reconstruction to guide video generation.
- Atlas is a multimodal world model for views, video and 3D; Holo4D focuses on reshooting one monocular video and swaps reconstructors without retraining.
References
- Bai et al. ReCamMaster: Camera-Controlled Generative Rendering from a Single Video. ICCV, 2025.
- Hu et al. EX-4D: Extreme Viewpoint 4D Video Synthesis via Depth Watertight Mesh. arXiv, 2025.
- Khazatsky et al. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset. RSS, 2024.
- Lai et al. CoWTracker: Tracking by Warping instead of Correlation. CVPR, 2026.
- Lin et al. Vista4D: Video Reshooting with 4D Point Clouds. CVPR, 2026.
- Luo et al. 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere. ICML, 2026.
- Pont-Tuset et al. The 2017 DAVIS Challenge on Video Object Segmentation. arXiv, 2017.
- Ren et al. GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control. CVPR, 2025.
- Sucar et al. V-DPM: Video Reconstruction with Dynamic Point Maps. CVPR, 2026.
- Wan Team. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv, 2025.
- Wang et al. VGGT-Ω. CVPR, 2026.
- Yu et al. TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models. ICCV, 2025.
- Zhang et al. Efficiently Reconstructing Dynamic Scenes One D4RT at a Time. CVPR, 2026.
BibTeX
@inproceedings{lan2026holo4d,
title = {{Holo4D}: Holistic 4D Reconstruction as Geometric
Control for Video Diffusion},
author = {Lan, Yushi and Jiang, Zeren and Namekata, Koichi and
Pan, Xingang and Zheng, Chuanxia and Vedaldi, Andrea},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}