Feed-forward models make multi-view 3D reconstruction fast and robust, but their predictions often remain locally inaccurate and geometrically inconsistent across views. Structure-from-motion (SfM) offers a complementary geometric signal through camera poses and 3D points estimated by correspondence filtering and optimization. To propagate this globally consistent SfM scaffold into dense geometry, we introduce Scaffold3D, an SfM-conditioned reconstruction framework that injects image patch-aligned SfM point tokens into a pairwise feed-forward pointmap predictor. Our approach preserves image-token reasoning while using explicit SfM structure to condition dense prediction. Because Scaffold3D predicts pairwise pointmaps in local frames, we fuse them with global alignment and an SfM anchoring term. Scaffold3D achieves stronger overall performance than feed-forward reconstruction models, point-token propagation, depth-completion baselines, and recent geometry-conditioned models on standard benchmarks (ScanNet++, ETH3D, and Tanks&Temples) and out-of-domain MV-dVRK, and is competitive on 4D-DRESS. On standard benchmarks, gains over point-token propagation are especially clear in low- and no-overlap regions. Our pipeline, built around a two-view SfM-conditioned pointmap predictor, outperforms several off-the-shelf multi-view geometry-conditioned baselines. Together, these results establish SfM-scaffold conditioning as a practical interface between learned pointmap prediction and classical geometric optimization.
Scaffold3D is a two-view SfM-conditioned pointmap predictor with a lightweight point encoder. For each image patch, all SfM 3D points whose projections fall into that patch are encoded together with their within-patch offsets, aggregated into a patch-aligned point token, refined by self-attention, and added to the image tokens before the first decoder layer. This preserves dense visual reasoning while letting the SfM scaffold guide predictions beyond directly triangulated support.
We first build an SfM scaffold from the input views. Dense correspondences are matched and filtered, bundle adjustment refines cameras and 3D points, and the surviving correspondences are triangulated into image-aligned 3D points.
For each view pair, Scaffold3D predicts complete pointmaps from RGB images and their SfM scaffolds. SfM observations in the same image patch are encoded as a patch-aligned point token and added to the image-token decoder, so dense visual reasoning stays active while the scaffold guides geometry beyond directly triangulated regions.
Fixed-camera dense global alignment fuses the pairwise pointmaps into one multi-view reconstruction. Pairwise terms complete dense geometry, and SfM anchors constrain pixels with triangulated scaffold support.
The paper evaluates Scaffold3D against feed-forward predictors (DUSt3R, MASt3R, MASt3R-SfM, VGGT, Pi3, Pi3X, Pow3R, and MapAnything), monocular depth-completion methods (OMNI-DC and Marigold-DC), point-token propagation (GGPT), and geometry-conditioned predictors (MVSAnywhere, MapAnything, Pi3X, and Pow3R). In the interactive viewer below, any evaluation scene from any dataset can be selected to compare full point clouds from a compact set of representative methods. Geometry-conditioned predictors receive the same SfM scaffold as Scaffold3D for each scene.
Scaffold3D uses the SfM scaffold as reliable but incomplete geometric guidance. Conditioning improves our backbone across the five main benchmarks. On challenging hand-object scenes with textureless surfaces and extreme perspective changes, we observed matching errors along epipolar lines that survive filtering and triangulate to incorrect depths. Conditioning still improves our backbone in these cases, but strong feed-forward priors such as Pi3 remain ahead. Another limitation is the released checkpoint's sensitivity to input resolution. We recommend using it at the evaluated resolution of 512×384 pixels, as development experiments showed reduced accuracy at input resolutions not seen during training.
This study was conducted within the national “Proficiency” research project (No. PFFS-21-19), funded by the Swiss Innovation Agency Innosuisse in 2021 as one of 15 flagship initiatives. This work was supported by the Swiss AI Initiative, with computational resources provided by the Swiss National Supercomputing Centre (CSCS) on the Alps infrastructure. This work was also supported by the ETH Future Computing Laboratory. Yutong Chen is supported by a Google PhD Fellowship. We thank Luigi Piccinelli, Yiming Wang, Sergey Prokudin, Nikita Karaev, and Xudong Jiang for valuable discussions and support.
@inproceedings{rajic2026scaffold3d,
title = {Scaffold3D: SfM-Conditioned Pointmap Prediction for Multi-View 3D Reconstruction},
author = {Raji{\v{c}}, Frano and Chen, Yutong and Xu, Haofei and Pataki, Zador and Pollefeys, Marc and Tang, Siyu},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}