Ben Yang

Fast Digitization in XR of Reality-Linked Gaussian-Splatted Proxies

Ben Yang, Shanying Liu, Kirthana Natarajan, Steven Feiner

IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2026. Research demo.

Best Research Demo Award (Honorable Mention).

Headset passthrough view of a lab desk. Gaussian-splatted proxies of a spray bottle, a bag and a laptop float in pink wireframe boxes above the real bag, bottles and laptop on the desk.
Fig. 1. Proxies of a spray bottle, a bag and a laptop, reconstructed by feed-forward 3D Gaussian splatting, float in boxes above the desk. Each can be grabbed, inspected and used to select the physical object it came from.

Abstract

Capturing real objects as interactive 3D assets in XR typically requires offline photogrammetry. We present Gaussian-Splatted Proxies, a system for fast object digitization in XR using feed-forward 3D Gaussian splatting. With a Quest 3 headset, users select objects detected by the system through the passthrough cameras, then capture images that an edge server reconstructs as interactive Gaussian splats. Our system extracts individual objects from scene reconstructions without any per-scene training by projecting detection bounding boxes into 3D space. The resulting assets can be grabbed, inspected, rotated, and used to select the corresponding objects in the physical world.

Video

Still from the video: Lego assemblies on a desk, seen through the headset. Opens on YouTube.
Video 1. Live capture and use of proxies on a Quest 3, with no cuts. 1 min 46 s; opens on YouTube.

Cite

B. Yang, S. Liu, K. Natarajan, and S. Feiner, “Fast Digitization in XR of Reality-Linked Gaussian-Splatted Proxies,” in Proc. IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2026, pp. 1429–1430, doi: 10.1109/VRW70859.2026.00389.

@inproceedings{yang2026fastdigitization,
  title={Fast Digitization in XR of Reality-Linked Gaussian-Splatted Proxies},
  author={Yang, Ben and Liu, Shanying and Natarajan, Kirthana and Feiner, Steven},
  booktitle={2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
  pages={1429--1430},
  year={2026},
  doi={10.1109/VRW70859.2026.00389}
}