Ben Yang, smiling, holds a bouquet of flowers wrapped in paper.

Ben Yang

PhD candidate in Computer Science at Columbia University, in the Computer Graphics and User Interfaces Lab, advised by Steven Feiner.

I build XR interfaces for robot teleoperation and human-robot collaboration, and I use 3D Gaussian splatting to bring physical objects into the headset. In 2026 I was a research intern on the Robotics team at Toyota Research Institute. I am looking for a research internship for Summer 2027, and I expect to finish my PhD in February 2028.

Research

Telling a Robot What to Build

Many assembly tasks only need parts placed relative to each other, not at fixed spots in the room. I design XR interfaces that let a person specify just those constraints and leave the rest to a robot, starting from relative 6DoF pose specification for task guidance [6]. Delegate the Details pairs such an interface with a robot planner and was tested in a 20-participant study and offline evaluation with a simulated UR5 robot [2]; a bimanual technique lets an operator nest constraint groups into assembly hierarchies for teleoperation [5].

VR view of pipe parts staged on a wooden pallet. A gray pipe and a brass flanged pipe, joined by a solid blue line, share one glowing orange hull, and a T-pipe with a red valve wheel sits in a second hull at right. Dashed orange lines run from the pallet to the gray pipe, the T-pipe and a small valve, and two controllers point at the gray pipe and the T-pipe.
Fig. 1. Parts staged on a pallet. Solid blue links mark poses the author prescribes, and dashed orange links mark poses delegated to the robot. Making the flanged pipe a prescribed child of the delegated gray pipe extends the gray pipe’s bounding hull around it [2].

Bringing Physical Objects into the Headset

With a Meta Quest 3, our pipeline uses feed-forward 3D Gaussian splatting to digitize a real object in two to four seconds, with no per-scene training. The result is a proxy that can be grabbed, inspected, and used to select the physical object it came from [4]. An earlier study found that proxies kept in the scene after use were more efficient than temporary ones for hard reselection, and participants preferred them [7].

Headset passthrough view of a lab desk. Proxies of a spray bottle, a bag and a laptop float in wireframe boxes. A controller ray points at the bag proxy, whose box turns yellow, and a yellow line runs from it to the real bag, which is outlined by a yellow box labeled Handbag/Satchel.
Fig. 2. Gaussian-splatted proxies float above the desk. Pointing at one draws a line to the physical object it came from, here a bag, and outlines it [4].

Toolkits for Building XR Systems

XR applications wire many scene objects together with intricate logic, which makes them hard to build and debug. Our XR GUI for the MercuryMessaging toolkit lets developers view and edit the messages and relationships between scene entities from inside the headset [9].

Headset view of a city street crossing. Traffic lights are outlined in orange nearby and in bluish green farther away, red arrowheads run along the links between them, and blue arrowheads mark messages in transit.
Fig. 3. Message routes in a street scene, drawn in the headset by the XR GUI. Outlines show how far a message has propagated, red arrowheads trace the links, and blue arrowheads are messages in transit [9].

Software

MercuryMessaging is a network routing and event-handling library for Unity. I released version 2.0, which adds networking APIs and speeds up VR app and user study development up to two-fold.

I made a Linux/GCC port of Meta’s CLOD-3DGS, a continuous level-of-detail renderer for 3D Gaussian splatting from Eurographics 2025.

Publications

  1. 2026

    [1]Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidence

    Z. Li, X. He, H. Chen, C. Zou, S. Bai, B. Yang, M. Wu, J. Ledner, Y.-J. Cheng, A. Yamauchi, D. G. Turakhia, S. Feiner, P. Sajda

    ACM UIST 2026, to appear

  2. [2]Delegate the Details: Flexible Assembly Constraint Authoring for Collaborative XR

    Ben Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, Steven Feiner

    IEEE ISMAR 2026

  3. [3]SwEYEpinch and Beyond: Exploring Intuitive, Efficient Text Entry for Extended Reality via Eye and Hand Tracking

    Ziheng Li, Xichen He, Mengyuan Wu, Zeyi Tong, Haowen Wei, Ben Yang, Steven Feiner, Paul Sajda

    ACM CHI 2026

  4. [4]Fast Digitization in XR of Reality-Linked Gaussian-Splatted Proxies

    Ben Yang, Shanying Liu, Kirthana Natarajan, Steven Feiner

    IEEE VR 2026 Workshops (VRW), research demo. Best Research Demo Award (Honorable Mention).

  5. [5]Bimanual XR Specification of Relative and Absolute Assembly Hierarchies for Teleoperation

    Ben Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, Steven Feiner

    IEEE VR 2026 Workshops (VRW), poster

  6. 2025

    [6]Virtual Reality Task Guidance Through Relative 6DoF Pose Specification

    Ben Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, Steven Feiner

    IEEE ISMAR 2025 Adjunct

  7. [7]Permanent Proxies for Bimanual Selection and Manipulation in VR

    Ben Yang, Jen-Shuo Liu, Xichen He, Barbara Tversky, Steven Feiner

    IEEE VR 2025 Workshops (VRW), poster

  8. 2024

    [8]VHard: An XR UI for Kinesthetic Rehearsal of Rock Climbing Moves

    Joel Salzman, Jace Li, Ben Yang, Steven Feiner

    IEEE ISMAR 2024 Adjunct, demo

  9. [9]An XR GUI for Visualizing Messages in ECS Architectures

    Ben Yang, Xichen He, Jace Li, Carmine Elvezio, Steven Feiner

    IEEE ISMAR 2024 Adjunct, demo

  10. 2023

    [10]Augmented Reality and Virtual Reality for Ice-Sheet Data Analysis

    Alexandra Boghosian, S. Isabel Cordero, Carmine Elvezio, Sofia Sanchez-Zarate, Ben Yang, Shengyue Guo, Qazi Ashikin, Joel Salzman, Kirsty Tinto, Steven Feiner, Robin Bell

    IEEE IGARSS 2023

Preprints

  1. 2026

    [11]PointGrade: Geometric Priors for Grading MoonBoard Problems

    Beatrice Stotz, Ningna Wang, Daria Nogina, Caroline Zhang, Jiyang Yin, Amy Huang, Ben Yang, Jace Li, Joel Salzman, Steven Feiner, Silvia Sellán

    arXiv preprint arXiv:2609.17770

Under Review

  1. First-author paper under review at IEEE ICRA 2027 (double-blind); decision expected January 2027.
  2. First-author paper under review at ACM CHI 2027 (double-blind); decision expected December 2026.
  3. First-author paper under review at IEEE VR 2027 (double-blind); decision expected November 2026.

News

  1. Co-authored Augmenting Human Performance with an XR Agent, to appear at ACM UIST 2026 in Detroit, MI.
  2. Presented Delegate the Details at IEEE ISMAR 2026 in Bari, Italy.
  3. Started a research internship on the Robotics team at Toyota Research Institute in Cambridge, MA.
  4. Co-authored SwEYEpinch and Beyond, on XR text entry, published at ACM CHI 2026 in Barcelona, Spain.
  5. Fast Digitization in XR received the Best Research Demo Award (Honorable Mention) at IEEE VR 2026 in Daegu, South Korea.
  6. Presented a research demo on Gaussian-splatted proxies and a poster on bimanual teleoperation at IEEE VR 2026 in Daegu, South Korea.
  7. Presented Virtual Reality Task Guidance at IEEE ISMAR 2025 in Daejeon, South Korea.
  8. Received the Columbia–Dream Sports AI Grant.
  9. Started as head TA for COMS 6173: VR and AR Seminar.
  10. Presented Permanent Proxies for Bimanual Selection and Manipulation in VR at IEEE VR 2025 in Saint-Malo, France.
  11. Received the EGSC Professional Development Scholarship.
  12. Presented two demos at IEEE ISMAR 2024 in Bellevue, WA: an XR GUI for visualizing ECS messages and VHard, for rehearsing climbing moves.
  13. Started as head TA for COMS 4172: 3D User Interfaces and Augmented Reality.
  14. Started my PhD in Computer Science at Columbia University, advised by Steven Feiner in the CGUI Lab.

Awards

  1. Best Research Demo Award (Honorable Mention), IEEE VR 2026
  2. Columbia–Dream Sports AI Grant
  3. EGSC Professional Development Scholarship
  4. Greenwoods Fellowship
  5. Grand Prize, AGU Michael H. Freilich Student Visualization Competition

Talks

  1. Paper talk, IEEE ISMAR 2026, Bari, Italy: Delegate the Details
  2. Research demo and poster, IEEE VR 2026, Daegu, South Korea: Fast Digitization in XR and Bimanual XR Specification
  3. Presentation, IEEE ISMAR 2025, Daejeon, South Korea: Virtual Reality Task Guidance
  4. Poster, IEEE VR 2025, Saint-Malo, France: Permanent Proxies
  5. Demos, IEEE ISMAR 2024, Bellevue, WA: An XR GUI for Visualizing Messages in ECS Architectures and VHard

Teaching & Service

  1. Fall 2025Head TA, COMS 6173: VR and AR Seminar
  2. Spring 2024Head TA, COMS 4172: 3D User Interfaces and Augmented Reality
  3. 2021TA, COMS 4771: Machine Learning, Summer and Fall
  4. ReviewingACM CHI (2024, 2025, 2027), ACM VRST (2024), IEEE ISMAR (2024)

Miscellany

When not working on research, I like to ski, climb, and play volleyball. I also enjoy fantasy, sci-fi, and literary fiction. Some of my favorites are Piranesi (Susanna Clarke), The Last Samurai (Helen DeWitt), and The Mountain in the Sea (Ray Nayler).

About

Ben Yang is a PhD candidate in Computer Science at Columbia University, where he works with Steven Feiner in the Computer Graphics and User Interfaces Lab and is part of Columbia HCI. His research is on XR interfaces for robot teleoperation and human-robot collaboration, and on 3D Gaussian splatting for bringing physical objects into XR. In 2026 he was a research intern on the Robotics team at Toyota Research Institute. He received an M.S. in Computer Science from Columbia University in 2022, working with Junfeng Yang, and a B.S. in Computer Science, Mathematics, and Cognitive Science from Rutgers University in 2020.