Ben Yang

Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidence

Z. Li, X. He, H. Chen, C. Zou, S. Bai, B. Yang, M. Wu, J. Ledner, Y.-J. Cheng, A. Yamauchi, D. G. Turakhia, S. Feiner, P. Sajda

ACM Symposium on User Interface Software and Technology (UIST), 2026. To appear.

Two panels of a diagram. Left, labeled b: evidence-quality sliders feed an online EM loop between a foundational VLM and a virtual prompt, above a grid of drone images marked red or blue. Right, labeled c: a person in a headset aims a controller among floating drones while red outlines and red curved lines mark targets, under a box that reads Agent provides task cues.
Fig. 1. (b) Weighted evidence drives an online EM loop between a foundational VLM and a virtual prompt. (c) The agent’s task cues outline likely targets in red and draw lines toward them for a player in a headset.

Cite

Z. Li, X. He, H. Chen, C. Zou, S. Bai, B. Yang, M. Wu, J. Ledner, Y.-J. Cheng, A. Yamauchi, D. G. Turakhia, S. Feiner, and P. Sajda, “Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidence,” in Proc. ACM Symposium on User Interface Software and Technology (UIST), 2026, to appear.

@inproceedings{li2026xragent,
  title={Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidence},
  author={Li, Z. and He, X. and Chen, H. and Zou, C. and Bai, S. and Yang, B. and Wu, M. and Ledner, J. and Cheng, Y.-J. and Yamauchi, A. and Turakhia, D. G. and Feiner, S. and Sajda, P.},
  booktitle={2026 ACM Symposium on User Interface Software and Technology (UIST)},
  year={2026}
}