Automated Generation of Three-Dimensional Virtual Worlds for Task Explanation

Table of Contents:

  1. Principal Investigator.
  2. Productivity Measures.
  3. Summary of Objectives and Approach.
  4. Detailed Summary of Technical Progress.
  5. Transitions and DOD Interactions.
  6. Software and Hardware Prototypes.
  7. List of Publications.
  8. Invited and Contributed Presentations.
  9. Honors, Prizes or Awards Received.
  10. Project personnel promotions obtained.
  11. Project Staff.
  12. Misc Hypermedia URL.
  13. Keywords.

Principal Investigator.

Productivity Measures.

Summary of Objectives and Approach.

  1. Knowledge-based graphics. Our work investigates the use of AI techniques to generate high-quality interactive graphics automatically for tasks ranging from equipment maintenance and training to data visualization. Automated generation addresses a number of serious problems with the otherwise conventional authoring approaches that underly hypermedia and multimedia documentation and editor-based visualization environments: Despite technological advances, hand-crafted presentations remain time-consuming and expensive to author, and are not adequately customized to the needs of individual users and situations.

  2. Virtual worlds. Our previous work in knowledge-based graphics generation has stressed the automated design of static and animated graphics for explaining 3D phenomena. In this research, we developed a principled approach to generating graphics that fulfill a set of formally expressed communicative goals that are provided as input. Our current research extends our previous work by investigating the use of 3D displays and interaction devices to allow the user to explore a virtual world that the system designs and generates on the fly to explain objects and tasks to the user in both concrete and abstract domains. As the user moves around in, or is guided about the world, the system monitors their actions and redesigns the information being presented to better fulfill the set of communicative goals that it has been assigned. We are not only investigating self-contained virtual worlds, but are also addressing the development of augmented realities in which computer-generated material enriches, rather than replaces, the user's view of the world

Detailed Summary of Technical Progress.

  1. Knowledge-based multivariate visualization. Interactive visualization systems provide a powerful means to explore complex data, especially when coupled with 3D interaction and display devices to produce virtual worlds. Although designing a quality static 2D visualization is already a difficult task for most users, designing an interactive 3D one is even more challenging. To address this problem, we continued our work on AutoVisual, a rule-based system that designs interactive virtual worlds for visualizing and exploring multivariate relations. AutoVisual uses ``worlds within worlds,'' an interactive visualization technique that we developed previously, which exploits nested, heterogeneous coordinate systems to map multiple variables onto each of the three spatial dimensions. AutoVisual's design process is guided by user-specified visualization tasks, and by a set of design principles encoded using a rule-based language. The virtual worlds that it designs are input to a new version of our n-Vision multivariate visualization system, which is controlled from AutoVisual through Tcl/Tk.

  2. Augmented reality. We continued our work on augmented reality, using a see-through head-mounted display to overlay a virtual world on the real world. Complementing our work on KARMA (Knowledge-based Augmented Reality for Maintenance Assistance), in Architectural Anatomy, we developed a prototype environment for exposing the hidden structural support systems in a building, making it possible for the user to view columns and joists within the walls. In this project, we applied our work on incorporating 2D window management tools in a virtual environment to display the 2D output of a commercial structural analysis application in a 2D X11 window anchored in the 3D virtual environment.

    Our original prototype environment was developed in C/C++, and used X11R5, modified by us to support a memory-mapped frame buffer, running under Mach. Much of our work during the past year has concentrated on a new implementation that uses X11R6 on Solaris, allowing us to take advantage of a wider range of processors. We decided to step back and redesign the system so that it would let us grow in directions that would be difficult to pursue with our previous prototype. In general, these directions address the support of multiple, mobile users, who interact with both virtual and real objects, and who collaborate as they move into and out of each other's physical space. At the core of this work is a large-scale distributed support infrastructure, whose software we are building in Modula-3. As this work progresses, we expect to use Modula-3 network objects to support distribution of objects, connection creation and teardown, and recovery from and tolerance to crashes, in conjunction with the integrated, interpreted language Obliq to support rapid prototyping. The tracker servers, graphics database, etc., can all be threads, whose interfaces are advertised with network objects, accessible from Modula-3 and Obliq.

  3. A virtual world for network management. We are investigating how a virtual world can be used to support the management of large, fast computer networks. We have been developing an automated visualization design component for this domain that will determine which presentation techniques to use to meet the user's informational needs. Over the past year, we developed a preliminary version of this component. It includes a collection of visualization techniques (mapping of data to 3D objects and their properties, camera motion, object motion, transparency, cutaway views, insets) that can be used to explore a small computer network, currently by hand.

Transitions and DOD Interactions.

  1. DEC SRC. Met on several occasions with Mark Manasse and Marc Brown of DEC SRC to discuss support for our augmented reality research (discussions still in progress).
  2. Digital Image Design. Met on many occasions with Brad Paley and Juey Ong of Digital Image Design, who have donated to our lab a Cricket 3D interaction device. The Cricket is currently being used in our multivariate visualization and network visualization projects.
  3. McDonnell Douglas Aerospace. Met with John Pridgeon and David McMillan of McDonnell Douglas Aerospace, on June 3, 1994 to discuss applications of augmented reality to military maintenance and repair, and later prepared a joint proposal with James Curtin of McDonnell Douglas Training Systems and Peter Tinker of Xtensory Inc.
  4. NYNEX Science & Technology. Met with Alison Lee of NYNEX Science & Technology on May 11, 1994 to discuss our work on 3D user interfaces.
  5. Sun Microsystems. Bruce Reichlen of Sun Microsystems donated several FPGA SBUS boards for use with our current see-through HMD, and access to a Sun design compilation service that we will be using to generate code for the boards.
  6. Virtual I/O. Met on several occasions with Greg Amadon (Pres.) and Linden Rhodes (Vice Pres.) of Virtual I/O, a Seattle-based vendor of a new, relatively inexpensive, see-through HMD, shown at SIGGRAPH '94. We currently have a developer's kit on order from Virtual I/O and have discussed our requirements for future display products.

Software and Hardware Prototypes.

  1. Augmented reality testbed. This environment provides software and hardware support for a head-tracked see-through HMD. and multiple tracked physical objects. In addition to 3D virtual objects, we support 2D X11 windows that can be arbitrarily positioned relative to the HMD, relative to the user's body, or relative to the 3D world.
  2. Multivariate visualization. Our n-Vision multivariate visualization system supports user exploration of multivariate relations in a ``fish-tank virtual world.'' It is written in C++ and Tcl/Tk, and uses StereoGraphics CrystalEyes stereo eyewear, in conjunction with VPL DataGlove and Digital Image Design Cricket interaction devices. The AutoVisual automated visualization design prototype, is written in Tcl/Tk and the CLIPS production system language. It interactively creates virtual worlds for n-Vision, based on a specification input by the user.
  3. Network visualization. Our network visualization testbed uses StereoGraphics Crystal Eyes eyewear and Digital Image Design Cricket and Logitech 6DOF mouse interaction devices. It is written in SGI Inventor, with a knowledge-based design component that is being developed in CLIPS.

List of Publications.

  1. Kurlander, D. and Feiner, S. Inferring constraints from multiple snapshots. ACM Transactions on Graphics, 12(4), October 1993, 277-304. 335KB PostScript file
  2. Feiner, S., MacIntyre, B., Haupt, M., and Solomon, E. Windows on the world: 2D windows for 3D augmented reality. Proc. UIST '93 (ACM Symp. on User Interface Software and Technology), Atlanta, GA, November 3-5, 1993, 145-155. 8.1MB PostScript file
  3. Encarnação, J., Foley, J., Bryson, S., Feiner, S., Gershon, N. Research issues in perception and user interfaces. In IEEE Computer Graphics and Applications, 14(2), March 1994, 67-69. (Reprinted in L. Rosenblum, R. A. Earnshaw, J. Encarnação, H. Hagen, A. Kaufman, S. Klimenko, G. Nielson, F. Post, D. Thalmann (eds.), Scientific Visualization: Advances and Challenges, Academic Press, London, 1994, pp. 467-472.)
  4. Beshers, C. and Feiner, S. Automated design of data visualizations. In L. Rosenblum, R. A. Earnshaw, J. Encarnacao, H. Hagen, A. Kaufman, S. Klimenko, G. Nielson, F. Post, D. Thalmann (eds.), Scientific Visualization: Advances and Challenges, Academic Press, London, 1994, 87-102.
  5. McKeown, K., Millman, D., Donnelly, B., Hoover, J., McClintock, R., Scholten, W., Anastassiou, D., Chang, S., Crosswell, A., Dalal, M., Feiner, S., Kantor, P., Klavans, J., and Schwartz, M. The Janus Digital Library. Digital Libraries '94, College Station, TX, June 19-21.
  6. Singh, G., Feiner, S., and Thalmann, D. (eds.). Virtual Reality Software & Technology: Proceedings of the VRST '94 Conference, World Scientific, Singapore, 1994.
  7. Feiner, S., Krueger, T., Webster, T., MacIntyre, B., and Keller, E. Architectural anatomy. To appear in Presence, 1995.
  8. Crutcher, L., Lazar, A., Feiner, S., and Zhou, M. Management of broadband networks using a 3D virtual world. To appear in IEEE Parallel and Distributed Technology, 1995.
  9. MacIntyre, B. and Feiner, S. Future multimedia user interfaces. To appear in R. Herrtwich (ed.), Fundamentals of Multimedia Systems, Morgan Kaufmann, Los Altos, CA, 1995. (To be published simultaneously in conventional book form and through WWW.)

Invited and Contributed Presentations.

  1. Virtual worlds for visualizing information. Gartner Group Symposium '93, Lake Buena Vista, FL, October 4-8, 1993.
  2. Knowledge-based multimedia and virtual worlds, New Horizons in Physics Lecture Series: Multimedia, SUNY, New Paltz, NY, October 14, 1993.
  3. Virtual worlds for visualizing information. Department of Computer Science Industrial Partners Program Day, Brown University, Providence, RI, October 15, 1993.
  4. Virtual worlds and augmented reality at Columbia University. Virtual Reality Systems Fall '93, New York, NY, October 18-21, 1993.
  5. Virtual worlds for visualizing information. Apple Computer, Inc., Cupertino, CA, October 22, 1993.
  6. Tutorial: Virtual reality for visualization (with S. Bryson). Visualization '93, San Jose, CA, October 24-29, 1993.
  7. Augmented reality. 5th EFDPMA Washington DC Virtual Reality Conference, Education Foundation of the Data Processing Management Association, Washington, DC, November 1-2, 1993.
  8. Virtual worlds for visualizing information. Columbia University Center for Medical Informatics Seminar, New York, NY, February 10, 1994.
  9. Virtual worlds for visualizing information. Institut für Informatik, Freie Universität, Berlin, Germany, March 14, 1994.
  10. Virtual worlds for visualizing information. Fraunhofer Institute for Computer Graphics, Rostock, Germany, March 15, 1994.
  11. Virtual worlds for visualizing information. Department of Computer Science, Otto-von-Guericke Universität Magdeburg, Magdeburg, Germany, March 16, 1994.
  12. Keynote address: Knowledge-based graphics and virtual worlds. Simulation & Integration '94, Magdeburg, Germany, March 17-18 1994.
  13. Virtual worlds for visualizing information. Institute for the Learning Sciences, Northwestern University, Evanston, IL, April 15, 1994.
  14. User interfaces for the 21st century: Knowledge-based graphics and virtual worlds. Educating America for the 21st Century: Driving Forces, Institute for Learning Technologies, Columbia University, April 21, 1994.
  15. Discussant, Interacting in 3-D, CHI '94, Boston, MA, April 24-28, 1994.
  16. Redefining the user interface: Augmented reality. Universal Personal Communications: Achieving Global Wireless Connectivity, Columbia University, New York, NY, May 13, 1994.
  17. Keynote address: Knowledge-based virtual worlds and augmented reality. Virtual Reality Applications '94, Leeds, UK, June 7-9, 1994.
  18. Tutorial: Virtual reality and its applications (with S. Bryson), Virtual Reality Applications '94, Leeds, UK, June 7-9, 1994.
  19. Multimedia User Interfaces: Augmented Reality and Ubiquitous Computing, Seminar on Fundamentals and Perspectives of Multimedia Systems, Dagstuhl, Germany, July 4-8, 1994.
  20. Tutorial: Developing advanced virtual reality applications (with S. Bryson, R. Pausch, D. Proffitt, and H. Sowizral). ACM SIGGRAPH '94, Orlando, FL, July 24-29, 1994.
  21. Panel: Research frontiers in virtual reality. Co-chair (with S. Bryson). Panelists: F.P. Brooks, Jr., P. Hubbard, R. Pausch, A. van Dam, ACM SIGGRAPH '94, Orlando, FL, July 24-29, 1994. (Published as Bryson, S., Feiner, S., Brooks, F., Hubbard, P., Pausch, R., and van Dam, A. Research frontiers in virtual reality. Proc. SIGGRAPH '94, Orlando, FL, July 24-29, 1994, 473-474.)
  22. Tutorial: Introduction to virtual reality design (with S. Bryson). VRST '94, Singapore, August 23-26, 1994.

Honors, Prizes or Awards Received.

  1. Associate editor, ACM Transactions on Information Systems.
  2. Member of editorial board, IEEE Transactions on Visualization and Computer Graphics.
  3. Member of editorial board, Electronic Publishing (Wiley).
  4. Member of editorial board, Advances in HCI (Ablex).
  5. Member of executive board, IEEE Technical Committee on Computer Graphics.
  6. Co-chair of symposium, 1993 IEEE Symposium on Research Frontiers in Virtual Reality.
  7. Co-chair of program committee, VRST '94 (Virtual Reality Software and Technology).
  8. Chair of program committee, ACM UIST '94.
  9. Co-chair of program committee, IEEE VRAIS '95 (Virtual Reality Annual International Symposium).
  10. Associate papers chair, ACM CHI '94.
  11. Member of organizing and program committees, AAAI Spring '94 Symposium on Intelligent Multi-Media Multi-Modal Systems.
  12. Member of program committee, AVI '94 (Advanced Visual Interfaces).
  13. Member of program committee, IEEE Visualization '94.
  14. Member of editor-in-chief search committee, IEEE Transactions on Visualization and Computer Graphics.
  15. Member, ACM Scholars' Advisory Group.
  16. Invited Participant, SIGGRAPH 1994 Future Search Conference.
  17. Member of program committee, ACM CHI '95.
  18. Member of program committee, 1995 Workshop on Interactive 3D Graphics.
  19. Member of papers committee, SIGGRAPH '95.
  20. Member of program committee, Eurographics '95.
  21. Member of program committee, IEEE Visualization '95.
  22. Member of program committee, ISOTAS '96 (International Symposium on Object Technologies for Advanced Software).

Project Personnel Promotions Obtained.

Project Staff.

  1. Steven K. Feiner, PI, Associate Professor of Computer Science
  2. Cliff Beshers, Ph.D. student
  3. Blair MacIntyre, Ph.D. student
  4. Michelle Zhou, Ph.D. student
  5. Diane Beuschel, M.S. student
  6. Alexandra Ewert, M.S. student
  7. Marcus Haupt, M.S. student
  8. Peter Kamali, M.S. student
  9. Michael Kraizman, M.S. student
  10. Miriam Roiter, M.S. student

Misc Hypermedia.

  1. EOYL 1994
  2. QUAD 1994 (1)
  3. QUAD 1994 (2)
  4. QUAD 1994 (3)
  5. QUAD 1994 (4)
  6. Columbia University Graphics and User Interfaces Laboratory
  7. Architectural Anatomy videotape (online in .html storyboard and QuickTime and MPEG formats). This animation depicts early results from an ongoing collaboration between members of Columbia University's Department of Computer Science and School of Architecture. In this project, we expose a building's ``architectural anatomy,'' allowing the user to see its otherwise hidden structural systems. We build on our support software for combining 2D X11 windows and 3D graphics in augmented reality. The animation was recorded with a video camera that ``wears'' the prototype see-through head-mounted display that we developed.

    Our prototype application overlays a graphical representation of portions of a building's structural systems over a user's view of the room in which they are standing. The overlaid virtual world typically shows the outlines of the concrete joists, beams, and columns surrounding the room. We have built a partial model of Columbia's Schapiro Center for Physical Science and Engineering Research, which contains portions of the joists, beams, and columns that are in and near the lab in which our work is being performed. The model is based on the construction drawings provided by the building's architects.

    The animation starts with a pan over part of our laboratory, showing wireframe (without hidden-line removal) views of several of the building's structural elements, including vertical columns and part of the floor, which are seen overlaid on a corner of our lab.

    The view then pans up to the ceiling, revealing the top of one of the major columns.

    As the user pans down the column, the column is selected with the mouse, indicating that the user wishes to view its internal structure in conjunction with a structural analysis. When the user selects the column, it changes line style.

    After a brief pause, the internal reinforcing steel bars inside the selected column appear, along with an X11 window that contains the output of a commercially available structural analysis and design program.

    The window's position is fixed relative to the 3D physical world so that it appears to be attached to the selected structural element as the user moves.

Keywords.

  1. knowledge-based graphics
  2. virtual environment, virtual world, virtual reality
  3. augmented reality
  4. 3D user interface
  5. head-mounted display