Augmented Reality for Task Assistance and Training

Table of Contents:

  1. Principal Investigator.
  2. Productivity Measures.
  3. Summary of Objectives and Approach.
  4. Detailed Summary of Technical Progress.
  5. Transitions and DOD Interactions.
  6. Software and Hardware Prototypes.
  7. List of Publications.
  8. Invited and Contributed Presentations.
  9. Honors, Prizes or Awards Received.
  10. Project Personnel Promotions.
  11. Project Staff.
  12. Multimedia URL.
  13. Keywords.
  14. Business Office.
  15. Expenditures.
  16. Students.
  17. Book Plans.
  18. Sabbatical Plans.
  19. Related Research.
  20. History.

Principal Investigator.

Productivity Measures.

Summary of Objectives and Approach.

  1. Our objectives are to develop virtual environment technology that can be used for both training and task assistance. Our emphasis is on augmented reality systems that use see-through head-worn displays to supplement, rather than replace, the real world, enriching it with additional information. Augmented realities can be used for training by synthesizing physical objects that are overlaid on the real world. They can also be used for task assistance by overlaying real equipment with information that highlights relevant parts, reveals hidden internal structure, displays live test results, or provides narrative instructions.
  2. Our approach is two-pronged: we are building virtual environment support infrastructure and creating testbed applications with which to test this infrastructure. Our infrastructure is designed for fast prototyping of applications with three important features:
    1. Integration of multiple displays and interaction devices. This makes it possible for a single application to use a synergistic combination of hardware, including head-worn, hand-held, desk-top, and wall-mounted devices.
    2. Support for multiple users. Data sharing among users is possible at any level, ranging from a complete environment to an individual item, and is facilitated by language-level support for sharing arbitrary data structures.
    3. Knowledge-based virtual world design. We are developing techniques for the automated generation of virtual environments that communicate desired information to the user.

Detailed Summary of Technical Progress.

  1. We developed COTERIE (COlumbia Testbed for Exploratory Research in Interactive Environments), a toolkit that provides language-level support for building distributed virtual environments. COTERIE is based on the distributed data-object paradigm for distributed shared memory, and is implemented in Modula-3. Any data object in COTERIE can be declared to be a shared object that is replicated fully in any process that is interested in it. These shared objects support asynchronous data propagation with atomic serializable updates, and asynchronous notification of updates. Unlike other VE toolkits, COTERIE is based on a set of general-purpose parallel and distributed language constructs designed to accommodate the needs of virtual environments research. (See [7].)
  2. We are using COTERIE to build a set of testbed augmented-reality applications:
    1. construction assistance. The user is given instructions for assembling a structure composed of space-frame components.
    2. architectural anatomy. The user is presented with information about campus buildings. Two versions are being developed. An indoors version is based on our earlier standalone application, which is being ported to our new infrastructure. It will show building support systems within the context of our lab. (See [3].) An outdoors version, currently being constructed, uses a backpack containing a computer, packet-radio modem, and differential GPS system, connected to a see-through head-worn display with magentometer/inclinometer orientation tracking, and a hand-held tablet. It will overlay information in the vicinity of (i.e., coarsely registered with) surrounding landmarks.
    3. equipment maintenance. An electronic junction box is overlaid with notes that describe the status of connections.
    4. personal information space. A mobile augmented reality system has been constructed for offsite demonstrations. A preliminary test application, shown at OOPSLA '95, uses a notebook computer and a see-through head-worn display with built-in orientation tracking to overlay information about cities relative to the user's location.
  3. In our research on knowledge-based support for virtual environments, we have continued development of AutoVisual, which constructs virtual worlds for multivariate data visualization. Our recent work has concentrated on integrating a rule-based engine being implemented in CLIPS with a front-end implemented in STk (Tk in Scheme).

Transitions and DOD Interactions.

  1. We contributed numerous bug fixes and enhancements to the upcoming DEC SRC Modula-3 environment distribution, including a complete rewrite of the package used for persistent object I/O so that it would support heterogeneous architectures, and extension of the graphics packages to support Criterion RenderWare.
  2. We are working with Critical Mass (a newly formed software company that will be developing and distributing an enhanced version of the Modula-3 development environment) to coordinate bug fixes and enhancements.
  3. We submitted a proposal to ARPA on virtual environments for science education, coauthored with Ann Lasko-Harvill and Jaron Lanier (Talisman Dynamics), Douglas Smith and Cordell Green (Kestrel Institute), Coco Conn (Digital Circus Productions), Robert McClintock (Teachers College) and Ray Sambrotto (Lamont-Doherty Earth Observatory)
  4. We gave a talk on our work at the NRL Workshop on Virtual Reality (May 23-24, 1995), and will be meeting with Paul Radtke of NTSC in December.

Software and Hardware Prototypes.

  1. Prototype Name: COTERIE
  2. Prototype Name: AutoVisual
  3. Prototype Name: Architectural Anatomy
  4. Prototype Name: Construction Assistant
  5. Prototype Name: Equipment Maintenance Assistant
  6. Prototype Name: Personal Information Space

List of Publications.

  1. Crutcher, L., Lazar, A., Feiner, S., and Zhou, M. Management of broadband networks using a 3D virtual world. IEEE Parallel and Distributed Technology, 3(2), Summer 1995, 4-13.
  2. Feiner, S. Research in 3D user interface design at Columbia University. To appear in CHI '96 (Conf. on Human Factors in Computing Systems) Conf. Companion, Vancouver, British Columbia, Canada, April 14-18, 1996. Download a PostScript copy (23,388 bytes)
  3. Feiner, S., Krueger, T., Webster, T., MacIntyre, B., and Keller, E. Architectural anatomy. Presence, (4)3, Summer 1995, 318-325.
  4. Foley, J., van Dam, A., Feiner, S., and Hughes, J. Computer Graphics: Principles and Practice, Second Edition in C, Addison-Wesley, Reading, MA, 1995.
  5. MacIntyre, B. PC 3D Graphics Accelerators FAQ, v 0.9, November 1995, http://www.cs.columbia.edu/~bm/3dcards/3d-cards1.html, http://www.cs.columbia.edu/~bm/3dcards/3d-cards2.html,
  6. MacIntyre, B. and Feiner, S. Future multimedia user interfaces. To appear in R. Herrtwich (ed.), Fundamentals of Multimedia Systems, Morgan Kaufmann, Los Altos, CA, 1995.
  7. MacIntyre, B. and Feiner, S. Language-level support for exploratory programming of distributed virtual environments. Submitted. Download a PostScript copy (136,198 bytes)
  8. Rosenblum, L., Bryson, S., and Feiner, S. Virtual reality unbound. IEEE Computer Graphics & Applications (special issue on IEEE VRAIS '95), 15(5), September 1995, 19-21.
  9. Singh, G. and Feiner, S. Introduction to the special issue on virtual reality software and technology. ACM Transactions on Computer-Human Interaction (special issue on VRST '94), 2(3), September 1995, 177-178.

Invited and Contributed Presentations.

  1. Virtual reality for visualization (with S. Bryson). Contributed tutorial presentation, IEEE Visualization '94, Tysons Corner, VA, October 17-21, 1994.
  2. Setting up a virtual reality system, Invited tutorial presentation, New York Virtual Reality Expo '94, New York, NY, November 29-December 2, 1994.
  3. Augmented reality for task training and assistance. Invited presentation, ONR Workshop on Virtual Environments, Arlington, VA, March 21-22, 1995.
  4. Augmented reality: Seeing on top of the world. Invited colloquium presentation, IBM Almaden Research Center, CA, April 13, 1995.
  5. Augmented reality: Seeing on top of the world. Invited colloquium presentation, Polytechnic University, New York, NY, April 24, 1995.
  6. Discussant, Innovative Interaction, CHI '95, Denver, CO, May 7-11, 1995.
  7. Augmented reality. Invited presentation, NRL Workshop on Virtual Reality, Naval Research Lab, Washington, DC, May 23-24, 1995.
  8. Virtual worlds for visualizing information. Invited conference presentation, Colloquium on High Performance Scientific Computation 1995, Laboratorio Nacional de Computacao Cientifica, Rio de Janeiro, Brazil, July 31-August 4, 1995.
  9. Virtual worlds for visualizing information. Invited presentation, IBM T.J. Watson Research Center, Hawthorne, NY, August 24, 1995.
  10. Distributed objects for virtual environments. Contributed workshop presentation by Blair MacIntyre, Workshop on Reliability and Scalability in Distributed Object Systems, OOPSLA '95, Austin, TX, October 15, 1995.
  11. Augmented reality in Modula-3. Contributed presentation and demonstration by Blair MacIntyre, Modula-3 User Group Meeting, OOPSLA '95, Austin, TX, October 17, 1995.
  12. Augmented reality. Invited conference presentation, International Symposium in Electronics and Computing Technology 1995, Instituto Tecnologico de La Laguna, Torreon, Coahuila, Mexico, October 19-21, 1995.
  13. Virtual reality for visualization (with S. Bryson). Contributed tutorial presentation, IEEE Visualization '95, Atlanta, GA, October 30-November 3, 1995.
  14. Seeing on top of the world. Invited colloquium presentation, Media Lab Colloquium Series, Media Lab, MIT, Cambridge, MA, February 7, 1996.
  15. Research in 3D user interface design at Columbia University. Contributed conference presentation, CHI '96 (Conf. on Human Factors in Computing Systems), Vancouver, British Columbia, Canada, April 14-18, 1996.
  16. Augmented reality. Invited tutorial presentation. Swiss Federal Institute of Technology, Lausanne, Switzerland, May 29-31, 1996
  17. Topic to be determined. Invited conference presentation. Computer Animation '96, Geneva, Switzerland, June 3-5, 1996.

Honors, Prizes or Awards Received.

  1. Associate editor, ACM Transactions on Graphics, 1995-present
  2. Member of editorial board, IEEE Transactions on Visualization and Computer Graphics, 1994-present
  3. Chair of program committee, ACM UIST '94
  4. Co-chair of program committee, IEEE VRAIS '95 (Virtual Reality Annual International Symposium)
  5. Member of program committee, ACM CHI '95.
  6. Member of program committee, 1995 Workshop on Interactive 3D Graphics
  7. Member of papers committee, SIGGRAPH '95
  8. Member of program committee, Eurographics '95
  9. Member of program committee, IEEE Visualization '95
  10. Member of program committee, ISOTAS '96 (International Symposium on Object Technologies for Advanced Software)
  11. Member of papers committee, ACM CHI '96
  12. Member of program committee, IEEE VRAIS '96 (Virtual Reality Annual International Symposium)
  13. Member of papers committee, SIGGRAPH '96
  14. Member of program committee, Graphics Interface '97

Project Personnel Promotions.

Project Staff.

  1. Name: Dr Steven K. Feiner

Multimedia URL.

  1. EOYL FY95
  2. QUAD FY95 (PowerPoint)
  3. ACHIEVEMENT FY95 (PowerPoint)
  4. EOYL FY94
  5. Virtual Reality for Manufacturing - Case Studies
  6. Virtual Interface Technology Bibliography
  7. 3D Accelerator Page
  8. Architectural Anatomy QuickTime
  9. (New COTERIE videotape is being made.)

Keywords.

  1. Augmented Reality
  2. Virtual Environments
  3. Knowledge-Based Graphics
  4. Distributed Systems
  5. Mobile Computing

Business Office

Expenditures

  1. Est. FY96: 75% by 01 Oct 96
  2. FY95: 0% billed (23% actual) Current delay is due to a combination of charges for student support and offset occurring in batches and being posted late, and some vendors being slow to deliver and invoice. I am currently finishing negotiations for two large equipment deals that will consume the remainder of our 1995 funds by the end of this calendar year.
  3. FY94: 78% billed (100% actual)

Students

  1. Name: Mr Blair MacIntyre
  2. Name: Mr Clifford Beshers
  3. Name: Mr Xinshi Sha
  4. Name: Mr Alejandro Jaimes

Book Plans

Sabbatical Plans

  1. Person: Steven K. Feiner

Related Research

  1. U. Alberta MR Toolkit
  2. CMU VuMan Project
  3. Registration Errors in Augmented Reality, UNC Chapel Hill
  4. U. Toronto ETC-Lab
  5. DCIEM Robotics Lab
  6. MIT Wearable Computing
  7. NPSNET
  8. U. Virginia ALICE
  9. Modula-3
  10. comp.graphics
  11. sci.virtual-worlds

History

  1. As a graduate student of Andy van Dam at Brown University, I was supported in part by funding from Marv Denicoff. With the goal of using computers to improve technical documentation, we developed the Interactive Graphical Document system (IGD) from 1979-82. IGD supported the creation of graphical hypermedia documents. A document's pages not only presented static pictures and text, but served as the interface to interactive animations and simulations. Pages could be linked to other pages and nested in a recursive chapter hierarchy. IGD included a window-based layout system, through which authors viewed, created, and edited iconic representations of the pages, chapters, and links.
  2. IGD was the first system to allow users to view and edit a hypertext as a directed graph, whose nodes were depicted as scaled miniatures of the pages, complete with all their graphical content. It was also the first system to support automatically generated navigable graphical displays of user history (the forerunner of HyperCard's display of recently seen pages), of all links into and out of a page, and of the document's index. Hierarchical containment within chapters was used to reduce the visual complexity of large documents to a manageable level by limiting the number of nodes and links that were drawn, while allowing the user to drill down arbitrarily deeply where desired. (Today this would be called a user-controlled fisheye view.) As well, IGD was one of the first systems to allow non-programmers to create a graphical interface entirely through a direct manipulation, WYSIWIG, graphical interface. An author drew pictures, placed them on pages, created ``buttons'' within the pictures, and attached to the buttons assorted actions, such as linking to another page or running an animation.
  3. This work is reflected in current and past commercial ``card-based'' hypertext systems, such as HyperCard. With the current popularity of the web, many of the navigation facilities originally developed for IGD are being rediscovered by a new generation of researchers.
  4. IGD's documents were only as good as the abilities and efforts of the human authors and designers who created them. Recognizing this bottleneck to effective technical documentation prompted my dissertation work on APEX (Automated Pictorial EXplanations), a knowledge-based system that synthesized sequences of static 3D pictures that illustrate the actions performed by a problem solver. This work, coupled with drastic improvements in the performance and size of graphics processors and displays, led to my current research on knowledge-based virtual environments, and augmented reality.