Augmented Reality for Task Assistance and Training


Table of Contents:

  1. Principal Investigator.
  2. Productivity Measures.
  3. Summary of Objectives and Approach.
  4. Detailed Summary of Technical Progress.
  5. Transitions and DOD Interactions.
  6. Software and Hardware Prototypes.
  7. List of Publications.
  8. Invited and Contributed Presentations.
  9. Honors, Prizes or Awards Received.
  10. Project Personnel Promotions.
  11. Project Staff.
  12. Multimedia URL.
  13. Keywords.
  14. Business Office.
  15. Expenditures.
  16. Students.
  17. Book Plans.
  18. Sabbatical Plans.
  19. Related Research.
  20. History.


Principal Investigator.


Productivity Measures.


Summary of Objectives and Approach.

  • Our objectives are to develop virtual environment technology that can be used for both training and task assistance. Our emphasis is on augmented reality systems that use see-through head-worn displays to supplement, rather than replace, the real world, enriching it with additional information. Augmented realities can be used for training by synthesizing physical objects that are overlaid on the real world. They can also be used for task assistance by overlaying real equipment with information that highlights relevant parts, reveals hidden internal structure, displays live test results, or provides narrative instructions.
  • Our approach is two-pronged: we are building virtual environment support infrastructure and creating testbed applications with which to test this infrastructure. Our infrastructure is designed for fast prototyping of applications with three important features:
    1. Integration of multiple displays and interaction devices. This makes it possible for a single application to use a synergistic combination of hardware, including head-worn, hand-held, desk-top, and wall-mounted devices.
    2. Support for multiple users. Data sharing among users is possible at any level, ranging from a complete environment to an individual item, and is facilitated by language-level support for sharing arbitrary data structures.
    3. Knowledge-based virtual world design. We are developing techniques for the automated generation of virtual environments that communicate desired information to the user.


    Detailed Summary of Technical Progress.

    1. We have continued to develop COTERIE (Columbia Object-oriented Testbed for Exploratory Research in Interactive Environments), a toolkit that provides language-level support for building distributed virtual environments. COTERIE is based on the distributed data-object paradigm for distributed shared memory, and is implemented in Modula-3. Any data object in COTERIE can be declared to be a shared object that is replicated fully in any process that is interested in it. These shared objects support asynchronous data propagation with atomic serializable updates, and asynchronous notification of updates. Unlike other VE toolkits, COTERIE is based on a set of general-purpose parallel and distributed language constructs designed to accommodate the needs of virtual environments research. This year we completed the basic infrastructure, including the code generator, enhancements to the 3D graphics layer, and added support for new interaction devices.

      Over the last few months, we exposed COTERIE's shared data objects to Obliq (the distributed, interpreted language used in COTERIE), to create a new version of the language called ``Obliq*''. In Obliq, all data items are available as client-server distributed objects. As references to data items are passed between processes, they refer back to the original object and access it via remote method calls. However, there are two other kinds of data sharing semantics that are useful: replicated data, and simple data. We implemented replicated data in Obliq* by using COTERIE's shared data objects. Simple data items are objects that retain no connection to the original object when they are copied between machines.

    2. We extended the set of COTERIE prototype augmented-reality applications that we began to implement last year, and demonstrated them at several conferences:

      1. construction assistance. We demonstrated ARC (Augmented Reality for Construction), which provides overlaid graphics and sound to assist in the construction of spaceframe structures, at the ASCE Third Congress on Computing in Civil Engineering, Anaheim, CA, June 17-19, 1996.
      2. architectural anatomy. We started to explore multi-user applications by creating a virtual world in which one user wears a tracked HMD to explore our lab, augmented with a view of its hidden support infrastructure, while multiple remote users see alternate representations of the lab. This is our first experiment that uses COTERIE to investigate how distributed users with heterogeneous displays could interact. It relies on Prof. Nayar's Omnicam, which provides a wide-angle, true perspective (single center-of-projection) view of the environment, without the need for a moving camera mount. We demonstrated this application at ACM UIST '96, Seattle, WA, November 6-8, 1996.
      3. equipment maintenance. We used our telecommunications junction box domain for a preliminary exploration of how vision-based approaches can provide high-quality tracking in a controlled environment. (Our trial system tracks a set of colored markers on the junction box at over 10Hz in a software-only implementation on a 200MHz Pentium Pro.)
      4. personal information space. We demonstrated a mobile augmented reality system that displays overlaid information about the user's current environment at several venues, including ACM CHI '96, Vancouver, BC, April 14-18, 1996 and the ONR 50th Anniversary Symposium, Washington, DC, May 22, 1996,

    3. In our research on knowledge-based support for virtual environments, we continued development of n-Vision, our multivariate data visualization tool, and AutoVisual, which constructs virtual worlds for n-Vision. Patent 5,524,187 was awarded for the worlds-within-worlds visualization technique used in n-Vision on June 4, 1996.

      We designed and implemented a new specification language for n-Vision that provides a simpler, more flexible way to specify interactive virtual worlds, while at the same time ensuring a correct and efficient implementation. The language is processed by a specialized interpreter to which we have applied modern language implementation techniques and domain-specific optimizations to achieve high performance.

      We specify n-Vision visualizations as programs in a simple functional language, and use program transformation techniques to make possible efficient evaluation. Each visualization program is an expression that computes a graphical scene based on the current user input. In the functional representation, data dependencies among scene components are represented explicitly. This allows us to transform the expressions into cached dataflow graphs, which, when evaluated, track all changes in the system and recompute only those scene components that are affected. In addition, we use partial evaluation and dynamic compilation techniques to further optimize the most intensive part of the scene computation--the sampling of user functions. These optimizations have resulted in a visualization system that is simpler, faster and more reliable.

      While these optimizations are well-known to the programming language community, the highly dynamic nature of our application makes more traditional language implementations impractical, because none of the necessary information is available until run-time. The primary benefit of the improved target language is that AutoVisual, our visualization design component, can now be made both simpler and more powerful in turn: the space of possible visualizations is larger, because worlds are now composed of smaller components that are linked together dynamically; the specifications are smaller and simpler, and hence easier to generate; and the n-Vision subsystem now provides stronger guarantees of performance, relieving AutoVisual of some of the burden of optimization.


    Transitions and DOD Interactions.

    1. Member of NRL external review panel for 6.1 Research Option Proposal on Advanced Interfaces for Synthetic Environments (J. Templeman, PI), January 29, 1996.
    2. Gave demonstration of COTERIE at ONR 50th Anniversary Symposium, May 22, 1996.
    3. Hosted Paul Radtke (NTSC), for demonstrations of our ONR-funded research, October 4, 1996.
    4. Submitted pre-proposal for use of augmented reality in siting temporary structures to TriService CADD/GIS Technology Center, and were contacted on November 22, 1996 to arrange a technology demonstration,
    5. Worked with Larry Rosenblum (NRL) and Chandrajit Bajaj (Purdue) to prepare a CREW pre-proposal to ONR for research on virtual environments, October-November, 1996.
    6. Continue to work with Critical Mass (vendors of an enhanced version of the Modula-3 development environment), coordinating bug fixes and enhancements. Demonstrated sample COTERIE applications at Critical Mass's booth at ObjectWorld, May 1996.


    Software and Hardware Prototypes.

    1. Prototype Name: COTERIE
      • Type: software
      • URL: http://www.cs.columbia.edu/graphics/projects/coterie/
      • Availability: Currently internal. Will be publicly available soon through the WWW.
      • Description: Infrastructure for rapid prototyping of distributed virtual environments, emphasizing flexible language-level support for sharing arbitrary data among users.
    2. Prototype Name: AutoVisual
    3. Prototype Name: OpenGL Interface for Modula-3
    4. Prototype Name: Architectural Anatomy
    5. Prototype Name: ARC (Augmented Reality for Construction)
    6. Prototype Name: Equipment Maintenance Assistant
      • Type: software
      • Availability: Internal to project
      • Description: Developed to test COTERIE. Information about a telecommunications junction box is overlaid near termination blocks.
    7. Prototype Name: Personal Information Space
      • Type: software
      • Availability: Internal to project
      • Description: A portable application designed to demonstrate our COTERIE infrastructure at conferences. It uses a notebook computer and see-through head-worn display to situate information around the user.


    List of Publications.

    1. MacIntyre, B. and Feiner, S. Future multimedia user interfaces. Multimedia Systems, 4(5), 1996, 250-268. Download a PostScript file (2,342,912 bytes)
    2. Feiner, S. Research in 3D user interface design at Columbia University. CHI '96 (Conf. on Human Factors in Computing Systems) Conference Companion, Vancouver, British Columbia, Canada, April 14-18, 1996, 129-130.
    3. Singh, G., Feiner, S., and Thalmann, D. Introduction to the special section on virtual reality software and technology. Communications of the ACM, 39(5), May 1996, 35-36.
    4. Webster, A., Feiner, S., MacIntyre, B., Massie, W., and Krueger, T. Augmented reality in architectural construction, inspection and renovation. Proc. ASCE Third Congress on Computing in Civil Engineering, Anaheim, CA, June 17-19, 1996, 913-919.
    5. Dalal, M., Feiner, S., McKeown, K., Jordan, D., Allen, B., and alSafadi, Y. MAGIC: An experimental system for generating multimedia briefings about post-bypass patient status. Proc. 1996 AMIA Annual Fall Symp (Journal of the American Medical Informatics Association Symposium Supplement), Washington, DC, October 26-30, 1996, 684-688. Download a gzip'ed PostScript file (403,522 bytes compressed)
    6. Zhou, M. and Feiner, S. Data characterization for automatically visualizing heterogeneous information. Proc. INFOVIS '96 (IEEE Symp. on Information Visualization), San Francisco, CA, October 28-29, 1996, 13-20. Download a gzip'ed PostScript file (345,906 bytes compressed)
    7. MacIntyre, B. and Feiner, S. Language-level support for exploratory programming of distributed virtual environments. Proc. UIST '96 (ACM Symp. on User Interface Software and Technology), Seattle, WA, November 6-8, 1996, 83-94. Download a PostScript file (2,128,122 bytes)
    8. Dalal, M., Feiner, S., McKeown, K., Pan, S., Zhou, M., Höllerer, T., Shaw, J., Feng, Y., and Fromer, J. Negotiation for automated generation of temporal multimedia presentations. Proc. ACM Multimedia '96, Boston, MA, November 18-22, 1996, 55-64. Download a gzip'ed PostScript file (285,346 bytes compressed)
    9. Webster, A., Feiner, S., MacIntyre, B., Massie, W., and Krueger, T. Augmented reality applications in architectural construction. To appear in D. Bertol (ed.), Designing Digital Space: An Architect's Guide to Virtual Reality, John Wiley & Sons, New York, 1996, 193-200.
    10. Feiner, S. Foreword. To appear in C. Strothotte and T. Strothotte, Seeing Between the Pixels, Springer Verlag, Heidelberg, 1996.
    11. Zhou, M. and Feiner, S. Top-down hierarchical planning of coherent visual discourse. To appear in Proc. IUI '97 (1997 Int. Conf. on Intelligent User Interfaces), Orlando, FL, January 6-9, 1997. Download a gzip'ed PostScript file (334,434 bytes compressed)
    12. Beshers, C. and Feiner, S. Generating efficient virtual worlds for visualization using partial evaluation and dynamic compilation. Submitted. Download a PostScript file (1,433,128 bytes).)


    Invited and Contributed Presentations.

    1. Feiner, S. Seeing on top of the world. Invited talk. Media Lab Colloquium Series, Media Lab, MIT, Cambridge, MA, February 7, 1996.
    2. Beshers, C. Multivariate visualization. Invited keynote talk. 1996 Association for Business Simulation and Experiential Learning Conference, Orlando, FL, March 28-29, 1996.
    3. Feiner, S. Seeing on top of the world. Invited talk. Digital Equipment Corporation, Systems Research Center, Palo Alto, CA, March 29, 1996.
    4. Feiner, S. New horizons in virtual reality. Panel (with M. Bolas, S. Bryson, H. Fuchs, S. Stansfield, and M. Zyda). IEEE VRAIS '96, Santa Clara, CA, March 30-April 3, 1996.
    5. MacIntyre, B. Distributed programming in COTERIE. Demonstration. Critical Mass booth, ObjectWorld, Boston, MA, May 6, 1996.
    6. Feiner, S. and MacIntyre, B. Augmented reality. Demonstration. ONR 50th Anniversary Symposium, National Academy of Sciences, Washington, DC, May 22, 1996.
    7. Feiner, S. Augmented reality. Tutorial. Computer Science Postgraduate Course on Virtual Reality. Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland, May 29-31, 1996.
    8. Feiner, S. Adding insight through animation in augmented reality. Invited keynote talk. Computer Animation '96, Geneva, Switzerland, June 3-4, 1996. (Abstract appears as Feiner, S. Adding insight through animation in augmented reality, Proc. Computer Animation '96, June 3-4, 1996, Geneva, Switzerland, IEEE Comp. Soc. Press, 14-15.)
    9. Feiner, S., Webster, A., and MacIntyre, B. Augmented reality system for spaceframe assembly. Demonstration. Third ASCE Congress for Computing in Civil Engineering, Anaheim, CA, June 17-19, 1996.
    10. Feiner, S. Augmented reality for visualization. Tutorial. Eighth International School for Computer Science Researchers. Lipari, Italy, July 1-12, 1996.
    11. Feiner, S. Augmented reality. Panel (with R. Azuma, G. Bishop, M. Dailey, S. Ellis, and U. Neumann). Workshop on Wearable Computer Systems, Renton, WA, August 19-21, 1996.
    12. Feiner, S. Agents and system intelligence. Panel (chair: A. Biermann, with C. Knoblock, M. Maybury, J. Moore, K. Sycara, and K. Wittenburg). Computer Science and Telecommunications Board Workshop: Toward an Every-Citizen Interface to the National Information Infrastructure, Washington, DC, August 22-23, 1966.
    13. Feiner, S. Seeing on top of the world. Invited talk. Microsoft Research, Redmond, WA, November 5, 1996.
    14. MacIntyre, B. and Feiner, S. Language-level support for exploratory programming of distributed virtual environments. Demonstration. UIST '96 (ACM Symp. on User Interface Software and Technology), Seattle, WA, November 6-8, 1996.
    15. Feiner, S. Seeing on top of the world. Invited talk. Department of Computer Science Seminar Series, University of British Columbia, Vancouver, Canada, January 30, 1997.
    16. Feiner, S., Webster, A., and MacIntyre, B. ARC (Augmented Reality for Construction). Demonstration. To be presented at ACM '97, Anaheim, CA, March 1-4, 1997.


    Honors, Prizes or Awards Received.

    1. Honorable Mention, 1996 American Institute of Architects/Architecture Architectural Research Awards Program (with A. Webster, B. MacIntyre)
    2. Member of IBM Mobile Scientific Advisory Board, 1996-present
    3. Member of executive board, IEEE Technical Committee on Computer Graphics, 1993-present
    4. Member of editorial board, IEEE Transactions on Visualization and Computer Graphics, 1994-present
    5. Member of editorial board, The Virtual Reality Society, 1994-present
    6. Member of editorial board, Electronic Publishing, 1988-present
    7. Member of program committee, ISOTAS '96 (International Symposium on Object Technologies for Advanced Software)
    8. Member of papers meta-review committee, ACM CHI '96
    9. Member of program committee, IEEE VRAIS '96 (Virtual Reality Annual International Symposium)
    10. Member of papers committee, SIGGRAPH '96
    11. Member of paper committee, IEEE Visualization '96
    12. Member of extended program committee, ACM Multimedia '96
    13. Member of program committee, Graphics Interface '97
    14. Member of program committee, Computer Graphics International '97
    15. Member of technical sketches committee, SIGGRAPH '97
    16. North-American co-chair of program committee, ACM VRST '97 (Virtual Reality Software and Technology)
    17. Member of program committee, WearCon '97 (1st Int. Symp. on Wearable Computers)
    18. Member of symposium committee, IEEE InfoVis '97


    Project Personnel Promotions.


    Project Staff.

    1. Name: Dr Steven K. Feiner
      • Homepage
      • Position: Associate Professor
      • Task: Principal Investigator


    Multimedia URL.

    1. EOYL FY96
    2. QUAD FY96 (PowerPoint)
    3. ACHIEVEMENT FY96 (PowerPoint)
    4. EOYL FY95
    5. QUAD FY95 (PowerPoint)
    6. ACHIEVEMENT FY95 (PowerPoint)
    7. EOYL FY94

    8. MacIntyre, B. PC 3D Graphics Accelerators FAQ, v 1.2, March 1996,
      http://www.cs.columbia.edu/~bm/3dcards/3d-cards1.html
      http://www.cs.columbia.edu/~bm/3dcards/3d-cards2.html

      Selected References to work by Columbia's Computer Graphics and User Interfaces Lab

    9. Applying Virtual Environments to Manufacturing
    10. Smart Clothes: The MIT Wearable Computing Web Page
    11. WWW References for OpenGL
    12. Computer Graphics Research
    13. Research in Visualization
    14. VR References
    15. Virtual Reality: Architecture Applications
    16. Virtual Reality: Simulation and Training Applications
    17. Virtual Reality on the Net: Research Centers
    18. Augmented Reality and Computer Augmented Environments
    19. HCI Resources: Research Labs and Projects
    20. Pathfinder for Human-Computer Interaction (HCI)
    21. Augmented Reality
    22. HCI Project Home Page: Links to Other Sites
    23. Human Factors and Ergonomics Society Virtual Environments Technical Group
    24. Virtual Interface Technology Bibliography
    25. The Attribute Explorer
    26. Externalising Abstract Mathematical Models
    27. Virtual Environments Annotated Bibliography
    28. Annotated Visualization Bibliography
    29. Financial Viewpoints: Using point-of-view to enable understanding of information
    30. The Mind's Eye--An Approach to Understanding Large Complex Information-Bases through Visual Discourse
    31. Visualizing Retrieved Information: A Survey (Special Report, Computer Graphics and Visualization in the Global Information Infrastructure), IEEE CG&A, 16(2), March 1996
    32. Tutorial on Visualization
    33. Space of query devices
    34. Thesis: Data Visualisation
    35. Dimension 3D links to related work
    36. comp.sys.ibm.pc.hardware.video FAQ Part 3/4
    37. Guide to 3D User Interfaces
    38. 3D Graphics and the X Window System
    39. CGI and Animation Literature, Reference and Discussions
    40. VR Hardware


    Keywords.

    1. Augmented Reality
    2. Virtual Environments
    3. Virtual Worlds
    4. Knowledge-Based Graphics
    5. Distributed Systems
    6. Mobile Computing
    7. Wearable Computing


    Business Office


    Expenditures

    1. Est. FY97: 100%
    2. FY96: 100%
    3. FY95: 100%
    4. FY94: 100%
    5. FY93: 100%


    Current and Former Students

    1. Name: Mr Blair MacIntyre
      • Homepage
      • Position: Ph.D. student, research assistant
      • Nationality: Canada
      • Task: Develop COTERIE augmented reality infrastructure
      • Thesis: Building and interacting with distributed multi-user augmented reality systems
    2. Name: Mr Clifford Beshers
      • Homepage
      • Position: Ph.D. student, research assistant
      • Nationality: USA
      • Task: Develop knowledge-based virtual environments
      • Thesis: Knowledge-based virtual environments for multivariate visualization
    3. Name: Mr Xinshi Sha
      • Homepage
      • Position: Ph.D. student, research assistant
      • Nationality: PRC
      • Task: Develop high-performance 3D graphics backends for COTERIE
      • Thesis: Undecided (3rd semester student)
    4. Name: Mr Alejandro Jaimes
      • Homepage
      • Position: M.S. student
      • Nationality: Colombia
      • Task: Use of Stk to build new UI for AutoVisual
    5. Name: Dr Dorée Seligmann
      • Homepage
      • Nationality: USA
      • Task: Automated design of 3D graphics for interactive manuals
      • Thesis: Interactive Intent-Based Illustration: A Visual Language for 3D Worlds
      • Graduated: Ph.D., 1993
      • Job: Member, Multimedia Research Department, Bell Labs, Lucent Technologies
    6. Name: Dr David Kurlander
      • Homepage
      • Nationality: USA
      • Task: Novel techniques for example-based editing of illustrations
      • Thesis: Graphical Editing by Example
      • Graduated: Ph.D., 1993
      • Job: Member, User Interfaces Group, Microsoft Research


    Book Plans

    1. Topic: Computer graphics
      • Publisher: Addison-Wesley
      • Publication Year: 1997-2000
      • Type of Publication: textbook (3rd edition of Foley, J., van Dam, A., Feiner, S., and Hughes, J., Computer Graphics: Principles and Practice)
      • Additional Support: No


    Sabbatical Plans


    Related Research

    1. CMU VuMan Project
    2. Registration Errors in Augmented Reality, UNC Chapel Hill
    3. U. Toronto ETC-Lab
    4. DCIEM Robotics Lab
    5. NPSNET
    6. U. Virginia ALICE
    7. Modula-3
    8. Applying Virtual Environments to Manufacturing
    9. Research on comparative effectiveness of recall using augmented reality, virtual reality, and stationary CRTs
    10. Smart Clothes: The MIT Wearable Computing Web Page
    11. Research in Visualization
    12. Virtual Reality: Architecture Applications
    13. Virtual Reality: Simulation and Training Applications
    14. ECRC Research on Augmented Reality
    15. Augmented Reality and Computer Augmented Environments
    16. comp.graphics
    17. sci.virtual-worlds


    History

    1. Microsoft's Comic Chat automatically generates a comic-strip representation of the IRC ``chat session'' in which a user is participating. It uses heuristics to determine the characters to place in each panel, and when to start a new panel; the characters' gestures, expressions, positions, and orientations; the shape and layout of word balloons; and panel zoom level. Comic Chat was created by former Ph.D. student David Kurlander, based in part on the idea of a graphical history, developed from 1988-1993 and reported on in his Ph.D. dissertation.
    2. As a graduate student of Andy van Dam at Brown University, I was supported in part by funding from Marv Denicoff. With the goal of using computers to improve technical documentation, we developed the Interactive Graphical Document system (IGD) from 1979-82. IGD supported the creation of graphical hypermedia documents. A document's pages not only presented static pictures and text, but served as the interface to interactive animations and simulations. Pages could be linked to other pages and nested in a recursive chapter hierarchy. IGD included a window-based layout system, through which authors viewed, created, and edited iconic representations of the pages, chapters, and links.
    3. IGD was the first system to allow users to view and edit a hypertext as a directed graph, whose nodes were depicted as scaled miniatures of the pages, complete with all their graphical content. It was also the first system to support automatically generated navigable graphical displays of user history (the forerunner of HyperCard's display of recently seen pages), of all links into and out of a page, and of the document's index. Hierarchical containment within chapters was used to reduce the visual complexity of large documents to a manageable level by limiting the number of nodes and links that were drawn, while allowing the user to drill down arbitrarily deeply where desired. (Today this would be called a user-controlled fisheye view.) As well, IGD was one of the first systems to allow non-programmers to create a graphical interface entirely through a direct manipulation, WYSIWIG, graphical interface. An author drew pictures, placed them on pages, created ``buttons'' within the pictures, and attached to the buttons assorted actions, such as linking to another page or running an animation.
    4. This work is reflected in current and past commercial ``card-based'' hypertext systems, such as HyperCard. With the current popularity of the WWW, many of the navigation facilities originally developed for IGD are being rediscovered by a new generation of researchers.
    5. IGD's documents were only as good as the abilities and efforts of the human authors and designers who created them. Recognizing this bottleneck to effective technical documentation prompted my dissertation work on APEX (Automated Pictorial EXplanations), a knowledge-based system that synthesized sequences of static 3D pictures that illustrate the actions performed by a problem solver. This work, coupled with drastic improvements in the performance and size of graphics processors and displays, led to my current research on knowledge-based virtual environments, and augmented reality.