As computation becomes widely accessible, transparent, wearable and personal, it becomes a useful tool to augment everyday activities. Certain human capabilities such as daily scheduling need not remain the responsibility of a user when they can be easily transfered to personal digital assistants. This is especially important for tasks that are excessively cumbersome to humans yet involve little computational overhead. An important one is memory or information storage. It is well-known that some things are better stored using external artifacts (such as handwritten or electronic notes) than in a human brain. However, it is also critical that the transfer of information to be processed (i.e. by a digital assistant) proceeds in a natural, seamless way. Often, it is more cumbersome for a user to input data and functionality into a computer than to manually perform a task directly. In other words, the transfer from reality into a virtual space is often too distracting to the user and reduces a digital assistant's effectiveness. In such cases it would be helpful that the assistant operates autonomously without user intervention. DyPERS is a 'Dynamic Personal Enhanced Reality System' which is motivated by the above issues. It acts as an audio-visual memory assistant which reminds the user at appropriate times using perceptual cues as opposed to direct programming. Using a head-mounted camera and a microphone, DyPERS sees and hears what the user perceives to collect a fully audio-visual memory. The resulting multimedia database can be indexed and played back in real-time. The user then indicates to DyPERS which visual objects are important memory cues such that it learns to recognize them in future encounters and associate them with the recorded memories.
When a cue is recognized at some later time, DyPERS automatically overlays the appropriate audio-video clip on the user's world through a heads-up-display (HUD) [Feiner et al. , 1992], as a reminder of the content. This process is triggered when a relevant object is detected by the video camera system which constantly scans the visual field to detect objects which are associated with the memories.