Current efforts are being made to port the system to run locally and efficiently on an untethered wearable computer with only on-board processing. This would enable multiple users to engage in a usability study where the system can be tested on a variety of pool tables in more diverse situations.
One important addition in progress is the inclusion of a 3D estimation stage which would analyze object locations over time to compute a 3D structural estimate of the scene. This would maintain virtual positions of balls and pockets that fell out of the current image view. 3D recursive estimation and 3D alignment algorithms could increase the precision and stability of the output   .
In addition, we are using the recovery of 3D structure to combine the system's output on the user's natural binocular field (instead of the video camera's images). This allows a much more natural display since the distortion due to the camera's awkward placement on the side of the head can be factored out and a virtual camera view (centered at each eye) can be simulated. Figure depicts the recovery of 3D information and its use for calibrated graphical overlay reprojection.
Figure: Recovery and Synthesis of 3D Models
The user sees the world through the stereo, transparent HUD and the computer sees it through its video camera. However, using a recursive structure from motion estimator, we can accumulate multiple 2D views of the objects in the scene to converge to a 3D model of the scene. The 3D model can then be analyzed and used to generate appropriate synthetic 3D overlays. A simple 3D transformation maps the synthetic 3D model into the view point of each eye. This 3D transformation is determined manually during initialization by having the user adjust the 3D rendering software's view point. Assume that Figure (a) is the image seen by the user's left eye. Figure (b) is a recovered 3D model of the pool table synthesized in a real-time 3D software package and displayed on the left CRT of the HUD. The user changes the view in the 3D display such that it matches the image seen behind the CRT (of the true pool table) and the system is calibrated to synthesize images for the left eye as shown in Figure (c). The calibration stays constant and the user can freely move while the synthetic overlays rotate to compensate with the head rotation. The system, however, requires very accurate, frame to frame tracking of the pool table to estimate its 3D pose and structure (which is necessary for synthesizing the 3D overlays).
Figure: Calibration of 3D Output
We are also investigating more sophisticated methods for strategy and planning which would consider higher-order effects such as rebounds, multiple collisions as well as preparatory placement for subsequent shots.
Other sensors could be added to the system to increase precision. For example, if a laser pointer could be mounted in the pool cue, we could determine the exact impact point on the cue ball and compute english and spin to predict more complex trajectories. Also, tracking the pool stick would allow the system to realize when its suggested shot is being disregarded so that it may assist the user in completing another shot.