Current efforts are being made to port the system to run locally and efficiently on an untethered wearable computer with only on-board processing. This would enable multiple users to engage in a usability study where the system can be tested on a variety of pool tables in more diverse situations.
One important addition in progress is the inclusion of a 3D estimation stage which would analyze object locations over time to compute a 3D structural estimate of the scene. This would maintain virtual positions of balls and pockets that fell out of the current image view. 3D recursive estimation and 3D alignment algorithms could increase the precision and stability of the output [9] [2] [1].
In addition, we are using the recovery of 3D structure to combine the
system's output on the user's natural binocular field (instead of the
video camera's images). This allows a much more natural display since
the distortion due to the camera's awkward placement on the side of
the head can be factored out and a virtual camera view (centered at
each eye) can be simulated. Figure 
depicts the recovery
of 3D information and its use for calibrated graphical overlay
reprojection.
Figure: Recovery and Synthesis of
3D Models
The user sees the world through the stereo, transparent HUD and the
computer sees it through its video camera. However, using a recursive
structure from motion estimator, we can accumulate multiple 2D views
of the objects in the scene to converge to a 3D model of the
scene. The 3D model can then be analyzed and used to generate
appropriate synthetic 3D overlays. A simple 3D transformation maps the
synthetic 3D model into the view point of each eye. This 3D
transformation is determined manually during initialization by having
the user adjust the 3D rendering software's view point. Assume that
Figure (a) is the image seen by the user's left
eye. Figure (b) is a recovered 3D model of the pool table
synthesized in a real-time 3D software package and displayed on the
left CRT of the HUD. The user changes the view in the 3D display such
that it matches the image seen behind the CRT (of the true pool table)
and the system is calibrated to synthesize images for the left eye as
shown in Figure (c). The calibration stays constant and
the user can freely move while the synthetic overlays rotate to
compensate with the head rotation. The system, however, requires very
accurate, frame to frame tracking of the pool table to estimate its 3D
pose and structure (which is necessary for synthesizing the 3D
overlays).
Figure: Calibration of 3D Output
We are also investigating more sophisticated methods for strategy and planning which would consider higher-order effects such as rebounds, multiple collisions as well as preparatory placement for subsequent shots.
Other sensors could be added to the system to increase precision. For example, if a laser pointer could be mounted in the pool cue, we could determine the exact impact point on the cue ball and compute english and spin to predict more complex trajectories. Also, tracking the pool stick would allow the system to realize when its suggested shot is being disregarded so that it may assist the user in completing another shot.