Biz & IT —

Augmented reality gets to work—and gets past the “Glassholes”

Mobile tech, Internet of things, cloud delivers info to workers' fingers, eyeballs.

The "Sword of Damocles" head-mounted display, the original augmented reality headset, circa 1968.
Enlarge / The "Sword of Damocles" head-mounted display, the original augmented reality headset, circa 1968.
Ivan Sutherland

Augmented reality (AR) is a technology that has been on the cusp of becoming the next big thing for over 20 years. But the technology—the projection of data or digital imagery over real-world objects—has largely remained the stuff of fighter cockpits at the high end and of mobile games and art projects on the low. The promise of Google Glass—real augmented reality for the masses—failed to materialize.

That doesn't mean the technology won't fly at all. While many organizations experimented with Glass, other devices already in the hands—and on the heads—of companies and software developers have been pushing forward augmented reality in multiple industries. Work is being done today to integrate corporate cloud applications and data from intelligent machines connected to the "Internet of Things" into applications for mobile and wearable devices. And all this could help make humans on the factory floor, on the flight line, in hospitals, and in the field more effective and efficient. With Microsoft's HoloLens promising a standard development platform for AR, the cost of building those applications could plummet in the next few years.

At the same time, there are alternatives to traditional AR that can be just as effective in some cases—information systems tied to mobile devices by geolocation, proximity to systems, and other triggers that don't require the data to be in your face. As back-end analytics systems begin to drive things like when maintenance is done on systems and matching up the right person to the job, having an augmented view of the world doesn't necessarily mean looking at it through a piece of glass.

"It's going to take a while for consumers to adopt these things," said Arnie Lund, Technology Leader for the Connected Experience Labs (CEL) research organization at GE Software. "But when we look in the industrial space, there are incredible opportunities to enhance productivity with these technologies and open up new kinds of experiences with the evolution of [mobile] devices."

Through a glass, not-so-darkly

A mechanic wearing a tracked head-worn display performs a maintenance task on a Rolls Royce DART 510 Engine using the ARMAR system developed by researchers at Columbia University.
Enlarge / A mechanic wearing a tracked head-worn display performs a maintenance task on a Rolls Royce DART 510 Engine using the ARMAR system developed by researchers at Columbia University.

Though it wasn't yet known by that name, basic AR was first implemented as part of a head-mounted display called "The Sword of Damocles" by computer scientist Ivan Sutherland at MIT's Lincoln Laboratory starting in 1966. Sutherland's "ultimate display" could be used for basic virtual reality or augmented reality—the binocular displays were partially see-through—but it was limited to wireframe graphics and was tethered to a mechanical arm to provide motion tracking. It would be nearly 30 years before AR got to the point where it could be used practically.

There have been a number of studies in the use of AR for manufacturing and repair tasks. Over the last 20 years, AR has been used by Boeing to construct 747s, by astronauts to help maintain the International Space Station, and in "heads up" displays and helmet reticles for military aircraft. But with the propagation of mobile devices with wireless connectivity, cameras, geolocation and gyroscopes, AR has become relatively more accessible to developers.

Cloud computing technology has helped raise the bar, allowing smaller devices to outsource much of the heavy lifting. New cloud-based software systems designed for mobile devices, such as Qualcomm's Vuforia, have made it easier to offload object recognition and tracking tasks from devices and tie them to other computing resources in the cloud. Other networked software developer kits, such as OptiTrack's NatNet software developer kit, have already been used in trials of AR for technical tasks, such as the Augmented Reality for Maintenance and Repair (ARMAR)research done by US Army and Columbia University researchers Steve Henderson and Steven Feiner.

But augmented reality doesn't necessarily have to use see-through glasses to be effective. There are applications in the field today that do almost all of what most industrial applications require of AR—with no headset required.

"Just enough AR"

One of those is iQagent, a set of mobile and server applications from iQuest, an Alpharetta, Georgia based software company that focuses on plant automation and control systems. Bob Meads, president of iQuest, told Ars that iQagent is "just enough augmented reality" for the company's industrial customers——while it doesn't try to overlay visualizations or images on top of the real world through an iPad's camera, it does associate data with locations and machinery.

iQagent is a sort of first step into the cloud for AR in that it uses a server application back-end as a broker to a variety of different data sources already in most manufacturing plants. It uses ODBC connections to database servers, the OPC Data Access (OPC DA) specification for pulling data from human-machine interface (HMI) and supervisory control and data acquisition (SCADA) systems, and connections to unstructured data on Web servers and Microsoft SharePoint servers.

The information on each piece of instrumented equipment—both live and historical, as well as maintenance instructions, videos, and other content—can be tied to each physical piece of equipment. The data can then be accessed from the mobile client by pointing its camera at a QR code associated with it. It could also be linked to geographic locations. While iQagent is used mostly in manufacturing environments now, the same capabilities could be applied to field service or repair operations or any other task where a technician needs access to live or historical data about a piece of equipment.

Another way to link data to the physical world is through beacons. Radio frequency identification (RFID) tags and newer technologies such as Bluetooth Low Energy (BLE) 4.0 allow objects to essentially broadcast their identity and location—and potentially a lot more information—which can be pulled up into a corporate cloud application for tracking and analysis.

Passive RFID has been used in sensing of equipment and inventory for over a decade, often to help keep track of high-value tools on the shop floor and keep them from going out the door. But new wide-area RFID systems can be used to track tagged tools, products, and systems across facilities, helping a technician or worker with a mobile application to track down the tool they need for a job or a shipment that went astray in receiving.

BLE 4.0, however, could take that capability further, broadcasting live data to mobile devices that "pair" with a piece of equipment to give on-the-spot diagnostics or simply help someone find the right access hatch. The new Simblee chip from RF Digital is an example of this, and with a pre-built mobile app that bridges between the chip's BLE connection and a connection to a cloud or Web application, it could be used as an enhancement or substitute for some types of augmented reality applications.

The next level

When beacons and cloud data are tied to full augmented reality, however, the results could have an even bigger effect on the productivity and efficiency of workers. Image processing tied to beacons and geolocation, for example, could help technicians get a quick grasp of the status of various pieces of equipment, find the equipment that needs their attention, and locate what they need to fix the problem. For example, researchers at ARATLab at the National University of Singapore have combined augmented reality and RFID for tasks like AR-assisted assembly of objects with embedded RFID tags, showing people which order parts go together and how.

A demonstration of AR-assisted assembly using RFID to identify objects at ARATlabs, National University of Singapore.

Many of these pieces are being pulled together in GE Software's labs in San Ramon. GE's Lund is overseeing research that pulls historic and real-time machine data from an enterprise cloud platform down to a mobile device—be it an AR headset or a tablet computer. Lund and his team demonstrated an application for Ars at GE Software's headquarters in San Ramon that tied not only into information about manufacturing machinery and its maintenance status, but also into analytics that could help dispatch the right technician to make repairs and guide the technician by beacons to the right tools for the job.

Many organizations, including GE's CEL, sank time and effort into working with Google Glass as a testbed for these technologies. Microsoft's recently announced HoloLens and Windows 10 Holographic operating system promise to do more than pick up where Google Glass left off, however, because they bring augmented reality capabilities to a development platform that many developers in the space are already comfortable with. iQuest's Meads said that his company was already in the process of shifting development efforts toward Windows tablets, and while he hasn't had the opportunity to lay hands on a HoloLens yet, he said it would be a natural extension of that effort.

But for now, AR is less of revolution than a simple evolution of industrial computing. As technologies like HoloLens become more mainstream, true AR may seep into more and more tasks in manufacturing, healthcare, and other industries, but "just enough" AR will likely be part of more and more of our daily work experience.

Channel Ars Technica