Two professors in the Computer Science Department at Columbia University have been named Fellows of the Institute of Electrical and Electronics Engineers (IEEE): Steven Feiner for “contributions to augmented reality and computer graphics,” and Salvatore Stolfo for “contributions to machine learning-based computer security.” IEEE is the world’s leading professional association for advancing technology for humanity, and the IEEE Fellow is the highest grade of IEEE membership, limited every year to one-tenth of one-percent of the total voting membership.
Steven Feiner is one of the earliest researchers to investigate and publish scholarly papers on augmented reality (AR), in which computer-generated media—text, graphics, video, sound—are integrated and geometrically aligned with a user’s experience of the real world, one example being by overlaying 3D virtual objects on what the user sees. Feiner joined the Columbia faculty in 1985 and started the Computer Graphics and User Interfaces Lab, addressing a broad range of topics in designing and interacting with synthesized pictures. Several years later, Feiner extended that work to virtual reality, in which virtual objects are viewed on their own, independent of the real world, and since 1990, he and his students have been investigating AR interaction and visualization techniques, and applications of AR to tasks such as equipment maintenance.
The lab’s first AR systems used a “see-through” head-worn display that Feiner and his students assembled to optically combine the real and virtual worlds, connected to desktop and deskside computers. Their earliest project augmented the user’s view of a small flat-panel display running a desktop window system by immersing it in a larger virtual environment, viewed through the head-worn display. The user could see the windows on the flat-panel display surrounded by the outlines of windows above, below, and to the sides of the flat panel. Feiner called this a “hybrid user interface” because it combined different display technologies in a way that benefited from their respective advantages.
The lab’s next project tracked in 3D selected parts of a laser printer, overlaying maintenance instructions directly on relevant parts as the user manipulated them. This posed questions central to the effective design of many kinds of AR experiences: What virtual content should be presented to help the user and what should be suppressed to avoid confusion? How should virtual content be positioned and sized relative to physical objects? How should rendering style be controlled to make content easy to understand? How should the user’s attention be directed toward content that was outside the field of view (e.g., to make the user move or turn to see it)? Feiner’s team developed rules that automatically designed and generated virtual content based on the task to be performed, the positions and orientations of the physical objects being manipulated, and the position and orientation of the user’s head.
Feiner soon decided that it was essential to explore how the lab’s AR systems could be made mobile and be taken outdoors. In 1996, Feiner and his students created the “Touring Machine,” which included a head-tracked, see-through, head-worn display, a hand-held tablet, and a backpack containing a computer and a very accurate kind of GPS. Both the backpack computer and tablet had digital radio modems for wireless access to the internet.
Before the era of smartphones, ubiquitous GPS, and Wi-Fi, and sixteen years before Google Glass in 2012, the Touring Machine allowed a user to walk around campus, be directed to a building or academic department of interest, view a building’s name and its department names overlaid on that building, and call up a department’s web page on the tablet.
Over time, the systems that Feiner and his lab built have become smaller and lighter weight, often being replaced with or built on top of purely commercial hardware and software. With support from NSF, ONR, and industry, the lab has had productive collaborations with colleagues in cognitive science, architecture, journalism, archaeology, and medicine from rehabilitation to surgery, investigating how to use augmented reality to assist users in performing many kinds of skilled tasks. Approaches designed for one person are complemented by ones that address multiple people, whether co-located or remote, and extended to asymmetric scenarios in which a remote expert working entirely in virtual reality can assist a local user performing a task in the real world guided by augmented reality. In most of the lab’s research, members create and refine new techniques, and evaluate them by designing, running, and analyzing user studies to compare those techniques with existing ones.
Being named an IEEE Fellow is his latest award. Just this fall, the paper introducing the Touring Machine received the Early Innovator Award from the International Symposium on Wearable Computers (ISWC), celebrating the paper from the first ISWC conference in 1997 that has had the most impact, and this year also Feiner received the Career Impact Award from the IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Three years ago, Feiner received the IEEE VGTC 2014 Virtual Reality Career Award, and he was elected to the CHI Academy in 2011. Together with his students, he has won the ACM UIST 2010 Lasting Impact Award for early work on supporting 2D windows and hypermedia in augmented reality, and best paper awards at ACM UIST, ACM CHI, ACM VRST, IEEE ISMAR, and IEEE 3DUI.
“It is a great honor to be elevated to IEEE Fellow, and one that I owe in large part to the incredible students and colleagues with whom I have had the pleasure to work,” says Feiner. “It is fun to be in a field in which we must continually remind ourselves that the technologies we use now will, in just a few years, be eclipsed in power by ones that are a tiny fraction of the size and price. Our job is to understand how to use these ever more powerful tools to make the future a better place than the present for all of us.”
Steven Feiner did his undergraduate degree at Brown in 1973 and received his
PhD in Computer Science also from Brown in 1987
Computer security is not just about technology and systems. It is also about the people who use those systems and how their vulnerable behaviors can be exploited. That premise informs much of Salvatore Stolfo’s research into computer and network security. A Columbia professor since 1979, Stolfo starts with the assumption that data will always be a target. The only unknown is the specific path hackers will take to breach security.
One previously unforeseen path, exposed by Stolfo working with his student Ang Cui, is through embedded devices such as printers, routers, drones, webcams, medical devices, and the growing legions of IoT devices, very few of which come with built-in security. Stolfo and Cui made headlines by demonstrating that certain HP LaserJet printers would accept firmware updates without verifying their authenticity, allowing hackers to upload their own software to take over control of the printer. Later Stolfo and Ang would find serious vulnerabilities in Cisco’s VolP phones that could turn the phones into listening devices.
Stolfo did not start out concentrating in security. An early proponent of artificial intelligence and machine learning, Stolfo in the early 1980s developed a large-scale expert data analysis system, called ACE (Automated Cable Expertise) that was likely the first system to combine rule-based inference (an AI expert system) with a relational database management system; ACE was widely distributed to a telephone wire centers across the US. Stolfo also created machine learning algorithms to detect patterns of fraudulent credit card transactions, research that was adopted throughout the financial industry. Stolfo also co-developed the DADO computer system, a “fifth generation computer” sponsored by DARPA’s high performance parallel computing initiative. It was a fully functional 1023-processor for performing acoustic analyses and keyword spotting capabilities, and the parallel broadcast and resolve/report function that it introduced apparently influenced part of the design of the IBM Blue Gene parallel computer, is the basis of the Apache Hadoop MapReduce functionality for processing very large datasets.
DADO had one other distinction: it was the first invention claimed by Columbia University for ownership of a faculty member’s intellectual property under the Baye Dole Act. With outside investors, the technology was later commercialized and adapted for large-scale speech recognition services.
Combining research with entrepreneurship is a hallmark of Stolfo’s career. It’s a way of being able to experiment with new technology in a lab with students while being guided at the same time by practical, real-world concerns.
With funding from DARPA’s Cyber Panel program in the early 90s, Stolfo and his students began work on detecting network attacks and e-commerce fraud; it was the challenge of detecting the activities of creative but malicious fraudsters that Stolfo became hooked on security. In 1996 he established the Intrusion Detection System (IDS) lab at Columbia, which has received many millions of dollars in research funds. The lab would pioneer the use of data analysis and machine learning techniques for computer security and created the field of Adversarial Machine Learning. With support from DARPA and other federal agencies, Stolfo launched two start-ups to commercialize his research in the areas of active user authentication, deception security, and embedded systems security. Red Balloon Security commercialized the Symbiote technology for embedded systems and was named one of Popular Science’s Best of What’s New for 2016. Allure Security Technology has received millions of dollars in funding from the US government and private investors to innovate technology to stop data breaches.
To date, Stolfo has been granted over 73 patents and has published several books and well over 250 scientific papers on many diverse topics in computer science: parallel computing, AI knowledge-based systems, data mining, computer security and intrusion detection systems. Two of his security papers are among the top 20 most-cited security papers; several others have earned best paper awards, including the RAID Most Influential Paper and Usenix Security Distinguished Paper awards. His research has been supported by DARPA, NSF, ONR, and numerous companies and state and federal agencies.
“I’m truly honored to be elevated to IEEE Fellow,” says Stolfo. “My science has been driven by making an impact both inside and outside the academy, and I’ve had the good fortune to work alongside my colleagues and students towards that goal. I believe the internet should be secure for enterprises, and for everyday people in their everyday lives. I look forward to continuing my work to making the internet safe.”
Salvatore Stolfo received his Ph.D. from The Courant Institute of Mathematical Sciences,
New York University, in 1979.