The paper presented the first outdoor augmented reality system using GPS position
tracking with a see-through head-worn display. Before the era of smartphones,
ubiquitous GPS, and Wi-Fi, the Touring Machine, which included a backpack stuffed
with electronics, let users navigate Columbia’s campus, overlaying names of buildings
and academic departments on buildings viewed through the user’s head-worn display,
and allowing users to select a department to explore its website on a hand-held display
wirelessly connected to the internet.
Cited 1257 times as of November 2017, the paper was authored by Professor Steven Feiner, Blair MacIntyre (PhD ’99, and now Professor in the School of Interactive Computing, Georgia Tech), Tobias Höllerer (PhD ’04 and now Professor of CS, UC Santa Barbara), and Tony Webster (now Lecturer in Finance, SEAS and Clinical Associate Professor of Real Estate, NYU). Both MacIntyre and Höllerer were Feiner’s PhD students at the time of the paper.
Six SEAS students traveled this summer to Ghana to train university students to lead CS outreach that will impact hundreds of Ghanian students. The trip was organized by a nonprofit headed by Chelsey Roebuck (BS’10).
The paper, widely covered in the press, describes an automatic method for error-checking thousands to millions of neurons in a deep-learning neural network. Authors are Kexin Pei, Yinzhi Cao (Lehigh), Junfeng Yang, Suman Jana.
Developed by Kexin Pei, Yinzhi Cao (Lehigh), Junfeng Yang, and Suman Jana, DeepXplore error-checks thousands to millions of neurons in deep learning neural networks. In tests, DeepXplore found thousands of bugs missed by previous methods.
To uniquely identify and encode information in printed objects, Columbia researchers Dingzeyu Li, Avinash S. Nair, Shree K. Nayar, and Changxi Zheng have invented a process that embeds carefully designed air pockets, or AirCode tags, just below the surface of an object. By manipulating the size and configuration of these air pockets, the researchers cause light to scatter below the object surface in a distinctive profile they can exploit to encode information. Information encoded using this method allows 3D-fabricated objects to be tracked, linked to online content, tagged with metadata, and embedded with copyright or licensing information. Under an object’s surface, AirCode tags are invisible to human eyes but easily readable using off-the-shelf digital cameras and projectors.
The AirCode system has several advantages over existing tagging methods, including the highly visible barcodes, QR codes, and RFID circuits: AirCode tags can be generated during the 3D printing process, removing the need for post-processing steps to apply tags. Being built into a printed object, the tags cannot be removed, either inadvertently or intentionally; nor do they obscure any part of the object or detract from its visual appearance. Invisibility of the tags also means that the presence of information can remain hidden.
“With the increasing popularity of 3D printing, it’s more important than ever to personalize and identify objects,” says Changxi Zheng, who helped develop the method. “We were motivated to find an easy, unobtrusive way to link digital information to physical objects. Among their many uses, AirCode tags provide a way for artists to authenticate their work and for companies to protect their branded products.”
One additional use for AirCode tags is robotic grasping. By encoding both an object’s 3D model and its orientation into an AirCode tag, a robot would just need to read the tag rather than rely on visual or other sensors to locate the graspable part of an object (such as the handle of a mug), which might be occluded depending on the object’s orientation.
AirCode tags, which work with existing 3D printers and with almost every 3D printing material, are easy to incorporate into 3D object fabrication. A user would install the AirCode software and supply a bitstring of the information to be encoded. From this bitstring, AirCode software automatically generates air pockets of the right size and configuration to encode the supplied information, inserting the air pockets at the precise depth to be invisible but still readable using a conventional camera-projector setup.
How the AirCode system works
The AirCode system takes advantage of the scattering of light that occurs when light penetrates an object. Subsurface scattering in 3D materials, which is not normally noticeable to people, will be different depending on whether the light hits an air pocket or hits solid material.
Computational imaging techniques are able to differentiate reflective light from scattered light and decompose a photo into two separate components: a direct component produced by reflected light, and a global component produced from the scattered light waves that first penetrate an object. It’s this global component that AirCode tags manipulate to encode information.
Decomposing an image into direct and global components. Green vectors in this schematic represent the direct component produced from light that reflects off the surface; this component resembles the majority of light rays perceived by our eyes. Orange vectors represent the global component produced by light that first penetrates an image before reaching the camera; the global component is barely visible but can be isolated and amplified.
Other innovations are algorithmic, falling into three main steps:
Analyzing the density and optical properties of a material. Most plastic 3D printer materials exhibit strong subsurface scattering, which will be different for each printing material and will thus affect the ideal depth, size, and geometry of an AirCode structure. For each printing material, the researchers analyze the optical properties to model how light penetrates the surface and its interactions with air pockets.
“The technical difficulty here was to create a physics-based model that can predict the light scattering properties of 3D printed materials especially when air pockets are present inside of the materials,” says PhD student Dingzeyu Li, a coauthor on the paper. “It’s only by doing these analyses were we able to determine the size and depth of individual air pockets.”
Analyzing the optical properties of a material is done once with results stored in a database.
Constructing the AirCode tag for a given material. Like QR codes, AirCode tags are made up of cells arranged on a grid, with each cell representing either a 1 (air-filled) or a 0 (filled with printing material), according to the user-supplied bitstring. Circular-shaped markers, easily detected by computer vision algorithms, help orient the tag and locate different cell regions.
Unlike QR codes, which are clearly visible with sharp distinctions between white and black, AirCode tags are often noisy and blurred, with many levels of gray. To overcome these problems, the researchers insert predefined bits to serve as training data for calibrating in real time the threshold for classifying cells as 0s or 1s.
AirCode layout and its corresponding image: data bits (1s are air-filled cells, 0s are cells with solid material) for encoding user-supplied information; markers and rotation cells for orientation; classifier training bits for on-the-fly calibration. Size is variable; a 5cmx5cm tag stores about 500 bits.
Detecting and decoding the AirCode tag. To read the tag, a projector shines light on the object (multiple tags might be used to ensure the projector easily finds a tag) and a camera captures images of the object. A computer vision algorithm previously created by coauthor Shree Nayar and adapted for AirCode tags separates each image into its direct component and global component, making the AirCode tag clearly visible so it can be decoded.
While the AirCode system has certain limitations—it requires materials to be homogeneous and semitransparent (though most 3D printer materials fit this description) and the tags become unreadable when covered by opaque paint—tagging printed objects during fabrication has substantial benefits in cost, aesthetics, and function.
Interviewed by Columbia Technology Ventures (CTV), Allen discusses his recent work on surgical robots and how incorporating big data, machine learning, and computer vision is contributing to smarter robots able to cope with the unexpected.
The timing isn’t great—mid-semester with early projects coming due and midterms beginning—but still they come; 37 from Columbia and Barnard traveled to Orlando earlier this month to join 18,000 other women in tech for the Grace Hopper Celebration, an annual gathering co-produced by AnitaB.org (formerly the Anita Borg Institute) and the Association for Computing Machinery.
For three days (Oct 4-6) attendees listened to 16 keynote speakers—among them Melinda Gates, Fei-Fei Li (Professor and Director of Stanford’s AI Lab and Chief Scientist at Google Cloud), and Megan Smith (Former US Chief Technology Officer). They signed up for technical panels on AI, wearable technologies, data science, software engineering, and dozens of other innovative technologies. They networked with their peers and listened to pitches from recruiters who flock to the event.
But for many, the main draw is just being among so many other women who share their interests in computer science and engineering. In fields dominated by men, the Grace Hopper Celebration (GHC) is one of the few tech venues where women run the show. Here they are the speakers, panelists, and attendees, sharing what they love about technology and what they hope to accomplish in their careers or in their research. They share also stories of workplace discrimination, slights, and sometimes blatant sexism as well as tangible recommendations for what works to keep women in technology.
Click image to hear this year’s GHC keynote speeches. Many speakers told inspiring stories of overcoming adversity to pursue their careers in computing and technology.
Columbia CS major Tessa Hurr, attending for the fourth time, describes it this way: “GHC is a community of women who are there to support one another and lift one another up and encourage one other to pursue a career in STEM.” A senior about to embark on a career, she especially wanted to hear from women about their work experiences. “Coming from Columbia, where the engineering school and computer science department have done a lot of work to balance the ratio of males to females, you see a lot of other women and you don’t feel alone. But in industry, you see the problem of gender imbalance so glaringly. Being at GHC, I know there are support systems if I need them.”
Women at GHC may be excited about supporting other women in technology, but they’re just as enthusiastic about technology itself and the good it can do in the world. Says Hurr, “Sometimes when you’re learning different concepts in class you don’t necessarily see how they translate over to the real world; GHC tech talks help bridge that gap so you better understand how you can have an impact on the world and work towards social good through tech.”
Myra Deng, a CS student attending for the first time, appreciated the emphasis on new technologies, especially AI. A talk by keynote speaker Fei-Fei Li linking AI and diversity was especially inspiring to Deng, who is on the Intelligent Systems track. “Professor Li talked about how AI is supposed to represent the entire human experience but you can’t really model or build that with just a small section of the human population. Diversity isn’t just being nice in the workplace; it’s essential to getting the technology right.”
This mix of technology and support system is a powerful thing, and GHC has been growing by leaps and bounds. In four years, GHC has grown from 8000 to 18,000 participants.
Many attend by taking advantage of scholarships offered by some of the big tech companies. “If the concern is finances, there are lots of resources, including a Github page listing scholarships,” says Julia Di, president of WiCS, which also sponsors students. This year WiCS raised enough funding to send 16 students, though only six were able to purchase tickets in the few hours it took before tickets sold out. Next year, WiCS may follow the lead of tech companies and make a donation to pre-purchase tickets.
Some scholarships require students take the entire week off, not just the three days of the conference, making GHC even more of a time commitment as students scramble to get school work done ahead of time, and scramble again to catch up when they return to campus. That so many do shows how much importance they attach to continuing in tech and supporting others who want to do the same.
Deng encourages women to make the most of the opportunity offered by GHC. “Every now and then, it’s good to zoom out from school and see what’s going on in the world. At GHC you meet so many incredible people you might not otherwise meet. I came back a lot more motivated because I know what I’m working on is important. It’s why I’m in Tech. You can always catch up on school work later.”
Dean Boyce's statement on amicus brief filed by President Bollinger
President Bollinger announced that Columbia University along with many other academic institutions (sixteen, including all Ivy League universities) filed an amicus brief in the U.S. District Court for the Eastern District of New York challenging the Executive Order regarding immigrants from seven designated countries and refugees. Among other things, the brief asserts that “safety and security concerns can be addressed in a manner that is consistent with the values America has always stood for, including the free flow of ideas and people across borders and the welcoming of immigrants to our universities.”
This recent action provides a moment for us to collectively reflect on our community within Columbia Engineering and the importance of our commitment to maintaining an open and welcoming community for all students, faculty, researchers and administrative staff. As a School of Engineering and Applied Science, we are fortunate to attract students and faculty from diverse backgrounds, from across the country, and from around the world. It is a great benefit to be able to gather engineers and scientists of so many different perspectives and talents – all with a commitment to learning, a focus on pushing the frontiers of knowledge and discovery, and with a passion for translating our work to impact humanity.
I am proud of our community, and wish to take this opportunity to reinforce our collective commitment to maintaining an open and collegial environment. We are fortunate to have the privilege to learn from one another, and to study, work, and live together in such a dynamic and vibrant place as Columbia.
Sincerely,
Mary C. Boyce
Dean of Engineering
Morris A. and Alma Schapiro Professor
Add to GMail