
Research Fair Spring 2025
The Spring 2025 Research Fair will be held on Thursday, January 23rd, from 12:00 to 14:00 in Carlton Commons. This is an opportunity to meet faculty and Ph.D. Students working in areas of interest to you and possibly work on these projects.
Please read their requirements carefully! There will be a couple of Zoom sessions and recordings available – see below for all details. |
In Person
Faculty/Lab: Prof. Vishal Misra, Vice Dean of Computing and Artificial Intelligence
Brief Research Projects Description & Required/preferred prerequisites and qualifications:
1) AI Course Assistant Project
The AI Course Assistant project aims to enhance education by creating an intelligent teaching assistant for university courses. The project involves designing an NLP-powered chatbot to address student queries, provide guided learning, and generate study materials from lecture transcripts. Advanced instructor tools will help in course planning and question generation, integrating seamlessly with existing learning management systems. Additionally, the project ensures scalability for university-wide deployment while focusing on cost and resource optimization.
Skill Sets Required: Proficiency in Python, experience with NLP (Hugging Face), machine learning (TensorFlow/PyTorch), web development (React/HTML), RESTful APIs, database management, and familiarity with cloud platforms like AWS or GCP, Git.
2) AI Advising Assistant Project
Intuitive and accurate chatbot system that provides personalized support, helping students efficiently find answers to their questions. This Project is designed to streamline information retrieval through a modular chatbot system that employs retrieval-augmented generation (RAG) for accurate and efficient responses. By integrating OpenAI vector embeddings, student-specific FAQ filtering, FAISS similarity search, and concise prompting, the system delivers tailored answers while ensuring scalability through its modular architecture and shared knowledge bases. Analytics monitor usage and refine chatbot accuracy, while cost-tracking mechanisms, such as API monitoring and caching strategies, optimize resource utilization and reduce repeated queries. Additionally, an out-of-scope query detection feature leverages unsupervised learning on embeddings to identify queries beyond the knowledge base, ensuring continuous adaptability and accurate performance for high-traffic periods.
Skill Sets Required: Strong Python skills, expertise in NLP, vector databases (Faiss/Pinecone), RAG techniques, chatbot frameworks (Rasa), web development, RESTful APIs, and serverless cloud architectures.
3) Find the Expert Tool
The Find the Expert tool is designed to help users locate researchers with relevant expertise. It leverages NLP-based text analysis and advanced search algorithms to create detailed researcher profiles. The system includes web scraping for data collection, topic modeling for profiling, and an intuitive search interface with filtering and sorting options. Future goals include integrating with university systems, automating profile updates, and developing analytics for search result quality. The tool emphasizes data privacy and security while ensuring scalability and efficient resource use through caching and monitoring strategies.
Skill Sets Required: Proficiency in Python, experience in web scraping, NLP, search algorithms, database management, academic research metrics, RESTful APIs, and cloud deployment.
Faculty/Lab: Prof Simha Sethumadhavan
Brief Research Project Description: 1) Computer Architecture related projects 2) LLM applications for security
Required/preferred prerequisites and qualifications: For 1) must have taken 4824, and for 2) must have experience or ability to fine tune LLMs. Must have also taken Security 1
Faculty/Lab: Dr. Corey Toler-Franklin, Graphics Imaging & Light Measurement Lab (GILMLab)
Brief Research Project Description: https://coreytolerfranklin.com/gilmlab/
AI for Computer Graphics
Physics-based Material Simulation for Real-time Rendering, Animation and Gaming
Seeking graduate students with experience in graphics and/or physics and machine learning for projects focused on physics-based photorealistic material simulation for computer graphics.
AI for Cancer Detection
Identifying Cancer Cells and Their Biomarker Expressions
Cell quantitation techniques are used in biomedical research to diagnose and treat cancer. Current quantitation methods are subjective and based mostly on visual impressions of stained tissue samples. This time-consuming process causes delays in therapy that reduce the effectiveness of treatments and add to patient distress. Our lab is developing computational algorithms that use deep learning to model changes in protein structure from multispectral observations of tissue. Once computed, the model can be applied to any tissue observation to detect a variety of protein markers without further spectral analysis. The deep learning model will be quantitatively evaluated on a learning dataset of cancer tumors.
AI for Neuroscience
Deep Learning for Diagnosing and Treating Neurological Disorders
Advances in biomedical research are based upon two foundations, preclinical studies using animal models, and clinical trials with human subjects. However, translation from basic animal research to treatment of human conditions is not straightforward. Preclinical studies in animals may not replicate across labs, and a multitude of preclinical leads have failed in human clinical trials. Inspired by recent generative models for semi-supervised action recognition and probabilistic 3D human motion prediction, we are developing a system that learns animal behavior from unstructured video frames without labels or annotations. Our approach extends a generative model to incorporate adversarial inference, and transformer-based self-attention modules.
AI for Quantum Physics & Appearance Modeling
Quantum Level Optical Interactions in Complex Materials
The wavelength dependence of fluorescence is used in the physical sciences for material analysis and identification. However, fluorescent measurement techniques like mass spectrometry are expensive and often destructive. Empirical measurement systems effectively simulate material appearance but are time consuming, requiring densely sampled measurements. Leveraging GPU processing and shared super computing resources, we develop deep learning models that incorporate principles from quantum mechanics theory to solve large scale many-body problems in physics for non-invasive identification of complex proteinaceous materials.
AI for Multimodal Data & Document Analysis
Deciphering Findings from the Tulsa Race Massacre Death Investigation
The Tulsa Race Massacre (1921) destroyed a flourishing Black community and left up to 300 people dead. More than 1000 homes were burned and destroyed. Efforts are underway to locate the bodies of victims and reconstruct lost historical information for their families. Collaborating with the Tulsa forensics team, we are developing spectral imaging methods (on-site) for deciphering information on eroded materials (stone engravings, rusted metal, and deteriorated wood markings), novel multimodal transformer networks to associate recovered information on gravestones with death certificates and geographical information from public records, and graph networks for reconstructing social networks.
Required/preferred prerequisites and qualifications: High priority to students with computer graphics and/or physics experience.
General requirements: Python and/or C/C++, Machine Learning experience
Faculty/Lab: Prof. Asaf Cidon
Brief Research Project Description: Retrieval-augmented generation (RAG) enhances LLMs with external knowledge, improving context relevance and reducing hallucinations. RAG uses vector databases to index documents as semantic vectors, enabling efficient retrieval for user queries. Unlike traditional indexes, vector databases often rely on in-memory graph indexes, which become challenging to manage at billion-scale vectors. This project explores the systems challenges of large-scale vector databases and investigates how emerging hardware can help. We will study existing research, evaluate performance through experiments, and implement new ideas.
Required/preferred prerequisites and qualifications:
* Interested in systems programming and familiar with C++
* Have at least one year left in your program
* Have taken Operating Systems class
* Familiar with basic graph algorithms
* Can contribute 15 hours a week
Faculty/Lab: Prof. Junfeng Yang
Brief Research Project Description: *LLM for security analysis*: Security is a critical aspect of software systems, but vulnerabilities can compromise their integrity. Program analysis and testing techniques play a key role in strengthening software by identifying and addressing vulnerabilities before attackers can exploit them. This project aims to advance conventional program analysis techniques by leveraging language models, pushing the boundaries of vulnerability discovery, software debugging, and beyond. We will investigate how LLM could help with static analysis (e.g., taint analysis), symbolic execution, and fuzz testing.
Required/preferred prerequisites and qualifications: Proficient in at least one programming language; Knowledgeable of LLM/AI and/or program analysis
Faculty/Lab: Prof. Eugene Wu, Prof. Lydia Chilton, Daniel Osgood
Brief Research Project Description: We are seeking student programmers to assist with interface and database development for an NSF project with the Climate School, related to climate risk crowdsourcing and visualization – project website here: https://columbia-desdr.github.io/
Required/preferred prerequisites and qualifications: Preference for students with prior experience with FLASK, Postgres, Python or DBT. Experience with R is also a plus. Preference for students with prior experience in working as part of a development team using GitHub.
Faculty/Lab: Prof. Yunzhu Li – Robotic Perception, Interaction, and Learning Lab (RoboPIL)
Brief Research Project Description:
Project #1: Develop mobile robots capable of long-horizon robotic manipulation tasks using foundation models
Project #2: Learning simulatable digital twins from and for real-world robotic interactions: (1) Collect real world videos from teleoperation or internet data, (2) Train object dynamics models and couple learned model with physics-based simulators
Required/preferred prerequisites and qualifications:
Required:
1. Strong programming skills in one or more programming languages such as Python, C++, etc.
2. Understanding basic concepts of ROS
3. Understanding basic concepts of robotics and computer vision, e.g. robot kinematics, pinhole camera model, etc.
4. Experience with machine learning and computer vision, especially 3D computer vision
Preferred:
1. Experiences of developing robotic systems and using ROS
2. Understanding common deep learning algorithms, e.g. MLP, CNN, etc.
3. Understanding the basics of physics-based simulation
4. Experience of working with robots
Faculty/Lab: Prof. Julia Hirschberg, Speech Lab
Brief Research Project Description:
Our work focuses on developing advanced techniques, models, and strategies for speech applications, with a primary emphasis on assessing speech signals. Examples include:
• Automated Health Assessment: Building systems that analyze speech signals to evaluate health conditions and diagnose diseases, such as respiratory illnesses and dysarthria.
• Pronunciation Assessment: Creating models to help non-native English speakers improve their English pronunciation.
• Speech Assessment for Broader Applications: Utilizing automated speech assessment frameworks to evaluate and enhance the performance of speech applications, such as assessing the outputs of voice conversion systems.
• Noise Robust Speech Emotion Recognition: Investigating the impact of noise on emotion recognition performance and developing strategies to enhance the robustness of emotion recognition systems in noisy environments.
Required/preferred prerequisites and qualifications: Familiarity with fundamental machine learning concepts. Proficiency in Python, with experience in PyTorch, including the ability to design, train, and evaluate machine learning models.
Faculty/Lab: Prof. Sharon Di
Brief Research Project Description: At the intersection of selfdriving, AI, LLM, VLM, and game theory. https://docs.google.com/forms/d/e/1FAIpQLSdweArZn_m8pT1u7PPQef8TPHywrBoaTpIVvlj1EjozfCDygQ/viewform
Required/preferred prerequisites and qualifications: background in ML, RL, CV
Faculty/Lab: Prof. Asaf Cidon & Prof. Junfeng Yang
Brief Research Project Description: Satellite-backed services have become an essential component of everyday life, in areas such as navigation, Internet connectivity and imaging. The collapsing cost of launching to space has disrupted the way satellites are deployed, shifting the industry from a model of few expensive fault-tolerant high-orbit satellites to arrays of commodity low-cost SmallSats in low-Earth orbit. However, satellite software hasn’t kept up with the hardware trends, and missions are still using the ad-hoc flight software infrastructure built for expensive one-off missions in high-altitude orbits, wherein operators manually deploy software to each satellite individually. This approach is woefully inadequate in the new emerging SmallSat operational model, where an operator needs to manage hundreds of “wimpy” satellites with varying hardware capabilities under intermittent communication. The goal of this project will be to design a novel software runtime for fleets of heterogeneous satellites built on top of the eBPF ISA.
Required/preferred prerequisites and qualifications:
classes:
– Operating Systems I
– Programming Languages
experience:
– LLVM/Clang
– eBPF
Faculty/Lab: Prof. Matthew Connelly
Brief Research Project Description:
We have two research projects:
1) Using Streamlit to improve and extend the search functions on the History Lab website: https://lab.history.columbia.edu/content/foiarchive-search
2) Using already collected Named Entity Linking data and a node.js framework to create a RAG system.
Required/preferred prerequisites and qualifications: Python; node.js, postgres, LLM, Streamlit
Faculty/Lab: Prof. Venkat Venkatasubramanian
Brief Research Project Description:
Mechanistic interpretability – Inner workings of LLMs
Data-driven AI models learn from the vast amounts of data that they are able to process. Yet, we do not know how that knowledge is organized within their vast network, though it may be a form of geometrical organization in their hyperspace. Previously, we investigated geometric relationships between the most common n-grams, training different size sparse-autoencoders to uncover “monosemanticity” (i.e., how parts of a neural network are activated by specific, single ideas), PyTorch Hooks, and feature clamping. We performed these analyses in small LLMs and now have the capacity to repeat analyses for much larger ones via a GPU cluster and aim to build visualization tools to uncover patterns of organization within their network. We also aim to explore in what ways we can infuse first principles knowledge into LLMs to build reasoner tools into this geometric organization to solve problems pertinent to science and engineering.
Prerequisites: Python programming, linear algebra, machine learning, data structures, interest in probability, statistics, and optimization, and curiosity to learn about a new field.
Accelerating drug discovery through neural-symbolic models incorporating domain knowledge
Scientists must process thousands of unstructured documents to obtain key information and infer results when researching new drugs. These documents are rich in technical information and domain-specific terms which, despite their success in other fields, large language models like ChatGPT have difficulty tackling. We developed SUSIE, an ontology-based pharmaceutical information extraction tool that is built to extract semantic triples and present them to the user as knowledge graphs (KGs). The ontology that the student will be interacting with is the Columbia Ontology of Pharmaceutical Engineering (COPE). We have previously explored different methods to interface the generated knowledge graphs and the ontology, finding the best methods for knowledge graph embedding. We are now interested in populating the ontology with the KGs and using neural-symbolic models to query the populated ontology using first-order logic.
Prerequisites: Python programming, machine learning, data structures, interest in symbolic AI tools, and curiosity to learn about a new field.
Zipf’s law for drug discovery: exploring AlphaFold 2
DeepMind made the full source code for AlphaFold2, the Nobel Prize-winning AI tool for predicting protein structures, publicly available. We aim to build tools to understand the neural network architecture and model parameters that made the breakthroughs achieved by the model possible, to then leverage them for new applications within drug discovery. We aim to uncover laws that govern the inner workings of transformer-based models generally, and AlphaFold2 specifically: is there the equivalent of a Zipf’s law for biological entities? How is biological knowledge represented within this Neural Network (for instance, with geometric reasoning)? Can we use them to then identify specific drug targets or create personalized solutions?
Prerequisites: knowledge of Python programming, machine learning, interest in geometric deep learning, and curiosity to learn about a new field.
Emergent Properties of LLMs
This project is concerned with the discovery of global properties that consistently emerge in various Transformer-based Large Language Models (LLMs). Of particular interest are general organization principles of knowledge within a “well-trained” network of this sort; to this end, we will explore the geometry and topology of latent spaces within commercial-scale networks. We will also develop sampling methods inspired by statistical physics, aiming to infer the knowledge embedded in arbitrary latent vectors.
Previous experience with LLMs, Python, and/or computer vision is required.
Emergent Properties of Energy Based Models
Energy-Based Models (EMBs) provide an alternative to the standard feed-forward neural network. Following recent work within the CRIS lab, we will develop a so-called “teleodynamic” theory of EBMs, borrowing from the principles of both thermodynamics and game theory. We are interested in the emergent properties of neural networks at equilibrium states, their behavior during inference, as well as the descent to equilibrium during training.
Previous Experience with Convolutional networks, Python, and/or computational physics is required.
Emergent Properties of Nano-Robot Swarms
This project is concerned with agent-based simulations of magnetic robot swarms, which have demonstrated success in a variety of real-world tasks. We are interested in a “teleodynamic” theory of these robot swarms, involving game-theoretic models at the individual robot level that yield emergent behaviors in many-robot swarms. Afterwards, we will implement our models in large-scale agent-based simulations. By discovering a general theory of the robot swarms, we hope to identify novel behaviors and use cases.
Previous Experience with Game Theory, Thermodynamics, and/or swarm optimization is required.
Dynamics of Tokens in LLM
This project aims to achieve a deeper understanding of Large Language Models (LLMs) and the influence of the tokens. By analyzing the findings of several research papers and Anthropic AI’s dictionary learning, it can be inferred that tokens are essentially the data points lying in an extremely high dimensional space. This vector-like property of the token can be compared to the famous example, “King – Man + Woman = Queen”.
The objective of this project is to study the embedding vector of the tokens, to understand the relationship between individual tokens and a group of tokes. By understanding that “similar tokens appear closer” in the embedding space, these embedding vectors will be studied for their clustering ability.
Finally, the token will be modeled as a system using the Statistical and Game theoretical framework called Statistical- Teleodynamics.
Prerequisites: Python, Statistics, Neural Networks, LLM, Game Theory and Machine learning techniques
Traditional Neural Network:
It is known that each layer in a fully connected neural network contributes incrementally by reducing the complexly distributed data points into linearly separable points, allowing the last layer to simply draw the boundary for classification.
Previous analysis by our group has shown that each layer in a fully trained Neural Network model exhibits a Log-Normal trend in its neuronal weights. Realizing that the Lognormal weights represent the perfectly trained state or ideal state, this project aims to understand the ability of these ideal layers to reduce complexly shaped data into linearly separable data points. This can be compared with matrix rotation or operation.
The second aim of this project is to use the recently found trends in neuronal weights as a way for weight initialization. The two major advantages of this approach are its potential to reduce the training time and the ability to use lesser data to train. This could potentially reduce the training time by a huge factor!
The third aim of this project involves analysing the topological aspects of this lognormal Initialization by understanding the Betti number to analyse the data resolving capabilities of these layers.
Prerequisites: Python, Statistics, Neural Networks, ML
Modeling Microrobotic Dynamics
Microrobots, in this context, refer to a collection of magnetic robotic particles that can be stacked axially, like Legos, to form large Microrobots. Due to their magnetic nature, an external influence can help them move, rotate, stack further or even break stacking. These external fields can make the microrobots coordinate collectively to form a colony or a system of microrobots. This can be compared to the ant/bird colonies and their collective ability to sustain or solve problems. With this system along with an external field, robotic swarms can execute terrain reconnaissance, pattern formation, and cargo transportation.
This project aims to model the Microrobots and their self gathering phenomenon using a Statistical and Game theoretical framework. The modeling will focus on specific applications of the system of Microrobots under external magnetic influence.
The overall steps includes:
Theoretical comparison
Simulation and optimization
This theory of modeling can be extended further along with the existing models on Financial/ Economic segregation, Ant colonies, Bird flock and so on.
Prerequisites: Statistics, Game Theory, Modeling, Simulation, and Python
Required/preferred prerequisites and qualifications: The prerequisites for each project in the descriptions above.
Faculty/Lab: Prof. Steve Feiner
Brief Research Project Description:
The Computer Graphics and User Interfaces Lab (Prof. Feiner, PI) does research in the design of 3D and 2D user interfaces, including augmented reality (AR) and virtual reality (VR), and mobile and wearable systems, for people interacting individually and together, indoors and outdoors. We use a range of displays and devices: head-worn, hand-held, and table-top, including Varjo XR-3, HoloLens 2, and Magic Leap 2, in addition to consumer headsets such as Meta Quest 3. Multidisciplinary projects potentially involve working with faculty and students in other schools and departments, from medicine and dentistry to earth and environmental sciences and social work.
Required/preferred prerequisites and qualifications: We’re looking for students who have done excellent work in one or more of the following courses or their equivalents elsewhere: COMS W4160 (Computer graphics), COMS W4170 (User interface design), COMS W4172 (3D user interfaces and augmented reality), and COMS E6173 (Topics in VR & AR, or its earlier 6998 version), and who have software design and development expertise. For those projects involving 3D user interfaces, we’re especially interested in students with Unity experience.
Zoom Session info below
Virtual
Please check back for more possible Zoom session details
Faculty/Lab: Prof. Brian Plancher
Brief Research Project Description:
Constrained Parallel-DDP for any Robot:
I designed and implemented Parallel-DDP a few years ago, since then there have been significant changes made to CUDA that enable much higher performance (and I have learned a lot more about writing better CUDA code). As such, we are rewriting the code. We also need to expand the software to support the inclusion of additional constraints. This can be done through many formulations including previous work that I have done. As such we are currently building a constraint taxonomy and also integrating a number of methods to support constraints and working to compare them to find the optimal formulation. We currently have a draft taxonomy and a number of Python implementations validating these constraint approaches. We now need to consolidate those Python approaches into one single unified algorithm and develop the final constrained CUDA codebase.
An End-to-End Edge Robotics Stack for the Crazyflie:
In our recent, award-winning, TinyMPC work, we developed a novel control algorithm and implementation for micro-drones (e.g., the Crazyflie). However, while we were able to make really nice demos, they relied on external motion-capture setups. Looking forward we would like to move the entire robotics stack onboard the Crazyflie. With the addition of the AI deck, we think this is possible, and aim to recreate the kind of experiments done in this perceptive drone racing paper on the Crazyflie using a modified version of TinyMPC plus some computer vision. At the moment the key next step is to develop an integrated proof-of-concept implementation of such a system as we have independently validated the various parts of the approach. This will require compressing the computer vision pipeline onto the AI deck and integrating the AI deck with the onboard microcontroller on the base Crazyflie platform.
Required/preferred prerequisites and qualifications:
Constrained Parallel-DDP for any Robot:
Ideal backgrounds for this project would include experience with: (CUDA) C++, parallel programming, numerical optimization, optimal control, and open-source software development.
An End-to-End Edge Robotics Stack for the Crazyflie:
Ideal backgrounds for this project would include experience with: (embedded) C++, quadrotor firmware, embedded systems integration, numerical optimization, optimal control, and open-source software development.
Zoom: 1/22/2025 8:00:00 PM
https://brianplancher.com/zoom
Faculty/Lab: Prof. Shalmali Joshi, ReAIM Lab https://reaim-lab.github.io/
Brief Research Project Description:
(Project 1) Principled design and benchmarking for multimodal AI in health: Multimodal learning in health is more challenging that training vision-language models because of missingness in paired data, source of labels and reliability of contextual information. If you are interested in developing new deep learning methods suitable for such data, with real impact in areas of cardiology and electronic health record data, this is the project for you!
(Project 2) Improving representation learning for Electronic Health Record data: Foundation models that model patient data from electronic health records do not generalize due to many challenges from missingness, non-harmonized representations, and the current state-of-the-art architectures. We are developing new representations for EHR data amenable to foundation models for health. We test out these foundation models in real health data.
(Project 3) Evaluating algorithmic fairness and equity in health: Existing literature in algorithmic fairness has not borne out successful fair models in healthcare AI. Our lab is developing and validating new methods to evaluate fairness in health settings.
All our projects include collaborations are with expert clinicians who do research in each area.
Required/preferred prerequisites and qualifications: Strong computational background in machine learning, have completed some coursework in probabilistic modelling, causal inference or statistical learning. Students should be passionate about using AI to have a broad impact in health and medicine with a strong interest in using fundamental and methodological contributions in multi-modal healthcare, improving the reliability of large language modelling, deep learning, and/or algorithmic fairness in health and medicine. The lab does not train simple models for some healthcare task, we strive hard to improve the reliability and robustness of our AI models, working with clinicians. We do work that will have methodological impact as well as clinical impact. We have multiple projects ongoing in the lab, and students who tend to get most out of it are those who express their interest, take the initiative to self-learn and discuss background material with group members, participate in meetings, and contribute proactively, including reaching out to the PI as necessary.
Zoom:
1/23/2025 | 2:00:00 PM | https://columbiacuimc.zoom.us/my/shalmalij |
Faculty/Lab: Prof. Baishakhi Ray, Prof. Junfeng Yang, PhD students Jinjun Peng, Alex Mathai & Yangruibo Ding
Brief Research Project Description: LLM for Code: we study how to improve the coding capabilities of LLMs, in terms of functionality (https://arxiv.org/abs/2406.01006) or security (https://arxiv.org/abs/2501.08200). We will explore various training techniques (e.g. SFT/DPO/RL) and data engineering methods (e.g. synthetic data generation).
Required/preferred prerequisites and qualifications: proficient Python programming; strong coding/engineering/debugging capability; understanding of neural network training and the Transformer architecture; willingness to read research papers
Meet:
See online instructions for applying: “https://co1in.me/apply_ra”. Joining the Google Meet is not needed if you do not have any questions.
1/22/2025 | 3:00:00 PM | https://meet.google.com/riv-bbhp-jyz |
Faculty/Lab: Prof. Steve Feiner
Brief Research Project Description: The Computer Graphics and User Interfaces Lab (Prof. Feiner, PI) does research in the design of 3D and 2D user interfaces, including augmented reality (AR) and virtual reality (VR), and mobile and wearable systems, for people interacting individually and together, indoors and outdoors. We use a range of displays and devices: head-worn, hand-held, and table-top, including Varjo XR-3, HoloLens 2, and Magic Leap 2, in addition to consumer headsets such as Meta Quest 3. Multidisciplinary projects potentially involve working with faculty and students in other schools and departments, from medicine and dentistry to earth and environmental sciences and social work.
Required/preferred prerequisites and qualifications: We’re looking for students who have done excellent work in one or more of the following courses or their equivalents elsewhere: COMS W4160 (Computer graphics), COMS W4170 (User interface design), COMS W4172 (3D user interfaces and augmented reality), and COMS E6173 (Topics in VR & AR, or its earlier 6998 version), and who have software design and development expertise. For those projects involving 3D user interfaces, we’re especially interested in students with Unity experience.
Zoom:
Faculty/Lab: Prof. Brian Smith, Computer-Enabled Abilities Lab (CEAL)
Brief Research Project Description:
Street View Imagery for Enhanced Blind Navigation
We will develop an iOS application that provides real-time, audio-based descriptions of the user’s surroundings by leveraging street view imagery. As users walk, the app will intelligently gather environmental details—like landmarks, obstacles, and points of interest—and convert this information into concise, meaningful audio cues for people who are blind or have low vision. By combining deep learning approaches, large language models (LLMs), and computer vision, the app aims to offer safe and efficient navigation assistance in a variety of outdoor settings.
Required/preferred prerequisites and qualifications:
Prior experience in at least one of the following is required:
• iOS development (Swift, SwiftUI)
• Python and deep learning frameworks (e.g., PyTorch, TensorFlow), especially for computer vision techniques
• Familiarity with large language models (LLMs)
• User study design and data analysis
More info & To Join: If you’re interested, please fill out this form: https://forms.gle/5rJb1eYPjzyP3Tft7 Gaurav Jain (https://ceal.cs.columbia.edu/)