Moawia Eldeeb (SEAS ’18) and Josh Augustin (SEAS ’18) created Pivot, a connected in-home gym. With seed funding, they built SmartSpot, a successful B2B forerunner of Pivot, which they sold to hundreds of gyms around the world, generating over $1 million in revenue. This past July, they closed a $17 million round of Series A funding and expect to launch a consumer version next year.
Risser’s software company, Artomatix, developed AI that automates the creation of 3D content. The deal is believed to be valued up to $60 million.
Nora Wixom (BC ’13) shares how she got to work on the visual effects of blockbuster films Jurassic World, Captain America : Civil War, Kong: Skull Island, and Star Wars Episode VIII.
Shortly after Wixom graduated from Barnard College with a computer science degree, she found herself working at Industrial Light & Magic (ILM), the visual effects company founded by George Lucas.
On IMDB it says your role was creature technical director for Episode VIII. What did you have to do?
At ILM, a creature technical director (TD) is a visual effects artist who governs the motion of a digital asset, creature or otherwise.
Having an animator control every moving piece of a movie by hand would be way too tedious and time consuming so we step in and help deform certain elements programmatically, like clothing, flesh, and hair. If you think of an animated character as a puppet, a creature artist first “rigs” the character, or puts the strings on the puppet so the animators can move it.
We then turn the character or asset over to animation, where the animators move the bones of the character in the shots throughout the movie. Those shots then come back to creature, where we use the animated primary motion to govern secondary motion, or anything that moves in response to animation. An animator might move the bones of an arm into a flexing motion, and in return we would simulate bicep muscles flexing, skin creasing and sliding, clothing wrinkling, or fur moving.
Creature technical directors at ILM also simulate non-creature assets such as crowds, leaves on trees, and major pieces of explosions or crumbling buildings. On Episode VIII in particular, I was working on hair and flesh simulations for the fathiers and the crystal foxes, as well as rigs for the Resistance ski speeder, and ship explosions for fight sequences.
Were you part of a team, how big or small was it?
We always work as a team, but the size of the team depends heavily on the movie. For smaller or less demanding films, I’ve worked with creature crews as small as two. On large shows that are centered around digital characters, we can have 30+ creature artists in San Francisco alone with more assisting from ILM offices around the world.
Creature artists also rely heavily on many other kinds of artists to bring our characters to life: modellers to sculpt the creatures, texture artists to control their look and feel, lighters and compositors to integrate the digital character into the filmed footage. Even the smallest creature crew requires a support team in order to make our work shine.
How long did you have to work on the project?
A creature TD typically spends four to six months doing “shot work” on a film, but can spend one to two years on a film if they are involved early on with rigging assets since that has to come before any animation. For Episode VIII, I was exclusively doing shot work, so I was crewed to the movie for a period of around six months. However, a movie can spend much longer inside the doors of ILM from concept art to final delivery; multiple years is not unheard of.
What was your favorite part or scene to work on? Why?
My favorite part of working on Episode VIII had to be simulating TIE fighter explosions for the final fight sequence on Crait. That’s such a classic Star Wars moment, and it felt incredible to be able to contribute in a fun way to the Star Wars canon.
What was a major challenge that you encountered while working on visual effects?
One major challenge of Episode VIII was the inclusion of the most adorable character – the Porg. Originally, the director wanted the Porg to be a puppet, and all the footage was shot with a practical, stuffed Porg. Unfortunately, once we started reviewing the footage, the director decided he no longer liked the look and feel of the puppet and wanted it replaced with a digital Porg.
Removing and replacing a character is much more difficult than just shooting without one and adding it later, and keeping the look consistent between the shots with the puppet and the shots with the digital character can be a challenge as well – not to mention this entire digital character was entirely unplanned.
In the end, it was the right choice, as the digital character was much more expressive and able to emote more fully, but it certainly would have helped to know that from the start!
You graduated in 2013, how did you get to work for ILM just two years after graduating?
Landing at ILM so quickly after graduation was a combination of good timing, good connections, and being well prepared. I had a friend, Victor Frenkel, who had graduated from the 3:2 Combined Plan Program a year ahead of me and went to work for ILM as a technical assistant (an entry level job where you support render farm utilization across the studio).
Two months after my December 2013 graduation, he contacted me to share that he was getting promoted out of his role and into research and development, and knew I may be interested in filling his position. He recommended me to the team at ILM, and I proceeded to interview and be offered the job. So, although I did have a strong network, I also had the experience and the knowledge to be the right candidate for the job at the right time.
Once at ILM, I worked as a technical assistant for around eight months before transitioning into the creature department. I was able to make that transition by reaching out and working closely with the creature supervisor on Jurassic World; she would sit down with me and teach me skills, and I would have to go back on my own time and replicate what we did to prove to her that I understood.
After two months of working on simple assets and shot work, she offered to bring me on to her show as a full time creature artist. My very first shot as a creature TD was the shot in Jurassic World where (spoiler alert!) the facilities worker gets eaten by the Indominous Rex after it escapes from the enclosure – I simulated the clothing on the digital double of the employee.
How did your CS degree help you prepare for work after graduation?
Studying CS at Columbia made me a much better creature TD because it gave me a low-level understanding of how our software fundamentally worked.
A creature TD will run in-shot simulations by tweaking solver parameters like time steps and spring strengths. Since I had taken COMS 4167, I had first-hand experience building a physics engine and had deep knowledge of how these solvers ran their calculations. COMS 4160 taught me how raytracing worked and how images were rendered, and COMS 4170 gave me the user interface background to design effective tools for other artists to interact with my characters. The linear algebra skills I learned were also incredibly applicable as all movement in 3D space is represented with vector and matrix notation; to this day, I’m amazed by how this knowledge comes back again and again.
How did you become interested in visual effects?
Believe it or not, I initially declared chemistry as my major as I intended on going to medical school. However, it turns out I was both impatient and pretty terrible at organic chemistry, so I reevaluated my options after about a year. I had realized that perhaps my interest in medicine was at its heart an interest in anatomy, and maybe there was another way to indulge that.
Reaching back, I did not grow up in a gaming-friendly household, but I remembered playing The Sims on my computer and thinking that the game had a sense of humor and must have been fun to work on. Working on characters for games and film seemed like it could be a good way to combine that sense of humor with STEM, art, and my anatomical interest, so I looked at character-based job listings at companies like ILM and Pixar to see what kinds of degrees and skills they required.
The shortest path between Columbia and there seemed to be through the CS department along the vision and graphics track, which taught courses in image generation, vision algorithms, and the technical aspects of animation. Although I had absolutely no CS experience, I loved the idea that computer science was a major where I could be creative and constructive and have some cool projects to show for it at the end of the day instead of a stack of papers and problems sets. I changed my major the next week, and the rest is history.
What advice would you give to students who want to get into visual effects? And how to get that first job?
One important piece of advice I’d have for students pursuing visual effects (or any non-traditional career path for that matter) is to not be shy about blazing your own trail!
Jobs like this kind of fall out of the scope of what the predefined CS tracks prepare you for, so you may need to do a bit of legwork on your own to learn the skills you’ll need. In my case, I needed to have some experience using creative software packages like Autodesk Maya and Adobe Photoshop, so I took classes at the School of Visual Arts and the University of Michigan to make sure I had those bases covered. It’s safe to assume you’ll learn a ton on the job, but coming in with enough knowledge to be comfortable using the necessary tools is critical to your job application and to your ability to succeed once you’re there.
Admittedly, I struggled to find the kinds of jobs I was interested in through the Columbia employment resources. That being said, I would also highly, highly recommend that any students interested in applied computer graphics take the time to attend an industry conference like GDC or SIGGRAPH. These conferences have job fairs, portfolio reviews, and interviews on-site, and they are great places to discover hot topics in the industry, connect with like-minded individuals, and see where your skills can take you after graduation.
Although these conferences are expensive, they often offer discounted pricing for students, or have positions open for students to volunteer at the conference in exchange for free admission. SIGGRAPH typically rotates between Anaheim, Vancouver, and Los Angeles, but will make a rare visit to the East Coast (Washington DC) for the 2020 conference, so I strongly recommend going next year if it sounds interesting to any students out there.
The 33rd Conference on Neural Information Processing Systems (NeurIPS 2019) fosters the exchange of research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects.
The annual meeting is one of the premier gatherings in artificial intelligence and machine learning that featured talks, demos from industry partners as well as tutorials. Professor Vishal Misra, with colleagues from the Massachusetts Institute of Technology (MIT), held a tutorial on synthetic control.
At this year’s NeurIPS, 21 papers from the department were accepted to the conference. Computer science professors and students worked with researchers from the statistics department and the Data Science Institute.
Noise-tolerant Fair Classification
Alex Lamy Columbia University, Ziyuan Zhong Columbia University, Aditya Menon Google, Nakul Verma Columbia University
Fairness-aware learning involves designing algorithms that do not discriminate with respect to some sensitive feature (e.g., race or gender) and is usually done under the assumption that the sensitive feature available in a training sample is perfectly reliable.
This assumption may be violated in many real-world cases: for example, respondents to a survey may choose to conceal or obfuscate their group identity out of fear of potential discrimination. In the paper, the researchers show that fair classifiers can still be used given noisy sensitive features by simply changing the desired fairness-tolerance. Their procedure is empirically effective on two relevant real-world case-studies involving sensitive feature censoring.
Poisson-randomized Gamma Dynamical Systems
Aaron Schein UMass Amherst, Scott Linderman Columbia University, Mingyuan Zhou University of Texas at Austin, David Blei Columbia University, Hanna Wallach MSR NYC
This paper presents a new class of state space models for count data. It derives new properties of the Poisson-randomized gamma distribution for efficient posterior inference.
Using Embeddings to Correct for Unobserved Confounding in Networks
Victor Veitch Columbia University, Yixin Wang Columbia University, David Blei Columbia University
This paper address causal inference in the presence of unobserved confounder when proxy is available for the confounders in the form of a network connecting the units. For example, the link structure of friendships in a social network reveals information about the latent preferences of people in that network. The researchers show how modern network embedding methods can be exploited to harness the network estimation for efficient causal adjustment.
Variational Bayes Under Model Misspecification
Yixin Wang Columbia University, David Blei Columbia University
The paper characterizes the theoretical properties of a popular machine learning algorithm, variational Bayes (VB). The researchers studied the VB under model misspecification, which is the setting that is most aligned with the practice, and show that the VB posterior is asymptotically normal and centers at the value that minimizes the Kullback-Leibler (KL) divergence to the true data-generating distribution.
As a consequence, they found that the model misspecification error dominates the variational approximation error in VB posterior predictive distributions. In other words, VB pays a negligible price in producing posterior predictive distributions. It explains the widely observed phenomenon that VB achieves comparable predictive accuracy with MCMC even though VB uses an approximating family.
Poincaré Recurrence, Cycles and Spurious Equilibria in Gradient-Descent-Ascent for Non-Convex Non-Concave Zero-Sum Games
Emmanouil-Vasileios Vlatakis-Gkaragkounis Columbia University, Lampros Flokas Columbia University, Georgios Piliouras Singapore University of Technology and Design
The paper introduces a model that captures a min-max competition over complex error landscapes and shows that even a simplified model can provably replicate some of the most commonly reported failure modes of GANs (non-convergence, deadlock in suboptimal states, etc).
Moreover, the researchers were able to understand the hidden structure in these systems — the min-max competition can lead to system behavior that is similar to that of energy preserving systems in physics (e.g. connected pendulums, many-body problems, etc). This makes it easier to understand why these systems can fail and gives new tools in the design of algorithms for training GANs.
Near-Optimal Reinforcement Learning in Dynamic Treatment Regimes
Junzhe Zhang Columbia University, Elias Bareinboim Columbia University
Dynamic Treatment Regimes (DTRs) are particularly effective for managing chronic disorders and is arguably one of the key aspects towards more personalized decision-making. The researchers developed the first adaptive algorithm that achieves near-optimal regret in DTRs in online settings, while leveraging the abundant, yet imperfect confounded observations. Applications are given to personalized medicine and treatment recommendation in clinical decision support.
Paraphrase Generation with Latent Bag of Words
Yao Fu Columbia University, Yansong Feng Peking University, John Cunningham University of Columbia
The paper proposes a latent bag of words model for differentiable content planning and surface realization in text generation. This model generates paraphrases with clear steps, adding interpretability and controllability of existing neural text generation models.
Adapting Neural Networks for the Estimation of Treatment Effects
Claudia Shi Columbia University, David Blei Columbia University, Victor Veitch Columbia University
This paper addresses how to design neural networks to get very accurate estimates of causal effects from observational data. The researchers propose two methods based on insights from the statistical literature on the estimation of treatment effects.
The first is a new architecture, the Dragonnet, that exploits the sufficiency of the propensity score for estimation adjustment. The second is a regularization procedure, targeted regularization, that induces a bias towards models that have non-parametrically optimal asymptotic properties “out-of-the-box”. Studies on benchmark datasets for causal inference show these adaptations outperform existing methods.
Efficiently Avoiding Saddle Points with Zero Order Methods: No Gradients Required
Emmanouil-Vasileios Vlatakis-Gkaragkounis Columbia University, Lampros Flokas Columbia University, Georgios Piliouras Singapore University of Technology and Design
The researchers prove that properly tailored zero-order methods are as effective as their first-order counterparts. This analysis requires a combination of tools from optimization theory, probability theory and dynamical systems to show that even without perfect knowledge of the shape of the error landscape, effective optimization is possible.
Metric Learning for Adversarial Robustness
Chengzhi Mao Columbia University, Ziyuan Zhong Columbia University, Junfeng Yang Columbia University, Carl Vondrick Columbia University, Baishakhi Ray Columbia University
Deep networks are well-known to be fragile to adversarial attacks. The paper introduces a novel Triplet Loss Adversarial (TLA) regulation that is the first method that leverages metric learning to improve the robustness of deep networks. This method is inspired by the evidence that deep networks suffer from distorted feature space under adversarial attacks. The method increases the model robustness and efficiency for the detection of adversarial attacks significantly.
Efficient Symmetric Norm Regression via Linear Sketching
Zhao Song University of Washington, Ruosong Wang Carnegie Mellon University, Lin Yang Johns Hopkins University, Hongyang Zhang TTIC, Peilin Zhong Columbia University
The paper studies linear regression problems with general symmetric norm loss and gives efficient algorithms for solving such linear regression problems via sketching techniques.
Rethinking Generative Coverage: A Pointwise Guaranteed Approach
Peilin Zhong Columbia University, Yuchen Mo Columbia University, Chang Xiao Columbia University, Pengyu Chen Columbia University, Changxi Zheng Columbia University
The paper presents a novel and formal definition of mode coverage for generative models. It also gives a boosting algorithm to achieve this mode coverage guarantee.
How Many Variables Should Be Entered in a Principal Component Regression Equation?
Ji Xu Columbia University, Daniel Hsu Columbia University
The researchers studied the least-squares linear regression over $N$ uncorrelated Gaussian features that are selected in order of decreasing variance with the number of selected features $p$ can be either smaller or greater than the sample size $n$. And give an average-case analysis of the out-of-sample prediction error as $p,n,N \to \infty$ with $p/N \to \alpha$ and $n/N \to \beta$, for some constants $\alpha \in [0,1]$ and $\beta \in (0,1)$. In this average-case setting, the prediction error exhibits a “double descent” shape as a function of $p$. This also establishes conditions under which the minimum risk is achieved in the interpolating ($p>n$) regime.
Adaptive Influence Maximization with Myopic Feedback
Binghui Peng Columbia University, Wei Chen Microsoft Research
The paper investigates the adaptive influence maximization problem and provides upper and lower bounds for the adaptivity gaps under myopic feedback model. The results confirm a long standing open conjecture by Golovin and Krause (2011).
Towards a Zero-One Law for Column Subset Selection
Zhao Song University of Washington, David Woodruff Carnegie Mellon University, Peilin Zhong Columbia University
The researchers studied low-rank matrix approximation with general loss function and showed that if the loss function has several good properties, then there is an efficient way to compute a good low-rank approximation. Otherwise, it could be hard to compute a good low-rank approximation efficiently.
Average Case Column Subset Selection for Entrywise l1-Norm Loss
Zhao Song University of Washington, David Woodruff Carnegie Mellon University, Peilin Zhong Columbia University
The researchers studied how to compute an l1-norm loss low-rank matrix approximation to a given matrix. And showed that if the given matrix can be decomposed into a low-rank matrix and a noise matrix with a mild distributional assumption, we can obtain a (1+eps) approximation to the optimal solution.
A New Distribution on the Simplex with Auto-Encoding Applications
Andrew Stirn Columbia University, Tony Jebara Spotify, David Knowles Columbia University
The researchers developed a surrogate distribution for the Dirichlet that offers explicit, tractable reparameterization, the ability to capture sparsity, and has barycentric symmetry properties (i.e. exchangeability) equivalent to the Dirichlet. Previous works have used the Kumaraswamy distribution in a stick-breaking process to create a non-exchangeable distribution on the simplex. The method was improved by restoring exchangeability and demonstrating that approximate exchangeability is efficiently achievable. Lastly, the method was showcased in a variety of VAE semi-supervised learning tasks.
Discrete Flows: Invertible Generative Models of Discrete Data
Dustin Tran Google Brain, Keyon Vafa Columbia University, Kumar Agrawal Google AI Resident, Laurent Dinh Google Brain, Ben Poole Google Brain
While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. The researchers extend normalizing flows to discrete events, using a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Empirically, they find that discrete flows obtain competitive performance with or outperform autoregressive baselines on various tasks, including addition, Potts models, and language models.
Characterization and Learning of Causal Graphs with Latent Variables from Soft Interventions
Murat Kocaoglu MIT-IBM Watson AI Lab IBM Research, Amin Jaber Purdue University, Karthikeyan Shanmugam MIT-IBM Watson AI Lab IBM Research NY, Elias Bareinboim Columbia University
This work is all about learning causal relationships – the classic aim of which is to characterize all possible sets that could produce the observed data. In the paper, the researchers provide a complete characterization of all possible causal graphs with observational and interventional data involving so-called ‘soft interventions’ on variables when the targets of soft interventions are known.
This work potentially could lead to discovery of other novel learning algorithms that are both sound and complete.
Identification of Conditional Causal Effects Under Markov Equivalence
Amin Jaber Purdue University, Jiji Zhang Lingnan University, Elias Bareinboim Columbia University
Causal identification is the problem of deciding whether a causal distribution is computable from a combination of qualitative knowledge about the underlying data-generating process, which is usually encoded in the form of a causal graph, and an observational distribution. Despite the obvious need for identifying causal effects throughout the data-driven sciences, in practice, finding the causal graph is a notoriously challenging task.
In this work, the researchers provide a relaxation of the requirement of having to specify the causal graph (based on substantive knowledge) and allow the input of the inference to be an equivalence class of causal graphs, which can be inferred from data. Specifically, they propose the first general algorithm to learn conditional causal effects entirely from data. This result is particularly useful for evaluating the impact of conditional plans and stochastic policies, which appear both in AI (in the context of reinforcement learning) and in the data-driven sciences.
Efficient Identification in Linear Structural Causal Models with Instrumental Cutsets
Daniel Kumor Purdue University, Bryant Chen Brex Inc., Elias Bareinboim Columbia University
Regression analysis is one of the most common tools used in modern data science. While there is a great understanding and powerful technology to perform regression analysis in high dimensional spaces, the output of such a method is purely associational and devoid of any causal interpretation.
The researchers studied the problem of identification of structural (causal) coefficients in linear systems (deciding whether regression coefficients are amenable to causal interpretation, etc). Building on a technique called instrumental variables, they developed a new method called Instrumental Cutset, which partitions the systems into tractable components such that identification can be decided more efficiently. The resulting algorithm was efficient and strictly more powerful than the current state-of-the-art methods.
Assistant Professor Allison Bishop takes a look at failure and how people can learn from “unsuccessful” research.
When it comes to research and getting papers into cryptography conferences, there usually has to be a “positive” result — either a new theorem must be proven, a new algorithm must be presented, or a successful attack on an existing algorithm must be obtained. If researchers try to accomplish a lofty goal and fall short, but manage to achieve a smaller goal, they typically present only the smaller goal as if it was the point on its own.
“I’ve found that not every research paper magically comes together and has a “great” result,” said Allison Bishop, who has been teaching since 2013. “Our community doesn’t really talk about the research process and I wanted to highlight research where even if it “failed” there is still something to learn from it.”
Through the years Bishop noticed the lack of a venue to talk about all kinds of research. When she and other researchers studied obfuscation it resulted in a paper “In Pursuit of Clarity In Obfuscation”. In the paper they talked about how they “failed” but managed to still learn from their mistakes. Their topic on failure was not considered a “standard” that could be published and they were not able to submit it to a conference. But Bishop, along with PhD students Luke Kowalczyk and Kevin Shi, really wanted to get their findings out and share it with other researchers.
And so, a conference dedicated to disseminating insightful failures of the cryptology research community was born. The Conference for Failed Approaches and Insightful Losses in Cryptology or CFAIL featured seven previously unpublished papers for a day of talks by computer scientists on insightful failures spanning the full range from cryptanalysis (trying to break systems) to cryptographic theory and design (constructing new systems and proving things about specific systems or about abstract systems, etc.).
“CFAIL is great for our field in that it promotes openness and accessibility for these kinds of ideas which are typically sort of intimate,” said Luke Kowalczyk, who completed his PhD in November of last year. “When approaching new problems, it’s always helpful to see the approaches of other researchers, even if they were not successful. However, it’s rare to see failed approaches explained in a public and formal setting.”
They were not alone in thinking about the lack of dialogue on research failures. At the time of the conference, a thread on Hacker News (a tech news aggregator) discussed the incentive structures of academia. Shared Kowalczyk, “I was proud to see CFAIL cited as an example of a scientific field with a formal venue to help promote this kind of openness.”
“There is a deeply ingrained human tendency to fear that being open about failure will make other people think you are dumb,” said Bishop. On the contrary, the researchers at CFAIL were some of the “most creative, bold, and deeply intelligent people.” And the atmosphere it created was energizing for the participants — the audience got pretty involved and felt comfortable asking questions, and even started thinking about some of the open research problems in real time. Continued Bishop, ”I think talking about failure is probably the best scientific communication strategy left that is severely underused.”
Bishop will continue to promote openness in scientific research with another CFAIL at Crypto 2020. This time around it will be a workshop at the conference and a call for papers will be out soon.
IBM has selected assistant professor Baishakhi Ray for an IBM Faculty Award. The highly selective award is given to professors in leading universities worldwide to foster collaboration with IBM researchers. Ray will use the funds to continue research on artificial intelligence-driven program analysis to understand software robustness.
Although much research has been done, there are still countless vulnerabilities that make system robustness brittle. Hidden vulnerabilities are discovered all the time – either through a system hack or monitoring system’s functionalities. Ray is working to automatically detect system weaknesses using artificial intelligence (AI) with her project, “Improving code representation for enhanced deep learning to detect and remediate security vulnerabilities”.
One of the major challenges in AI-based security vulnerability detection is finding the best source code representation that can distinguish between vulnerable versus benign code. Such representation can further be used as an input in supervised learning settings for automatic vulnerability detection and fixes. Ray is tackling this problem by building new machine-learning models for source code and applying machine learning techniques such as code embeddings. This approach could open new ways of encoding source code into feature vectors.
“It will provide new ways to make systems secure,” said Ray, who joined the department in 2018. “The goal is to reduce the hours of manual effort spent in automatically detecting vulnerabilities and fixing them.”
A successful outcome of this project will produce a new technique to encode source code with associated trained models that will be able to detect and remediate a software vulnerability with increased accuracy.
IBM researchers Jim Laredo and Alessandro Morari will collaborate closely with Ray and her team on opportunities around design, implementation, and evaluation of this research.
With multiple grants, Professor Luca Carloni works toward developing design methodologies and system architectures for heterogeneous computing. He foresees computing systems both in the cloud and at the edge of the cloud will become more heterogeneous.
For a while, many systems in the cloud, in servers, and in computers, were based on homogeneous multi-core architectures where multiple processors are combined in a chip and multiple chips on a board. But all, in the first approximation, are copies of the same type of processor.
“Now, it is not the case,” said Carloni, who has worked on heterogeneous computing for the past 10 years. “And we have been one of the first groups to really, I think, understand this transition from a research viewpoint and change first our research focus and then our teaching efforts to address all the issues.”
Heterogeneous means that a system is made of components and each component has a different nature. Some of these components are processors that execute software – application software and system software while other components are accelerators. An accelerator is a hardware module specialized to execute a particular function. Specialization provides major benefits in terms of performance and energy efficiency.
Heterogeneous computing, however, is more difficult. Compared to its homogeneous counterpart, heterogeneous systems bring new challenges in terms of hardware-software interactions, access to shared resources, and diminished regularity of the design.
Another aspect of heterogeneity is that components often come from different sources. Let’s say that a company builds a new system-on-chip (SoC), a pervasive type of integrated circuit that is highly heterogeneous. Some parts may be designed anew inside the company, some reused from previous designs, while others may be licensed from other companies. Integrating these parts efficiently requires new design methods.
Carloni’s lab, the System-Level Design Group, tackles these challenges with the concept of embedded scalable platforms (ESP). A platform combines a flexible computer architecture and a companion computer-aided design methodology. The architecture defines how the computation is organized among multiple components, how the components are interconnected, and how to establish the interface between what is done in software and what is done in hardware. Because of the complexity of these systems, many important decisions must be made early, while room must be left to make adjustments later. The methodology guides software programmers and hardware engineers to design, optimize, and integrate their novel solutions.
By leveraging ESP, the SLD Group is developing many SoC prototypes, particularly with field programmable gate arrays (FPGAs). With FPGAs, the hardware in the system can be configured in the field. This allows the chance to explore several optimizations for the target heterogeneous system before committing to the fabrication of expensive silicon.
All of these topics are covered in System-on-Chip Platforms, a class Carloni teaches each fall semester. Students have to design an accelerator — not just for one particular system, but also with a good degree of reusability so that it can be leveraged across multiple systems. Earlier this year, Carloni presented a paper that describes this course at the 2019 Workshop on Computer Architecture Education.
“In developing System-on-Chip Platforms we put particular emphasis on SoC architectures for high-performance embedded applications,” he said. “But we believe that the course provides a broad foundation on the principles and practices of heterogeneous computing.”
A new framework that tests deep learning systems selected as a “Research Highlight” in the Communications of the ACM.
Add to GMail