Tool predicts how fast code will run on a chip

Tool predicts how fast code will run on a chip

Trending Technologies   January 06, 2020

The machine-learning system should enable developers to improve computing efficiency in a range of applications.

MIT researchers have invented a machine-learning tool that predicts how fast computer chips will execute code from various applications.  

To get code to run as fast as possible, developers and compilers — programs that translate programming language into machine-readable code — typically use performance models that run the code through a simulation of given chip architectures. 

Compilers use that information to automatically optimise code, and developers use it to tackle performance bottlenecks on the microprocessors that will run it. But performance models for machine code are handwritten by a relatively small group of experts and are not adequately validated. As a consequence, the simulated performance measurements often deviate from real-life results. 

In a series of conference papers, the researchers describe a novel machine-learning pipeline that automates this process, making it easier, faster, and more accurate. In a paper presented at the International Conference on Machine Learning in June, the researchers presented Ithemal, a neural-network model that trains on labelled data in the form of “basic blocks” — fundamental snippets of computing instructions — to automatically predict how long it takes a given chip to execute previously unseen basic blocks. Results suggest Ithemal performs far more accurately than traditional hand-tuned models. 

Then, at the November IEEE International Symposium on Workload Characterization, the researchers presented a benchmark suite of basic blocks from a variety of domains, including machine learning, compilers, cryptography, and graphics that can be used to validate performance models. They pooled more than 300,000 of the profiled blocks into an open-source dataset called BHive. During their evaluations, Ithemal predicted how fast Intel chips would run code even better than a performance model built by Intel itself. 

Ultimately, developers and compilers can use the tool to generate code that runs faster and more efficiently on an ever-growing number of diverse and “black box” chip designs. “Modern computer processors are opaque, horrendously complicated, and difficult to understand. It is also incredibly challenging to write computer code that executes as fast as possible for these processors,” says co-author Michael Carbin, an assistant professor in the Department of Electrical Engineering and Computer Science (EECS) and a researcher in the Computer Science and Artificial Intelligence Laboratory (CSAIL). “This tool is a big step forward toward fully modelling the performance of these chips for improved efficiency.”

Most recently, in a paper presented at the NeurIPS conference in December, the team proposed a new technique to automatically generate compiler optimisations.  Specifically, they automatically generate an algorithm, called Vemal, that converts certain code into vectors, which can be used for parallel computing. Vemal outperforms hand-crafted vectorisation algorithms used in the LLVM compiler — a popular compiler used in the industry.

Learning from data

Designing performance models by hand can be “a black art,” Carbin says. Intel provides extensive documentation of more than 3,000 pages describing its chips’ architectures. But there currently exists only a small group of experts who will build performance models that simulate the execution of code on those architectures. 

“Intel’s documents are neither error-free nor complete, and Intel will omit certain things, because it’s proprietary,” Mendis says. “However, when you use data, you don’t need to know the documentation. If there’s something hidden, you can learn it directly from the data.”

To do so, the researchers clocked the average number of cycles a given microprocessor takes to compute basic block instructions — basically, the sequence of boot-up, execute, and shut down — without human intervention. Automating the process enables rapid profiling of hundreds of thousands or millions of blocks. 

Domain-specific architectures

In training, the Ithemal model analyses millions of automatically profiled basic blocks to learn exactly how different chip architectures will execute computation. Importantly, Ithemal takes raw text as input and does not require manually adding features to the input data. In testing, Ithemal can be fed previously unseen basic blocks and a given chip and will generate a single number indicating how fast the chip will execute that code. 

The researchers found Ithemal cut error rates in accuracy — meaning the difference between the predicted speed versus real-world speed — by 50 per cent over traditional hand-crafted models. Further, in their next paper, they showed that Ithemal’s error rate was 10 per cent, while the Intel performance-prediction model’s error rate was 20 per cent on a variety of basic blocks across multiple different domains.

The tool now makes it easier to quickly learn performance speeds for any new chip architectures, Mendis says. For instance, domain-specific architectures, such as Google’s new Tensor Processing Unit used specifically for neural networks, are now being built but aren’t widely understood. “If you want to train a model on some new architecture, you just collect more data from that architecture, run it through our profiler, use that information to train Ithemal, and now you have a model that predicts performance,” Mendis says.

Next, the researchers are studying methods to make models interpretable. Much of machine learning is a black box, so it’s not really clear why a particular model made its predictions. “Our model is saying it takes a processor, say, 10 cycles to execute a basic block. Now, we’re trying to figure out why,” Carbin says. “That’s a fine level of granularity that would be amazing for these types of tools.”

They also hope to use Ithemal to enhance the performance of Vemal even further and achieve better performance automatically.

This article was originally published by MIT News. Follow this link to read related articles.

Specialized AI Chips Hold Both Promise and Peril for Developers

Specialized AI Chips Hold Both Promise and Peril for Developers

Trending Technologies   August 12, 2019

When it comes to the compute-intensive field of AI, hardware vendors are reviving the performance gains we enjoyed at the height of Moore’s Law. The gains come from a new generation of specialized chips for AI applications like deep learning. But the fragmented microchip marketplace that’s emerging will lead to some hard choices for developers. 

The new era of chip specialization for AI began when graphics processing units (GPUs), which were originally developed for gaming, were deployed for applications like deep learning. The same architecture that made GPUs render realistic images also enabled them to crunch data much more efficiently than central processing units (CPUs). A big step forward happened in 2007 when Nvidia released CUDA, a toolkit for making GPUs programmable in a general-purpose way.

AI researchers need every advantage they can get when dealing with the unprecedented computational requirements of deep learning. GPU processing power has advanced rapidly, and chips originally designed to render images have become the workhorses powering world-changing AI research and development. Many of the linear algebra routines that are necessary to make Fortnite run at 120 frames per second are now powering the neural networks at the heart of cutting-edge applications of computer vision, automated speech recognition, and natural language processing.  

Now, the trend toward microchip specialization is turning into an arms race. Gartner projects that specialized chip sales for AI will double to around the US $8 billion in 2019 and reach more than $34 billion by 2023. Nvidia’s internal projections place the market for data centre GPUs (which are almost solely used to power deep learning) at $50 billion in the same time frame. In the next five years, we’ll see massive investments in custom silicon come to fruition from Amazon, ARM, Apple, IBM, Intel, Google, Microsoft, Nvidia, Qualcomm. There is also a slew of startups in the mix. CrunchBase estimates that AI chip companies, including Cerebras, Graphcore, Groq, Mythic AI, SambaNova Systems, and Wave Computing, have collectively raised more than $1 billion. 

To be clear, specialized AI chips are both important and welcomed, as they’re catalysts for transforming cutting-edge AI research into real-world applications. However, the flood of new AI chips, each one faster and more specialized than the next, will also seem like a throwback to the rise of enterprise software. We can expect cut-throat sales deals and software specialization aimed at locking developers into working with just one vendor. 

Imagine if, 15 years ago, the cloud services AWS, Azure, Box, Dropbox, and GCP all came to market within 12 to 18 months. Their mission would have been to lock in as many businesses as possible—because once you’re on one platform, it’s hard to switch to another. This type of end-user gold rush is about to happen in AI, with tens of billions of dollars, and priceless research, at stake. 

Chipmakers won’t be short on promises, and the benefits will be real. But it’s important for AI developers to understand that new chips that require new architectures could make their products slower to market—even with faster performance. In most cases, AI models are not going to be portable between different chip makers. Developers are well aware of the vendor lock-in risk posed by adopting higher-level cloud APIs, but in the past, the actual compute substrate has been standardized and homogeneous. This situation is going to change dramatically in the world of AI development.

It's quite likely that more than half of the chip industry’s revenue will soon be driven by AI and deep learning applications. Just as software begets more software, AI begets more AI. We’ve seen it many times: Companies initially focus on one problem, but ultimately solve many. For example, major automakers are striving to bring autonomous cars to the road, and their cutting-edge work in deep learning and computer vision is already having a cascading effect; the research is leading to such offshoot projects as Ford’s delivery robots.

As specialized AI chips come to market, the current chip giants and major cloud companies will probably strike exclusive deals or acquire top performing startups. This trend will fragment the AI market rather than unifying it. All that AI developers can do now is understand what’s about to happen and plan how they’ll weigh the benefits of a faster chip with the costs of building on new architectures.

Evan Sparks is CEO of Determined AI. He holds a PhD in computer science from the University of California, Berkeley, where his research focused on distributed systems for data analysis and machine learning.

Drag-and-drop data analytics

Drag-and-drop data analytics

Trending Technologies   August 04, 2019

The system lets nonspecialists use machine-learning models to make predictions for medical research, sales, and more.

In the Iron Man movies, Tony Stark uses a holographic computer to project 3-D data into thin air, manipulate them with his hands, and find fixes to his superhero troubles. In the same vein, researchers from MIT and Brown University have now developed a system for interactive data analytics that runs on touchscreens and lets everyone — not just billionaire tech geniuses — tackle real-world issues.

For years, the researchers have been developing an interactive data-science system called Northstar, which runs in the cloud but has an interface that supports any touchscreen device, including smartphones and large interactive whiteboards. Users feed the system datasets, and manipulate, combine, and extract features on a user-friendly interface, using their fingers or a digital pen, to uncover trends and patterns.

In a paper being presented at the ACM SIGMOD conference, the researchers detail a new component of Northstar, called VDS for “virtual data scientist,” that instantly generates machine-learning models to run prediction tasks on their datasets. Doctors, for instance, can use the system to help predict which patients are more likely to have certain diseases, while business owners might want to forecast sales. If using an interactive whiteboard, everyone can also collaborate in real-time.

The aim is to democratize data science by making it easy to do complex analytics, quickly and accurately.

“Even a coffee shop owner who doesn’t know data science should be able to predict their sales over the next few weeks to figure out how much coffee to buy,” says co-author and long-time Northstar project lead Tim Kraska, an associate professor of electrical engineering and computer science in at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and founding co-director of the new Data System and AI Lab (DSAIL). “In companies that have data scientists, there’s a lot of back and forth between data scientists and nonexperts, so we can also bring them into one room to do analytics together.”

VDS is based on an increasingly popular technique in artificial intelligence called automated machine-learning (AutoML), which lets people with limited data-science know-how train AI models to make predictions based on their datasets. Currently, the tool leads the DARPA D3M Automatic Machine Learning competition, which every six months decide on the best-performing AutoML tool. 

Joining Kraska on the paper are: first author Zeyuan Shang, a graduate student, and Emanuel Zgraggen, a postdoc and main contributor of Northstar, both of EECS, CSAIL, and DSAIL; Benedetto Buratti, Yeounoh Chung, Philipp Eichmann, and Eli Upfal, all of Brown; and Carsten Binnig who recently moved from Brown to the Technical University of Darmstadt in Germany.

An “unbounded canvas” for analytics

The new work builds on years of collaboration on Northstar between researchers at MIT and Brown. Over four years, the researchers have published numerous papers detailing components of Northstar, including the interactive interface, operations on multiple platforms, accelerating results, and studies on user behavior.

Northstar starts as a blank, white interface. Users upload datasets into the system, which appear in a “datasets” box on the left. Any data labels will automatically populate a separate “attributes” box below. There’s also an “operators” box that contains various algorithms, as well as the new AutoML tool. All data are stored and analyzed in the cloud.

The researchers like to demonstrate the system on a public dataset that contains information on intensive care unit patients. Consider medical researchers who want to examine co-occurrences of certain diseases in certain age groups. They drag and drop into the middle of the interface a pattern-checking algorithm, which at first appears as a blank box. As input, they move into the box disease features labeled, say, “blood,” “infectious,” and “metabolic.” Percentages of those diseases in the dataset appear in the box. Then, they drag the “age” feature into the interface, which displays a bar chart of the patient’s age distribution. Drawing a line between the two boxes links them together. By circling age ranges, the algorithm immediately computes the co-occurrence of the three diseases among the age range.

“It’s like a big, unbounded canvas where you can layout how you want everything,” says Zgraggen, who is the key inventor of Northstar’s interactive interface. “Then, you can link things together to create more complex questions about your data.”

Approximating AutoML

With VDS, users can now also run predictive analytics on that data by getting models custom-fit to their tasks, such as data prediction, image classification, or analyzing complex graph structures.

Using the above example, say the medical researchers want to predict which patients may have blood disease based on all features in the dataset. They drag and drop “AutoML” from the list of algorithms. It’ll first produce a blank box, but with a “target” tab, under which they’d drop the “blood” feature. The system will automatically find best-performing machine-learning pipelines, presented as tabs with constantly updated accuracy percentages. Users can stop the process at any time, refine the search, and examine each model’s errors rates, structure, computations, and other things.

According to the researchers, VDS is the fastest interactive AutoML tool to date, thanks, in part, to their custom “estimation engine.” The engine sits between the interface and the cloud storage. The engine leverages automatically creates several representative samples of a dataset that can be progressively processed to produce high-quality results in seconds.

“Together with my co-authors, I spent two years designing VDS to mimic how a data scientist thinks,” Shang says, meaning it instantly identifies which models and preprocessing steps it should or shouldn’t run on certain tasks, based on various encoded rules. It first chooses from a large list of those possible machine-learning pipelines and runs simulations on the sample set. In doing so, it remembers results and refines its selection. After delivering fast approximated results, the system refines the results in the back end. But the final numbers are usually very close to the first approximation.

“For using a predictor, you don’t want to wait four hours to get your first results back. You want to already see what’s going on and, if you detect a mistake, you can immediately correct it. That’s normally not possible in any other system,” Kraska says. The researchers’ previous user study, in fact, “show that the moment you delay giving users results, they start to lose engagement with the system.”

The researchers evaluated the tool on 300 real-world datasets. Compared to other state-of-the-art AutoML systems, VDS’ approximations were as accurate but were generated within seconds, which is much faster than other tools, which operate in minutes to hours.

Next, the researchers are looking to add a feature that alerts users to potential data bias or errors. For instance, to protect patient privacy, sometimes researchers will label medical datasets with patients aged 0 (if they do not know the age) and 200 (if a patient is over 95 years old). But novices may not recognize such errors, which could completely throw off their analytics.

“If you’re a new user, you may get results and think they’re great,” Kraska says. “But we can warn people that there, in fact, maybe some outliers in the dataset that may indicate a problem.”