industry news
Subscribe Now

MLPerf Releases Over 500 Inference Benchmark Results, Showcasing a Wide Range of Machine Learning Solutions

Mountain View, CA – November 6, 2019 – After introducing the first industry-standard inference benchmarks in June of 2019, today the MLPerf consortium released over 500 inference benchmark results from 14 organizations. These benchmarks measure how quickly a trained neural network can process new data for a wide range of applications (autonomous driving, natural language processing, and many more) on a variety of form factors (IoT devices, smartphones, PCs, servers and a variety of cloud solutions). The results of the benchmarks are available on the MLPerf website at https://mlperf.org/

“All released results have been validated by the audits we conducted,” stated Guenther Schmuelling, MLPerf Inference Working Group Co-chair from Microsoft. “We were very impressed with the quality of the results. This is an amazing number of submissions in such a short time since we released these benchmarks this summer. It shows that inference is a growing and important application area, and we expect many more submissions in the months ahead.” 

“Companies are embracing these benchmark tests to provide their customers with an objective way to measure and compare the performance of their machine learning solutions,” stated Carole-Jean Wu, Inference Working Group Co-chair from Facebook. “There are many cost- performance tradeoffs involved in inference applications. These results will be invaluable for companies evaluating different solutions.” 

Of the over 500 benchmark results released today, 182 are in the Closed Division intended for direct comparison of systems. The results span 44 different systems. The benchmarks show a 5-order-of-magnitude difference in performance and a 3-order-of-magnitude range in estimated power consumption and range from embedded devices and smartphones to large-scale data center systems. The remaining 429 open results are in the Open Division and show a more diverse range of models, including low precision implementations and alternative models. 

Companies in China, Israel, Korea, the United Kingdom, and the United States submitted benchmark results. These companies include: Alibaba, Centaur Technology, Dell EMC, dividiti, FuriosaAI, Google, Habana Labs, Hailo, Inspur, Intel, NVIDIA, Polytechnic University of Milan, Qualcomm, and Tencent. 

Future versions of MLPerf will include additional benchmarks such as speech-to-text and recommendation, and additional metrics such as power consumption. MLPerf is also developing a smartphone app that runs inference benchmarks for use with future versions. “We are actively soliciting help from all our members and the broader community to make MLPerf better,” stated Vijay Janapa Reddi, Associate Professor, Harvard University, and MLPerf Inference Working Group Co-chair. 

“Having independent benchmarks help customers understand and evaluate hardware products in a comparable light. MLPerf is helping drive transparency and oversight into machine learning 

performance that will enable vendors to mature and build out the AI ecosystem. Intel is excited to be part of the MLPerf effort to realize the vision of AI Everywhere,” stated Dr Naveen Rao, Corp VP Intel, GM AI Products. 

Additional information about these benchmarks are available at https://mlperf.org/inference- overview/. The MLPerf Inference Benchmark whitepaper is available at https://edge.seas.harvard.edu/files/edge/files/mlperf_inference.pdf. The MLPerf Training Benchmark whitepaper is available at https://arxiv.org/abs/1910.01500

About 

MLPerf’s mission is to build fair and useful benchmarks for measuring training and inference performance of ML hardware, software, and services. MLPerf was founded in February, 2018 as a collaboration of companies and researchers from educational institutions. MLPerf is presently led by volunteer working group chairs. MLPerf could not exist without open source code and publicly available datasets others have generously contributed to the community.

Leave a Reply

featured blogs
Dec 8, 2022
You will notice a big change when you try to download the latest version of Innovus, Genus, or Joules on our Cadence download site, downloads.cadence.com . Instead of the expected INNOVUS221 or GENUS221, or JOULES221 releases, you will find DDI221, which includes the 22.1 ver...
Dec 7, 2022
We explore hyperscale datacenters & internet traffic's impact on climate change and discuss how energy-efficient system design shapes a sustainable future. The post How the Electronics Industry Can Shape a More Sustainable, Energy-Efficient World appeared first on From ...
Dec 7, 2022
By Karen Chow When Infineon needed to select a field solver for the development of their next-generation power semiconductor products,… ...
Nov 18, 2022
This bodacious beauty is better equipped than my car, with 360-degree collision avoidance sensors, party lights, and a backup camera, to name but a few....

featured video

How to Harness the Massive Amounts of Design Data Generated with Every Project

Sponsored by Cadence Design Systems

Long gone are the days where engineers imported text-based reports into spreadsheets and sorted the columns to extract useful information. Introducing the Cadence Joint Enterprise Data and AI (JedAI) platform created from the ground up for EDA data such as waveforms, workflows, RTL netlists, and more. Using Cadence JedAI, engineering teams can visualize the data and trends and implement practical design strategies across the entire SoC design for improved productivity and quality of results.

Learn More

featured chalk talk

Enabling Digital Transformation in Electronic Design with Cadence Cloud

Sponsored by Cadence Design Systems

With increasing design sizes, complexity of advanced nodes, and faster time to market requirements - design teams are looking for scalability, simplicity, flexibility and agility. In today’s Chalk Talk, Amelia Dalton chats with Mahesh Turaga about the details of Cadence’s end to end cloud portfolio, how you can extend your on-prem environment with the push of a button with Cadence’s new hybrid cloud and Cadence’s Cloud solutions you can help you from design creation to systems design and more.

Click here for more information about Cadence Cloud Portfolio