industry news
Subscribe Now

MLPerf Releases Over 500 Inference Benchmark Results, Showcasing a Wide Range of Machine Learning Solutions

Mountain View, CA – November 6, 2019 – After introducing the first industry-standard inference benchmarks in June of 2019, today the MLPerf consortium released over 500 inference benchmark results from 14 organizations. These benchmarks measure how quickly a trained neural network can process new data for a wide range of applications (autonomous driving, natural language processing, and many more) on a variety of form factors (IoT devices, smartphones, PCs, servers and a variety of cloud solutions). The results of the benchmarks are available on the MLPerf website at https://mlperf.org/

“All released results have been validated by the audits we conducted,” stated Guenther Schmuelling, MLPerf Inference Working Group Co-chair from Microsoft. “We were very impressed with the quality of the results. This is an amazing number of submissions in such a short time since we released these benchmarks this summer. It shows that inference is a growing and important application area, and we expect many more submissions in the months ahead.” 

“Companies are embracing these benchmark tests to provide their customers with an objective way to measure and compare the performance of their machine learning solutions,” stated Carole-Jean Wu, Inference Working Group Co-chair from Facebook. “There are many cost- performance tradeoffs involved in inference applications. These results will be invaluable for companies evaluating different solutions.” 

Of the over 500 benchmark results released today, 182 are in the Closed Division intended for direct comparison of systems. The results span 44 different systems. The benchmarks show a 5-order-of-magnitude difference in performance and a 3-order-of-magnitude range in estimated power consumption and range from embedded devices and smartphones to large-scale data center systems. The remaining 429 open results are in the Open Division and show a more diverse range of models, including low precision implementations and alternative models. 

Companies in China, Israel, Korea, the United Kingdom, and the United States submitted benchmark results. These companies include: Alibaba, Centaur Technology, Dell EMC, dividiti, FuriosaAI, Google, Habana Labs, Hailo, Inspur, Intel, NVIDIA, Polytechnic University of Milan, Qualcomm, and Tencent. 

Future versions of MLPerf will include additional benchmarks such as speech-to-text and recommendation, and additional metrics such as power consumption. MLPerf is also developing a smartphone app that runs inference benchmarks for use with future versions. “We are actively soliciting help from all our members and the broader community to make MLPerf better,” stated Vijay Janapa Reddi, Associate Professor, Harvard University, and MLPerf Inference Working Group Co-chair. 

“Having independent benchmarks help customers understand and evaluate hardware products in a comparable light. MLPerf is helping drive transparency and oversight into machine learning 

performance that will enable vendors to mature and build out the AI ecosystem. Intel is excited to be part of the MLPerf effort to realize the vision of AI Everywhere,” stated Dr Naveen Rao, Corp VP Intel, GM AI Products. 

Additional information about these benchmarks are available at https://mlperf.org/inference- overview/. The MLPerf Inference Benchmark whitepaper is available at https://edge.seas.harvard.edu/files/edge/files/mlperf_inference.pdf. The MLPerf Training Benchmark whitepaper is available at https://arxiv.org/abs/1910.01500

About 

MLPerf’s mission is to build fair and useful benchmarks for measuring training and inference performance of ML hardware, software, and services. MLPerf was founded in February, 2018 as a collaboration of companies and researchers from educational institutions. MLPerf is presently led by volunteer working group chairs. MLPerf could not exist without open source code and publicly available datasets others have generously contributed to the community.

Leave a Reply

featured blogs
May 18, 2022
Learn how award-winning ARC processor IP powers automotive functional safety tech, from automotive sensors to embedded vision systems, alongside AI algorithms. The post Award-Winning Processors Drive Greater Intelligence and Safety into Autonomous Automotive Systems appeared...
May 18, 2022
The Virtuoso Education Kit has just been released and now there is already a new kit available: The Organic Printed Electronics PDK Education Kit ! This kit also uses Virtuoso as the main Cadence... ...
May 12, 2022
By Shelly Stalnaker Every year, the editors of Elektronik in Germany compile a list of the most interesting and innovative… ...
Apr 29, 2022
What do you do if someone starts waving furiously at you, seemingly delighted to see you, but you fear they are being overenthusiastic?...

featured video

EdgeQ Creates Big Connections with a Small Chip

Sponsored by Cadence Design Systems

Find out how EdgeQ delivered the world’s first 5G base station on a chip using Cadence’s logic simulation, digital implementation, timing and power signoff, synthesis, and physical verification signoff tools.

Click here for more information

featured paper

Intel Agilex FPGAs Deliver Game-Changing Flexibility & Agility for the Data-Centric World

Sponsored by Intel

The new Intel® Agilex™ FPGA is more than the latest programmable logic offering—it brings together revolutionary innovation in multiple areas of Intel technology leadership to create new opportunities to derive value and meaning from this transformation from edge to data center. Want to know more? Start with this white paper.

Click to read more

featured chalk talk

Power Profiler II

Sponsored by Mouser Electronics and Nordic Semiconductor

If you are working on a low-power IoT design, you are going to face power issues that can get quite complicated. Addressing these issues earlier in your design process can save you a lot of time, effort, and frustration. In this episode of Chalk Talk, Amelia Dalton chats with Kristian Sæther from Nordic Semiconductor about the details of the new Nordic Power Profiler Kit II - including how it can measure actual current, help you configure the right design settings, and show you a visualized power profile for your next design.

Click here for more information about the Nordic Semiconductor Power Profiler Kit II (PPK2)