industry news
Subscribe Now

MLPerf Releases Over 500 Inference Benchmark Results, Showcasing a Wide Range of Machine Learning Solutions

Mountain View, CA – November 6, 2019 – After introducing the first industry-standard inference benchmarks in June of 2019, today the MLPerf consortium released over 500 inference benchmark results from 14 organizations. These benchmarks measure how quickly a trained neural network can process new data for a wide range of applications (autonomous driving, natural language processing, and many more) on a variety of form factors (IoT devices, smartphones, PCs, servers and a variety of cloud solutions). The results of the benchmarks are available on the MLPerf website at https://mlperf.org/

“All released results have been validated by the audits we conducted,” stated Guenther Schmuelling, MLPerf Inference Working Group Co-chair from Microsoft. “We were very impressed with the quality of the results. This is an amazing number of submissions in such a short time since we released these benchmarks this summer. It shows that inference is a growing and important application area, and we expect many more submissions in the months ahead.” 

“Companies are embracing these benchmark tests to provide their customers with an objective way to measure and compare the performance of their machine learning solutions,” stated Carole-Jean Wu, Inference Working Group Co-chair from Facebook. “There are many cost- performance tradeoffs involved in inference applications. These results will be invaluable for companies evaluating different solutions.” 

Of the over 500 benchmark results released today, 182 are in the Closed Division intended for direct comparison of systems. The results span 44 different systems. The benchmarks show a 5-order-of-magnitude difference in performance and a 3-order-of-magnitude range in estimated power consumption and range from embedded devices and smartphones to large-scale data center systems. The remaining 429 open results are in the Open Division and show a more diverse range of models, including low precision implementations and alternative models. 

Companies in China, Israel, Korea, the United Kingdom, and the United States submitted benchmark results. These companies include: Alibaba, Centaur Technology, Dell EMC, dividiti, FuriosaAI, Google, Habana Labs, Hailo, Inspur, Intel, NVIDIA, Polytechnic University of Milan, Qualcomm, and Tencent. 

Future versions of MLPerf will include additional benchmarks such as speech-to-text and recommendation, and additional metrics such as power consumption. MLPerf is also developing a smartphone app that runs inference benchmarks for use with future versions. “We are actively soliciting help from all our members and the broader community to make MLPerf better,” stated Vijay Janapa Reddi, Associate Professor, Harvard University, and MLPerf Inference Working Group Co-chair. 

“Having independent benchmarks help customers understand and evaluate hardware products in a comparable light. MLPerf is helping drive transparency and oversight into machine learning 

performance that will enable vendors to mature and build out the AI ecosystem. Intel is excited to be part of the MLPerf effort to realize the vision of AI Everywhere,” stated Dr Naveen Rao, Corp VP Intel, GM AI Products. 

Additional information about these benchmarks are available at https://mlperf.org/inference- overview/. The MLPerf Inference Benchmark whitepaper is available at https://edge.seas.harvard.edu/files/edge/files/mlperf_inference.pdf. The MLPerf Training Benchmark whitepaper is available at https://arxiv.org/abs/1910.01500

About 

MLPerf’s mission is to build fair and useful benchmarks for measuring training and inference performance of ML hardware, software, and services. MLPerf was founded in February, 2018 as a collaboration of companies and researchers from educational institutions. MLPerf is presently led by volunteer working group chairs. MLPerf could not exist without open source code and publicly available datasets others have generously contributed to the community.

Leave a Reply

featured blogs
Dec 11, 2019
A popular demonstration at Super Computing 2019 (SC19) was a collaboration between Samtec and eSilicon that emulates next generation data center architectures for typical 19 inch rack mount applications. In the video above, Ralph Page, Systems Architect at Samtec, walks us th...
Dec 11, 2019
Do you want to quickly view a schematic, layout, or IC package without installing the complete OrCAD product suite? Get read-only access to design data created in OrCAD Capture , PCB Editor , or... [[ Click on the title to access the full blog on the Cadence Community site. ...
Dec 10, 2019
I now expect widespread deployment of fully autonomous cars by about 2030. Do you agree with this assessment, or do you think I'm being overly optimistic?...
Dec 6, 2019
[From the last episode: We looked at the role of interrupts in a processor, and how they help save energy in battery-powered IoT devices.] OK, time to take a breath. We'€™ve covered a wide range of computing concepts; time for a quick review. We started by looking at the ro...