[Tel Aviv, Israel, September 8, 2022] – Deci, the deep learning company harnessing Artificial Intelligence (AI) to build better AI, announced results for its Natural Language Processing (NLP) inference model submitted to the MLPerf Inference v2.1 benchmark suite under the open submission track. Generated by Deci’s Automated Neural Architecture Construction (AutoNAC) technology, the NLP model, dubbed DeciBERT-Large, ran on Dell-PowerEdge-R7525-2 hardware using the AMD EPYC^TM 7773X processor. The resulting model outperformed both the throughput performance of the BERT-Large model by 6.46x and achieved a 1% boost in accuracy.

The model was submitted under the offline scenario in MLPerf’s open division in the BERT 99.9 category. The goal was to maximize throughput while keeping the accuracy within a 0.1% margin of error from the baseline, which is 90.874 F1 (SQUAD). The DeciBERT-Large model far exceeded these goals, reaching a throughput of 116 QueriesPer Second (QPS) and an F1 score of 91.08 for accuracy.

For the submission, Deci leveraged its proprietary automated Neural Architecture Construction technology (AutoNAC) engine to generate a new model architecture tailored for the AMD processor. AutoNAC, an algorithmic optimization engine generating best-in-class deep learning model architectures for any task, data set, and inference hardware, typically powers up to a 5X increase in inference performance with comparable or higher accuracy relative to state-of-the-art neural models.

“While the key optimization objective when generating the DeciBERT model was to optimize throughput, AutoNAC also managed to significantly reduce the model size – an important accomplishment with a number of benefits including the ability to run multiple models on the same server and better utilize cache memory,” said Prof.  Ran El-Yaniv, Deci’s chief scientist and co-founder. “These results confirm once again the exceptional performance of our AutoNAC technology, which is applicable to nearly any deep learning domain and inference hardware”.

MLPerf gathers expert deep learning leaders to build fair and useful benchmarks for measuring training and inference performance of ML hardware, software, and services.

The Impact of Faster NLP Inference 

Deci’s NLP inference acceleration directly translates into cloud cost reduction as it enables more processes to run on the same machine in less time or alternatively it enables teams to use a more cost efficient machine while retaining the same throughput performance. For some NLP applications such as question answering, higher throughput also means better user experience as the queries are processed faster and insights can be generated in real time.

Table 1: Deci Submission Results

About Deci

Deci enables deep learning to live up to its true potential by using AI to build better AI. With the company’s deep learning development platform, AI developers can build, optimize, and deploy faster and more accurate models for any environment including cloud, edge, and mobile, allowing them to revolutionize industries with innovative products. The platform is powered by Deci’s proprietary automated Neural Architecture Construction technology (AutoNAC), which empowers data scientists to build best-in-class deep learning models that are tailored for any task, data set and target inference hardware. Leading AI teams use Deci to accelerate inference performance, enable new use cases on limited hardware, shorten development cycles and reduce computing costs. Founded by Yonatan Geifman, PhD, Jonathan Elial, and Professor Ran El-Yaniv, Deci’s team of deep learning engineers and scientists are dedicated to eliminating production-related bottlenecks across the AI lifecycle.