editor's blog
Subscribe Now

Making the Line Move Faster

No one likes standing in line, but if you’re going to be doing any serious parallel processing, you’ll run into many queues as a way for threads or processes to send messages to each other. Usually they’re implemented in software, which adds a level of overhead to the programs, in particular when putting things on or taking them off the queue.

For instance, when putting a new item into the queue, you have to check to see if there is room in the queue, and you have to upgrade the head afterwards, and, by the time all is said and done, you’ve chewed up 10 instructions. In addition, the queue information may or may not be in the cache, and you have to ensure exclusive access where critical, and, well, it can get slow enough to where you want to limit how many things work in parallel and how much they communicate. If the program is split up too finely, then too much time is lost to the not-so-instant messaging between processes or threads

In order to alleviate this, hardware queues have been proposed. But these interrupt the micro-architecture and require custom interconnect. Worse yet, they’re not well handled by operating systems, especially when it comes to context switches. And architects have the unenviable job of deciding how big to make them and how many to provide: you know they’ll never get that right in everyone’s eyes.

In one of those random encounters of something interesting, I ran across a recent report by Lee et al from the University of North Carolina that takes a middle road for single-producer/single-consumer queues: hardware-accelerated software queues. In so doing, there were four primary criteria they wanted to meet:

  1. The frequent enqueue and dequeue tasks have to be as efficient as possible. Hardware queues meet this; software queues don’t.
  2. The time between one entity putting something on the queue and something else taking it off should be as short as possible. Again, hardware queues can do this; software queues, not so much.
  3. They should be as easy to program as software queues (quantity, synchronization, etc.). Software queues meet this by definition; hardware queues don’t at the very least by virtue of their limited quantity and size.
  4. They have to work without changing the OS. Because software queues work in the application memory space, they can do this; hardware queues can’t.

The abridged version of what they do can be summarized by four primary points:

  • While still implementing the queue in memory, cache a local copy of the queue head, tail, and size in a separate dedicated hardware table. This table stores some number of active queues much the way a memory cache stores some number of active memory addresses. This means that the various bookkeeping steps can be done without going to memory and without contention from anyone else.
  • Pipeline the queue operations into three steps: address generation, the actual store or load, and index updating.
  • Use dedicated hardware to calculate the address of the store or load. Again, this happens in private hardware without interference; multiple addresses can be generated in a single operation. The actual load or store can happen when the address is ready, meaning that the queue operations may happen out of order (if an early one takes longer to have its address resolved, for instance).
  • Accumulate index updates and store a bunch of them at the same time to reduce the amount of access to the cache.

Of course, as with everything, the details hide much devilry, so they have special considerations to handle misspeculation, precise interrupts, a full queue cache, fence (memory barrier) instructions, and avoiding livelock (since, in theory, the various queue indices could reside on three different memory pages that an OS may not be able to keep in memory at the same time).

With this solution, Criterion 1 is met because of the hardware acceleration, as is Criterion 2. Because the actual queues are still implemented in the application memory space, with no specific size or quantity limits, Criteria 3 and 4 are met.

For details on all of this as well as the results of their testing, you can check out the paper courtesy of James Tuck, one of the authors.

Leave a Reply

featured blogs
Sep 21, 2020
Technology is changing the strategies we use to do things - oh so fast that 2010 seems like a distant past- within many spaces -- including the way we do our current topic of interest - Timing... [[ Click on the title to access the full blog on the Cadence Community site. ]]...
Sep 21, 2020
Semicon, the world’s largest semiconductor conference and exhibition, is September 23-25 in Taiwan. Like most shows of its size and caliber, Semicon boasts a long and illustrious list of exhibitors (500+), and countless forums, symposiums, and workshops. Of course Semic...
Sep 18, 2020
[From the last episode: We put the various pieces of a memory together to show the whole thing.] Before we finally turn our memory discussion into an AI discussion, let'€™s take on one annoying little detail that I'€™ve referred to a few times, but have kept putting off. ...
Sep 16, 2020
In addition to the Great Highland (Scottish) bagpipes, the Uilleann (Irish) bagpipes, and the Northumbrian (English) bagpipes, there are myriad other offerings spanning the globe....

Featured Video

Four Ways to Improve Verification Performance and Throughput

Sponsored by Cadence Design Systems

Learn how to address your growing verification needs. Hear how Cadence Xcelium™ Logic Simulation improves your design’s performance and throughput: improving single-core engine performance, leveraging multi-core simulation, new features, and machine learning-optimized regression technology for up to 5X faster regressions.

Click here for more information about Xcelium Logic Simulation

Featured Paper

Helping physicians achieve faster, more accurate patient diagnoses with molecular test technology

Sponsored by Texas Instruments

Point-of-care molecular diagnostics (PoC) help physicians achieve faster, more accurate patient diagnoses and treatment decisions. This article breaks down how molecular test technology works and the building blocks for a PoC molecular diagnostics analyzer sensor front end system.

Read the Article

Featured Chalk Talk

Thermal Bridge Technology

Sponsored by Mouser Electronics and TE Connectivity

Recent innovations can make your airflow cooling more efficient and effective. New thermal bridges can outperform conventional thermal pads in a number of ways. In this episode of Chalk Talk, Amelia Dalton chats with Zach Galbraith of TE Connectivity about the application of thermal bridges in cooling electronic designs.

More information about TE Thermal Bridge Technology