As it happens, the key instruction difference between AMD and NVIDIA GPUs was that AMD GPUs had a rotate instruction and NVIDIA GPUs did not.

Rotate, of course, is the family of instructions closely related to the shift instructions, where instead of shifting predefined values such as 0’s (for unsigned shifts) or sign bits (for signed or arithmetic shifts), the bits shifted out of one “end” of the number are shifted into the other. If shifting right, for example, the least significant bits are shifted into the most significant end of the word.

How did NVIDIA respond to this competitive pressure?

Did they leave AMD’s challenge unanswered?

No, of course not.

Did they add a rotate instruction?

Also no.

What NVIDIA added in their Kepler (SM 3.0) architecture was a set of funnel shift instructions: instructions that concatenate two (2) registers and shift them as a unit.

One application for funnel shift instructions is to implement memory copies where the source and destination pointers are misaligned with respect to one another. In classic RISC, architectures such as the Alpha would signal an exception if the program attempted a 32-bit read of an address that was not 32-bit aligned (least significant 2 bits set to zero). The modern equivalent to this anachronism: on x86, instructions such as MOVAPS will signal an exception if the effective address is not naturally aligned with respect to the operand size (32- and 64-byte for AVX2 and AVX512, respectively).

On such architectures, memory copies where the source and destination pointers both are aligned are trivial, and if both pointers are “relatively aligned” (misaligned in the same way), the misaligned portion of the copy can be dealt with by small amounts of prologue and epilogue code. But for memory copies where the source and destination pointer are not even relatively aligned, there’s no getting away from misaligned memory accesses of either the source or destination pointer – unless you happen to have a funnel shift in your arsenal. In that case, a memory copy can be formulated entirely in terms of aligned memory transactions, with the shifted operands dealt with in the innermost loop of the copy.

But another, more subtle application for funnel shift is that if the two input operands are the same register, it becomes a rotate.

So NVIDIA was able to close the ISA gap with AMD, not by emulating exactly what AMD had done, but by implementing something else that subsumed the target capability.

Enter LOP3.LUT

It turns out NVIDIA, home to some of the most capable CPU architects in the world, is pretty good at designing such machine instructions: expansive new capabilities in exchange for modest hardware cost. Such instructions work best when the compiler can use them without forcing developers to access them using intrinsic functions.

Another instruction set innovation, not unlike funnel shifts, that NVIDIA deployed in the Maxwell generation: the FPGA-like LOP3.LUT instruction. Like the age-old rasterops (ROPs) used to specify how BitBlt (bit block transfer) operations manipulate pixels, LOP3.LUT implements a general-purpose family of logic operations that can be defined by an 8-bit immediate operand. The usual suspect logic operations AND, OR, XOR, NOT, and variants thereof (e.g. NAND) all are implementable in terms of LOP3.LUT, but so are novel amalgams of other Boolean expressions.

Why is it called LOP3.LUT? Well.. remember when Boolean operations were first introduced to you, and you learned about truth tables? Consider for example the truth tables for A|B and A&B, shown here in code form to work around limitations of Substack:

  A    B    A|B  A&B
  0    0     0    0
  0    1     1    0
  1    0     1    0
  1    1     1    1

With 2 inputs that can have 2 possible values, there are four possible outputs.

Now consider the truth table for a 3-operand logic operation:

  A	B    C	Result
  0	0    0    ?
  0	0    1	  ?
  0	1    0	  ?
  0	1    1	  ?
  1	0    0	  ?
  1	0    1	  ?
  1	1    0	  ?
  1	1    1	  ?

With 3 bits’ input, there are 2³ or 8 bits’ worth of possible outputs, and this output may be specified as an 8-bit immediate to the instruction - filling in the rightmost column of the truth table above. As described here on StackOverflow, the immediate needed for any generalized Boolean function of 3 inputs can be computed by throwing three magical values (0xF0, 0xCC, and 0xAA), derived from the left-hand side of the truth tables, through said function.

So confident is NVIDIA that their compiler can exploit this instruction when developers write complicated logical expressions, that it’s not even available as an intrinsic. (It is, of course, available via inline PTX if necessary.)

Byte Permutes

The first generation of CUDA-capable hardware had some ISA properties that did not make it to the second generation. One such was the ability to address sub-registers: the top and bottom 16-bit halves of the 32-bit registers. For applications like image processing, where 16-bit pixel values often are more than adequate, this hardware capability enabled kernels to be built with a smaller register footprint. But such applications weren’t numerous enough to justify continued support for the hardware feature, especially in light that NVIDIA’s compiler at the time was having difficulty taking advantage of the feature. That prioritization is understandable given that NVIDIA was otherwise occupied, adding major architectural features like caches and critical instructions like fused multiply-add.

Later, as CUDA found application in broader and broader market segments, the need to access subregisters found its way back onto NVIDIA’s priority list. Instead of half-registers, which required opcode space to be allocated for any instructions designed to consume them, NVIDIA instead provided instructions to pick apart 32-bit registers and put them back together in 8-bit chunks: the so-called byte permute instructions, which take two (2) 32-bit registers and return a 32-bit mashup of the inputs, where the 8-bit lanes are selected from the two inputs based on an immediate.

Although available as intrinsics, NVIDIA’s compiler team has done a nice job of exploiting the byte permute instructions when possible. Whenever you write code that OR’s, shifts, and/or masks in 8-bit increments, you’re likely to find some combination of byte permutes (or funnel shifts, or LOP3.LUT!) in the instruction mix.

Coding with Claude is like having a talented Ph.D. computer science candidate who can type really fast: a very good coder tactically, with at-time huge blind spots that must be dealt with intentionally. It is excellent at relatively self-contained coding tasks, but still needs close air support to do a really good job.

Updating the chapter on Normalized Cross Correlation was mostly free beer, though. I had wanted to get rid of the 24-bit multiplies (they were rendered obsolete with Fermi c. 2011), but Claude had the insight that the DP4A instruction (added with Pascal) was a great fit for the workload. It ported the key kernels to use the new instruction and, when we saw that it was no faster, refactored the kernel to use ILP for a 2.4x speedup. The increased parallelism and latency tolerance from this refactor unlocked the latent performance opportunities of DP4A: re-applying DP4A increased performance by another 3x! I’d call a >7x performance improvement a big win, and Claude deserves the bulk of the credit.

More recently, I undertook to rewrite the Scan chapter, and that set us on a journey that’s worth recounting. By the way, the opportunity cost of writing this article is why I haven’t been documenting similar journeys as they occurred. When you can get a month’s worth of work done in a day, pausing to document progress feels like a real cost. But my readers deserve to know at least a little bit about how I’ve been using Claude to update The CUDA Handbook and the accompanying source code.

Read more

This week has been busy. The Scan and Reduction chapters have been rewritten to reflect current reality: warp intrinsics have now been around for long enough that we start our coverage with warp-sized problems before leading into block- and grid-wide ones.

The Scan chapter needed a significant update: we now lead with the Algebra Of Scans framing and, for clarity, got rid of the awkward Blelloch (upsweep/downsweep) formulation and now start with the algorithm of Duane Merrill’s Ph.D. thesis, which was the best-in-class algorithm at the time of publication of the First Edition. Merrill has since rendered that algorithm obsolete with the bandwidth-limited decoupled lookback formulation, published with Michael Garland, now available via the CUB and Thrust utility libraries. With Claude’s help, I was able to write a sample implementation of the decoupled lookback and port the stream compaction sample to use that algorithm, as well.

The Software Architecture chapter has been updated for currency. It now mentions the debugger and profiler, and coverage of versioning, cloud vendors, and other topics has been updated. I may add some coverage of profiling in other parts of the book, though profiling tutorials tend to seem rhetorical as opposed to possessing the investigative quality that motivates most usage of profilers.

I added coverage of NVRTC with a sample in the Reduction chapter and sample code, which prompted a repository-wide refactor of the error handling macros. Having a macro that imposed the policy of naming a local variable status had outstayed its welcome (the NVRTC sample caused namespace collisions in collocated CUDART and driver API code), so we renamed to status_cudart and Error_cudart. The driver API and NVRTC variants are status_cuda/Error_cuda and status_nvrtc/ Error_nvrtc, respectively.

I also got rid of some dated utility APIs in chLib: the threading API has been replaced by std::thread, and the high-resolution timing API has been replaced by std::chrono::steady_clock.

In terms of coverage of the topics outlined in the Table of Contents, the biggest gap now in the text and code is coverage of multi-GPU programming. The state of the art has advanced a great deal since publication of the First Edition, when multiple GPUs typically were made available on PCI Express via switches and bridge chips. Since then, NVIDIA has invested heavily in both scale-up and scale-out technologies, developing NVLink for cache-coherent chip interconnects and acquiring Mellanox for their best-in-class Infiniband network cards. Multi-GPU programming deserves a rewrite to cover these hardware technologies and the corresponding APIs.

One especially divisive coding practice is one that I’ve adopted since the late 1990s, after I asked one of my developers (I was his manager) why he was writing the code that way. He’d written something like:

if ( NULL == wndptr )

which, to some developers, looks awkward because they expect the more complex expression to be on the left of the == operator.

The awkwardness led to the name Yoda Notation, after the odd word ordering in the Jedi Master’s speech. (And now’s as good a time as any to mention that there are articles arguing against use of Yoda notation, though this one brims with strawman energy.)

At the time, when I asked, Why lead with the 0? Why not put wndptr on the left?, my employee said, Well there’s a typo and I only put one equal sign in the expression, if 0 is on the left, the code won’t compile because it’s not an l-value. That’s an oft-cited reason that generally is no longer true – most compiler will catch that error – but I still tend to put the constants on the left side in such expressions for another reason. When I write:

if ( 0 == threadIdx.x ) {

it’s as much for dramatic effect as to guard against the possibility of typos.

It turns out there’s a name for the deliberate misordering of words for strategic effect: it’s a literary device known as anastrophe.

Another, less visible example (you’ll soon see why) of anastrophe that finds expression in my code is when I’m writing instrumentation that should never find its way into production code: I don’t indent it at all. Sometimes it stands out like a sore thumb – but that’s the point – I want it to stand out so it gets removed before getting pushed.

The final argument in favor of Yoda Notation is that you have to write the code somehow, so any additional information you can bake into the source code is useful.

Anyway, once my employee explained his reasoning, I embraced the idea and, left to my own devices, I use it most of the time. It’s a low-grade way to Write Code For The Next Guy. As with all coding conventions, rules are made to be broken and your mileage may vary! But if you have ever wondered why my code features Yoda Notation, now you know.

With Claude’s help, I re-ran all the benchmarks in the sample source code and updated the text and figures accordingly. A reverse tunnel on my home network enabled Claude, running in VSCode on a non-GPU-capable Amazon EC2 instance, to operate the Linux machine in my basement. Once that setup was finalized, I was able to edit the text, figures, and source code with a heavy hand.

That exercise featured some notable exchanges with Claude:

• Predication v divergence: The example I chose for the first edition of The CUDA Handbook to highlight the different ways GPU handle conditional code used a loop emitted by the compiler to perform atomic updates to shared memory. Since more-capable hardware has made that loop unnecessary, I told Claude that we needed a better example of predication versus divergence, and it immediately suggested the Collatz Conjecture. I did not even know what the Collatz Conjecture was, but as soon as I read up on it, I agreed that it was the perfect example to serve the pedagogical purposes of that section of the book.

• The chapter on Normalized Cross-Correlation was in need of updating (NVIDIA long ago obsoleted the need to use 24-bit multiply intrinsics for performance), and amidst that exercise, Claude identified __dp4a() as a performance optimization opportunity. This instruction computes lane-wise multiplications between the four 8-bit lanes of the 32-bit operands. After profiling the application, we realized that using __dp4a() did not increase performance because the kernel was occupancy-limited. After refactoring the kernel to leverage TLP (compute multiple outputs per GPU thread) for a 2.4x speed increase, computing the sums with __dp4a() increased performance by another 3x for a 7x speed increase on the target hardware.

I am still busy updating the contents of The CUDA Handbook, but enough has changed over the last few weeks that I asked Claude this morning:
Can you give an overview of the changes we’ve made to the book text since first moving it to the website?

and it replied with this summary (delineated here by horizontal dividers):

Here’s the arc of what’s happened to the book text since it went to Markdown on July 4 — 209 commits across ~3½ weeks, touching every chapter and appendix. It breaks into a few distinct campaigns:

1. Migration fidelity (Jul 4–9)

Getting the Markdown to faithfully match the print book: restoring values and whole sentences lost in conversion (checked against the print copy), fixing off-by-one listing numbers, duplicate/out-of-sequence section numbers, empty-heading artifacts, ad-slot corruption, and 13+ dead external links. Listings stopped being flowed text and are now sourced verbatim from the code repo at build time (31 listings) and linked to GitHub.

2. Copy-editing & typography (Jul 6–10)

Several sweeping passes — “~230 mechanical fixes,” a numbering/cross-reference repair pass (~35), plus ongoing work: code-font for identifiers/filenames/program names, italicized function mentions, real pipe tables (fixing split-word artifacts like “Re/turns”), and IMPORTANT/NOTE rendered as callout boxes.

3. Idiom & API modernization (book-wide)

Adopted the cuda()/cu() error-check macros throughout; replaced cudaThreadSynchronize() → cudaDeviceSynchronize(); and dropped dated CUDA-version qualifiers from feature descriptions across the whole book.

4. Web/interactive features

Glossary hover-cards, an interactive memcpy picker (ch5, replacing Table 5-10 on screen), line-number gutters on all listings, and inlined/clickable citation footnotes.

5. Benchmark refresh on modern hardware (RTX 3060 / Ampere)

Re-ran the book’s in-place benchmarks chapter by chapter: ch5, ch6, ch7, ch10, ch11, ch14, ch15, ch16 — replacing book-era numbers, noting where old cliffs (register spilling, mapped-pinned anomaly) no longer reproduce.

6. Technical modernization & brand-new sections

The heaviest content work, by chapter:

• ch2 — new §2.7 GPU Interconnects, rewritten §2.2 Integrated GPUs and §2.3 Multiple GPUs, a per-generation §2.6 GPU Architecture rewrite, and (just now) §2.1.5 Packaging & Coherent Interconnects.

• ch8 (42 touches) — new Integer Dot Products / DPX, LOP3, _sync warp primitives + participation mask, deep-learning FP formats, and unified SASS tables Tesla→Blackwell.

• ch7 — new Thread Block Clusters, Dynamic Parallelism rewritten for CDP2, TLP/ILP naming.

• ch5 (47 touches) — new Managed Memory section, demand-paging update, GDRCopy.

• ch10 — rewritten for texture/surface objects; ch16 — GPU-resident Radix Sort + native shared-mem atomics; ch14/15 — modern N-body & correlation kernels; appc — recast as a warp-synchrony deprecation note.

7. Figures

Chapter 2 fully vectorized — all 57 figures redrawn in a consistent house style (the work we just finished). Elsewhere, missing performance charts were generated from the manuscript’s own data (ch6, ch16).

8. Voice & editorial

A standing rule — never refer to the author in the third person — plus retrospective callouts where the first edition’s predictions either came true (source-JIT modules) or died (texturing-as-a-read-path).

Most-touched files: ch5 (47), ch8 (42), ch2 (32), ch6 (32), ch7 (27).

Net: the migration was the easy part — most of the effort has gone into making the 2nd-edition text current (Ampere→Blackwell, modern intrinsics, new architectural features) while tightening typography and wiring it for the web.

(The Ch13 research pass is still running in the background; I’ll bring the restructuring plan once it lands.)

That last parenthetical refers to the next task on our to-do list: like the Normalized Cross-Correlation work, Chapter 13 (Parallel Prefix Sum) needs more extensive revision, because the pedagogical value of the Blelloch formulation of Scan was questionable even during the first edition’s timeframe; and Merrill and Garland have since shipped the best-in-class, bandwidth-limited formulation that is now the industry standard.

There is more work to do, like modernizing usage of C++ (possibly retiring some of the chLib functionality in favor of std::thread, for example), adding coverage of Thrust and CUB and other utility libraries, adding a chapter on Matrix Multiplication and how to target Tensor Cores, and improving coverage of topics like Independent Thread Scheduling. Time and again, Claude has proven its usefulness in this endeavor. I am working on an article that will discuss the broader implications for labor and intellectual property, as I see them, but for now I will say that

1. I think coding tasks (the actual typing) can be automated and that trend will continue, and

2. Claude is at least as good at copy editing as humans - faster, more thorough, compliant (I can just tell Claude that “data is” and that’s the beginning and end of the discussion) - so although I think AI technology has dire implications for makework labor, in my experience it has amplified the scope of my aspiration and enabled me to build things, not just faster or more easily than I would have without its help, but things that I wouldn’t otherwise have been able to build at all.

The technology has evolved in eye-popping fashion. The first CUDA-capable GPU was a then-whopping 684M transistors – the biggest chip TSMC feasibly could manufacture at the time – and its main market goal was to run 3D games. Today, a single Rubin chiplet is 168B transistors, 245x bigger than G80, for a growth rate of 30% per year across 20 years. Rubin chiplets also are designed to collaborate with their peers, with packaging innovations that enable multiple chiplets to emulate a single huge GPU (“two GPUs in a trenchcoat”) with performance and seamlessness that GPU architects of yore could only imagine. The addition of TensorCores requires special attention, especially in light that starting with Blackwell, they now have their own on-chip memory.

The state of the art has advanced in other areas. Duane Merrill, the NVIDIA researcher whose Ph.D. thesis (cited in The CUDA Handbook) was on optimized GPU scan algorithms, improved on that state of the art with this work, done in collaboration with Michael Garland. Merrill and Garland’s implementation is nicely packaged for use by developers in the CUB and Thrust libraries, which are not mentioned anywhere in The CUDA Handbook. In light of these developments, the chapter on Scan serves more as a historical reference than resource that can be brought to bear on practical applications.

Another area where the state of the art has evolved: The C++11 standard was still young when the first edition of The CUDA Handbook was published. Modern C++ is more mature, NVIDIA has built several libraries that leverage its features to make available CUDA capabilities in a more ergonomic fashion, and std::thread has withstood the test of time and enjoys widespread toolchain and platform support; so some of the CUDA Handbook library code could stand to be refactored.

Finally, although most of the benchmarking code is as relevant today as when it was written, the benchmark numbers themselves are laughably out of date. I have some ideas on how to proceed, but won’t share further details at this time.

So, there’s no shortage of work to do.

Here are the steps taken so far:

• The full text of The CUDA Handbook, including listings and figures, is now live on the website, and some measures have been taken to bring it onto the Internet and further into the 21^st Century: for example, the errata fixes have been applied to the source material, and listing links now reference the GitHub repository.

• All the ParallelProgrammer Substack posts have been moved to the Blog section, so the website is the canonical reference for the CUDA Handbook blog; and

• I worked with the good folks at ethicalads.ai to transition the website to be ad-supported, with the option to remove ads for paid members. Tentatively, membership is priced the same as a subscription to my Substack ($10/mo), with an annual fee of $99 and a lifetime fee of $199.

Paying Substack subscribers will be “grandfathered” in to the ad-free experience on cudahandbook.com, and we’ll be emailing you all soon.

For those wondering Why is there a revenue model at all?:

By making the text of The CUDA Handbook available in this way – easily accessible, and more dynamic/easily updated, but copyrighted with no authorization to present it elsewhere without express written permission – I am reclaiming ownership of the material in a way that we’ve never enjoyed before. When the book was just a one-time byproduct of a person-year in nights and weekends, the content was more passive, like a big thick business card, and it is no longer even in print1.

With this new model, I’m hoping it can become something more.

Additionally, besides the obvious quid pro quo between authors and their readers, a revenue model enables copyright claims to be asserted that allege real economic damages. There is at least one GitHub repository with an unauthorized PDF scan of The CUDA Handbook available for download, and Microsoft has refused to take it down despite my filing a specific complaint and ticket with a link to the file. If I threaten to sue for damages, they may rethink their strategic neglect.

Every author has to navigate the tension between notoriety and being fairly compensated for their work. In software, we often bias toward the former, with open source initiatives supported by indirect economic models. For the sample source code, making The CUDA Handbook’s source code freely available was never in question. For the text and the presentation of material, I’m hoping this avenue will enable more activity and for the material to be updated and to remain current.

If you have thoughts on the matter, don’t hesitate to email me. You may be surprised at how few people do!

…sometimes, and it's as true of authors as it is of readers, you have a life. People in your world get sick or die. You fall in love, or out of love. You move house. Your aunt comes to stay. You agreed to give a talk half-way around the world five years ago, and suddenly you realise that that talk is due now.....

In my case, I’ve been having adventures of every stripe, from health issues to family issues to job issues. In the spirit of Harlan Ellison’s nonfiction, I’ll bias for candor and say 1) I’m fine, I donated a kidney, which required about a three-month recovery, 2) my mother passed away, and it’s complicated, and 3) I left one job and started another, lucked into yet another job opportunity, and started work at the newest job last week. There are other issues that have hindered my productivity, and postings to this Substack have suffered accordingly. For that, I apologize and I’ll endeavor to do better moving forward. But, life keeps happening…

Anyway I wrote a long-form post on a different social media platform commemorating the life of one of my favorite author, Dan Simmons, who passed away recently from a stroke.

From this tweet I learned of the passing of the inimitable Dan Simmons, whose first fantasy novel, Song of Kali, won the World Fantasy Award; whose first horror novel, Carrion Comfort, won the Bram Stoker award; and whose first science fiction novel, Hyperion, won the Hugo Award. Just as Lord Of The Rings was a single work published in 3 volumes, Hyperion was published as The Hyperion Cantos (Hyperion and The Fall Of Hyperion), followed by Endymion and The Rise Of Endymion.

Simmons was not a hard SF author, not exactly. (For SF fans, ‘hard’ means SF written within the confines of science as we understood it when the work was written, though the distinction must be qualified by one of Arthur C. Clarke’s Laws: “Any sufficiently advanced technology is indistinguishable from magic.”) But his most transcendent work, like Clarke’s, tested the limits of our scientific understanding to explore how technology intersects with the human condition. Hyperion is widely-cited as a prescient take on artificial intelligence, with even members of Congress weighing in.

Simmons came into the field via a most unusual journey. He majored in English, an influence that found expression in his work through his modeling his own novels on famous literary works (Hyperion, for example—the first book, not The Hyperion Cantos—is modeled on Chaucer’s Canterbury Tales, with its stories within stories). And he entered the field later in life than most, submitting and getting rejections so many times that he registered for a writing seminar at his wife’s urging. By his account, if this seminar didn’t result in his learning how to become a professional writer, he would abandon the endeavor and focus on being a high school teacher.

That seminar changed his life. Because one of the teachers was Harlan Ellison, the pugnacious and incandescently gifted author whose work defies categorization. Ellison was a genius in every sense of the word and did not suffer fools gladly.

When I wrote about Ellison’s legacy upon learning of his passing, I mentioned that he’d “discovered” Dan Simmons. The story of his discovery may be found in the Simmons story collection Prayers To Broken Stones. Ellison walked into the hallway with a slush pile of the execrable stories that get submitted to entry-level seminars, (unlike the famed Clarion West, the one Simmons was attending and that Ellison was teaching did not require attendees to be published authors). He’d gotten so frustrated at the poor quality of the submitted work that he had exploded at one of the attendees and mercilessly criticized their work in front of the whole class: the characters were cardboard, the plot predictable, the story wooden, etc.

I’ll try again to find Ellison’s account of Simmons’s discovery because I am not doing it justice here. By his account, he tore the hapless author of this work to shreds, then called for a break, not because he’d likely reduced a grown man to tears in front of a crowd of strangers, but because he’d asked the poor man how many books he’d written and found out that it was 60, or something. So what had sent Ellison into the hallway wasn’t just that his latest victim’s work was terrible, but that he had spent many years toiling away, getting feedback from editors that his work needed work, but never improving his craft enough to publish.

So Ellison was in the hallway, having publicly humiliated one of the participants in this seminar, when he hit upon the story Simmons had submitted. He started reading and fully finished it in the hallway, with tears streaming down his face. Once he’d recovered his composure and re-convened the seminar, he held up the story and asked the class: “Who. Wrote. This?!”

Simmons, with understandable reluctance, raised his hand. And Ellison delivered them a speech in praise of Simmons’s story (later his first publication, “The River Styx Runs Upstream”) that was at least equal to the one he’d just given about the other participant’s lesser work. In Ellison’s telling, it almost reads as if he condemned Simmons to be a writer, a calling where, by Ellison’s account, one does not choose the profession so much as it chooses you.

So Simmons got that story published in 1983 or so, the year he turned 35. And I wouldn’t say he was prolific, exactly, but he was productive, and acclaim followed him into every genre he tried.

This linked tweet is about Hyperion, which is getting renewed attention as AI takes over our economy and our collective imagination. And I can’t say too much about why Hyperion is so relevant to this day and age because Simmons had a gift for writing work that was 1) impossible to discuss without spoiling it, and 2) near-impossible to adapt to the screen. Hyperion has been rumored to be slated for adaptation, but Simmons issued a rare public statement to the effect that Disney had hired some of the best story doctors on the planet to solve that problem, and even they had come up empty. (The Terror, a fictionalized account of the disappearance of two British ships attempting to discover the Northwest Passage, was capably adapted by SyFy into a miniseries.)

Simmons reportedly had taken up the novel Carrion Comfort on a dare by editor Ellen Datlow, and his later work reads as if it, too, had been inspired by dares. Or perhaps he was just trying to integrate his various story ideas into unified works for the sake of efficiency. His blog Writing Well, now relegated to the Wayback Machine, features complaints about the folks who approach writers with ideas, as if writers want for ideas as opposed to time.

Simmons got started as a professional writer later in life than most, and he was taken from us by a stroke earlier than some. I wish he’d had more time to share his creativity with the world, but we can console ourselves by picking up and reading or re-reading Hyperion, or Carrion Comfort, or really any of the work that he published. One of the redeeming qualities of writing that is difficult to adapt, is that it invites back into a world where we read.

• It never ceases to amaze me how few people write with feedback, even on microblogging platforms like Twitter. So we authors are left to consult metrics to determine which topics caught readers’ attention.

• Given that I’m best known for my work on CUDA and The CUDA Handbook, I suppose it’s surprising that most of my technical Substack posts involved SIMD instructions like AVX2, not CUDA.

• I’m gratified at the number of people who’ve subscribed, especially the paid subscribers; as a professional writer since the late 1980s, I have unique perspective on the way the Internet has been incredible for content, and mostly terrible for content providers, whose pay has plummeted as they struggle to differentiate from high quality content and be heard through the noise of mediocre content. I expect this problem to get worse, not better, as AI tools improve at mimicking human writers.

With no further ado: this article will reflect on the most popular articles and most popular tweets of 2025, with some bonus content if you read to the end.

Most Popular Articles

The most popular article by far, fueled by a post to Hacker News, was Why We Need SIMD (The Real Reason), which recapitulated the history of SIMD instructions on x86 and spoke to the engineering tradeoffs that have prompted CPU vendors to pursue SIMD instuctions as a preferred mechanism to accelerate data parallel workloads.

The second most popular article was my reading list: Perennial Technical Reads, a walk down memory lane invoking Fred Brooks, Jon Bentley, and others. If I were writing it today, I might have remembered to include Joel Spolsky’s spicy blog along with the other Internet resources mentioned.

The third most popular article was a send-up of a l33t coding question, Third Largest Element, which implemented an AVX2-accelerated solution that no reasonable interviewer possibly could expect a candidate to write in real time.

Almost as popular: An article with the enticing title This ML Workload Runs 30x Faster w AVX512, an implementation of the float-to-NF4 conversion that uses permute instructions for register-to-register lookups. It also alludes to a cool optimized Binary Sort implementation that I first learned almost 40 years ago.

Understandably, none of my paid posts were as popular:

• Book Review: The NVIDIA Way, a review of Tae Kim’s bestselling hagiography of Jensen Huang that is a fun read but an incomplete oral history of NVIDIA,

• Implications of Google v. Oracle, an article on the most consequential Supreme Court decision in software engineering history,

• Standardized or Proprietary, a reflection on the tradeoffs between standardized APIs designed by committees such as the OpenGL ARB (Architectural Review Board) and proprietary APIs such as Microsoft’s Direct3D and NVIDIA’s CUDA, both of which I had a central hand in designing,

• AMD’s GPU Software: A Software Architect’s Take, a spicy summary of some (by no means all!) of the issues in AMD’s GPU software stack,

• Finding And Fixing Direct3D and A Missive From The RISC/CISC War, reflections on my time at Microsoft excerpted from an autobiography in progress.

Most of my articles are free, but I’ll keep leavening with paid content to keep things interesting for my most committed subscribers.

Most Popular Tweets

The runaway viral tweet of the year, viewed 1.8M times, was an anecdote about how consumption of soda dropped 90% when NVIDIA started charging $0.25/soda. NOTE: this was a comment on economics and demand elasticity, not NVIDIA’s confidential business practices c. 2002!

Another notable tweet, with >400k views, was a pedestrian observation about NVIDIA’s software stack, one I have made many times: that because CUDA’s driver API is portable across both operating systems and CPU architectures, it enables NVIDIA to meet developers on platforms they have chosen.

A tweet on game developers’ elite status on the software engineering community got quite a few views:

Some posts that didn’t get as much traction as I expected include Don’t Move The Data!, an updated version of a 2017 article on how data movement has become the limiting reagent of all compute. I may keep updating this one, because the goalposts keep moving as packaging and other technologies continue to develop at a breakneck pace.

Conclusion and 2026

I have been blocking accounts to curate my feed to be constructive and technical, and it has mostly worked. At its best, the Internet and platforms like Twitter serve to facilitate wholesome exchanges and opportunities for learning.

If any of the articles mentioned above sounded interesting, I’d encourage you to take a look through the archives. There are articles on C++ programming idioms, CUDA programming practices, API design, the evolving technological landscape, and more.

As a final note, I have all but decided to move the entire CUDA Handbook text, and related content like the 2013 article on Histograms, to the Web site. Everyone seems to pirate the book anyway! AI tools seem accomplished at performing the stultifying task of converting Word text to Markdown, so look for updates as that project moves forward.

A few weeks ago, the folks at Qt Group reached out to see if they could contract with me to write about the new CUDA support for Axivion, their code analyzer tool that is designed to complement (not replace!) seasoned developers’ adherence to coding standards such as MISRA, AUTOSAR, and NVIDIA’s own CUDA Security guidance.

Code analyzers have a long history in our industry. In the 1970s, when K&R’s The C Programming Language was new, a program called lint was developed by a computer scientist at Bell Labs named Stephen C. Johnson. lint itself is a historical curiosity, with functionality that has long since been rendered obsolete by language design (e.g., mismatched argument types) or subsumed into compilers (e.g., pedantic warnings about portability issues). But the idea persists that there’s a role to play for tools that complement compilers’ core task of translating source code into machine code and, in an homage to its origins, such tools are called linters.

My first experience with using a linter in production was at Microsoft in the 1990s, which acquired PREfast, a static code analyzer that visited much more rigorous scrutiny on the code than the compiler. Management was concerned about security vulnerabilities, especially ones caused by buffer overflow bugs, and PREfast was designed to detect such bugs through holistic analysis of the source code. A team was assigned to run the entire Windows code base through PREfast, and a bug was opened for every issue it found.

That exercise had a bigger influence on how I write code than any other single event in my career as a software developer—because some of those bugs were damn near impossible to fix without regression risk.

If a function neglected to check the return value from malloc(), then passed the resulting pointer into a call stack, where only some code paths dereferenced the bad pointer… PREfast would catch that bug, and outline the exact set of conditions needed for it to reproduce. But fixing the bug sometimes would require touching code in many places, often in ways that made the fix hard to verify by inspection.

The bugs opened by PREfast generally could be dropped into one of three categories:

• Most often, PREfast raised legitimate bugs that could be fixed with minimal disruption to the source code.

• Occasionally, we were able to prove that PREfast had raised a spurious concern.

• Every so often, we had an unsatisfying resolution: it seemed like there was a bug, but without a repro case, and if the affected code was distributed throughout source code, our attempts to ‘fix’ the ‘bug’ seemed just as likely to introduce regressions.

Microsoft trusted its engineers; if the person investigating a PREfast issue attested that it had been fixed (or that it was not a bug), the bug would be closed. For bugs that were deemed spurious, there was a mechanism to mark them so PREfast would not raise those issues again. If memory serves, marking a bug that way required sign-off from multiple people, akin to a code review.

The experience of triaging, fixing, confirming the fixes for these bugs, and figuring out how to improve test coverage of the code, had a profound effect on my approach to software development. The latter two categories, in particular, motivated me to think about how I could write code that 1) does not raise spurious concerns with the static code analyzer and, more importantly, 2) enables bug fixes to be more local, e.g. by handling resource allocation failures at the call site, and propagating errors from unified code paths.

As computers have become more pervasive, they increasingly have found their way into safety-critical applications. For software running on those computers, the stakes are higher and the software must be designed and built more carefully. We can’t go about building software for medical devices, airplanes, weapons systems, or automobiles in the same way that we build software for computer games or e-commerce Web sites. NASA has long adhered to coding standards and software development practices that bias strongly for robustness over speed of development. Prominent failures, such as the notorious Therac-25 radiation therapy machine1, have motivated development of coding standards that are more formal, and have more force in effect, than the advisory coding standards that most professional software engineering organizations have internally negotiated.

In the late 1990s, the automobile industry, recognizing the inevitability of computers continuing to find more applications throughout their products, convened the Motor Industry Software Reliability Association (MISRA) to formalize coding standards. Now in its third edition, MISRA has found its way into other application domains, such as the Joint Strike Fighter (JSF) C++ Coding Standards2. More recently, the AUTOSAR (AUTomotive Open System ARchitecture) was developed to standard software development for Electronic Control Units (ECUs), and the International Standards Organization (ISO) incorporated Product Development At The Software Level as Section 6 of the ISO 26262 standard for Functional Safety of Road Vehicles.

The simple presence of MISRA, or claimed adherence to its standards, won’t protect companies if employees fail to adhere to best practices. Around 15 years ago, as if to underscore why the automotive industry was leading the field of software engineering in developing such standards, Toyota Corporation was investigated for UA (unintended acceleration) events that had killed an estimated 89 people in May 2010. A related class action lawsuit was settled for $1.6B in December 2012, and in 2014, Toyota was further fined $1.2B for concealing safety defects. Expert witness Michael Barr spent more than 20 months reviewing Toyota’s source code and testified about its quality, even citing an internal Toyota document that characterized it as “spaghetti code.” Barr and a NASA team that was contracted to do an evaluation both checked the code against MISRA standards and found thousands of violations. For those interested, this article has a (lengthy) summary as well as links to the hundreds of pages of testimony by expert witnesses Koopman and Barr.

By now, it should be clear why there is an appetite for tools such as Axivion, which analyzes source code and flags possible violations of the various safety standards, citing them by chapter and verse. They are intended to complement, not replace, seasoned engineers’ judgment. Like the leaders at Microsoft who trusted their developers to triage and address the issues raised by PREfast, the architects of these coding standards recognized that it’s more realistic to be pragmatic than didactic. That said, if I were an engineering director at a company building software for safety-critical systems, I’d consider thoughtfully incorporating Axivion into the CI/CD workflow.

And as far as I know, Axivion is the first and only offering that performs this service, not only for C/C++ and C# code, but also for CUDA C++ code3. Only recently, NVIDIA published CUDA C++ Guidelines for Robust and Safety Critical Programming, and Qt Group has incorporated the guidance from that document into their tool. Such expanded support is needed as CUDA continues to find applications in robotics, autonomous vehicles, and other safety-critical domains.

As a first exercise, the Qt folks ran the source code for The CUDA Handbook through their tool, and sent a report summarizing its findings. The CUDA Handbook isn’t intended for safety-critical applications, per se, but it’s an open source code base we can usefully examine to get a sense of the error reporting and how a developer would triage and address the issues raised by the tool.

In our next article, we’ll put Axivion through its paces and take a look at its findings.

“Imagine if the CUDA SDK was $1k/year/seat,” the discussion began. Rhetorically, of course, since history has taught us that charging for platform support hinders adoption. @opdroid1234 wondered, “i never understood Intel charging for Vtune :(", the profiling tool that made it easy to profile applications to improve their performance on x86. As shown above, the discussion reminded me of a story that Jerry Pournelle, the legendary BYTE Magazine columnist1, had told at the height of the contest for mindshare between Window NT and OS/2.

It seems ridiculous today, but OS/2 once was considered as credible a successor to MS-DOS as Windows NT, which had originally been positioned as a microkernel operating system for which both Windows and OS/2 would be peer client operating systems. The competition referenced by Jerry Pournelle in the September 1993 issue of BYTE Magazine was the by-product of a so-called “divorce” between the two companies that started, according to this June 1992 New York Times article, in September 1990. Surprisingly, to me at least, the break occurred amidst an FTC investigation of collusion between the two companies that had started in the late 1980s2.

So by 1992, with Windows NT development going ahead full steam without any collaboration with IBM, the two companies were vying for market share and developer mindshare at conferences like COMDEX. Developers and consumers alike were being asked: operating system would provide the bigger return on investment?

And that is where distant past history intersects with our present discussion. I’ll let Pournelle take it from here (link to a PDF of the original pages of BYTE Magazine):

“Develop Your Device Drivers for OS/2!” proclaimed a sign at IBM’s booth. “IBM Device Driver Source Kit for OS/2 Now Available on CD!” A flier announced “Free. WIN-OS/2 source code for seamless VGA Display Device Driver when you order the DDK!” Microsoft has long used CD-ROMs to distribute their development kits, and will even get developers a discount on a CD-ROM drive.

Aha, I thought, now IBM has learned that trick. With any luck, we’ll soon see a flood of new device drivers for both OS/2 and Windows under OS/2, and maybe it won’t be long before I can hook up my Pioneer CD-ROM drive on an OS/2 system. By gum golly, it looks like IBM is doing something right.

Then I got closer, and to my horror, I saw the price. You can get the Device Driver Source Kit for only $4993. It wasn’t clear whether that was the regular price or a show special. I went up to the young woman who was passing out the fliers.

“I thought the big problem with OS/2 was the lack of device drivers,” I said. She gave me a smug look and handed me a flier. “Yeah,” I said, “but isn’t 500 bucks a bit steep? If I go over to Microsoft and whisper about writing device drivers for NT, they’ll stuff kit disks into my briefcase.” She drew herself up to her full height and said, “ Thank you very much for the information.” She was too polite to say, “We’re IBM. We don’t care. We don’t have to,” and maybe she didn’t think it, but it sure looked like she did.

I went around to some of the sharp OS/2 programmer troops and told that story. They were disturbed. They hadn’t known the price was that high. The kit, it seems, is sold by a different part of IBM. I spent an hour looking for any policy-level IBM official, or a PR officer, but I never found one to discuss this with. When I went back to the OS/2 station, the young woman, still smug, said, “ Back again?” It was pretty clear she had no use for my observations, so I didn’t offer any.

I didn’t know how much Microsoft charged for their kits, so I went over to find out. I’d no more than set foot on their carpet when one of the PR people recognized me4 and asked if she could help. “If I wanted to write NT device drivers, what should I do?” I asked. “ You’ re in the wrong booth,” she said. “Let’s go over to the NT booth. I don’t know anything about NT, but we’ll find someone.”

We walked over-no question that Microsoft’s PR people knew where each and every Microsoft booth and display was and in 7 minutes-I timed it-I was talking to one of the NT product managers. “Well, first you need the developer’s kit, with the compilers and source code and stuff,” he said. “That’s on a CD-ROM for $69. Then you get the DDK. It’s another CD-ROM, same price.”

“Little steep, isn’t it?” I said. “A little,” he said, “but there’s a lot there, all the sources we could find, not just ones we wrote but any we could talk people out of. Want a copy?” I think I just may have located the reason why there’s a shortage of OS/2 device drivers, and why there probably won’t be a shortage of NT drivers.

OS/2 was not a market success, and is not a viable platform today, in part because it did not deliver a high enough return on investment. Back then, the return came in the forms of application and, as Pournelle underscores above, hardware support; the investment required modest purchases of machines to develop the software, attending conferences, and learning how to use the tools and APIs. The smaller that denominator (that is, the lower the barrier to entry for hardware and software developers), the larger the return on investment. OS/2 made its mark on history, and the competitive pressures it exerted did influence operating system technology, but history will record it as a platform that failed in the marketplace.

There are notable examples of successful platforms with a high barrier to entry. I plan to write about those sometime soon.

I thought of the clown car full of limit order book implementations I had lying around; some AVX2, some AVX512, some using different memory layouts, some using different metadata to accelerate key cases. But the various implementations were resident in separate Git branches. It would have been easy to replace the various methods in the order_book class with virtual functions, and indirect through them to test the various implementations; but that would have incurred a Draconian performance hit, and the point of the exercise had been to explore the design space and find the fastest combination. Giving up performance by going across an unnecessary procedure call1 was unacceptable.

So I took an action to explore CRTP as a possible solution to this problem: “flatten” the source code so scalar, AVX2, and AVX-512 code can coexist and be compared side-to-side in the code base, without compromising performance. Tentatively, it’s working out really well, although I can’t yet share the final AVX-512 implementation. I thought now would be a good time to go over CRTP and why it was such a good fit for this exercise of exploring different CLOB implementations.

Whither CRTP?

You may be wondering: what the heck is CRTP? In this case, the Wikipedia article has an excellent article on the topic, including the history that the term was invented by James Coplien in his book, Advanced C++ Programming Styles and Idioms (Copyright 1992). Sean McBride’s recent (June 2025) review of the book characterized Coplien’s book as “the most difficult C++ book I’ve read,” and having read and struggled with it 30 years ago, I share that sentiment. CRTP is such a counterintuitive idiom that the Wikipedia article straightfacedly recounts how Microsoft developer Christian Beaumont “initially thought it could not compile in the Microsoft compiler available at the time”:

The Microsoft Implementation of CRTP in Active Template Library (ATL) was independently discovered, also in 1995, by Jan Falkin, who accidentally derived a base class from a derived class. Christian Beaumont first saw Falkin’s code and initially thought it could not compile in the Microsoft compiler available at the time. Following the revelation that it did work, Beaumont based the entire ATL and Windows Template Library (WTL) design on this mistake.

Formally known as “F-bounded quantification,” CRTP entails having a class derive from a class template instantiation using itself as template argument. The base class can then define static functions that invoke member functions of the derived class; when the compiler is given all these class definitions, plus code that uses the static functions, it has all the information needed to compile the code knowing the type of the derived class. Code that creates and uses instances of the derived classes then transparently benefit from this form of static polymorphism, without even knowing about the enabling linguistic legerdemain.

Backgrounder: Runtime Polymorphism

It may be helpful to give a concrete example, starting with a more straightforward implementation of polymorphism. (You might say as the language designers intended.)

In C++, an abstract base class defines pure virtual functions that specify operations that may be performed on the object, but defer implementation onto their derived classes; in fact, unlike (impure?) virtual functions, derived classes must provide implementations of their parent classes’ pure virtual functions.

The Ray Tracing In One Weekend series (RTOW), whose sample source code is available on GitHub, has a good example of this traditional type of polymorphism.

If you don’t know what ray tracing is, but like to program and enjoy pretty pictures, it may be worth checking out RTOW. Ray tracing is a brute force method of rendering that creates images by modeling the behavior of light rays in the environment. A key abstraction at the center of ray tracing is the ray intersection calculation that determines whether a ray of light intersects with an object and, if the intersection occurs, computes the location of that intersection. To keep the separation of concerns clean between the casting of rays and intersecting those rays with objects, the RTOW code defines an abstract base class hittable, with derived class sphere:2

class hittable {
public:
virtual ~hittable() = default;
virtual bool hit(const ray& r) const = 0;
};

class sphere : public hittable {
sphere(const point3& center, double radius, shared_ptr<material> mat)
: center(center), radius(std::fmax(0,radius)), mat(mat) {}
bool hit(const ray& r) const override {
…
}
};

class triangle : public hittable {
triangle(const point3 vertices[3]): …
bool hit(const ray& r, interval ray_t, hit_record& rec) const override {

… // ray-triangle intersection routine
}
};

When a programmer using this class hierarchy creates an instance of sphere, it also is an instance of hittable; and the code can operate on hittable objects without knowing whether they are spheres or triangles. This function, for example, returns the objects that a ray hit:

std::vector<hittable *>
return_hits( const ray& r, const std::vector<hittable *> objects ) {
    std::vector<hittable *> ret;
    for ( auto obj : objects ) {
        if ( obj->hit( ray ) ) {
            ret.emplace_back( obj );
        }
    }
}

This polymorphism is enabled by a virtual table, also known as a vtable or vtbl: an array of pointers-to-function, one per class, that the compiler creates behind the scenes. When you call obj->hit(), the compiler uses the pointer to obj to find the vtbl and call the corresponding function.3

This type of runtime polymorphism – where clients interact with object strictly through interfaces, with no knowledge of their internal representation4 - is the bedrock underpinning many important frameworks and technologies, from native language support (e.g., Java interfaces) to technologies like Microsoft’s Common Object Model (COM), which enables language-independent API deployment, to the Service Oriented Architecture (SOA) that Amazon embraced to improve on the inefficiencies of what had been a huge, monolithic code base5.

But the pattern depicted above – where the sphere and triangle classes inherit from, so have an IS-A relationship with hittable (i.e., sphere * and triangle * both are hittable *) – is hard to achieve with CRTP. Chris gave a concise example showing how, which involves using the C++17 type trait std::is_base_of:6

// Boring, Java-esque OOP stuff         // CRTP equivalent
                                        template<typename T>
class Shape {};                         class Shape : public T {};

class Polygon : public Shape {};        template<typename T>
                                        class Polygon : public Shape<T> {};

class Circle : public Shape {};         template<typename T>
// These “are” polygons, of course.     class Circle : public Shape<T> {};

class Square : public Polygon {};       // These “are” polygons, of course.
class Pentagon : public Polygon {};     template<typename T>
                                        class Square : public Polygon<T> {};
                                        class Pentagon : public Polygon<T> {};

// The generic one.                     // The generic one.
void doSomething(Shape& foo);           template<typename T>
                                        void doSomething(T& foo);

// The special overload for polygons.   // The special overload for polygons.
void doSomething(Polygon& foo);         template<typename T>
                                        requires(std::is_base_of_v<Polygon<T>, T>)
                                        void doSomething(T& foo);

In Chris’s words: “The key point is: notice how we can overload doSomething for the category of classes “Polygon”, even though we used CRTP!”

For our purposes, CRTP is preferable because although we want to explore the design space of different instruction sets, memory layouts, and implementation strategies, we do not want to give up any performance by calling across virtual function boundaries. We could mark the derived classes as final, which empowers the compiler to bypass the vbtl call when it can prove that an object is of the expected type; but CRTP is a more certain path.

So how does CRTP work? Our usage, for static polymorphism, will echo this example from the Wikipedia article:

template <typename T>
struct Base {
    void call() {
        // ...
        static_cast<T*>(this)->implementation();
        // ...
    }

    static void staticFunc() {
        // ...
        T::staticSubFunc();
        // ...
    }
};

Consider the order_book class in our sample code. The class, as published by Charles Cooper, already was arranged in a way that was amenable to refactoring for CRTP, with static functions such as add_order calling member functions such as ADD_ORDER:

static void add_order(order_id_t const oid, book_id_t const book_idx,
sprice_t const price, qty_t const qty)
{
#if TRACE
printf(”ADD %lu, %u, %d, %u”, oid, book_idx, price, qty);
#endif // TRACE
    ...
}

void ADD_ORDER(order_t *order, sprice_t const price, qty_t const qty)
{
    ...
}

add_order looks up the correct book (each ‘symbol’ in the exchange - MSFT for Microsoft, for example - has its own instance of order_book), and dispatches the order by calling that book’s ADD_ORDER method.

Let’s say we are exploring the tradeoffs between AOS (array of structures) and SOA (structure of arrays) memory layouts of the limit order book data. Using CRTP, we can templatize the order_book class, then create derived classes order_book_scalar (the original scalar implementation, which uses the AOS layout and will serve as our canonical test reference) and order_book_soa, which rearranges the price levels to be a tuple of std::vector instead of a std::vector of tuples.

The reason we are experimenting with SOA is because it’s needed for optimal SIMD performance. I’ll explain in a later post, when we look at the AVX implementations of order_book.

For the CRTP refactoring, we also need to move some of the order_book members to the derived class – namely the m_bids and m_asks members. These are the per-book snapshots of the aggregate orders at each price level.

static constexpr size_t MAX_BOOKS = 1 << 14;
static constexpr size_t NUM_LEVELS = 1 << 20;
static order_book *s_books; // can we allocate this on the stack?
static oidmap<order_t> oid_map;
using level_vector = pool<level, level_id_t, NUM_LEVELS>;
using sorted_levels_t = std::vector<price_level>;

// A global allocator for all the price levels allocated by all the books.
static level_vector s_levels;
sorted_levels_t m_bids;
sorted_levels_t m_offers;
using level_ptr_t = level_vector::__ptr;

The CRTP changes entail templatizing order_book:

template<typename Derived, LAYOUT layout, TARGET_ISA isa, bool TRACE = false>
class order_book
{
    ...
};

and creating derived class order_book_scalar:

class order_book_scalar : public order_book<order_book_scalar, LAYOUT::ARRAY_OF_STRUCTS, TARGET_ISA::GENERIC_C>
{
};

The “curiously recurring” aspect of this inheritance structure is that order_book_scalar inherits from itself.

Note that we took this opportunity to templatize the TRACE compile-time option (it had been a preprocessor macro) and the memory layout and ISA (instruction set architecture), to future-proof this class hierarchy to experimenting with SOA memory layouts and AVX2 and AVX-512 implementations of the limit order book.

We moved the m_bids/m_asks members and the ADD_ORDER method from order_book to order_book_scalar, and modified the static function order_book::add_order to specifically reference the Derived class:

s_books[size_t(book_idx)].ADD_ORDER(order, price, qty);
static_cast<Derived *>(&s_books[size_t(order->book_idx)])->ADD_ORDER(order, price, qty);

The client code also needs to be refactored, and here, the refactoring of main.cpp had the beneficial side effect of moving almost all of main() into a templatized function to test and measure the performance of an order_book implementation. We introduced a new function template timeBacktest():

template<typename T>
double
timeBacktest( const std::string filename )
{
   ...
}

that handles the ADD_ORDER message from the exchange by specifically referencing the template parameter T:

case (itch_t::ADD_ORDER): {
    ...
    T::add_order(order_id_t(pkt.oid), book_id_t(pkt.stock_locate), ... );
    break;
}

main() then can invoke timeBacktest() with the order_book class instantiation to test:

timeBacktest<order_book_scalar>( filename );

Later refactorings introduce an SOA implementation of order_book, a cross-check function to make sure our speculative CLOB implementations actually work, and command-line options to invoke the full Cartesian product of possible combinations of template parameters.

Before, I had separate GitHub branches with AVX2 and AVX-512 implementations, with various heuristics and metadata controlled by preprocessor constructs. I had to check out a specific branch and configure the preprocessor macros to test a given combination. Using CRTP, we’ll be able to flatten the code base so we can do side-by-side comparisons of the various implementations’ source code – and their performance.

Thanks again to Chris Kitching of Spectral Compute for putting me onto the possibilities of this refactor!

Some perceived barriers, like the 1-micron barrier (or in today’s parlance, the 1,000-nm barrier) turned out essentially to be psychological, like the 4-minute mile. The perception that Moore’s Law might come to an unwelcome end has motivated investments in alternative technologies, like Josephson junctions or gallium arsenide as a substrate, for decades.

For the last few years, as process improvements continued to roll out (albeit more unevenly than they did in the halcyon 1990s), an alarming trend started to be observed: logic was benefiting markedly more from improved scaling than SRAM. In December 2022, Tom’s Hardware published “TSMC’s 3nm Node: No SRAM Scaling Implies More Expensive CPUs and GPUs,” and in February 2024, semiengineering.com published: “SRAM Scaling Issues, And What Comes Next.”

Why should we care? What is SRAM to you and me?

Well, for more than 30 years, the majority of the transistor budget allocated to CPUs has been SRAM in the form of caches. Modern CPUs have dozens of cores and each core has a very fast L1 (usually 64K each for code and data) and a moderately fast L2; the L3, or “LLC” (last level cache), or “uncore,” consists of an L3 cache that arbitrates external memory traffic, be it across the cache coherency interconnect or the integrated memory controllers on the CPU die. In short, the caches are comprised of SRAM and are designed to make memory seem faster (lower latency). SRAM has been such a dominant percentage of CPUs’ transistor budgets for the last 30 years that even significant instruction set improvements such as MMX (52 new instructions, c. 1996) and x64 (c. 2002) cost less than 10% total die area.

We’ve had scares like this before. A little over 20 years ago, a big inflection point in the history of Moore’s Law occurred: the end of Dennard scaling, where improvements in transistor density also delivered improvements in clock rates. The 1990s had been a decade of free beer, with Intel and AMD leading the charge from 25 MHz 80486 chips to 1,000-MHz chips by the end of the decade. In 2002, then-Intel VP Pat Gelsinger observed that it was becoming infeasible to parlay improvements in density into higher clock rates, and predicted multicore CPUs as a likely answer to the questions posed by this technical challenge. Intel introduced its first multicore in the Pentium Duo (c. 2006), and today, modern CPUs have many dozens of cores and peak performance is impossible to attain without multithreading.

But with benefit of hindsight, we now see that the primary beneficiary of the end of Dennard scaling was not any CPU vendor, but NVIDIA, with its GPUs and the CUDA technology underpinning their AI hardware business. NVIDIA GPUs, the story went, were designed for throughput, not latency; so they were able to target lower clock rates and, instead of caches designed to reduce the effective latency of memory, the SRAM in GPUs is allocated to:

• The humongous register file (largest memory on the chip), SRAM that helps cover memory latencies and instruction latencies via thread level parallelism (TLP);

• Shared memory, a software-managed cache, occupies a significant portion of each Streaming Multiprocessor (SM), of which there are 140 on a modern GPU;

• The L1 caches for each SM, which typically are aliased on to the shared memory (and the proportion of L1/shared memory is under software control); and

• the L2 cache, dozens of megabytes, is yet more SRAM.

The design of each of these hardware units reflects the different types of memory traffic serviced by the SRAM.

But the point is, the transistor budgets of both CPUs and GPUs is dominated by SRAM.

And for a scary few years, SRAM did not seem to benefiting from scaling to the same degree as logic. Quoting from the above-linked semiengineering.com article:

“In traditional scaling of planar devices, gate length and gate oxide thickness were scaled down together to improve performance and control of the short-channel effect. A thinner oxide enabled the performance gain in lower VDD level, which is advantageous for SRAM in reducing both leakage and dynamic power,” said Jongsin Yun, memory technologist at Siemens EDA. “However, in recent technology node migrations, we’ve barely seen further scaling oxide or VDD levels. Moreover, the geometric shrinkage of transistors results in thinner metal interconnects, leading to increased parasitic resistance and, consequently, more power loss, and RC delay. As AI design increasingly demands more internal memory access, it has become a significant challenge for SRAM to further scale its power and performance benefits in technology node migration.”

What does this mean?

It’s as big a question for hardware and software designers as the end of Dennard scaling, or the end of Moore’s Law itself. One of the more obvious consequences is that, coupled with advances in packaging technology, chiplets that are especially RAM-intensive (e.g., I/O dies with cache) can target less-dense, more-economical fabrication processes. As the article mentions, AMD’s 3D V-Cache technology “allows additional SRAM cache memory to be stacked on top of processors, increasing the amount of cache available to the processor cores.”

A less-obvious innovation would be to trade compute for effective bandwidth; can tightly-integrated compression technology reduce the SRAM footprint needed to service a given memory traffic load?

Of course, the staggering investments in continued advancement of semiconductor fabrication technology, which have fueled that ongoing innovation, may forestall the apocalyptic scenario.

TSMC is hiring more memory designers to improve SRAM density, but whether they can eke more out of SRAM remains to be seen. “Sometimes you can make things better by applying more people, but only up to a point,” said Tate. “Over time, customers will need to think about architectures that don’t use SRAM as intensively as they do now.”

And tentatively, TSMC’s investments may have paid off: the February 15, 2024 article is followed by the November 2024 “SRAM Scaling Isn’t Dead After All.” All it will cost you is the very latest and most expensive TSMC process!

Moore’s Law is bound to run out sometime. To be honest, it has outlasted everyone’s reasonable expectation! We keep getting previews of what it might be like, and having to design hardware (and, sometimes, the software that runs on it) accordingly. Without impending physical laws to constrain and inform our designs, what would we do to keep our industry fun and exciting?

At the same time, some content distinguishes itself because I find myself referencing it over, and over and over across the years. The small list I’ve compiled reflects how I have spent most of my career twiddling bits in user mode.

Without further ado, and in no particular order, here’s a list of resources that have been on my shelves, serving me well, in some cases for decades.

Textbooks

Introduction to Algorithms, 4th ed. by Cormen, Leiserson, Rivest and Stein.

Now in its fourth edition, this book has been the best text on the subject since it came out in 1990, just in time for the undergraduate course in algorithms I took as a sophomore. Since the book was first published, algorithm design necessarily has evolved to center parallel architectures to a greater degree; and these days, memory operations matter a lot more than the number of instructions executed. But the math and the tools described to prove algorithms correct, and analyze their performance, are timeless. The chapters naturally progress in complexity, from simple sorting algorithms and data structures such as binary trees, to chapters on Dynamic Programming, Greedy Algorithms, and Amortized Analysis, to advanced topics where they really can only give an overview and point the reader at resources.

The only nit I can pick with this book is that early editions had a chapter on parallel algorithms that covered PRAM, a good fit for massively parallel architectures like GPUs; but it has since been replaced by a chapter on threading, perhaps due to co-author Charles Leiserson’s affinity for the Cilk programming language. But parallel programming is such a broad topic that it could only get an introductory chapter with references, anyway.

This book is a gem.

Programming Pearls, 2nd ed. by John Bentley

Back in the 1980s, CMU professor Jon Bentley wrote a monthly column for Communications of the ACM called “Programming Pearls,” and this book is a compendium of the best of those columns. As a freshman at Duke University (c. 1988), this book was on the must-buy list for the mandatory intro-to-programming course, even though it did not figure into the curriculum as far as I could tell. I was immediately hooked: he talks about everything from DSLs (though he calls them “little languages”) to binary heaps (though he just calls them “heaps”). Bentley’s evident love for the craft of programming, coupled with clear explanations in an approachable style, still serve as an inspiration.

Hacker’s Delight, 2nd ed. by Henry S. Warren

With a title like that, who can resist? This book is the densest collection of bit twiddling recipes you can find anywhere. Whether you need a branchless conditional negate, need to figure out how to enlist floating point hardware to find the most significant set bit, or want to understand Gray codes or why integer division is so damn difficult, this book’s got your back! It’s well-organized, but if you are anything like me, you may lose yourself just perusing it, then going back to find the little kernels of wisdom when needed.

Some of the tricks described in this book have been subsumed into hardware (his panoply of population count algorithms has been obsoleted on some platforms by native machine instructions), and compilers have improved at making it unnecessary to spell out some of the optimizations (see ‘branchless conditional negate’ above) - largely because compiler writers have spent the last few decades poring over this book and figuring out ways to translate code to use the idioms described here.

By the way, if you enjoy Hacker’s Delight, you need to know about the Stanford bit twiddling hacks maintained by Sean Eron Anderson.

The Mythical Man-Month, by Fred Brooks

This oft-cited text on software engineering is worth a re-read every couple of years. If your first thought on hearing the title is, Oh that’s the one where he discovered that doubling the size of a software engineering team causes schedules to slip, because of communications overhead… drop everything, buy a copy if you don’t already have one, and reread this book. Every time I read it, I’m struck anew by some durable truth Brooks articulates, and I think, should we be struck at how much software engineering has changed in the last fifty years, or how little? Reading this book will incline you to the latter. No one was talking about “scrums” or “agile programming” when Brooks wrote this book, but many key insights he shares still hold true today. His observation that some software engineers are substantially more productive (one or two orders of magnitude) than average (we’ve all seen this) is immediately followed by the observation that there just aren’t enough such engineers to do all the required work. In his discussion of roles, from QA to software architects to language lawyers (and he uses that term), is as relevant today as it was back then.

Come to think of it, I have this book on my Kindle and it has been a while since I reread it2.

Performance Aides

A Primer on Memory Consistency and Cache Coherence, by Vijay Nagarajan, Daniel J. Sorin, Mark D. Hill, and David A. Wood

When the first edition of this novella-sized treatise was published, John Montrym handed it to me and said: “Read. This.” Montrym was a legendary GPU architect before he joined NVIDIA in the late 1990s, and he’s as warm and generous with his time as he is brilliant; so if he tells you to read something, you drop everything and comply.

This masterwork is required reading for anyone interested in parallel computing architectures, whether they be multicore CPUs, multi-socket servers, GPUs, or modern servers that feature both. It starts by defining Consistency and Coherence, their motivations, scalability problems that have arisen over the years, and solutions to those scalability problems. It includes a detailed description of TSO/x86, Intel’s “total store ordering” memory consistency model that was esoteric enough that it was nothing more than poorly-documented tribal knowledge before the Linux kernel team considered an optimization of their spin lock. According to this account (which has reference links):

Various experts went back and forth over whether the final MOV that sets a lock variable to 1 needed to be prefixed by LOCK or not. The discussion ended when Linus Torvalds said “I know that it is needed”. Only to see an Intel architect finally intervene and say “you know, really, it isn’t needed”. This was followed by a series of releases of Intel manuals documenting the x86 memory model, with increasing precision in each release. Intel also actually changed the published rules along the road, withdrawing some optimizations as they realized that they would break existing software.

The Primer is now in its second edition, which added a chapter on “accelerators” (mostly GPUs). It includes ample references for you to chase down for further study.

Intel Intrinsics Guide, Intel Corporation.

This resource is for every developer interested in accessing esoteric x86 instructions - especially SIMD instructions - unless you are a masochist, or have a solid engineering reason to bypass the compiler’s ability to use intrinsics to automate register allocation and instruction scheduling.

These days, writing SIMD code for x86 chips is akin to writing a legal brief: extensive research is needed, in part because there are annoying orthogonality misses in the ways the instructions were implemented. You have to confirm that the instruction set actually supports a feature (for example: until AVX512, there was no SIMD double-to-int64 conversion instruction. You could only do one of those at a time!), and that it’s supported across the full SIMD width of the instruction set.

The Intrinsics Guide lets you filter by ISA and search for either instruction mnemonics or intrinsic names, and once you find them, it has details on exactly how the instruction behaves, which Intel chips implemented it, which header file to include, and performance guidance (latency in clocks and throughput in CPI (clocks per instruction).

Optimization Manuals, Agner Fog.

If his Web site is any indication, Agner Fog is an accomplished Renaissance man; but for me, his manuals on x86 microarchitecture and x86 instruction tables are worth their weight in platinum.

To convey the scope of achievement encapsulated in these works, consider the microarchitecture manual is copyrighted 1996-2025. (It was last updated in September!) Execution pipelines, micro-op caches, instruction decoding, branch prediction, cache architectures, partial register access, store forwarding… these are just a few of the details of CPU implementation you can glean from these pages.

Historical References

The Story of ISPC, by Matt Pharr.

ISPC (“Implicit SPMD Program Compiler”) is a toolchain designed to simplify SIMD instruction set programming: in contrast with vectorizing compilers, it strikes a more natural balance between enabling programmers to express their intent, and the compiler’s ability to generate the corresponding code. It hasn’t achieved as much adoption as I expected, though it has notched some significant wins of mindshare: DreamWorks has open-sourced their MoonRay renderer that uses ISPC.

Computer graphics nerds know Matt Pharr for his masterful book Physically Based Rendering, now in its fourth edition; but in ISPC, he also created one of the only viable ways to program SIMD instruction sets with human-readable syntax and without intrinsics3.

To me, this blog series is an important historical reference on the early GPU computing landscape, when CUDA was new. Starting with the early history of Larrabee (Intel’s manycore entree into the parallel computing competition of the late 2000s), Pharr details how his experiments with vectorizing compiler technology led to what we now know as ISPC, and how ISPC came to be open sourced by Intel. There are some interesting stories in there.

Oral history of Intel, by Bob Colwell

Bob Colwell was a CPU architect at Intel, starting in 1990, and worked on the Pentium Pro, the first CPU to implement a 4/1/1 decoder for the x86 instruction set and one that substantially improved on the Pentium’s limited superscalar features. He was the chief architect for the Pentium 4 and reportedly included 64-bit features in that chip which then were disabled.

This oral history is the size of a medium-sized book. Where the discussion touches on Itanium is particularly interesting to me, but there’s much more.

No matter your opinion of Intel, its place in history is secure, and Colwell both had an outsized hand in writing that history and also bore firsthand witness to many events that are recounted here.

A “limit order book” is a list of orders to buy and sell “symbols,” the generic name for stocks, bonds, options, futures, or whatever other commodity is being traded on the exchange (there is a glossary at the bottom of this article). Market participants submit orders with the price and quantity of the symbol that they wish to buy or sell, and the exchange has rules to match these buyers and sellers and execute the requested trades. As one might expect, the exchange’s specialized technology to perform this task is called the “matching engine.”

The term “limit” in “limit order book” refers to the price that a trader specified along with their order. A buy order (“bid”) at a given price means the trader does not wish the order to be filled above that price; for sell orders (“asks”), the seller does not wish to sell below a given price. Traders often submit bids that are a bit lower, or asks that are a bit higher, than the prices currently being traded on the exchange; a “market order” is a request that the exchange simply fill the order at the best-available price, but often that is a construct of the retail trading market, not a primitive operation offered by the exchange. If an order cannot be immediately filled, it stays in the book until filled or cancelled.

The current price of a commodity is difficult to define: one only knows the price at which the most recent trade was fulfilled, but by definition, the price may be moving. The lowest ask price is always higher than the highest bid price, inviting the definition of a market’s “spread”: the difference between the two. As a rule, the smaller the spread, the more liquidity is in the market. (Also as a rule, HFT firms’ profit is highly correlated with the size of the spread; the bigger the spread, the more opportunity there is for them to submit profitable trades.) In any case, whether the prices you see scrolling across the CNBC chyron are “midpoints” (the average between the lowest ask and the highest bid) or the last price filled, obviously it’s subject to change, possibly quickly.

Computers that are plugged into the exchange get updates with tremendous frequency – depending on the exchange, up to millions of times per second: mostly new orders (whether they be to buy or sell a symbol), cancellations of existing orders, and “fills,” notices that the exchange’s matching engine has filled corresponding bid and ask orders that had been on the books. Since the CLOB is huge in comparison to an order, most exchanges require market participants to maintain their own accurate mirrors of the exchange’s CLOB as updates pour into their trading systems.

Although details vary by exchange, updates typically include ADD, DELETE, and EXECUTE orders, with each order containing only the security, price, quantity, and whether the order is a bid or ask at the given price. As a consequence, efficiently updating the levels corresponding to the security in question is the stock in trade (dyswidt) of every high frequency market participant, preferably with the levels sorted by price and canonicalized to a form more amenable to processing by the strategy code.

Often, HFT firms maintain an aggregate book that distills all the pending orders into price levels, with the total quantities of bid and ask orders resting at each level. The “top” of the book, with the orders closest to the bid-ask spread, is the main focus of many trading algorithms. Some are as simple as observing that the book is “weighted” too heavily in one direction or the other: for example, if there are more asks than bids, to enough of a degree, then the commodity is likely to decline.

The topic of this blog series will be to maintain an aggregate book, minimizing latency of updates to maximize a prospective strategy’s opportunity to process the book and make trading decisions.

Having worked on production HFT code, I took an interest in fast CLOB updates, with an especial focus on SIMD-friendly algorithms to do the updates quickly. CLOB code is fun to optimize and also makes for a good illustration of the transformations that need to be applied to enable SIMD processing to work best.

To start, I looked around the Internet for an open source implementation, hoping to find one specialized to the ITCH feed used by NASDAQ to trade equities. The high volume and low latency requirements to successfully trade on NASDAQ are legion, ITCH is well-documented, and you can reach out to NASDAQ to obtain sample ITCH files to use to simulate processing. This implementation is a good example of a ‘textbook’ implementation, using data structures like hashmaps for order lookup and a doubly-linked list for easy removal of the order upon fulfillment or cancellation. The Order structure looks like this:

struct Level {
    uint32_t price;
    uint32_t limitVolume;
    // either both are nullptr or both are populated
    // i.e. if orders in Level == 1, both pointers are ==
    Order* first;
    Order* last;
    Level(uint32_t _price);
};

To the author’s credit, they use a pool for allocation, not the default C++ heap, but my intuition leads me away from using this type of data structure for a CLOB – on 64-bit systems, the pointers alone are 8B each. Since data movement is the limiting reagent of all compute, a smaller memory footprint often leads to faster code.

An implementation closer to my starting point was built by a Charles Cooper, whose implementation uses std::vector to hold the aggregate limit order book. Cooper’s implementation is both older (work started c. 2014) and claims much lower per-trade latency than the previously-linked repository (61ns v. 92ns). Quoting from the readme:

This is a very fast implementation of the ITCH order book, clocking in at around 61ns / tick (or 16 million messages / second, tested on a 2012 i7-3820), offering fast updates and O(1) access to any price level (to get the price is a single dereference, the aggregate quantity is another dereference). It only calculates the aggregate quantities at each price and does not track the queue depth for each order.

The order structure in Cooper’s repository looks like this:

enum class qty_t : uint32_t {};
enum class book_id_t : uint16_t {};
enum class level_id_t : uint32_t {};
typedef struct order {
  qty_t m_qty;
  level_id_t level_idx;
  book_id_t book_idx;
} order_t;

The two sides then are instantiated in the flagship order_book structure as follows:

class price_level
{
 public:
  price_level() {}
  price_level(sprice_t __price, level_id_t __ptr)
      : m_price(__price), m_ptr(__ptr)
  {
  }
  sprice_t m_price;
  level_id_t m_ptr;
};

using sorted_levels_t = std::vector<price_level>;
sorted_levels_t m_bids;
sorted_levels_t m_offers;

Adding a new order consists of doing an Insertion Sort into one or the other of m_bids / m_asks, depending on whether the order is to buy or sell. The ADD_ORDER method in order_book:

  void ADD_ORDER(order_t *order, sprice_t const price, qty_t const qty)
  {
    sorted_levels_t *sorted_levels = is_bid(price) ? &m_bids : &m_offers;
    // search descending for the price
    auto insertion_point = sorted_levels->end();
    bool found = false;
    while (insertion_point-- != sorted_levels->begin()) {
      price_level &curprice = *insertion_point;
      if (curprice.m_price == price) {
        order->level_idx = curprice.m_ptr;
        found = true;
        break;
      } else if (price > curprice.m_price) {
        // insertion pt will be -1 if price < all prices
        break;
      }
    }
    if (!found) {
      order->level_idx = s_levels.alloc();
      s_levels[order->level_idx].m_qty = qty_t(0);
      s_levels[order->level_idx].m_price = price;
      price_level const px(price, order->level_idx);
      ++insertion_point;
      sorted_levels->insert(insertion_point, px);
    }
    s_levels[order->level_idx].m_qty = s_levels[order->level_idx].m_qty + qty;
  }

starts by assigning sorted_levels to be a pointer to the sorted vector for the side:

sorted_levels_t *sorted_levels = is_bid(price) ? &m_bids : &m_offers;

The inline function is_bid() relies on the sign of the price:

bool constexpr is_bid(sprice_t const x) { return int32_t(x) >= 0; }

The loop scans the sorted vector of price_level structures, looking for one with the same price as the incoming order; if it finds one, it terminates with found==true. Otherwise, a new index is allocated for the price level and inserted into the vector.

The DELETE method reduces the aggregate quantity for the given price level and, if it is now zero, removes the corresponding price_level structure from the side (again using a variable sorted_levels for the std::vector of bids or asks.

  // shared between delete and execute
  void DELETE_ORDER(order_t *order)
  {
    assert(MKPRIMITIVE(s_levels[order->level_idx].m_qty) >=
           MKPRIMITIVE(order->m_qty));
    auto tmp = MKPRIMITIVE(s_levels[order->level_idx].m_qty);
    tmp -= MKPRIMITIVE(order->m_qty);
    s_levels[order->level_idx].m_qty = qty_t(tmp);
    if (qty_t(0) == s_levels[order->level_idx].m_qty) {
      // DELETE_SORTED([order->level_idx].price);
      sprice_t price = s_levels[order->level_idx].m_price;
      sorted_levels_t *sorted_levels = is_bid(price) ? &m_bids : &m_offers;
      auto it = sorted_levels->end();
      while (it-- != sorted_levels->begin()) {
        if (it->m_price == price) {
          sorted_levels->erase(it);
          break;
        }
      }
      s_levels.free(order->level_idx);
    }
  }

Let’s be clear. This code runs pretty fast on modern CPUs! It uses arrays, albeit “array of structures” as opposed to the “structure of arrays” memory layout favored for processing by SIMD instructions; but modern CPUs have hardware prefetchers that make array processing very fast.

Before we take a serious pass at optimizing this code, we’ll want to refactor it a bit, partly for housekeeping to improve its processing of ITCH files (the multi-gigabyte files from Nasdaq that summarize a day’s trading), partly to improve the emulated I/O performance, and partly to lay the groundwork for SIMD optimization. An SOA layout for the sorted vectors is de rigueur!

Roadmap

I’ll be honest - this blog series is a writeup on coding work I’ve already done. I’m waist-deep in past commits and work branches, trying to hammer the code into a form with pedagogical value.

So in rough order, here are some of the changes to the repository that I am planning:

• A refactor of the buf_t class in bufferedreader.h, to use mapped file I/O instead of the POSIX open/read/close API set; in my opinion, mapped file I/O delivers a much closer approximation to getting orders off the wire;

• An update to the ITCH processing code to read the initial symbols, so we can decode which stock symbol (e.g., MSFT) corresponds to the 16-bit ‘locate’ field in the order;

• Duplication of the existing m_bids / m_offers vectors with SOA layout, with optional cross-checking.

• I’ll likely break out the bigger ADD_ORDER / DELETE_ORDER functions into a separate implementation file instead of leaving them in headers.

• One refactor that I have not completed is to flatten the source code for various SOA layouts and instruction sets that currently reside in branches. I want to flatten the code and make all the variants visible simultaneously, and I have an implementation strategy in mind. We’ll cross that bridge when we get to it.

Once these changes are in place, we’ll be able to pursue a proper SIMD optimized version of the limit order book. And as long as the cross-checking is indeed optional, we’ll be able to see how much faster it runs! (Also on my to-do list: try the optimized code on a recent AVX-512 implementation, instead of my slow AMD machine.)

For now, here is a link to my fork of Cooper’s itch-order-book repository.

Glossary

ask – order to sell. Typically accompanied by a quantity, and the lowest price the buyer would accept.

bid – order to buy. Typically accompanied by a quantity, and the highest price the buyer would be willing to pay.

central limit order book (CLOB) – the central data structure

exchange – an entity that facilitates the buying and selling of some set of commodities, often using a central limit order book (CLOB)

level – a price; in the context of a CLOB, multiple orders may come in at the same price level.

price – the amount of money a trader is willing to pay (if buying) or accept (if selling). Typically accompanies a bid or ask. Importantly, prices are not always specified using currencies – the generic term is ‘tick.’ For some exchanges, prices are 64-bit integers!

price level – see level.

side – generic term for whether an order is a bid (to buy) or ask (to sell).

spread – difference between the lowest ask and highest bid price.

symbol – a commodity being traded on the exchange, be it an equity, option, or future.

tick – the smallest unit of price.

NVIDIA just delivered their first Volta-enabled DGX-1 systems – great news for those who need additional compute power: each V100 chip delivers 15.7 TFLOPS of single precision performance, compared to its predecessor that only had 10.6 TFLOPS.

Wait, you say, that’s an interesting qualifier. Who doesn’t “need the additional compute power…?” Did someone hack into Nick’s blog account and post on his behalf? Or has he become a Luddite in his dotage?

Nope, no, I still think more compute is generally better; but it is past time to question the architecture of these systems with huge, discrete GPUs connected to the world by buses. The problem with DGX-1 is that those GPUs are hungry! They need to be fed! And they can only sip data through the tiny soda straw known as the PCI Express bus. (For a 2025 perspective, consider that NVIDIA claims 8,000GB/s per GB200 chip, compared to PCIe 6.0’s 128GB/s unidirectional bandwidth, for a 60x disparity between HBM bandwidth and bus bandwidth. Both of these numbers can be doubled: the Blackwell number because it is a two-die configuration, and the PCIe number because it supports 256GB/s bidirectional bandwidth. —Ed.)

For perspective, let’s compare these chips to G80, the first CUDA-capable GPU. Let’s set the stage by observing that G80 was the largest ASIC NVIDIA could feasibly design and fabricate in 2006, straining the limits of contemporary fabrication technology – a classic “win” chip. It had 684M transistors, a theoretical maximum performance of 384GFLOPS for single precision, and no support at all for double precision. GP100 and GV100 respectively have 22x and 31x more transistors, and 27x and 39x more single precision performance than G80. (In 2025, a Blackwell chip contains 104B transistors, 5x more than 2017’s V100, and they utilize packaging to effectively double that count. So each of the two chiplets has >150x as many transistors as the original CUDA-capable chip! —Ed.) But the bandwidth to deliver data to and from these GPUs has not been increasing commensurately with that performance.

                Transistors   SPFOPS   Bandwidth
Chip  Date       (billions)  (TFLOPS)    (G/s)    FLOPS/byte
 G80  11/8/2006      0.7        0.38       86.4      4.4
GT200 6/16/2008      1.4        0.93      141        6.6
GF100 3/26/2010      3.1        1.35      177        7.6
GK110 11/12/2012     7.0        4.3       288       14.9
P100  4/5/2016      15.3       10.6       732       14.5
V100  6/21/2017     21.2       15.7       900       17.4
A100  5/14/2020     54.2       19.5      2000        9.8 
H100  3/22/2022     80         67        3350       20.0
H200  11/28/2024    80         67        4800       14.0
B200  1/30/2025    104         80        8000       10.0

There is an inflection point in the chart with the introduction of HBM (P100, c. 2016), the memory technology invented to keep GPUs from starving. HBM was able to keep the FLOPS/byte from continuing to diverge at the same rate.

Several other important developments are not reflected in this table:

• TensorCores, which greatly increase the theoretical FLOPS counts deliverable by chips since V100 (c. 2017),

• NVIDIA has pivoted their product designs (and marketing) to focus on peak FLOPS numbers deliverable by TensorCores, not single precision FLOPS, and

• NVLink, which displaced PCIe as the GPU-GPU interconnect within nodes (“scale-up”) - the need for this investment was motivated because the industry standard PCIe bus plateaued at the 3.0 standard for almost a decade (c. 2013-2022).

As the number of FLOPS per byte of I/O diverges, the number of workloads that benefit from more FLOPS diminishes. Googling around for literature on FLOPS/byte, I ran across this 2011 presentation by Peter Kogge entitled “Hardware Evolution Trends of Extreme Scale Computing.” For anyone in the GPU business, the first sign that something’s amiss crops up in Slide 3, which cites “1 byte/s per FLOPS/s” as the “Classical Measure of Performance” (slide added —Ed.):

Even G80’s device memory fell well short of that goal with 1 byte/4.5 FLOPS. I prefer this framing because it adopts the viewpoint of scarcity (bytes/FLOP – getting data in and out for processing) rather than abundance (FLOPS/byte – having lots of processing power to bring to bear on data once it is in hand).

The presentation is from 2011, but still very relevant: after reviewing Moore’s Law and the rise and fall of Dennard scaling, and the preeminent importance of power dissipation in modern computing, the concluding slide reads in part:

• World has gone to multi-core to continue Moore’s Law

• Pushing performance another 1000X will be tough

• The major problem is in energy

• And that energy is in memory & interconnect

• We need to begin rearchitecting to reflect this …

• DON’T MOVE THE DATA!

“DON’T MOVE THE DATA” has been good advice to everyone who’s had the data for decades (in 1992 I wrote a Dr. Dobb’s Journal article that focused on hand-coding x87 assembly to keep intermediate results in registers)… but the advice has more currency now.

Moving The Data on CPUs

The data/compute conundrum finds expression on modern multi-core CPUs, too. Each core on a modern x86 CPU has ILP (instruction level parallelism) of 5, meaning it can detect parallelism opportunities between non-dependent instructions and execute up to 5 instructions in a single clock cycle. Latency to the L3 cache is about 50 clock cycles. So a CPU core can perform at least dozens, and possibly hundreds, of FLOPS on data in registers during the time it takes for the L3 to service a load (2 of the 5 pipelines can do 8 FLOPS per instruction via AVX). And that’s assuming the data was in cache!

As an aside, this observation helps explain why “optimized” numerical Python code is still dead slow. Python is interpreted, so has a library called Numpy that wraps vectorized implementations of operations that do things like element-wise addition or multiplication between arrays. But for the reasons explained above, it is very inefficient to do multiple passes over the data if the computation could have been fused into a single pass. The code spends all of its time moving data, and very little time processing it. (Python developers wishing to optimize such code can use Numba, which enables JIT compilation for operator fusion and other performance optimizations. —Ed.)

DON’T MOVE THE DATA!

A Gift From Heaven: Deep Learning

Which workloads, pray tell, require endless FLOPS per byte of I/O? Or turn it around and ask, which workloads still thrive when there is barely any I/O per FLOP? NVIDIA hasn’t been shy about trumpeting its solution to this problem: deep learning! Training a deep learning network entails refining floating point weights that roughly represent neurons that “learn” as they are trained on the data. As long as the weights can reside in device memory, only a modest amount of I/O is needed to keep the GPU busy. In retrospect, NVIDIA is extremely fortunate that deep learning cropped up. Without it, it’s not clear what workload could soak up all those FLOPS without the GPUs starving. (2025 update: At the end of 2017, NVIDIA’s market capitalization was some $117B. It is now $4.2T, 35x higher.. so to say NVIDIA is “extremely fortunate that deep learning cropped up” turned out to be the understatement of the millenium! —Ed.) The importance of machine learning as a workload helps explain why GV100 contains purpose-built hardware for machine learning, in the form of TensorCores. But that hardware actually exacerbates the GPU starvation problem, by increasing FLOPS without increasing bandwidth.

NVIDIA probably isn’t comfortable betting the farm on a single workload – especially one where their main customers are enterprises that can invest in their own machine learning hardware and that is attracting VC money for application-specific hardware. How do you hedge? How can NVIDIA relieve the bottleneck? Unless some workload materializes that is as compute-intensive (per byte of I/O) as machine learning, NVIDIA must seek out ways to address their GPUs’ I/O bottleneck. (And certainly they did, by purchasing Mellanox for their Infiniband controllers and driving continued investments into their existing NVLink and GPUDirect technologies. —Ed.)

I/O: NVIDIA’s Strategic Landscape

The problem confronted by NVIDIA is that they are hindered by some business and legal challenges. According to the terms of their 2011 settlement with Intel, 1) They do not have a license to Intel’s industry-leading cache coherency protocol technology, and 2) they do not have a license to build x86 CPUs, or even x86 emulators. (How times have changed! The details we know about the Intel/NVIDIA deal appear to be squarely aimed at granting NVIDIA access to x86 processors and Intel’s cache coherency protocol. NVIDIA can leverage the learnings from their own Grace Hopper superchip, preceded by the partnership with IBM to bring coherent access to GPU memory to servers with IBM’s POWER architecture. I don’t know about Jensen, but for NVIDIA’s sake, I was pining for access to QPI back in the late aughts! —Ed.)

NVIDIA has done what they can with the hand they were dealt – they built GPUDirect to enable fellow citizens of the bus (typically Infiniband controllers [and, since, they have added mass storage such as NVMe —Ed.]) to access GPU memory without CPU intervention; they built NVLINK, a proprietary cache coherency protocol. They have licensed NVLINK to IBM for the POWER architecture and signaled a willingness to license it to ARM licensees. The problem is that POWER and ARM64 are inferior to Intel’s x86, whose high-end CPU performance is unmatched and whose “uncore” enables fast, cache coherent access across sockets. NVIDIA itself, though an ARM licensee, has announced that they will not be building a server-class ARM chip. (In April 2021, they reversed this decision and announced Grace. NVIDIA also tried and failed to buy ARM, a $40B purchase that was announced in September 2020 and abandoned in February 2022 amidst regulatory headwinds. In a quirk of coincidence, NVIDIA paid a $1.25B penalty to Softbank after terminating the agreement, an amount reminiscent of the $1.5B paid to NVIDIA by Intel for the private antitrust settlement. —Ed.)

I’m not sure why NVIDIA announced they would not be building their own ARM to drive their GPUs, because that seems like an obvious way for them to own their destiny. It may be that NVIDIA concluded that ARM64 cores simply will never deliver enough performance to drive their GPUs. That’s too bad, because there is a lot of low-hanging fruit in NVIDIA’s driver stack. If they made the software more efficient, it could either run faster on the same hardware or run at the same speed on lesser hardware – like ARM64 cores.

Not being able to coordinate with Intel on the cache coherency protocol has cost NVIDIA big-time in at least one area: peer-to-peer GPU traffic. Intel could, but chooses not to, service peer-to-peer traffic between NVIDIA GPUs at high performance (Intel and NVIDIA give different stories as to the reason, and these conversations happen indirectly because the two companies do not seem to have diplomatic relations). As things stand, if you have a dual-CPU server (such as NVIDIA’s own DGX-1) with cache coherency links between the CPUs, any peer-to-peer GPU traffic must be carefully routed past the CPUs, taking care not to cross the cache coherency link. If Intel could license QPI to Altera, they could license it to NVIDIA. Failing to do so is a matter of choice and a by-product of the two companies’ respective positions in the business and legal landscapes.

As things stand, NVIDIA is dependent on Intel to ship great CPUs (in 2025, of course, AMD has since eaten Intel’s lunch! -Ed.) with good bus integration, and peer-to-peer-capable GPU servers have to be designed to steer traffic around the QPI link. The announcement that NVIDIA would not build ARM64 SOCs was done in 2014, so now that the competitive landscape has evolved (and though I can remember when Intel’s market capitalization was 12x NVIDIA’s, it is now only about 1.7x), it would not surprise me if NVIDIA revisited that decision. (Of course, NVIDIA did enter the server-class ARM business with Grace so my crystal ball was working that day. But if you’d told me that in 2025, NVDA would be worth $4.5T and INTC $185B, I am not sure I would have believed you. —Ed.)

One Path Forward: SoCs

One partial solution to the interconnect problem is to build a System on a Chip (SoC): put the CPU and GPU on the same die. (In 2025, packaging technology has advanced to the point where hardware designers build multi-chip implementations of such architectures, such as AMD’s MI300A or NVIDIA’s GB10. —Ed.) Intel and AMD have been building x86 SOCs for many years; it is Intel’s solution to the value PC market, and AMD has behaved like their life depended on it since 2006, when they acquired GPU vendor ATI. NVIDIA’s Tegra GPUs are all ARM SoCs. The biggest downside of SoCs is that the ratio of CPU/GPU performance is fixed years before the hardware becomes available, causing workloads to suffer if they are more CPU- or GPU-intensive than the SoC was designed to address. (This downside risk of SOC design is mitigated by chiplets, since they are separately fabricated and can even target different semiconductor nodes. —Ed.) And if the device doesn’t have enough performance, scaling performance across multiple chips may be more difficult because GPUs require such high bandwidth. A conspicuous success story for big SoCs has been in the gaming console market, where the target workload is better-understood and, in any case, game developers will code against whatever hardware is in the console.

So I suspect that as workloads continue to tap out the FLOPS and balance out the bandwidth/FLOPS, big SoCs will start to make more sense. In sizing the CPU/GPU ratio, hardware designers can create a device with the biggest possible GPU that doesn’t starve with the available bandwidth.

SoCs are just a stopgap, though. As the laws of physics continue to lower the boom, the importance of system design will continue to increase, as Kogge pointed out in his 2011 presentation. The fundamental problem of the speed of light isn’t going away… ever.

(The DGX Spark [né DIGITS] device announced at GTC 2025, of course, is a system-in-package with ARM cores and a Blackwell GPU. As far as packaging goes, NVIDIA has been late to the chiplet party, possibly due to lingering trauma from the bump crack fiasco of the 2000s; but they are on the way. —Ed.)

When Tim Dettmers et al. published QLoRA (Quantized Low-Rank Adaptation), a memory-efficient method for fine-tuning large language models, they introduced the 4-bit NormalFloat (NF4) representation for quantization. Quoting from their paper, NF4 is “an information theoretically optimal quantization data type for normally distributed data that yields better empirical results than 4-bit Integers and 4-bit Floats.” Section 3 of the paper describes how they arrived at the sixteen (16) values representable by NF4, and Appendix E enumerates them:

constexpr float NF4_LUT[16] = {
    -1.0,
    -0.6961928009986877,
    -0.5250730514526367,
    -0.39491748809814453,
    -0.28444138169288635,
    -0.18477343022823334,
    -0.09105003625154495,
    0.0,
    0.07958029955625534,
    0.16093020141124725,
    0.24611230194568634,
    0.33791524171829224,
    0.44070982933044434,
    0.5626170039176941,
    0.7229568362236023,
    1.0
};

It is pretty wild to consider that the venerable FP32 data type can be usefully distilled into 4 bits by simply picking 16 (sixteen) of the 4 billion or so values representable by FP32, and picking the closest one; but when you think about it, that is what all FP4 representations do. It’s just a bit jarring to see such arbitrary-seeming values written out to double-precision accuracy.

So, how does one convert to and from this representation? From is easy: a simple lookup into the NF4_LUT table above. What about the other direction (“quantization”), i.e. how to convert a value in the range [-1.0, 1.0] to the closest of the 16 encodings of NF4 enumerated above? The value 0.6, for example, falls between NF4_LUT[13]=0.5626 and NF4_LUT[14]=0.7230 and is closest to NF4_LUT[13]. So the (zero-based) index, and the correct NF4 encoding for 0.6, is 13=0xd.

A brute force implementation would simply go through all 16 entries of the lookup table, compute the distance to the input value, and report the index that was closest. That code looks like this:

int
float_to_NF4_0( const float *base, size_t N, float f )
{
    int ret = 0;
    float diff = fabsf( base[0]-f );
    for ( size_t i = 1; i < 16; i++ ) {
        float thisdiff = fabsf( base[i]-f );
        if ( thisdiff < diff ) {
            diff = thisdiff;
            ret = i;
        }
    }
    return ret;
}

This algorithm works for any lookup table - obviously you pass NF4_LUT as the first parameter and 16 as the second. (The parameter N is left over from implementations that supported problem sizes other than 16.)

But looking at the NF4_LUT array, and considering the problem from first principles, reasonably would expect that their structure (they are in sorted order) could be exploited somehow. Binary Search comes to mind; and for my part, I was reminded of a trick that I learned from Jon Bentley’s Programming Pearls (1986), a compendium of his popular monthly column that used to run in Communications of the ACM. Bentley’s optimization, intended for fixed-sized arrays, enables the Binary Search probes to be done with steadily decreasing powers of 2, in an elegantly unrolled loop. For NF4, that looks like this:

int
float_to_NF4_1( const float *base, size_t N, float f )
{
    int i = 0;
    if ( f >= base[i+8] ) i += 8;
    if ( f >= base[i+4] ) i += 4;
    if ( f >= base[i+2] ) i += 2;
    if ( f >= base[i+1] ) i += 1;
    return i + ((base[i+1]-f)<(f-base[i]));
}

After the Binary Search is complete, 0<=i<15 and the correct return value must be i or i+1. The final return value is computed by incrementing i if the input value is closer to NF4_LUT[i+1]. This replaces the exhaustive search of the brute force algorithm with Binary Search.

Are further improvements possible? I thought you would never ask!

Midpoints FTW

The bitsandbytes implementation of the same quantization algorithm does something similar, but more clever, because it eliminates the need for that final check and conditional increment: this code precomputes the 15 midpoints between the 16 LUT entries, and constructs the 4-bit index by evaluating a decision tree (at most 4 if statements):

inline
unsigned char
dQuantizeNF4_0(float x)
{
  // the values for this tree was generated by test_normal_map_tree
  // in the file tests/test_functional.py
  if(x > 0.03979014977812767f)
    if(x > 0.3893125355243683f) // 1
      if(x > 0.6427869200706482f) // 11
        if(x > 0.8614784181118011f) // 111
          return 0b1111;
        else
          return 0b1110;
      else
        if(x > 0.5016634166240692f) // 110
          return 0b1101;
        else
          return 0b1100;
    else
      if(x > 0.2035212516784668f) // 10
        if(x > 0.2920137718319893f) // 101
          return 0b1011;
        else
          return 0b1010;
      else
        if(x > 0.1202552504837513f) // 100
          return 0b1001;
        else
          return 0b1000;
  else
    if(x > -0.33967943489551544f) // 0
      if(x > -0.13791173323988914f) // 01
        if(x > -0.045525018125772476f) // 011
          return 0b0111;
        else
          return 0b0110;
      else
        if(x > -0.23460740596055984f) // 010
          return 0b0101;
        else
          return 0b0100;
    else
      if(x > -0.6106329262256622f) // 00
        if(x > -0.4599952697753906f) // 001
          return 0b0011;
        else
          return 0b0010;
      else
        if(x > -0.8480964004993439f) // 000
          return 0b0001;
        else
          return 0b0000;
}

I don’t know about you, but I find this code as clear as mud. It clarifies things to explicitly create a new lookup table from the midpoints:

constexpr float NF4_LUT_mid[16] = {
    0.0f,
    NF4_LUT[ 0] + 0.5f * ( NF4_LUT[ 1] - NF4_LUT[ 0] ),
    NF4_LUT[ 1] + 0.5f * ( NF4_LUT[ 2] - NF4_LUT[ 1] ),
    NF4_LUT[ 2] + 0.5f * ( NF4_LUT[ 3] - NF4_LUT[ 2] ),
    NF4_LUT[ 3] + 0.5f * ( NF4_LUT[ 4] - NF4_LUT[ 3] ),
    NF4_LUT[ 4] + 0.5f * ( NF4_LUT[ 5] - NF4_LUT[ 4] ),
    NF4_LUT[ 5] + 0.5f * ( NF4_LUT[ 6] - NF4_LUT[ 5] ),
    NF4_LUT[ 6] + 0.5f * ( NF4_LUT[ 7] - NF4_LUT[ 6] ),
    NF4_LUT[ 7] + 0.5f * ( NF4_LUT[ 8] - NF4_LUT[ 7] ),
    NF4_LUT[ 8] + 0.5f * ( NF4_LUT[ 9] - NF4_LUT[ 8] ),
    NF4_LUT[ 9] + 0.5f * ( NF4_LUT[10] - NF4_LUT[ 9] ),
    NF4_LUT[10] + 0.5f * ( NF4_LUT[11] - NF4_LUT[10] ),
    NF4_LUT[11] + 0.5f * ( NF4_LUT[12] - NF4_LUT[11] ),
    NF4_LUT[12] + 0.5f * ( NF4_LUT[13] - NF4_LUT[12] ),
    NF4_LUT[13] + 0.5f * ( NF4_LUT[14] - NF4_LUT[13] ),
    NF4_LUT[14] + 0.5f * ( NF4_LUT[15] - NF4_LUT[14] ),
};

and rewrite the function, replacing the constants with references to NF4_LUT_mid. The resulting code highlights that we’re building indices one bit at a time:

inline
unsigned char
dQuantizeNF4_1(float x)
{
  // the values for this tree was generated by test_normal_map_tree
  // in the file tests/test_functional.py
  if(x > NF4_LUT_mid[0b1000] )
    if(x > NF4_LUT_mid[0b1100]) // 1
      if(x > NF4_LUT_mid[0b1110]) // 11
        if(x > NF4_LUT_mid[0b1111]) // 111
          return 0b1111;
        else
          return 0b1110;
      else
        if(x > NF4_LUT_mid[0b1101]) // 110
          return 0b1101;
        else
          return 0b1100;
    else
      if(x > NF4_LUT_mid[0b1010] ) // 10
        if(x > NF4_LUT_mid[0b1011] ) // 101
          return 0b1011;
        else
          return 0b1010;
      else
        if(x > NF4_LUT_mid[0b1001] ) // 100
          return 0b1001;
        else
          return 0b1000;
  else
    if(x > NF4_LUT_mid[0b0100]) // 1
      if(x > NF4_LUT_mid[0b0110]) // 11
        if(x > NF4_LUT_mid[0b0111]) // 111
          return 0b0111;
        else
          return 0b0110;
      else
        if(x > NF4_LUT_mid[0b0101]) // 110
          return 0b0101;
        else
          return 0b0100;
    else
      if(x > NF4_LUT_mid[0b0010] ) // 10
        if(x > NF4_LUT_mid[0b0011] ) // 101
          return 0b0011;
        else
          return 0b0010;
      else
        if(x > NF4_LUT_mid[0b0001] ) // 100
          return 0b0001;
        else
          return 0b0000;
}

If the pattern doesn’t immediately jump out at you, look at an innermost if statement:

        if(x > NF4_LUT_mid[0b1111]) // 111
          return 0b1111;
        else
          return 0b1110;

and note that if the condition is met, the bit that was tested is set. The same is true of the outer if statements; they are just further removed from the index construction. All of the statements within a given conditional, set the bits controlled by all of the if statements that led to that code path.

Another way to look at this code: each if statement conditionally OR’s a bit into the final output.

Sound familiar? It should - it is akin to Bentley’s optimized Binary Search!

Enter AVX512

Aside: The bitsandbytes code referenced above is CUDA code, not CPU code; but we were able to lift it verbatim to use in our CPU benchmarks. I am not sure if the nested if statements compile to the best GPU code – I suspect that putting the LUT in shared memory or using warp shuffles would do the same calculation more efficiently – but for now, I want to take a look at how we can combine Bentley’s awesome Binary Search optimization with the VPERMPS instruction from AVX512 to compute 16 of these indices at a time for a humongous speedup.

VPERMPS, which may be accessed via intrinsics such as _mm512_permutexvar_ps(), essentially performs a register-to-register lookup, using the least significant 4 bits of the SIMD lanes in one register to index into another register. On both AVX2 and AVX512, VPERMPS is one of the few lane-crossing instructions that can be brought to full effect across the entire register. (Many AVX2 instructions, in particular, operate separately on the top and bottom halves of the register.)

An AVX-512 implementation of the quantization algorithm may be implemented as follows:

void
float_to_NF4_16( uint32_t *out, const float *base, size_t N, const float *f )
{
    __m512i v_i = _mm512_setzero_si512();
    __m512 v_f = _mm512_load_ps( f );
    const __m512 v_lut = _mm512_loadu_ps( &NF4_LUT_mid[0] );
    auto round = [v_lut, &v_i, v_f]( int N2 ) -> void {
        __m512i v_i_N2 = _mm512_add_epi32( v_i, _mm512_set1_epi32( N2 ) );
        __m512 v_lut_i_N2 = _mm512_permutexvar_ps( v_i_N2, v_lut );
        __mmask16 mask_gt = _mm512_cmp_ps_mask( v_f, v_lut_i_N2, _CMP_GE_OS );
        v_i = _mm512_mask_add_epi32( v_i, mask_gt, v_i, _mm512_set1_epi32( N2 )     );
    };
    round( 8 );
    round( 4 );
    round( 2 );
    round( 1 );
    _mm512_store_si512( (__m512i *) out, v_i );
}

The lambda is used for a DRY (do not repeat yourself) pattern, echoing the unrolled Binary Search from before. It’s as clear as intrinsics-based AVX512 code can be, which is to say, not very. But it is fast as hell! The whole function compiles to about 18 machine instructions, with no loops, just a fallthrough execution constructing 16 output values in parallel:

        vmovaps zmm0, zmmword ptr [rcx]
        vmovdqa64       zmm1, zmmword ptr [rip + NF4_LUT_mid]
        vcmpgeps        k1, zmm0, dword ptr [rip + NF4_LUT_mid+32]{1to16}
        vpbroadcastd    zmm2 {k1} {z}, dword ptr [rip + .LCPI0_0]
        vpord   zmm3, zmm2, dword ptr [rip + .LCPI0_1]{1to16}
        vpermd  zmm4, zmm3, zmm1
        vcmpleps        k1, zmm4, zmm0
        vmovdqa32       zmm2 {k1}, zmm3
        vpord   zmm3, zmm2, dword ptr [rip + .LCPI0_2]{1to16}
        vpermd  zmm4, zmm3, zmm1
        vcmpleps        k1, zmm4, zmm0
        vmovdqa32       zmm2 {k1}, zmm3
        vpord   zmm3, zmm2, dword ptr [rip + .LCPI0_3]{1to16}
        vpermd  zmm1, zmm3, zmm1
        vcmpleps        k1, zmm1, zmm0
        vmovdqa32       zmm2 {k1}, zmm3
        vmovdqa64       zmmword ptr [rdi], zmm2
        vzeroupper
        ret

On my old AMD CPU, it is about 30x faster than the fastest scalar implementation, which ironically is the brute force algorithm.

A 30x Speedup

The output from my test program, which is checked into the Parallel Programmer GitHub repository, reads as follows and shows a 29.55x speedup from AVX-512:

gold: 23.64 clocks/iteration
binsearch: 74.22 clocks/iteration
bitsandbytes: 48.63 clocks/iteration
AVX512: 0.80 clocks/iteration

(The “clocks” are just RDTSC ticks, which are an apples-to-apples comparison as long as the comparisons are being done on the same CPU and the CPU implements invariant RDTSC. On Linux, you can check for this CPU feature by searching /proc/cpuinfo for constant_tsc.)

A more recent AVX-512 implementation might be even faster. My testing was done on my trusty Ryzen 7 7700X, the first available AVX-512 capable CPU from AMD.

An AVX2 implementation would be more complicated, because although we do have a register-to-register lookup, there are only 8 32-bit lanes, so the LUT does not fit in a single register. So on the one hand, probably you’d have to select between two possible LUT values in each of the 4 rounds of the calculation; on the other, AVX2 implementations tend to have such great microarchitectures that it’d still be very fast. The VPBLEND instruction would be our friend, maybe for some other day.

On Intrinsics v Hand-Coding

One final word: just yesterday, a Twitter commentator intimated that anyone using intrinsics could instead be hand-coding in assembly language.

I used to belong in that camp, sort of; though in my defense 1) I was writing image processing code where the C programming language’s compulsion to promote short integers to the machine’s natural word length was constantly inhibiting my efforts at SWAR, and 2) I was competing with 1980s-era compiler technology, before SIMD was widely available, on architectures that hadn’t benefited from decades of hardware/software codesign between compilers and hardware ISAs. In the intervening years, I’ve grown to appreciate compilers that will do register allocation and instruction scheduling on my behalf.

Apropos to the topic at hand, the exercise I invite you to undertake – hypothetically, if not literally – is to first, look over the generated code and see if you could improve upon it by hand-coding.

Next, imagine how we transform this function into one that generates the actual intended output of 4 bits per element. Currently it computes 16x32b elements. To compress these further into 4b elements invites a 4x unroll of this function that demotes and interleaves 32b->16b->8b->4b, which would unlock rich ILP opportunities. It might be a day’s work with intrinsics, but it would be many times more work to hand-code, and unlikely to be faster.

Now.. I am here to tell you that everyone is doing this wrong.

The CUDA Handbook has settled on a fairly rigid set of rules around error handling, which offer the best combination of conciseness, correct error handling, and portability to HIP without extra build steps:

1. Check every error code.

2. Use cudaGetLastError() sparingly.

3. Use goto for error handling and cleanup. (see “goto statement considered occasionally useful”)

4. Do not check error codes for API calls that free resources.

A template for a function that allocates and frees resources looks something like this:

cudaError_t
allocateAndUseEvent( size_t N, … )
{
    cudaError_t status_cudart;
    cudaEvent_t event = 0; // IMPORTANT - code not correct w/o initialization
    cuda(EventCreate( &event ) );
    …
Error_cudart:
    cudaEventDestroy( event ); // note - error code not checked!
    return status_cudart;
}

If the function allocates resources on behalf of its caller, the code immediately above the Error_cudart label would free resources as needed, then return cudaSuccess.

Your Mileage May Vary™. The CUDA Handbook macros impose some policy on their users: the status_cudart variable and Error_cudart label must be defined, for example. You may wish to implement different policies (the SDK samples typically print an error and exit the process, rather than propagating the error to a caller), give your error label a different name (we recently changed the names of both the status variable and the error label to enable the admixture of different API families in the same function), or make other changes based on your application’s needs.

For conciseness, our macros prepend ‘cuda’ to the front of the function on behalf of the caller, so functions such as the above one actually cannot be called with our default error handling macros. That said, it is uncommon to write our own functions that propagate native CUDA errors – more typically, when an error is encountered, we transmute it into our own error handling regime before propagating it. For those rare cases, I left the old CUDART_CHECK macro in place in chError.h.

Under no circumstances, however, should you incorporate cudaGetLastError() into your daily error-checking routine. It is just as amenable to invocation via the error handling macros as other CUDA functions, and almost never needs to be called explicitly. The only circumstance when cudaGetLastError() must be called is when a kernel launch may have been misconfigured. In other cases, such as to detect when a running kernel has encountered a memory fault, you can rely on functions such as cudaDeviceSynchronize() . For more context, take a look at my previous article on asychronous error handling.

If all you wanted was a page-long description of the CUDA Handbook’s philosophy on error handling, you can stop reading now. The rest of the article gives an overview of the state of affairs and, for those building applications portable to AMD’s ROCm platform, an honorable mention to our error handling macros’ provisions for stealth HIPification.

State Of Affairs

We’ll start with a quick overview of the CUDA SDK Sample code.

The interval arithmetic sample has a typical error handling macro that prints a message to stderr and exits the process if CUDA returns an error:

#define CHECKED_CALL(func)  \
    do {                    \
        cudaError_t err = (func); \
        if (err != cudaSuccess) { \
            printf(               \
                “%s(%d): ERROR: %s returned %s (err#%d)\n”, __FILE__, __LINE__, #func, cudaGetErrorString(err), err); \
            exit(EXIT_FAILURE);   \
        }                         \
    } while (0)

The NPP samples have error handling macros of their own:

#define NPP_CHECK_CUDA(S) do {cudaError_t eCUDAResult; \
        eCUDAResult = S; \
        if (eCUDAResult != cudaSuccess) std::cout << “NPP_CHECK_CUDA - eCUDAResult = “ << eCUDAResult << std::endl; \
        NPP_ASSERT(eCUDAResult == cudaSuccess);} while (false)

But most samples call a macro checkCudaErrors() that is defined in a variety of places, resulting in code that looks like this fragment from graphConditionalNodes.cu:

    checkCudaErrors(cudaGraphAddNode(&bodyNode, bodyGraph, NULL, NULL, 0, &params));
    checkCudaErrors(cudaGraphInstantiate(&graphExec, graph, NULL, NULL, 0));
    // Initialize device memory and launch the graph
    checkCudaErrors(cudaMemset(dPtr, 0, 1)); // Set dPtr to 0
    printf(”Host: Launching graph with device memory set to 0\n”);
    checkCudaErrors(cudaGraphLaunch(graphExec, 0));
    checkCudaErrors(cudaDeviceSynchronize());
    // Initialize device memory and launch the graph
    checkCudaErrors(cudaMemset(dPtr, 1, 1)); // Set dPtr to 1
    printf(”Host: Launching graph with device memory set to 1\n”);
    checkCudaErrors(cudaGraphLaunch(graphExec, 0));
    checkCudaErrors(cudaDeviceSynchronize());

There is little to no need for such verbosity in our error handling.

Unobtrusive Error Handling

If you define a macro cuda that prepends “cuda” to the function being called, your API calls become more succinct with no loss of correctness. Let’s rewrite the series of CUDA calls from graphConditionalNodes.cu:

    cuda(GraphAddNode(&bodyNode, bodyGraph, NULL, NULL, 0, &params));
    cuda(GraphInstantiate(&graphExec, graph, NULL, NULL, 0));
    // Initialize device memory and launch the graph
    cuda(Memset(dPtr, 0, 1)); // Set dPtr to 0
    printf(”Host: Launching graph with device memory set to 0\n”);
    cuda(GraphLaunch(graphExec, 0));
    cuda(DeviceSynchronize());
    // Initialize device memory and launch the graph
    cuda(Memset(dPtr, 1, 1)); // Set dPtr to 1
    printf(”Host: Launching graph with device memory set to 1\n”);
    cuda(GraphLaunch(graphExec, 0));
    cuda(DeviceSynchronize());

The refactored code is doing exactly the same thing, but more concisely.

For The CUDA Handbook source code, we don’t handle errors the same way that NVIDIA’s sample code does, which typically is to print an error and exit the process. Instead, we use a goto-based error handling scheme to clean up, if necessary, before returning the error to our caller. The macro looks like this:

#define cuda( fn ) do { \
        (status_cudart) =  (cuda##fn); \
        if ( cudaSuccess != (status_cudart) ) { \
            fprintf( stderr, “CUDA Runtime Failure (line %d of file %s):\n\t” \
                “%s returned 0x%x (%s)\n”, \
                __LINE__, __FILE__, #fn, status_cudart, chGetErrorString(status_cudart) ); \
            goto Error_cudart; \
        } \
    } while (0)

The macro enables long sequences of CUDA calls, some of which may fail, to be invoked concisely and correctly. See for example this series of resource allocations and memory copies in stream1Device.cu:

cuda(Malloc( &dptrOut, N*sizeof(float) ) );
cuda(Memset( dptrOut, 0, N*sizeof(float) ) );
cuda(Malloc( &dptrY, N*sizeof(float) ) );
cuda(Memset( dptrY, 0, N*sizeof(float) ) );
cuda(Malloc( &dptrX, N*sizeof(float) ) );
cuda(Memset( dptrY, 0, N*sizeof(float) ) );
cuda(EventCreate( &evStart ) );
cuda(EventCreate( &evHtoD ) );
cuda(EventCreate( &evKernel ) );
cuda(EventCreate( &evDtoH ) );

This code is taking advantage of the timing features in CUDA events to separately measure the host-to-device, kernel execution, and device-to-host runtimes. If we were dedicating three lines of code to the error handling, the code fragment would be much harder to understand.

Stealth HIPification

For those aspiring to port their workloads to HIP, AMD’s rough equivalent to CUDA, we need only modify the macro to prepend “hip” instead of “cuda” to the API call. In the CUDA Handbook source code, this prompted me to split the headers into a chError_cuda.h and chError_hip.h. When using this approach to HIPify, the preprocessor also must be enlisted to transform error codes and constants such as cudaMemcpyHostToDevice. Additionally, our goto-based error handling scheme requires that we explicitly ignore the error codes from functions such as cudaFree() and cudaStreamDestroy(); so the HIP edition of our error handling header file must include preprocessor transformations for those functions, as well.

I wish the itinerant maintainer of the HIPify-perl and HIPify-clang tools would focus instead on a canonical header file to solve this problem in an officially-supported manner; in the meantime, would-be adopters of HIP are left to either use the preprocessor, as I have chosen to do, or use source-to-source translation to preprocess their source files.

Conclusion

Error handling is at once pervasive and prone to pitfalls. CUDA developers can write incorrect code, or code that is doing superfluous error checks, with disturbing ease. I’d encourage anyone maintaining a significant CUDA code base to periodically do an “error handling audit,” and make sure errors are being checked and handled correctly.

Read more

This followup article describes the SIMD implementations of the original “Third Largest” question, with performance results. In the earlier article, we wanted to know: what values of k cause the asymptotically-faster Heap version of the algorithm to actually be faster than the Sort version? The question explored in this article: With the aid of SIMD, how much faster can the Sort version go? And, it is worth the effort?

Quoting from the previous article:

So far, we’ve explored the design space that a good coder might be able to explore during a coding interview. We’ve been good computer science students and used our algorithm analysis skills, replacing an O(Nk) algorithm with an O(Nlgk) algorithm. But what if we give ourselves permission to revisit the sorting algorithm, with the help of SIMD? Can the insertion step for Insertion Sort be implemented in O(1) time on a small array?

Yes. Yes, it can. But exploring that implementation strategy will be left for another article.

And here we are.

It bears mention that the SIMD-accelerated implementations of this function only work for small k, namely k<=8. We’ll touch on implementation strategies for any k, and I believe SIMD instruction sets will deliver higher performance than equivalent scalar implementations, but today we are only talking about small k.

An AVX register can hold a maxSoFar array of up to size 8, and with the right instruction sequence, we can identify where a candidate value must be inserted, conditionally shift the SIMD lanes of the maxSoFar register, and insert the incoming value in sorted order—all using a handful of SIMD instructions.

inline void insertNewMax_x4 ( __m128i& v_maxSoFar, __m128i x ) {
    __m128i v_cmplt = _mm_cmplt_epi32( v_maxSoFar, x );
    __m128i v_insertion_mask = _mm_xor_si128( v_cmplt, _mm_srli_si128( v_cmplt, 4 ) );
    __m128i v_shift1 = _mm_srli_si128( v_maxSoFar, 4 );

    v_maxSoFar = _mm_blendv_epi8( v_maxSoFar, v_shift1, v_cmplt );
    v_maxSoFar = _mm_blendv_epi8( v_maxSoFar, x, v_insertion_mask );
}

inline void insertNewMax( __m128i& v_maxSoFar, int32_t x ) {
    insertNewMax_x4( v_maxSoFar, _mm_set1_epi32( x ) );
}

NOTE: The _mm_srli_si128() intrinsic has two properties that are not obvious at first blush. First, the immediate parameter specifies a byte count. Secondly, the 256-bit analog of this function performs the requested shift on the upper and lower halves of the 256-bit register, placing _mm256_srli_si256() firmly in the Set Of Annoying Nonorthogonal x86 SIMD instructions.

The above function updates maxSoFar, not in O(lgk) time, but in a few instructions: we broadcast the candidate value across one SIMD register, then compare against the values already in the maxSoFar register. If the candidate is “less than” all of the values in maxSoFar, the result is a mask of 0’s, and the subsequent operations do nothing; but if the candidate needs to be inserted into maxSoFar, the mask is 1’s in the lanes where the maxSoFar values are less than the candidate, then 0’s. The lane where the candidate must be inserted can be isolated with a shift and XOR operation. We then use the VBLENDPD instruction (wrapped with the _mm_blend_epi8() intrinsic) to conditionally overwrite maxSoFar, first with a shifted version of itself to make room for the candidate if it will be inserted, then with the candidate value in the lane where it must land.

As with most SIMD coding, this code is not work-efficient! It is comparing and shifting and masking and blending 4 SIMD lanes at a time, zeroing in on the lanes that need to be shifted and/or replaced (and if we needed a bigger maxSoFar array, we could rewrite to use AVX2 or AVX512 sized registers, which would hold 8 or 16 values, respectively).

The AVX2 formulation of the third-largest problem, using an SSE2-valued maxSoFar array, is as follows:

int32_t
thirdLargest_avx2( const std::vector<int32_t>& v )
{
    int32_t minMax;
    __m128i v_maxSoFar;

    {
        v_maxSoFar = _mm_set_epi32( std::numeric_limits<int32_t>::max(),
                                    std::numeric_limits<int32_t>::min(),
                                    std::numeric_limits<int32_t>::min(),
                                    std::numeric_limits<int32_t>::min() );
        insertNewMax( v_maxSoFar, v[0] );
        insertNewMax( v_maxSoFar, v[1] );
        insertNewMax( v_maxSoFar, v[2] );

    }

    minMax = _mm_cvtsi128_si32( v_maxSoFar );
    size_t i = 3;
    while ( i < v.size() ) {
        if ( v[i] > minMax ) {
            insertNewMax( v_maxSoFar, v[i] );
            minMax = _mm_cvtsi128_si32( v_maxSoFar );
        }
        ++i;
    }

    return minMax;
}

The initialization of v_maxSoFar uses sentinels to ensure the first 3 input values are inserted correctly:

        v_maxSoFar = _mm_set_epi32( std::numeric_limits<int32_t>::max(),
                                    std::numeric_limits<int32_t>::min(),
                                    std::numeric_limits<int32_t>::min(),
                                    std::numeric_limits<int32_t>::min() );

For k==2, we’d instead initialize as:

        v_maxSoFar = _mm_set_epi32( std::numeric_limits<int32_t>::max(),
                                    std::numeric_limits<int32_t>::max(),
                                    std::numeric_limits<int32_t>::min(),
                                    std::numeric_limits<int32_t>::min() );

The code then scans the input array, checking to see if the new candidate element v[i] should be inserted into v_minMax:

    minMax = _mm_cvtsi128_si32( v_maxSoFar );
    size_t i = 3;
    while ( i < v.size() ) {
        if ( v[i] > minMax ) {
            insertNewMax( v_maxSoFar, v[i] );
            minMax = _mm_cvtsi128_si32( v_maxSoFar );
        }
        ++i;
    }

Using clang per Godbolt, this code features only 9 AVX2 instructions in the inner loop:

.LBB0_2:
        mov     edi, dword ptr [rcx + 4*rsi]
        cmp     edi, eax
        jle     .LBB0_4
        vmovd   xmm1, edi
        vpbroadcastd    xmm1, xmm1
        vpcmpgtd        xmm2, xmm1, xmm0
        vpsrldq xmm3, xmm2, 4
        vpsrldq xmm4, xmm0, 4
        vblendvps       xmm0, xmm0, xmm4, xmm2
        vpxor   xmm2, xmm3, xmm2
        vpblendvb       xmm0, xmm0, xmm1, xmm2
        vmovd   eax, xmm0
        jmp     .LBB0_4

With SIMD, of course a further optimization is possible: Instead of checking for v[i] > minMax, why not check the next eight (8) incoming values to see if any of them require insertion? If none of them do, we can skip ahead by 8 values, not 1. But if any of them do, we have to find the value or values inside the AVX2 register that need to be inserted. Superficially, it seems the likelihood of each new candidate requiring insertion into v_maxSoFar should go down as the algorithm progresses.

The loop structure looks something like this:

__m256i v_minMax = _mm256_broadcastd_epi32( v_maxSoFar );
size_t Nleft = v.size() - 3;
while ( Nleft >= 8 ) {
    __m256i v_next = _mm256_loadu_si256( (__m256i *) (v.data() + i) );
    __m256i v_cmplt = _mm256_cmpgt_epi32( v_next, v_minMax );
    int mask_lt = _mm256_movemask_ps( _mm256_castsi256_ps( v_cmplt ) );
        
    // if mask_lt has any set bits, some lanes must be inserted into v_minMax

    v_minMax = _mm256_broadcastd_epi32( v_maxSoFar );
    Nleft -= 8;
    i += 8;
}

This loop structure assumes that all 8 lanes of v_next will be processed in each loop iteration (hence the decrement of Nleft and increment of i by 8 at the end of the loop). That needn’t necessarily be the case; we could just insert the “first” (since x86 is little-endian, the least-significant lane), and increment the loop such that the next iteration of the loop restarts just after the just-inserted element. Since we are using unaligned loads, the code would work; it would do some redundant comparisons, but those comparisons would be against the updated v_maxSoFar array, so may result in less work being done. (There’s that pesky data dependence again!)

Here’s the loop with that logic filled in - the g++ intrinsic ffs() returns 0 for inputs of zero, otherwise the bit position of the least significant set bit, plus one. For example, ffs(12)==3. The resulting code - I call it kthLargest_avx2_least, because it inserts the smallest lane of the candidate input - looks like this:

while ( Nleft >= 8 ) {
    __m256i v_next = _mm256_loadu_si256( (__m256i *) (v.data() + i) );
    __m256i v_cmplt = _mm256_cmpgt_epi32( v_next, v_minMax );
    int mask_lt = _mm256_movemask_ps( _mm256_castsi256_ps( v_cmplt ) );

    int lsb = ffs( mask_lt );
    if ( lsb ) {
        __m256i v_newCandidate = _mm256_permutevar8x32_epi32( v_next,
            _mm256_broadcastd_epi32( _mm_cvtsi32_si128( lsb-1) ) );
        insertNewMax_x8( v_maxSoFar, v_newCandidate );
        minMax = _mm_cvtsi128_si32( _mm256_castsi256_si128( v_maxSoFar ) );
        v_minMax = _mm256_set1_epi32( minMax );

        Nleft -= lsb;
        i += lsb;
    }
    else {
        Nleft -= 8;
        i += 8;
    }
}

Alternatively, if mask_lt is nonzero, indicating that there is at least one lane whose value must be inserted into v_maxSoFar, we can fully process the candidate input, inserting each lane for which the corresponding bit in mask_lt is nonzero. The while loop in this code never executes if mask_lt==0. I call this variant kthLargest_avx2_peel(), because it peels and inserts the smallest lane in turn:

while ( Nleft >= 8 ) {
    __m256i v_next = _mm256_loadu_si256( (__m256i *) (v.data() + i) );
    __m256i v_cmplt = _mm256_cmpgt_epi32( v_next, v_minMax );
    int mask_lt = _mm256_movemask_ps( _mm256_castsi256_ps( v_cmplt ) );

    int mask_done = -1;
    while ( 0 != (mask_lt&mask_done) ) {
        int lsb = ffs( mask_done & mask_lt );
        __m128i v_newCandidate = _mm256_castsi256_si128(
            _mm256_permutevar8x32_epi32( v_next,
                _mm256_broadcastd_epi32( _mm_cvtsi32_si128( lsb-1) ) ) );
        insertNewMax_x4( v_maxSoFar, v_newCandidate );
        mask_done = ~((1<<lsb)-1);
    }
    v_minMax = _mm256_broadcastd_epi32( v_maxSoFar );
    Nleft -= 8;
    i += 8;
}

A final variant would unconditionally insert all 8 lanes, if any of them would trigger insertion. This variant relies heavily on the insertNewMax_x4() function being a NOP for inputs that should not be inserted. The code uses a lambda to avoid code duplication:

while ( Nleft >= 8 ) {
    __m256i v_next = _mm256_loadu_si256( (__m256i *) (v.data() + i) );
    __m256i v_cmplt = _mm256_cmpgt_epi32( v_next, v_minMax );
    int mask_lt = _mm256_movemask_ps( _mm256_castsi256_ps( v_cmplt ) );
    auto insert_lane = [&v_maxSoFar, v_next]( int i ) {
        __m128i v_nexti = _mm256_castsi256_si128( 
            _mm256_permutevar8x32_epi32( v_next,
                _mm256_broadcastd_epi32( _mm_cvtsi32_si128( i ) ) ) );
        insertNewMax_x4( v_maxSoFar, v_nexti );
    };

    if ( mask_lt ) {
        insert_lane( 0 );
        insert_lane( 1 );
        insert_lane( 2 );
        insert_lane( 3 );
        insert_lane( 4 );
        insert_lane( 5 );
        insert_lane( 6 );
        insert_lane( 7 );
    }
    v_minMax = _mm256_broadcastd_epi32( v_maxSoFar );
    Nleft -= 8;
    i += 8;
}

All of these AVX2 implementations have the following properties in common:

• they hold the maxSoFar ‘array’ in an AVX2 register (which we call v_maxSoFar);

• they can insert a candidate value into the correct lane of v_maxSoFar in a fixed handful of instructions - no looping required.

• the technique works for any k<=8 because an AVX2 register can hold up to 8xint32_t.

For k>8, multiple AVX2 registers may be used (up to 16, if running in 64-bit mode); or AVX512 may be used, doubling the number of elements per register to 16 and the number of registers to 32. If worse comes to worst, an array of __m256i or __m512i elements can be used, but for any k<=512, an AVX512 implementation could keep v_maxSoFar in registers.

Next: Let’s find out whether all (any?) of this SIMD optimization was worth the effort!

Performance Analysis

We measure performance of the scalar implementations as well as the AVX2 implementations: the register formulation thirdLargest(); kthLargest_heap() and kthLargest_sort() with k==3, even though those reasonably can be expected to perform similarly; and the AVX2 implementations described in the previous section:

• thirdLargest_avx2() inserts each input element in turn,

• thirdLargest_avx2_least() reads the next 8 input elements and, if any need to be inserted, inserts the first and advances the array index to point to the next, and

• thirdLargest_avx2_peel() processes 8 input elements at a time, inserting each element that was greater than the “minimum maximum”. Note, depending on the inputs, this formulation performs some redundant work, since processing elements earlier in the input array may obviate the need to process some subsequent elements.

For random inputs, across ten (10) runs, the various implementations run at the following speeds (in gigaelements per second):

nth - nth_element() STL
heap - kthElement_heap<3>
sort - kthElement_sort<3>
swap - thirdElement<false> - 
dlgt - thirdElement<true>, the Hacker's delight conditional swap
avx2 - naive AVX2 implementation, uses an AVX2 register only for v_maxSoFar

_x8 - AVX2 implementation that checks 8 elements at a time, and skips ahead by 8 if none of the incoming 8 candidates need to be inserted into v_maxSoFar

peel - AVX2 implementation that processes elements out of the incoming candidates, if any of them need to be inserted into v_maxSoFar

least - AVX2 implementation that checks 8 elements at a time and inserts the first into v_maxSoFar, then restarts the scan at the next element

Run  nth   heap  sort  swap  dlgt  avx2   _x8   peel   least
 1   0.27  5.43  5.43  2.76  2.75  2.75  14.55  14.41  14.51
 2   0.90  5.44  5.43  2.75  2.76  2.76  14.45  14.50  14.47
 3   0.28  5.43  5.43  2.75  2.76  2.75  14.62  14.49  14.61
 4   0.41  5.43  5.44  2.75  2.75  2.75  14.55  14.42  14.41
 5   0.25  5.43  5.44  2.75  2.76  2.75  14.51  14.41  14.44
 6   0.48  5.43  5.44  2.75  2.76  2.76  14.64  14.33  14.52
 7   0.69  5.43  5.44  2.75  2.76  2.76  14.62  14.45  14.56
 8   0.45  5.43  5.44  2.75  2.76  2.75  14.60  14.46  14.43
 9   0.43  5.44  5.42  2.75  2.75  2.75  14.69  14.49  14.51
10   0.29  5.44  5.44  2.76  2.76  2.76  14.45  14.34  14.55

Some takeaways from this performance data:

• nth_element() exhibits the most variable performance, with performance on a full-sized data set ranging from 0.26-0.86 billion elements per second (average 0.44, standard deviation of 0.20).

• The benefits of sweeping the input array are clear: even the slowest implementation is about 10x faster than nth_element().

• The AVX2 implementations that “look ahead,” and skip 8 elements at a time when possible, all are about 50x faster than nth_element() and about 5x faster than thirdElement().

• The futility of the Hacker’s Delight cleverness (dlgt as opposed to swap) is laid bare: it is no faster than the ‘naive’ swap, which is eminently more readable.

• Interestingly, the generic kthElement_heap() and kthElement_sort() both run at the same speed, and both are noticeably faster than the thirdElement() versions. One would expect the implementations specific to k==3 to be a bit faster!

For reverse-sorted input (minimum work per element), the results are similar:

Run  nth   heap  sort  swap  dlgt  avx2   _x8   peel   least
 1   2.03  5.42  5.43  2.76  2.76  2.76  14.54  14.56  14.46
 2   2.08  5.42  5.42  2.75  2.76  2.75  14.67  14.59  14.38
 3   2.08  5.42  5.43  2.76  2.75  2.76  14.59  14.47  14.29
 4   2.08  5.41  5.43  2.76  2.75  2.75  14.52  14.44  14.44
 5   2.08  5.42  5.43  2.76  2.75  2.75  14.67  14.57  14.34
 6   2.08  5.40  5.42  2.75  2.75  2.75  14.65  14.27  14.43
 7   2.08  5.43  5.42  2.75  2.75  2.75  14.62  14.59  14.47
 8   2.08  5.42  5.43  2.75  2.75  2.76  14.71  14.49  14.48
 9   2.07  5.41  5.42  2.75  2.76  2.75  14.59  14.47  14.41
10   2.08  5.42  5.42  2.75  2.76  2.76  14.53  14.52  14.48

The main takeaways from this data are:

• The scalar, and especially the AVX2 implementations that “look ahead,” barely run any faster than on randomized data; this is a surprise, since they should be running through the input, detecting that no updates to v_MaxSoFar are needed after initial setup; and

• intriguingly, nth_element() is both more level (not as much variance in speed) and markedly faster; is that because a median-of-three pivot selection on a sorted array amounts to pivoting on the actual median every time?

For sorted input (maximum work per element), the timing results are even more perplexing:

Run  nth   heap  sort  swap  dlgt  avx2   _x8  peel  least
 1   1.82  0.27  1.20  2.10  2.09  0.97  1.00  0.62  0.20
 2   1.82  0.27  1.17  2.09  2.11  0.98  1.00  0.62  0.20
 3   1.82  0.27  1.17  2.10  2.10  0.98  1.00  0.62  0.20
 4   1.82  0.27  1.20  2.10  2.09  0.97  1.00  0.62  0.20
 5   1.82  0.27  1.17  2.10  2.10  0.98  1.00  0.62  0.20
 6   1.82  0.27  1.20  2.10  2.09  0.98  1.00  0.62  0.20
 7   1.82  0.27  1.20  2.07  2.07  0.98  1.00  0.62  0.20
 8   1.82  0.27  1.17  2.09  2.06  0.98  1.00  0.62  0.20
 9   1.82  0.27  1.20  2.06  2.06  0.97  1.00  0.62  0.20
10   1.82  0.27  1.20  2.10  2.10  0.98  1.00  0.62  0.20

Holy smokes! On this CPU, on an already-sorted input, the STL’s nth_element() is faster than any of the AVX2 implementations! Why might that be? One explanation is that the clever handful of AVX2 instructions needed to insert a new candidate into v_maxSoFar is slower than the instructions needed to rearrange the tiny 3-element arrays on the stack for kthElement_sort() and kthElement_heap(). Note that since a new maximum is being inserted into the tiny 3-element heap or sorted array on every iteration, the CPU likely is executing few, if any, mispredicted branches.

Another explanation for the performance difference is that SIMD instructions typically cause CPUs to reduce their clock rate - an argument against a hybrid approach that would do the “lookahead” eight elements at a time, and use scalar instructions to rearrange the maxSoFar array. I haven’t measured the slowdown for the CPU used for these measurements (an 8-core AMD Ryzen 7 7700X, which is one of AMD’s earlier AVX512-capable CPUs).

Conclusions

I’ll leave it to you to decide whether the SIMD implementation was worth pursuing; the workload is synthetic, after all. For random and reverse-sorted inputs (where the top 3 are encountered early in the input), the AVX2 implementations are 5-7x faster. For already-sorted inputs, their performance suffers, in ways that bear investigation.

The source code for these articles has been pushed to the parallelprogrammer repository on GitHub.

If you want to play with holding v_maxSoFar in multiple registers, investigating the perverse behavior on sorted input, and/or porting the code to AVX512, have at it! And let me know how it goes.

In one such case, an interviewer asked me to write a function that returned the third-highest value in an array. For candidates familiar with the C++ STL, it may be tempting to invoke the C++ STL function nth_element() that does just this:

std::nth_element( v.begin(), v.begin()+v.size()-3, v.end() );

The problem is that nth_element() appears to wrap an implementation of Quickselect, the cousin to Quicksort that runs in expected-linear time because it only recurses on one of the two subarrays after the partitioning step. Quickselect, though fast (not optimal – like Quicksort, it suffers from O(N²) worst-case runtime), is terrible for modern CPUs because of the type of unpredictable memory traffic it triggers: data-dependent, conditional swaps of array elements. Modern CPUs hate nothing more than permuting array elements, especially if they can be in far-flung parts of memory, sparsely referencing different cache lines! Availability of data is the limiting reagent of all compute these days, so using 64B cache lines (the x86 standard) to swap 4B elements is exceptionally wasteful. In this context, any reuse within the cache is somewhat coincidental.

Happily, when asked this third-largest question, I wrote a function that kept the top three values in registers, and updated them with a decision tree on each loop iteration. I believe this met the interviewer’s goal of filtering out candidates for the type of performance engineering role I was interviewing for: Such an algorithm scans the input array, rather than trying to rearrange it, and enlists hardware prefetchers when they are available and, as a result, runs much faster.

int32_t
thirdLargest( const std::vector<int32_t>& v )
{
    int32_t v0, v1, v2;

    auto swap_if = []( int32_t& x, int32_t& y, bool predicate ) -> void {
        if ( predicate ) {
            std::swap( x, y );
        }
    };

    auto insertNewMax = [&v0, &v1, &v2, swap_if]( int32_t v3 ) -> void {
        v0 = v3;
        swap_if( v0, v1, v0>v1 );
        swap_if( v1, v2, v1>v2 );
    };
    {
        v0 = v[0];
        v1 = v[1];
        v2 = v[2];

        swap_if( v0, v2, v0>v2 );
        swap_if( v0, v1, v0>v1 );
        swap_if( v1, v2, v1>v2 );
    }

    for ( size_t i = 3; i < v.size(); ++i ) {
        if ( v[i] > v0 ) {
            insertNewMax( v[i] );
        }
    }
    return v0;
}

On my laptop, this function runs 3-11x faster than nth_element(), with an average speedup of 7.4x and standard deviation of 2.8.

As an aside, I tested using the Hacker’s Delight formulation of swap_if, and it ran at exactly the same speed as the more ergonomic code shown above. Using a template parameter fancySwap to enable the branchless, logical-op intensive conditional swap, that code would look like this:

auto swap_if = []( int32_t& x, int32_t& y, bool predicate ) -> void {
    if ( fancySwap ) {
        int32_t mask = -int32_t(predicate);
        x ^= y;
        y ^= (x&mask);
        x ^= y;
    }
    else {
        if ( predicate ) std::swap( x, y );
    }
};

We can thank the architects of modern CPUs for making the easy-to-write, easy-to-understand version just as fast. You can’t blame me for trying, though! I am old enough to remember when a taken branch was 16 clocks versus 4 clocks for a branch not taken. The code is still in the sample, in case you want to run your own tests.

What About Larger k?

I don’t recall my interviewer doing this, but the obvious escalation once a successful solution to the Third-Largest problem has been implemented is to ask: What about the k’th largest value, for arbitrary values of k? What if we want to know, say, the 100^th largest value in the array instead of the 3^rd-largest? We still have nth_element(), of course, but we also can replace our 3 explicit values with an array maxSoFar of size k, and keep them sorted in increasing order. Any incoming value larger than maxSoFar[0], the “minimum maximum” if you will, will cause maxSoFar[0] to be replaced and the incoming value then must be placed in the correct sorted order in the array. This operation amounts to the insertion step in the bedrock Insertion Sort algorithm, which may be asymptotically slow, but is still an important and useful algorithm. Let’s see what our algorithm looks like with a maxSoFar array of size k instead of our explicit 3 values:

template<uint32_t k>
int32_t
kthLargest_sort( const std::vector<int32_t>& v )
{
    // Local array we keep sorted in increasing order
    // The first element is smallest, so any incoming element
    // that is larger must be replace that element, then get
    // moved into position
    std::array<int32_t,k> maxSoFar;

    for ( size_t i = 0; i < k; i++ ) {
        maxSoFar[i] = v[i];
    }
    std::sort( maxSoFar.begin(), maxSoFar.end() );
    int32_t minMax = maxSoFar[0];
    for ( size_t i = k; i < v.size(); i++ ) {
        int32_t x = v[i];

        if ( minMax < x ) {
            size_t j;
            for ( j = 1; j < k && maxSoFar[j] < x; j++ ) {
                maxSoFar[j-1] = maxSoFar[j];
            }
            maxSoFar[j-1] = x;
            minMax = maxSoFar[0];
        }
    }
    return maxSoFar[0];
}

Since maxSoFar is small and local to the stack, shifting its elements around doesn’t have the cache line problem we alluded to earlier – even for large k, it exhibits very cache-friendly behavior. But the number of operations performed for each input element grows linearly with k. At some point, for sufficiently large k, nth_element() likely will run faster. We’ll get into more detail in the Performance Analysis section, but I was surprised at how big k (and maxSoFar) could get before performance stopped being competitive with the STL’s nth_element().

Enter Binary Heaps

Do we need to keep the elements sorted? If you intuition says No, your intuition is correct! For small k, the question makes little difference, but for larger k, there’s a known data structure – the binary heap – that can hold the k largest elements found in the array so far, but the heap only takes O(lgk) time to update, because it does not maintain the elements in sorted order.

My first introduction to binary heaps was in Programming Pearls by Jon Bentley (c. 1986)1. Although heaps conceptually include nodes with parents and children, they are defined to be complete binary trees (no missing children), so can be densely packed into arrays. A clever indexing scheme, not pointers, is used to navigate between parent and child nodes in the heap. When using zero-based indexing, the index of the parent of the i’th heap element is (i-1)/2, and its children are 2i+1 and 2i+2. For a heap to be valid, the values of each child node must be greater than or equal to its parent.

Here's an example heap of size 6, which is represented in memory simply as

{1, 3, 5, 7, 9, 11}

          1
       /     \
      3       5
     / \     /
    7   9  11

To “heapify” an array, we call an operation SiftUp on each array element in turn. SiftUp checks to see if a given element is “smaller” than its parent and, if so, swaps with the parent, working its way toward the “top” of the heap (element zero. The algorithm can early-out: SiftUp stops comparing and swapping as soon as it sees that a node’s parent is less than the node value.

template<typename T, std::size_t k>
inline void
SiftUp( std::array<T,k>& x, size_t i )
{
    while ( i ) {
        size_t p = (i-1)>>1;
        if ( x[p] < x[i] )
            break;
        std::swap( x[i], x[p] );
        i = p;
    }
}

For our application, we are not heapifying the entire array; we are creating a heap of size k, to hold the k largest elements found so far in the input array. Once our k-sized heap is created, the first element with index 0 is known to be the smallest value in the heap; so it is at once:

• the value that must be compared with incoming array elements, to see if they belong in the heap, and

• the value that must be replaced as soon as such an array element is identified.

Once that first element has been replaced, the heap property may no longer hold true, so we perform an operation called SiftDown() that moves downward in the heap, swapping each element with the smaller of its two children until the heap property holds again.

template<typename T, std::size_t k>
inline void
SiftDown( std::array<T,k>& x, size_t i=0 )
{
    size_t c;
    while ( (c = i+i+1) < k ) {
        if ( (c+1)<k && (x[c+1]<x[c]) )
            c += 1;
        if ( x[i] < x[c] ) break;
        std::swap( x[i], x[c] );
        i = c;
    }
}

Here, SiftDown takes the place of the Insertion Sort step, but instead of performing O(k) operations to make the array sorted again, we SiftDown to perform O(lgk) make the heap property hold again. For larger k, SiftDown() should be much faster than the insertion step (e.g. for k=1000, up to 10 swaps instead of up to 1,000 insertion steps).

Now that the basic operations of creating and updating the heap are defined, we can implement our algorithm for any k. Each update of the heap takes O(lgk) steps, and for small k, the heap stays local in very fast L1 cache for the CPU.

The code we’re using in this article to build and update the heap, is from my second book, Classical Algorithms in C++ (c. 1995). My treatment of binary heaps relied heavily on Jon Bentley’s coverage of binary heaps in his Programming Pearls. The only change to the code shown here is that I’ve templatized the heap size as well as the array element type.

Using SiftUp() to build our k-element heap and SiftDown() to update it when necessary, the kthLargest() function may be implemented as follows:

template<uint32_t k>
int32_t
kthLargest_heap( const std::vector<int32_t>& v )
{
    std::array<int32_t,k> heap;

    for ( size_t i = 0; i < k; i++ ) {
        heap[i] = v[i];
        SiftUp( heap, i );
    }
    int32_t minMax = heap[0];
    for ( size_t i = k; i < v.size(); i++ ) {
        int32_t x = v[i];

        if ( minMax < x ) {
            heap[0] = x;
            SiftDown( heap );
            minMax = heap[0];
        }
    }
    return heap[0];
}

Performance Analysis

Before we delve into the tradeoffs between the different algorithms described above, let’s observe that it’s difficult to implement one whose performance is independent of the input.

For example, if the first k values are, in fact, the largest in the input array, then all of our algorithms will quickly scan through the remaining input, rejecting candidates until the end of the input array is reached. From the CPU’s perspective, that becomes a performance test for its hardware prefetchers and branch prediction.

On the opposite end of the spectrum, if the array is sorted, then every input element will trigger a rearrangement of the maxSoFar array (whether an Insertion Sort step or a SiftDown of the heap), echoing the worst-case performance of Quicksort, where naïve implementations run in O(N²) on already-sorted inputs.

The bulk of our performance analysis will be done across 10 randomized input arrays of size 100*2²⁰, but we’ll also gather best- and worst-case performance numbers to confirm our suspicions about how they will behave on sorted inputs.

For nth_element(), the algorithm ran at the same speed for all k, about 37M elements per second. It’s by far the slowest option, so we’ll take it as the baseline and report the speedups of the different formulations.

For the k’th largest, we only have the kthLargest_sort() and kthLargest_heap() variants, which feature expected runtime of O(Nk) and O(Nlgk), respectively. Algorithmic analysis would dictate that nth_element(), the Quickselect implementation in the STL, will run faster than kthLargest_sort() for some k. How big does k have to get before our cache-friendlier kthLargest_sort() gets overtaken by its inferior algorithmic complexity?

In a similar vein, we know that kthLargest_heap() has better algorithmic complexity than kthLargest_sort(), so can ask the same question: where is the crossover? For k==1000, lgk is 10. Shuffling elements up to 1,000 times seems like it would take a lot longer than comparing and swapping elements up to 10 times. But as we already know from our foray into the Hacker’s Delight formulation of a conditional swap, modern CPUs can surprise us and the only way to find out for sure is with real-world testing.

The test program for this article times the kth-largest algorithms with varying values of k: 100…1000, 1000…10,000, and 10,000-100,000. (I may have just spoiled the findings)

For k<=1,000, only slight differences in performance are noted:

      Units: 100M's of elements/s

  k    sort   heap  % faster
 100   5.43   5.42  0%
 200   5.41   5.42  0%
 300   5.39   5.40  0%
 400   5.35   5.40  1%
 500   5.31   5.37  1%
 600   5.26   5.39  2%
 700   5.20   5.35  3%
 800   5.14   5.37  5%
 900   5.07   5.33  5%
1000   5.00   5.36  7%

Amazingly, maintaining a sorted array of the 1,000 largest elements in the input array only runs 7% slower than maintaining a heap. (Both formulations are about 15x faster than nth_element().) The trend is more clear if we chart it, though; Substack has lame support for tables and charts, and Excel’s heuristics for choosing scales and offsets amount to a flair for the dramatic, but you can see where this is going:

It’s interesting, but I have not investigated, why the heap implementation seesaws back and forth for ‘odd’ values (k=600 is slightly faster than k=500, for example).

Let’s see how sort’s performance disadvantage changes if we increase k some more:

  k     sort   heap  % faster
 1000   5.00   5.36  7%
 2000   4.35   5.22  20%
 3000   3.61   5.10  41%
 4000   2.88   5.03  75%
 5000   2.36   5.02  113%
 6000   1.95   4.94  153%
 7000   1.58   4.87  208%
 8000   1.35   4.66  246%
 9000   1.12   4.72  320%
10000   0.97   4.62  378%

For k=10000, the heap is 4.78x faster, and the performance disparity continues to diverge.

The final question on my mind was what value of k would cause kthLargest_sort() to be even slower than the STL’s nth_element(), and on this CPU at least, the answer is somewhere between k=10,000 and k=20,000. (Much larger than I would have expected.)

What’s Next? SIMD, of course

This is the Parallel Programmer Substack, after all.

So far, we’ve explored the design space that a good coder might be able to explore during a coding interview. We’ve been good computer science students and used our algorithm analysis skills, replacing an expected-O(N) algorithm with bad cache behavior with an O(Nk) algorithm with an O(Nlgk) algorithm. But what if we give ourselves permission to revisit the sorting algorithm, with the help of SIMD? Can the insertion step for Insertion Sort be implemented in O(1) time on a small array?

Yes. Yes, it can. But an exploration of how will have to wait until the next article.