Optimizing code for AMD Zen 4

Fetch reads 32 bytes a cycle, and one decoder turns at most four instructions per cycle into macro-ops, through a queue of 24 entries of 16 bytes; with two threads, each owns 12 of them and both share the decoder. The decoder reads 32 bytes from a 16-byte boundary, so branch targets aligned to 16 bytes give it its full width. [AMD 57647, Section 2.9, pages 25 and 26]

Of the four decode slots, just the first takes an instruction of more than 10 bytes, so code that runs from the decoders should keep such instructions, long NOPs included, to one in every four. [AMD 57647, Sections 2.8.3.1 and 2.9, pages 25 and 26]

An op cache entry holds up to nine macro-ops of consecutive instructions from at most two adjacent 64-byte lines, with eight slots for 32-bit immediates and displacements (two for a 64-bit immediate). More than three such constants, more than one EVEX instruction or any microcoded instruction lowers how many macro-ops the entry takes. [AMD 57647, Section 2.9.1, page 26]

The front end moves from the decoders to the op cache only at a taken branch and back on an op cache miss. Hot code should fit in the op cache, or in half of it per thread when two threads share the core, which limits unrolling. Only code with a flat code segment, as every 64-bit program has, uses the op cache. [AMD 57647, Section 2.9.1, pages 26 and 27]

Padding is cheapest in a few long NOPs: Zen 4 decodes a NOP of up to 15 bytes, the 8-byte form with 66h prefixes added, without a penalty, while code that also runs on earlier AMD processors such as Family 15h stops at 11 bytes. Extra prefixes on an instruction that already has one pad without adding instructions. [AMD 57647, Section 2.8.3.1, pages 24 and 25; Agner Fog, Section 24.3, page 248]

A NOP and the integer instruction next to it, or two NOPs, fuse into one macro-op when together at most 15 bytes long, so padding costs less than on earlier cores. Zen 5 no longer fuses NOPs. [AMD 57647, Section 2.9.5, page 29; Agner Fog, Section 25.3, page 255]

The L1 BTB has 1,536 entries and predicts direct jumps, conditional or not, with no bubble, and calls, returns and indirect branches with one; the 7,680-entry L2 BTB costs three bubbles when it overrides the L1. The bubble on every call and return makes inlining small functions, or those with few callers, pay off in hot loops. The L1 BTB of Zen 5 is ten times larger. [AMD 57647, Sections 2.8.1.2 and 2.8.1.3.3, pages 21 and 22]

A BTB entry holds two branches, so code entering a 64-byte line best meets at most two branches in it before leaving: a third takes another entry and more cycles to predict, unless an earlier branch is unconditional or nearly always taken. [AMD 57647, Section 2.8.3, pages 23 and 24]

Loop alignment rarely matters. For a hot loop, AMD suggests starting it at a 64-byte boundary or, slightly better, ending it on the last byte of a 64-byte line. [AMD 57647, Section 2.8.3, page 23]

A conditional branch that has never been taken is not in the BTB and is predicted not taken; once taken it is predicted taken until it falls through, and only then does the global history predictor handle it. Branches that are mostly not taken are the cheapest. [AMD 57647, Section 2.8.1.5, pages 22 and 23]

The return address stack has 32 entries per thread, against 52 on Zen 5. A call to the next instruction does not unbalance it, though lea rax, [rip] reads RIP more simply, and a one-byte ret is as fast as the longer form once advised. [AMD 57647, Section 2.8.1.3, pages 21 and 22; Agner Fog, Section 24.7, page 249]

Indirect branches with one target are predicted from the BTB; those with several use a 3,072-entry table indexed with global history, so fewer targets per branch predict better. Far targets cost capacity: a branch to the other side of a 1 MB boundary needs a BTB entry of its own, and only a few targets beyond a 128 GB boundary, or indirect ones beyond 64 MB, can be tracked at all. [AMD 57647, Sections 2.8.1.4 and 2.8.2, pages 22 and 23]

A cmp, test, add, sub, inc, dec, and, or or xor followed by a conditional branch becomes one macro-op, also with a memory operand, unless it carries a displacement together with an immediate, addresses relative to RIP or makes the pair longer than 15 bytes; inc and dec do not fuse with a branch on the carry flag. Zen 4 has none of the mov fusion of Zen 5. [AMD 57647, Section 2.9.3, page 28; Agner Fog, Section 24.4, page 248]

xor edx, edx before DIV, or CDQ or CQO before IDIV, with the same operand size and a divisor in a register other than rDX, saves a macro-op: the pair is dispatched as one fewer. [AMD 57647, Section 2.9.4, page 29]

Register-to-register moves of general registers of 32 and 64 bits, xchg of them and moves of whole vector registers take no execution cycle, nor do the zeroing idioms such as xor eax, eax and vpxor, which also break the dependency on the old value; sbb eax, eax depends only on the carry flag. [AMD 57647, Sections 2.9.2 and 2.9.6, pages 27 and 29]

Byte and word registers are not renamed apart from the full register, so writing AL or AX waits for the last write to any part of RAX; a 32-bit write clears the upper half and does not wait. [Agner Fog, Section 24.13, page 251]

PUSH, POP, CALL and RET update RSP without a dependency, and loads, stores, lea and mov from RSP can use the tracked value. Other instructions that read or write RSP cost an extra macro-op and reset the tracking. [AMD 57647, Section 2.9.7, pages 29 and 30]

Dispatch keeps its full width when any six consecutive macro-ops hold at most one integer division, two multiplications, two branches, two vector loads, three memory operations without ALU work, three with it, and four ALU operations in all. [AMD 57647, Section 2.9.8, page 30]

PDEP and PEXT run natively on one ALU, one per cycle with a latency of three cycles, against three per cycle on Zen 5; code with a separate path for processors where they are slow should take the fast path. [AMD 57647, Section 2.10.2, page 33; LLVM 23.1.2, X86ScheduleZnver4.td, line 568]

Only one ALU multiplies, 64 by 64 bits in three cycles and pipelined; a multiplication that writes two registers delivers the second one a cycle later. [AMD 57647, Section 2.10.2, page 33]

Division takes about 8 cycles plus 1 for every 9 bits of the quotient, and a typical 64-bit division can start every 8 cycles. [AMD 57647, Section 2.10.2, page 33]

An lea with a base, an index and a displacement is two macro-ops, where one with two of them is one. [AMD 57647, Section 2.10.2, page 33]

SHLD and SHRD have a latency of three cycles. [AMD 57647, Section 2.10.2, page 33]

Vector data travels on 256-bit paths. A 512-bit operation is one macro-op that the scheduler issues over two cycles, so it does as much work per cycle as two 256-bit ones while taking half the scheduler, register and queue entries; AMD expects 512-bit code to be faster and use less power. On Zen 5 the paths are 512 bits wide and 512-bit operations run at full rate, while Zen 3 has no AVX-512 at all. [AMD 57647, Sections 2.11 and 2.11.2, pages 34 to 36; AMD 56665, Section 2.1, page 18; Agner Fog, Section 24.21, page 254]

LLVM’s model gives VPADDD a latency of one cycle on any of the four pipes, and VFMADD231PS four cycles on pipe 0 or 1; the 512-bit forms keep their pipe for two cycles, so they finish half as many per cycle. [LLVM 23.1.2, X86ScheduleZnver4.td, line 1137; LLVM 23.1.2, X86ScheduleZnver4.td, line 1151; LLVM 23.1.2, X86ScheduleZnver4.td, line 951; LLVM 23.1.2, X86ScheduleZnver4.td, line 952]

Of the four vector pipes, 0 and 1 multiply, run FMA and compare floats, 2 and 3 add and convert, 1 divides and takes square roots, and all four run moves, logic and integer additions. Shuffles and shifts run on 1 and 2, integer multiplications on 0 and 3. [AMD 57647, Section 2.11.1, Table 2, page 36]

An FMA, or an AVX-512 operation with merge masking, needs a third source and competes for the source bus of pipe 4 or 5, which may stall it unless the value comes straight from a result of the same cycle. [AMD 57647, Section 2.11, page 35; AMD 57647, Appendix A.1, page 48]

A result that a float instruction hands to an integer vector one, or the reverse, such as vmulps feeding vpand, takes one more cycle; keeping to one data type, vandps after vmulps, avoids it. LLVM’s model adds the cycle. [AMD 57647, Sections 2.11.1 and 2.11.2, Table 2, pages 36 and 37; LLVM 23.1.2, X86ScheduleZnver4.td, line 499]

An AVX-512 compare that writes a mask register, such as VPCMPD, takes 3, 4 or 5 cycles for 128, 256 or 512 bits, where the older compares into a vector register take one cycle for integers and two for floats. [Agner Fog, Section 24.10, page 250; AMD 57647, latency spreadsheet, row 2068 (VPCMPD)]

VPCOMPRESSD and the other compress instructions are microcoded when they write to memory: compressing into a register and then storing it is faster. [AMD 57647, Section 2.11.2, page 37]

LLVM’s model makes a 256-bit VMASKMOVPS store 42 macro-ops that hold the store pipe for 12 cycles, while the masked load is an ordinary load, and so is an AVX-512 store under a mask register such as vmovups [rdi]{k1}, ymm0. [LLVM 23.1.2, X86ScheduleZnver4.td, line 876; LLVM 23.1.2, X86ScheduleZnver4.td, line 857; LLVM 23.1.2, X86InstrSSE.td, line 7413; LLVM 23.1.2, X86InstrAVX512.td, line 3388; LLVM 23.1.2, X86ScheduleZnver4.td, line 869]

Legacy SSE instructions are slower while bits 511:128 of the vector registers are not zero, since each then has to merge them; vzeroupper after AVX code avoids it. On Zen 3 the same mix traps and costs about 100 cycles each time. [AMD 57647, Section 2.11.6, page 39; AMD 56665, Section 2.11.6, pages 44 and 45; Agner Fog, Section 24.11, page 251]

Scalar instructions such as SQRTSS or cvtsi2sd keep the upper part of their destination and so wait for its last write, except when the core knows that part is zero. Zeroing a vector register when done with it gives that, and frees its physical register. [AMD 57647, Sections 2.11.2 and 2.11.5, pages 37 and 38]

Multiplication, division and square root may run again with a longer latency when an input or result is denormal; setting FTZ and DAZ in MXCSR avoids it, at the cost of IEEE 754 results. x87 code has no such switch. [AMD 57647, Sections 2.11.2 and 2.11.4, pages 37 and 38; Agner Fog, Section 24.10, page 250]

An FMA shortens a chain that runs through a multiplication into an addition, but lengthens one that runs through the addend: there a separate addition, or unrolling with several accumulators, is faster. [AMD 57647, Section 2.11.7, page 39]

A gather is slower than vector loads and shuffles when the indices are known in advance, and a register loaded in pieces is slower than one full-width load. [AMD 57647, Section 2.11.2, page 37]

The load-store unit runs three memory operations each cycle, against four on Zen 5. All three may be loads, two of them vector loads; two may be stores, and only one a vector store. [AMD 57647, Sections 2.6.2 and 2.12, pages 16 and 39; Agner Fog, Section 24.9, page 250]

A 512-bit load takes its 256-bit load pipe for two cycles and one more cycle of latency, and a 512-bit store is two 256-bit stores, so 512-bit code moves no more data per cycle than 256-bit code. [AMD 57647, Section 2.6.2.2, page 17; Agner Fog, Section 24.10, page 250]

A load with a base and a displacement, or a base and an index without scale, reaches an integer register in 4 cycles and a vector register in 7. A scaled index, a base with an index and a displacement, or a segment base that is not zero adds one cycle, but only one when there are several. [AMD 57647, Section 2.12, pages 40 and 41]

A load that crosses a 64-byte line, or a store that crosses a 32-byte boundary, takes the cache for two cycles instead of one, and crossing a 4 KB page can cost much more; 256-bit data is best aligned to 32 bytes. Aligned data is as fast with the unaligned instructions, such as movups, as with the aligned ones. [AMD 57647, Sections 2.6.2 and 2.6.2.2, pages 16 and 17]

The L1 data cache is banked by address bits 5:3. Aligned loads of neighbouring addresses never conflict, so a loop that reads several streams does best to load up to three consecutive values, two for vector data, from each stream together; narrow or misaligned stores use the banks worst. [AMD 57647, Section 2.6.2.1, page 17]

The L1 data cache picks its way from a hash of the linear address, so two virtual mappings of the same memory used at once miss in the L1 in turn, as can two unrelated lines whose addresses hash alike. [AMD 57647, Section 2.6.2.3, page 18]

A load gets its value from an earlier store still in the store queue when that one store holds every byte of the load, wherever the load starts within it. [AMD 57647, Section 2.12, pages 39 and 40]

Store forwarding matches on bits 11:0 of the address, so several stores in flight to addresses that differ by multiples of 4 KB slow down a load that needs one of them. [AMD 57647, Section 2.12, page 40]

The load queue tracks 48 loads that have not completed, the store queue holds 64 stores and 24 cache misses can be outstanding at once, against 104 stores and 124 misses on Zen 5. [AMD 57647, Section 2.12, pages 39 and 40]

The L1 prefetchers follow sequential lines up or down, a constant stride per instruction, and recurring patterns within a region; the L2 ones follow sequential lines and fetch the adjacent line. Data laid out so that misses follow one of these patterns misses less, and some server models can switch single prefetchers off for random access. A line that PREFETCHNTA brings into the L2 leaves it early and never enters the L3, which suits data read once but not small blocks that are reused. [AMD 57647, Section 2.12.1, pages 41 and 42; AMD 57647, Section 2.6.4, pages 18 and 19]

Streaming stores fill 64-byte write-combining buffers, of which eight lines can be open, seven of them for one thread: streams written whole and aligned to 64 bytes, and few at a time, go fastest. [AMD 57647, Sections 2.13.3 and 2.13.4, pages 43 and 44]

Locked instructions are fastest when their memory operand does not cross a 16-byte boundary, a tighter rule than the 64-byte line of Zen 5, and when floating-point instructions stay away from them. [AMD 57647, Section 2.15, page 46]

With two threads, the store queue, the retire queue and the op queue are split in half; the schedulers, register files, load queue and write-combining buffers are shared up to a limit per thread; the caches, TLBs and the op cache are shared freely. [AMD 57647, Section 2.14, Table 4, page 45]

Two threads that run the same code share BTB and op cache entries only if that code sits at identical linear and physical addresses; threads that run different code collide less in the BTB when it is in different linear pages. [AMD 57647, Section 2.14, page 45]

PAUSE stops the thread’s dispatch for about 64 cycles, which saves power and leaves resources to the other thread while a spin loop waits. [AMD 57647, Section 2.9.9, page 31]