Optimizing code for AMD Zen 3

Fetch reads 32 bytes a cycle, a 16-byte aligned window within one 64-byte line, into a queue of 24 entries of 16 bytes, 12 for each thread when two run. The decoder looks at two entries at once and turns at most four instructions into macro-ops, so branch targets on 16-byte boundaries decode fastest. [AMD 56665, Section 2.9, page 33]

Just one of the four decode slots, the first, handles an instruction longer than 8 bytes, where Zen 4 allows 10. Code that runs from the decoders should have at most one such instruction, long NOPs included, in any four. [AMD 56665, Sections 2.8.3.1 and 2.9, pages 32 and 33]

An op cache entry holds up to eight macro-ops of consecutive instructions, with eight slots for 32-bit immediates and displacements (two for a 64-bit one); each pair of such constants beyond four costs a macro-op of room. An entry with microcoded instructions holds at most four of them and five microcoded instructions and other macro-ops together. [AMD 56665, Section 2.9.1, pages 33 and 34]

The front end moves from the decoders to the op cache only at a branch target, and back on an op cache miss. Hot code should fit in the op cache, or in half of it per thread when two threads share the core, which limits unrolling. The op cache is tagged by physical address, so threads running the same code share its entries, and only code with a flat code segment uses it. [AMD 56665, Section 2.9.1, pages 33 and 34]

Padding is cheapest in a few long NOPs: Zen 3 decodes a NOP of up to 15 bytes, the 8-byte form with 66h prefixes added, without a penalty, while code that also runs on earlier AMD processors such as Family 15h stops at 11 bytes. Extra prefixes on an instruction that already has one pad without adding instructions. [AMD 56665, Section 2.8.3.1, pages 31 to 33; Agner Fog, Section 23.3, page 242]

The L1 BTB has 1,024 entries and predicts direct jumps, conditional or not, with no bubble, and calls, returns and indirect branches with one; the 6,656-entry L2 BTB costs three bubbles when it overrides the L1. Agner Fog gives the same sizes, and Zen 4’s L1 BTB is half as large again. [AMD 56665, Section 2.8.1.2, pages 28 and 29; Agner Fog, Section 3.13, page 37]

A BTB entry holds two branches when both end in the same 64-byte line and the first is conditional, and a target far from the branch can leave room for only one. Code entering a line best meets at most two predicted branches in it before leaving it. [AMD 56665, Sections 2.8.1.2 and 2.8.3, pages 28 and 31]

Agner Fog measured one taken jump or branch a cycle, twice the rate of earlier AMD cores, two not-taken branches a cycle and a call or a return every two cycles, fused compare-and-branch pairs included. [Agner Fog, Sections 23.4 and 23.15, pages 242 and 245]

Loop alignment rarely matters. For a hot loop, AMD suggests ending it on the last byte of a 64-byte line; a branch to a target near the end of a line wastes prediction bandwidth on a short fetch block. [AMD 56665, Sections 2.8.1.1 and 2.8.3, pages 28 and 31]

A conditional branch that has never been taken is not in the BTB and is predicted not taken; once taken it is predicted taken until it falls through, and only then does the global history predictor handle it. Branches that are mostly not taken are the cheapest, and a small loop that runs twice is better unrolled. Agner Fog found loops of up to 64 iterations predicted without a miss, as on Zen 4. [AMD 56665, Section 2.8.1.5, page 30; Agner Fog, Section 3.13, page 36]

The return address stack has 32 entries per thread. A call to the next instruction does not unbalance it, though lea rax, [rip] reads RIP more simply, and a one-byte ret is as fast as the longer form once advised. The bubble on every call and return makes inlining small functions, or those with few callers, pay off in hot loops. [AMD 56665, Sections 2.8.1.3 to 2.8.1.3.3, pages 29 and 30; Agner Fog, Section 3.13, page 37]

Indirect branches with one target are predicted from the BTB; those with several use a 1,536-entry table indexed with global history, half the size of Zen 4’s, so fewer targets per branch predict better. Far targets cost capacity: a branch to the other side of a 1 MB boundary needs a BTB entry of its own, and only a few targets beyond a 128 GB boundary, or indirect ones beyond 64 MB, can be tracked at all. [AMD 56665, Sections 2.8.1.4 and 2.8.2, pages 30 and 31; Agner Fog, Section 3.13, page 37]

A cmp, test, add, sub, inc, dec, and, or or xor followed by a conditional branch becomes one macro-op, also with a memory operand, unless it carries a displacement together with an immediate, addresses relative to RIP or makes the pair longer than 15 bytes; inc and dec do not fuse with a branch on the carry flag, and jrcxz never fuses. A cmp of a register with memory is best written with the memory operand second, cmp eax, [rdi] (opcode 3Bh) rather than cmp [rdi], eax. [AMD 56665, Section 2.9.3, pages 35 and 36; Agner Fog, Section 23.4, page 242]

Register-to-register moves of general registers of 32 and 64 bits, xchg of them and moves of whole XMM and YMM registers take no execution cycle, nor do the zeroing idioms such as xor eax, eax and vpxor, which also break the dependency on the old value; sbb eax, eax depends only on the carry flag, and pcmpeqd xmm0, xmm0 sets all ones without waiting. [AMD 56665, Sections 2.9.2 and 2.9.4, pages 34 to 36; Agner Fog, Sections 23.13 and 23.14, pages 244 and 245]

Byte and word registers are not renamed apart from the full register, so writing AL or AX waits for the last write to any part of RAX; a 32-bit write clears the upper half and does not wait. The flags are tracked in parts, so an instruction that writes only the carry flag does not wait for the zero flag. [Agner Fog, Section 23.13, page 245]

PUSH, POP, CALL and RET update RSP without a dependency, and loads, stores, lea and mov from RSP can use the tracked value. Other instructions that read or write RSP cost an extra macro-op and reset the tracking. [AMD 56665, Section 2.9.5, pages 36 and 37; Agner Fog, Section 23.7, page 243]

Dispatch keeps its full width when any six consecutive macro-ops hold at most one integer division, two multiplications, two vector loads, two branches, three memory operations without ALU work, three with it, and four ALU operations in all. [AMD 56665, Section 2.9.6, pages 37 and 38]

An integer instruction or store that is one macro-op becomes two when its address has a base and an index, so AMD advises compilers to avoid that addressing form. [AMD 56665, Section 2.3, Table 1, pages 21 and 22; AMD 56665, Appendix A.2, page 56]

PDEP and PEXT run natively on ALU 1, one per cycle with a latency of three cycles; code with a separate path for processors where they are slow, such as Zen 2, should take the fast path. [AMD 56665, Section 2.10.2, page 40; LLVM 23.1.2, X86ScheduleZnver3.td, line 555]

Only ALU 1 multiplies, 64 by 64 bits in three cycles and pipelined; a multiplication that writes two registers delivers the second one a cycle later. [AMD 56665, Section 2.10.2, page 40; LLVM 23.1.2, X86ScheduleZnver3.td, line 620]

Division runs on ALU 0 and takes about 8 cycles plus 1 for every 9 bits of the quotient, and a typical 64-bit division can start every 8 cycles. [AMD 56665, Section 2.10.2, page 40]

An lea with a base, an index and a displacement is two macro-ops, where one with two of them is one. [AMD 56665, Section 2.10.2, page 40]

SHLD and SHRD have a latency of three cycles, and LLVM’s -march=znver3 no longer treats them as slow, as it does for the Zen cores before. [AMD 56665, Section 2.10.2, page 40; LLVM 23.1.2, X86.td, line 1720]

Vector data travels on 256-bit paths, so YMM code does twice the work of XMM code at the same rate. There is no AVX-512 at all: code built for Zen 4, such as with -march=znver4, may fault, and AVX2 is the widest path to dispatch to. [AMD 56665, Sections 2.1 and 2.11.2, pages 18 and 43; Agner Fog, Sections 23.11 and 23.20, pages 244 and 247; LLVM 23.1.2, X86.td, line 1729]

Of the four vector pipes, 0 and 1 multiply, run FMA, compare floats and run AES and carry-less multiplication, 2 and 3 add and convert, 1 divides and takes square roots with two in flight, and all four run moves, logic and integer additions. Shuffles and shifts run on 1 and 2, integer multiplications on 0 and 3, and only one vector store a cycle leaves on the two store pipes. [AMD 56665, Sections 2.11 and 2.11.1, pages 41 and 43]

LLVM’s model gives a float addition or multiplication three cycles and an FMA four, each on two pipes, and VPADDD one cycle on any of the four. [LLVM 23.1.2, X86ScheduleZnver3.td, line 867; LLVM 23.1.2, X86ScheduleZnver3.td, line 904; LLVM 23.1.2, X86ScheduleZnver3.td, line 939; LLVM 23.1.2, X86ScheduleZnver3.td, line 1078; Agner Fog, Section 23.10, page 244]

An FMA on pipe 0 or 1 takes its third source over a bus of pipe 3, which then stalls unless its own input comes straight from a result of the same cycle; on Zen 4 the third source competes with pipes 4 and 5 instead. [AMD 56665, Section 2.11, page 42]

A result that a float instruction hands to an integer vector one, or the reverse, such as vmulps feeding vpand, takes one more cycle; keeping to one data type, vandps after vmulps, avoids it. Agner Fog measured the cycle, which he did not find on Zen 4, and LLVM’s model adds it. [AMD 56665, Sections 2.11.1 and 2.11.2, pages 43 and 44; Agner Fog, Section 23.12, page 244; LLVM 23.1.2, X86ScheduleZnver3.td, line 486]

Running a legacy SSE instruction while any YMM register holds a nonzero upper half, or AVX after it, traps to save or restore those bits of all 16 registers at a cost of about 100 cycles; vzeroupper after AVX code avoids it. On Zen 4 such an instruction is only slower. [AMD 56665, Section 2.11.6, pages 44 and 45]

Scalar instructions such as SQRTSS or cvtsi2sd keep the upper part of their destination and so wait for its last write, except when the core knows that part is zero. Zeroing a vector register when done with it gives that, and frees its physical register. [AMD 56665, Sections 2.11.2 and 2.11.5, pages 43 to 45]

Multiplication, division and square root may run again with a longer latency when an input or result is denormal; setting FTZ and DAZ in MXCSR avoids it, at the cost of IEEE 754 results. x87 code has no such switch. [AMD 56665, Sections 2.11.2 and 2.11.4, pages 43 and 44; Agner Fog, Section 23.10, page 244]

An FMA shortens a chain that runs through a multiplication into an addition, but lengthens one that runs through the addend: there a separate addition, or unrolling with several accumulators, is faster. [AMD 56665, Section 2.11.7, pages 45 and 46]

Gathers such as VPGATHERDD are microcoded, and a register loaded in pieces is slower than one full-width load, so vector loads and shuffles beat both when the indices are known in advance. [AMD 56665, latency spreadsheet, row 1000 (VGATHERDD with YMM registers); AMD 56665, Section 2.11.2, page 43]

LLVM’s model makes a 256-bit VMASKMOVPS store 42 macro-ops that hold the store pipe for 12 cycles, while the masked load is an ordinary load; Zen 3 has none of the AVX-512 stores under a mask register that are ordinary stores on Zen 4. [LLVM 23.1.2, X86ScheduleZnver3.td, line 864; LLVM 23.1.2, X86ScheduleZnver3.td, line 845]

The load-store unit runs three memory operations each cycle, as on Zen 4. All three may be loads, two of them vector loads; two may be stores, and only one a vector store. [AMD 56665, Sections 2.6.2 and 2.12, pages 24 and 46; Agner Fog, Sections 23.9 and 23.10, pages 243 and 244]

A load with a base and a displacement, a base and an index without scale, or a displacement alone reaches an integer register in 4 cycles and a vector register in 7. A scaled index, a base with an index and a displacement, or a segment base that is not zero adds one cycle, but only one when there are several. [AMD 56665, Section 2.12, pages 46 and 47; Agner Fog, Section 23.16, page 246]

A load that crosses a 64-byte line, or a store that crosses a 32-byte boundary, takes the cache for two cycles instead of one, and crossing a 4 KB page can cost much more; 256-bit data is best aligned to 32 bytes. Aligned data is as fast with the unaligned instructions, such as movups, as with the aligned ones. [AMD 56665, Section 2.6.2, pages 24 and 25]

The L1 data cache is banked by address bits 5:3. Aligned loads of neighbouring addresses never conflict, so a loop that reads several streams does best to load up to three consecutive values, two for vector data, from each stream together; byte accesses next to each other are better merged into wider ones, and narrow or misaligned stores use the banks worst. [AMD 56665, Section 2.6.2.1, page 25]

A load gets its value from an earlier store still in the store queue when that one store holds every byte of the load, wherever the load starts within it; a load that only partly overlaps a store waits about 10 cycles more. [AMD 56665, Section 2.12, page 46; Agner Fog, Section 23.17, page 246]

Agner Fog found that Zen 3 lacks the memory operand renaming of Zen 2 and Zen 4, which hands a stored value straight to a later load of it; he judges the loss small in 64-bit code, which keeps most variables in registers. [Agner Fog, Sections 22.18 and 23.20, pages 239 and 247]

Store forwarding matches on bits 11:0 of the address, so several stores in flight to addresses that differ by multiples of 4 KB slow down a load that needs one of them; Agner Fog measured the same. [AMD 56665, Section 2.12, page 46; Agner Fog, Section 23.16, page 246]

The load-store unit tracks 72 loads out of order, the store queue holds 64 stores and 24 cache misses can be outstanding at once, as LLVM’s model also takes. [AMD 56665, Section 2.12, page 46; LLVM 23.1.2, X86ScheduleZnver3.td, line 372; LLVM 23.1.2, X86ScheduleZnver3.td, line 383]

The L1 prefetchers follow sequential lines up or down, a constant stride per instruction, and recurring patterns within a region; the L2 ones follow sequential lines and fetch the line before or after. Data laid out so that misses follow one of these patterns misses less, and some server models can switch single prefetchers off for random access. A line that PREFETCHNTA brings into the L2 leaves it early and never enters the L3, which suits data read once but not small blocks that are reused. [AMD 56665, Section 2.12.1, pages 48 and 49; AMD 56665, Section 2.6.4, page 26; Agner Fog, Section 23.16, page 246]

Streaming stores fill 64-byte write-combining buffers, of which eight lines can be open, seven of them for one thread: streams written whole and aligned to 64 bytes, and few at a time, go fastest. [AMD 56665, Sections 2.13.3 and 2.13.4, pages 50 and 51]

Locked instructions are fastest when their memory operand does not cross a 16-byte boundary, when floating-point instructions stay away from them and with last branch recording off. [AMD 56665, Section 2.15, page 53]

With two threads, the store queue, the retire queue and the op queue are split in half; the schedulers, the integer register file, the load queue and the write-combining buffers are shared up to a limit per thread; the caches, TLBs, the op cache and the vector register file are shared freely. [AMD 56665, Section 2.14, Table 3, page 52]

Two threads that run the same code share BTB entries only if that code sits at identical linear and physical addresses; threads that run different code collide less in the BTB when it is in different linear pages. Switching between one and two active threads is costly, so a thread that is about to finish is better left running. [AMD 56665, Section 2.14, page 53]

PAUSE stops the thread’s dispatch for about 64 cycles, which saves power and leaves resources to the other thread while a spin loop waits. [AMD 56665, Section 2.9.7, page 38]