Optimizing code for AMD Zen and Zen+

Fetch takes a 32-byte aligned block a cycle into a queue of 20 entries of 16 bytes, 10 of them per thread when two run. The decoder looks at two entries at once and decodes up to four instructions from a 32-byte window that starts on a 16-byte boundary, so branch targets aligned to 16 bytes decode best. [AMD 55723, Section 2.9, page 25]

Of the four decode slots, just the first accepts an instruction longer than 8 bytes, so code decoded rather than read from the op cache should keep such instructions to one in any four. [AMD 55723, Section 2.9, page 25]

An op cache entry keeps up to eight consecutive instructions ending in one 64-byte line, with eight slots for 32-bit immediates and displacements (a 64-bit one takes two) and at most four microcoded instructions. An entry ends with its line, so a jump to one of the last instructions of a line builds a short entry. [AMD 55723, Section 2.9.1, page 26]

The front end switches from the decoders to the op cache only at a branch target, and back when the op cache misses. Hot code that fits in the op cache, or in half of it for each of two threads, avoids the switches, which puts a limit on unrolling; only code with a flat code segment uses it at all. [AMD 55723, Section 2.9.1, page 26]

Padding costs least as a few long NOPs: Zen decodes NOPs of up to 15 bytes, the 8-byte form with 66h prefixes in front, without a penalty, while code that may also run on Family 15h processors should stop at 11 bytes. Extra prefixes on an instruction that already carries one pad without adding instructions. [AMD 55723, Section 2.8.3.1, pages 23 to 25; Agner Fog, Section 22.3, page 233; LLVM 23.1.2, X86.td, line 1693]

Agner Fog found that an instruction of more than two macro-ops, which comes from microcode, takes at least two cycles to decode, so the decoders deliver at most one such instruction every two cycles; prefixes of any kind and number cost nothing. [Agner Fog, Section 22.3, page 233]

The BTB has three levels: the L0 keeps 8 taken branches, 4 of them forward and 4 backward, and predicts them with no bubble, the 256-entry L1 costs one bubble when it overrides the L0, and the 4,096-entry L2 four. Zen 2’s levels hold 16, 512 and 7,168. [AMD 55723, Section 2.8.1.2, page 21]

One BTB entry holds two branches when both end in the same 64-byte line and the first is conditional. From where code enters a line, at most two predicted branches before it leaves is best; a third needs another entry and more cycles. [AMD 55723, Sections 2.8.1.2 and 2.8.3, pages 21 and 23]

Agner Fog measured one taken jump, branch, call or return every two cycles and two branches not taken a cycle; a loop of no more than five instructions that crosses no 64-byte boundary runs an iteration a cycle. [Agner Fog, Section 22.15, pages 236 and 237]

Loop alignment seldom matters. For a hot loop, AMD suggests placing its end on the last byte of a 64-byte line, since the core reads one aligned 64-byte block a cycle. [AMD 55723, Section 2.8.3, page 23]

A conditional branch never yet taken has no BTB entry and is predicted not taken; once taken it is predicted taken until it falls through, and only then does the global history predictor take it over. As not-taken branches leave that history alone, mostly not-taken branches are the cheapest, and a small loop run twice is better unrolled. Agner Fog found loops of up to 12 iterations predicted well. [AMD 55723, Section 2.8.1.5, pages 22 and 23; Agner Fog, Section 3.13, page 36]

The return address stack has 32 entries, of which 31 can be used, and 15 per thread with two threads. A call to the next instruction keeps it balanced, though lea rax, [rip] reads RIP more simply, and a one-byte ret is as fast as the longer form once advised. Calls and returns never enter the L0 BTB, so inlining small functions, or those with few callers, pays in hot loops. [AMD 55723, Sections 2.8.1.3 to 2.8.1.3.3, pages 21 and 22; Agner Fog, Section 3.13, page 37]

An indirect branch that has had one target is predicted from its BTB entry; one with several picks among them in a 512-entry table by global history, at the latency of the L2 BTB, so a correct prediction takes about 5 - 3/N cycles for N targets and fewer targets per branch predict faster. A branch whose target lies across a 512 KB boundary can neither enter the L0 BTB nor share an entry. Zen 2 doubles the table. [AMD 55723, Sections 2.8.1.4 and 2.8.2, pages 22 and 23]

A 64-entry queue, 32 per thread with two threads, tracks each fetched line from fetch to retirement, one entry per line without branches and more where one BTB entry does not cover the line’s branches; when it fills, fetch waits. [AMD 55723, Section 2.8.1.6, page 23]

Only a cmp or test directly followed by a conditional branch fuses into one macro-op, and not when it has a displacement together with an immediate, or an address relative to RIP; no other arithmetic or logic instruction fuses. LLVM tunes -march=znver1 for that kind of fusion alone. [Agner Fog, Section 22.4, page 233; LLVM 23.1.2, X86.td, line 1694]

Register-to-register moves of 32-bit and 64-bit general registers and of XMM registers, and xchg of general registers, take no execution cycle, while a YMM move renames only its lower half and moves the upper one in a pipe. Zeroing idioms such as xor eax, eax and pxor break the dependency on the old value, and sbb eax, eax waits only for the carry flag, which LLVM knows for -march=znver1. [AMD 55723, Section 2.11.2, page 31; Agner Fog, Sections 22.13 and 22.14, page 236; LLVM 23.1.2, X86.td, line 1703]

Byte and word registers are not renamed apart from the whole register, so a write to AL or AX waits for the last write to any part of RAX, while a 32-bit write clears the upper half and does not wait. A write to part of an XMM register waits for the whole register, though on Zen the upper and lower halves of a YMM register are renamed apart. The flags are tracked in parts: an instruction that writes only the carry flag does not wait for the zero flag. [Agner Fog, Section 22.13, page 236]

The decoder tracks RSP, so chains of PUSH and POP do not wait for each other; Agner Fog found PUSH, POP and RET to be one macro-op each, with no latency on RSP. [AMD 55723, Section 2.9, page 25; Agner Fog, Section 22.7, page 233]

An lea with two operands takes one cycle on any ALU; one with three goes to an address unit and takes two cycles, its result coming back through ALU 2 or 3. [AMD 55723, Section 2.10.2, page 28; AMD 55723, latency spreadsheet, row 293 (LEA); AMD 55723, latency spreadsheet, row 294 (LEA)]

One ALU multiplies, up to 64 by 64 bits in three cycles, pipelined; a multiplication with two destination registers delivers the second a cycle later and starts only every other cycle. [AMD 55723, Section 2.10.2, page 28; AMD 55723, latency spreadsheet, row 261 (IMUL); AMD 55723, latency spreadsheet, row 387 (MUL)]

The divider, on ALU 2, yields two bits of the quotient a cycle: AMD’s spreadsheet gives a 64-bit DIV 41 cycles and a 32-bit one 25, as LLVM’s model does. LLVM tunes -march=znver1 to divide 64-bit values that fit in 32 bits with the 32-bit instruction. [AMD 55723, Section 2.10.2, page 28; AMD 55723, latency spreadsheet, row 141 (DIV); AMD 55723, latency spreadsheet, row 142 (DIV); LLVM 23.1.2, X86ScheduleZnver1.td, line 240; LLVM 23.1.2, X86ScheduleZnver1.td, line 241; LLVM 23.1.2, X86.td, line 1701]

PDEP and PEXT are microcoded, in AMD’s spreadsheet and in LLVM’s model, though the core supports BMI2; code with a choice uses shifts and masks instead. [AMD 55723, latency spreadsheet, row 466 (PDEP); AMD 55723, latency spreadsheet, row 467 (PEXT); AMD 55723, Section 2.1, page 13; LLVM 23.1.2, X86ScheduleZnver1.td, line 709]

SHLD is microcoded, taking four cycles in AMD’s spreadsheet, and LLVM treats double shifts as slow for -march=znver1. [AMD 55723, latency spreadsheet, row 648 (SHLD); LLVM 23.1.2, X86.td, line 1702]

BSF and BSR are six macro-ops with a latency of three and four cycles, while LZCNT and POPCNT are one macro-op of one cycle and TZCNT two of two cycles. [AMD 55723, latency spreadsheet, row 54 (BSF); AMD 55723, latency spreadsheet, row 55 (BSR); AMD 55723, latency spreadsheet, row 319 (LZCNT); AMD 55723, latency spreadsheet, row 527 (POPCNT); AMD 55723, latency spreadsheet, row 685 (TZCNT); LLVM 23.1.2, X86ScheduleZnver1.td, line 223; LLVM 23.1.2, X86ScheduleZnver1.td, line 226]

Vector pipes, loads and stores all work on 128 bits at a time, so a 256-bit instruction such as vaddpd with YMM registers is two macro-ops, and a 256-bit load or store two 128-bit ones. YMM code does the same work per cycle as XMM code, which Zen 2 doubles. [AMD 55723, Sections 2.3, 2.11 and 2.11.2, Table 1, pages 15, 28 and 30; Agner Fog, Sections 22.10 and 22.11, page 235]

Compilers differ on 256-bit vectors here. GCC 14.2, tuning for znver1, vectorizes loops with 128-bit vectors, splits 256-bit operations in two and splits unaligned 256-bit stores, while LLVM keeps 256-bit vectors that are cheap to run. [GCC 14.2, config/i386/x86-tune.def, line 546; GCC 14.2, config/i386/x86-tune.def, line 550; GCC 14.2, config/i386/x86-tune.def, line 555; LLVM 23.1.2, X86.td, line 1705]

Of the four vector pipes, 0 and 1 multiply, run FMA and AES and compare floats, 2 and 3 add floats, 3 converts and divides and takes square roots with two in flight, and 0, 1 and 3 add integers. Shuffles run on 1 and 2, shifts on 2, integer multiplications on 0, and stores and moves to integer registers leave from pipe 2. [AMD 55723, Sections 2.11 and 2.11.1, pages 29 and 30]

A float addition takes three cycles, a multiplication three in single precision and four in double, and an FMA five, each on two pipes; AMD’s spreadsheet, LLVM’s model and Agner Fog agree. Zen 2 multiplies doubles in three. [AMD 55723, latency spreadsheet, row 695 (VADDPS with XMM registers); AMD 55723, latency spreadsheet, row 1039 (VMULPD with XMM registers); AMD 55723, latency spreadsheet, row 1041 (VMULPS with XMM registers); AMD 55723, latency spreadsheet, row 780 (VFMADD132PS with XMM registers); LLVM 23.1.2, X86ScheduleZnver1.td, line 282; LLVM 23.1.2, X86ScheduleZnver1.td, line 346; LLVM 23.1.2, X86ScheduleZnver1.td, line 350; LLVM 23.1.2, X86ScheduleZnver1.td, line 354; Agner Fog, Section 22.10, page 234]

Division and square root run on pipe 3: AMD’s spreadsheet gives divps 10 cycles, DIVSD 13, sqrtss 14 and SQRTSD 20, as LLVM’s model does, and LLVM tunes -march=znver1 to compute square roots with these instructions rather than an estimate. [AMD 55723, latency spreadsheet, row 145 (DIVPS); AMD 55723, latency spreadsheet, row 146 (DIVSD); AMD 55723, latency spreadsheet, row 659 (SQRTSS); AMD 55723, latency spreadsheet, row 658 (SQRTSD); LLVM 23.1.2, X86ScheduleZnver1.td, line 320; LLVM 23.1.2, X86ScheduleZnver1.td, line 324; LLVM 23.1.2, X86ScheduleZnver1.td, line 364; LLVM 23.1.2, X86ScheduleZnver1.td, line 368; LLVM 23.1.2, X86.td, line 1695]

An FMA on pipe 0 or 1 reads its third source over a bus of pipe 3, which then stalls unless its own input comes straight from a result of the same cycle. [AMD 55723, Section 2.11, page 29]

A result passed from a float instruction to an integer vector one, or the reverse, such as mulps feeding por, takes one more cycle; staying with one data type, orps after mulps, avoids it. Agner Fog measured the cycle. [AMD 55723, Sections 2.11.1 and 2.11.2, pages 30 and 31; Agner Fog, Section 22.12, page 235]

VPERM2F128 is microcoded, three cycles in AMD’s spreadsheet and eight macro-ops in LLVM’s model, while VINSERTF128 and VEXTRACTF128, which move one 128-bit half, take a cycle; moving halves with them beats a full permute. [AMD 55723, latency spreadsheet, row 1128 (VPERM2F128); AMD 55723, latency spreadsheet, row 968 (VINSERTF128); AMD 55723, latency spreadsheet, row 776 (VEXTRACTF128); LLVM 23.1.2, X86ScheduleZnver1.td, line 980]

Mixing legacy SSE and AVX instructions costs nothing, since each YMM register is two 128-bit registers; Zen 2 brings the penalty back. vzeroupper is microcoded, yet LLVM still places it after AVX code for -march=znver1. [Agner Fog, Section 22.11, page 235; AMD 55723, latency spreadsheet, row 1366 (VZEROUPPER); LLVM 23.1.2, X86.td, line 1704]

Scalar instructions such as SQRTSS or cvtsi2sd keep the upper part of their destination, and so wait for its last write, unless the core knows that part to be zero. Zeroing a vector register when done with it gives that, and frees its physical register. [AMD 55723, Sections 2.11.2 and 2.11.5, pages 31 and 32]

A register is best loaded or stored whole, with movapd rather than movlpd and movhpd; when it must be split, the pieces are best back to back. [AMD 55723, Section 2.11.2, page 31]

Multiplication, division and square root may run again with a longer latency when an input or the result is denormal; FTZ and DAZ in MXCSR avoid it, at the cost of IEEE 754 results, while x87 code has no such switch. [AMD 55723, Sections 2.11.2 and 2.11.4, page 31; Agner Fog, Section 22.10, page 235]

Gathers such as VPGATHERDD and VGATHERDPS are microcoded, 20 cycles with YMM registers in AMD’s spreadsheet, and GCC 14.2 builds none when tuning for znver1: vector loads and shuffles do better where the indices are known. [AMD 55723, latency spreadsheet, row 945 (VGATHERDD with YMM registers); AMD 55723, latency spreadsheet, row 949 (VGATHERDPS with YMM registers); LLVM 23.1.2, X86ScheduleZnver1.td, line 946; GCC 14.2, config/i386/x86-tune.def, line 485; GCC 14.2, config/i386/x86-tune.def, line 496; GCC 14.2, config/i386/x86-tune.def, line 507]

Carry-less multiplication with PCLMULQDQ and the half-precision conversions VCVTPH2PS and VCVTPS2PH are microcoded, in AMD’s spreadsheet and in LLVM’s model. [AMD 55723, latency spreadsheet, row 447 (PCLMULQDQ); AMD 55723, latency spreadsheet, row 746 (VCVTPH2PS with YMM registers); AMD 55723, latency spreadsheet, row 752 (VCVTPS2PH with YMM registers); LLVM 23.1.2, X86ScheduleZnver1.td, line 497; LLVM 23.1.2, X86ScheduleZnver1.td, line 1208; LLVM 23.1.2, X86ScheduleZnver1.td, line 1218]

FMA4, XOP, TBM and LWP are gone, along with 3DNow!, so code built for the Bulldozer family needs another path. [AMD 55723, Section 2.1, page 13; Agner Fog, Section 22.10, page 235]

In x87 code, fxch swaps registers far faster than a store and a load, and nothing should come between fcom and the fnstsw that reads its result. [AMD 55723, Section 2.11.3, page 31]

The load-store unit runs two 128-bit loads and one 128-bit store a cycle in three pipelines, and a 256-bit access takes two of them; Zen 2 widens all three to 256 bits. [AMD 55723, Sections 2.6.2 and 2.12, pages 17 and 32]

A load with a base and a displacement, a base and an index without scale, or a displacement alone reaches an integer register in 4 cycles and a vector register in 7. A scaled index, an index together with a displacement, or a segment base other than zero costs one more cycle, once however many apply, so latency-bound code avoids them. [AMD 55723, Section 2.12, page 33; LLVM 23.1.2, X86ScheduleZnver1.td, line 90]

A load that crosses a 32-byte boundary, or a store that crosses a 16-byte one (or a 64-byte one for the TLB and tags), takes the cache for two cycles instead of one. As 256-bit data moves in 128-bit halves, 16-byte alignment is all it needs, and aligned data is as fast with the unaligned instructions, such as movups, as with the aligned ones. [AMD 55723, Section 2.6.2, pages 17 and 18]

The L1 data cache is banked; which banks a load reads depends on address bits 5:2, its size and its way, and two loads to the same bank in a cycle make one of them run again, which looks like a slower load. [AMD 55723, Section 2.6.2.1, page 18]

The L1 data cache predicts the way from a hash of the linear address, so two virtual mappings of the same memory in use at once keep missing in the L1 in turn, as do two unrelated lines whose addresses hash alike at one index; such a miss costs about an L2 hit. [AMD 55723, Section 2.6.2.2, page 18]

A load takes its value from an earlier store still in the store queue when that one store holds every byte of the load, whatever their alignment. [AMD 55723, Section 2.12, page 33]

Store forwarding matches on bits 11:0 of the address, so stores in flight to addresses that differ by multiples of 4 KB slow a load that needs one of them. A load best follows its store closely, with the same base and index registers unchanged in between and a displacement that fits in 8 bits. Agner Fog saw the false dependence at multiples of 4 KB. [AMD 55723, Section 2.12, page 33; Agner Fog, Section 22.16, page 237]

A value stored to memory and read back goes through store forwarding: Agner Fog measured 15 cycles for a store, an add to memory and a load of one location, where Zen 2 takes two. Values that are read back soon belong in registers. [Agner Fog, Section 22.18, pages 237 and 238]

The load queue and the store queue have 44 entries each, and 22 cache misses can be outstanding at once; Zen 2 has 48 store entries. [AMD 55723, Section 2.12, pages 32 and 33]

Two page table walkers serve misses in the L2 TLBs; the L2 data TLB also keeps page directory entries and a 64-entry cache keeps the two levels above, which lets most walks skip to the last level. A larger guest page backed by smaller host pages is held as the smaller size. [AMD 55723, Chapter 1, page 10, and Section 2.7.3, page 20]

Hardware prefetchers fill the L1 instruction and data caches and the L2, and Agner Fog found them better than software prefetching in most cases. A line that PREFETCHNTA brings into the L2 leaves it early and never enters the L3, which suits data used once but not small blocks that are reused. [AMD 55723, Sections 2.1 and 2.6.4, pages 13 and 19; Agner Fog, Section 22.16, page 237]

Streaming stores such as MOVNTI fill 64-byte write-combining buffers, of which eight lines can be open, seven of them for one thread: streams written whole and aligned to 64 bytes, and few at a time, go fastest. A fence, a locked instruction, an interrupt or a serializing instruction closes the buffers. [AMD 55723, Sections 2.13.3 and 2.13.4, Table 2, pages 35 to 37]

Locked instructions are fastest when their memory operand does not cross a 16-byte boundary, when floating-point instructions keep away from them and with last branch recording off. [AMD 55723, Section 2.15, page 39]

With two threads, the op queue, the store queue and the retire queue are split in half; the floating-point scheduler and the memory request buffers are shared up to a limit per thread; the caches, TLBs, op cache, scheduler entries and register files are shared freely. Agner Fog found that each thread gets about half of fetch and decode, and that turning SMT off may help code that keeps a core busy. [AMD 55723, Section 2.14, Figure 8, page 38; Agner Fog, Sections 22.19 and 22.21, pages 240 and 241]

Two threads that run the same code share BTB entries only if that code sits at identical linear and physical addresses; threads that run different code collide less in the BTB when it is in different linear pages. Going from one active thread to two and back is costly, so a thread about to finish is better left running. [AMD 55723, Section 2.14, pages 38 and 39]

Four cores share an L3 and form a complex; the L3 keeps a copy of the tags of their L2 caches and hands a line from one L2 to another within the complex. Agner Fog found threads talk faster within a complex than across, so threads that share data are best kept in one. [AMD 55723, Section 2.6.4, page 19; Agner Fog, Sections 22.19 and 22.21, pages 240 and 241]