Optimizing code for AMD Zen 2

Fetch reads a 32-byte aligned block a cycle into a queue of 20 entries of 16 bytes, 10 for each thread when two run. The decoder scans two entries at once and decodes at most four instructions; its window is 32 bytes from a 16-byte boundary, so branch targets aligned to 16 bytes decode fastest. [AMD 56305, Section 2.9, page 31]

Just the first of the four decode slots takes an instruction longer than 8 bytes, so code that runs from the decoders should have at most one such instruction in any four. [AMD 56305, Section 2.9, page 31]

An op cache entry holds up to eight consecutive instructions that end in the same 64-byte line, with eight slots for 32-bit immediates and displacements (two for a 64-bit one) and at most four microcoded instructions. An entry stops at the end of the line, so a branch to one of a line’s last instructions builds a short one. [AMD 56305, Section 2.9.1, pages 31 and 32]

The front end moves from the decoders to the op cache only at a branch target, and back on an op cache miss. Hot code should fit in the op cache, or in half of it per thread when two threads share the core, which limits unrolling, and only code with a flat code segment uses it. [AMD 56305, Section 2.9.1, pages 31 and 32]

Padding is cheapest in a few long NOPs: Zen 2 decodes a NOP of up to 15 bytes, the 8-byte form with 66h prefixes added, without a penalty, while code that also runs on Family 15h processors stops at 11 bytes. Extra prefixes on an instruction that already has one pad without adding instructions. [AMD 56305, Section 2.8.3.1, pages 29 to 31; Agner Fog, Section 22.3, page 233; LLVM 23.1.2, X86.td, line 1693]

The BTB has three levels: the L0 keeps 16 taken branches, half of them backward, and predicts them with no bubble, the 512-entry L1 costs one bubble when it overrides the L0, and the 7,168-entry L2 four. Zen 1’s levels hold 8, 256 and 4,096 entries, and Zen 3 drops the L0 for a larger L1. Agner Fog gives the same sizes and latencies. [AMD 56305, Section 2.8.1.2, page 27; AMD 55723, Section 2.8.1.2, page 21; AMD 56665, Section 2.8.1.2, pages 28 and 29; Agner Fog, Section 3.13, page 37]

A BTB entry holds two branches when both end in the same 64-byte line and the first is conditional. Code entering a line best meets at most two predicted branches in it before leaving it; a third costs another entry and more cycles to predict. [AMD 56305, Sections 2.8.1.2 and 2.8.3, pages 27 and 29]

Agner Fog measured one taken jump, branch, call or return every two cycles, half the rate of Zen 3, and two not-taken branches a cycle; a loop of at most five instructions that crosses no 64-byte boundary runs an iteration a cycle. [Agner Fog, Section 22.15, pages 236 and 237]

Loop alignment rarely matters. For a hot loop, AMD suggests ending it on the last byte of a 64-byte line, since the core reads one aligned 64-byte block a cycle. [AMD 56305, Section 2.8.3, page 29]

A conditional branch that has never been taken is not in the BTB and is predicted not taken; once taken it is predicted taken until it falls through, and only then does the global history predictor handle it. Branches that are mostly not taken are the cheapest, and a small loop that runs twice is better unrolled. Agner Fog found loops of up to 12 iterations predicted well, where Zen 3 manages 64. [AMD 56305, Section 2.8.1.5, page 28; Agner Fog, Section 3.13, page 36]

The return address stack has 32 entries, 31 of them usable, and 15 per thread with two threads. A call to the next instruction does not unbalance it, though lea rax, [rip] reads RIP more simply, and a one-byte ret is as fast as the longer form once advised. Calls and returns never use the L0 BTB, which makes inlining small functions, or those with few callers, pay off in hot loops. [AMD 56305, Sections 2.8.1.3 to 2.8.1.3.3, pages 27 and 28; Agner Fog, Section 3.13, page 37]

Indirect branches with one target are predicted from the BTB; those with several use a 1,024-entry table indexed with global history, twice Zen 1’s and smaller than Zen 3’s, at the latency of the L2 BTB, so fewer targets per branch predict faster. A branch whose target lies across a 512 KB boundary can neither use the L0 BTB nor share an entry. [AMD 56305, Sections 2.8.1.4 and 2.8.2, pages 28 and 29; AMD 55723, Section 2.8.1.4, page 22]

Only a cmp or test followed by a conditional branch fuses into one macro-op, not one with a displacement together with an immediate or an address relative to RIP; the arithmetic and logic instructions that fuse on Zen 3 do not. LLVM tunes -march=znver2 for that kind of fusion alone. [Agner Fog, Section 22.4, page 233; LLVM 23.1.2, X86.td, line 1694; LLVM 23.1.2, X86.td, line 1710; LLVM 23.1.2, X86.td, line 1718]

Register-to-register moves of general registers of 32 and 64 bits, xchg of them and moves of whole XMM and YMM registers take no execution cycle, where Zen 1 moves the upper half of a YMM register in a pipe. Zeroing idioms such as xor eax, eax and pxor break the dependency on the old value, and sbb eax, eax depends only on the carry flag. [AMD 56305, Section 2.11.2, page 37; Agner Fog, Sections 22.13 and 22.14, page 236]

Byte and word registers are not renamed apart from the full register, so writing AL or AX waits for the last write to any part of RAX; a 32-bit write clears the upper half and does not wait. A write to part of an XMM register waits for the whole register. The flags are tracked in parts, so an instruction that writes only the carry flag does not wait for the zero flag. [Agner Fog, Section 22.13, page 236]

A stack engine tracks RSP, so chains of PUSH and POP do not wait for each other; Agner Fog found PUSH, POP and RET to be one macro-op each with no latency on RSP. [AMD 56305, Section 2.9, page 31; Agner Fog, Section 22.7, page 233]

An lea with two operands takes one cycle on any ALU; one with three, a base, an index and a displacement, goes to an address unit and takes two cycles, with its result returning through ALU 2 or 3. [AMD 56305, Section 2.10.2, page 34; AMD 56305, latency spreadsheet, row 299 (LEA); AMD 56305, latency spreadsheet, row 300 (LEA)]

One ALU multiplies, 64 by 64 bits in three cycles and pipelined; a multiplication that writes two registers delivers the second a cycle later and starts only every other cycle. [AMD 56305, Section 2.10.2, page 34; AMD 56305, latency spreadsheet, row 266 (IMUL); AMD 56305, latency spreadsheet, row 395 (MUL)]

The divider produces two bits of the quotient a cycle, so division is slow: AMD’s spreadsheet gives a 64-bit DIV 41 cycles and a 32-bit one 25, as LLVM’s model does, far more than on Zen 3. LLVM tunes -march=znver2 to divide 64-bit values that fit in 32 bits with the 32-bit instruction. [AMD 56305, Section 2.10.2, page 34; AMD 56305, latency spreadsheet, row 142 (DIV); AMD 56305, latency spreadsheet, row 143 (DIV); LLVM 23.1.2, X86ScheduleZnver2.td, line 240; LLVM 23.1.2, X86.td, line 1701]

PDEP and PEXT are microcoded, as in LLVM’s model, though CPUID reports BMI2; Zen 3 runs them on an ALU in three cycles. [AMD 56305, latency spreadsheet, row 476 (PDEP); AMD 56305, latency spreadsheet, row 477 (PEXT); AMD 56305, Section 2.1, page 19; LLVM 23.1.2, X86ScheduleZnver2.td, line 719]

SHLD and SHRD are microcoded, and LLVM treats them as slow for -march=znver2, as it stops doing for Zen 3. [AMD 56305, latency spreadsheet, row 676 (SHLD); AMD 56305, latency spreadsheet, row 679 (SHRD); LLVM 23.1.2, X86.td, line 1702; LLVM 23.1.2, X86.td, line 1720]

BSF and BSR are six macro-ops with a latency of three and four cycles, while LZCNT and POPCNT are one macro-op of one cycle and TZCNT two; LLVM’s model agrees. [AMD 56305, latency spreadsheet, row 55 (BSF); AMD 56305, latency spreadsheet, row 56 (BSR); AMD 56305, latency spreadsheet, row 326 (LZCNT); AMD 56305, latency spreadsheet, row 547 (POPCNT); AMD 56305, latency spreadsheet, row 716 (TZCNT); LLVM 23.1.2, X86ScheduleZnver2.td, line 222; LLVM 23.1.2, X86ScheduleZnver2.td, line 225]

Vector data travels on 256-bit paths, so a 256-bit instruction is one macro-op where Zen 1 splits it in two, and YMM code does twice the work of XMM code at the same rate. There is no AVX-512: AVX2 is the widest path to dispatch to, as on Zen 3. [AMD 56305, Sections 2.1, 2.3 and 2.11.2, Table 1, pages 18, 21 and 37; AMD 55723, Section 2.3, Table 1, page 15; Agner Fog, Sections 22.10 and 22.11, page 235]

Of the four vector pipes, 0 and 1 multiply, run FMA, compare floats and run AES, 2 and 3 add floats, 3 converts and divides and takes square roots with two in flight, and 0, 1 and 3 add integers. Shuffles run on 1 and 2, shifts on 2, integer multiplications on 0, and stores and moves to integer registers leave from pipe 2. [AMD 56305, Sections 2.11 and 2.11.1, pages 35 and 36]

A float addition or multiplication takes three cycles and an FMA five, each on two pipes, and VPADDD one cycle on any of three; AMD’s spreadsheet, LLVM’s model and Agner Fog agree. Zen 3 cuts the FMA to four. [AMD 56305, latency spreadsheet, row 728 (VADDPS with YMM registers); AMD 56305, latency spreadsheet, row 816 (VFMADD132PS with YMM registers); AMD 56305, latency spreadsheet, row 1080 (VMULPS with YMM registers); AMD 56305, latency spreadsheet, row 1104 (VPADDD with YMM registers); LLVM 23.1.2, X86ScheduleZnver2.td, line 281; LLVM 23.1.2, X86ScheduleZnver2.td, line 345; LLVM 23.1.2, X86ScheduleZnver2.td, line 353; LLVM 23.1.2, X86ScheduleZnver2.td, line 420; Agner Fog, Section 22.10, page 234]

An FMA on pipe 0 or 1 takes its third source over a bus of pipe 3, which then stalls unless its own input comes straight from a result of the same cycle. [AMD 56305, Section 2.11, page 36]

A result that a float instruction hands to an integer vector one, or the reverse, such as vmulps feeding vpand, takes one more cycle; keeping to one data type, vandps after vmulps, avoids it. Agner Fog measured the cycle. [AMD 56305, Sections 2.11.1 and 2.11.2, pages 36 and 37; Agner Fog, Section 22.12, page 235]

AMD’s spreadsheet makes the AVX2 permutes that cross the 128-bit lanes, VPERMD, vpermps, vpermq and vpermpd, microcoded, while vperm2f128 is one macro-op of three cycles on pipe 2. [AMD 56305, latency spreadsheet, row 1176 (VPERM2F128); AMD 56305, latency spreadsheet, row 1178 (VPERMD); AMD 56305, latency spreadsheet, row 1191 (VPERMPD); AMD 56305, latency spreadsheet, row 1192 (VPERMPS); AMD 56305, latency spreadsheet, row 1193 (VPERMQ)]

Running a legacy SSE instruction while any YMM register holds a nonzero upper half, or AVX after it, traps to save or restore those bits of all 16 registers at a cost of about 100 cycles; vzeroupper after AVX code avoids it. Zen 1, whose YMM registers are two halves, has no such penalty, and Zen 3 has it too. [AMD 56305, Section 2.11.6, pages 38 and 39; Agner Fog, Section 22.11, page 235]

Scalar instructions such as SQRTSS or cvtsi2sd keep the upper part of their destination and so wait for its last write, except when the core knows that part is zero. Zeroing a vector register when done with it gives that, and frees its physical register. [AMD 56305, Sections 2.11.2 and 2.11.5, pages 37 and 38]

Multiplication, division and square root may run again with a longer latency when an input or result is denormal; setting FTZ and DAZ in MXCSR avoids it, at the cost of IEEE 754 results. x87 code has no such switch. [AMD 56305, Sections 2.11.2 and 2.11.4, pages 37 and 38; Agner Fog, Section 22.10, page 235]

Gathers such as VPGATHERDD are microcoded, in AMD’s spreadsheet and LLVM’s model, so vector loads and shuffles beat them when the indices are known in advance. [AMD 56305, latency spreadsheet, row 980 (VGATHERDD with YMM registers); AMD 56305, latency spreadsheet, row 984 (VGATHERDPS with YMM registers); LLVM 23.1.2, X86ScheduleZnver2.td, line 956; LLVM 23.1.2, X86ScheduleZnver2.td, line 1051]

A masked load with VPMASKMOVD or vpmaskmovq is one or two macro-ops, but the masked store is microcoded, in AMD’s spreadsheet and in LLVM’s model. [AMD 56305, latency spreadsheet, row 1250 (VPMASKMOVD load with XMM registers); AMD 56305, latency spreadsheet, row 1251 (VPMASKMOVD load with YMM registers); AMD 56305, latency spreadsheet, row 1253 (VPMASKMOVD store with YMM registers); AMD 56305, latency spreadsheet, row 1257 (VPMASKMOVQ store with YMM registers); LLVM 23.1.2, X86ScheduleZnver2.td, line 934]

The load-store unit runs two 256-bit loads and one 256-bit store a cycle, where Zen 1 does two memory operations of which at most one is a store; Zen 3 runs three. [AMD 56305, Section 2.12, page 39; Agner Fog, Section 22.9, page 234]

A load with a base and a displacement, a base and an index without scale, or a displacement alone reaches an integer register in 4 cycles and a vector register in 7. A scaled index, an index with a displacement, or a segment base that is not zero adds one cycle, but only one when there are several, so latency-sensitive code avoids them. [AMD 56305, Section 2.12, pages 39 and 40; LLVM 23.1.2, X86ScheduleZnver2.td, line 91; LLVM 23.1.2, X86ScheduleZnver2.td, line 94]

A load that crosses a 64-byte line, or a store that crosses a 32-byte boundary, takes the cache for two cycles instead of one. 256-bit data gains from 32-byte alignment, which Zen 1, loading it in 128-bit halves, does not need. Aligned data is as fast with the unaligned instructions, such as movups, as with the aligned ones. [AMD 56305, Section 2.6.2, page 23; AMD 55723, Section 2.6.2, pages 17 and 18]

The L1 data cache is banked by address bits 5:3, and two loads to the same bank in a cycle make one of them run again later, which looks like a slower load. [AMD 56305, Section 2.6.2.1, pages 23 and 24]

The L1 data cache picks its way from a hash of the linear address, so two virtual mappings of the same memory used at once miss in the L1 in turn, as can two unrelated lines whose addresses hash alike; such a miss costs at most about an L2 hit, as on Zen 4. [AMD 56305, Section 2.6.2.2, page 24]

A load gets its value from an earlier store still in the store queue when that one store holds every byte of the load, wherever the load starts within it. [AMD 56305, Section 2.12, page 39]

Store forwarding matches on bits 11:0 of the address, so several stores in flight to addresses that differ by multiples of 4 KB slow down a load that needs one of them. A load best follows its store with the same base and index registers, unchanged in between, and a displacement that fits in 8 bits. [AMD 56305, Section 2.12, page 39]

Agner Fog found that Zen 2 hands a stored value straight to a later load or read-modify-write of it, with little or no latency, when both write the memory operand identically, as a base register plus an 8-bit displacement, and the operand is a general register of 32 or 64 bits; a wrong guess, such as a store through another pointer to the same place, costs about 20 cycles. Zen 3 lacks it. [Agner Fog, Section 22.18, pages 237 to 239]

The load queue has 44 entries and the store queue 48, and 22 cache misses can be outstanding at once, against 72, 64 and 24 on Zen 3; Agner Fog gives the same buffer sizes. [AMD 56305, Section 2.12, page 39; Agner Fog, Section 22.16, page 237]

Hardware prefetchers fill the L1 instruction and data caches and the L2, and Agner Fog found them better than software prefetching in most cases. A line that PREFETCHNTA brings into the L2 leaves it early and never enters the L3, which suits data read once but not small blocks that are reused. [AMD 56305, Sections 2.1 and 2.6.4, pages 19 and 25; Agner Fog, Section 22.16, page 237]

Streaming stores such as MOVNTI fill 64-byte write-combining buffers, of which eight lines can be open, seven of them for one thread: streams written whole and aligned to 64 bytes, and few at a time, go fastest. A fence, a locked instruction, an interrupt or a serializing instruction closes the buffers. [AMD 56305, Sections 2.13.3 and 2.13.4, Table 2, pages 41 to 43]

Locked instructions are fastest when their memory operand does not cross a 16-byte boundary, when floating-point instructions stay away from them and with last branch recording off. [AMD 56305, Section 2.15, page 45]

With two threads, the op queue, the store queue and the retire queue are split in half; the floating-point scheduler and the memory request buffers are shared up to a limit per thread; the caches, TLBs, op cache, scheduler entries and register files are shared freely. Agner Fog found that each thread gets about half of fetch and decode. [AMD 56305, Section 2.14, Figure 8, page 44; Agner Fog, Sections 22.19 and 22.21, pages 240 and 241]

Two threads that run the same code share BTB entries only if that code sits at identical linear and physical addresses; threads that run different code collide less in the BTB when it is in different linear pages. Switching between one and two active threads is costly, so a thread that is about to finish is better left running. [AMD 56305, Section 2.14, pages 44 and 45]