Optimizing code for AMD Bulldozer to Excavator

The two cores of a module share the instruction cache, fetch, branch prediction, the decoder (Steamroller has two), the FPU and the L2; each keeps its integer scheduler, ALUs, AGLUs, load-store unit and L1 data cache. Two threads on one module thus compete for the front end and the vector pipes, and Agner Fog warns that an operating system may put them there while another module idles. [AMD 47414, Sections 1.6.7, 2.1, 2.12 and 7.1, pages 27, 30, 41 and 121; Agner Fog, Section 21.18, page 230]

The floating-point unit takes up to four ops a cycle, all from one thread though it may switch threads from one cycle to the next, and up to two loads, possibly of both threads; once inside, ops of both threads run side by side. [AMD 47414, Section 2.11, page 39]

For threads that share data, AMD suggests two per module when the work is mostly integer, so that they meet in the L2, but one per module for floating-point-heavy work, so that they do not compete for the single FPU. Threads on independent data are best spread one per module across the sockets first. [AMD 47414, Section 11.1.1, pages 198 and 199]

Since the L1 data caches write through to the shared L2, two threads on one module can pass data through the L2 with no lag between writer and reader. Across modules, data reaches the L3 only when the writer’s L2 evicts it, so the reader should trail the writer by more than the L2’s size, but not by much more than that pair’s share of the L3. [AMD 47414, Section 11.6, pages 213 and 214]

The instruction cache evicts a line when the same physical code is fetched through a virtual address that differs in bits 14:12, so processes on one module that map a shared library, a position-independent program under ASLR or a page merged between virtual machines at such addresses slow each other. AMD suggests keeping those bits equal in every mapping; large pages are unaffected. [AMD 47414, Section 6.12, pages 118 and 119]

Agner Fog measured L1 data cache throughput several times lower whenever more than one thread ran, even on different modules, and found no reason; L2 throughput is shared only by the two cores of a module. [Agner Fog, Section 21.16, page 229]

Fetch reads aligned 32-byte blocks, one a cycle on Bulldozer and Piledriver and one every other cycle on Steamroller, which has two fetch windows. The decoder looks at two 16-byte windows and decodes up to four instructions that lie wholly within them, so branch targets and hot loops aligned to 16 bytes, or 32 on the earlier models, decode best. [AMD 47414, Sections 1.6.4 and 2.7, pages 26 and 36; AMD 47414, Section 7.1.1, page 121]

Instructions that make one or two macro-ops (single and double) decode in hardware and longer ones come from microcode, during which the decoder does nothing else, so on the shared decoders of Bulldozer and Piledriver a microcoded instruction also holds up the other core. Agner Fog found Piledriver and later decode four singles, a double and two singles, or two doubles a cycle, Bulldozer not two doubles. [AMD 47414, Sections 1.4.3 and 8.1, pages 21 and 131; Agner Fog, Section 21.3, page 221]

An instruction with more than three prefixes decodes over ten times slower, so padding is best made of long NOPs (0F 1F) with at most three 66h prefixes, 11 bytes each at most, as LLVM pads for these cores. Agner Fog measured 14 to 15 extra cycles for four to seven prefixes, counting a three-byte VEX as one and a two-byte VEX as none. [AMD 47414, Sections 5.8 and 7.1.1, pages 94, 95 and 121; Agner Fog, Section 21.3, page 222; LLVM 23.1.2, X86.td, line 1621]

Steamroller replays small loops from its op queue: a loop with fewer than 16 branches that follow a steady pattern is detected, and one of fewer than 40 ops on the paths it takes runs from the buffer, so an unrolled loop is best kept under 40 ops, counting 256-bit instructions twice. GCC fits its unrolling to the buffer for Steamroller and Excavator. [AMD 47414, Sections 1.6.7 and 8.2, pages 28, 135 and 136; Agner Fog, Section 21.4, page 222; GCC 14.2, config/i386/x86-tune.def, line 347]

The BTB has two levels: 512 entries (128 sets of 4) that cost a taken branch one bubble and 5,120 (1,024 sets of 5) that cost four. A separate 512-entry array predicts indirect branches with several targets. Agner Fog reports the predictor shared by the two cores of a module and probably 10,240 L2 entries on Steamroller. [AMD 47414, Section 2.6, page 35; Agner Fog, Section 3.12, page 36]

Conditional branches go to a hybrid predictor, a local and a global one (with 12 bits of history) and a chooser for each branch. Agner Fog found long repeating patterns predicted well after some learning, but nested loops less well, as there is no loop counter. [AMD 47414, Section 2.6, page 35; Agner Fog, Section 3.12, page 36]

A correctly predicted taken branch costs at least one bubble and one not taken none, so code is best laid out to fall through on the likely path, with at most one branch, especially a conditional one, in each 32-byte fetch window. Agner Fog measured one taken branch every two cycles. [AMD 47414, Sections 7.1.1 and 7.9, pages 121, 122 and 129; Agner Fog, Section 21.18, page 232]

The return address stack has 24 entries, so each call should have its ret: a deeper chain loses the oldest and an unpaired call desynchronises it, except a call to the very next instruction, which the core recognises. In 64-bit code, lea rax, [rip] reads RIP without a call. [AMD 47414, Sections 2.6 and 7.4, pages 35, 124 and 125; Agner Fog, Section 3.12, page 36]

A cmp or test directly followed by a conditional branch fuses into one op, unless the compare uses a RIP-relative address or has a displacement as well as an immediate; no other instruction fuses. GCC fuses such pairs when tuning for these cores. [AMD 47414, Section 7.2, pages 122 and 123; Agner Fog, Section 21.5, page 222; GCC 14.2, config/i386/x86-tune.def, line 128; GCC 14.2, config/i386/x86-tune.def, line 133]

LOOP takes 7 cycles, 8 in 64-bit mode, where a dec and a jnz take one each. The branch predictor ignores far jumps, calls and returns, and in 32-bit code a nonzero CS base adds two cycles to every misprediction. [AMD 47414, Sections 7.5, 7.7 and 7.8, pages 126, 128 and 129]

With two ALUs a core runs at most two integer ALU instructions a cycle, one fewer than the AMD cores before it; the AGLUs compute addresses and, on later models, register moves. Agner Fog found pure integer code can gain by moving part of its work to vector registers, even using one element of each. [AMD 47414, Section 2.10.2, Figure 4, pages 37 and 38; Agner Fog, Sections 21.8 and 21.18, pages 223 and 231]

An lea with one or two address parts takes one cycle; one with base, index and displacement is a double instruction of 2 cycles, and the 16-bit form is slower still. Agner Fog and LLVM’s model also count a scaled index as slow. [AMD 47414, Section 5.2 and Appendix B, Table 10, pages 84 and 249; Agner Fog, Section 21.8, page 223; LLVM 23.1.2, X86ScheduleBdVer2.td, line 550; LLVM 23.1.2, X86ScheduleBdVer2.td, line 557]

EX1’s pipelined multiplier takes 4 cycles for operands of up to 32 bits, starting one every 2 cycles, and 6 for 64-bit operands, starting one every 4; LLVM’s model agrees. [AMD 47414, Appendix B, Table 10, pages 248 and 251; Agner Fog, Section 21.8, page 223; LLVM 23.1.2, X86ScheduleBdVer2.td, line 427; LLVM 23.1.2, X86ScheduleBdVer2.td, line 430]

DIV and IDIV are microcoded, and their time grows with the width of the dividend’s value on Bulldozer and of the quotient’s from Piledriver on, up to 76 and 72 cycles for 64-bit operands, so a division by a constant is best done by multiplying. [AMD 47414, Section 9.1 and Appendix B, Table 17, pages 141, 348 and 349]

POPCNT and LZCNT, new with Family 15h, beat any sequence that does their work: POPCNT takes 4 cycles, 6 with 64-bit operands, and starts every 2 or 4 cycles, and LZCNT is a double instruction of 2 cycles. [AMD 47414, Section 9.7 and Appendix B, Table 10, pages 157, 250 and 252; LLVM 23.1.2, X86ScheduleBdVer2.td, line 585; LLVM 23.1.2, X86ScheduleBdVer2.td, line 586]

SHLD and SHRD are microcoded, 4 cycles in AMD’s table. Where a register can be given up, an add and an adc shift by one, and a shr and an lea by two or three, in fewer cycles; LLVM treats double shifts as slow on these cores. [AMD 47414, Section 5.7 and Appendix B, Table 10, pages 92, 93 and 254; LLVM 23.1.2, X86.td, line 1619]

A write to an 8-bit or 16-bit register merges into the whole register and so waits for its last write, while a 32-bit write clears the upper half and does not; movzx eax, bl is better than mov al, bl. Agner Fog found the flags tracked in parts, so an instruction that writes only the carry flag does not wait for the others. [AMD 47414, Section 5.5, pages 86 to 89; Agner Fog, Section 21.13, page 227]

xor or sub of a 32-bit or 64-bit register with itself, and vector forms such as pxor, xorps, pandn, psubd or pcmpeqd, do not wait for the old value, though they still use a pipe, and sbb eax, eax waits only for the carry. Agner Fog found PCMPEQQ and PCMPGTQ recognised only from Steamroller on, and LLVM’s model, made for Piledriver, leaves PCMPEQQ out. [AMD 47414, Section 9.4, page 150; Agner Fog, Section 21.14, pages 227 and 228; LLVM 23.1.2, X86ScheduleBdVer2.td, line 1392; LLVM 23.1.2, X86ScheduleBdVer2.td, line 1445]

rep movs and rep stos have a fast microcode path that moves 16 bytes at a time when the direction flag is clear, the blocks do not overlap and the addresses are aligned to the operand size. It pays for counts in the hundreds, up to about the size of the L1 data cache; below eight, plain moves or a loop do better, and a quadword operand size beats smaller ones. [AMD 47414, Section 9.3, pages 148 and 149]

Every vector pipe is 128 bits wide, so a 256-bit instruction is a double of two macro-ops and does no more per cycle than two 128-bit ones. GCC vectorizes with 128-bit vectors for these cores, and splits 256-bit operations and unaligned 256-bit stores. [AMD 47414, Section 2.11, page 39; Agner Fog, Sections 21.9 and 21.10, pages 224 and 225; GCC 14.2, config/i386/x86-tune.def, line 546; GCC 14.2, config/i386/x86-tune.def, line 550; GCC 14.2, config/i386/x86-tune.def, line 555]

Pipes 0 and 1 hold the two FMA units, which also add, multiply and divide, with integer multiply-add and conversions on pipe 0 and shuffles, packs and permutes on pipe 1; pipes 2 and 3 run integer vector arithmetic and logic, and pipe 3 stores. Steamroller merges pipes 0 and 2 and adds a shuffle unit to the last pipe, so AMD advises it to keep packed integer and floating-point work apart, prefer instructions that more than one pipe can run and move stores out of inner loops. [AMD 47414, Appendix B, Tables 7 and 8, page 241; AMD 47414, Sections 1.6.2 and 1.6.6, pages 25 and 27]

Both FMA pipes add, multiply and fuse multiply-add with a latency of 6 cycles, one less when an FMA feeds the next one. Bulldozer has only FMA4, as in VFMADDPS; Piledriver and later add FMA3, as in VFMADD231PS, in the same pipes. [AMD 47414, Sections 1.6.1 and 2.1, pages 23 and 30; AMD 47414, Appendix B, Tables 12, 13 and 15, pages 277, 303 and 347]

XOP adds integer instructions of three and four operands: VPPERM, which selects bytes from two registers, VPCMOV, a bitwise select, rotates, shifts by a count per element, compares into masks and multiply-accumulates. The permutes and shifts run on pipe 1 in 2 or 3 cycles, and AMD suggests VPCMOV with a compare to replace short branches. [AMD 47414, Sections 1.6.1 and 7.6, pages 23, 126 and 127; AMD 47414, Appendix B, Table 12, pages 280 to 284]

A result passed between the FMA pipes and the integer vector units (logic, shuffles, the multiplier) costs a cycle more in either direction, so xorps or andps on float data adds two. Agner Fog measured the same delays and suggests grouping instructions so that fewer of these crossings fall on the critical path. [AMD 47414, Appendix B, Tables 15 and 16, pages 347 and 348; Agner Fog, Section 21.11, pages 225 and 226]

A move of a whole XMM register with movaps or movapd is done by renaming, with no pipe and no latency, and a computation that follows reads the original register. A YMM move renames its lower half only and copies the upper half in 2 cycles, so the three-operand AVX forms beat a move. [AMD 47414, Section 10.16, pages 183 and 184; AMD 47414, Appendix B, Table 12, pages 266 and 278; Agner Fog, Section 21.12, pages 226 and 227]

Moving a general register to an XMM register with MOVD is a double instruction of 8 cycles, so AMD advises a store and a reload instead, and GCC makes no direct moves to vector registers when tuning for these cores; the way back, MOVD to a general register, is a single instruction. [AMD 47414, Section 10.4, pages 165 and 166; AMD 47414, Appendix B, Table 12, page 267; GCC 14.2, config/i386/x86-tune.def, line 443]

Division and square root run in each FMA unit’s own divider: DIVPD takes 27 cycles and SQRTPD 38 on Bulldozer and Piledriver, while AMD’s table for Steamroller gives single precision fewer cycles, 18 for DIVPS, and double precision more, 33 and 44. [AMD 47414, Section 2.11, page 39; AMD 47414, Appendix B, Tables 12 to 14, pages 263, 274, 288, 299, 318 and 329]

256-bit stores are slow: Agner Fog measured one aligned store every 3 cycles on Bulldozer (10 unaligned) and every 17 to 20 on Piledriver, where he advises against 256-bit registers altogether, and one every 2 cycles (4 unaligned) on Steamroller and Excavator. LLVM’s model holds the store pipe 36 cycles for each. [Agner Fog, Sections 21.9 and 21.10, pages 224 and 225; LLVM 23.1.2, X86ScheduleBdVer2.td, line 735; LLVM 23.1.2, X86ScheduleBdVer2.td, line 1077]

AMD advises against mixing legacy SSE and AVX code: an SSE instruction keeps bits 255:128 of its destination and so waits for their last write, which a VZEROUPPER before the SSE code avoids. LLVM places it after AVX code for these cores, though its model gives it 46 cycles. [AMD 47414, Sections 1.6.2 and 5.5, pages 25, 87 and 91; LLVM 23.1.2, X86.td, line 1625; LLVM 23.1.2, X86ScheduleBdVer2.td, line 1320]

Agner Fog measured about 175 extra cycles on Bulldozer and Piledriver when a floating-point result is subnormal or underflows, unless flush-to-zero is on, and no such cost on Steamroller and Excavator. [Agner Fog, Section 21.9, page 224]

Scalar instructions such as SQRTSS and CVTSI2SD keep the rest of their destination and wait for its last write, unless the core knows that part to be zero, which it tracks from the instructions that cleared it. Zeroing the register before a conversion in a loop, loading scalars with vmovss or vmovsd from memory, and copying them between registers with vmovaps rather than vmovss avoid the wait. [AMD 47414, Sections 5.5 and 10.2, pages 88, 89 and 164; AMD 47414, Sections 10.14 and 10.15, pages 180 to 182; GCC 14.2, config/i386/x86-tune.def, line 64]

Each core’s load-store unit runs two 128-bit loads and one 128-bit store a cycle in two pipelines. [AMD 47414, Sections 2.11 and 2.12, pages 39 and 41]

The L1 data cache has 16 banks of 16 bytes, and two loads to one bank in a cycle delay one of them; Agner Fog saw it for addresses a multiple of 256 bytes apart, unless both are in one line, found it frequent and at times the main limit. [AMD 47414, Section 2.5.2, page 34; Agner Fog, Sections 21.16 and 21.18, pages 229 and 232]

Misalignment costs little within a line: Agner Fog measured one access a cycle, one every 2 or 3 cycles across a 64-byte line and one every 21 across a page. With MXCSR’s misaligned exception mask set, SIMD instructions with a memory operand accept unaligned data, which AMD recommends over separate loads, and movups is as fast as movaps on aligned data. [Agner Fog, Section 21.16, page 229; AMD 47414, Sections 1.6.2, 1.6.3 and 5.1.2, pages 24, 26 and 83]

A load takes its data from a queued store only when both begin at one address, the store is at least as wide and neither is misaligned; otherwise it waits until the store has retired and written the cache, which can take tens of cycles. Two narrow stores into one aligned doubleword followed by a load of either one also fail. [AMD 47414, Section 6.3, pages 100 to 103]

Forwarding first matches address bits 11:4 and the doubleword within a 16-byte bank, and the remaining bits only later, so a load a multiple of 4 KB away from a store in flight starts to forward, is cancelled and runs again. rep movs whose source and destination lie such a distance apart repeats this on every iteration; AMD suggests other addresses, or a loop when bits 6:4 or 11:3 of the two match. [AMD 47414, Sections 6.4 and 9.3, pages 104, 105 and 149; Agner Fog, Section 21.16, page 229]

The load queue holds 40 entries on Bulldozer, 44 on Piledriver and 48 on Steamroller, the store queue 24, or 32 on Steamroller, and up to 16 cache misses, prefetches included, can be outstanding. LLVM’s model takes 40 and 24. [AMD 47414, Sections 2.12 and 6.5, pages 41 and 109; LLVM 23.1.2, X86ScheduleBdVer2.td, line 135; LLVM 23.1.2, X86ScheduleBdVer2.td, line 143]

An L1 prefetcher follows strides of up to 504 bytes from one instruction, as many as 12 at once, until a page ends; an L2 prefetcher tracks up to 4,096 streams, strided or merely correlated. GCC turns on software prefetching at -O3 for these cores. [AMD 47414, Section 6.5, page 106; GCC 14.2, config/i386/x86-tune.def, line 237]

PREFETCHT0, T1 and T2 all fill the L1, like PREFETCH, and PREFETCHW fills it for writing. PREFETCHNTA fills the L1 too, but the line skips the L2 on eviction unless it came from there, which suits large arrays read once, not arrays only written, where streaming stores fit. Prefetching more than two arrays a multiple of 4 KB apart thrashes the 4-way L1. [AMD 47414, Section 6.5, pages 105 to 108]

Streaming stores such as MOVNTI go through four 64-byte write-combining buffers. A line written only in part may stay open long and slow the stream, which AMD says can be as much as six times slower than on earlier AMD cores, so streams should fill whole aligned lines; fences, locked and serializing instructions, I/O, uncacheable reads and interrupts close the buffers. [AMD 47414, Section 6.5, page 109; AMD 47414, Appendix A.3 and A.5, Table 6, pages 234 and 235]

Code written with ordinary stores leaves modified lines in the L2, which the instruction cache cannot take, so they go out to memory before the code runs; a JIT compiler that writes code with streaming stores avoids the round trip. [AMD 47414, Section 6.10, page 117]

MFENCE, SFENCE and serializing instructions hold up the core until all earlier work and memory accesses finish, where a locked instruction does not. As a store-load barrier, AMD suggests an xchg with an aligned location the core holds alone, or lock or of zero to the stack, and GCC avoids MFENCE for these cores; a misaligned lock costs a bus lock. [AMD 47414, Section 11.5.1, pages 211 and 212; GCC 14.2, config/i386/x86-tune.def, line 357]

Piledriver (model 02h and models 10h to 1Fh) adds FMA3, F16C, BMI and TBM, register moves on the AGLUs, a 64-entry L1 data TLB, a 44-entry load queue and division timed by the quotient. Agner Fog found its L2 much better than Bulldozer’s and its 256-bit stores much worse. [AMD 47414, Sections 2.1, 2.9.2, 2.10.2 and 2.12, pages 30, 37, 38 and 41; AMD 47414, Appendix B, Section B.6, page 348; Agner Fog, Sections 21.9 and 21.16, pages 224 and 229; LLVM 23.1.2, X86.td, line 1628]

Steamroller (models 30h to 4Fh) gives each core a decoder, grows the instruction cache to 96 KB, adds the loop buffer, 176 vector registers, a 48-entry load queue, a 32-entry store queue and XSAVEOPT, and cuts the L2 latency to 19 cycles, but has three vector pipes instead of four. Agner Fog adds that it fixed the 256-bit stores and partial store forwarding, yet ran only one 128-bit addition a cycle for one thread. LLVM schedules it with its Piledriver model, which X86.td calls approximate. [AMD 47414, Sections 1.6.1, 1.6.7, 2.5.1, 2.5.3, 2.11 and 2.12, pages 23, 27, 28, 34, 39 to 41; Agner Fog, Sections 21.9 and 21.17, pages 224 and 230; LLVM 23.1.2, X86.td, line 2208]

Excavator (models 60h to 7Fh) has no AMD guide. Agner Fog reports AVX2, one decoder per module again, a 32 KB 8-way L1 data cache of 3 cycles, a 1 MB L2 of 17 cycles, AGLUs that run the common ALU operations and floating-point latencies one cycle shorter. LLVM adds AVX2, BMI2, MOVBE, RDRAND and MWAITX to Steamroller’s features for it and schedules it with its Zen 1 model, which X86.td calls approximate. [Agner Fog, Sections 21.3, 21.8, 21.9, 21.10 and 21.16, pages 221 to 225 and 228; LLVM 23.1.2, X86.td, line 1647; LLVM 23.1.2, X86.td, line 2212; GCC 14.2, common/config/i386/cpuinfo.h, line 210]

Zen drops FMA4, XOP, TBM and LWP, so code built for this family needs another path there; Zen 1 says what replaced them. [AMD 55723, Section 2.1, page 13]