Optimizing code for Intel Haswell

Prefixes cost nothing in the decoders however many there are, except one: an arithmetic or logic instruction whose operand size override makes its immediate 16 bits wide, such as add ax, 1234, loses 2 to 3 cycles wherever it lies. A MOV with the same prefix decodes without that cost. [Agner Fog, Section 10.2, page 140]

The loop stream detector replays loops of up to 56 micro-ops straight out of the queue that feeds renaming. With Hyper-Threading the two threads share that one 56-entry queue, where Sandy Bridge gave each thread 28 entries of its own. [Intel 356477, Section 1.2, page 1-2]

CMP, ADD and SUB fuse with a following conditional jump that tests carry, zero, or a signed or unsigned comparison; INC and DEC only with one that tests zero or a signed comparison; TEST and AND with any. The first instruction may take a register, an immediate or a memory source, but not memory and an immediate together, not a memory destination and not a RIP-relative address. Two pairs can fuse in the same cycle, and a pair may now straddle a 16-byte boundary. [Intel 248966, Section 3.4.2.2, Table 3-2, pages 3-13 and 3-14; Agner Fog, Section 10.6, pages 141 and 142]

A micro-fused instruction is split in two in the micro-op queue only when it counts more than three sources: the base and the index of its address (the base counts even when absent, once there is an index), each register it reads and each it writes, the flags included, and twice a register both read and written. So cmp ecx, [rdx+8] (ecx, the flags and rdx) stays fused, while cmp ecx, [rdx+rsi*4] (four sources) is split. This is looser than Sandy Bridge, which splits every address with an index. [Intel 356477, Section 1.4.2, Table 1-6, page 1-6]

XOR, SUB, PXOR, XORPS, XORPD and every PSUB and PCMPGT form of a 32-bit, 64-bit, XMM or YMM register with itself, VEX forms included, zero it at renaming without an execution unit, up to four a cycle. An 8-bit or 16-bit register gets no such treatment, an MMX register is zeroed without waiting but still on a unit, PCMPEQ of a register with itself sets all ones without waiting for it but on a unit, and CMP, SBB, ANDN, PANDN, ANDNPS, ANDNPD, CMPEQPS and CMPEQPD of a register with itself get nothing special. [Agner Fog, Section 10.8, pages 142 and 143; Intel 356477, Section 1.3, page 1-3; LLVM 23.1.2, X86SchedHaswell.td, line 1766]

Most moves between 32-bit or 64-bit general-purpose registers or between XMM or YMM registers are removed at renaming, with zero latency and no port, more than 80% of the time by Agner Fog’s count, chains of moves included; they still take decode bandwidth. Never removed are moves of 8-bit, 16-bit and MMX registers, copies of a register onto itself and, unlike on Ivy Bridge, MOVZX, though mov eax, ebx and vmovaps xmm0, xmm1 are, even though they clear the upper bits. [Agner Fog, Section 10.8, page 143; Intel 356477, Section 1.3, page 1-3]

Intel does not document the branch predictor, which Agner Fog finds redesigned from Sandy Bridge’s: it combines tables of local and global history, predicts long repeating patterns of a conditional jump with no limit he could find (a study he cites puts its history at 29 branches or more) and predicts loops of up to about 32 iterations, nested ones reasonably. Indirect jumps and calls also predict well, and the return stack holds 16 entries. [Agner Fog, Section 3.8, page 31]

A jump or a predicted-taken branch costs one cycle for up to 128 branches that lie at least 16 bytes apart, and two cycles when more than one branch shares 16 bytes of code, while two branches predicted not taken can pass in one cycle. Code laid out so that the frequent path falls through goes faster. [Agner Fog, Section 3.8, page 31; Agner Fog, Section 10.14, page 150]

Two ports run branches: port 6, the main branch unit, and port 0, a second one. A fused compare and branch also runs on port 0 or 6 as one micro-op. [Intel 356477, Section 1.3.1, Table 1-1, page 1-3; Agner Fog, Section 10.6, page 141]

Intel’s table puts a cycle of delay, or an extra micro-op, on every value passed between vector integer and floating-point instructions, and between general-purpose and vector registers. [Intel 356477, Section 1.3.1, Table 1-3, pages 1-4 and 1-5]

Writing AL and then reading EAX costs nothing visible, unlike on Sandy Bridge; writing AH, BH, CH or DH and then reading the wider register still adds a merging micro-op and a cycle of latency. The flags behave the same way: a branch or CMOV that reads flags two instructions set separately (INC then JBE, or BT then CMOVBE) gets an extra micro-op, while no flag read after a shift or a rotate costs anything. [Agner Fog, Section 10.10, pages 146 and 147]

An LEA with one or two inputs runs on port 1 or 5 in a cycle. LLVM’s tuning for both cores avoids the slow kinds: LEA with base, index and displacement together, or with RBP, EBP or R13 as base beside an index; its Haswell model notes that these run only on port 1 and need two cycles there to read their inputs. [Intel 356477, Section 1.3.1, Table 1-1, page 1-3; LLVM 23.1.2, X86.td, line 578; LLVM 23.1.2, X86.td, line 1068; LLVM 23.1.2, X86SchedHaswell.td, line 200]

LLVM’s tuning marks POPCNT, LZCNT and TZCNT on both cores as waiting for whatever last wrote the register they overwrite, though they never use its value, as on Skylake. Its models run all three on port 1 with a latency of 3 cycles. [LLVM 23.1.2, X86.td, line 1075; LLVM 23.1.2, X86.td, line 590; LLVM 23.1.2, X86SchedHaswell.td, line 208; LLVM 23.1.2, X86SchedHaswell.td, line 210]

A 64-bit integer divide is much slower than a 32-bit one, so LLVM, tuning for Haswell or Broadwell, uses the 32-bit divide when both 64-bit operands turn out positive and below 2^32. [LLVM 23.1.2, X86.td, line 539; LLVM 23.1.2, X86.td, line 1069]

AESENC, AESENCLAST, AESDEC and AESDECLAST become one micro-op each, with a little less latency than before, which helps the modes that run several blocks at once, such as CBC decryption, and code that encrypts several buffers side by side. LLVM’s models run them on port 5 with a latency of 7 cycles. [Intel 248966, Section 6.10.3.1, page 6-49; LLVM 23.1.2, X86SchedHaswell.td, line 574]

Two FMA units, on ports 0 and 1, take a 256-bit FMA each a cycle with a latency of 5 cycles, so one core finishes 16 single-precision or 8 double-precision multiply-adds a cycle, 32 or 16 floating-point operations: twice Sandy Bridge’s peak. [Intel 248966, Section 15.15, page 15-52; Intel 356477, Section 1.3, page 1-3; Agner Fog, Section 10.9, page 144; LLVM 23.1.2, X86SchedHaswell.td, line 324]

Floating-point addition has a single unit, on port 1, at one a cycle with a latency of 3, while multiplication and FMA have two, so code made mostly of additions runs at half the rate of code made of multiplications. Agner Fog suggests turning some additions into FMAs that multiply by 1.0 to use both units, which, at 5 cycles each, needs up to 10 independent accumulators to keep them busy. [Agner Fog, Section 10.14, page 150; Intel 356477, Section 1.3.1, Table 1-1, page 1-3; LLVM 23.1.2, X86SchedHaswell.td, line 265]

Every instruction that can move data from one 128-bit half of a YMM register to the other takes 3 cycles, even when it does not: VEXTRACTI128 of the low half still takes 3, where VMOVDQA, which copies the same 128 bits, takes 0 or 1. Moves that cannot cross between the halves take 1 cycle. [Agner Fog, Section 10.9, pages 145 and 146]

Only port 5 shuffles, packs and unpacks, where Sandy Bridge had two shuffle units. In Intel’s example, 128-bit SIMD code whose share of port-5 micro-ops rose from about a third to almost half ran about 25% slower on Haswell than on Sandy Bridge, while its 256-bit AVX2 version beat both. [Intel 248966, Section 15.16, pages 15-57 to 15-59]

From Haswell on, one 32-byte load is better than two 16-byte halves, which Sandy Bridge preferred, and 32-byte loads no longer suffer more than 16-byte ones from line replacements in the L1. A 32-byte load that would cross a cache line, though, is still better split into two loads from 16-byte aligned addresses. [Intel 248966, Section 15.6.2, Example 15-12, page 15-24; Intel 248966, Section 15.7, page 15-25; Intel 248966, Section 15.16.3.5, page 15-63]

An operation on normal numbers that gives a subnormal result, or a multiplication of a normal by a subnormal number, costs about 124 cycles; adding a subnormal number costs nothing, nor do overflow, infinities or NaNs. Setting both FTZ and DAZ in MXCSR avoids the cost. [Agner Fog, Section 10.9, page 146]

Mixing SSE with 256-bit AVX code works as on Sandy Bridge: the switches save and restore the upper halves of all the YMM registers, each costing on the order of an XSAVE in Intel’s table and about 70 cycles in Agner Fog’s tests, and VZEROUPPER at the end of AVX code avoids them. LLVM inserts VZEROUPPER for both cores and models it as four micro-ops that use no port. [Intel 248966, Section 15.3, Table 15-3, page 15-10; Agner Fog, Section 10.9, page 146; LLVM 23.1.2, X86.td, line 1078; LLVM 23.1.2, X86SchedHaswell.td, line 1358]

Apart from division, square root, AES and the SSE4.2 string compares, the vector units are 256 bits wide. LLVM’s Haswell model gives a 256-bit VDIVPS or VSQRTPS 21 cycles, one every 14, and VDIVPD or VSQRTPD 35 cycles, one every 28; Intel’s table for Broadwell gives 17 and 10 for VDIVPS, 21 and 14 for VSQRTPS, 23 and 16 for VDIVPD and 35 and 28 for VSQRTPD. [Intel 356477, Section 1.3, page 1-3; Agner Fog, Section 10.9, page 144; LLVM 23.1.2, X86SchedHaswell.td, line 295; LLVM 23.1.2, X86SchedHaswell.td, line 318; Intel 248966, Section 17.25.2, Table 17-13, page 17-79]

Haswell is the first core with gathers, and each one runs as many micro-ops; Broadwell runs them much faster. LLVM’s models give an eight-element VPGATHERDD 34 micro-ops and 22 cycles on Haswell, 14 micro-ops and 21 cycles on Broadwell, against five on Skylake. [Intel 248966, Section 15.16.4, page 15-70; Intel 356477, Section 1.6, page 1-7; Agner Fog, Section 10.14, page 150; LLVM 23.1.2, X86SchedHaswell.td, line 1720; LLVM 23.1.2, X86SchedBroadwell.td, line 1435]

For Haswell, Intel recommends plain vector loads, shuffles and blends over a gather when the elements are consecutive, fewer than four, a short stride apart, the columns of a transpose, or used more than once, and a gather when the mask of four or more elements depends on the data or the indices of eight are computed in a vector; the vectors chapter has the reasons. [Intel 248966, Section 15.16.4, Examples 15-43 and 15-44, pages 15-70 and 15-71]

LLVM’s tuning for Haswell and Broadwell leaves out its fast-gather flag, which its sources reserve for Skylake client and AVX-512 cores because gathers cost too much on Haswell; the vectors chapter says what the flag decides. [LLVM 23.1.2, X86.td, line 785; LLVM 23.1.2, X86.td, line 1067]

Loads and stores move 256 bits at once, twice what Sandy Bridge managed, and the buffers behind them have entries of that width, so two 32-byte loads and a 32-byte store can complete each cycle and a copy can move 32 bytes a cycle from L1 to L1; the L2 delivers far less. Code tuned for Sandy Bridge whose loads and stores were the limit gains without being rebuilt. [Intel 248966, Section 3.6.1, page 3-41; Intel 248966, Section 15.16.3, pages 15-60 and 15-61; Agner Fog, Section 10.14, page 150]

A store has its address computed on port 2, 3 or 7, and port 7, a simple address unit, serves stores alone, leaving ports 2 and 3 to the loads more often. [Agner Fog, Section 10.9, page 145; Intel 356477, Section 1.3.1, Table 1-1, page 1-3; LLVM 23.1.2, X86SchedHaswell.td, line 123]

The L1 data cache bank conflicts of Sandy Bridge are gone: two loads can always proceed in the same cycle. A load from an address a multiple of 4 KB away from an earlier store still waits for it, though, as if the two overlapped. [Intel 248966, Section 3.6.1.3, page 3-43; Agner Fog, Section 10.11, page 148]

A store forwards to a load of the same size and address at any alignment up to 64 bits and when aligned at 128 and 256 bits, and to a smaller load inside it when the store is 64 bits or less, when the load takes a naturally aligned half or quarter of an aligned store, or when it reads 64 bits or less of an aligned wider store without crossing an 8-byte boundary; crossing a cache line adds 2 cycles. A load that is wider than the store, overlaps it in part, or crosses the middle of a 128-bit store or a 64-bit boundary of a 256-bit one waits about 10 cycles more, and about 50 after a 128-bit or 256-bit store that is not 16-byte aligned. [Agner Fog, Section 10.12, pages 148 and 149]

rep movsb copies more than 20 bytes a cycle when the copy is long, using the 256-bit store path, but loses to SIMD code below a few hundred bytes; rep stosb fills up to 32 bytes a cycle when the count is large. A loop of aligned 32-byte vector stores fills close to 30 bytes a cycle, limited by port 4, which takes each store’s data. Agner Fog found that REP MOVS reaches full speed only with both source and destination 32-byte aligned. [Intel 248966, Sections 15.16.3.2 and 15.16.3.3, page 15-62; Agner Fog, Section 10.11, page 148]

Haswell is Intel’s first core with TSX. A transaction keeps track of what it reads and writes in the 8-way L1 data cache, so writing nine lines that map to one set always aborts it and fewer may do so as well; with Hyper-Threading the other thread of the core uses the same cache and shrinks what a transaction can hold. [Intel 356477, Section 8.5, page 8-27; Intel 356477, Section 8.2.4.2, page 8-8]

XBEGIN and XEND are meant to cost about what a LOCK-prefixed instruction does, not a load, and each transaction adds a mostly fixed overhead that shows on Haswell when a tiny critical section repeats in a tight loop; touching the lines the transaction uses early in it hides part of the cost. Broadwell lowers the cost of a commit and keeps better track of reads evicted from the L1, so fewer transactions abort on conflicts that are not real. [Intel 356477, Section 8.5, page 8-27; Intel 356477, Section 8.2.4.2, page 8-8]

With two threads, each gets half of the op cache and the decoders serve them in turn, unless one is idle; the 56-entry micro-op queue, and with it the loop buffer, is shared between them rather than split. [Intel 356477, Section 1.2, page 1-2; Agner Fog, Section 10.13, page 149]

The other resources are shared as on Sandy Bridge, so a second thread helps only when neither saturates them: code meant to run over two instructions a cycle, or whose data barely fits the caches, gains nothing, and software cannot give one thread priority over the other. [Agner Fog, Section 10.13, page 149]

The Xeon E5 v3 keeps the same core and widens the uncore: up to 18 cores on two rings around the L3, up to two memory controllers, 40 lanes of PCI Express 3.0 and several QPI links for systems of more than one socket. [Intel 356477, Section 1.5, pages 1-6 and 1-7]

Broadwell’s server parts pair the 256 KB L2 with up to 2.5 MB of L3 per core that holds copies of every L2’s lines, and their L1 sustains 93 bytes a cycle of its peak 96; Skylake-SP changed this balance. [Intel 248966, Section 2.6.1, Table 2-11 and Figure 2-6, pages 2-27 and 2-28]

Broadwell’s Xeons already pick each core’s frequency by the mix of vector instructions it runs, though less than Skylake-SP, which has 512-bit instructions to draw more power (Skylake). [Intel 248966, Section 2.6.3, page 2-29]

Broadwell cuts the latency of floating-point multiplication from 5 cycles to 3, for SSE and AVX code and, by Intel’s manual, x87 as well; FMA stays at 5. LLVM’s Broadwell model has the same numbers for SSE and AVX. [Intel 356477, Section 1.6, page 1-7; Agner Fog, Section 10.9, Table 10.1, pages 143 and 144; LLVM 23.1.2, X86SchedBroadwell.td, line 282; LLVM 23.1.2, X86SchedBroadwell.td, line 284; LLVM 23.1.2, X86SchedBroadwell.td, line 324]

A micro-op of Haswell’s reads three inputs only for FMA, so ADC, SBB and CMOV, which read two registers and the flags, are two micro-ops each; Broadwell makes them one, though stores and blends stay at two. LLVM’s models agree: ADC and CMOV from registers are two micro-ops with 2 cycles of latency on Haswell and one with 1 on Broadwell, except CMOVA and CMOVBE, which read two flags and take one micro-op more on each. [Agner Fog, Section 10.9, pages 144 and 145; LLVM 23.1.2, X86SchedHaswell.td, line 138; LLVM 23.1.2, X86SchedBroadwell.td, line 136; LLVM 23.1.2, X86SchedHaswell.td, line 183; LLVM 23.1.2, X86SchedBroadwell.td, line 181; LLVM 23.1.2, X86SchedHaswell.td, line 1867; LLVM 23.1.2, X86SchedBroadwell.td, line 1603]

PCLMULQDQ becomes a single micro-op with lower latency and higher throughput: LLVM’s models give it three micro-ops and 11 cycles on Haswell, one and 5 on Broadwell. [Intel 356477, Section 1.6, page 1-7; LLVM 23.1.2, X86SchedHaswell.td, line 608; LLVM 23.1.2, X86SchedBroadwell.td, line 606]

Broadwell adds ADCX and ADOX, RDSEED and PREFETCHW, which -march=broadwell enables in LLVM; code built that way can fault on Haswell, which has none of them. [LLVM 23.1.2, X86.td, line 1086; Intel 356477, Section 1.3.1, Table 1-2, page 1-4; Intel 248966, Section 9.3.2, page 9-4]