Optimizing code for Intel Sandy Bridge

A length-changing prefix (a 66H operand-size override before a word immediate, mainly) costs 3 cycles in the predecoder, down from a 6-cycle penalty per 16-byte chunk on earlier cores; an address-size override and the Intel 64 REX prefix cost nothing, even though REX can change an instruction’s length. [Intel 356477, Section 2.1.1.1, page 2-4; Agner Fog, Section 9.2, page 125]

Sandy Bridge widens macro-fusion from Nehalem’s CMP and TEST to also fuse ADD, SUB, AND, INC and DEC with a following conditional branch, fusing when the first instruction’s first operand is a register and its second operand, if any, is an immediate, a register or memory that is not RIP-relative; the pair does not fuse only when the first instruction ends on the last byte of a 64-byte line and the branch begins the next line’s first byte. [Intel 356477, Section 2.1.1.1 (MacroFusion), page 2-5; Intel 248966, Section 3.4.2.2, Table 3-1, page 3-13]

Every micro-op in one way comes from one aligned 32-byte block of code; an instruction whose micro-ops number more than one is never split between two ways, an unconditional branch always ends a way, and one run through the MSROM fills a whole way alone. Code the op cache cannot hold this way still runs from the decoders. [Intel 356477, Section 2.1.1.2, page 2-6]

Once the front end switches to the decoders because the op cache missed, it does not return to the op cache until the following micro-op that is a branch, and Intel’s manual warns that frequent switches cost cycles; it does not say how many. [Intel 356477, Section 2.1.1.2, page 2-6]

The loop stream detector locks a loop into the micro-op queue and stops fetching, decoding and reading from any cache for it, so long as the loop reaches no further than eight 32-byte fetch blocks, has at most 28 micro-ops, takes no more than eight branches with none of them a CALL or RET, and keeps PUSH and POP balanced; a loop that overflows this is often still faster unrolled. [Intel 356477, Section 2.1.1.4, page 2-7]

The renamer removes NOP, VZEROUPPER, FXCHG and the zeroing idioms (XOR, SUB and PSUBx of a register with itself, PXOR and XORPS/XORPD likewise) from the pipeline at no cost, since their result needs no input; a so-called ones idiom such as cmpeqps xmm1, xmm1 still has to run, but waits for no source since its result does not depend on the data either. [Intel 356477, Section 2.1.2.1, pages 2-8 and 2-9]

A load folded into an arithmetic instruction, or a store, that addresses memory with an index register is split back into two micro-ops (one for the load or the store address, one for the rest) once it reaches the micro-op queue, using up rename and retirement bandwidth though not a decode slot; addressing by base, or base and displacement, avoids the split and keeps the pair fused. [Intel 356477, Section 2.1.1.4, page 2-7]

One branch prediction unit, using the branch’s address and the path that reached it, predicts conditional branches, direct and indirect calls and jumps, and returns; unlike the Core design before it, it has no predictor built specifically for loops. [Intel 356477, Section 2.1.1.3, page 2-7; Agner Fog, Section 3.7, page 30]

Neither source documents the branch target buffer’s size: Agner Fog could only say it looked bigger than Nehalem’s and could not tell whether it kept one level or two. A way holds at most two branches, and only when the first is conditional and both end in the same 64-byte line, so a third predicted branch in that line costs another entry and more cycles; up to four calls can be decoded from 16 bytes of code. [Agner Fog, Section 3.7, page 30; Intel 356477, Section 2.1.1.2, page 2-6]

The return stack buffer holds 16 near return addresses. [Agner Fog, Section 3.7, page 30]

Indirect calls and jumps predict through the same two-level, global-history predictor as conditional branches; returns do not use it. [Agner Fog, Section 3.7, page 30]

Every micro-op’s latency is one of a few fixed values tied to where it may run: 1 cycle on port 0, 1 or 5, 3 cycles only on port 1, 5 cycles only on port 0 (division and square root aside), so two micro-ops of different latency never compete for the same writeback bus in the same cycle, at the cost of a few micro-ops taking longer than their work strictly needs. [Agner Fog, Section 9.10, page 134]

Moving a result between a general-purpose register and a vector register, with MOVD or a flag test such as COMISS or PTEST, costs nothing extra; handing a float result to an integer shuffle or Boolean instruction such as PSHUFD or PADDD, or handing one back the other way, each add a cycle (PSHUFD right after ADDPS, say), though not every such pair does, and a load or store pays no such cost regardless of the data it carries. [Agner Fog, Section 9.10, page 134]

Agner Fog found the register-read-port limit that had bound every Intel core since the Pentium Pro gone here: his tests turned up no practical limit on how many registers one instruction could read at once. [Agner Fog, Section 9.9, page 131]

Reading a whole register after writing part of it (mov al, 1 then mov ebx, eax) costs an extra micro-op that joins the parts, generated, Agner Fog believes, when the reorder buffer is read; the same happens for the flags register, and for reading the flags right after a rotate. [Agner Fog, Section 9.11, page 135]

Separate multipliers serve general-purpose and vector registers: a general-purpose multiply runs on port 1 at a latency of 3 cycles, a vector one on port 0 at 5 cycles for any precision, and both run one operation a cycle. A 64-by-64 integer multiply with a 128-bit result has its low 64 bits ready in 3 cycles and its high 64 bits one cycle later, so code that uses both is best written to read the low half first. [Agner Fog, Section 9.10, page 131; Intel 248966, Section 13.2.3, page 13-1]

One divider, on port 0, serves integer division and also the square root and floating-point division operations, so only one of them runs at a time; LLVM’s model gives an 8-, 16-, 32- or 64-bit integer divide the same 25-cycle latency and a throughput of one every 10 cycles. [Agner Fog, Section 9.10, page 131; LLVM 23.1.2, X86SchedSandyBridge.td, line 154]

An LEA with one or two sources runs on port 1 or 5, doubling the throughput earlier cores gave it; a three-operand LEA (two registers plus a displacement, say) is limited to port 1 and needs two cycles there to read all its inputs. [Intel 248966, Section 3.5.1.2, page 3-21; LLVM 23.1.2, X86SchedSandyBridge.td, line 196]

256-bit AVX add and multiply run on the full-width units of ports 0 and 1; divide and square root still use the 128-bit units, so a 256-bit AVX divide or square root takes two passes through them. [Intel 248966, Chapter 15 intro, page 15-1]

ADDPD, MULPD, DIVPD and CVTPD2PS, and their AVX forms, no longer take a slow microcode assist for a denormal input or an underflowing result in double precision, unlike on earlier cores; the assist still costs about 140 to 150 cycles where it still applies, which FTZ and DAZ in MXCSR avoid at the price of IEEE 754 correctness. [Intel 356477, Section 2.1.3, page 2-9; Agner Fog, Section 9.10, page 134]

Once any YMM register’s upper half holds AVX data, the first legacy SSE instruction after it splits the whole YMM register file into a scratchpad, and the first AVX instruction after that joins it back together, not the per-register scheme Skylake later uses; Agner Fog measured about 70 cycles for each of those two switches. VZEROUPPER, which clears the upper halves and returns to the cheap state, is instead removed by the renamer and costs no execution cycle. [Intel 248966, Section 15.3, pages 15-7 and 15-8; Intel 356477, Section 2.1.2.1, page 2-8; Agner Fog, Section 9.12, page 135]

Moving data between a YMM register’s two 128-bit halves always costs one extra cycle: VINSERTF128, which does this, has a latency of 2 cycles against the 1 cycle of a shuffle or blend that stays within one half. [Agner Fog, Section 9.10, page 134]

An integer load reaches its register in 4 cycles with a base and a displacement under 2048; adding an index register, or a larger displacement, instead takes 5. A vector load of up to 128 bits takes one cycle more in each case, and a 256-bit AVX load always takes 7, whatever the addressing mode. [Intel 356477, Section 2.1.5.1, Table 2-8, page 2-13]

The L1 data cache is split into eight banks of 16 bytes each across every 128 bytes; a bank conflict recycles one of two same-cycle loads that land in the same bank at a different set, though two loads to the very same line and bank, such as a sequential pop, run together without one. [Intel 356477, Section 2.1.5.4, page 2-16; Agner Fog, Section 9.13, page 136]

A store forwards straight to a later load of the same address once the store is the last one to write there, from writeback memory, and covers every byte the load reads; neither may be a non-temporal access. Agner Fog found forwarding worked in more cases still, including most misaligned ones, so long as the load’s size does not exceed the store’s. [Intel 356477, Section 2.1.5.3, page 2-14; Agner Fog, Section 9.14, page 137]

A load that may run before an earlier store’s address is known is allowed to anyway, taking that store’s data as a guess, unless it crosses a 16-byte boundary without being part of an aligned 32-byte load, or is an unaligned 32-byte AVX load; such a load instead waits for every earlier store’s address to resolve. A guess that turns out wrong forces the load and everything after it to run again. [Intel 356477, Section 2.1.5.4, page 2-16]

A load whose address shares bits 11:0 with an earlier store in flight, 4 KB or a multiple of it away, pays about a 5-cycle penalty, more if it crosses two cache lines, since the processor cannot yet tell the addresses apart; aligning data to 32 bytes, or changing the offset between buffers, avoids it. [Intel 248966, Section 15.8, page 15-26]

Two hardware prefetchers feed the L1 data cache: one follows an ascending stream of recent accesses, the other tracks up to 2 KB strides per load instruction, forward or backward. Two more feed the L2 and last-level cache: a spatial prefetcher completes a line’s 128-byte pair, and a streamer that watches L1 traffic runs up to 20 lines ahead, adjusting how far by how many requests are already outstanding. [Intel 356477, Section 2.1.7, page 2-17]

Where Hyper-Threading runs two threads in a core, fetch and decode alternate evenly between them a cycle at a time, and register renaming does the same; the caches, TLBs, branch predictor, reorder buffer, reservation station and execution ports are instead shared competitively, so a thread that needs little of a resource leaves more for the other. [Agner Fog, Section 9.15, page 137]

Two threads help most when a lot of time goes to cache misses or mispredicted branches that would otherwise leave a core idle; when a shared resource is already the bottleneck for one thread, running two instead tends to evict each other’s cache and branch-predictor state and can run slower than one thread alone. [Agner Fog, Section 9.15, page 137]

Ivy Bridge adds three features: RDRAND, a hardware random number generator meeting NIST SP 800-90A; VCVTPH2PS and VCVTPS2PH, converting half-precision floats; and reading or writing the FS and GS base registers from any ring. [Intel 356477, Section 2.3, page 2-19]

Some register-to-register MOV instructions now complete during decode rather than reaching the scheduler at all, costing neither a rename slot nor an execution cycle; Intel’s manual does not say which forms qualify. [Intel 356477, Section 2.3, page 2-19]

The micro-op queue is split 28 entries to a logical processor on Sandy Bridge regardless of how many threads run; on Ivy Bridge, a core with only one active logical processor gives that thread all 56 entries, so the loop stream detector can then hold a longer loop. [Intel 356477, Section 2.3, page 2-19]

Sandy Bridge decodes the multi-byte NOP (0F 1F) only on the first of its four decoders, while a NOP padded with 66H prefixes decodes on any of them; Ivy Bridge removes that limit, decoding either form on all four at up to four a cycle. [Agner Fog, Section 9.2, page 125]

Ivy Bridge inserts the partial-register-joining micro-op only when a high-byte register (AH, BH, CH or DH) was written, not for every partial write Sandy Bridge catches; it also drops the extra micro-op Sandy Bridge adds for reading the flags right after a rotate. [Agner Fog, Section 9.11, page 135]

A next-page prefetcher, triggered by a sequential access nearing a 4 KB boundary in either direction, is new on Ivy Bridge. [Intel 356477, Section 2.3, page 2-19]

256-bit packed divide and square root run faster on Ivy Bridge than on Sandy Bridge, and so do ROL and ROR, though Intel’s manual gives no cycle counts for either. [Intel 356477, Section 2.3, page 2-20]

Agner Fog found Ivy Bridge’s software prefetch instructions wasteful when repeated on data already cached: a repeated PREFETCHNTA to the same address managed only one prefetch every 43 cycles there, against two a cycle on Sandy Bridge. [Agner Fog, Section 9.13, page 136]

Ivy Bridge is the first core with enhanced REP MOVSB and STOSB, flagged by CPUID leaf 7’s EBX bit 9, which make a byte-granularity REP string fast as well as flexible for memory copies and fills; on Ivy Bridge itself, though, Intel’s manual notes they may still not reach the throughput of a hand-written 128-bit or 256-bit AVX copy. [Intel 248966, Sections 3.7.6 and 3.7.6.2, page 3-63]