Optimizing code for Intel Skylake

Fetch brings 16 bytes of code a cycle into four decoders, of which only the first takes instructions of more than one micro-op (up to four); a run of two-micro-op instructions leaves the other three idle. Any number of prefixes decodes in one cycle, except that an arithmetic instruction with a 16-bit immediate (add ax, 1234) loses 2 to 3 cycles to its length-changing operand-size prefix; a mov does not. [Agner Fog, Section 11.2, pages 152 and 153]

The micro-op cache spends at most three of its 6-micro-op lines on each aligned 32-byte block of code and averages no more than 4 micro-ops or 32 bytes a cycle. A hot loop of up to about a thousand instructions runs from it at full speed; beyond that the 16-byte fetch limits code with long instructions, so unrolling loops further than needed costs speed. [Agner Fog, Sections 11.3 and 11.15, pages 153 and 161]

On the client parts, the 64-entry micro-op queue of each thread replays small loops itself, with no limit from instruction length. Skylake-SP has this loop stream detector disabled, so its small loops run from the micro-op cache. [Intel 248966, Section 2.7.1, page 2-34; Intel 248966, Section 2.6, page 2-26; Agner Fog, Section 11.4, page 153]

test and and fuse with any following conditional jump; cmp, add and sub with the jumps on equality and on the unsigned and signed comparisons, but not with js, jp or jo and their inverses; inc and dec only with those on equality and the signed comparisons. The pair becomes one micro-op that runs on port 0 or 6, even across a 16-byte boundary and two pairs a cycle. The first instruction may have an immediate or a memory source but not both, and no RIP-relative or memory destination operand; it is best placed right before its jump. [Agner Fog, Section 11.6, page 154; Agner Fog, Section 9.6, Table 9.2, page 129; Intel 248966, Section 3.4.2.2, Tables 3-1 and 3-2, pages 3-13 and 3-14; LLVM 23.1.2, MCTargetDesc/X86BaseInfo.h, line 290]

Under rare conditions a jump that crosses a 64-byte line could make these cores misbehave (erratum SKX102). The microcode update that fixes it keeps out of the micro-op cache every jump that crosses a 32-byte boundary or ends exactly on one: conditional jumps and macro-fused pairs, direct and indirect jumps, every call and every return. Such jumps then come from the legacy decoders, and the front end switches to them and back each time one runs. [Intel 341810, Sections 1.1 and 2.1, pages 5 and 6]

Intel lists Skylake on the desktop, in laptops and in servers (Xeon Scalable, W, D and E3 v5, Core X), Kaby Lake, Coffee Lake, Whiskey Lake, Amber Lake, the U-series Comet Lake and Cascade Lake. Cooper Lake, released after the list, is not on it, nor is any later core. [Intel 341810, Section 4.0, pages 13 to 15]

Intel measured losses of 0 to 4% on common benchmarks with the update, and more in parts of them. The loss comes from switching out of the micro-op cache and from the slower legacy decoders behind it, so code that takes more than 40% of its micro-ops from the cache stands to lose most. [Intel 341810, Sections 2.2 and 2.3, pages 6 to 8]

DSB2MITE_SWITCHES.PENALTY_CYCLES over core cycles gives the share lost to the switches, IDQ.DSB_UOPS over all delivered micro-ops the share from the micro-op cache, and the precise event FRONTEND_RETIRED.DSB_MISS_PS points at the branches that miss it. [Intel 341810, Section 2.3, pages 7 and 8]

Rebuilding with the GNU assembler’s -mbranches-within-32B-boundaries pads earlier instructions with up to five segment prefixes, or with NOPs, until no conditional jump, fused pair or unconditional jump touches a 32-byte boundary; Intel found this recovers most of the loss. LLVM’s integrated assembler has -x86-branches-within-32B-boundaries for the same three kinds, padding with NOPs unless -x86-pad-max-prefix-size allows prefixes. Silvermont and Goldmont are slow to decode an instruction carrying over three or four prefixes, so the option belongs in builds tuned for these cores, not in generic ones. [Intel 341810, Sections 3.0 and 3.1.1, page 11; LLVM 23.1.2, MCTargetDesc/X86AsmBackend.cpp, line 100; LLVM 23.1.2, MCTargetDesc/X86AsmBackend.cpp, line 142]

Intel does not document the predictor. Agner Fog found it predicts long repeating patterns and loops of up to about 32 iterations well, and indirect jumps well; the return stack holds 16 entries, against 22 on Ice Lake. [Agner Fog, Section 3.8, page 31]

Taken branches run at one or one every two cycles, not-taken ones at up to two a cycle. The fast rate holds for up to 128 branches at no more than one per 16 bytes of code, so hot code should be laid out with the common path falling through. [Agner Fog, Section 3.8, page 31; Agner Fog, Section 11.15, page 162]

Ports 0, 1, 5 and 6 each have an integer ALU, so simple integer instructions run four a cycle; shifts and branches use ports 0 and 6, and the bit manipulation instructions such as andn and bzhi two ports. [Intel 248966, Section 2.7.2, Tables 2-13 and 2-14, pages 2-34 and 2-35; Agner Fog, Section 11.9, page 155]

An lea with a base, an index and a displacement runs on port 1 only; the simpler forms run on ports 1 and 5. LLVM’s tuning for these cores counts these forms, and those with RBP or R13 as base and an index, as slow and avoids them. [Intel 248966, Sections 2.6 and 2.7.2, Table 2-13, pages 2-26 and 2-34; LLVM 23.1.2, X86.td, line 581; LLVM 23.1.2, X86.td, line 1100]

xor, sub and the vector pxor, xorps, pandn, psub and pcmpgt of a register with itself zero it without an execution unit, four a cycle, for 32-bit and 64-bit and all vector registers. Most register moves of these sizes are removed when renamed, but not movzx, not moves of 8 or 16 bits and not a register moved to itself. [Agner Fog, Section 11.8, pages 154 and 155]

Port 1 alone runs integer multiplication, PDEP and PEXT, popcnt, lzcnt, tzcnt, bsf, bsr and the double shifts, with a latency of three cycles and one a cycle in LLVM’s models, as on Golden Cove. [Intel 248966, Section 2.7.2, Table 2-14, page 2-35; LLVM 23.1.2, X86SchedSkylakeClient.td, line 192; LLVM 23.1.2, X86SchedSkylakeClient.td, line 783; LLVM 23.1.2, X86SchedSkylakeClient.td, line 201]

popcnt waits for the old value of its destination register, which it overwrites; compilers tuning for these cores zero the destination first to break the chain. [LLVM 23.1.2, X86.td, line 590; LLVM 23.1.2, X86.td, line 1108]

A MUL of 64 by 64 bits into RDX:RAX is two micro-ops on ports 1 and 5 and takes four cycles in LLVM’s models, against three for an IMUL that keeps only the low half. [LLVM 23.1.2, X86SchedSkylakeClient.td, line 134; LLVM 23.1.2, X86SchedSkylakeClient.td, line 137]

Integer division is slow: LLVM’s models give a 64-bit DIV 76 cycles and 32 micro-ops and a 64-bit IDIV 102 cycles, against 18 on Golden Cove. Compilers tuning for these cores test whether both operands fit in 32 bits and then use the much faster 32-bit division. [LLVM 23.1.2, X86SchedSkylakeClient.td, line 153; LLVM 23.1.2, X86SchedSkylakeClient.td, line 161; LLVM 23.1.2, X86.td, line 541; LLVM 23.1.2, X86.td, line 1101]

Writing part of a register no longer stalls: the core merges AL, AH or AX into RAX by itself. Operations on AH, BH, CH and DH run on ports 1 and 5 with a latency of one cycle, but what depends on them waits one cycle more; on Golden Cove they take three cycles on port 1. [Intel 248966, Section 3.5.2.3, page 3-32]

Floating-point addition, multiplication and FMA share two units on ports 0 and 1, all with a latency of four cycles, so up to two of them start each cycle. A chain of dependent ones is limited by latency: unrolling with several accumulators, at least eight for two units of four cycles, keeps both busy. [Intel 248966, Sections 15.15 and 15.15.1, pages 15-52 to 15-54; LLVM 23.1.2, X86SchedSkylakeClient.td, line 256; LLVM 23.1.2, X86SchedSkylakeClient.td, line 315]

Unlike Golden Cove, which adds in two or three cycles, an addition here takes as long as an FMA, so folding an addition into an FMA costs no latency. [Intel 248966, Section 15.15.1, page 15-52; Agner Fog, Section 11.15, page 162]

Shuffles and permutes run on port 5 only, one a cycle; those that move data between 128-bit lanes, such as VPERMD, take three cycles. Vector shifts by an immediate or a vector of counts run on ports 0 and 1, so a shift can stand in for a shuffle when port 5 is the bottleneck, which LLVM’s tuning for the servers does. [Intel 248966, Section 2.7.2, Table 2-14, page 2-35; Intel 248966, Section 15.15.2, page 15-54; Agner Fog, Section 11.9, Table 11.1, page 156; LLVM 23.1.2, X86SchedSkylakeClient.td, line 611; LLVM 23.1.2, X86.td, line 693; LLVM 23.1.2, X86.td, line 1143]

Mixing legacy SSE with AVX no longer costs a switch of the whole register state. Instead, while the upper bits of any register hold data, every SSE instruction depends on its destination’s old value and adds a micro-op to merge them. vzeroupper after AVX code avoids it, as does keeping that code in registers 16 to 31, which only AVX-512 encodings reach. The metric of UOPS_ISSUED.VECTOR_WIDTH_MISMATCH over all issued micro-ops shows it; Intel suggests a look above 5%. Golden Cove runs a slow microcode assist instead. [Intel 248966, Section 15.3, pages 15-8 to 15-10; Intel 248966, Section 22.1.11.2, page 22-7; Agner Fog, Section 11.10, pages 158 and 159]

The upper halves of the vector units are powered off while unused. The first 256-bit instruction takes 150 to 250 cycles, and then 256-bit code runs about 4.5 times slower, on the lower half twice, for some 56,000 cycles until they are on; they turn off again about 2.7 million cycles after the last such instruction. The same holds for 512-bit code. A dummy instruction on a YMM or ZMM register well ahead of a critical loop wakes them. [Agner Fog, Section 11.9, page 158]

An operation on normal numbers that gives a subnormal result, and a multiplication or division with a subnormal input, costs about 129 cycles of microcode; additions do not. Setting both FTZ and DAZ in MXCSR avoids it, as on Zen 4. [Agner Fog, Section 11.9, page 158]

Division and square root are pipelined, and 256-bit forms have the latency of 128-bit ones: LLVM’s models give DIVPS 11 cycles, SQRTPS 12, DIVPD 14 and SQRTPD 18. A reciprocal estimate with a Newton-Raphson step pays mainly when the code does little but divide. [Intel 248966, Sections 15.12 and 15.12.1, pages 15-38 to 15-40; LLVM 23.1.2, X86SchedSkylakeClient.td, line 286; LLVM 23.1.2, X86SchedSkylakeClient.td, line 290; LLVM 23.1.2, X86SchedSkylakeClient.td, line 295; LLVM 23.1.2, X86SchedSkylakeClient.td, line 299]

Gathers are faster than on Broadwell and beat a sequence of scalar loads unless they fetch only two elements or the data comes from memory rather than the caches. LLVM’s models give an eight-element VPGATHERDD five micro-ops and 22 cycles. [Intel 248966, Section 15.16.4, pages 15-71 and 15-73; LLVM 23.1.2, X86SchedSkylakeClient.td, line 1449]

Many MMX instructions, among them the saturating additions and subtractions, comparisons, pmaxsw, pminsw, pavgb and pabsb, run on one port at half Broadwell’s rate. Old MMX code gains from a rewrite in SSE2 or AVX2. [Intel 248966, Section 15.16.5, page 15-76]

Only the server cores and the Xeon W and Core X parts built from them run AVX-512 (F, CD, BW, DQ and VL); the client parts stop at AVX2 and FMA. Cascade Lake adds AVX-512 VNNI and Cooper Lake AVX-512 BF16. [Agner Fog, Chapter 11, page 152; Intel 248966, Chapter 17, Table 17-1, page 17-1; LLVM 23.1.2, X86.td, line 1118; LLVM 23.1.2, X86.td, line 1152; LLVM 23.1.2, X86.td, line 1158]

The 256-bit units of ports 0 and 1 join to run a 512-bit operation through port 0, beside a 512-bit unit on port 5. While the scheduler holds any 512-bit micro-op, vector operations of every width go to ports 0 and 5 only, so a few ZMM instructions in YMM code take port 1 away from it and may slow it. Port 1 still runs scalar work meanwhile. [Intel 248966, Section 17.20, pages 17-66 and 17-67; Intel 248966, Section 2.6, page 2-26; Agner Fog, Section 11.9, page 156]

Some parts have a second 512-bit FMA unit on port 5 and run two 512-bit FMAs, multiplications or additions a cycle; the others run one, on port 0, no more than two 256-bit ones. Agner Fog found that early Skylake-X parts with fewer than ten cores lack it, and that the CPUID signature does not tell them apart: Intel’s manual gives a timing test (Example 17-30) that does. Golden Cove servers and Zen 5 run two 512-bit FMAs a cycle, Zen 4 one. [Intel 248966, Section 17.21, pages 17-67 and 17-68; Agner Fog, Section 11.9, page 156]

A 512-bit FMA on port 0 takes four cycles and one on port 5 six, and forwarding between them adds between minus two and one, so the latency of 512-bit FMA, multiplication, addition and conversion code is four to seven cycles. [Intel 248966, Section 17.18, page 17-64]

The servers cap each core’s turbo frequency at one of three levels set by what it runs: the highest for scalar, SSE, 128-bit AVX and AVX2 without floating-point or multiply work; a middle one for AVX2 floating-point, multiplication and FMA and for AVX-512 without them; the lowest for AVX-512 floating-point, multiplication and FMA. Each cap is lower the more cores run at that level, and CORE_POWER.LVL0_TURBO_LICENSE to LVL2 count the cycles a core spends under each. [Intel 248966, Section 2.6.3, Table 2-12, pages 2-29 and 2-30]

A core that needs a lower level runs with reduced throughput for up to 500 microseconds until it is granted, and returns to a higher one only after some 2 ms without such code. A mispredicted path can trigger the change, and so can a dense mix of lighter instructions: on a Xeon Platinum 8180, 110 light 512-bit and 20 heavy 256-bit instructions within 65 cycles did. Short bursts of AVX-512 in otherwise scalar code can thus slow the code around them. [Intel 248966, Section 2.6.3, page 2-31]

Whether 512-bit vectors pay depends on the program, so GCC and LLVM prefer 256-bit vectors when tuning for these cores even where AVX-512 is enabled, and Intel advises building with 256 and with 512 (-mprefer-vector-width=512) and measuring; it saw 1.3 to 1.5 times the speed of AVX2 in deep learning code dominated by 512-bit FMAs. On Zen 4 and Zen 5 AMD recommends 512-bit vectors. [Intel 248966, Section 2.6.3, pages 2-31 and 2-32; LLVM 23.1.2, X86.td, line 1139]

Masking costs nothing in most instructions, but a masked load adds a micro-op and a cycle, a masked store about 10 cycles of latency, and a masked move between registers is never removed by renaming. Merge masking depends on the old destination; zero masking does not and should be used where zeros are fine. [Agner Fog, Section 11.9, page 157]

A broadcast of a 32-bit or 64-bit element from memory runs on the load ports, sparing port 5; a 16-bit one still goes through port 5, so it gains nothing from the memory form. [Intel 248966, Section 17.9.2, pages 17-34 and 17-35]

Two loads and one store a cycle, each up to 512 bits wide where AVX-512 exists, with 72 load and 56 store buffers. A store whose address goes to port 2 or 3 instead of 7 takes a load slot. Golden Cove loads three a cycle and stores two. [Agner Fog, Sections 11.9 and 11.12, pages 157 and 160; Intel 248966, Section 3.6.1, page 3-41]

The client L1 data cache peaks at 96 bytes a cycle and sustains about 81; Skylake-SP’s peaks at 192 and sustains about 133. The L2 delivers at most 64 bytes a cycle, about 29 sustained on the client parts and 52 on the servers. [Intel 248966, Section 2.7.3, Table 2-16, page 2-36; Intel 248966, Section 2.6.1.3, Table 2-11, pages 2-27 and 2-28]

Bank conflicts are gone, but a load from an address 4 KB, or a multiple of it, away from a pending store still waits for it as if they overlapped. [Agner Fog, Section 11.12, page 160]

A store forwards to a load of the same address in 4 cycles for 32-bit and 64-bit data and 5 for other sizes. Forwarding costs up to 3 cycles more when a store of 128 bits or more is misaligned and 4 to 5 more across a cache line; a load wider than the store, or covering bytes it did not write, waits about 11 cycles. [Agner Fog, Section 11.13, pages 160 and 161]

Loads after a vmaskmov store no longer wait for its mask to be known. A masked store that writes nothing to an invalid address still needs a microcode assist. [Intel 248966, Section 15.9.1, page 15-27]

Skylake-SP quadrupled the private L2 and shrank the L3 per core from 2.5 MB to 1.375 MB, which no longer keeps copies of the L2’s lines. Code that sizes its blocks or decides between cached and streaming stores from the L3 per core alone should count L2 and L3 together, and may now tile for the L2. [Intel 248966, Sections 2.6.1.2 and 2.6.1.3, pages 2-27 to 2-29]

Non-temporal stores occupy core resources longer than on Broadwell-EP, so a long series of them limits the write bandwidth of a core, and using them for blocks that would have fit in L2 and L3 adds misses and memory traffic. [Intel 248966, Section 2.6.2, page 2-29]

rep movs reaches full speed only with source and destination both aligned to 32 bytes; otherwise a copy with YMM or ZMM registers is faster. Golden Cove made short string operations fast. [Agner Fog, Section 11.12, page 160]

PAUSE takes up to about 140 cycles, against about 10 on earlier cores, to leave the core to the sibling thread and save power. A spin loop that counts PAUSEs therefore waits far longer here; one that waits for a time read with rdtsc, with exponential backoff under contention, waits as long whatever a PAUSE costs. On Zen 4 and Zen 5 PAUSE waits about 64 cycles. [Intel 248966, Section 2.7.4, pages 2-37 to 2-39]

Two threads on a core share the caches, the decoders and the execution units, each getting half of the throughput. Code that already runs more than two instructions a cycle, or is limited by cache size or instruction fetch, gains nothing from a second thread on its core. [Agner Fog, Sections 11.1 and 11.14, pages 152 and 161]