Optimizing code for Intel Golden Cove

The decoders read 32 bytes a cycle, twice as many as on Ice Lake, and six of them decode up to six instructions a cycle, against four on Zen 4. Only the first decoder takes an instruction of more than one micro-op, so code run from the decoders goes fastest with such instructions spread out. Agner Fog found the decoders fast enough that the micro-op cache counts for less than on earlier Intel cores. [Intel 248966, Section 2.4.1, page 2-11; Agner Fog, Chapter 13 and Section 13.1, pages 173 and 175]

The micro-op queue that feeds allocation holds 144 entries for one thread, or 72 for each of two. Its loop stream detector, which Intel says covers more loops than before, replays a loop from the queue without the decoders or the micro-op cache when the loop fits, runs many times, has no call or ret and keeps its pushes and pops balanced. [Intel 248966, Section 2.4.1, page 2-12; Intel 248966, Section 3.4.2.4, page 3-18]

Intel says only that the branch predictor’s tables grew and the code prefetcher improved, for programs with a large code footprint; neither its manual nor Agner Fog gives sizes for the BTB, the return stack or the indirect predictor of Golden Cove. [Intel 248966, Section 2.4.1, pages 2-11 and 2-12; Agner Fog, Chapter 13, pages 173 to 175]

An indirect jmp or call through a register or memory whose target lies more than 2 GB away takes more of the predictor and mispredicts more often. Code and the libraries it calls are best kept within 2 GB of each other: linked statically, mapped low with glibc’s LD_PREFER_MAP_32BIT_EXEC=1, or with JIT code placed near the program’s text. Intel measured 6% on a PHP server from this alone, and names the wider core and such branches as reasons for more bad speculation than before. [Intel 248966, Section 3.11.7, pages 3-82 and 3-83]

A conditional branch the predictor holds no history for is predicted not taken, so the rarely run branches that are usually taken cost a misprediction. Golden Cove ignores the 3Eh taken hint, which Intel introduced with its successor, Redwood Cove. [Intel 248966, Section 2.1.1.1, page 2-2]

Each of the five integer ports has an ALU and an LEA unit, so simple integer instructions run five a cycle; shifts and branches use ports 0 and 6. [Intel 248966, Section 2.4.1, Figure 2-2 and Table 2-3, pages 2-11 to 2-13; Agner Fog, Chapter 13, page 173]

Register-to-register moves and zeroing idioms are handled when micro-ops are renamed, and adding a small constant to a register sometimes takes no cycle either. LLVM’s models give mov between 64-bit registers and add or sub of an 8-bit immediate to one no micro-op, and a 32-bit mov a latency of zero. [Intel 248966, Section 2.4.1, Figure 2-2, page 2-11; Agner Fog, Chapter 13, page 173; LLVM 23.1.2, X86SchedAlderlakeP.td, line 572; LLVM 23.1.2, X86SchedAlderlakeP.td, line 1431]

Port 1 alone runs the slower integer operations: multiplication, PDEP and PEXT, popcnt, bsr, crc32 and the double shifts with an immediate count, which LLVM’s models give a latency of three cycles and one per cycle. PDEP and PEXT are as fast as on Zen 4, while Zen 5 runs three a cycle. [Intel 248966, Section 2.4.1, Table 2-4, page 2-13; LLVM 23.1.2, X86SchedAlderlakeP.td, line 346; LLVM 23.1.2, X86SchedAlderlakeP.td, line 658; LLVM 23.1.2, X86SchedAlderlakeP.td, line 390; LLVM 23.1.2, X86SchedAlderlakeP.td, line 405]

A MUL of 64 by 64 bits into RDX:RAX is two micro-ops: the low half on port 1 and the high half on port 5, four cycles in LLVM’s models against three for an IMUL that keeps only the low half. [Intel 248966, Section 2.4.1, Figure 2-2, page 2-11; LLVM 23.1.2, X86SchedAlderlakeP.td, line 344; LLVM 23.1.2, X86SchedAlderlakeP.td, line 346]

Integer division runs on port 1; LLVM’s models give a 32-bit DIV 15 cycles and a 64-bit one 18, the latter three micro-ops that keep port 1 busy for three cycles. [Intel 248966, Section 2.4.1, Figure 2-2, page 2-11; LLVM 23.1.2, X86SchedAlderlakeP.td, line 206; LLVM 23.1.2, X86SchedAlderlakeP.td, line 207]

An operation on AH, BH, CH or DH runs on port 1 only, with a latency of three cycles, where Skylake to Ice Lake took one cycle plus one on ports 1 and 5. Code that packs bytes through the high byte registers, as some compilers and older assembly do, is better rewritten with shifts. [Intel 248966, Section 3.5.2.3, page 3-32]

Only Sapphire and Emerald Rapids run AVX-512, AVX-512 FP16 and AMX. Alder Lake, like Raptor Lake after it, stops at AVX2 and AVX-VNNI, the set its E-cores share: the P-cores carry the AVX-512 units, but Intel disabled them, and later microcode ended a BIOS option of early boards that turned them on with the E-cores off. [Intel 248966, Sections 2.2, 2.3.1 and 2.4, pages 2-5, 2-6 and 2-10; Agner Fog, Chapter 13 and Section 13.1, pages 173 and 175]

Vector operations of up to 256 bits run on ports 0, 1 and 5: FMA and multiplication on 0 and 1, division and square root on 0, shuffles on 1 and 5, including those that cross 128-bit lanes. 512-bit operations use ports 0 and 5 only: where port 5 has the unit, as for FMA, addition, logic and shuffles on the servers, they do twice the work per cycle of 256-bit code, and where only port 0 has it, as for shifts and division, the same. [Intel 248966, Section 2.4.1, Tables 2-3 and 2-4, pages 2-12 to 2-14]

Sapphire and Emerald Rapids have a 512-bit FMA unit on port 5 besides the one on port 0, two 512-bit FMAs a cycle with a latency of four cycles, as many as Zen 5 and twice the 512-bit rate of Zen 4, which runs each over two cycles. [Intel 248966, Section 2.4.1, Tables 2-3 and 2-4, pages 2-13 and 2-14; LLVM 23.1.2, X86SchedSapphireRapids.td, line 269]

Floating-point additions and subtractions, addsub included, have a unit of their own on ports 1 and 5 with a latency of three cycles, cut to two when one addition feeds the next. A chain of additions is best left as additions, not turned into FMAs, which take four. On the servers, 512-bit additions get the shorter latency on port 5 only. [Intel 248966, Section 2.4.1, Table 2-5, pages 2-14 and 2-15; Agner Fog, Chapter 13, page 173; LLVM 23.1.2, X86SchedAlderlakeP.td, line 263]

A result that moves between the FMA or multiply units and the one-cycle vector integer operations on ports 0 and 1, either way, arrives a cycle late, as does a value moved from a general register into the FMA units. [Intel 248966, Section 2.4.1, Table 2-5, page 2-14]

VPERMD, VPERMQ, VPERMPS, VPERMPD, VPMULLQ, vrange, vgetmant and the half-precision complex multiplications wait for the old value of their destination, which they then overwrite. Zeroing the destination first with vpxord lets them start early. [Intel 248966, Section 2.4.1.2, pages 2-15 and 2-16]

A legacy SSE instruction run while the upper halves of the vector registers hold data needs a microcode assist that merges them: Intel measured a three-instruction sequence at 388 cycles where Ice Lake took 1. vzeroupper before calling or returning to code that may use SSE avoids it, and the ASSISTS.SSE_AVX_MIX event finds where it happens. The same vzeroupper rule holds on Zen 4. [Intel 248966, Section 3.11.6, Tables 3-12 and 3-13, pages 3-80 to 3-82]

A gather is a load micro-op for each element plus a few more: LLVM’s models give an eight-element VPGATHERDD 12 micro-ops and a latency of 30 cycles, and the 16-element one of Sapphire Rapids 19. [LLVM 23.1.2, X86SchedAlderlakeP.td, line 2224; LLVM 23.1.2, X86SchedSapphireRapids.td, line 3915]

On Sapphire Rapids, VPCOMPRESSD to memory is four micro-ops in LLVM’s model, not a microcoded sequence as on Zen 4, so compressing straight into memory is fine here. [LLVM 23.1.2, X86SchedSapphireRapids.td, line 2757]

AVX-512 FP16 adds and multiplies 32 half-precision values per 512-bit register with a latency of four cycles; division and square root take about 31. A complex multiplication goes through the multiplier twice as one micro-op, 8 cycles, and rounds its intermediate products to half precision, which keeps it symmetric but costs accuracy. [Agner Fog, Chapter 13, pages 173 and 174]

On Sapphire and Emerald Rapids, a tile multiplication (tdp*) can start every 16 cycles and takes 52; a tile load every 8, taking 45; and loading the tile configuration takes about 204 cycles, worth doing once rather than in every call. AMX has a frequency license of its own, usually lower than the AVX-512 one. [Intel 248966, Sections 20.2.1 and 20.3, Table 20-2, pages 20-2 and 20-3]

A tile load cannot take its data from stores still in the store buffer: it waits until they reach the cache. Data written just before it is loaded into a tile, such as a freshly decompressed or converted block, should be written early. [Intel 248966, Section 20.15, page 20-41]

Three loads a cycle, against two on Ice Lake, but only two when they are 512-bit, AMX, x87 or MMX loads, and two stores a cycle, one when 512-bit. Zen 4 also loads three a cycle, Zen 5 four. [Intel 248966, Section 2.4.1.1, page 2-15; Agner Fog, Chapter 13, page 173]

The third load port brings bank conflicts: two loads in the same cycle may share a bank, given by bits 5:2 of the address, but a third one waits, even when all three read the same line. Values read again and again in a tight loop are better kept in a register; 512-bit loads, two a cycle at most, never conflict. [Intel 248966, Section 3.6.1.3, page 3-44]

A load that a store covers only in part still gets that part from the store, if both start at the same address. [Intel 248966, Section 2.4.1.1, page 2-15]

16 L1 misses can be in flight per core, and 48 in the L2’s miss queue, against 32 on Ice Lake; Redwood Cove, the successor, has 64. [Intel 248966, Section 2.4, page 2-11; Intel 248966, Section 2.1.1, page 2-2]

Short rep movsb, rep stosb, rep cmpsb and rep scasb are fast: with the data in the L1 cache, any length up to 128 bytes takes a flat 9, 12, 15 and 15 cycles, zero included except for the compares. CPUID leaf 7, subleaf 1, EAX bits 10 to 12 report these features, so a memcpy or memset can test for them before using the string instruction for small sizes. [Intel 248966, Section 3.8, page 3-66]

Sapphire Rapids added MOVDIRI and MOVDIR64B, weakly ordered stores for ringing a device’s doorbell without stalling the stores behind them. One sfence before a group of them orders the descriptors they announce; the 64 bytes MOVDIR64B copies are best written with one aligned 512-bit store, so that it can forward them. [Intel 248966, Sections 3.12.1 and 3.12.1.1, pages 3-84 and 3-85]

Emerald Rapids added a self-tuning L2 prefetcher (AMP) that helps most code but slowed some AMX deep-learning code by up to about 7%; a firmware setting turns it off. [Intel 248966, Section 20.14, page 20-41]

With two threads on a core, each gets half of the 144-entry micro-op queue. On the hybrid parts, an E-core runs a thread faster than a P-core whose two threads are both busy, so filling the E-cores pays before doubling up on P-cores. [Intel 248966, Sections 2.3.2.2 and 2.4.1, pages 2-9 and 2-12]

On Alder Lake and Raptor Lake, Thread Director tells the operating system how each thread would fare on either kind of core. Intel advises against pinning threads to cores, and for waiting with UMWAIT, TPAUSE or PAUSE rather than plain spinning, which lets the scheduler see the waiting. [Intel 248966, Sections 2.3.2.1 and 2.3.3, pages 2-6 to 2-10]

Two AMX threads on one core, or an AMX thread beside another busy thread, slow each other down. Intel advises one AMX thread per core with its sibling idle, and idle workers that wait with UMWAIT or TPAUSE in its C0.2 state rather than spinning. [Intel 248966, Section 20.17.2, page 20-52]