The front end of x86-64 cores

A core feeds its out-of-order engine from up to three places: the legacy decoders, which turn instruction bytes into operations; a cache of operations decoded earlier, which Intel calls the decoded ICache or DSB and AMD the op cache; and, on Intel’s cores, a loop buffer that replays a small loop from the queue in front of rename. The decoders are the slowest of the three and the cache spares both their work and their power. [Intel 248966, Sections 3.4.2.4 and 3.4.2.5, page 3-18; AMD 58455, Section 2.9.1, page 34]

The decoders get a limited number of bytes a cycle. Intel’s cores before Golden Cove fetch 16 bytes a cycle for them and Golden Cove 32. AMD’s Zen 3 and Zen 4 read 32 bytes from a 16-byte boundary within one 64-byte line, and Zen 5 two aligned blocks of 32 bytes for its two decode pipes. [Intel 248966, Section 2.4.1, page 2-11; Agner Fog, Section 11.2, page 152; AMD 56665, Section 2.9, page 33; AMD 57647, Section 2.9, page 25; AMD 58455, Section 2.9, page 33]

Skylake, Zen 3 and Zen 4 decode at most four instructions a cycle. Golden Cove has six decoders. Zen 5 has two decode pipes of four, but one thread decodes at most four instructions a cycle; with two threads, each has a pipe of its own. [Agner Fog, Section 11.2, page 152; AMD 57647, Section 2.9, page 25; Intel 248966, Section 2.4.1, page 2-11; AMD 58455, Section 2.9, page 33]

Sixteen bytes a cycle keep four decoders busy only while instructions average four bytes or less. Longer code is limited by fetch instead, and that is where the decoded cache gains most: Agner Fog found that it delivers the equivalent of 32 bytes of code a cycle on Sandy Bridge and Skylake. [Agner Fog, Sections 9.3 and 11.3, pages 126 and 153]

On Intel’s cores only the first decoder takes an instruction of more than one micro-op. On Skylake it handles up to four and the other three decoders one each; longer instructions come from microcode. Instructions of two to four micro-ops are best spread among single-micro-op ones, and Golden Cove keeps the rule with its six decoders. [Agner Fog, Sections 9.2 and 11.2, pages 125 and 152; Intel 248966, Section 3.5.1, page 3-20; Agner Fog, Chapter 13, page 173]

Code that runs from the decoded cache skips fetch and decode: its path through the pipeline is shorter, it delivers more operations a cycle and no power goes into decoding it again. Zen 5’s op cache delivers up to 12 instructions a cycle, against four from the decoders for one thread. [Intel 248966, Section 3.4.2.5, page 3-18; AMD 58455, Section 2.9.1, page 34]

The caches have grown with each generation: 1,536 micro-ops from Sandy Bridge to Skylake, 2,304 on Ice Lake and 4,096 on Golden Cove; 4K macro-ops on Zen 3, 6.75K on Zen 4 and 6K instructions on Zen 5, whose entries count a fused pair as one instruction. [Agner Fog, Sections 9.3, 11.3 and 12.3, pages 126, 153 and 164; Agner Fog, Table 13.1, page 174; AMD 56665, Section 2.9.1, page 33; AMD 57647, Section 2.9.1, page 26; AMD 58455, Section 2.9.1, page 34]

Intel’s decoded cache, from Sandy Bridge on (Skylake), files the code of each aligned 32-byte block in at most three lines of six micro-ops. A block that decodes to more than 18 micro-ops, or whose micro-ops do not fit in three lines, is never cached and always runs from the decoders. [Agner Fog, Section 9.3, page 126; Intel 248966, Section 3.4.2.5, page 3-19; Intel 248966, Section 22.5.7.3, page 22-61]

A lookup in Intel’s decoded cache covers a whole 32-byte block: if any of the block’s micro-ops is missing, all of them come from the decoders. Each move from the cache to the decoders costs up to three cycles, so a hot block that keeps missing costs more than its own decoding. [Intel 248966, Section 22.5.7.3, page 22-61]

Short instructions that decode to several micro-ops, such as read-modify-write instructions on memory, can put more than 18 micro-ops in a 32-byte block. Intel suggests writing them as a load, the operation and a store, which takes more bytes for the same micro-ops, shifting the dense stretch until it spans two blocks, or spreading it with NOPs. [Intel 248966, Section 3.4.2.5, page 3-19]

Each unconditional jump or call ends a line of Intel’s decoded cache, so at most three of them in a 32-byte block can be cached. A jump table of 5-byte jumps, or a switch statement compiled to a run of jumps, has more; long NOPs between the jumps, which never execute, make room for them. [Intel 248966, Section 3.4.2.5, page 3-19; Agner Fog, Section 9.3, page 126]

Large constants take room in every decoded cache. On Intel’s cores from Sandy Bridge on, a micro-op has 32 bits for its displacement and immediate, and one that needs more, such as a 64-bit constant or a large displacement with an immediate, takes a second entry and an extra cycle to read. AMD’s op cache entries have a few slots for immediates and displacements, of which a 64-bit immediate takes two (see Zen 5, Zen 4 and Zen 3). [Agner Fog, Section 9.3, pages 126 and 127; AMD 58455, Section 2.9.1, page 34; AMD 57647, Section 2.9.1, page 26; AMD 56665, Section 2.9.1, pages 33 and 34]

AMD’s front end runs in one mode or the other: it switches to the op cache only at a taken branch and back to the decoders when the op cache misses, so code that keeps missing pays for the switches as well as for decoding (Zen 5). [AMD 58455, Section 2.9.1, page 34; AMD 57647, Section 2.9.1, pages 26 and 27]

Both vendors advise keeping hot code within the cache and not unrolling loops past it. Intel puts the limit at about 750 instructions per hot block when two threads share the core, or 1,500 with one; AMD at the op cache’s capacity with one thread and half of it with two, as the threads share it. [Intel 248966, Section 3.4.2.5, page 3-19; AMD 58455, Section 2.9.1, page 34]

Intel’s loop stream detector replays a small loop from the micro-op queue, so that neither the decoders nor the decoded cache need to work: with one thread this saves power, with two it leaves the front end to the other thread. A loop qualifies when it runs more than about 20 times, has no call or ret, pushes as much as it pops and fits the queue; Intel’s general rules cap it at 60 micro-ops, 15 taken branches and 15 lines of 64 bytes, and suggest splitting a long loop body into loops that fit. [Intel 248966, Section 3.4.2.4, page 3-18]

The queue, and with it the largest loop, has grown: 64 micro-ops per thread on Skylake, 70 on Ice Lake, 144 for one thread or 72 for each of two on Golden Cove, and 192 or 96 on Redwood Cove. Skylake’s server cores have the detector turned off. [Intel 248966, Section 2.7.1, page 2-34; Intel 248966, Section 2.5.1.1, page 2-18; Intel 248966, Section 2.4.1, page 2-12; Intel 248966, Section 2.1.1, page 2-1; Intel 248966, Section 2.6, page 2-26]

The queue sends micro-ops to allocation in groups of a fixed size, so a loop whose length is not a multiple of it would leave slots empty at the end of every pass. The detector avoids most of this loss by keeping several copies of a short loop in the queue. [Intel 248966, Section 3.4.2.4, page 3-18]

AMD’s guides for Zen 3 to Zen 5 describe no loop buffer, and Agner Fog found only a queue of unknown size in front of rename on Zen 3 and Zen 4. On AMD’s cores a small loop runs from the op cache, and keeping it there is what counts. [AMD 58455, Section 2.9, pages 33 to 39; Agner Fog, Sections 23.6 and 24.6, pages 243 and 249]

Macro-fusion turns an instruction that sets the flags and the conditional branch right after it into one operation, which takes one slot in decode, rename and retirement. The two must be adjacent: an instruction scheduled between them prevents it, so a compare belongs right before its branch. [Intel 248966, Section 3.4.2.2, page 3-12; AMD 58455, Section 2.9.3, page 36; Agner Fog, Section 11.6, page 154]

Intel’s cores since Sandy Bridge (Skylake) fuse test and and with any conditional branch; cmp, add and sub with branches on the equal, unsigned and signed comparisons, but not on the sign, parity or overflow flag alone; and inc and dec only with branches on equality and the signed comparisons. [Intel 248966, Section 3.4.2.2, Tables 3-1 and 3-2, pages 3-13 and 3-14; LLVM 23.1.2, MCTargetDesc/X86BaseInfo.h, line 290]

AMD’s Zen 3 to Zen 5 fuse more: cmp, test, add, sub, and, or, xor, inc and dec with any conditional branch, except that on Zen 3 and Zen 4 inc and dec do not fuse with a branch that reads the carry flag, which Intel’s cores do not fuse either (Zen 5, Zen 4, Zen 3). [AMD 58455, Section 2.9.3, page 36; AMD 57647, Section 2.9.3, page 28; AMD 56665, Section 2.9.3, page 35]

With a memory operand the vendors differ. Intel’s Sandy Bridge to Skylake fuse an instruction that reads memory or has an immediate, but not both, nor one that writes memory or addresses relative to RIP. AMD’s Zen 3 and Zen 4 fuse cmp and test with memory even with an immediate, unless the instruction carries a displacement as well as an immediate or addresses relative to RIP. [Agner Fog, Sections 9.6 and 11.6, pages 129 and 154; Intel 248966, Section 3.4.2.2, page 3-13; AMD 57647, Section 2.9.3, page 28; AMD 56665, Section 2.9.3, pages 35 and 36]

Zen 5 relaxes the rule on displacements: an instruction that carries a displacement as well as an immediate still fuses when one of them is at most 2 bytes and the other at most 4. [AMD 58455, Section 2.9.3, page 36]

On Intel’s cores the choice of instruction decides fusion more often, so Intel prefers test to cmp for a comparison with zero, as test fuses with every condition. Its advice to use unsigned comparisons dates from Core 2: since Nehalem cmp fuses with the signed conditions too. On AMD’s cores all of these fuse. [Intel 248966, Section 3.4.2.2, pages 3-12 to 3-16; AMD 58455, Section 2.9.3, page 36]

Newer cores also fuse a register move with the operation after it, which gives the non-destructive three-operand form that the instruction set lacks: Zen 5 for operations on registers and immediates, and Intel’s Redwood Cove for a move or a load followed by an operation, though only when the code runs from the micro-op cache. [AMD 58455, Section 2.9.4, pages 36 and 37; Intel 248966, Section 2.1.1.3, pages 2-3 and 2-4]

A 66h or 67h prefix that changes the length of what follows it, as in add ax, 1000, whose 16-bit immediate is two bytes shorter than the 32-bit one without the prefix, defeats the quick length decoding of Intel’s cores. Intel gives the slow path 6 cycles, and 11 for some instructions that cross a 16-byte fetch line. [Intel 248966, Section 3.4.2.3, pages 3-16 and 3-17; Agner Fog, Section 8.2, page 109]

From Sandy Bridge on the penalty is narrower: a move with a 16-bit immediate pays nothing, while arithmetic and logic instructions with one pay 2 or 3 cycles in the decoders, wherever they are placed. Ice Lake charges them 3 cycles, and 16-bit moves pay again. [Agner Fog, Sections 9.2, 11.2 and 12.2, pages 125, 152 and 163]

The penalty is paid in the decoders only: code that runs from the decoded cache or a loop buffer has had its lengths found already. [Intel 248966, Section 22.5.7.3, page 22-61; Agner Fog, Section 8.2, page 110]

Hot code that may run from the decoders should use 8-bit or 32-bit immediates rather than 16-bit ones, working on the full register. LLVM does so when it compares a 16-bit value with a constant that does not fit in 8 bits, except for the processors it marks as fast with 16-bit immediates, such as AMD’s Zen and Intel’s Atom. [Intel 248966, Section 3.4.2.3, page 3-17; LLVM 23.1.2, X86ISelLowering.cpp, line 24008; LLVM 23.1.2, X86.td, line 1700]

AMD’s cores have no such penalty: Agner Fog found that prefixes of any kind and number decode without delay on Zen 3, Zen 4 and Zen 5. [Agner Fog, Sections 23.3, 24.3 and 25.2, pages 242, 248 and 255]

Long instructions slow the decoders of both vendors. On AMD’s cores only the first of four decode slots takes an instruction longer than 8 bytes on Zen 3 or 10 bytes on Zen 4 and Zen 5, so at most one in any four should be that long. Intel’s rules ask for instructions shorter than 8 bytes and say that only the first decoder takes an instruction with several prefixes. [AMD 56665, Section 2.9, page 33; AMD 57647, Section 2.9, page 26; AMD 58455, Section 2.9, page 33; Intel 248966, Section 3.4.2.6, page 3-20]

Alignment matters most for code that runs from the decoders, whose windows start on 16-byte boundaries on AMD’s cores and on Intel’s older ones. Code that runs from the decoded cache is held in its own units instead: blocks of 32 bytes on Intel’s cores, entries within at most two adjacent 64-byte lines on AMD’s. [AMD 58455, Sections 2.9 and 2.9.1, pages 33 and 34; Intel 248966, Section 3.4.1.4, page 3-8; Intel 248966, Section 3.4.2.5, page 3-19]

Intel’s rule for a branch that is usually taken: with the decoded cache, keep the whole branch in one 64-byte line, near its end, and put its target near the start of a line; with the decoders, the same within aligned 16-byte blocks. [Intel 248966, Section 3.4.1.4, page 3-8]

On the Skylake family with the microcode update for its JCC erratum (Skylake), 32-byte boundaries matter for every jump: one that crosses such a boundary or ends on it is no longer cached and runs from the decoders. Padding the code so that no jump, fused pair included, touches a boundary avoids most of the loss (mitigation); Intel lists no later core as affected. [Intel 341810, Sections 2.1 and 2.4, pages 6 and 8; Intel 341810, Section 4.0, pages 13 to 15]

AMD finds that loop alignment rarely matters on Zen 3 to Zen 5. For a hot loop it suggests starting it on a 64-byte boundary or, slightly better, ending it on the last byte of a line (Zen 5), and branch targets on 16-byte boundaries for the decoders (Zen 5). [AMD 58455, Sections 2.8.3 and 2.9, pages 30 and 33; AMD 57647, Section 2.8.3, page 23]

LLVM aligns functions and loops to 16 bytes on every x86 target unless told otherwise. [LLVM 23.1.2, X86ISelLowering.cpp, line 2865; LLVM 23.1.2, X86ISelLowering.cpp, line 2871]

Padding that runs is not free: a NOP uses no execution unit but still passes through decode and the retire queue, so a few long NOPs cost less than many short ones, and only a very long stretch of padding is better jumped over. [AMD 58455, Section 2.8.3.1, pages 31 and 32]

How long a NOP may be depends on the core: up to 15 bytes, made by adding 66h prefixes to the 8-byte form, on Zen 3 to Zen 5 (Zen 5), while AMD’s Family 15h slows down with more than three 66h prefixes, so code for every AMD processor stops at 11 bytes. LLVM pads with NOPs of up to 15 bytes when it tunes for Intel’s cores since Sandy Bridge, for AMD’s Zen or for its generic target, of up to 11 for Family 15h and of up to 10 for the plain -march=x86-64 tuning. [AMD 58455, Section 2.8.3.1, Tables 6 and 7, pages 31 and 32; LLVM 23.1.2, MCTargetDesc/X86AsmBackend.cpp, line 952; LLVM 23.1.2, X86.td, line 1043; LLVM 23.1.2, X86.td, line 1621; LLVM 23.1.2, X86.td, line 1693; LLVM 23.1.2, X86.td, line 1893; LLVM 23.1.2, X86.td, line 953]

Zen 4 fuses a NOP with the instruction next to it, or two NOPs together, which makes padding that runs cheaper there; Zen 5 fuses no NOPs. [AMD 57647, Section 2.9.5, page 29; Agner Fog, Section 25.3, page 255]

Instead of NOPs, instructions already in the code can be lengthened with prefixes that change nothing, such as another 66h on an instruction that has one: the padding then adds no instruction to the cache or the pipeline. It works only where such instructions come before the place to align. [AMD 58455, Section 2.8.3.1, page 32; Agner Fog, Section 11.2, page 152]