Caches and memory on x86-64 cores

Each core has its own L1 data and instruction caches and its own L2, and shares an L3 with other cores. Intel reckons that every level further out costs roughly three to ten times as much as the one inside it, and that processors spend much of their time waiting for misses even where few accesses miss. The sizes and latencies of each core are on its page: Skylake, Golden Cove, Zen 4 and Zen 5. [Intel 248966, Section 3.6.10, page 3-55; AMD 58455, Section 2.6, page 19]

On AMD’s cores the L3 holds what the L2 caches of its complex evict. A hit there removes the line from the L3 when it was for a store or when only one core has read the line, and keeps it for code and for lines several cores read; on a miss in the L3, its copies of the L2 tags let it fetch the line from another core’s L2 in the same complex. Agner Fog found that threads exchange data faster within one complex of eight cores than between complexes. Intel’s Skylake servers also stopped keeping copies of the L2’s lines in the L3 (Skylake). [AMD 58455, Section 2.6.4, page 23; Agner Fog, Section 24.19, page 253]

Two threads on one core share its caches. On Zen 5 the L1 caches, the L2, the L3 and the TLBs go to whichever thread asks, while the store queue and the write-combining buffers are divided between the threads when the second one runs. Agner Fog advises running only one thread per core when cache size limits a program. [AMD 58455, Section 2.14, Table 10, pages 59 and 60; Agner Fog, Section 11.14, page 161]

Code that sweeps over more data than a cache holds gets fewer misses when it works on blocks that fit: tiling a loop nest, swapping inner and outer loops so that accesses run along cache lines, and skewing loops so that they become contiguous. Intel suggests sizing the working set to half the L1 data cache, or else to the L2, and expects little from sizes between the two. [Intel 248966, Section 3.6.10, Rule 8, pages 3-55 and 3-56]

LLVM’s x86 cost model gives fixed cache sizes for every target: a 32 KB L1 data cache and a 256 KB L2, both 8-way, whatever core the code is compiled for. Those are the sizes of Skylake’s client parts (Skylake); they are a model, not a description of other cores, and Zen 5’s L1 holds 48 KB and its L2 1 MB (Zen 5). [LLVM 23.1.2, X86TargetTransformInfo.cpp, line 114; LLVM 23.1.2, X86TargetTransformInfo.cpp, line 144; AMD 58455, Sections 2.6.2 and 2.6.3, pages 20 and 23]

Caches move data in lines of 64 bytes on all these cores. An access that crosses from one line into the next needs both, so it takes two turns of the cache where an aligned access takes one: on Zen 5 misaligned loads and stores across a 64-byte boundary run at half the rate of aligned ones, and on Zen 3 and Zen 4 a store pays this already when it crosses 32 bytes (Zen 4). [AMD 58455, Section 2.6.2, page 20; Intel 248966, Section 3.6.7, page 3-53]

In a stream of consecutive accesses that all start at the same misalignment, the share that splits a line grows with the width: one 16-byte access in four, every second 32-byte AVX access and every 64-byte AVX-512 access. Intel therefore advises aligning data to the width of the vectors that read it. On AMD’s cores the unaligned instructions, such as vmovups, are as fast as the aligned ones when the address happens to be aligned. [Intel 248966, Section 3.6.6, page 3-52; Intel 248966, Section 15.6.1, page 15-22; Intel 248966, Section 17.23.1, page 17-75; AMD 58455, Section 2.6.2, page 20]

An access that crosses a 4 KB page boundary needs two translations as well as two lines. Skylake cut the extra cost of such a load to about 5 cycles from about 100 on earlier cores (Skylake); AMD warns that on Zen 5 it can still cost significantly more than a split within a page. [Intel 248966, Section 2.7.3, page 2-36; AMD 58455, Section 2.6.2, page 20]

The 64-bit Linux and Windows ABIs keep the stack aligned to 16 bytes, so a function with wider objects on the stack, such as 32-byte vectors, must align its own frame, which costs a register to track the padding. Intel weighs that against the line splits of unaligned accesses and advises aligning to the widest register the code stores; a value that cannot be aligned is best copied once to aligned storage or a register. [Intel 248966, Section 3.6.6, Rule 48, pages 3-52 and 3-53]

Padding every field to its natural alignment avoids splits, and ordering the fields from largest to smallest keeps that padding small. When a loop reads only some fields of many records, separate arrays per field (a structure of arrays) bring in only the bytes it uses and suit vector code, at the price of more address streams; grouping the fields that are always used together in their own arrays sits between the two. Intel advises laying data out so that it is read in sequence, which the hardware prefetchers follow. [Intel 248966, Section 3.6.5, Rule 6 and Rule 47, pages 3-50 to 3-52]

The L1 data cache is split into banks, and loads in the same cycle that need one bank can delay each other. Agner Fog found no bank conflicts on Skylake; Golden Cove’s third load port brought them back for a third load to one bank (Golden Cove); AMD’s cores pick the bank by address bits 5:3 and advise grouping consecutive loads from each stream, three per stream on Zen 4 (Zen 4) and four on Zen 5, two for vector loads, and merging byte accesses into wider ones. [Agner Fog, Section 11.12, page 160; AMD 58455, Section 2.6.2.1, page 21]

AMD’s L1 data cache chooses its way from a hash of the linear address before translation. Two virtual addresses that map the same memory, as with a buffer mapped twice, therefore find each other’s lines missing when they are used together, and pay about an L2 hit each time; two unrelated addresses with the same hash and index evict each other in the same way (Zen 4). [AMD 58455, Section 2.6.2.3, pages 21 and 22]

Skylake runs two loads and one store a cycle (Skylake), Golden Cove three loads and two stores (Golden Cove), Zen 3 and Zen 4 three memory operations of which two may be stores (Zen 4) and Zen 5 four (Zen 5), with fewer for 512-bit data where the core has it. Intel finds loads and stores in up to four of every ten instructions of common programs, so these limits, not the arithmetic, set the speed of many loops. [Intel 248966, Section 3.6.1, page 3-41; Intel 248966, Section 2.4.1.1, page 2-15; AMD 58455, Section 2.12, page 50]

A load’s latency depends on its address. AMD’s cores add a cycle for a scaled index or for a base, an index and a displacement together (Zen 5); on Sandy Bridge the 4-cycle best case needed a lone base register or a base with a displacement below 2048. In a pointer-chasing loop, where every load waits for the one before, following a pointer held in a register is therefore faster than computing the address from an index. [AMD 58455, Section 2.12, pages 51 and 52; Intel 248966, Section 3.6.1.2, pages 3-42 and 3-43]

The widest load and store the core supports moves the most data per operation, and Intel advises using it. Byte-sized neighbours are better read and written as one wider access, which also uses AMD’s banks better. Width pays only where the path is that wide: on Zen 4 a 512-bit access takes a 256-bit pipe twice and moves no more per cycle than 256-bit code (Zen 4). [Intel 248966, Section 3.6.1, page 3-41; AMD 58455, Section 2.6.2.1, page 21]

A core keeps working past a miss and can have many misses under way at once, so independent misses overlap and cost little more than one. Golden Cove tracks 16 L1 misses and 48 in its L2 (Golden Cove), Zen 4 24 (Zen 4) and Zen 5 124 (Zen 5). Misses that depend on each other, as in a linked list, cannot overlap and pay the full latency each. [Intel 248966, Section 2.4, page 2-11; AMD 58455, Section 2.12, page 51; Intel 248966, Section 3.6.10, page 3-55]

A store reaches the cache only after it retires, so a load soon after it would find the old value there. The core instead hands the load its data straight from the store buffer, which costs a few cycles where going through the cache would cost many; Intel adds that only the loads that forward wait until the store’s data is in the buffer, and unrelated loads go ahead. [Intel 248966, Sections 3.6.4 and 3.6.4.1, pages 3-45 and 3-46; AMD 58455, Section 2.12, page 50]

Every core here forwards only from one store that holds all the bytes the load reads; a load assembled from two stores, or reaching past one, waits until the stores have reached the cache. Within that rule they differ: Intel’s general rules also want the load to start at the store’s first byte, AMD’s guides for Zen 4 and Zen 5 allow any offset inside the store (Zen 5), and Golden Cove forwards the covered part of a wider load that starts where the store does (Golden Cove). [Intel 248966, Section 3.6.4.1, Rules 41 and 42, page 3-46; AMD 58455, Section 2.12, page 50; AMD 57647, Section 2.12, pages 39 and 40; Intel 248966, Section 2.4.1.1, page 2-15]

The usual failures are several narrow stores followed by one wide load of the same bytes, such as bytes written one by one and read as a doubleword, and a wide store followed by narrow loads of its parts. The load then waits about 11 cycles on Skylake and about 12 on Zen 5 (Skylake). Intel advises writing the data with one store, and reading a part of stored data as one aligned load that is then shifted and masked; a compiler’s byte spill reloaded with a zero-extending load avoids the wide reload. [Intel 248966, Section 3.6.4.1, Rules 43 and 44, pages 3-47 and 3-48; Agner Fog, Section 11.13, page 161; Agner Fog, Section 25.17, page 259]

A load that forwards cannot finish before the store’s data exists, so a value that passes through memory adds the store and the load to its dependency chain. Intel advises keeping such values in registers, passing arguments in registers where the calling convention allows, not forwarding the result of a slow instruction such as a division, keeping forwarding out of chains that run from one iteration to the next, and computing store addresses early; it even suggests spilling to vector registers instead of memory. [Intel 248966, Section 3.6.4.2, Rules 45 and 46, pages 3-49 and 3-50; Intel 248966, Section 3.6.4, Rule 40, page 3-46; Intel 248966, Section 3.6.2, page 3-44]

Zen 2, Zen 4 and Zen 5, but not Zen 3, go further: when a load names the same address as a recent store, written the same way, the core predicts the match and passes the value on before either has run, so a store followed by a load of it, or an instruction that reads and writes a memory variable, adds no memory latency at all (Zen 3). Agner Fog found that it works for general registers only and not with absolute or RIP-relative addresses; Zen 4 allows 32-bit displacements where Zen 2 allowed 8-bit ones, and Zen 5 runs two such read-modify-write instructions a cycle with 8-bit to 64-bit operands. A wrong guess is costly: on Zen 2, a store through a second pointer to the same address made the sequence take about 20 cycles. [Agner Fog, Section 22.18, pages 237 to 239; Agner Fog, Section 24.18, page 253; Agner Fog, Section 25.18, page 259]

A load often reaches execution while older stores still lack their addresses. Since Intel’s Core microarchitecture such a load runs anyway on the guess that it does not overlap them, and is run again with everything after it if an older store turns out to write its bytes; AMD’s load-store unit also lets loads pass older stores that do not conflict. Intel names the SSBD bit of the IA32_SPEC_CTRL MSR as the way to disable this speculative store bypass. Computing store addresses early leaves fewer loads to guess about. [Intel 248966, Section 3.6.3, page 3-45; Intel 248966, Section 3.6.4.2, Rule 46, page 3-50; AMD 58455, Section 2.12, page 50]

The first check of a load against older stores looks only at the low bits of the addresses, so a load from an address a multiple of 4 KB away from a store still in flight is treated as if it overlapped. Intel puts the cost at about five cycles of load latency, more when the load also splits a line, and AMD’s Zen 4 and Zen 5 compare bits 11:0 for forwarding and advise against several such stores in flight. Two buffers walked in step, such as the source and destination of a copy whose start addresses differ by a multiple of 4 KB, meet this on every iteration; Intel advises aligning the data and changing the offset between the buffers (Skylake, Zen 5). [Intel 248966, Section 15.8, page 15-26; AMD 58455, Section 2.12, page 50; AMD 57647, Section 2.12, page 40; Agner Fog, Section 11.12, page 160]

Every access needs its page’s translation, and the TLBs hold a few thousand: the 4,096 entries of Zen 5’s L2 data TLB cover 16 MB in 4 KB pages. Large pages stretch them; in Intel’s example 20 MB takes 5,120 entries in 4 KB pages and 10 in 2 MB pages. The sizes of each core’s TLBs are on its page (Skylake, Golden Cove, Zen 4, Zen 5). [Intel 248966, Appendix B.1.2, page B-2; AMD 58455, Section 2.7.2, page 24]

A miss in the last TLB level starts a page walk, which reads four or five levels of page tables through the data caches. Walks run in parallel: two at once on Skylake, four on Golden Cove and six on Zen 3, Zen 4 and Zen 5, whose walkers keep the upper levels’ entries in caches of their own so that a walk can skip two or three levels. [Intel 248966, Section 2.7.3, page 2-36; Intel 248966, Section 2.4.1.1, page 2-15; AMD 58455, Section 2.7.3, page 25; AMD 57647, Section 2.7.3, page 19; AMD 56665, Section 2.7.3, page 27]

Zen 3, Zen 4 and Zen 5 can hold four 4 KB pages in one TLB entry when they form an aligned 16 KB block that is also contiguous and aligned in physical memory and has the same attributes, in long mode only. Memory the operating system hands out in contiguous runs thus reaches further in their TLBs even without large pages. [AMD 58455, Section 2.7.1, page 24; AMD 57647, Section 2.7.1, page 19; AMD 56665, Section 2.7.1, page 27]

On Linux a program gets 2 MB pages either from hugetlbfs, a pool reserved for them, or from transparent huge pages, which the kernel provides automatically or, in its madvise setting, for the regions a program marks. Large pages need contiguous physical memory: when it is fragmented, the kernel must first move pages about, which delays the allocation, unless the pages are reserved in advance. [Intel 248966, Appendix B.3.1, page B-7; Intel 248966, Appendix B.5, page B-11]

Hardware prefetchers watch the misses and fetch the lines that the pattern so far predicts. Intel’s cores follow the stride of each load instruction and the next line into the L1, and into the L2 fetch the other half of an aligned 128-byte pair and follow streams of misses; AMD’s follow sequential lines up or down, strides per instruction and recurring patterns within a region into the L1, and sequential and neighbouring lines into the L2 (Zen 4). Data laid out so that its misses follow such a pattern misses less. [Intel 248966, Sections 3.7.2 and 3.7.3, pages 3-58 to 3-60; AMD 58455, Section 2.12.1, page 53]

The prefetchers need a few misses before they start, run past the end of an array, and do not cross a 4 KB page boundary, so short arrays and data scattered over many pages gain little. Strides must stay below the distance at which the prefetcher gives up. Intel’s per-instruction prefetcher tells loads apart by the low 8 bits of their instruction address, so in a loop longer than 256 bytes two loads can look like one. [Intel 248966, Section 9.5.2, pages 9-12 and 9-13; Intel 248966, Section 3.7.2, page 3-60]

Prefetching that guesses wrong costs cache space and memory bandwidth, as with random or irregular access such as some database code. Some of AMD’s server parts can turn single prefetchers off through an MSR; Intel suggests shrinking the data used at once below half the L2 where its L1 prefetcher fetches lines that are not needed, and Emerald Rapids’ adaptive L2 prefetcher can be switched off in firmware (Golden Cove). [AMD 58455, Section 2.12.1, page 53; Intel 248966, Section 3.7.2, pages 3-58 and 3-59]

A prefetch instruction asks for the 64-byte line holding its address and changes nothing a program can see. It does not fault (a LOCK prefix aside): when the access would fault, it is dropped, and a line already in a closer cache is left where it is. Intel warns against prefetching addresses that are not mapped, a null pointer above all, which can take a long time. [Intel 248966, Sections 9.3.1 and 9.3.2, page 9-3; Intel 248966, Section 9.3.3, page 9-5; Intel 248966, Section 3.7.1, page 3-57]

The hint names a level, but each core decides what it means. On Skylake’s client parts PREFETCHT0 fills the L1 and L2, PREFETCHT1 and PREFETCHT2 the L2 alone and PREFETCHNTA the L1 alone; on Intel’s Xeon Scalable parts each hint also fills the L3, NTA’s with a line likely to be replaced early, and Intel treats T1 and T2 alike on all its cores. Zen 5 keeps them apart: T1 skips the L1 and T2 the L1 and L2, while a line that NTA brings in leaves the L2 early and never enters the L3 (Zen 5). [Intel 248966, Section 9.3.2, Tables 9-2 and 9-3, page 9-4; AMD 58455, Section 2.13.1, Table 8, page 54; AMD 58455, Section 2.6.4, page 23]

Software prefetch pays where the hardware cannot see the pattern and the data is not in the cache already. Agner Fog found the hardware prefetchers of Zen 4 and Zen 5 better than explicit prefetching in most cases. Intel’s advice for the rest: prefetch far enough ahead to cover the latency but not so far that the line is evicted again before use, issue as few prefetches as possible, one per line, and spread them among the other instructions. [Agner Fog, Section 24.16, page 252; Agner Fog, Section 25.16, page 258; Intel 248966, Section 9.1, pages 9-1 and 9-2; Intel 248966, Sections 9.5.6 and 9.5.8, pages 9-16 and 9-17]

A normal store to a line not in the cache first reads the line, so that the rest of it is there when the store lands. Streaming, or non-temporal, stores skip that read and write around the caches through write-combining buffers. They save the read’s bandwidth and leave the cached data alone, but their ordering is weak, like that of write-combining memory. [Intel 248966, Sections 9.4.1 and 9.4.1.2, pages 9-5 and 9-6; Intel 248966, Section 3.6.11, page 3-56; AMD 58455, Section 2.13, page 54]

The buffers send a line as one 64-byte write when it is complete. A stream that writes whole lines in order gets the full bandwidth; one that leaves lines partly written sends several partial writes instead, which costs far more than the bytes saved, and on AMD’s cores a partly filled buffer can stay open a long time. Intel’s cores have 10 buffers from Nehalem and 12 from Ice Lake, Zen 4 eight and Zen 5 twelve, one fewer for a thread that shares the core (Zen 4); Intel advises splitting a loop that writes more than four arrays. [Intel 248966, Section 3.6.9, Rule 51, page 3-55; Intel 248966, Section 3.6.11, page 3-56; AMD 58455, Sections 2.13.4 and 2.13.5, pages 56 and 57]

Because they are weakly ordered, streaming stores may become visible to other cores after later ordinary stores, and their data may stay in a buffer. Code that fills a buffer with them and then sets a flag for another thread needs an SFENCE between the two; on AMD’s cores SFENCE and MFENCE also empty the write-combining buffers. An LFENCE and an SFENCE together are not an MFENCE, since neither orders loads against stores. [Intel 248966, Section 9.4.1.1, page 9-6; Intel 248966, Sections 9.4.5.1 and 9.4.5.3, pages 9-8 and 9-9; AMD 58455, Section 2.13.4, Table 9, page 57]

Streaming stores suit data written once and not read again soon, in amounts larger than the caches. Intel puts the point at about half the last-level cache and has the size read from CPUID; below it, or when the program reads the data back soon after, they add misses instead. Skylake’s servers keep them longer in the core, which limits a long run of them (Skylake), and Zen 5 switches rep movs and rep stos to streaming stores by itself from the size of the L3 (Zen 5). [Intel 248966, Section 3.7.5, pages 3-61 and 3-62; Intel 248966, Section 9.6.1, page 9-25]

CLFLUSH writes a line back if it is dirty and removes it from every cache. CLFLUSHOPT does the same but is ordered only against writes to its own line, fences and locked instructions, so many of them overlap: Intel measured flushing a buffer up to 9 times faster with it on Skylake, and advises an SFENCE before and after a series of them where order matters. The MFENCE that older manuals required after CLFLUSH is not needed. [Intel 248966, Section 9.4.6, page 9-9; Intel 248966, Section 9.4.7, Rule 13, pages 9-10 and 9-11]

Since Ivy Bridge, rep movsb and rep stosb are fast enough for memcpy and memset, which CPUID leaf 7 reports in EBX bit 9; Ice Lake made rep movsb fast for 1 to 128 bytes (EDX bit 4), and Golden Cove added fast empty rep movsb and fast short rep stosb, rep cmpsb and rep scasb (Golden Cove). Copying backwards with the direction flag set is much slower, and Intel does not advise rep scas and rep cmps for speed. [Intel 248966, Sections 3.7.6 and 3.7.6.1, page 3-63; Intel 248966, Section 3.7.6.3, page 3-65; Intel 248966, Section 3.8, page 3-66; Intel 248966, Section 3.7.5, page 3-63]

Even so, the string instructions are not always the fastest copy. On Ivy Bridge a copy under 128 bytes was slower than with 128-bit AVX and a misaligned destination cost it more; on Skylake rep movs reaches full speed only with both buffers aligned to 32 bytes (Skylake). Intel advises using the compiler’s library routines, which choose by size and alignment. [Intel 248966, Section 3.7.6.2, Tables 3-5 and 3-6, page 3-65; Intel 248966, Section 3.7.5, page 3-62]

Caches keep coherent by whole lines, so two threads that write different variables in one 64-byte line take the line from each other on every write, as if they shared the data. Intel calls the spacing that avoids it the false-sharing threshold and has it read from the CLFLUSH line size of CPUID leaf 1, or taken as 64 bytes; padding or aligning data and locks that different threads write to it removes the traffic, at the cost of memory. On Linux, perf c2c finds such lines: one line with several offsets written by different threads. [Intel 248966, Section 11.4.5, Rule 17, pages 11-15 and 11-16; Intel 248966, Sections 3.11.5.1 to 3.11.5.3, pages 3-77 and 3-78]

Many threads that update one line, a lock above all, queue for it: on Sapphire Rapids systems such contention fills a queue of requests, and the cores are then slowed down. Intel advises spinning on a plain load with PAUSE until the lock looks free, trying the cmpxchg only then and backing off exponentially after a failure, and replacing shared counters by per-thread copies that are summed now and then. [Intel 248966, Section 3.11.1, page 3-73; Intel 248966, Section 3.11.3, pages 3-75 and 3-76]

A locked instruction is fastest when its operand stays within one line, or within 16 bytes on Zen 3 and Zen 4 (Zen 5). AMD also advises keeping floating-point instructions away from it and leaving last branch recording off, and Intel notes that heavy use of the lock prefix slows memory access whether the data is cached or not. [AMD 58455, Section 2.15, page 60; Intel 248966, Section 3.6.10, page 3-56]