Vector code on x86-64 cores

The vector width a core runs at full rate has changed with every generation. Skylake’s client parts and Alder Lake’s Golden Cove cores stop at AVX2 and 256-bit units, while the Skylake servers and Sapphire Rapids add AVX-512 with one or two 512-bit FMA units (Skylake, Golden Cove). Zen 3 has 256-bit paths and no AVX-512 (Zen 3); Zen 4 runs AVX-512 on 256-bit paths, each 512-bit operation in two halves (Zen 4); Zen 5 has 512-bit paths throughout (Zen 5). [Agner Fog, Chapter 11, page 152; Intel 248966, Sections 2.2, 2.3.1 and 2.4, pages 2-5, 2-6 and 2-10; Intel 248966, Section 17.21, page 17-67; AMD 56665, Section 2.1, page 18; AMD 57647, Sections 2.11 and 2.11.2, pages 34 to 36; AMD 58455, Section 2.11, page 43]

On the Skylake servers wide vector code costs clock speed. Each core’s turbo ceiling drops one level for AVX2 floating-point or multiply work and for light AVX-512, and two for heavy AVX-512; a core waits up to 500 microseconds for a lower level and keeps it for some 2 ms after the last such instruction (Skylake, Skylake). Intel still saw deep-learning code built for AVX-512 run 1.3 to 1.5 times as fast as its AVX2 build, and gives AMX on Sapphire Rapids a level of its own, usually below the AVX-512 one. [Intel 248966, Section 2.6.3, Table 2-12, pages 2-29 to 2-31; Intel 248966, Section 20.2.1, page 20-2]

On the Skylake servers the widest micro-op waiting in the scheduler decides which ports vector work may use: while a 512-bit one waits, code of every width shares ports 0 and 5, and port 1 runs none of it (Skylake). AVX-512 instructions on YMM registers count as 256-bit there, and with AVX-512VL nearly every AVX-512 instruction has 128-bit and 256-bit forms, so masks and the registers above 15 do not require 512-bit vectors; only static rounding implies 512 bits. [Intel 248966, Section 17.20, page 17-66; Intel 248966, Section 17.14, page 17-46]

AMD recommends 512-bit vectors on Zen 4 and Zen 5 alike: the same work takes fewer instructions and macro-ops, which it expects to raise speed and lower power, even on Zen 4, which issues each 512-bit operation over two cycles on 256-bit paths. Agner Fog comes to the same view for Zen 4: a 512-bit instruction is a single micro-op, which counts wherever the decoders or the six micro-ops a cycle of the pipeline are the limit rather than the vector units. [AMD 57647, Section 2.11.2, page 36; AMD 58455, Section 2.11.2, page 46; Agner Fog, Section 24.21, page 254]

A VEX or EVEX instruction writes all of its destination, while a legacy SSE instruction leaves the bits above 127 alone, so the processor tracks whether the upper parts of the registers may hold data, and SSE code that runs while they do pays for it. What it pays depends on the core. Haswell, Broadwell and Ice Lake client save or restore the upper halves of every register at each switch, at roughly the cost of an XSAVE; Skylake instead makes each SSE instruction depend on its destination and merge into it (Skylake); Golden Cove takes a microcode assist (Golden Cove). Zen 3 takes a micro-fault of about 100 cycles that saves or restores the upper halves of all 16 registers (Zen 3), and Zen 4 and Zen 5 widen each such SSE operation to merge the upper bits (Zen 5). [Intel 248966, Section 15.3, Table 15-3, pages 15-8 to 15-10; Intel 248966, Section 3.11.6, Tables 3-12 and 3-13, pages 3-80 to 3-82; AMD 56665, Section 2.11.6, pages 44 and 45; AMD 58455, Section 2.11.6, page 49]

VZEROUPPER zeroes bits 128 and up of the vector registers and returns the state to clean, at no latency: Intel says so, and AMD’s spreadsheets give it a latency of 0 on Zen 3 and Zen 4. Intel’s rules for code that has written the upper bits are to run it before calling anything that may use SSE, at the end of each such function, and before starting a thread, which would otherwise inherit the dirty state. [Intel 248966, Sections 15.3 and 15.3.1, pages 15-10 and 15-11; Intel 248966, Section 17.19, pages 17-65 and 17-66; AMD 56665, latency spreadsheet, row 1487 (VZEROUPPER); AMD 57647, latency spreadsheet, row 3359 (VZEROUPPER)]

Only registers 0 to 15 can make the state dirty. SSE code, 128-bit VEX and EVEX instructions, AVX-512 instructions on registers 16 to 31, writes to mask registers and general-purpose code do not make it dirty, so vector code kept in registers 16 to 31, which only EVEX encodings reach, needs no VZEROUPPER; LLVM’s pass that inserts it looks at YMM0 to YMM15 and ZMM0 to ZMM15 only. [Intel 248966, Section 17.19, page 17-65; LLVM 23.1.2, X86InsertVZeroUpper.cpp, line 121]

Compilers insert VZEROUPPER on their own. GCC 14.2 does it with -mvzeroupper, before control leaves a function, and its tuning turns that on for every x86 target except Knights Landing; LLVM has a pass that adds it before calls and returns in functions that used the upper bits, for the targets whose tuning asks for it, among them Skylake, Ice Lake, x86-64-v3 and every Zen. [GCC 14.2, doc/invoke.texi, line 35521; GCC 14.2, config/i386/x86-tune.def, line 722; LLVM 23.1.2, X86InsertVZeroUpper.cpp, line 10; LLVM 23.1.2, X86InsertVZeroUpper.cpp, line 204; LLVM 23.1.2, X86.td, line 1109; LLVM 23.1.2, X86.td, line 1214; LLVM 23.1.2, X86.td, line 993; LLVM 23.1.2, X86.td, line 1704]

Recompiling SSE code for AVX removes the switches altogether: each SSE instruction becomes its 128-bit VEX form, which zeroes the upper bits instead of keeping them, and 128-bit and 256-bit AVX code mix at no cost. Intel names GCC’s -mavx for C and assembly sources and its assembler’s -msse2avx for hand-written SSE. [Intel 248966, Section 15.3, pages 15-7 and 15-10]

AVX-512 makes each element of an operation depend on a bit of a mask register, k1 to k7 (k0 cannot serve as one). Where the bit is clear the element is not computed, so it raises no floating-point exception and sets no MXCSR flag, and for some instructions a memory access for it cannot fault; its destination keeps its old value (merge masking) or becomes zero (zero masking). Gathers, scatters and stores to memory can only merge. [Intel 248966, Section 17.2, pages 17-7 and 17-8]

Merge masking adds the old destination as an input, and with it a dependency. In Intel’s example on a Skylake server with two FMA units, a merge-masked multiplication into a register carried over from the previous iteration managed two every four cycles, and the zero-masked form two every cycle; Intel’s advice is to mask only where needed and to zero rather than merge where possible. On Zen 4 and Zen 5 that extra input also competes for a source bus of the vector pipes (Zen 5), and on Skylake masked moves between registers are never eliminated (Skylake). [Intel 248966, Section 17.2.2, pages 17-12 and 17-13; AMD 58455, Section 2.11, page 44; Agner Fog, Section 11.9, page 157]

For a condition with work on both sides, computing each side unmasked over all elements and merging the results once, with a masked move before the store, can beat masking every instruction, as fewer instructions wait on each other: Intel measured 23% with 11% more instructions and one more register. Unlike masking, the unmasked work does not suppress exceptions from the elements whose results are thrown away. [Intel 248966, Section 17.2.3, pages 17-13 and 17-14]

Mask registers have logic instructions of their own, such as kandw, so nested conditions combine without going back through vector registers. Moving a mask in from a general-purpose register takes two macro-ops on Zen 4 and Zen 5, with a latency of three cycles on Zen 4 and two on Zen 5, and kortestw is two macro-ops on both. [Intel 248966, Section 17.2.4, page 17-15; AMD 57647, latency spreadsheet, row 344 (KMOVW from a general register); AMD 57647, latency spreadsheet, row 356 (KORTESTW); AMD 58455, latency spreadsheet, row 974 (KMOVW from a general register); AMD 58455, latency spreadsheet, row 990 (KORTESTW)]

AVX’s masked moves, VMASKMOVPS and VPMASKMOVD with their kin, find their mask in the top bits of the elements of a second vector. On Intel’s cores a masked load can take an assist of several hundred cycles when masked-off elements lie at an invalid address, or when the mask is all zero and the address uses an index; on Skylake an all-zero masked store to an invalid address does too. Intel uses them only where an unmasked move will not do; AVX-512 masked loads take no assist when the mask is all zero. LLVM’s models make a 256-bit VMASKMOVPS store 42 macro-ops on Zen 3 and Zen 4 (Zen 4), while AMD’s Zen 5 spreadsheet gives it two. [Intel 248966, Section 15.9, pages 15-26 and 15-27; Intel 248966, Section 17.2.5, Table 17-2, page 17-16; LLVM 23.1.2, X86ScheduleZnver3.td, line 864; LLVM 23.1.2, X86ScheduleZnver4.td, line 876; AMD 58455, latency spreadsheet, row 1315 (VMASKMOVPS store)]

A masked store hands its data to a later load only when its mask is all ones. With a partial mask a load that overlaps it waits until the store has reached the cache, and with an all-zero mask the load does not depend on it. For a remainder known to be very short and followed by a load from the same line, Intel prefers scalar code to a masked remainder. [Intel 248966, Sections 17.3 and 17.4, pages 17-19 and 17-20]

Masks also take the place of scalar loops before and after the vector loop: a loop can handle the elements up to the first aligned address under a mask, run its body unmasked on aligned data and finish the rest under a mask. Intel measured 4% over an unaligned body with a masked end, at the cost of code size; the memory chapter says why alignment matters. [Intel 248966, Section 17.2.6, pages 17-16 to 17-18]

A gather fetches each element on its own, so where the layout of the data is known, vector loads followed by shuffles bring in more per load. AMD advises against gathers whose indices are known in advance, and Intel replaces gathers of short strides, such as the two halves of each complex number, and of adjacent pairs by loads and permutes. [Intel 248966, Section 15.16.4, page 15-70; Intel 248966, Section 17.22, pages 17-71 to 17-74; AMD 58455, Section 2.11.2, page 46]

Where a gather is the right tool, Intel’s advice is to issue it well before its result is needed and to gather many elements at once, since a 256-bit gather takes less than twice as long as a 128-bit one. For 32-bit elements, doubleword indices are better than quadword ones, which fill only half the register without halving the latency. A gather merges into its destination, so zeroing that first breaks the dependency on its old value when no element is masked off. [Intel 248966, Section 15.16.4, page 15-70]

What a gather costs depends on the core. On Skylake it beats a sequence of scalar loads unless it fetches only two elements or the data comes from memory (Skylake), and on Golden Cove it is a load per element plus a few micro-ops (Golden Cove). AMD’s spreadsheets list the gathers as microcode on Zen 3, Zen 4 and Zen 5 (Zen 3). [Intel 248966, Section 15.16.4, page 15-73; LLVM 23.1.2, X86SchedAlderlakeP.td, line 2224; AMD 56665, latency spreadsheet, row 1000 (VGATHERDD with YMM registers); AMD 57647, latency spreadsheet, row 1476 (VGATHERDD with YMM registers); AMD 58455, latency spreadsheet, row 1935 (VGATHERDPS); AMD 58455, latency spreadsheet, row 1943 (VPGATHERDD)]

The compilers decide per target whether to vectorize with gathers. LLVM uses them only for targets with AVX-512, or with AVX2 and a tuning that marks gathers as fast, which Skylake has and Zen 3 has not. GCC 14.2 builds no gathers at all when tuning for Zen 1, Zen 2 or Zen 4, for Alder Lake, for generic x86-64, or for Intel’s cores from Skylake to Rocket Lake; for Zen 3 it builds only those of eight or more elements, and for Zen 5 and Sapphire Rapids all of them. [LLVM 23.1.2, X86TargetTransformInfo.cpp, line 6688; LLVM 23.1.2, X86.td, line 789; LLVM 23.1.2, X86.td, line 1098; GCC 14.2, config/i386/x86-tune.def, line 485; GCC 14.2, config/i386/x86-tune.def, line 496; GCC 14.2, config/i386/x86-tune.def, line 507; GCC 14.2, config/i386/i386-options.cc, line 154]

A scatter issues a store for every element, so it should never write elements that one vector store could, and Intel replaces scatters with short strides by permutes and full stores. AMD’s spreadsheet lists the scatters of Zen 5 as microcode, and GCC 14.2 builds none when tuning for Zen 4. [Intel 248966, Section 17.11, page 17-39; Intel 248966, Section 17.22.2, page 17-72; AMD 58455, latency spreadsheet, row 1947 (VPSCATTERDD); AMD 58455, latency spreadsheet, row 1952 (VSCATTERDPS); GCC 14.2, config/i386/x86-tune.def, line 491]

Most 256-bit AVX instructions treat their registers as two 128-bit lanes, each handled like the SSE form, and only a few move data from one lane to the other, such as vperm2f128, vinsertf128 and vextractf128. Intel’s way of widening SSE code is to give each lane an iteration of its own: filling the two halves of a register with 16-byte loads and vinsertf128 puts the data in the right lane without any shuffle across lanes. [Intel 248966, Sections 15.4, 15.4.1 and 15.4.2, pages 15-12 to 15-14]

Crossing lanes costs latency. Skylake runs VPERMD in three cycles (Skylake); AMD’s spreadsheet for Zen 5 gives VPSHUFB two cycles at any width but VPERMD four with YMM and five with ZMM registers, and on Zen 4 the VEX form of VPSHUFB takes one. LLVM marks lane-crossing shuffles with indices in a register as fast when tuning for Intel’s cores from Skylake on, and not for any Zen. [LLVM 23.1.2, X86SchedSkylakeClient.td, line 611; AMD 58455, latency spreadsheet, row 384 (VPSHUFB); AMD 58455, latency spreadsheet, row 569 (VPERMD with YMM registers); AMD 58455, latency spreadsheet, row 570 (VPERMD with ZMM registers); AMD 57647, latency spreadsheet, row 2874 (VPSHUFB with YMM registers); LLVM 23.1.2, X86.td, line 653; LLVM 23.1.2, X86.td, line 1106; LLVM 23.1.2, X86.td, line 1691]

Intel’s cores up to Skylake run every shuffle on port 5, which often becomes the bottleneck; Ice Lake and Golden Cove add a unit on port 1 for shuffles within lanes (Golden Cove). Intel relieves port 5 by using blends, which also run on port 0, in place of shuffles (almost 40% faster in its matrix transpose), by inserting halves straight from memory with vinsertf128, and by letting the load ports do simple shuffles such as broadcasts and vmovsldup from memory (Skylake). Zen 4 and Zen 5 run most shuffles on two of their four vector pipes (Zen 5). [Intel 248966, Sections 15.11 to 15.11.3, pages 15-32 to 15-37; Intel 248966, Section 2.4.1, Tables 2-3 and 2-4, pages 2-12 to 2-14; AMD 57647, Section 2.11.1, Table 2, page 36; AMD 58455, Section 2.11.1, page 45]

AVX-512 adds permutes that take any element of two registers, VPERMI2 and VPERMT2 (the first overwrites the index register, the second a table register), and VBMI adds VPERMB, which moves bytes across the whole register. Intel transposes an 8 by 8 matrix of words with them. On Zen 5 each is one macro-op: the two-source permutes of doublewords take three, four and five cycles for 128, 256 and 512 bits, VPERMB two, four and five. [Intel 248966, Sections 17.8 and 17.8.1, pages 17-31 to 17-33; Intel 248966, Section 17.17, page 17-56; AMD 58455, latency spreadsheet, row 571 (VPERMB with XMM registers); AMD 58455, latency spreadsheet, row 573 (VPERMB with ZMM registers); AMD 58455, latency spreadsheet, row 585 (VPERMI2D with XMM registers); AMD 58455, latency spreadsheet, row 587 (VPERMI2D with ZMM registers); AMD 58455, latency spreadsheet, row 603 (VPERMT2D with XMM registers); AMD 58455, latency spreadsheet, row 605 (VPERMT2D with ZMM registers)]

VPCOMPRESSD packs the elements a mask selects to the low end of a register and VPEXPANDD spreads them out again, which replaces the tables of shuffle controls that AVX and AVX2 code needs for the same job. Compressing straight to memory is microcode on Zen 4 (Zen 4), and AMD’s Zen 5 spreadsheet still lists that form as microcode, while LLVM’s model of Sapphire Rapids runs it in four micro-ops (Golden Cove). LLVM also gives the zero-masked forms a false dependency on their destination on Zen 4 and Zen 5, which it no longer assumes for Zen 6. [Intel 248966, Sections 17.5 to 17.6.1, pages 17-20 to 17-26; AMD 57647, Section 2.11.2, page 37; AMD 58455, latency spreadsheet, row 1940 (VPCOMPRESSD to memory); LLVM 23.1.2, X86SchedSapphireRapids.td, line 2757; LLVM 23.1.2, X86.td, line 620; LLVM 23.1.2, X86.td, line 1724; LLVM 23.1.2, X86.td, line 1761]

SIMD code runs best vertically, with each element of a vector belonging to a different item, as in a structure of arrays. Computing on an array of structures leads to horizontal operations that spend a whole vector on one result, so Intel suggests laying the data out by field, or converting it once, and reducing across a vector only at the end, with shuffles such as movhlps and ordinary additions. [Intel 248966, Section 5.5.1, page 5-24; Intel 248966, Section 7.5.1.4, page 7-9]

The horizontal additions of SSE3 save instructions but not time. AMD’s spreadsheets list HADDPS as microcode on Zen 4 and give VHADDPS three or four macro-ops on Zen 5, with a quarter of the throughput of VADDPS; on its Core microarchitecture Intel found a dot product built on HADDPS slightly slower than one built on shuffles, for the latency of HADDPS. LLVM builds them from shuffles and additions only when tuning for AMD’s Jaguar, the one target it marks as running them faster than the sequence they replace. [AMD 57647, latency spreadsheet, row 278 (HADDPS); AMD 57647, latency spreadsheet, row 1518 (VHADDPS); AMD 58455, latency spreadsheet, row 3 (VADDPS); AMD 58455, latency spreadsheet, row 1402 (VHADDPS with XMM registers); AMD 58455, latency spreadsheet, row 1428 (VHADDPS with YMM registers); Intel 248966, Section 7.6.1, page 7-12; LLVM 23.1.2, X86.td, line 835; LLVM 23.1.2, X86.td, line 1588]

A sum or dot product kept in one accumulator runs at the latency of the addition, not at its throughput. Intel’s rule is to keep at least as many independent accumulators as the latency of the critical operation times the number of pipes that run it, eight for Skylake’s four-cycle additions on two ports (Skylake); a gain of up to that factor is possible. [Intel 248966, Section 15.15.1, pages 15-53 and 15-54]

An FMA rounds once, so replacing a multiplication and an addition by one changes results slightly, which iterative code must allow for. Where the chain runs through the addend, a separate addition is faster (Zen 5), and GCC 14.2’s tuning keeps the vectorizer from building loops around tight chains of 128-bit FMAs for Zen 1 to Zen 4 and generic x86-64, and of 256-bit ones for Zen 2 to Zen 4, Alder Lake, Sapphire Rapids and generic x86-64. [Intel 248966, Section 15.15, page 15-52; AMD 58455, Section 2.11.7, page 49; GCC 14.2, config/i386/x86-tune.def, line 516; GCC 14.2, config/i386/x86-tune.def, line 521]

How alignment affects vector loads and stores is in the memory chapter: the wider the vector, the more unaligned accesses split a line. From Skylake on Intel no longer advises splitting unaligned 32-byte loads into 16-byte halves, unless the data is 16-byte aligned. GCC 14.2 still splits unaligned 256-bit loads when tuning for Nehalem and Sandy Bridge, and stores also for Bulldozer and Zen 1, and -mavx256-split-unaligned-load and -store ask for it elsewhere. [Intel 248966, Section 15.6.2, page 15-23; GCC 14.2, config/i386/x86-tune.def, line 541; GCC 14.2, config/i386/x86-tune.def, line 546; GCC 14.2, doc/invoke.texi, line 36080]

When only some buffers can be aligned, Intel aligns the ones that are stored to: a store that splits a line across a page boundary costs about 150 cycles and waits for retirement (memory). In Intel’s tests, misaligning the buffers of a memory-bound AVX-512 loop by 4 bytes cut its speed to between 0.59 and 0.77 of the aligned case, the destination alone to 0.66. [Intel 248966, Section 15.6.3, page 15-25; Intel 248966, Section 17.23.1, Table 17-10, pages 17-75 and 17-76]

Arithmetic on denormal (subnormal) numbers can be slow: Skylake spends about 129 cycles of microcode on a subnormal result, or on a multiplication or division with a subnormal input (Skylake), and AMD’s cores redo multiplications, divisions and square roots with a longer latency (Zen 5). Two MXCSR bits avoid it at the cost of IEEE 754 conformance: FTZ turns results that would be denormal into zeros, DAZ reads denormal inputs as zeros. Intel and AMD both advise setting both where the precision of denormals does not matter; x87 code has no such switch. [Intel 248966, Section 3.9.2.3, pages 3-68 and 3-69; AMD 58455, Section 2.11.4, page 47; Agner Fog, Section 11.9, page 158]

The modes are meant to be set once rather than switched around pieces of code: a read of MXCSR has a long latency and a write serializes the processor, as Intel expects programs to keep the same FTZ and DAZ settings throughout. One MXCSR serves single and double precision alike. [Intel 248966, Section 3.9.3, page 3-69]

Compilers can set the bits when a program starts: GCC 14.2’s -mdaz-ftz sets FTZ and DAZ, though not in shared libraries unless given explicitly, and -ffast-math, through -funsafe-math-optimizations, may link start-up code that changes the default floating-point control word. [GCC 14.2, doc/invoke.texi, line 35081; GCC 14.2, doc/invoke.texi, line 15012]

AVX-512 FP16 handles denormal half-precision values at full speed and ignores FTZ and DAZ, so half-precision code neither needs nor heeds them (Golden Cove). [Intel 248966, Section 18.5.4, page 18-12]

GCC 14.2 and LLVM keep the vectorizer to 256-bit vectors when tuning for Intel’s AVX-512 cores, from the Skylake servers to Granite Rapids, and LLVM also for -march=x86-64-v4; neither does so for Zen 4 or Zen 5, where LLVM uses 512-bit vectors. -mprefer-vector-width=128, 256 or 512 overrides the choice in both, and Intel advises building with each and measuring (Skylake); its own compiler’s -qopt-zmm-usage defaults to low for the Skylake servers. [GCC 14.2, config/i386/x86-tune.def, line 560; GCC 14.2, config/i386/i386-options.cc, line 136; GCC 14.2, config/i386/i386-options.cc, line 3043; GCC 14.2, doc/invoke.texi, line 35533; LLVM 23.1.2, X86.td, line 1139; LLVM 23.1.2, X86.td, line 1213; LLVM 23.1.2, X86.td, line 1013; LLVM 23.1.2, X86.td, line 1727; LLVM 23.1.2, X86Subtarget.cpp, line 300; LLVM 23.1.2, X86TargetTransformInfo.cpp, line 217; Intel 248966, Section 2.6.3, page 2-32; Intel 248966, Section 17.27, page 17-89]

-march=x86-64-v3 (AVX2, FMA, BMI1 and BMI2) and -march=x86-64-v4 (AVX-512 F, BW, CD, DQ and VL) select the psABI’s levels rather than a core: GCC 14.2 tunes such code generically, and LLVM schedules it with its Haswell and Skylake server models. Code built for x86-64-v4 or for Zen 4 may fault on Zen 3, which has no AVX-512 (Zen 3). [GCC 14.2, doc/invoke.texi, line 34250; LLVM 23.1.2, X86.td, line 977; LLVM 23.1.2, X86.td, line 997; LLVM 23.1.2, X86.td, line 2269]

A program can carry several builds of a hot function and choose one when it runs. GCC 14.2’s target_clones attribute compiles a function once for each listed target, such as sse4.1 and avx, and adds a resolver that picks the clone for the processor at hand; C++ code can also declare the versions one by one with the target attribute. Intel’s compiler builds such paths with -ax, but takes the specialized ones only on Intel’s processors and the default path elsewhere. [GCC 14.2, doc/extend.texi, line 4221; GCC 14.2, doc/extend.texi, line 29431; Intel 248966, Section 17.27, page 17-89]