VPGATHERDQ

VPGATHERDQ loads quadwords from scattered addresses, each a base plus a signed 32-bit index times a scale of 1, 2, 4 or 8. It is the integer twin of VGATHERDPD, whose page shows how half as many index bytes as data bytes shape the registers: 2, 4 or 8 quadwords take their indices from an XMM, XMM or YMM register. VGATHERDPS explains the rest, the mask that selects elements and is cleared as they load, the #UD register rules and the faults it resumes after; VPGATHERQQ compares the index widths.

Recorded on Zen 5#

• Negative indices. vpgatherdq ymm4, qword ptr [base + xmm9*8], ymm11 loaded elements 1 to 3 from indices 5, 7 and -5, the last 40 bytes below the base, kept element 0, whose mask element had a clear sign bit, and cleared all of YMM11.
• A mask register. vpgatherdq zmm1{k5}, qword ptr [base + ymm3*8] loaded elements 0, 6 and 7 by C1h, the low byte of K5, and left all 64 bits of K5 zero; the other five elements kept their values.

In compiled code#

For d[i] = t[idx[i]] with int indices and int64_t data, Clang 23 at -O2 gathers four quadwords at a time from XMM indices with -march=x86-64-v4 and eight from YMM indices with znver5, each after a KXNORW or KXNORB has set the mask to all ones; GCC 14 keeps the loop scalar. For _mm512_i32gather_epi64 GCC sets the mask with KXNORB for x86-64-v4 but loads it from a constant in memory for znver5.

AMD’s Zen 5 latency spreadsheet lists VPGATHERDQ as microcoded, with no latency, and its guide advises vector loads and shuffles instead of a gather when the indices are known in advance (Section 2.11.2).

In the optimization guide

What the optimization guide says about VPGATHERDQ, a line per claim. Each links to the claim in the guide, with its sources and where they differ.

Vectors

• Gathers and scatters: Where the layout of the data is known, vector loads and shuffles usually beat a gather.
• Gathers and scatters: Gathers beat scalar loads on Skylake, while AMD lists them as microcode on Zen 3, Zen 4 and Zen 5.

Crestmont and Gracemont

• Vectors and AVX2: Gathers are microcoded on these cores and take about 50 cycles.

Golden Cove and Raptor Cove

• Vectors, AVX-512 and AMX: A gather is a load micro-op for each element plus a few more: LLVM’s models give an eight-element VPGATHERDD 12 micro-ops and a latency of 30 cycles, and the 16-element one of Sapphire Rapids 19 micro-ops.

Haswell and Broadwell

• Gathers: Haswell is the first core with gathers, and each one runs as many micro-ops; Broadwell runs them much faster.

Skylake to Cooper Lake

• Vectors on all parts: Gathers are faster than on Broadwell and beat a sequence of scalar loads unless they fetch only two elements or the data comes from memory rather than the caches.

Zen 2

• Vectors and floating point: Gathers such as VPGATHERDD are microcoded, in AMD’s spreadsheet and LLVM’s model, so vector loads and shuffles beat them when the indices are known in advance.

Zen 3

• Vectors and floating point: Gathers such as VPGATHERDD are microcoded, and a register loaded in pieces is slower than one full-width load, so vector loads and shuffles beat both when the indices are known in advance.

Zen 4

• Vectors and floating point: A gather is slower than vector loads and shuffles when the indices are known in advance, and a register loaded in pieces is slower than one full-width load.

Zen 5

• Vectors and floating point: A gather is slower than vector loads and shuffles when the indices are known in advance.

Forms

Latency and throughput

Data flow

Where the elements of the sources go, element 0 on the right; results that depend on a control value are drawn for an example. How to read.

Operation

What the forms do, in pseudocode that our tests run against results recorded on real processors. How to read.

x = mem.index.number
if x in (ymm1.number, ymm3.number):
    fault(UD)
if ymm1.number == ymm3.number:
    fault(UD)
for i in range(VL // 64):
    if ymm3.sq[i] < 0:
        a = mem.index.sd[i] * mem.scale
        ymm1.q[i] = load(mem.base + a, 64)
ymm3.write(0)
# ymm1 is zeroed from bit VL up

if mem.index.number == zmm1.number:
    fault(UD)
for i in range(VL // 64):
    if k1[i]:
        a = mem.index.sd[i] * mem.scale
        zmm1.q[i] = load(mem.base + a, 64)
k1.write(0)  # all of it
# zmm1 is zeroed from bit VL up

Operands

Flags

None: XED records no flag that the forms test or change.

Exceptions

The exception class is XED's name for the set of exception conditions that a form follows. How to read.

VEX forms

avx-type-12

EVEX forms

AVX512-E12

Attributes

What else XED records about the forms. How to read.

All forms

gather

EVEX forms

write masking; memory fault suppression

Intrinsics

The C intrinsics of Clang 23.1.2 that compile to VPGATHERDQ, each with the form it compiles to and the Clang options it needs. The forms come from compiling every intrinsic and decoding the result with Intel XED; all intrinsics A–Z explains the method.

CPUID and processors

Each form belongs to an XED ISA set. A form can be used when the processor reports every CPUID bit of one of its ISA set's alternatives.

More AVX2 instructions

• VPBLENDVB
• VPBLENDW
• VPBROADCASTB
• VPBROADCASTD
• VPBROADCASTQ
• VPBROADCASTW
• VPCMPEQB
• VPCMPEQD
• VPCMPEQQ
• VPCMPEQW
• VPCMPGTB
• VPCMPGTD
• VPCMPGTQ
• VPCMPGTW
• VPERM2I128
• VPERMD
• VPERMPD
• VPERMPS
• VPERMQ
• VPGATHERDD
• VPGATHERQD
• VPGATHERQQ
• VPHADDD
• VPHADDSW
• VPHADDW
• VPHSUBD
• VPHSUBSW
• VPHSUBW
• VPMADDUBSW
• VPMADDWD
• VPMASKMOVD
• VPMASKMOVQ
• VPMAXSB
• VPMAXSD
• VPMAXSW
• VPMAXUB
• VPMAXUD
• VPMAXUW
• VPMINSB
• VPMINSD

All 138 AVX2 instructions

More AVX10.1 instructions

• VPERMILPS
• VPERMPD
• VPERMPS
• VPERMQ
• VPERMT2B
• VPERMT2D
• VPERMT2PD
• VPERMT2PS
• VPERMT2Q
• VPERMT2W
• VPERMW
• VPEXPANDB
• VPEXPANDD
• VPEXPANDQ
• VPEXPANDW
• VPEXTRB
• VPEXTRD
• VPEXTRQ
• VPEXTRW
• VPGATHERDD
• VPGATHERQD
• VPGATHERQQ
• VPINSRB
• VPINSRD
• VPINSRQ
• VPINSRW
• VPLZCNTD
• VPLZCNTQ
• VPMADD52HUQ
• VPMADD52LUQ
• VPMADDUBSW
• VPMADDWD
• VPMAXSB
• VPMAXSD
• VPMAXSQ
• VPMAXSW
• VPMAXUB
• VPMAXUD
• VPMAXUQ
• VPMAXUW

All 686 AVX10.1 instructions