AVX, AVX-512, APX and AMX in the calling conventions

The x86-64 ABIs were written for SSE, and each extension since has added registers that calls must deal with: wider vector registers with AVX and AVX-512, mask registers, sixteen more general-purpose registers with APX, and the tiles of AMX. This page collects the rules for them from both 64-bit ABIs, System V and Microsoft x64; the extension pages describe the instructions.

__m256 and __m512 arguments#

In System V, a __m256 argument is one SSE eightbyte followed by three SSEUP ones, and so fills a YMM register; a __m512 fills a ZMM register. That holds only when the code is compiled for a processor with those registers. Compiled without AVX, the same function takes the vector in memory, so caller and callee must be compiled with the same vector options. GCC and Clang warn about such functions (-Wpsabi): “AVX vector argument without AVX enabled changes the ABI”. Variadic calls pass these types on the stack in any case.

/* A 32-byte vector, like AVX's __m256. Whether it travels in a YMM
   register depends on the processor the code is compiled for. */
typedef float v8sf __attribute__((vector_size(32)));

v8sf add8(v8sf a, v8sf b)
{
	return a + b;
}

#ifdef _WIN32
v8sf __vectorcall add8_vc(v8sf a, v8sf b)
{
	return a + b;
}
#endif

add8:
        movaps  8(%rsp), %xmm0
        addps   40(%rsp), %xmm0
        movq    %rdi, %rax
        movaps  %xmm0, (%rdi)
        movaps  24(%rsp), %xmm0
        addps   56(%rsp), %xmm0
        movaps  %xmm0, 16(%rdi)
        ret

add8:
        movaps  xmm0, XMMWORD PTR 8[rsp]
        addps   xmm0, XMMWORD PTR 40[rsp]
        mov     rax, rdi
        movaps  XMMWORD PTR [rdi], xmm0
        movaps  xmm0, XMMWORD PTR 24[rsp]
        addps   xmm0, XMMWORD PTR 56[rsp]
        movaps  XMMWORD PTR 16[rdi], xmm0
        ret

add8:
        vaddps  %ymm1, %ymm0, %ymm0
        ret

add8:
        vaddps  ymm0, ymm0, ymm1
        ret

add8:
        vmovaps (%rcx), %ymm0
        vaddps  (%rdx), %ymm0, %ymm0
        retq

add8_vc@@64:
        vaddps  %ymm1, %ymm0, %ymm0
        retq

add8:
        vmovaps ymm0, ymmword ptr [rcx]
        vaddps  ymm0, ymm0, ymmword ptr [rdx]
        ret

add8_vc@@64:
        vaddps  ymm0, ymm0, ymm1
        ret

/* A 64-byte vector, like AVX-512's __m512, in ZMM registers. */
typedef float v16sf __attribute__((vector_size(64)));

v16sf add16(v16sf a, v16sf b)
{
	return a + b;
}

add16:
        vaddps  %zmm1, %zmm0, %zmm0
        ret

add16:
        vaddps  zmm0, zmm0, zmm1
        ret

add16:
        vaddps  %zmm1, %zmm0, %zmm0
        retq

add16:
        vaddps  zmm0, zmm0, zmm1
        ret

Microsoft x64 passes every vector wider than 8 bytes by reference, whatever the processor; only __vectorcall passes __m256 values in YMM registers. Apple’s documentation for macOS points out the same dependence in System V: how a vector wider than 16 bytes is passed depends on the processor features the code is compiled for.

VZEROUPPER#

The ABIs do not say who owns the upper halves of the YMM and ZMM registers across a call: they are scratch. But processors run instructions of the older SSE encoding slowly while those upper halves hold data, so compilers clear them with VZEROUPPER before a function that used them returns or calls another function, which might use SSE. GCC and Clang do so by default when they compile for AVX; GCC’s manual gives the reason under its -mvzeroupper option.

/* After using the upper halves of the YMM registers, a function clears
   them with VZEROUPPER before it calls or returns to code that may use
   SSE instructions. */
typedef float v8sf __attribute__((vector_size(32)));

extern void next(void);

void square(v8sf *p)
{
	*p = *p * *p;
	next();
}

square:
        vmovaps (%rdi), %ymm0
        vmulps  %ymm0, %ymm0, %ymm0
        vmovaps %ymm0, (%rdi)
        vzeroupper
        jmp     next@PLT

square:
        vmovaps ymm0, YMMWORD PTR [rdi]
        vmulps  ymm0, ymm0, ymm0
        vmovaps YMMWORD PTR [rdi], ymm0
        vzeroupper
        jmp     next@PLT

square:
        vmovaps (%rdi), %ymm0
        vmulps  %ymm0, %ymm0, %ymm0
        vmovaps %ymm0, (%rdi)
        vzeroupper
        jmp     next@PLT

square:
        vmovaps ymm0, ymmword ptr [rdi]
        vmulps  ymm0, ymm0, ymm0
        vmovaps ymmword ptr [rdi], ymm0
        vzeroupper
        jmp     next@PLT

AVX-512 mask registers#

The eight mask registers K0 to K7 are scratch in System V: a function may change them, and none carries an argument or a result. A mask passed to a function is an integer, such as the 16-bit __mmask16, and travels in a general-purpose register. Microsoft’s convention does not mention the mask registers; it makes XMM16 to XMM31 scratch.

/* AVX-512 compares produce a mask in a K register, which selects
   elements; the mask registers are scratch in every ABI. */
typedef int v16si __attribute__((vector_size(64)));

v16si pick(v16si a, v16si b, v16si c)
{
	v16si m = a > b;

	return (m & a) | (~m & c);
}

pick:
        vpcmpd  $6, %zmm1, %zmm0, %k1
        vpblendmd       %zmm0, %zmm2, %zmm0{%k1}
        ret

pick:
        vpcmpd  k1, zmm0, zmm1, 6
        vpblendmd       zmm0{k1}, zmm2, zmm0
        ret

pick:
        vpcmpgtd        %zmm1, %zmm0, %k1
        vpblendmd       %zmm0, %zmm2, %zmm0 {%k1}
        retq

pick:
        vpcmpgtd        k1, zmm0, zmm1
        vpblendmd       zmm0 {k1}, zmm2, zmm0
        ret

APX: R16 to R31#

APX adds sixteen general-purpose registers. The two 64-bit ABIs treat them differently:

Microsoft adds a caveat: jmp_buf has no room for R30 and R31, so a function must not change them between setjmp and the call that leads to longjmp. None of the new registers carries arguments, so code compiled for APX calls code compiled without it, and the other way round. The APX extension page lists the instructions.

AMX tiles#

The eight tile registers TMM0 to TMM7 of AMX are scratch in both ABIs, and so is the tile configuration, TILECFG, in System V: a function that uses tiles configures them with LDTILECFG, and TILERELEASE returns them to their initial state.

Linux adds a step before the first use. The tile data is 8 KB of state that the kernel does not allocate for every thread, so a process must ask for it with arch_prctl(ARCH_REQ_XCOMP_PERM, 18), 18 being the number of the XSAVE component of the tile data. Until it has, the first AMX instruction raises SIGILL; afterwards, the kernel allocates the larger buffer for each thread when it first uses the tiles. The permission holds for the whole process, survives fork and is cleared by exec. See the AMX tile extension for the instructions.