ENQCMD

ENQCMD hands a 64-byte command to a device that many programs share, such as an accelerator with a shared work queue, without a system call. It reads the command from memory, stamps it with the thread’s process address space identifier (PASID), writes it in one piece to the device’s enqueue register, whose address is in a general-purpose register, and learns in ZF whether the device took it. It shares the opcode 0F 38 F8 with MOVDIR64B, whose 64-byte write it resembles, and with ENQCMDS, the kernel’s variant that takes the PASID and the privilege bit from the command itself.

What the command carries#

Bits 19:0 of the written command are the PASID from the IA32_PASID MSR, bit 31 is 0 to say the sender runs as a user, whatever the current privilege level, and bits 511:32 come from the source. The low 32 bits of the source must be 0, the MSR’s valid bit set and the destination aligned to 64 bytes, or ENQCMD raises #GP. Only the operating system writes the MSR, so a program cannot pose as another address space, and a device can rely on the PASID to tell whose addresses a command holds. The MSR is supervisor state component 10 of XSAVE.

Accepted or not#

ZF = 0 means the device accepted the command, not that it ran it. ZF = 1 asks for a retry: the queue was full, or the address was no enqueue register at all, in which case nothing is written anywhere. The other arithmetic flags are cleared. The intrinsic _enqcmd(portal, desc) (-menqcmd) returns ZF, so the usual loop retries while it returns 1: GCC 14 at -O2 compiles it to enqcmd rdi, zmmword ptr [rsi], SETE and a test, Clang 23 to ENQCMD and a JNE on the flag directly, its loop unrolled five times.

Like the direct stores, the write bypasses the caches and is not ordered after earlier stores to write-back or write-combining memory, so a command that points at data written just before needs an SFENCE first.

Where it runs#

The page lists Intel’s server processors from Sapphire Rapids on and no AMD processor; Zen 5 reports no ENQCMD (CPUID leaf 7, ECX bit 29), so the page’s pseudocode is written from Intel’s manual and was not run. APX adds an EVEX form for its new registers.

Forms

Latency and throughput

Operation

What the forms do, in pseudocode that our tests run against results recorded on real processors. How to read.

a = dest.read()                  # 64 bits, 32 with a 67 prefix; ES,
                                 # which no prefix overrides
if a % 64:
    fault(GP, 0)                 # 64-byte aligned
p = msr[0xD93]                   # IA32_PASID
if not p >> 31 & 1:
    fault(GP, 0)                 # the OS gave the thread no PASID
v = src.read()                   # 64 bytes, not read atomically
if v & 0xFFFFFFFF:
    fault(GP, 0)                 # bits 31:0 must be 0
c = v | p & 0xFFFFF              # the PASID in bits 19:0; bit 31,
                                 # 0: a user's command
zf = 1 - enqueue(a, c)
of = sf = af = pf = cf = 0

Operands

Flags

Exceptions

The exception class is XED's name for the set of exception conditions that a form follows. How to read.

Legacy forms

none in XED

EVEX forms

APX-EVEX-ENQCMD

CPUID and processors

Each form belongs to an XED ISA set. A form can be used when the processor reports every CPUID bit of one of its ISA set's alternatives.

More ENQCMD instructions

• ENQCMDS

More APX instructions

• ADCX
• ADD
• ADOX
• AND
• ANDN
• AOR
• AXOR
• BEXTR
• BLSI
• BLSMSK
• BLSR
• BZHI
• CCMPscc
• CFCMOVcc
• CMOVcc
• CMPccXADD
• CRC32
• CTESTscc
• DEC
• DIV
• ENQCMDS
• IDIV
• IMUL
• INC
• INVEPT
• INVPCID
• INVVPID
• JMPABS
• KMOVB
• KMOVD
• KMOVQ
• KMOVW
• LDTILECFG
• LZCNT
• MOVBE
• MOVDIR64B
• MOVDIRI
• MOVRS
• MUL
• MULX

All 88 APX instructions