System call conventions on x86-64 and x86

Programs usually reach the kernel through their C library, whose functions such as write move their arguments into place and execute the instruction that enters the kernel. Below the library, each system has its own convention: the instruction, the register that holds the number of the system call, the registers of the arguments, and how an error is reported. It resembles the C convention of the platform, but differs in the details that matter to anyone writing assembly, a runtime library or a sandbox.

Linux on x86-64#

A 64-bit Linux program executes syscall with the number of the system call in RAX and up to six arguments in RDI, RSI, RDX, R10, R8 and R9. The kernel returns the result in RAX; a value from -4095 to -1 is an error number, negated. Nothing is passed on the stack.

The argument registers are those of the C convention but for the fourth, R10 instead of RCX, since the syscall instruction itself overwrites RCX with the return address and R11 with the flags. Those two registers are lost; the kernel preserves all others except RAX. The numbers are in the Linux system call tables, where write is number 1.

/* A system call on x86-64 Linux: the number in RAX, arguments in RDI,
   RSI, RDX, R10, R8 and R9; the kernel returns in RAX and overwrites
   RCX and R11. The six arguments of the C function arrive in the
   registers of the C convention, which differ in the fourth. */
long raw_syscall6(long n, long a, long b, long c, long d, long e, long f)
{
	register long r10 __asm__("r10") = d;
	register long r8 __asm__("r8") = e;
	register long r9 __asm__("r9") = f;
	long ret;

	__asm__ volatile ("syscall"
			  : "=a" (ret)
			  : "a" (n), "D" (a), "S" (b), "d" (c), "r" (r10), "r" (r8), "r" (r9)
			  : "rcx", "r11", "memory");
	return ret;
}

/* write(1, buf, len): system call 1 on x86-64. */
long raw_write(const char *buf, unsigned long len)
{
	long ret;

	__asm__ volatile ("syscall"
			  : "=a" (ret)
			  : "a" (1L), "D" (1L), "S" (buf), "d" (len)
			  : "rcx", "r11", "memory");
	return ret;
}

raw_syscall6:
        movq    %rdi, %rax
        movq    %r8, %r10
        movq    %rsi, %rdi
        movq    %r9, %r8
        movq    %rdx, %rsi
        movq    8(%rsp), %r9
        movq    %rcx, %rdx
        syscall
        ret

raw_write:
        movq    %rdi, %r8
        movl    $1, %eax
        movq    %rsi, %rdx
        movq    %rax, %rdi
        movq    %r8, %rsi
        syscall
        ret

raw_syscall6:
        mov     rax, rdi
        mov     r10, r8
        mov     rdi, rsi
        mov     r8, r9
        mov     rsi, rdx
        mov     r9, QWORD PTR 8[rsp]
        mov     rdx, rcx
        syscall
        ret

raw_write:
        mov     r8, rdi
        mov     eax, 1
        mov     rdx, rsi
        mov     rdi, rax
        mov     rsi, r8
        syscall
        ret

raw_syscall6:
        movq    %r9, %r11
        movq    %rdi, %rax
        movq    8(%rsp), %r9
        movq    %rsi, %rdi
        movq    %rdx, %rsi
        movq    %rcx, %rdx
        movq    %r8, %r10
        movq    %r11, %r8
        syscall
        retq

raw_write:
        movq    %rsi, %rdx
        movq    %rdi, %rsi
        movl    $1, %eax
        movl    $1, %edi
        syscall
        retq

raw_syscall6:
        mov     r11, r9
        mov     rax, rdi
        mov     r9, qword ptr [rsp + 8]
        mov     rdi, rsi
        mov     rsi, rdx
        mov     rdx, rcx
        mov     r10, r8
        mov     r8, r11
        syscall
        ret

raw_write:
        mov     rdx, rsi
        mov     rsi, rdi
        mov     eax, 1
        mov     edi, 1
        syscall
        ret

Linux x32#

x32 programs use the same instruction and registers, and their own table of numbers: the kernel knows an x32 system call by bit 30 of the number, __X32_SYSCALL_BIT, 0x40000000. Most system calls are shared with x86-64 under the same number; those whose arguments contain pointers or long values of a different size have numbers of their own, from 512.

Linux on i386#

A 32-bit Linux program can enter the kernel with int 0x80, the number in EAX and up to six arguments in EBX, ECX, EDX, ESI, EDI and EBP; the result returns in EAX, with errors from -4095 to -1 as on x86-64. The C library prefers a faster path: it calls __kernel_vsyscall, a function of the vDSO that the kernel maps into every process, which enters the kernel with SYSENTER where the processor has it, and with int 0x80 otherwise.

/* write(1, buf, len) on i386 Linux: system call 4 through INT 0x80,
   the number in EAX and the arguments in EBX, ECX and EDX. The
   braces give the instruction in AT&T and in Intel syntax. */
long raw_write(const char *buf, unsigned long len)
{
	long ret;

	__asm__ volatile ("int {$0x80|0x80}"
			  : "=a" (ret)
			  : "a" (4L), "b" (1L), "c" (buf), "d" (len)
			  : "memory");
	return ret;
}

raw_write:
        pushl   %ebx
        movl    $4, %eax
        movl    8(%esp), %ecx
        movl    $1, %ebx
        movl    12(%esp), %edx
        int $0x80
        popl    %ebx
        ret

raw_write:
        push    ebx
        mov     eax, 4
        mov     ecx, DWORD PTR 8[esp]
        mov     ebx, 1
        mov     edx, DWORD PTR 12[esp]
        int 0x80
        pop     ebx
        ret

raw_write:
        pushl   %ebx
        movl    8(%esp), %ecx
        movl    12(%esp), %edx
        movl    $4, %eax
        movl    $1, %ebx
        int     $128
        popl    %ebx
        retl

raw_write:
        push    ebx
        mov     ecx, dword ptr [esp + 8]
        mov     edx, dword ptr [esp + 12]
        mov     eax, 4
        mov     ebx, 1
        int     128
        pop     ebx
        ret

A 64-bit program can execute int 0x80 too, and the kernel then treats the call as one of a 32-bit program: it looks the number up in the i386 table and takes the arguments from EBX, ECX, EDX, ESI, EDI and EBP, 32 bits of each. An int 0x80 copied into 64-bit code therefore makes a different system call than intended, with truncated pointers.

FreeBSD#

FreeBSD on x86-64 uses the registers of Linux: the number in RAX and the arguments in RDI, RSI, RDX, R10, R8 and R9, entered with syscall. Errors are reported differently: the kernel sets the carry flag and returns the positive error number in RAX, and the C library’s wrappers branch on the carry flag to set errno. On i386, FreeBSD’s C library executes int 0x80 inside a function called with the C convention, and the kernel reads the arguments from the user stack, where the call left them.

macOS#

macOS on x86-64 enters the kernel with syscall and passes the arguments in the registers of FreeBSD and Linux, but the number in RAX carries a class in bits 24 and up, since the kernel, XNU, combines a BSD kernel with Mach. The main classes are:

A BSD system call such as write, number 4, is therefore made with 0x2000004 in RAX. Like FreeBSD, macOS sets the carry flag on error.

OpenBSD#

Since OpenBSD 7.5, the kernel accepts each system call only from the place in the C library where that system call’s instruction is: the dynamic linker tells the kernel where they are with pinsyscalls, and a system call from anywhere else ends the program with SIGABRT. Programs therefore have to go through the C library.

Windows#

Windows documents no system call convention. Programs call the Windows API, and the native system services behind it, the Nt and Zw routines, are reached from user mode through system calls whose instruction and numbers Microsoft does not document. The numbers change between Windows releases: NtCreateUserProcess, for example, has had more than half a dozen of them between Windows Vista and Windows 11, according to a table of the numbers of each release. Code that executes syscall with a number of its own may make a different system call, or none, on another Windows build.