Measuring code on x86-64 cores

Three counts of time matter when timing code. The time-stamp counter (TSC) ticks at a fixed rate and goes on through halts, and where it is invariant through every sleep state; core cycles tick at the core’s current clock and stop while the logical processor is halted, so they speed up and slow down with the frequency; reference cycles stop in a halt too but tick at the TSC’s fixed rate. Intel calls core cycles divided by retired instructions the non-halted CPI, the figure for the code itself, and TSC ticks divided by retired instructions the nominal CPI, which includes the time spent waiting, say, for I/O. [Intel 325384, Section 22.7, pages 22-147 and 22-148; Intel 325384, Section 22.2.2, Table 22-1, page 22-7]

Where CPUID 80000007H reports an invariant TSC in EDX bit 8 (the same bit on Intel’s and AMD’s processors), the counter advances at one constant rate in every P-, C- and T-state, so operating systems use it as a clock. That rate is not the clock of the core: Intel puts it close to the highest frequency without turbo, and on Skylake at the guaranteed P1 frequency, and AMD at the processor’s baseline frequency. Under turbo a core therefore runs more cycles than TSC ticks, and a TSC difference measures time, not cycles. [Intel 325384, Sections 20.17.1 and 22.7.2, pages 20-43 and 22-148; Intel 248966, Section 2.7.4, page 2-38; AMD 24593, Section 13.2.4, pages 422 and 423]

On Intel’s processors the TSC runs at a fixed ratio to the core crystal clock. CPUID leaf 15H gives the ratio as EBX divided by EAX and the crystal’s frequency in Hz in ECX, so the TSC’s frequency is ECX times EBX divided by EAX; where ECX is 0 the crystal’s frequency comes from a table in Intel’s manual, 24 MHz for client Skylake parts and 25 MHz for Skylake servers. [Intel 325384, Section 20.17.4, page 20-44; Intel 325384, Section 22.7.3, Table 22-95, page 22-149]

The average frequency of a core over an interval comes from two more counters, which CPUID leaf 6 reports in ECX on both vendors: APERF counts at the frequency the core actually ran, MPERF at a fixed one, and both count only while the core is awake (C0). Their ratio over the interval, times the base (on AMD the P0) frequency, is the effective frequency; only the ratio is defined, not the counts themselves. Intel’s manual also suggests counting core cycles with the performance counters to get the average frequency. [Intel 325384, Section 17.2, page 17-2; Intel 325384, Section 20.17, page 20-43; AMD 24593, Section 17.3, pages 659 to 661]

MPERF and APERF are MSRs that normally only ring 0 can read. On the first Zen cores Agner Fog therefore read APERF and the TSC from a driver before and after each test, and scaled the TSC counts of the test to core cycles with their ratio. AMD’s RDPRU, which its guides for Zen 3 and Zen 5 list, lets a program read MPERF (ECX 0) and APERF (ECX 1) at any privilege level where CPUID 8000_0008H reports it in EBX bit 4; other values of ECX return 0. The CR4.TSD bit that confines RDTSC to ring 0 confines RDPRU too, but RDPRU then raises #UD instead of #GP. In a guest, reads of MPERF are scaled by the hypervisor’s TSC ratio. [Agner Fog, Section 22.20, page 240; AMD 56665, Section 2.1, page 19; AMD 58455, Section 2.1, page 13; AMD 24594, RDPRU, page 298; AMD 24593, Sections 17.3.2 and 17.3.4, pages 661 and 662]

RDTSC is not ordered with the code around it: it may read the counter before earlier instructions have finished, and later ones may start before it has read. Intel’s manual gives the fences that pin it down: an LFENCE just before it makes it wait until the earlier instructions have run and their loads have their data, MFENCE then LFENCE before it waits for earlier stores as well, and an LFENCE just after it keeps later instructions, memory accesses included, from starting before the read. [Intel 325383, RDTSC, page 4-559; Intel 325384, Section 20.17, page 20-43]

RDTSCP already waits for the instructions before it: on Intel’s processors until they have run and their loads have their data, though not until their stores are visible, and on AMD’s until they have retired. Later instructions can still start early, so Intel advises an LFENCE after it, and an MFENCE before it when earlier stores must be included. [Intel 325383, RDTSCP, page 4-561; AMD 24594, RDTSCP, page 435]

On AMD’s processors LFENCE holds back later instructions only where it is dispatch serializing: it then retires, with everything before it, before anything after it executes. CPUID 8000_0021H reports in EAX bit 2 that LFENCE always works that way; otherwise it depends on how the system has configured it, and LFENCE may order no more than loads. [AMD 24593, Sections 7.6.3 and 7.6.4, pages 211 and 212]

CPUID serializes: everything before it, stores included, completes before the following instruction is even fetched, and Intel’s manual suggests it around RDPMC for exact counts. It is a costly fence: it overwrites four registers, runs as a long microcode sequence, and inside a virtual machine on Intel’s processors it always exits to the hypervisor. Intel’s manual prefers SERIALIZE on processors that have it. [Intel 325383, CPUID, page 3-203; Intel 325384, Sections 11.3 and 28.1.2, pages 11-18 and 28-2; Intel 248966, Section 22.1.2, page 22-4]

A pair of TSC readings around a few instructions says little. AMD’s manual warns against using the TSC for very short intervals: repeated measurements of the same code differ, and even between serializing instructions the state of the system can add cycles before the value is read. The counter itself is reliable over long ones: Intel guarantees that it returns a larger value at every read and does not wrap within ten years of a reset. [AMD 24593, Section 13.2.4, page 422; Intel 325384, Section 20.17, page 20-43]

Every logical processor keeps a TSC of its own, which privileged software can write, so two processors’ counters need not agree; Intel’s TSC_ADJUST MSR helps the operating system keep them in step. RDTSCP returns the TSC_AUX value that the operating system keeps per processor in the same step as the count, which lets a program notice that it moved to another processor between two readings and discard the interval. [Intel 325384, Sections 20.17.2 and 20.17.3, pages 20-43 and 20-44; AMD 24594, RDTSCP, page 435]

An operating system can confine RDTSC and RDTSCP to ring 0 by setting CR4.TSD, after which they raise #GP in user code; Intel expects a system that does so to offer the time through a call of its own instead. [Intel 325384, Section 20.17, page 20-43; AMD 24593, Section 13.2.4, page 422]

Intel’s architectural performance monitoring is described by CPUID leaf 0AH: a version in EAX bits 7:0, how many general-purpose counters a logical processor has in bits 15:8 and their width in bits 23:16, and how many fixed counters in EDX. Every logical processor keeps counters and event selectors of its own, not shared with its sibling, and a hypervisor may show a guest fewer counters than the hardware has. [Intel 325384, Sections 22.2 and 22.2.1, pages 22-2 and 22-3; Intel 325383, RDPMC, page 4-551]

The fixed counters each count one event, with no event to choose: fixed counter 0 retired instructions, 1 core cycles outside a halt, 2 reference cycles, at the TSC’s rate, outside a halt, and 3, on cores that have it, the issue slots that the top-down method divides up. Fixed counter 2 approximates the time a thread spent awake; fixed counter 1, the cycles it took. [Intel 325384, Section 22.2.2, Table 22-1, pages 22-5 to 22-7; Intel 325384, Section 22.7.1, page 22-148]

AMD’s processors have four core counters, six where CPUID 8000_0001H sets PerfCtrExtCore, and CPUID 8000_0022H gives their number on newer ones; Zen 5 has six for each thread. Which events they count differs between processors, so AMD documents the events in each processor’s PPR rather than in its architecture manual, and its Zen 5 guide sends readers to a PPR for counters and profiling tools instead of describing a method of analysis like Intel’s top-down one. [AMD 24593, Sections 13.2.1 and 13.2.2, pages 411, 420 and 421; AMD 58455, Section 2.1, page 15; AMD 58455, Section 1.2, page 9]

RDPMC takes the counter’s number in ECX and returns the count in EDX:EAX; on Intel’s processors the bits above the counter’s width read as zero. The vendors number the counters differently: on Intel’s processors ECX bits 31:16 choose the kind, 0 for general-purpose counters, 4000H for fixed ones and 2000H for the top-down metrics, and bits 15:0 the counter; on AMD’s 0 to 5 are the core counters and higher numbers those of the northbridge and L3. [Intel 325383, RDPMC, page 4-551; AMD 24594, RDPMC, page 430]

Outside ring 0, RDPMC works only while the operating system has set CR4.PCE, and raises #GP otherwise or for a counter that does not exist. Linux decides with the rdpmc file of each CPU PMU in /sys/bus/event_source/devices: 0 forbids it, 2 allows it always, and 1, the default, allows it only while the program has a perf event’s page mapped. [Intel 325383, RDPMC, page 4-551; AMD 24593, Section 13.2.1, page 412; Linux 7.2, ABI/testing/sysfs-bus-event_source-devices-rdpmc, line 5]

RDPMC waits for nothing either: events of earlier instructions may not be counted yet, and later ones may already count. For an exact count Intel’s manual puts a serializing instruction such as CPUID before or after it, and between two fast reads, which otherwise need not come out in order. AMD’s warns that counts vary from run to run and are no tool for very short sequences. [Intel 325383, RDPMC, page 4-552; AMD 24593, Section 13.2.1, page 412]

Programs usually reach the counters through Linux’s perf events, whose sysctl kernel.perf_event_paranoid sets what a user without CAP_PERFMON may do: from 1 up such a user may not count the events of a whole CPU, at 2, the default, not profile the kernel either, and at -1 almost every event is open to all. [Linux 7.2, admin-guide/sysctl/kernel.rst, line 992]

A counter can also raise an interrupt when it overflows, which is how a profiler samples: it preloads the counter with minus the interval, and the interrupt handler records what it needs and reloads it. AMD’s Instruction-Based Sampling goes further: it picks a fetch block or an operation at random and has the hardware record what happened to it, such as the instruction cache and ITLB behaviour of a fetch, before interrupting. [AMD 24593, Sections 13.2.1 and 13.2.3.2, pages 412 and 421; AMD 24593, Section 13.3, page 423]

A sample lands near the instruction responsible, not always on it. Intel’s precise events land on the next instruction to retire (for a branch, its target); others drift further, less the closer the event is to retirement, and front-end events may land before the cause. Cycle samples tend to pile up on the instruction after a slow one, and within a basic block every instruction runs equally often, so averaging the samples over the block counts its executions better than any single instruction’s tally. [Intel 248966, Section 22.5.1, pages 22-45 and 22-46; Intel 248966, Section 22.4.1.2, page 22-21]

Intel’s manual uses Linux perf with its own event names: perf stat -e assists.sse_avx_mix to find transitions between VEX and legacy SSE code, perf mem record --ldlat 128 to sample loads that took over 128 cycles together with their addresses, and perf record -e frontend_retired.itlb_miss to find code that misses the ITLB; perf c2c finds false sharing (Memory). [Intel 248966, Section 3.11.6.2, page 3-81; Intel 248966, Section 3.11.2, pages 3-73 and 3-74; Intel 248966, Appendix B, Figure B-7, page B-15]

Intel’s top-down method (TMA) accounts for issue slots, the places for micro-ops at the allocation stage, as many per cycle as the narrowest stage of the pipeline is wide. Every slot falls in one of four classes, which add up to the whole: Retiring when it held a micro-op that retired, Bad Speculation when it held one that was thrown away or was lost to recovering from a wrong guess, Backend Bound when the back end could not take a micro-op, and Frontend Bound when the front end had none to give. [Intel 248966, Section 22.1.1, pages 22-2 and 22-3; Intel 325384, Section 22.2.2, Table 22-1, page 22-7]

The classes split further into a tree, and the analysis follows the largest share down to a leaf rather than looking at every event. Bad Speculation comes first, since counts taken on the wrong path distort the rest (Branches); front-end problems deserve attention only when Frontend Bound is high at the top level. Intel’s VTune applies the method, and Intel publishes the full tree with the events of each generation at download.01.org/perfmon. [Intel 248966, Sections 22.1 and 22.1.2, pages 22-1 to 22-4; Intel 248966, Section 22.1.6, pages 22-5 and 22-6; Intel 248966, Section 22.1.11, page 22-7]

Frontend Bound divides into fetch latency, when no micro-ops arrive at all, as after an instruction cache or ITLB miss or a redirect, and fetch bandwidth, when too few arrive, as from the decoders (Front end). Backend Bound divides into Memory Bound, stalls waiting for one of the cache levels or for DRAM, whose bandwidth Intel blames when the queue of requests to memory is over about 70% full, and Core Bound, busy execution units or too little parallelism in the code (Out of order, Memory). [Intel 248966, Sections 22.1.2 to 22.1.5, pages 22-4 and 22-5]

A high Retiring share does not mean the code is as fast as it can be. Micro-ops of microcode, such as floating-point assists, count as Retiring though they could often be avoided, and scalar code that retires well may gain most from vectors, which do more work per micro-op. [Intel 248966, Section 22.1.7, page 22-6]

From Ice Lake on, Intel’s cores keep the four top-level shares themselves, in the PERF_METRICS MSR, as fractions of 255 of the slots that fixed counter 3 counts; Golden Cove’s add four second-level shares, from which the other four follow by subtraction. RDPMC reads them with ECX 2000_0000H, and can clear them as it reads where the processor supports that. [Intel 325384, Section 22.3.9.3, pages 22-74 and 22-75; Intel 325384, Section 22.3.10.1.1, page 22-77; Intel 325383, RDPMC, pages 4-551 and 4-552]

Turbo makes the frequency a variable of the test. Intel’s cores run between P1 and P0 when the system asks for P0, reaching the top only with one busy core and less with more, but under normal conditions not below P1; AMD’s go above P0 while their estimated power stays under a limit, unless boost is switched off in the HWCR MSR. Agner Fog saw Zen cores run from 8% of nominal while waiting for a disk to 120% (on Zen 2) after long busy code, and lose the top speed once all cores were busy. [Intel 248966, Section 16.3.2, page 16-5; AMD 24593, Section 17.2, page 659; Agner Fog, Section 22.20, page 240]

A test should let the processor reach a steady speed first. Agner Fog advises running a long stretch of busy code before the tested code, or turning power saving off, and still found Zen’s clock counts somewhat inexact; on Skylake the upper halves of the vector units also need waking (Skylake). Intel’s own test of the FMA units of a Skylake server warms up with AVX-512 code and runs each test three times. [Agner Fog, Section 21.18, page 230; Agner Fog, Section 22.20, page 240; Intel 248966, Section 17.21, page 17-67]

What the code runs can set the frequency too. Skylake’s servers lower a core’s turbo limit for heavy AVX2 and AVX-512 code, take up to 500 microseconds to switch and some 2 ms to switch back (Skylake, switching), and AMX has a limit of its own, usually lower still, on Golden Cove’s servers (Golden Cove); a benchmark of such code alone can miss its cost to the scalar code around it. [Intel 248966, Section 2.6.3, pages 2-29 to 2-31; Intel 248966, Section 20.2.1, page 20-2]

Idle time leaves its marks as well. In deep C-states a core may lose the contents of its caches (on Intel’s mobile parts the L3 is flushed in C7), so code that runs after a pause starts cold, and a high rate of interrupts both keeps the processor out of deep sleep and lowers the turbo frequency it can reach. [Intel 248966, Sections 16.3.2 and 16.3.3, pages 16-4 and 16-5; Intel 248966, Section 16.4, page 16-15]

How a test calls the code changes the result. Intel’s example is memset: timing one loop of many identical calls trains the branch predictor on the routine’s branches and favours an unrolled vector version, while real programs call it with varying sizes and other code in between. Timing each call with its own pair of TSC readings, with their overhead measured beforehand, showed rep stosb faster below a few hundred bytes in the usual patterns. [Intel 248966, Section 15.16.3.3, pages 15-62 and 15-63]

A second thread on the same core changes what the first one measures, since they share the core’s caches and units (Memory). The counters belong to each logical processor, but not every event can tell the threads apart: Intel’s issue slots are shared among the threads of a core, and on its older cores the events of the load and store ports counted only per core, which makes an analysis per thread simplest with Hyper-Threading off. [Intel 325384, Section 22.2.2, Table 22-1, page 22-7; Intel 248966, Section 22.4.1, page 22-17]

In a virtual machine the TSC a guest reads may be the host’s scaled by a multiplier and moved by an offset that the hypervisor sets, on Intel’s processors through the TSC offset and multiplier of the VMCS and on AMD’s through the TSC offset of the VMCB and the TSC ratio, which scales MPERF as well. The guest may also see fewer counters than the hardware has, and every CPUID exits to the hypervisor. [Intel 325384, Section 27.6.5, page 27-15; AMD 24593, Section 15.30.5, page 585; Intel 325384, Section 22.2.1, page 22-3]