NX and the End of Injected Shellcode

Before hardware support for no-execute pages became standard, an exploit had one dominant shape: send bytes into a buffer, overwrite the return address with the address of that buffer, and let the processor execute the data. The payload was the input. Nothing in the program had to already exist for the attack to work, which meant that an overflow on a stack holding attacker-controlled text was usually enough on its own, with no library addresses to know and no code layout to study.

Non-executable memory broke that shape. A page marked non-executable raises a fault when the instruction pointer reaches it, and since every writable page a process owns is by default non-executable, the payload cannot run where it was written. This is often described as write-xor-execute: a page may be writable or executable, but not both at once.

The bit and where it lives

x86-64 puts the permission in the page table entry. Each level of the four-level page table contributes permission bits, and the execute-disable bit at each level is combined so that a page is executable only if every level on the path allows it. A single 1 in the NX bit of any entry along the walk makes the final page non-executable. The bit sits in the high part of the entry, alongside the present and writable bits, so changing it means walking to the correct table and modifying a 64-bit entry.

Page tables are per-process, so the marking is per-process, and the same physical page can be executable in one address space and non-executable in another. That property is what makes shared file-backed mappings workable: the text of libc is mapped executable in every process that uses it, while a heap page allocated by one process is marked non-executable in that process alone.

The initial marking is decided when the mapping is created. mmap with PROT_READ|PROT_WRITE produces a non-executable region; adding PROT_EXEC produces an executable one. mprotect can change the marking later, but only to values permitted by the kernel, and on systems that enforce W^X strictly the kernel refuses to make a mapping both writable and executable even when asked. The refusal is the policy, not the mechanism: the hardware would allow the combination, and the operating system declines to create it.

/* A new mapping is executable only if PROT_EXEC is requested. */
void *p = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

/* Writing code into p is legal; jumping to it is not. Turning the
   page executable afterwards is what a JIT does, and it is a syscall. */
mprotect(p, 4096, PROT_READ | PROT_EXEC);

The two calls involve a page-table walk, a TLB invalidation for the range, and on a multiprocessor system an inter-processor interrupt to every CPU that has the mapping cached. That last cost is what makes frequent permission changes expensive and why generated-code systems batch them.

What the fault looks like and what it means for the attacker

An attempt to execute from a non-executable page raises a protection fault with a specific error code. On Linux the process receives SIGSEGV with si_code set to SEGV_ACCERR, distinguishing it from an access to an unmapped page (SEGV_MAPERR). The distinction matters for debugging and for exploit development, because a crash on an unmapped page usually means the overwrite wrote a wrong address, while a fault on a mapped but non-executable page means the address was right and only the permission was wrong.

That second case is the interesting one. It confirms that the control-flow hijack succeeded and that the target address was computed correctly, and it fails only because the payload cannot run there. The natural response is to stop trying to execute new code and start reusing code that is already executable. A return into a library function does the same job as shellcode for many purposes: system("/bin/sh") needs no injected instructions at all, only a stack arranged so that the function receives its argument. That technique is the subject of return-to-libc chains, and it is the direct descendant of the shellcode era.

The reuse does not have to be a whole function. A sequence of instructions ending in ret, taken from anywhere in the executable’s text, can be chained so that each fragment performs one operation and returns to the next. Return-oriented programming generalises the idea to arbitrary computation, and it exists because NX made injection impossible while leaving the code already present fully usable.

The cost on legitimate programs

A just-in-time compiler produces machine code at runtime, and that code must be executed. Under strict W^X it cannot be written and executed in the same mapping, so the compiler must allocate a writable region, emit code into it, then change the permission to executable before calling into it. Every transition costs a system call and a TLB invalidation for the affected range.

The usual arrangement is a pair of mappings over the same physical pages: one writable and one executable. The compiler writes through the writable alias and executes through the executable one, and because the two mappings share physical memory, no copy is needed. This is what memfd_create with MFD_EXEC and the older dual-mapping trick accomplish, and it is why the kernel exposes a way to create an executable file descriptor rather than requiring callers to go through a temporary file.

The alias has a cost of its own: the instruction cache is indexed by virtual address, so the same code reached through two different virtual addresses can occupy two sets of cache lines. A JIT that writes through one alias and executes through the other pays for the code twice in the instruction cache, and for large generated code this is measurable.

Signals and debuggers add another wrinkle. A debugger that wants to set a breakpoint writes an int3 byte into the text of the process, which requires the text page to be writable for the duration of the write. The kernel’s ptrace interface handles this by temporarily changing the mapping’s permission, which is invisible to the traced program but means that text is not permanently immutable in a process under debug.

Threads, and the moment of the switch

The permission change is not atomic with respect to other threads. If one thread is executing code in a region while another calls mprotect to remove execute permission, the running thread may continue executing instructions already in the pipeline and then fault on the next fetch. Kernels handle this with a TLB shootdown: the calling thread sends inter-processor interrupts to every CPU that has the mapping in its TLB and waits for acknowledgement before returning.

The shootdown is expensive. Its cost scales with the number of CPUs running threads that share the address space, and a JIT that flips permissions frequently in a multithreaded process pays that cost repeatedly. This is the reason production JITs batch permission changes: they emit into a large writable region and switch the whole region at once, rather than flipping per function.

There is also a correctness requirement that is easy to miss. On architectures with weak memory ordering, the instructions written by one thread must be visible to the instruction fetch of another before the permission change takes effect. The kernel’s mprotect provides the necessary barrier on the paths that matter, but a user-space implementation that bypasses the kernel must issue its own cache and pipeline synchronisation. The ARM architecture has an explicit instruction-cache maintenance operation for exactly this reason, and omitting it produces code that runs correctly on one core and executes stale bytes on another.

Where the boundary actually is

NX is enforced per page, so it protects against executing data that lies on a page marked non-executable, and nothing more. It does not prevent a write to a page that is already executable, and it does not prevent control from being transferred into the middle of an existing function. A page of library text is executable and writable only in the sense that the mapping allows writes; on a normal system text is mapped read-only, so the write fails too, but that is a separate permission.

The gap that NX leaves open is the one that defines modern exploitation. Existing code is a large, dense space of instruction sequences, and every one of them is a legitimate jump target as far as the processor is concerned. The defence that follows from this is not about permissions on data but about restricting where ret and indirect jumps may go, which is what control-flow integrity attempts and what hardware shadow stacks enforce in silicon.

A system that maps everything writable also non-executable has removed the cheapest attack and left the expensive ones. The economics are worth stating plainly: an injected payload needs a buffer and a return address; a reuse chain needs the addresses of code fragments, which requires either a leak or a non-randomised binary, and the chain must survive whatever the function does to the stack on the way through. That is a large increase in work per exploit, and it is why the technique did not merely move but changed character.