Buffer Overflows and Shellcode
Explore the mechanics of buffer overflow vulnerabilities and how they can be exploited to inject and execute malicious shellcode.
Buffer Overflows and Shellcode is a free Assembly Language & x86 Low-Level Systems Programming lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Assembly Language & x86 Low-Level Systems Programming learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Intro: Buffer Overflows
Welcome to a critical topic in low-level security: Buffer Overflows. These are a type of software vulnerability that can allow attackers to gain control over a program.
Essentially, a buffer overflow occurs when a program tries to write more data into a fixed-size memory buffer than it was designed to hold. This excess data 'overflows' into adjacent memory regions.
What's a Buffer?
In programming, especially in languages like C or assembly, a buffer is simply a block of memory reserved for storing data. Think of it like a container with a specific capacity.
- Buffers are often used for temporary storage, like holding user input or network data.
- They can be declared as arrays of characters (strings) or other data types.
- For example,
char username[32];declares a buffer that can hold up to 31 characters plus a null terminator.
Code: A Vulnerable Buffer
Let's look at a simple C program with an intentional buffer overflow vulnerability. Pay close attention to the strcpy function.
strcpy copies a string from source to destination, but it does not check the destination buffer's size. This is a common source of overflows.
/* buffer_example.c */
#include <stdio.h>
#include <string.h>
// This function is intentionally vulnerable
void vulnerable_greet(char *name_input) {
char buffer[16]; // A small buffer, 16 bytes
// This is the vulnerability! strcpy doesn't check size.
// If name_input is longer than 15 chars (+ null terminator),
// it will overflow 'buffer'.
strcpy(buffer, name_input);
printf("Hello, %s!\n", buffer);
}
int main() {
// Let's call the vulnerable function with a safe input
vulnerable_greet("CoddyKit User");
printf("Program finished normally.\n");
return 0;
}The Stack Frame
To understand how overflows can be exploited, we need to revisit the call stack. When a function is called, a new stack frame is created.
This stack frame typically contains:
- Local variables for the function.
- The saved Base Pointer (EBP/RBP).
- Most importantly: the Return Address, which tells the CPU where to resume execution after the function finishes.
Overwriting the Return Address
When a buffer overflow occurs on the stack, the excess data doesn't just corrupt adjacent local variables. If enough data is supplied, it can overwrite the saved EBP and then the return address itself.
By changing the return address, an attacker can trick the program into jumping to an arbitrary memory location instead of returning to the legitimate caller. This is the core mechanism for many buffer overflow exploits.
Introducing Shellcode
So, where does the attacker want the program to jump? Usually, to a piece of code they control, known as shellcode.
- Shellcode is a small, self-contained piece of machine code.
- Its primary purpose is often to execute a 'shell' (a command prompt) on the target system.
- It can also perform other malicious actions, like creating new users, downloading files, or connecting to a remote server.
Anatomy of Simple Shellcode
Shellcode is typically written in assembly language to be as compact and efficient as possible. It avoids null bytes (\x00) because many string functions stop copying at the first null byte.
A common goal is to call the execve system call (on Linux) to launch /bin/sh.
; Example (conceptual) Linux x86 shellcode snippet:
; mov eax, 0x0b ; Syscall number for execve
; mov ebx, addr_of_sh ; Pointer to '/bin/sh' string
; mov ecx, 0 ; Arg 2 (argv) = NULL
; mov edx, 0 ; Arg 3 (envp) = NULL
; int 0x80 ; Invoke kernel (syscall)Injecting Shellcode
How does the shellcode get into the program? The attacker includes it as part of the malicious input that causes the buffer overflow.
When the buffer overflows, the shellcode is written into the program's memory, usually on the stack, alongside the overwritten return address.
Exploit Structure: NOP Sled
Attackers often use a NOP sled (No Operation sled) to increase the reliability of their exploit.
- A NOP sled is a sequence of 'No Operation' (NOP) instructions (e.g.,
\x90in x86). - If the attacker overwrites the return address to point anywhere within the NOP sled, the CPU will simply execute NOPs until it 'slides' into the actual shellcode.
- This compensates for slight inaccuracies in guessing the exact memory address of the shellcode.
Defenses & Mitigations
Operating systems and compilers have developed several defenses against buffer overflows:
- ASLR (Address Space Layout Randomization): Randomizes memory addresses to make guessing the return address or shellcode location harder.
- DEP/NX Bit (Data Execution Prevention/No-Execute): Marks memory regions (like the stack) as non-executable, preventing shellcode from running there.
- Stack Canaries: Compiler-generated random values placed on the stack; if overwritten, the program detects tampering and aborts.
- Safe Functions: Using functions like
strncpy,snprintf,fgets, or C++ strings that perform bounds checking.
Check Your Understanding
A stack buffer overflow allows an attacker to write past the end of a buffer. What is the primary goal an attacker aims to achieve by carefully crafting input to overwrite the return address?
Recap: Overflows & Shellcode
In this lesson, we explored the dangerous world of buffer overflows. We learned that they occur when too much data is written into a fixed-size buffer, corrupting adjacent memory.
Crucially, if this overflow reaches the return address on the stack, an attacker can hijack program control. They achieve this by injecting shellcode—small, malicious machine code—and redirecting execution to it, often aided by a NOP sled. We also touched upon essential mitigation techniques like ASLR, DEP, and stack canaries.
Frequently asked questions
Is the “Buffer Overflows and Shellcode” lesson free?
Yes — the full text of “Buffer Overflows and Shellcode” is free to read here on the web, and the Assembly Language & x86 Low-Level Systems Programming course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Assembly Language & x86 Low-Level Systems Programming course, upgrade to CoddyKit PRO.
What will I learn in “Buffer Overflows and Shellcode”?
Explore the mechanics of buffer overflow vulnerabilities and how they can be exploited to inject and execute malicious shellcode. You practise Assembly Language & x86 Low-Level Systems Programming with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Assembly Language & x86 Low-Level Systems Programming?
No prior experience is required. Assembly Language & x86 Low-Level Systems Programming on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Buffer Overflows and Shellcode” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Assembly Language & x86 Low-Level Systems Programming lesson?
Yes. Every Assembly Language & x86 Low-Level Systems Programming lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Cache Coherency and Performance
- Hand-Optimizing Critical Sections
- Buffer Overflows and Shellcode
- Branch Prediction and Speculative Execution