Automating Data Structure Recovery
Develop scripts to automatically identify and reconstruct complex data structures within obfuscated binaries.
Automating Data Structure Recovery is a free Reverse Engineering & Binary Analysis Basics lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Reverse Engineering & Binary Analysis Basics learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What are Data Structures?
In programming, a data structure is a way to organize and store data efficiently. Think of it like a neatly arranged filing cabinet for related information.
In reverse engineering, we often deal with compiled programs, which means the original source code is gone. Our goal is to "see" these hidden filing cabinets in the raw binary data.
The 'Why' of Data Recovery
Recovering data structures is crucial for understanding a program's logic. If you know how an object is laid out in memory, you can:
- Understand how different pieces of data relate.
- Identify important program variables.
- Pinpoint potential vulnerabilities more easily.
It turns a jumble of bytes into meaningful information!
Finding Hidden Structures
When a program is compiled, compilers often remove debugging information and optimize code. This makes it hard to automatically identify structures because:
- Original names are lost.
- Fields might be reordered or padded.
- Complex structures can be spread out.
It's like trying to rebuild a puzzle without the picture or edge pieces!
Simple Types in Binary
Before tackling complex structures, let's remember how basic data types look in raw bytes. A structure is just a collection of these simpler types.
For example, an integer might be 4 bytes, a character 1 byte. Their order and size matter!
import struct
# Simulate a small piece of binary data
binary_data = b'\x01\x00\x00\x00' + b'\x41' + b'\x02\x00\x00\x00'
print("Raw bytes:", binary_data)
# Interpret bytes 0-3 as a 32-bit integer (little-endian)
# '<I' means little-endian unsigned int
int_val = struct.unpack('<I', binary_data[0:4])[0]
print(f"Integer (offset 0): {int_val}")
# Interpret byte 4 as a character
char_val = chr(binary_data[4])
print(f"Character (offset 4): {char_val}")
# Interpret bytes 5-8 as another 32-bit integer
int_val2 = struct.unpack('<I', binary_data[5:9])[0]
print(f"Integer (offset 5): {int_val2}")What is a 'Struct'?
In languages like C, a struct is a user-defined data type that groups related variables into one single unit. Imagine a "User" struct that holds a user's ID (integer), name (string), and age (integer).
When compiled, this struct occupies a contiguous block of memory, with each field at a specific offset from the start of the block.
Spotting Structures Manually
When reverse engineering manually, you'd look for clues like:
- Repeated Access Patterns: Code that always reads/writes at
[reg + 0],[reg + 4],[reg + 8]. - Function Arguments: A large block of memory passed as a single argument to a function.
- Pointers: A field that points to another known structure or data type.
These patterns suggest a structured block of data.
Automating Pattern Search
Manually finding structures is tedious! This is where scripting shines. We can write scripts to automatically scan binary data for common patterns that might indicate a structure.
For example, a script could look for two integers followed by a null-terminated string, a very common pattern for simple objects.
def find_pattern(data_bytes: bytes, pattern_bytes: bytes):
"""Searches for a byte pattern within a larger byte string."""
indices = []
for i in range(len(data_bytes) - len(pattern_bytes) + 1):
if data_bytes[i:i+len(pattern_bytes)] == pattern_bytes:
indices.append(i)
return indices
# Simulate a binary's data section
simulated_binary_data = (
b'\xDE\xAD\xBE\xEF' + # random bytes
b'\x01\x00\x00\x00' + # int 1 (little-endian)
b'\x0A\x00\x00\x00' + # int 10
b'NAME\x00' + # string "NAME"
b'\x00\x00\x00\x00' + # padding
b'\x02\x00\x00\x00' + # int 2
b'\x0B\x00\x00\x00' + # int 11
b'ITEM\x00' # string "ITEM"
)
# Define a pattern to search for: int(1), int(10), string("NAME")
pattern_to_find = (
b'\x01\x00\x00\x00' +
b'\x0A\x00\x00\x00' +
b'NAME\x00'
)
found_at_offsets = find_pattern(simulated_binary_data, pattern_to_find)
if found_at_offsets:
print(f"Pattern found at offsets: {found_at_offsets}")
else:
print("Pattern not found.")XRefs for Structure Clues
In reverse engineering tools, a cross-reference (xref) shows you where a specific address or data is used in the code.
If many functions consistently access data starting at a particular address, and then at +0x4, +0x8, +0xC, these xrefs strongly suggest a data structure is being manipulated at that memory location.
Scripts can automate the analysis of these xrefs!
Scripting Structure Definitions
Once you've identified a potential data structure layout (e.g., through patterns or xrefs), your script can then define this structure within the reverse engineering tool itself.
This means telling the tool: "At this address, there's a structure named 'MyObject' with an integer 'ID' at offset 0, and a string 'Name' at offset 4."
This makes the disassembled code much more readable, replacing raw memory accesses with meaningful field names!
import struct
# Reusing simulated_binary_data from previous example
simulated_binary_data = (
b'\xDE\xAD\xBE\xEF' + # random bytes
b'\x01\x00\x00\x00' + # int 1 (little-endian)
b'\x0A\x00\x00\x00' + # int 10
b'NAME\x00' + # string "NAME"
b'\x00\x00\x00\x00' + # padding
b'\x02\x00\x00\x00' + # int 2
b'\x0B\x00\x00\x00' + # int 11
b'ITEM\x00' # string "ITEM"
)
class ItemStruct:
def __init__(self, data_bytes, start_offset):
# We assume the struct starts at start_offset in data_bytes
# Field 1: 4-byte integer (ID) at offset 0 from struct start
self.item_id = struct.unpack('<I', data_bytes[start_offset:start_offset+4])[0]
# Field 2: 4-byte integer (Quantity) at offset 4 from struct start
self.quantity = struct.unpack('<I', data_bytes[start_offset+4:start_offset+8])[0]
# Field 3: Null-terminated string (Name) at offset 8 from struct start
name_start = start_offset + 8
name_end = data_bytes.find(b'\x00', name_start)
if name_end == -1: # No null terminator, read until end
self.name = data_bytes[name_start:].decode('ascii', errors='ignore')
else:
self.name = data_bytes[name_start:name_end].decode('ascii', errors='ignore')
def __str__(self):
return (f"ItemStruct:\n"
f" ID: {self.item_id}\n"
f" Quantity: {self.quantity}\n"
f" Name: '{self.name}'")
# The "ITEM" pattern starts at offset 24 in simulated_binary_data
second_struct_offset = 24
found_item = ItemStruct(simulated_binary_data, second_struct_offset)
print(found_item)
# Let's also parse the first one to show it works
first_struct_offset = 4
found_name = ItemStruct(simulated_binary_data, first_struct_offset)
print("\n--- Another instance ---")
print(found_name)Other Structure Clues
Beyond simple patterns and xrefs, scripts can look for:
- Alignment: Data types often align to certain byte boundaries (e.g., 4-byte integers align to addresses divisible by 4).
- Vtables: In C++, objects often start with a pointer to a "virtual method table" (vtable), a strong indicator of an object.
- Common Function Args: If a library function expects a specific structure as input, scripts can identify calls to that function and infer the structure of its arguments.
Data Structure Challenge
Automating data structure recovery is about finding meaningful patterns in raw binary data.
Which of the following is NOT a primary reason why scripting is essential for identifying data structures in obfuscated binaries?
Recap: Automated Structure Recovery
We've learned that recovering data structures is vital for understanding compiled programs. Manual identification is hard due to lost source info and optimizations.
Scripting helps by:
- Scanning for byte patterns that indicate data types.
- Analyzing cross-references to memory locations.
- Programmatically defining structures within RE tools.
This transforms raw bytes into understandable program logic, making complex analysis much easier!
Frequently asked questions
Is the “Automating Data Structure Recovery” lesson free?
Yes — the full text of “Automating Data Structure Recovery” is free to read here on the web, and the Reverse Engineering & Binary Analysis Basics course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Reverse Engineering & Binary Analysis Basics course, upgrade to CoddyKit PRO.
What will I learn in “Automating Data Structure Recovery”?
Develop scripts to automatically identify and reconstruct complex data structures within obfuscated binaries. You practise Reverse Engineering & Binary Analysis Basics with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Reverse Engineering & Binary Analysis Basics?
No prior experience is required. Reverse Engineering & Binary Analysis Basics on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Automating Data Structure Recovery” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Reverse Engineering & Binary Analysis Basics lesson?
Yes. Every Reverse Engineering & Binary Analysis Basics lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- IDAPython and Ghidra Scripting
- Automating Data Structure Recovery
- Binary Patching Techniques
- FLIRT Signatures & Library Function Identification