Trace Surfing A tale of data structure recovering and other yerbas
By Agustin Gianni – Immunity Inc.
Problem Statement
Given a memory trace, what information does the trace gives us about the underlying data structures?
Road map ●
Investigation of previous approaches
●
Realization that they kind of suck
●
Enlightenment phase → how can we improve
Introduction ●
What is a memory trace? ●
A memory trace is a collection of all the memory accesses performed by an application. –
●
Both reads and writes
How can I obtain a memory trace? ●
Binary Instrumentation – –
●
pintool DynamoRIO
Full system emulation – –
QEMU BOCHS
Example Memory Trace # White listed image `calc.exe` # Loading hooks from file hooks.hks # Loaded hook alloc:test_custom_alloc:00000774:0:my_alloc_ # Loaded hook free:test_custom_alloc:000007b6:0:my_free_ L:calc.exe:0x003a0000:0x0045ffff # Thread 0x0 started # Instrumented malloc at 0x75619cee # Instrumented free at 0x75619894 # Instrumented realloc at 0x7561b10d # Instrumented calloc at 0x7561c456 W:0x003a76c6:0x01d125e0:0x01d125e0:0x00000004:0x0000000f W:0x003a76cc:0x01d125e0:0x01d125e4:0x00000004:0x0000000f … F:0x003b8f9a I:0x003b8f9a:0x00000031:0x00000000 F:0x003b8fdc I:0x003b8fdc:0x00000031:0x00000000 # Thread 0x1 did not finish but application exited.
Introduction ●
Why do we care about recovering data structures? ●
Large binaries are a pain to reverse –
Specially Object Oriented Code ●
●
Virtual Function Tables and friends
●
Makes reverse engineering happier
●
Saves time
Why not? ●
Computers got fast enough to trace every single memory access
HexRays – With data types
Introduction ●
Has anyone approached the problem? ●
Dynamic analysis – –
●
Howard: Dynamic Excavator for Reverse Engineering Data Structures Rewards: DDE, Dynamic Data Structure Excavation
Static analysis –
WYSINWYX: What You See Is Not What You eXecute ●
Based on abstract interpretation, blah, blah, blah!
The Rewards / Howard approach ●
●
Trace every single memory access ●
Heap
●
Stack
Define type sinks ●
System Calls
●
Library Calls
●
Special purpose instructions –
For instance, string manipulation instructions on Intel architecture.
●
Propagate recovered types
●
Analyze the memory trace
Type Sinks ●
●
A type sink is a function, syscall or instruction that we know which types it is taking System calls and standard libraries are the more verbose ●
For instance: – – –
ssize_t read(int fd, void *buf, size_t count); Leaks four types: ssize_t, int, void *, size_t Also we can extract semantics ● ● ●
We know that 'fd' is a descriptor 'buf' is a buffer Etc.
Type Sinks ●
Instructions can also leak types ●
Intel String Operations –
●
Intel Floating Point Instructions –
●
FADD, FDIV, FMUL, and so on.
Jumps – –
●
CMPS, INS, LODS, MOVS, OUTS, SCAS, STOS
JG / JL → Signed Integers JA / JB → Unsigned Integers
Memory dereferences –
Data dereferences leak half a type ●
–
We just know the dereferenced address is a pointer
Indirect calls leak function pointer types ●
We know that the dereferenced address contains a pointer to a function.
What do we want to recognize? ●
●
Things to recognize: ●
Structures / Classes
●
Arrays
●
Pointers
How? ●
Study how the memory is accessed
Identifying Pointers ●
Pointers are 'easy' to detect ●
Just see what instructions dereference memory
●
The dereferenced argument must be a valid pointer –
●
Otherwise the program would crash
Problem – –
We cannot yet know the type of the pointer If we are lucky enough, and by lucky I mean that we have sufficient code coverage, we will identify the type of the pointer.
Warning : we are entering the terrain of the incomplete and unsound assumptions.
Absolute correctness ●
Do we really care about absolute correctness? ● ●
●
Even if we could automatically identify a fraction of the types correctly, that saves us work.
Eventually decisions/corrections must be done ●
●
Hint → I don't
Inconsistent typing is detected by humans
We are not aiming to solve unsolvable problems ●
We cannot get back what is not there –
Compilation is not bidirectional ●
Although Rolf may argue this I've been told ;)
Identifying Structures ●
Typically structure fields are accessed in an indirect way ●
●
●
This depends heavily on the compiler and the optimization level. Often, access patterns will be similar.
Example ●
Let A be a base pointer
●
*(A + 0) is the first field
●
*(A + 8) is the second field
●
And so on
Identifying Structures ●
What we want to do is to detect indirect memory addresses. ●
●
We can obtain this from a memory trace
But … ●
What if A was not a structure – – – –
●
Let A be an array *(A + 0) is the first element *(A + 8) is the second element And … we are screwed
Also, sometimes structure fields are accessed directly –
There is no base pointer
Identifying Structures ●
●
There is no way we can decide, with certainty, whether a pointer points to a structure or an array ●
We have to make unsound assumptions
●
Rely on compiler specific constructs
●
Heuristics
●
And why not a bit of magic
In the end, manual work needs to be done ●
Still, less work than reversing manually
Identifying Structures ●
To distinguish between arrays and structures we use some heuristics ●
Memory accesses are generally scattered –
Example: ● ● ●
●
Access field at offset 0x00 Then offset 0x10 And so on
Size of the access is generally heterogeneous –
Example: ● ● ●
Access field 2 which is an integer Then access field 3 which is a short integer Etc.
Identifying Structures - Example ●
6
2
Memory accesses ●
1 – DWORD
●
2 – DWORD
●
3 – WORD
●
4 – WORD
●
5 – DWORD
●
6 – BYTE
●
7 – WORD
3 5
4
6
7
1
Identifying Structures ●
There are a considerable amount of cases where this will fail ●
The most trivial cases – –
●
Initializing a structure with “memset” Copying a structure with “memcpy”
How do we solve this –
If we have more than one access pattern, favor the more irregular
Identifying Arrays ●
We can identify arrays by watching memory accesses on loops ●
There are two cases – –
Sequential memory accesses Random memory accesses
Identifying Arrays ●
Sequential memory accesses ●
Let A be a pointer
●
We are on a loop
●
A is dereferenced at loop cycle one.
●
B is generated also at loop cycle one.
●
Next iteration
●
B is dereferenced.
●
A is likely an array pointer
Identifying Arrays ●
Random memory accesses ●
If all the accesses are of the same size we have a hint that we are dealing with an array.
●
But it is also likely that it could be an structure.
●
This is getting hairy.
So, where are we?
Where are we? ●
●
Detecting whether a pointer points to an array or a structure is essentially an educated guess. ●
We need to further “educate” ourselves
●
We need to have stronger assumptions that we can rely on.
Tracing stack memory accesses is tricky ●
What about address reutilization –
●
We need to tag every address with a TAG to differentiate two identical addresses accessed in different times
Tracing all memory accesses is painfully slow ●
We are interested in large binaries
Are we screwed then? ● ●
●
Not really We need to make our analysis a little bit more specific ●
Hence less complete
●
But more accurate
It is all about giving up a bit of generality for a bit more of accuracy
Looking for better waves
Focus on Heap Objects ●
Why? ●
Heap objects are shared. We like data that is shared –
●
We have more information –
●
It leads to good things from an vulnerability research point “malloc” like functions give us the size of the chunks
It is easier to track heap memory – –
Hook allocation routines and tag the returned memory with a unique id Hook also deallocation routines to keep track of valid memory chunks
Object Oriented Code ●
●
Objects are basically structures with methods ●
Each object method needs to somehow reference its underlying object.
●
Objects of a given class share a set of common characteristics
Most of them come from the heap ●
●
Or at least those object with shared state information
So if we focus on objects, the problem is a bit less complicated ●
We are dealing with structures of know size
●
Now the whole address space is reduced to a fraction of its size –
●
Just analyze the .heap
Keeping track of the life of a heap memory region is simple – –
Hook the allocation routine → The block is alive Hook the free routine → The block is dead
How to detect objects? ●
Not every single heap chunk is an object
●
Heuristics! ●
Take advantage of calling conventions – – –
●
●
Visual Studio: will set the 'ecx' register to the 'this' pointer GCC 32 bits: pushes as the first method argument the object GCC 64 bits: 'rsi' is set to the 'this' pointer
So we mark every tracked heap chunk that is on “ecx”, “rsi” or the first argument of a function as a possible object The object must be used inside the potential method
How to detect methods ●
There is no sound way
●
We have to trust our heuristics ●
●
We are going to miss some methods ●
●
Which are better than most Anti-Virus heuristics :P The dynamic nature of a trace makes us rely on code coverage.
We are going to mark some functions as methods ●
Sometimes the this pointer remains spuriously in 'ecx'
So, how are we now?
We are doing better! ●
●
We can detect “interesting objects” ●
We know its size
●
We know where they are being used
What else we need to do? ●
Detect fields
●
Detect relationships with other types – –
Inheritance Composition
Detecting Object Fields ● ●
●
●
We already have all heap memory accesses in our trace If the memory access is to one of our interesting objects we save the access offset and size Since we only track interesting objects the analysis is much quicker We can implement the algorithms used by Howard/Rewards ●
If we have information from type sinks, we can propagate it
Detecting methods ●
On each function call check if ECX points to a heap object. ●
If true – –
● ●
Mark the chunk as interesting Save the access offset for future usage
Mark the function as interesting Does this function get called again with the same conditions? –
That is, the same function gets called with a chunk of the same size as the 'this' parameter
How far can we go?
How far can we go? ●
With all the collected traces we can obtain quite a lot of information ●
Class Hierarchy
●
Virtual Function Tables
●
Types!
●
Bonus (not really related with type inference) – –
Code coverage information Indirect branch resolution
How can we achieve this?
Virtual Function Tables ●
●
Useful to help IDA Pro to discover more functions For each write to an interesting chunk ●
Is the value written referring to .text ? –
Is [value] also in .text? ●
–
This is for sure a Virtual Function Table
If not, it is just a field update
Types ●
Type reconstruction algorithm is divided in three phases ●
First Analysis Pass (FAP) –
Pun intended
●
Second Analysis Pass (SAP)
●
Third Analysis Pass (TAP)
First Analysis Pass ●
For each function ●
Get all its interesting chunks –
●
Mark the whole chunk as a composite type –
●
Set the composite type size to the size of the chunk
If 'this' does not point to the first byte of the chunk, get the offset –
●
That is chunks that were passed as the 'this' argument
Divide the composite chunk in two types at the calculated offset
Repeat the process with all the methods that used the chunk and subdivide the composite type
First Analysis Pass chunk_address = A ecx_address = A + 0 Composite Type
Composite Type TypeA
Chunk
In this case, TypeA fills the whole composite type
Offset = 0
First Analysis Pass chunk_address = A ecx_address = A + C Composite Type
Composite Type TypeA
TypeB
Offset = C
Chunk
In this case there are two types, we recognize this because there were two methods called with 'this' pointing at the same memory chunk but at a different offset.
First Analysis Pass – continued ●
Add the current function to a list of methods
●
For each write to the interesting chunk ● ●
Add a field at the offset of the write Mark the field with the corresponding basic type according to the write size –
For instance, a write of four bytes is marked as “uint32_t”
First Analysis Pass – continued ●
Collect a set of constraints ●
●
For each chunk that was received as the 'this' argument build a map from the method address to a list of all the types created. This will be later used build relationships between types and subsequent merging of identical types
method_at_0xcafecafe
Type_A
Size = X_1
Type_B
Size = X_1
Type_C
Size = X_2
Second Analysis Pass ●
Merge similar types ●
●
Cheat by first using the type constraints collected on the FAP phase
How do we define similar? ●
They have the same size –
●
They have equivalent fields –
●
Equal types with differing sizes will be addressed in the third pass That is, at offset O there is a type T of size S in both types
They share a set of methods –
How many? ● ● ● ● ●
Let N be the number of methods in Type1 Let M be the number of methods in Type2 Let S be the number of shared methods SimilarityIndex(N,M,S) = (S / (N+M)) * 100 If SimilarityIndex > SimilarityThreshold then they are similar
Third Analysis Pass ●
●
There are types that share methods and fields but they differ in size What is going on? ●
There are two possible scenarios –
Type2 in inherits from Type1 ●
–
len(Type2) > len(Type1) most of the times
The type has an internal buffer ●
This is the case of for example strings in some browsers
Inheritance / Composition ●
A simple inheritance relationship is translated into a composition of structures
Inheritance / Composition ClassA Field1 Field2 Field3 Field4 ClassB ClassA Field1 Field2 Field3 Field4 Field1 Field2 Field3
Inheritance / Composition ●
Two classes of different size use the same method
●
The bigger one is likely the child class
●
The smallest one is likely the parent class
●
This heuristic can fail ●
Say that we have a dynamically allocated buffer inside a class –
●
Rare, weird, but it can and will happen
Failure will generate an extra type but the relationships between the types will still be interesting and can be detected by a human once the information is imported into IDA Pro
Hard example :) ●
●
Example string class that will contain metadata and contents on the same chunk of memory Other recurring complex examples are hash tables
StringClass StringMetadata uint32_t len AAAAAAAAAAAAAAA AAAAAAAAAAAAAAA … AAAAAAAAAAAAAAA ???????????????
Increasing accuracy
Increasing accuracy ●
Accuracy of our approach is directly related with code coverage ●
●
The more code coverage, the more accuracy
Increasing code coverage ●
The “smart” way – –
We can tweak Klee (requires source code) We can code our version of SAGE ● ●
●
??? Profit
The “other” way – –
Fuzz the application like a 15 year old Gather a set of input files (if possible) and calculate the set of files that gets the maximum coverage
Static Analysis ●
How can we further validate our results? ●
●
Detecting calling convention
We have collected a fair amount of information, how can we propagate this information? ●
●
Propagating the type information into basic blocks not executed on the trace Or we can be lazy and let HexRays decompiler to do it for us :)
Calling convention detection ●
●
●
A spurious function calls can happen when a non method function is called on a method The function call can receive the 'this' pointer of the previous method call We avoid this case by ruling out all the function calls that do not behave as thiscall
Calling convention detection ●
Given a function get its CFG
●
Obtain a DAG (direct acyclic graph)
●
Do a topological sort
●
Assume ECX is a 'this' pointer ●
●
Add it to a list of 'this' aliases
For each basic block ●
If instruction kills any of the 'this' aliases ●
●
If the instruction aliases one of the 'this' pointers ●
●
If the alias list is empty return “not thiscall” Add the new alias to the list
If the instruction accesses memory using one of the aliases of 'this' then the function is likely 'thiscall'
Calling convention detection ● ●
This can fail too Generally it gives a correct answer in 90% of the analyzed function ●
●
These results were validated by analyzing binaries with symbols available
In practice this information allows us to detect spurious functions detected as methods of a class
Example: calc.exe types
Example: calc.exe types
Example: calc.exe types
References ●
http://www.pintool.org/
●
http://www.dynamorio.org/
●
http://wiki.qemu.org/Main_Page
●
http://bochs.sourceforge.net/
●
http://www.few.vu.nl/~asia/publications
●
http://www.cs.purdue.edu/homes/xyzhang/reverse.html
●
http://pages.cs.wisc.edu/~reps/
Thanks to ●
Juliano Rizzo
●
Nicolas Waisman
●
Pablo Sole
●
Sean Heelan
●
Topo Muñiz
