Before we talk about Buffer overflow, we need to understand how memory allocation happens in computer programs.
Local Memory Allocation on Stack
When we declare a local variable (variables that are declared in a function and can be used within that function only), a temporary memory space is allocated for storing the data. This memory space is automatically deallocated when the function returns. All such local variables are contained within a data structure known as Stack.
Stack not only stores these local variables but a return pointer (RP) as well. Return Pointer contains the memory address of that point in the program which it has to return to after the function has completed execution.
Global Memory Allocation on Heap
Global variables (not associated with the scope of a function and can be accessed anywhere in a program) are declared within a data structure known as Heap. The memory for global variables is allocated dynamically during the run time and in a more manual way. Also, after its use, if it is no longer needed, you have to free the memory space all by hand.
The function that is used to allocate the memory space for a global variable is called malloc () – memory allocator.
Buffer Overflow
The memory space which holds the data for local or global variable, declared within a stack or a heap, is called the buffer. Buffer Overflow is a condition that happens when more data is pushed than that the buffer is supposed to hold.
Stack-based Buffer Overflow
When a program doesn’t check and limit the amount of data pushed into the assigned space for a variable, the variable space overflows. As a result, the data stored in that variable space goes into the neighboring variable’s pace and eventually into the return pointer space. Attackers takes the advantage of this by overwriting the buffer space with malicious code and the return pointer with the new address of the space where the malicious code is stored. Thus, when the function tries to return to the main program, it instead runs the malicious code inserted by the attacker.
Heap-based Buffer Overflow
Heap Overflow is a type of Buffer Overflow where the buffer that can be overwritten is allocated in the heap portion of the memory (buffer that was allocated using malloc function). Heap Overflows are exploitable in a different way as compared to Stack Overflows. Exploitation is done by corrupting the heap data.
Out of above two types of Buffer Overflow, Stack-based Buffer Overflow is more common and widely exploited.
