Theory of reverse engineering

Why

• Because AV doesn't work

• You should think as malware to find malware
• It’s awesome

2
 Reverse engineering vs malware analysis

• You can use a huge amount of utilities and enterprise solutions, but there will come a day when
they don't help you.
• A reverse engineer can add utilities to his toolkit and use automatic solutions.
• A malware analyst will never be able to understand the malicious capabilities without
a reverse engineering if the sample bypasses analyst's tools.

3
 I hope you know

• Difference between 0xDEAD and 57005

• Pointers in C++ or C
• EXE and DLL
• Functions, arguments, subfunctions
• for (int i=0; i<20; i++)
{}

4
Programming cycle

linker
• Source • Executable
code • Object file • Running
code process
compiler OS loader

5
Reverse engineering cycle

decompiler
•55 8B EC •void func {
•Push ebp
•Mov ebp
esp
disassembler

6
The code does not differ from the data

Disassembled code

Disassembled data

7
Constant or offset?

8
Little endian

• The lower byte is written at the smaller addresses!

9
Code or data?

10
 Assembler as an instrument for reverse engineering

• You should know basic assembler instructions

• But you don’t have to know every command (Nobody analyzes tens of thousands of operations)
• It is important to be able to manage functions and their arguments
• The most difficult is math operations, but it’s not required for analysis in most cases
• We'll look at the most useful basic commands

11
Registers

Data Registers – EAX, EBX, ECX, EDX

Index Registers – ESI, EDI
Pointer Registers – EIP, ESP, EBP
Control Registers – one flag register
Segment Registers – CS, DS, SS

12
 General-Purpose Registers

• Accumulator register (EAX). Used in arithmetic operations, also stored return value.
• Counter register (ECX). Used in shift/rotate instructions and loops.
• Data register (EDX). Used in arithmetic operations and I/O operations.
• Base register (EBX). Used as a pointer to data.
• Stack Pointer register (ESP). Pointer to the top of the stack.
• Stack Base Pointer register (EBP). Used to point to the base of the stack.
• Source Index register (ESI). Used as a pointer to a source in stream operations.
• Destination Index register (EDI). Used as a pointer to a destination in stream operations.

13
 Instruction Pointer

• EIP - The EIP register contains the address of the next instruction to be executed if
no branching is done.

14
 Stack Pointers

• ESP is the current stack pointer. EBP is the base pointer for the current stack frame.

15
 EFLAGS

EFLAGS register holds many single bit flags. You should remember the following:

• Zero Flag (ZF) – Set of the result of some instruction is zero

• Sign Flag (SF) – Set equal to the most-significant bit of the result, which is the
sign bit of a single integer. (0 indicates a positive value and 1 indicates a negative value)

16
 Instructions

• Usually consist of two parts: operation and arguments (operands)

operation
arguments

• Destination first

Move into EDI register offset of stru_B926D4

17
 Operand types

• Registers: eax, ax, ebx

• Memory: [eax], offset stru_B90483
• Immediates: 0xB9
• Offset: [eax + 0x8]

18
 Addressing Memory
• There is a difference between address of data and data (like pointers)
• Square brackets mean dereference

[0x00B8F416] = CharUpperW
Mov eax, 0x00B8F416 eax = 0x00B8F416
Mov eax, offset aCharupperw eax = 0x00B8F416
Mov eax, [0x00B8F416] eax = 0x43686172

ASCII
0x43 C
0x68 h
0x61 a
0x72 r
0x55 U
0x79 p

19
Basic instructions
• nop
• mov
• lea
• add, sub
• cmp
• test
• xor
• jmp, jz, jnz, je
• push/pop
• call
• ret
• leave

20
 nop

• nop – no operation. Command that does nothing

• It’s very useful in RE for bytepatching, and gives

us the opportunity to skip any checks

21
 mov

• mov – move. Copies data from one location to another

(from right to the left)
• mov eax, ebx
• mov eax, [0x0100FA35]
• mov eax, 0x4F
• mov [0x003F0033], ebx
• mov [0x00324424], [0x3f229304]

22
 lea

• Lea – load effective address

• Move edx, [ebx + eax*8+4] edx stored data

• Lea edx, [ebx + eax*8+4] edx stored address

23
 add sub

• Add – add
• Sub – substraction

Add eax, 1
Sub ebx, eax

24
 cmp

• Cmp – compare. Compare operands and change flag register.

• This instruction basically subtracts one operand from the other for comparing
whether the operands are equal or not.
• It does not disturb the destination or source operands.
• It is used along with the conditional jump instruction for decision making.

• cmp eax, ecx

• cmp eax, 0x4f

25