3GAP Training Course

Analysing Crash Logs

Ian Hamilton, 21 March 2011 CONFIDENTIAL
 CONFIDENTIAL

Overview

This presentation describes the AP crash logs, and how to use the information
they contain to help debug AP crashes

– What the different crash log sections are for

– How the gcc compiler organises the stack
– How to find the location of a crash
– How to determine the immediate cause of the crash
– How to unwind the stack to find out the calling hierarchy
– How to extract local variables from the crash log

See M:\DOC\Application_Notes\OY3G_AN_0319_Using_libcrash_on_the_3GAP\Draft

2
 CONFIDENTIAL

The Crash Log

There are six sections in a crash log:

• Crash type
• Register dump
• Memory map
• Code dump
• Stack dump
• Stack unwind

3
 CONFIDENTIAL

Crash type

The first line of the crash log looks something like this:

Crash handler called, signal SIGABRT (6, 0xfffffffa)

The signal tells you why the AP crashed.

SIGABRT - Function abort() called, usually due to a failed assert()

SIGBUS – Illegal bus access, e.g. word access not word-aligned
SIGSEGV – Attempt to access an illegal memory location
SIGILL – Illegal instruction executed
SIGFPE – Floating point error

4
 CONFIDENTIAL

Register dump

This section shows what values were in the CPU registers when the crash
occurred.

• R0 to R10 – general purpose registers

• FP – frame pointer (used as a general purpose register)
• IP – scratch register
• SP – stack pointer
• LR – Link register (return address)
• PC – Program counter

[Register usage is a convention defined by the Arm Procedure Call Standard, APCS]

5
 CONFIDENTIAL

Memory Map

The Memory Map section shows all the memory segments used by the
application. Use of any other memory by the application is illegal and will cause a
SIGSEGV.
Flags show how each segment can be used:

• r-xp Read only, executable – CODE

• rwxp Read/write, executable – DATA (Heap, static data or stack)
• ---p Can’t read or write – GUARD SEGMENT

p means private
s means shared

Using the map, we can find which module the crash occurred in.

6
 CONFIDENTIAL

Code dump

A memory dump either side of the program counter.

– Can tell immediately if its proper code, or if the program counter has ‘got lost’
– Can check against a disassembly to make sure you have the right code
– In a pinch, you can disassemble the code

7
 CONFIDENTIAL

Stack dump

This is the most useful bit of the dump, but you need the other bits to interpret it

– Captures the state of the thread that crashed

– Calling backtrace
– Function arguments
– Local variables

8
 CONFIDENTIAL

Stack unwind

The least useful bit of the dump!!

– Relies on the frame pointer (fp register)

– The compiler optimisation we use eliminates the frame pointer
– Therefore, the unwinding invariably fails

9
 CONFIDENTIAL

Find the crash

For a SIGSEGV, the PC register in the stack dump points at the instruction which
caused the segment violation

PC: 0x4038aabc

Use the memory map to locate this address:

4031d000-4042b000 r-xp 00000000 1f:05 1816768 /lib/libc-2.3.6.so

So the crash occurred inside libc

10
 CONFIDENTIAL

Disassemble the code

Set up a directory somewhere to do the debugging in – local drive is best

Unpack the ipkg and disassemble:

bash$ ar -x /<path to your builds>/SR0.4.0_0.5.0_AP_379.8.0/ipkg/arm/libc_167.8.0_arm.ipk

bash$ tar -xvzf data.tar.gz

bash$ /opt/codesourcery/arm-2008q3/bin/arm-none-linux-gnueabi-objdump -dl lib/libc.so.6 > libc.txt

11
 CONFIDENTIAL

Locate the crash address

For a library, subtract the start address from the PC register to get the offset into the code

0x4038aabc – 0x4031d000 = 0x0006dabc

– Search for 6dabc:

0006dab4 <strcmp>:
6dab4: e1a0c000 mov ip, r0
6dab8: e3a02000 mov r2, #0 ; 0x0
6dabc: e7d2000c ldrb r0, [r2, ip]
6dac0: e7d23001 ldrb r3, [r2, r1]
6dac4: e3500000 cmp r0, #0 ; 0x0
6dac8: e2822001 add r2, r2, #1 ; 0x1
6dacc: 0a000003 beq 6dae0 <strcmp+0x2c>
6dad0: e1500003 cmp r0, r3
6dad4: 0afffff8 beq 6dabc <strcmp+0x8>
6dad8: e0630000 rsb r0, r3, r0
6dadc: e12fff1e bx lr
6dae0: e2630000 rsb r0, r3, #0 ; 0x0
6dae4: e12fff1e bx lr
12
 CONFIDENTIAL

Unwind the stack (1)

The first step in unwinding the stack in this example crash log is trivial. The
strcmp() function is very simple, and the compiler doesn’t bother setting up a
stack frame or pushing the return address on the stack. The return address for
this call is therefore still in the LR register, and shown in the register dump
(0x000cc6d4).

Looking at the map, this address is in the code segment of the main application:

00008000-000fb000 r-xp 00000000 1f:05 6700100 /opt/ipaccess/MgrApp/ipa-mgr_app.379.8.0

So, we now need to install and disassemble the application like this:

bash$ ar -x /<path to your builds>/SR0.4.0_0.5.0_AP_379.8.0/ipkg/arm/nostrip/mgr-app-nostrip_379.8.0_arm.ipk

bash$ tar -xvzf data.tar.gz
bash$ /opt/codesourcery/arm-2008q3/bin/arm-none-linux-gnueabi-objdump -dl opt/ipaccess/MgrApp/ipa-mgr_app > ipa-mgr_app.txt

13
 CONFIDENTIAL

Unwind the stack (2)

Load the disassembled application into an editor and search for “cc6d4:”. The
main application is not offset like the library was, so we use the address
unchanged, except for removing the leading zeroes and adding a colon.

/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/sdgpio.c:374
cc6c0: e59f10c8 ldr r1, [pc, #200] ; cc790 <SDGPIO_Write+0xdc>
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/sdgpio.c:367
cc6c4: e1a08002 mov r8, r2
cc6c8: e1a0a003 mov sl, r3
cc6cc: e1a04000 mov r4, r0
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/sdgpio.c:374
cc6d0: ebfd10f0 bl 10a98 <_init+0x814>
cc6d4: e2509000 subs r9, r0, #0 ; 0x0
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/sdgpio.c:383
cc6d8: e28d600e add r6, sp, #14 ; 0xe
cc6dc: e3a07001 mov r7, #1 ; 0x1
cc6e0: e1a01005 mov r1, r5
cc6e4: e3a02008 mov r2, #8 ; 0x8
cc6e8: e59f30a4 ldr r3, [pc, #164] ; cc794 <SDGPIO_Write+0xe0>

Note the source line information, thanks to the unstripped binary we used for the
disassembly.
14
 CONFIDENTIAL

Digression (1) – compiler optimisation (1)

– The compiler is set to optimize the code, so the order of execution of the assembly
code may not match the source code. There may be source lines out of order, source
lines split into two or more sub sections, source lines optimized out altogether, and
inlined functions.

/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/sdgpio.c:374
cc6c0: e59f10c8 ldr r1, [pc, #200] ; cc790 <SDGPIO_Write+0xdc>
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/sdgpio.c:367
cc6c4: e1a08002 mov r8, r2
cc6c8: e1a0a003 mov sl, r3
cc6cc: e1a04000 mov r4, r0
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/sdgpio.c:374
cc6d0: ebfd10f0 bl 10a98 <_init+0x814>
cc6d4: e2509000 subs r9, r0, #0 ; 0x0
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/sdgpio.c:383
cc6d8: e28d600e add r6, sp, #14 ; 0xe
cc6dc: e3a07001 mov r7, #1 ; 0x1
cc6e0: e1a01005 mov r1, r5
cc6e4: e3a02008 mov r2, #8 ; 0x8
cc6e8: e59f30a4 ldr r3, [pc, #164] ; cc794 <SDGPIO_Write+0xe0>

15
 CONFIDENTIAL

Digression (2) – Library calls

– This call is an example of a library call:

cc6d0: ebfd10f0 bl 10a98 <_init+0x814>

– The disassembly doesn’t show the name of the library function as its not in the
application’s symbol table (its in another module).
– If you look up <_init+0x814> in the disassembly, you see this:

10a98: e28fc600 add ip, pc, #0 ; 0x0

10a9c: e28ccaf1 add ip, ip, #987136 ; 0xf1000
10aa0: e5bcfcc0 ldr pc, [ip, #3264]!

– This is retrieving the function address from a table, and this is how the dynamic
linking is done.
– First time through, address in table is pointing at the dynamic linker. This inserts
real library function address in table so subsequent calls go straight to library.
16
 CONFIDENTIAL

Analyse the source (1)

– Here’s the relevant section of the source code. Again, it is vital to look at the right version of
the source code, otherwise the C code won’t line up with the disassembly:
363: SDGPIO_Error_t SDGPIO_Write(SDGPIO_Handle_t handle,
364: uint16_t devNum,
365: uint32_t gpioSelectMask,
366: uint32_t value)
367: {
368: uint16_t reg;
369:
370: DEBUG_PRINTF(DEBUG_TRACE, MODULE_SDGPIO,
371: "SDGPIO_Write(): Entered.\n");
372:
373: /* Check for a valid handle */
374: if (strcmp(SDGPIO_INFO(handle)->idString, SDGPIO_ID_STRING) != 0)
375: {
376: DEBUG_PRINTF(DEBUG_ERROR, MODULE_SDGPIO,
377: "SDGPIO_Config(): Error invalid handle.\n");
378:
379: return SDGPIO_ERROR_INVALID_HANDLE;
380: }
381:
382: /* Read existing value for PC20X_CFG_SDGPIO_OUTPUT_x */

– The bad pointer is passed into the function, so next we need to know where it was called
from to try and trace back where the bad pointer came from. For this, we need to identify this
function’s stack frame in the stack dump.
17
 CONFIDENTIAL

Stack frames (1)

To use the information in the stack dump, you need to know how the gcc
compiler arranges the stack into ‘stack frames’.

– Return address
– Context (saved registers)
– Local variables

18
 CONFIDENTIAL

Stack frames (2)

Stack frame set up at the start of a function:

000cc6b4 <SDGPIO_Write>:
SDGPIO_Write():
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/SDGPIO/src/
sdgpio.c:367
cc6b4: e92d47f0 push {r4, r5, r6, r7, r8, r9, sl, lr}
cc6b8: e1a05001 mov r5, r1
cc6bc: e24dd010 sub sp, sp, #16 ; 0x10

Stack dump rearranged:

0x4bd01de0: 0x4bd01dee 0x00000001 0x00000001 0x001ba7f4 <- local variables

r4 r5 r6 r7 r8 r9 r10(sl)
0x4bd01df0: 0x00000001 0x403aa810 0x00000001 0x403d2978 0x00000010 0x000000c3 0x00000000 <- saved context

lr
0x4bd01e0c: 0x402daac8 <- return address

19
 CONFIDENTIAL

Digression (3) – ARM register naming

– Apart from the program counter and link register, all ARM registers are ‘general
purpose’.
– Special functions (and names) are by convention only.

– r0 to r9 - general purpose (see below)

– r10 - aka sl (stack limit)
– r11 - aka fp (frame pointer)
– r12 - aka ip (inter-procedure call scratch register)
– r13 - aka sp (stack pointer)
– r14 - aka lr (link register)

– r0 to r3 - aka a1 to a4 – working registers/function arguments

– R4 to r9 - aka v1 to v6 – variables, must be preserved

20
 CONFIDENTIAL

Unwind the stack (3)

– The return address is 0x402daac8, which is in another library. We go back to the

memory map to find out which one:
402d3000-402ea000 r-xp 00000000 1f:05 11557752 /opt/ipaccess/lib/libsdgpioutil.so.0.167.8.0

– Calculate the offset into the library code:

0x402daac8 – 0x402d3000 = 0x00007ac8

– Install and disassemble this ipackage:

bash$ ar -x /<path to your builds>/SR0.4.0_0.5.0_AP_379.8.0/ipkg/arm/arm-devdrv_167.8.0_arm.ipk
bash$ tar -xvzf data.tar.gz
bash$ /opt/codesourcery/arm-2008q3/bin/arm-none-linux-gnueabi-objdump -dl opt/ipaccess/lib/libsdgpioutil.so > libsdgpioutil.txt

– Search the disassembled library for “7ac8:”:

SetData():
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/sdgpio_util/src/spi_bitbash.c:248
7abc: e3a01000 mov r1, #0 ; 0x0
7ac0: e1a00006 mov r0, r6
7ac4: ebfff91f bl 5f48 <_init+0xac>
readWordGeneric():
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/sdgpio_util/src/spi_bitbash.c:540
7ac8: e3a00001 mov r0, #1 ; 0x1
7acc: ebfff9b9 bl 61b8 <_init+0x31c>
SetCS():
/storage/rel_daily_builds/build379.8rebuild_sandbox/target/platform/PC202/arm/arm_sdgpio_util/sdgpio_util/src/spi_bitbash.c:333

21
 CONFIDENTIAL

Digression (4) – Compiler optimisation (2)

– The setData() function is very short, and declared static, so rather than incur the
overhead of a call, context save, context restore and return, the compiler has decided it
is more efficient to inline the code.

0000795c <readWordGeneric>:
readWordGeneric():
decodeClkType():
SetCS():
decodeClkType():
SetCS():
readWordGeneric():
SetClkMask():
SetClk():
SetData():
readWordGeneric():
SetCS():
.
.
.

22
 CONFIDENTIAL

Analyse the source (2)

– So the SDGPIO_Write function was called from line 248 of the spi_bitbash.c source file:

242: static void SetData( SDGPIO_Handle_t handle, int useTemperatureBus, int data )
243: {
244: unsigned int bit = 0;
245: unsigned int mask = 0;
246:
247: SetDataMask( useTemperatureBus, data, &mask, &bit );
248: SDGPIO_Write(handle,
249: PA_DEV_NUM,
250: mask,
251: bit );
252:
253: DEBUG_PRINTF(1,"", "set data:%d mask=0x%08X\r\n", bit, mask);
254:
255: }

23