coLinux debugging ================= Table of Contents colinux-debug-daemon.

exe Command Line Parameters Short guide to see more error messages Stepguide for colinux-daebug-daemon Example debuging over network Howto see debug prints from kernel driver Howto use Windows crash dump Howto see label names from minidump - script way Howto convert label names from minidump - manualy way Testing different versions colinux-debug-daemon.exe -----------------------This is a configurator and debug logger. You need to start it before you wish to see more internals from coLinux system. Start it after colinux driver is installed (normal case after the Installer was done, or after you runs colinux-daemon --install-driver) and before launch your Linux via colinux-daemon. Command Line Parameters -----------------------d Download debug information on the fly from driver. Without -d, uses standard input. -p Parse the debug information and output an XML -f filename File to append the output, instead of writing to standard output. Write to file with flush on every line. -s level=num,level2=num2,... Change the levels of the given debug facilities. A highter value give more outputs, 0 disable the facility. -n ipaddress Send logs as UDP packets to ipaddress:63000 (requires -d) -e exitcode Translate exitcode into human readable format. -h Shows a short help text

Short guide to see more error messages -------------------------------------Open a windows command prompt, change into coLinux install directory and run the debug-logger: C: cd \coLinux colinux-debug-daemon.exe -d -p -s prints=31,misc=31 -f debug.xml Than start colinux (as console application or as service). Whenever you means that the trace point was done, stop debug logger with key CTRL-C on the colinux-debug-daemon. Open the file debug.xml with Internet Explorer or notepad and view near the end of file. Search the text, you was seen from colinux-daemon and view near this linies in the debug.xml. Mostly the debug says more about your problem as the colinux-daemon console.

Stepguide for colinux-daebug-daemon ----------------------------------1) Open a Command Prompt or shell window and cd to the coLinux directory, e.g. "cd c:\colinux" 2) To see a list of options: colinux-debug-daemon -h 3) To see a list of "facility" names: colinux-debug-daemon -d (Type ^C to exit back to the cmd prompt.) * "facilities" are different types of messages that the daemon can capture. The current list of facility names is: misc, network, messages, prints, blockdev, allocations, context_switch, pip e * Each facility has an associated level number - higher numbers capture more messages. Facility can be turn off with value 0. * See below for how to set the facility level 4) To capture and display some messages: colinux-debug-daemon -d -p -f mylogfile.xml -s network=11,blockdev=9 * "-d" makes it get the raw debug messages in real time from a running colinux session. Without "-d", it gets the raw debug messages from standard input, presumably from a previously-captured file. * "-p" makes it decode (parse) the raw debug messages and display them in human-readable form (encapsulated in XML markup). Without "-p", I presume that it just sends the raw debug messages to the output, perhaps so you can capture them to a file in a compact form for later inspection. * "-f mylogfile.xml" makes it append the output to the file

"mylogfile.xml". Without "-f", it sends the output to standard output. As far as I can tell, "-f filename" is equivalent to ">>filename", but it flushes more file caches. * "-s network=11,blockdev=9" sets the "network" facility level to 11, thus capturing network messages sent at levels 0-11, and the "blockdev" facility level to 9. * To determine the desired facility level, either increase the number (trial and error) until you see the level of detail you need, or inspect the source code for the component you wish to debug to see the level at which the desired messages are sent. 5) There is also a "-n ip-address" switch that will send the raw logs to UDP port 63000 at the indicated IP address, instead of to standard output or a file. 6) There is a python program to listen on port 63000 and write the logs to a file. The python program is in the source tarball, in src/colinux/user/debug/server.py 7) There is a python program to decode and display the XML-formatted (preparsed) messages. It is in the source tarball, in src/colinux/user/debug/dump.py . It is not strictly necessary to use this program; the XML-formatted messages are in ASCII and are easy enough to read directly. With a small shell command under Linux you can extract the text: sed -ne 's|<string>\(.*\)</string>|\1|p' <debug.xml >debug.txt 8) The source code for colinux-debug-daemon is in the source tarball, in src/colinux/user/debug/main.c 9) The source code for various other colinux helper daemons (i.e. the things tha t ultimately send the debug messages) is in the source tarball in various place s. For example, the bridged network daemon source for Windows is in src/colinux/os/winnt/user/conet-bridged-daemon/ 10) Debug messages are sent by calls like the following: co_debug_lvl(network, 13, "packet sent (0x%x written)\n", write_size); (Mitch Bradley via Mail) Example debuging over network ----------------------------You can run coLinx on one PC (TARGET) and send all debugging messages over network to an other PC (DEVEL). This is very helpfull, if your colinux crashes your PC and you can not log into local filesystem. Run small Python script inside coLinux sources, to download all raw debugging datas into file: DEVEL: python src/colinux/user/debug/server.py debug.bin Or run Netcat (Linux) as debug server: DEVEL: netcat -l -u -p 63000 > debug.bin

On the other PC open command prompt, run debug logger with network option '-n' and the ipaddress to your 'server' PC: TARGET: colinux-debug-daemon.exe -d -s prints=31,misc=31 -n ip.of.devel.pc Open a second command prompt and run colinux: TARGET: colinux-daemon.exe @colinux.conf Can also run the debug-daemon after coLinux was started. The debug session should start, before you think is interesting. After finish your debug session stop colinux-debug-daemon with CTRL-C and stop the server.py (or netcat) with CTRL-C. Convert the binary data into readable XML format: DEVEL: colinux-debug-daemon -p < debug.bin > debug.xml Now can convert the XML file into a text file with one line per log entry: DEVEL: python src/colinux/user/debug/dump.py debug.xml > debug.txt Typicaly output for debug.txt are such lines: DEVEL: cat debug.txt 1|00000000.496810439|colinux-daemon|src/colinux/os/winnt/user/misc.c:co_term inal_printv:37|prints "Cooperative Linux Daemon, 0.8.0" 2|00000000.496824313|colinux-daemon|src/colinux/os/winnt/user/misc.c:co_term inal_printv:37|prints "Compiled on Jul 22 2007 21:59:05" 3|00000000.496826571|colinux-daemon|src/colinux/os/winnt/user/misc.c:co_term inal_printv:37|prints "" 4|00000000.496827397|colinux-daemon|src/colinux/user/config.c:co_parse_confi g_args:713|using 'vmlinux' as kernel image 5|00000000.496828810|colinux-daemon|src/colinux/user/config.c:parse_config_a rgs:635|configuring 32 MB of virtual RAM ... Howto see debug prints from kernel driver ----------------------------------------Run regedit, create a registry key "HKEY_LOCAL_MACHINE\SOFTWARE\coLinux", create a new DWORD with Name "Debug" and set the value to 1F (hex). Or create such file colinux-debug.reg and import it: Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\coLinux] "Debug"=dword:0000001f Download DbgView.exe from http://www.sysinternals.com ( http://technet.microsoft.com/en-us/sysinternals/bb896647.aspx ), run the DbgView.exe, relaod the coLinux driver with: cd CurentColinuxPath colinux-daemon.exe --remove-driver colinux-daemon.exe --install-driver

Start coLinux with "colinux-daemon @YourConfig" Watch the output from DbgView, typicaly will see such lines: [m:colinux-driver f:0 l:10 loaded to host kernel [m:colinux-driver f:0 l:10 machine has 480 MB of RAM [m:colinux-driver f:0 l:10 _limit] machine RAM use limit: src/colinux/kernel/manager.c:49:co_manager_load] src/colinux/kernel/manager.c:63:co_manager_load] src/colinux/kernel/manager.c:37:set_hostmem_usage 416 MB

Every line has the follow format: m: module f: facility (0=misc) l: level following by filename, linenumber, function name and the text. The registry key will only read on driver loading, typicaly on booting Windows. Debug outputs decreases coLinux running speed. Remember to remove the registry key Debug, or set it to "0" for normal running mode. Howto use Windows crash dump ---------------------------Enable minidump (the small option, 64KB). Not need a full core dump. Don't set the ckeckbox for "automatic reboot after crash dump". The menu will find via keys Windows+Pause (or Control Panel * System Properties) * Register: Extentions * Frame: Start and Recover * Button: Configure * Frame: System errors. If you think, coLinux would crash on start or with your next doing, then should restart Windows after your last changes on configuration and before you starts coLinux. This saves your last doing and write back all file buffers to disk. Typicaly BSOD have such lines: *** STOP: 0x1111777A (0x00000000, 0x00000000, 0x00000000, 0x00000000) *** linux.sys - Adress F799992F0 base at F7989000, DateStamp 46c6e118 0x1111777A in this case is an call of BugCheck, because coLinux detects an internal error. There exists some more with similar error code 0x11117777 to 0x1111777F. Grep into coLinux source and locate the "0x11117777 + n", where 'n' is the number '3' in the example. F7989000 and F799992F0 in this case needs to know to find any address from windows crash in the dumped linux.sys later. Note all the hex numbers exactly down to paper and restart Windows. In Event Manager finds mostly the hex numbers later, but not in all cases. Download from "Install Debugging Tools for Windows 32-bit Version" http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx the Current Release of "Download the Debugging Tools for Windows" http://msdl.microsoft.com/download/symbols/debuggers/dbg_x86_6.7.05.1.exe approx. 15MB. Windows will say, that no symbols was found. That is no problem. Windows will never seen the GCC symbols. XP Symbol names we not need. You can

download and install they from "Download Windows Symbol Packages" http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx approx. 200MB. But worth downloading and installing IMHO. Install they, if you wish to see more Windows names (see example follow). Symbols can online use. This loads symbols only for used files and stores it on local cache (typicaly 4MB). For example, to download symbols to c:\websymbols, you would add the following to your symbol path: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols Read more on http://www.microsoft.com/whdc/devtools/debugging/debugstart.mspx Please don't change or replace the file "linux.sys" from there you got the crash before the next step! Run WinDbg.exe, and "Open Crash-Dump ...". The file typicaly exists as C:\WINDOWS\Minidump\Mini081807-01.dmp (file name is coded by current date). In the prompt "kd>" run !analyze -v and than "Write windows text to file ..." (in the menu "edit"). CoLinux developers interest for the follow lines: BugCheck 1111777A, {f79b02f0, 8b606015, 0, 0} ... Probably caused by : linux.sys ( linux+9a48 ) ... STACK_TEXT: (... please all lines ...) This lets see the Stack, for example: STACK_TEXT: WARNING: Stack unwind information not available. Following frames may be wrong . f7a54b04 f79a9a48 1111777a f79b02f0 8b606015 nt+0x21925 f7a54b34 f79a8a07 00000000 f79b02f0 8b606015 linux+0x9a48 f7a54b54 f79a609d 8147e000 00000001 814844cc linux+0x8a07 f7a54b84 f79a206a 8155bf60 814e2000 f7a54bb4 linux+0x609d f7a54bc4 f79a7189 8155bf60 00000011 814e2000 linux+0x206a f7a54c34 804eddf9 8155bea8 81497ed0 806d02d0 linux+0x7189 f7a54c58 805749d1 8155bea8 81497ed0 814fbe78 nt+0x16df9 f7a54d00 8056d33c 000007c4 00000000 00000000 nt+0x9d9d1 If you have installed the Windows symbols and set the path, then all the labels with "nt+..." would see with the windows names, for example: f7a54b04 f7a54b34 f7a54b54 f7a54b84 f7a54bc4 f7a54c34 f7a54c58 f7a54d00 x6c6 f79a9a48 f79a8a07 f79a609d f79a206a f79a7189 804eddf9 805749d1 8056d33c 1111777a 00000000 8147e000 8155bf60 8155bf60 8155bea8 8155bea8 000007c4 f79b02f0 f79b02f0 00000001 814e2000 00000011 81497ed0 81497ed0 00000000 8b606015 8b606015 814844cc f7a54bb4 814e2000 806d02d0 814fbe78 00000000 nt!KeBugCheck+0x14 linux+0x9a48 linux+0x8a07 linux+0x609d linux+0x206a linux+0x7189 nt!IofCallDriver+0x32 nt!SeQueryAuthenticationIdToken+0

Is a liddle bit better to read. But names from linux.sys can't see.

Howto see label names from minidump - script way -----------------------------------------------Locate the file bin/mindump-repaint-labels.sh inside coLinux source. Unpack linux.sys from archive daemons-*.dbg.zip from same build date as the BSOD was triggered. If you have self build coLinux, then use the file from src/colinux/os/winnt/build/linux.sys. "minidump.txt" in the example is the saved output from "!analyze -v". Run the script with paths to your files: mindump-repaint-labels.sh linux.sys < minidump.txt > backtrace.txt View into file backtrace.txt, the caller stack should be more readable now: f7a54b04 f79a9a48 f7a54b34 f79a8a07 (linux+0x9a48) f7a54b54 f79a609d (linux+0x8a07) f7a54b84 f79a206a (linux+0x609d) f7a54bc4 f79a7189 (linux+0x206a) f7a54c34 804eddf9 (linux+0x7189) f7a54c58 805749d1 f7a54d00 8056d33c x6c6 1111777a f79b02f0 8b606015 nt!KeBugCheck+0x14 00000000 f79b02f0 8b606015 linux!co_os_malloc+0x28 8147e000 00000001 814844cc linux!co_monitor_os_init+0x17 8155bf60 814e2000 f7a54bb4 linux!co_monitor_create+0x18D 8155bf60 00000011 814e2000 linux!co_manager_ioctl+0x2E8 8155bea8 81497ed0 806d02d0 linux!dispatch_wrapper+0x419 8155bea8 81497ed0 814fbe78 nt!IofCallDriver+0x32 000007c4 00000000 00000000 nt!SeQueryAuthenticationIdToken+0

Please double check, that you use the right linux.sys from crash. Other case the label names would give function calls, that never would find in source. Howto convert label names from minidump - manualy way ----------------------------------------------------If the must convert by hands, you need the coLinux build environment under Linux. Or objdump.exe from mingw32 to run directly under windows. There start: i686-pc-mingw32-objdump -D linux.sys > linux.sys.dump Open linux.sys.dump and view the first lines: linux.sys: file format pei-i386 Disassembly of section .text: 00011000 <_co_monitor_get_pfn>: 11000: 55 11001: 89 e5

push mov

%ebp %esp,%ebp

You will see, that linux.sys starts with code 0x00011000. An other offset is the 0x1000 in the linker params. So, the real offset from windows runtime WinDbg to the dump is 0x10000. For example "linux+0x9a48" is 0x19a48 in the dump. Locate "19a48:" in linux.sys.dump and view some lines before to get the function name. The top of function give the start address. Sub the address 00019a20 from stack address 18a48 to have the relative offset 0x28. The stack have now

decoded as "linux!co_os_malloc+0x28". 00019a20 <_co_os_malloc>: ... 19a3d: 68 7a 77 11 11 push pushed the parameter for KeBugCheck 19a42: ff 15 74 31 02 00 call is the call to KeBugCheck 19a48: 83 c4 0c add address is shown in stack as 'linux+0x9a48'

$0x1111777a *0x23174 $0xc,%esp

<-- this <-- this <-- this

The next stack label is "linux+0x8a07". Add the 10000 to have 18a07 and locate this in the linux.sys.dump file, pick the function name from some lines before 18a07. Now have the next caller. Do the same steps for all. 000189f0 <_co_monitor_os_init>: ... 18a02: e8 19 10 00 00 18a07: 83 c4 10 00015f10 <_co_monitor_create>: ... 16098: e8 53 29 00 00 1609d: 83 c4 10 00011d80 <_co_manager_ioctl>: ... 12065: e8 a6 3e 00 00 1206a: 83 c4 10 00016d70 <_dispatch_wrapper@8>: ... 17184: e8 f7 ab ff ff 17189: 83 c4 20

call add

19a20 <_co_os_malloc> $0x10,%esp

call add

189f0 <_co_monitor_os_init> $0x10,%esp

call add

15f10 <_co_monitor_create> $0x10,%esp

call add

11d80 <_co_manager_ioctl> $0x20,%esp

Write down the coLinux names. The stack can better read now: f7a54b04 f79a9a48 1111777a f7a54b34 f79a8a07 00000000 _os_malloc+0x28 f7a54b54 f79a609d 8147e000 _monitor_os_init+0x17 f7a54b84 f79a206a 8155bf60 _monitor_create+0x18d f7a54bc4 f79a7189 8155bf60 _manager_ioctl+0x2e8 f7a54c34 804eddf9 8155bea8 spatch_wrapper+0x419 f7a54c58 805749d1 8155bea8 f7a54d00 8056d33c 000007c4 x6c6 f79b02f0 8b606015 nt!KeBugCheck+0x14 f79b02f0 8b606015 linux+0x9a48 00000001 814844cc linux+0x8a07 814e2000 f7a54bb4 linux+0x609d 00000011 814e2000 linux+0x206a 81497ed0 806d02d0 linux+0x7189 linux!co linux!co linux!co linux!co linux!di

81497ed0 814fbe78 nt!IofCallDriver+0x32 00000000 00000000 nt!SeQueryAuthenticationIdToken+0

The NT kernel calls are in the list of imports. For example the "call *0x23174" can find in the dump as: 00023174 <__imp__KeBugCheckEx@20>: 23174: 4c 23175: 33 01 dec xor %esp (%ecx),%eax

This example was written down by forcing BugCheck with a special hacked source via a "co_os_alloc(0)". In the real run you would have other addresses in the stack list. Testing different versions -------------------------If you can't analyse the stack self, you can send it to coLinux developers. Very interesting is the exactly build date of the file linux.sys. Please give the name or date of snapshort you used. Good idea is to note the file size. The stack can only properly view, if we use exactly the same linux.sys for objdump, that created the BSOD. If you are testing various coLinux versions, then should be shure, that you have clean load the driver linux.sys with the colinux-daemon.exe from same build. Check, that linux.sys and colinux-daemon.exe in your directory have the same build date. A file "overwrite" of the file linux.sys is not enouth to use this driver. To be shure, run: cd CurentColinuxPath colinux-daemon.exe --remove-driver colinux-daemon.exe --status-driver cd NewColinuxPath colinux-daemon.exe --install-driver colinux-daemon.exe --status-driver

<--- should say: Driver not installed

In some heavy cases needs to reboot Windows between remove and install.