Debugging

Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction
Debugging Tricks with Apache HTTP Server
What kinds of
issues
2.4
encountered

Using tools to
look inside the
web server
Jeff Trawick
Looking from
the outside http://emptyhammock.com/
What if you
trawick@emptyhammock.com
build the code
differently

Compare with
April 7, 2014
httpd 2.2

References
and further
reading
 Get these slides...

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered

Using tools to
http://emptyhammock.com/projects/info/slides.html
look inside the
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Table of Contents

Debugging
Tricks with
Apache HTTP
Server 2.4 1 Introduction
Jeff Trawick

Introduction
2 What kinds of issues encountered
What kinds of
issues
encountered
3 Using tools to look inside the web server
Using tools to
look inside the
web server
4 Looking from the outside
Looking from
the outside 5 What if you build the code differently
What if you
build the code
differently 6 Compare with httpd 2.2
Compare with
httpd 2.2
7 References and further reading
References
and further
reading
 Introduction — Who am I?

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
I’ve worked at
several large corporations, for over two decades
Introduction
my own one-person company, Emptyhammock, for the last
What kinds of
issues two years
encountered

Using tools to I’ve worked on

look inside the
web server several products which were primarily based on or
Looking from otherwise included Apache HTTP Server
the outside
lower-level networking products
What if you
build the code web applications
differently

Compare with
I’ve debugged many customer and user problems over the
httpd 2.2 years.
References
and further
reading
 Introduction — What will we attempt to cover?

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues Touch on all the basics.
encountered

Using tools to
Describe all the new httpd 2.4 debugging features.
look inside the
web server Summarize the techniques which are different with httpd
Looking from 2.2.
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 What kinds of issues encountered

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
Crash High CPU
Introduction
Hang of server High memory
What kinds of
issues
encountered Stall of individual requests High consumption of other
Using tools to
pooled resources
look inside the
Termination
web server Incorrect output - wrong
Bad response time
Looking from transformation
the outside
Limited concurrency
What if you
Incorrect output -
build the code without problem
differently
missing/bad protocol
symptoms
Compare with element
httpd 2.2

References
and further
reading
 Non-problems

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered
Validate behavior of new software/configuration
Using tools to Understand steady-state behavior for baseline when
look inside the
web server something is wrong
Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Using tools to look inside the web server

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction
Logging (the information itself, the timestamp, information
What kinds of
about other processing at about the same time)
issues
encountered OS-level tools (view use of resources, whether discrete
Using tools to items like files or continuous like CPU)
look inside the
web server CPU-, code-level tools (determine what code is running
Looking from
the outside
frequently, what is running for the request, analyze
What if you
memory references, walk through the processing of a
build the code
differently
request, etc.)
Compare with
httpd 2.2

References
and further
reading
 Careful with logging!

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick As you increase the level of logging, you increase the chances
that private data will be logged.
Introduction

What kinds of Passwords, session keys, etc.

issues
encountered

Using tools to Modules/log configurations of particular interest:

look inside the
web server mod dumpio, mod log config when configured to log
Looking from certain request or response header fields
the outside

What if you mod log forensic

build the code
differently http (the built-in module) when configured at higher trace
Compare with
httpd 2.2
levels
References
and further
reading
 Logging

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered

Using tools to
Error log
look inside the
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Error log records

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick Configurable content

Introduction Fields dropped when information is unavailable
What kinds of Third-party modules can implement their own fields
issues
encountered
Typical message:
Using tools to
look inside the
web server [Sun Oct 28 13:37:27.676386 2012] [:error]
Looking from [pid 14340:tid 140625844377344] [client 127.0.0.1:50837]
the outside
mod_wsgi (pid=14340): Target WSGI script
What if you
build the code '/home/trawick/myhg/apache/documents/AC20
differently 12EU/lookup.wsgi' does not contain WSGI
Compare with application 'application'.
httpd 2.2

References
and further
reading
 Hiding error log fields

Debugging The ErrorLogFormat directive can limit which fields are logged, but you could implement post-processing
Tricks with to remove fields as appropriate for what you are debugging.
Apache HTTP
Server 2.4 My own silly mechanism:

Jeff Trawick $ grep Accept-Ranges logs/error_log

[Thu Apr 03 07:26:49.605322 2014] [http:trace4] [pid 13680:tid 140130244732672] http_filters.c(83
Introduction $ grep Accept-Ranges logs/error_log | nots.pl
[http:trace4] [pid 13680:tid 140130244732672] http_filters.c(837): [client 192.168.1.207:60141]
What kinds of $ grep Accept-Ranges logs/error_log | nots.pl | nomodlevel.pl
issues [pid 13680:tid 140130244732672] http_filters.c(837): [client 192.168.1.207:60141] Accept-Ranges
encountered $ grep Accept-Ranges logs/error_log | nots.pl | nomodlevel.pl | nopidtid.pl
http_filters.c(837): [client 192.168.1.207:60141] Accept-Ranges: bytes
Using tools to $ grep Accept-Ranges logs/error_log | nots.pl | nomodlevel.pl | nopidtid.pl | noclient.pl
look inside the http_filters.c(837): Accept-Ranges: bytes
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Detailed logging only for specified client IP

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
LogLevel info
Introduction <If "%{REMOTE_ADDR} =~ /127.0.0/">
What kinds of LogLevel trace8
issues
encountered </If>
Using tools to
look inside the
web server
Only works once request processing has reached a certain
Looking from
the outside point. Connection-level issues which occur before that
What if you point won’t be logged.
build the code
differently Prior to 2.4.4, this expression needed to be placed inside a
Compare with Location container to be effective.
httpd 2.2

References
and further
reading
 Detailed logging for problematic requests

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues LogLevel info
encountered
<Location /problem/>
Using tools to
look inside the LogLevel trace8
web server
</Location>
Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 (Mostly) HTTP layer logging at different levels

Debugging
Tricks with [core:trace5] Request received from client: GET / HTTP/1.1
Apache HTTP [http:trace4] Headers received from client:
Server 2.4 [http:trace4] Connection: keep-alive
[http:trace4] Cache-Control: max-age=0
Jeff Trawick [http:trace4] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KH...
[http:trace4] Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Introduction [http:trace4] Accept-Encoding: gzip,deflate,sdch
[http:trace4] Accept-Language: en-US,en;q=0.8
What kinds of
[http:trace4] Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
issues
encountered [http:trace4] If-None-Match: \\"2d-4b1922bade1c0\\"
[http:trace4] If-Modified-Since: Sat, 12 Nov 2011 23:41:03 GMT
Using tools to [http:trace3] Response sent with status 304, headers:
look inside the [http:trace5] Date: Tue, 06 Nov 2012 12:18:57 GMT
web server [http:trace5] Server: Apache/2.4.4-dev (Unix) OpenSSL/1.0.0e mod_wsgi/3.4 Python...
Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 mod log debug

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
Configurable debug logging mechanism using new
Introduction LogMessage directive.
What kinds of
issues
Different ways to think of it:
encountered
Generate custom trace or error messages for processing of
Using tools to
look inside the
interest to you.
web server Track interesting values as they change (or not) during
Looking from request processing.
the outside

What if you
Conditional expression support with access to dynamic
build the code values is provided by the new ap expr support.
differently
http://httpd.apache.org/docs/2.4/expr.html
Compare with
httpd 2.2

References
and further
reading
 mod log debug – sample configuration

Debugging
Tricks with
Apache HTTP
# Log some module's request note at all phases
Server 2.4 # of processing (but only if set)
Jeff Trawick <Location />
LogMessage "%{note:mod_your_debug}" hook=all \
Introduction
"expr=-T %{note:mod_your_debug}"
What kinds of </Location>
issues
encountered

Using tools to
# Log when a location is requested as a subrequest
look inside the <Location /app/dash/>
web server
LogMessage "subrequest to /app/dash/" \
Looking from hook=type_checker "expr=-T %{IS_SUBREQ}"
the outside
</Location>
What if you
build the code
differently # Log when a particular error is encountered
Compare with LogMessage "Timeout from %{REMOTE_ADDR}" \
httpd 2.2 "expr=%{REQUEST_STATUS} = 408"
References
and further
reading
 mod dumpio

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
This is a way to trace the raw, unencrypted data exchange
Introduction

What kinds of
into the error log.
issues
encountered
A packet trace is usually preferable, but in some
Using tools to
environments it is simpler to modify the httpd
look inside the configuration to enable this module than it is to capture
web server
packets.
Looking from
the outside Also, if the person analyzing diagnostic data won’t have
What if you access to server keys, a packet trace can’t be used to
build the code
differently understand most application-layer issues.
Compare with
httpd 2.2

References
and further
reading
 mod dumpio configuration

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered LogLevel info dumpio:trace7
Using tools to DumpIOInput On
look inside the
web server
DumpIOOutput On
Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 mod dumpio output

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick You can see I/O operations on input as well as input and
Introduction
output data.
What kinds of
issues dumpio_in [getline-blocking] 0 readbytes
encountered
dumpio_in (data-HEAP): 20 bytes
Using tools to dumpio_in (data-HEAP): GET /dir/ HTTP/1.1\r\n
look inside the
web server dumpio_in [getline-blocking] 0 readbytes
Looking from dumpio_in (data-HEAP): 22 bytes
the outside
...
What if you
build the code dumpio_in (data-HEAP): Connection: keep-alive\r\n
differently

Compare with extraneous information removed

httpd 2.2

References
and further
reading
 Catching requests which do not finish

Debugging
Tricks with
(presumably due to a child process crash, though you could
Apache HTTP potentially identify hung requests if you don’t use mod status)
Server 2.4

Jeff Trawick LoadModule log_forensic_module modules/mod_log_forensic.so

Introduction
ForensicLog logs/forensic.log
What kinds of
issues This logs the start and end of the request along with all of the
encountered
request headers.
Using tools to
look inside the
web server
+UJggYn8AAQEAAAs1da4AAAAA|GET / HTTP/1.1|Host...
-UJggYn8AAQEAAAs1da4AAAAA
Looking from
the outside

What if you
build the code
check_forensic will scan the log and determine which
differently requests didn’t finish cleanly.
Compare with
httpd 2.2
Compare with mod whatkilledus, described later.
References
and further
reading
 Where did that error message come from?

Debugging
Tricks with module id in error log:
Apache HTTP
Server 2.4 [core:info] [pid 4373:tid 140043736946432] ...
Jeff Trawick AH00128: File does not exist: ...
whoops, missing module id:
Introduction
... [:info] [pid 8889:tid 140363200112416] mod_wsgi
What kinds of
issues (pid=8889): Initializing Python.
encountered
In this case it is obviously mod wsgi, but it isn’t always that
Using tools to
look inside the
easy. (FWIW, the fix is in mod wsgi issue 292.)
web server ... [:error] [pid 14883:tid 140625458312960] 1
Looking from ... [:error] [pid 14883:tid 140625458312960] 2
the outside
... [:error] [pid 14883:tid 140625458312960] 3
What if you
build the code
... [:error] [pid 14883:tid 140625458312960] 4
differently (That was mod wsgi logging stderr from a script.)
Compare with
httpd 2.2

References
and further
reading
 Where did that error message come from?

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction
The module id in the error log records is your hint on
What kinds of
issues controlling the log level to see or hide the message.
encountered
core:info Configure a specific LogLevel for module core to
Using tools to
look inside the see or hide this.
web server

Looking from :error No module is available, so this log message can’t

the outside
be controlled with a module-specific LogLevel.
What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 mod backtrace feature to identify message source

Debugging
Tricks with
Apache HTTP
Server 2.4 Consider this if no module id is available or you need to know
Jeff Trawick the caller of a utility function that logged a message.
Introduction mod backtrace has the capability of adding a backtrace to
What kinds of error log messages in certain conditions.
issues
encountered ErrorLogFormat ... [%{/AH00128/}B] ...
Using tools to
look inside the If the search string appears in the message, a
web server
mini-backtrace will appear as an additional field in the
Looking from
the outside error log record.
What if you ... [0x4453dd<ap_run_handler<ap_invoke_handler<
build the code
differently ap_process_async_request<ap_process_request] ...
Compare with AH00128:...
httpd 2.2

References
http://emptyhammock.com/projects/httpd/diag/
and further
reading
 Looking from the outside

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered examining resource use
Using tools to
look inside the tracing activity
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Resource use

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered

Using tools to
top/iostat/vmstat/etc. (even ps)
look inside the
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 System call trace

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered

Using tools to
strace/truss/dtruss
look inside the
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Higher level tools

Debugging
Tricks with
Apache HTTP
Server 2.4 Brendan Gregg’s DTrace Toolkit, at
Jeff Trawick http://www.brendangregg.com/dtracetoolkit.html
Introduction

What kinds of The DTrace Toolkit has been around for a while and
issues
encountered contains a number of analysis and reporting scripts based
Using tools to on DTrace.
look inside the
web server sysdig, at http://www.sysdig.org/
Looking from
the outside

What if you
sysdig was just announced last week. I haven’t played with
build the code
differently
it much yet; the Lua scripts, chisels, appear to operate at
Compare with
roughly the same layer as the scripts in the DTrace
httpd 2.2 Toolkit, and a lower-level command provides the basic
References
and further
collection features.
reading
 A simple DTrace Toolkit example...

Debugging
Tricks with
Apache HTTP $ sudo /usr/share/dtrace/toolkit/procsystime -n httpd
Server 2.4 ^C
Jeff Trawick
Elapsed Times for processes httpd,
Introduction

What kinds of SYSCALL TIME (ns)

issues
encountered ...
Using tools to accept4 25569461
look inside the close 29081544
web server
stat 36630193
Looking from
the outside munmap 41668446
What if you
writev 48378858
build the code shutdown 71471901
differently
gettimeofday 97454962
Compare with write 1000753076
httpd 2.2
select 8131189175
References
and further _umtx_op 22781598217
reading kevent 32804433802
 A simple sysdig example...

Debugging
Tricks with
Apache HTTP $ sudo sysdig -n 1000 -c topfiles_bytes proc.name=httpd
Server 2.4

Jeff Trawick Bytes Filename

------------------------------
Introduction
9.27KB /home/trawick/inst/24-64/logs/forensic.log
What kinds of 960B /home/trawick/inst/24-64/logs/access_log
issues
encountered 494B /home/trawick/inst/24-64/manual/mod/module-dict.html
Using tools to
look inside the
web server

Looking from
the outside
(I forgot that I had enabled mod log forensic...)
What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 DTrace one-liners, on several platforms

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered

Using tools to
look inside the
Jeff, this is where you view the document in the browser.
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Looking inside the process with a debugger

Debugging
Tricks with
Apache HTTP Basic information: Backtraces
Server 2.4

Jeff Trawick
gdb
Most platforms (even Windows, using MinGW gdb on
Introduction
MinGW build of httpd)
What kinds of
issues
Basic use:
encountered
gdb /path/to/httpd pid-or-corefile
Using tools to (gdb) thread apply all bt full
look inside the
web server (but other commands may be useful too)
Looking from
the outside pstack
What if you Solaris (I learned through bad experiences to pretend that
build the code
differently pstack isn’t available on Linux)
Compare with
Use:
httpd 2.2
pstack pid-or-corefile
References
and further (but pflags and pldd information is also good)
reading
 Getting more debugging information

Debugging
Tricks with
Apache HTTP
Server 2.4 The backtraces (with variables if available) are most
Jeff Trawick important, but more information is available if you ask for
Introduction it.
What kinds of gdb, more details:
issues
encountered (gdb) info sharedlibrary
Using tools to (gdb) info threads
look inside the
web server (gdb) thread apply all bt full
Looking from (gdb) thread apply all x/i $pc
the outside

What if you
Solaris /proc tools:
build the code
differently # pstack 13579
Compare with
# pldd 13579
httpd 2.2 # pflags 13579
References # pmap 13579
and further
reading
 httpd-specific gdb tricks

Debugging
Tricks with (gdb) source /path/to/2.4.x/.gdbinit
Apache HTTP (gdb) dump_table r->headers_in
Server 2.4
[0] 'Host'='127.0.0.1:8080' [0x7f8094003cb6]
Jeff Trawick [1] 'Connection'='close' [0x7f8094003cd4]
(gdb) dump_brigade b
Introduction dump of brigade 0x7f8094007320
What kinds of | type (address) | length | data addr | contents | rc
issues ------------------------------------------------------------------------------
encountered 0 | FILE (0x7f8094000b08) | 45 | 0x7f8094000ba8 | [**unp... | 1
Using tools to 1 | EOS (0x7f8094000c48) | 0 | 0x00000000 | | n/a
look inside the end of brigade
web server

Looking from Use of these macros requires some familiarity with the httpd
the outside implementation or module programming interface.
What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Example output

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered Jeff, this is where you show ubuntu64.core.collect.gdbout and
Using tools to
look inside the
solaris10.core.pstackout.
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Umm, what does that stuff mean?

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
Recognize normal behavior
encountered
Determine where crash likely occurred
Using tools to
look inside the
web server
Determine definitively where crash occurred
Looking from
the outside
(similar issues for hang)
What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Umm, what does that stuff mean? (cont.)

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
Perplexing (?) problem: Show that output to an httpd
developer and they can quickly determine the important
Introduction parts (i.e., pick the interesting thread)
What kinds of
issues
or determine that there’s nothing interesting, which can be
encountered just as important
Using tools to
look inside the Users typically report the least interesting thread from the
web server
core dump, which wastes their time and ours.
Looking from
the outside Some sort of automatic annotation/explanation would be
What if you useful.
build the code
differently Descriptions of normal activity
Compare with Bug numbers for backtraces that match known problems
httpd 2.2 et cetera
References
and further
reading
 Demo

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues Jeff, this is where you go to
encountered
http://emptyhammock.com/projects/httpd/explore/.
Using tools to
look inside the Try loading PR53870.pstackout and
web server
ubuntu64.core.collect.gdbout.
Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 What if you build the code differently

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of Improving general debuggability of the generated code by

issues
encountered
affecting code generation or symbols
Using tools to Enabling optional run-time checks
look inside the
web server
Enabling third-party exception hooks
Looking from
the outside Enabling third-party tracing of API hooks
What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Different code generation for debugging

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
Adding symbols, not stripping executable
Introduction

What kinds of Disabling in-lining of functions for better diagnosablity

issues
encountered Disabling other optimization so that more variables can be
Using tools to checked
look inside the
web server
Options like -funwind-table for tools like
Looking from
the outside mod backtrace to work on ARM
What if you
build the code (huge YMMV, with architecture, OS, compiler, and
differently
compiler/linker flags as variables)
Compare with
httpd 2.2

References
and further
reading
 General debug capabilities not built in by default

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered
Hook tracing
Using tools to DTrace probes in the server (DTrace provider ap)
look inside the
web server Exception hooks
Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Hook tracing

Debugging
Tricks with
Apache HTTP httpd hooks are what allow different modules to handle or
Server 2.4
otherwise affect processing of the different phases of
Jeff Trawick
execution.
Introduction
A module that needs to take part in a particular aspect of
What kinds of
issues connection or request processing uses a special hook
encountered
macro to save a callback pointer.
Using tools to
look inside the
web server
At the point where httpd core passes control to modules,
Looking from it invokes a special hook macro to continue calling module
the outside
callbacks until a failure occurs, a module elects to handle
What if you
build the code the request, or all callbacks have been serviced (depending
differently
on the hook).
Compare with
httpd 2.2 By tracing what happens inside the hook invocation, some
References
and further
types of failures can be quickly tracked to a particular
reading module.
 Hook tracing (cont.)

Debugging
Tricks with
Apache HTTP
Server 2.4 httpd now provides a way for third-party code to run
Jeff Trawick during the hook macros at the following points:
Introduction
Start of the hook execution
About to call a particular module’s hook function
What kinds of
issues Returned from that module’s hook function
encountered
End of the hook execution
Using tools to
look inside the Code inserted into the calling of different modules’ handler
web server

Looking from
functions can determine what module’s handler took
the outside ownership of this phase of request processing and/or
What if you
build the code
caused the request to fail.
differently
More generally, if some mysterious error occurs at any
Compare with
httpd 2.2 phase of processing, such as the notorious 500 with no log
References message, hook tracing could pinpoint the module.
and further
reading
 Enabling hook tracing

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
Configure argument --enable-hook-probes causes
Introduction ap_hook_probes.h to be included in files with hook
What kinds of
issues
definitions, making special macros active.
encountered
ap_hook_probes.h isn’t part of httpd, so it needs to be
Using tools to
look inside the copied into include or located via CPPFLAGS.
web server

Looking from Any code invoked by the macros in ap_hook_probes.h

the outside
has to be compiled into the server, so this can be handled
What if you
build the code by statically linking a module into the server if the desired
differently
logic can’t be implemented completely in a macro.
Compare with
httpd 2.2

References
and further
reading
 Enabling hook tracing (cont.)

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction Build mechanism for including this code

What kinds of
issues export CPPFLAGS=-I/path/to/module
encountered ./configure --enable-hook-probes \
Using tools to --with-module=debugging:/path/to/module/mod_foo.c \
look inside the
web server --other-args
Looking from
the outside After httpd is built, httpd -l will show mod_foo.c as
What if you built-in (like core.c and a few others).
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 mod hook ar — Experimental hook tracer

Debugging
Tricks with
Apache HTTP
Must be built into the server as with other hook trace
Server 2.4 code.
Jeff Trawick
Sets a request note to information about the active
Introduction module while a hook is active.
What kinds of
issues Sets a request note to information about the failing
encountered
module if a hook returns an error.
Using tools to
look inside the
web server
Logging the RequestFailer note in the access log:
Looking from 127.0.0.1 ..."GET /cgi-bin/printenva" \
the outside
404 215 mod_cgid.c/404/handler
What if you
build the code Can log the name of the ActiveModule note in the case
differently
of a crash:
Compare with
httpd 2.2 ... [pid 30568:tid 140369329334016] Crash state: \
References mod_crash.c/handler
and further
reading
Download from http://emptyhammock.com/downloads/
 Possible directions with hook tracers

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
How much performance degradation?
Introduction

What kinds of Can this be used to implement DTrace probes?

issues
encountered Can a built-in module provide a simple API for loadable
Using tools to hook debug modules?
look inside the
web server
Will someone write a script to help with generating the
Looking from
the outside right set of macros based on the hooks that need to be
What if you instrumented?
build the code
differently
(if indeed this is interesting to anyone)
Compare with
httpd 2.2

References
and further
reading
 DTrace probes

Debugging
Tricks with
Apache HTTP
Server 2.4 httpd-specific probes enabled via --enable-dtrace was
Jeff Trawick the goal for 2.4, but only part of the code was committed,
Introduction and it hasn’t been kept up to date with new hooks.
What kinds of
issues
Someone needs to take interest in getting it working on
encountered one of the several platforms with DTrace.
Using tools to
look inside the Existing DTrace providers can certainly help understand
web server
httpd processing.
Looking from
the outside The pid provider provides great info but it is problematic
What if you
build the code
with httpd because you have to specify a particular
differently
process id.
Compare with
httpd 2.2

References
and further Has anyone tried to use mod dtrace with 2.4?
reading
 Exception hooks

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick
sig_coredump() is the handler for fatal signals with
Introduction httpd on Unix since the httpd 1.3 days.
What kinds of
issues It changes to the configured core dump directory and
encountered
re-throws the signal, causing the process to exit; at this
Using tools to
look inside the point the system (possibly) creates a core file.
web server

Looking from
If the --enable-exception-hook configure option was
the outside specified, sig_coredump() will also call exception hooks.
What if you
build the code This allows third-party modules to clean up some resource
differently
or save diagnostic information in the event of a crash.
Compare with
httpd 2.2

References
and further
reading
 Example exception hook module —
mod whatkilledus
Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction
Like mod log forensic, this module saves information about
What kinds of
issues the client request in an early request processing hook.
encountered

Using tools to
Unlike mod log forensic, the info is kept in memory during
look inside the
web server
the life of the request, and only logged if a crash occurs.
Looking from Also, if mod backtrace is loaded it will capture a
the outside
backtrace for the crashing thread.
What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 mod whatkilledus report

Debugging
Tricks with **** Crash at 2012-09-06 14:48:23
Apache HTTP
Server 2.4 Process id: 23368
Jeff Trawick
Fatal signal: 11

Introduction ...
What kinds of
/home/trawick/inst/24-64/bin/httpd:ap_run_fatal_exception+0x5b 0x4305
issues ...
encountered
/home/trawick/inst/24-64/modules/mod_crash.so:0x7fecbd59e986
Using tools to /home/trawick/inst/24-64/modules/mod_crash.so:0x7fecbd59ead8
look inside the
web server /home/trawick/inst/24-64/bin/httpd:ap_run_handler+0x5b 0x45008e
/home/trawick/inst/24-64/bin/httpd:ap_invoke_handler+0x173 0x450966
Looking from
the outside /home/trawick/inst/24-64/bin/httpd:ap_process_async_request+0x264 0x4
/home/trawick/inst/24-64/bin/httpd:0x468dc4
What if you
build the code /home/trawick/inst/24-64/bin/httpd:0x468fb3
differently /home/trawick/inst/24-64/bin/httpd:ap_run_process_connection+0x5b 0x4
Compare with ...
httpd 2.2

References
and further
reading
 mod whatkilledus report (cont.)

Debugging
Tricks with Request line (parsed):
Apache HTTP
Server 2.4 GET :10080 /crash/
Jeff Trawick
Request headers:
Host:127.0.0.1%3a10080
Introduction User-Agent:ApacheBench/2.3
What kinds of
Accept:*/*
issues
encountered
Client connection:
Using tools to 127.0.0.1:44883->127.0.0.1:10080 (user agent at 127.0.0.1:44883)
look inside the
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 mod whatkilledus notes

Debugging
Tricks with
Apache HTTP
Server 2.4
mod whatkilledus and mod backtrace can actually work
Jeff Trawick
well on Windows, with great backtraces if the web server
Introduction
.pdb files are available. Uhhh, I don’t have mod backtrace
What kinds of
issues working for 64-bit httpd on Windows yet.
encountered
The original versions of mod whatkilledus and
Using tools to
look inside the mod backtrace worked somewhat differently:
web server
mod backtrace and mod whatkilledus acted independently.
Looking from
the outside Neither supported Windows, and mod backtrace supported
What if you
fewer Unix-y platforms.
build the code mod whatkilledus had no mechanism to filter out sensitive
differently
information.
Compare with
httpd 2.2
http://emptyhammock.com/projects/httpd/diag/
References
and further
reading
 Comparison with httpd 2.2 — error log

Debugging
Tricks with
Apache HTTP
Server 2.4
Error messages
Jeff Trawick No module id, pid, thread id, etc. unless the module
generating the message adds it explicitly.
Introduction
No control over the format.
What kinds of
issues
No sub-second timestamps.
encountered No traceXXX levels
Using tools to Some messages just aren’t present, because even LogLevel
look inside the
web server debug would be too noisy, or separate log files are used
Looking from (mod rewrite) which have to be managed independently.
the outside
No per-module LogLevel, no per-dir LogLevel (which is
What if you
build the code
what allows per-client LogLevel)
differently Custom scripting can be used to reduce the output to
Compare with something readable, though nothing can be done about
httpd 2.2
the volume, and that may necessitate a different scheme
References
and further for rotating logs during problem determination.
reading
 Comparison with httpd 2.2 — other logs

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction

What kinds of
issues
encountered

Using tools to
mod log debug isn’t available.
look inside the
web server

Looking from
the outside

What if you
build the code
differently

Compare with
httpd 2.2

References
and further
reading
 Recap of Jeff’s toys

Debugging
Tricks with
Apache HTTP
Server 2.4
Explore, collect.py
Jeff Trawick
mod backtrace and mod whatkilledus
Introduction
mod hook ar
What kinds of
issues
encountered
pgfiles.py (not mentioned; shows open files for a process
Using tools to group, organized to show which files are shared by
look inside the
web server different processes)
Looking from nots.pl, nomodlevel.pl, etc.
the outside

What if you Available from

build the code
differently
http://emptyhammock.com/projects/ and/or
Compare with
httpd 2.2 http://emptyhammock.com/downloads/ (or ask Jeff
References directly for nots.pl et al)
and further
reading
 httpd materials

Debugging
Tricks with
Apache HTTP
Server 2.4

Jeff Trawick

Introduction
httpd debugging guide in the reference manual
What kinds of
http://httpd.apache.org/dev/debugging.html
issues
encountered module debugging guide from Cliff Wooley
Using tools to http://www.cs.virginia.edu/˜jcw5q/talks/apache/
look inside the
web server apache2moddebugging.ppt
Looking from
the outside
httpd debugging guide from Prefetch Technologies
What if you
http://prefetch.net/articles/debuggingapache.html
build the code
differently In PDF, click on the title or cut and paste the URL.
Compare with
httpd 2.2

References
and further
reading
 More general information (all from Joyent?)

Debugging
Tricks with
Apache HTTP
Server 2.4 The DTrace Book
Jeff Trawick
(http://www.dtracebook.com/index.php/Main Page)
Introduction
DTrace one-liners from Brendan Gregg
What kinds of
issues (http://www.brendangregg.com/DTrace/
encountered
dtrace oneliners.txt)
Using tools to
look inside the “And It All Went Horribly Wrong...” talk from Bryan
web server

Looking from
Cantrill
the outside (http://www.joyent.com/content/06-developers/
What if you
build the code
01-resources/13-and-it-all-went-horribly-wrong-
differently debugging-production-systems/debugging-production-
Compare with
httpd 2.2
systems.pdf)
References
and further
In PDF, click on the title or cut and paste the URL.
reading