Debugging with GDB

Sakeeb Sabakka
 Agenda
 Introduction to GDB
 GDB commands
 Malloc debugging
 Introduction to disassembled code
 Other debugger tools
 Remote debugging
 What is GDB
 The Gnu DeBugger
 As gdb(1) man page says: gdb
“allow you to see what is going on
inside another program while it
executes—or what another program
was doing at the moment it crashed.
 Getting Started…

 Compile the source code for debugging with -g

option.

override CFLAGS +=-g

 Getting Started…

`gdb' to start GDB.
 quit or Ctrl-d to exit.

 The most usual way to start GDB is with one argument, specifying
an executable program:
# gdb a.out

 You can also start with both an executable program and a core file
specified:
# gdb a.out core

 You can, specify a process ID if you want to debug a running

process:
# gdb –p 4321
 
 Getting Started…
 Command Completion
(gdb) info bre < TAB >
(gdb) info breakpoints

gdb) b make_ TAB

make_a_section_from_file make_environ make_abs_section
make_function_type make_blockvector make_pointer_type
make_cleanup make_reference_type make_command
make_symbol_completion_list
(gdb) b make_
 Getting Started…
 (gdb) b 'bubble( M-?
bubble(double,double)
bubble(int,int)

(gdb) b 'bubble(

Type in quotes and press TAB or type M-?

Useful in function overloading scenarios

(gdb) shell [shell command]

execute the specified command in shell
 Getting Started…
 (gdb) help
 (gdb) help command
 (gdb) aprorpos < regex >
• Search for commands matching the given regex.

 (gdb) info
 (gdb) help info
 (gdb) info set
 (gdb) help show
 Getting Started…
 (gdb) run [ args ]
• (gdb) file /home/sakeeb/bin/gdb_ex1
• (gdb) run 87 178
 (gbd) set args [ args ]
• (gdb) file /home/sakeeb/bin/calculate_bms
• (gdb) set args 87 178
• (gdb) run
 (gdb) kill
stop the execution of the binary being debugged.
 Breakpoints…
 A breakpoint makes your program stop whenever a
certain point in the program is reached. For each
breakpoint, you can add conditions to control in finer detail
whether your program stops.
 Breakpoints are set using break command
(gdb) break location
The breakpoint location can be any of the following
linenum
filename:linenum
function
filename:function
 Breakpoints…
 Conditional Breakpoints
(gdb) break if (expression)
(gdb) condition bN (expression)
(gdb) until [location] – goto location or skip the loop
 Setting break points for a single time
(gdb) tbreak args
 Using regular expression to set breakpoints
(gdb) rbreak regex
e.g (gdb) rbreak link*
(gdb) rbreak
 Specify the commands to be executed at a breakpoint
(gdb) commands [ bN ]
 Breakpoints…
(gbd) break traverse_tree
(gdb) commands
> if(node == 0x0)
> print head
> else
> print *head
> end
> continue
> end

(gdb) condition 1 (node == 0x0)

(gdb) break random_tree if (branch_count <= 2)

 Breakpoints…
 continue [COUNT]
Continue execution, ignore the current break point
for COUNT-1 times.
 ignore <BP# [COUNT]>
Set ignore-count of breakpoint
 save-breakpoints
Save current breakpoint definitions as a script
 info breakpoints [N]
Display information about current breakpoint[s].
 Stepping…
 Single stepping through the function is done
through step or stepi command. Step steps into
the functions calls
 Stepping over the function call is done using next
or nexti command.
• stepi and nexti do stepping at assembly instruction
where step and next do stepping at source code level.

 The continue statement continues the execution

of the program till it reaches next break point or
program termination
 Function Manipulation…
 Returning from a function without executing the the
remaining code
(gdb) return
 Get the control after executing the function
(gdb) finish
 The value of the variable can be changed in the frame
where it is defined.
Set the value of i to 20 in the current frame/function
(gdb) print i=20
or
(gdb) set i=20
 Invoking a function from gdb.
(gdb) call <function name>
 Watchpoints……
 Watchpoint stop the program execution whenever the value of
expression changes

(gdb) watchpoint temp->next

(gdb) watchpoint *(int*)0x56ad52
(gdb) watchpoint ‘a+b / c+d’

 Read watch point

(gdb ) rwatch expr [thread threadnum]

 Read / Write watchpoint

(gdb) awatch expr [thread threadnum]

Information on watchpoints
 (gdb)info watchpoints
 Catchpoints….
 Catchpoints causes gdb to stop for certain kinds
of events.
(gdb) catch event
“event” can be
exec call to exec
fork | vfork call to fork or vfork
load libname dynamic loading of any shared library
unload libname dynamic unloading
thread_start | thread_exit | thread join
before create, before exit or after join of threads.
start | exit after create or before exit of proceses
throw throwing of c++ exception
catch catching of c++ exception
 Core files…
 Start gdb with executable and core file
$ gdb a.out core ( or gdb a.out –c core)
or
(gdb) core core
 Stopping execution just before crash happens.
$ gdb –crashdebug a.out
$ gdb –crashdebug –pid <pid>
 Generate the dump of the process
(gbd) dumpcore
use –lcore or set live-core=1 for using the gdb generated corefile
$ gdb –lcore a.out core
Or
(gdb) set live-core 1
(gdb) core core
 Core files…
 To set the core file size
# ulimit –c
# ulimit –c <size>

 To dump share memory of the process into the core dump.

$ adb -w -k /stand/vmunix /dev/mem
(adb) core_addshmem_read/W 1

 To append pid of the process to core file while application

dumps core
$ adb -w -k /stand/vmunix /dev/mem
(adb) core_addpid/W 1
 Stack Frames...
 When program stops GDB helps you in finding all the information
stored in the stack frame ( frame pointer register )
 Frame contains the arguments given to the function ,
functions local variables and the address at which the
function is executing
 GDB by default is in the currently executing frame (frame 0)
and displays information pertaining to that frame.
 Back trace – summary of how your program got to place where it
is.
 Function Arguments & Environment Variables

Stack
Collection of stack frames. Grows Downward.

Unused Memory

Heap
Dynamic memory for programs and libraries.
Grows upward.
Un-initialized Data Segment (BSS)
Executable stores only size required for this area, memory will
allocated and initialized to zero on process instantiation.
Initialized Data Segment
Initialized global variables from executable

Text Segment
Machine Code. Shared between all the processes executing the
same binary. Read-Only.
 Stack Frames...
 (gdb) backtrace / bt / where - Prints the entire stack
(gdb) bt n - prints the innermost n frames
(gdb) bt –n –Prints the outermost n frames
(gdb) bt full –Prints the information of local variables

 Selecting a frame
(gdb) frame <num>
(gdb) frame addr
(gdb) up n – towards outermost frame
(gdb) down n – towards innermost frame
 Stack Frames...
 Information about frames
(gdb) info frame / f
Print -
address of the current frame
address of previous and next frames
address of local variables and parameters
program counter
registers saved in the frame
(gdb) info args – Prints the argument of the selected frame
(gdb) info locals – Prints the local variables
(gdb) info scope <function> - info about locals in the function
(gdb) info catch - Prints all exception handlers active
 Threads…
 List of threads
(gdb) info thread
(gdb) thread <thread-id>
(gdb) thread apply <thread-id | all > command
Apply the command on the specified thread.

To view the backtrace of all the threads use the following

(gdb) thread apply all backtrace

(gbd) backtrace-other-thread
 Examining the Data...
 Display the value of an expression
(gdb) print[/FMT] (expression)
(gdb) print i
(gdb) print/s string
 (gdb) output (expression)
same as print, but don't put in value history and don't print
newline.
 Examine the value of memory
(gdb) x /FMT <address>
(gdb) x/s string
 Display the value of expression each time program stops
(gdb) display/FMT (expression)
(gdb) display i
 (gbd) help set radix
 Examining the Data...
Print format modifiers
 x Print in hexadecimal
 d Print as a signed decimal
 u Print as an unsigned decimal
 o Print as octal
 t Print in binary (t is for two)
 a Print as an address
 c Print as character
(gbd) x/FMT (expression)
(gdb) x/d &i
(gdb) x/s s
(gdb) x/c s
(gdb) x/4c s
(gdb) x/t s
(gdb) x/3x s
 Examining the Source…
 Display source code
(gdb) list LINENUM | FILE:LINENUM | FUNCTION |
FILE:FUNCTION | *ADDRESS ]
(gdb) list 5,
(gbd) list ,28
(gbd) list 21, 38
(gdb) list parse_args
(gbd) list main.c:doit

(gdb) set listsize 24

 Searching the source code

(gbd) search regex
(gdb) reverse-serach regex
 Examining the Source…
 (gdb) whatis insert::node
Display the type of the expression.
 (gdb) ptype insert::node
Display the description of the type of the expression.
Note: - insert::node refers to node variable in function insert

 `@' is a binary operator for treating parts of memory as arrays

eg:- int *array = (int *) malloc (len * sizeof (int))
(gdb) print *array@len
 {type}addr is used for display the value at addr as type
 (type)addr is used for display the value as type
(gdb) p/x (short[2])0x12345678
 Signals…
 GDB has ability to detect any occurrence of signal in execution
(gdb) info signals
(gdb) info handle
(gdb) info signal [Signal num]
Print table of how gdb handles signals
 Change the GDB behavior for signal handling
(gdb) handle signal [keywords]
signal can be signal name or Num.
Keywords can be
nostop – Not to stop Program execution
stop- Stop Program execution
print / noprint – should / should not print the signal info
pass / noignore – allow program to see the signal
nopass / ignore – should not allow program to see the signal
 Logging...
 Logging Output

 set logging on
• Enable logging.
 set logging off
• Disable logging.
 set logging file file
• Change the name of the current logfile. The default logfile is
`gdb.txt'.
 set logging overwrite [on|off]
• By default, GDB will append to the logfile. Set overwrite if you
want set logging on to overwrite the logfile instead.
 set logging redirect [on|off]
• By default, GDB output will go to both the terminal and the
logfile. Set redirect if you want output to go only to the log file.
 show logging
• Show the current values of the logging settings.
 The .gdbinit File…
 GDB will look for this file in two places
 First, In your home directory
 Second, In the current directory
 comments are started with "#” and blank lines are ignored.
 Put redundant commands into .gdbinit
Eg:- cat .gdbinit
set listsize 24
set break main
set sourcedir /home/sakeeb/imported_sources
set objectdir /home/sakeeb/impored_objects
set logging file /tmp/sakeeb.gdb.log
set loggin on
 Automate most of debugging tasks
 Use .gdbinit to automate debugging
 gdb commands stored in alternate files can be executed using –x option.
 Make use of –batch option to run gdb in batch mode
eg: - gdb –batch –x /home/sakeeb/audfilter/crash_debug_commands
 The child process…
(gdb) set follow-fork-mode [ parent | child | ask | serial ]
The unfollowed process will continue to run.
By default, the debugger will follow the parent process.

# inetd –k
# chatr +dbg enable /usr/sbin/inetd
# gdb /usr/sbin/inetd
(gdb) set follow-fork-mode child
(gdb) break execve
(gdb) continue
Invoke the service to be traced (eg;- telnet to the m/c to invoke and debug telnetd)
(gdb) step
!!! we are insided the spawned process..
 Convenient Breakpoints…

(gdb) rbp
Set breakpoints and actions for all functions.
(gdb) rdp
Delete breakpoints set by rbp.
(gdb) xbp
Set breakpoint for all function return point.
(gdb) xdp
Delete breakpoints set by xbp.
(gdb) xbreak <funcion>
Set breakpoint at the function exit.
 Debugging memory…
(gdb) set heap-check [option] <on | off>
option is one of :
leaks – leak detection, defaut with heap-check.
bounds – bounds check, defaut with heap-check.
free - validation of free, defaut with heap-check.
string – validation of arguments to str* functions.
scramble – scramble the deallocated blocks with
value 0xfeedface

(gdb) xbp
(gdb) set heap-check on
(gdb) run
<-- at break-point at function return -->
(gdb) info leaks
 Debugging memory…

(gdb) set heap-check watch <address>

watch allocation/deallocation of address
(gdb) set heap-check null-check-size <num>
malloc returns null after allocation of num bytes
(gdb) set heap-check null-check <n | random>
instruct malloc to return null after nth invocation
(gdb) set heap-check block-size <num-bytes>
Break when allaction size is > num-bytes

(gdb) catch nomem

catch null pointer returned by malloc.
 Debugging memory…

(gdb) info corruption

Checks for corruption in the currently allocated heap.

(gdb) info heap <process | arenas>

report high level memory usage of a process/arena.

(gdb) help info heap

(gdb) info heap

(gdb) help set heap-check

 Debugging memory…
 Use +check=[option] compiler options (IA compiler) to
enable runtime checks
Options are:
• all | none
• bounds – check out of boundary array reference.
• malloc – memory leak and heap corruption
• stack – writing outside the stack frame
• uninit – reading un-initialized variables.

 A failed check results in the program abort at runtime with

stack trace on stderr.

 RTC_NO_ABORT env variable must be set to 1 to continue

the program execution after a failed runtime check.
 Low level debugging…

(gdb) info shared

details about shared library mapping into memory.
(gdb) info registers
details about registry contents.
(gdb) disassemble [address | function]
disassemble the program to assembly instructions.
 Low level debugging…
 Procedure calling in PA32 binary
• The first 4 arguments are passed in registers
(r26==arg0, r25==arg1, r24==arg2, r23==arg3)
• Arg4 and above are passed on the stack.
• Minimum stack frame size if 0x40. This includes space
for up to 4 optional arguments.
• rp (return pointer) register is the only thing most
functions save onto the stack.
• Leaf functions ( function which does not call other
functions) doesn’t allocate stack frame.
• Structures are passed as pointer to its base address.
 Low level debugging…
(gdb) disass foo
Dump of assembler code for function foo:;;;
File: args.c;;;  
17 {
<-- saves return pointer
0x29ec <foo>:  stw %rp,-0x14(%sp)
<-- creates stack frame (0x40 + space for locals)
0x29f0 <foo+0x4>:      ldo 0x140(%sp),%sp
<-- stores the argument to stack ( at 0x24 + frame size)
0x29f4 <foo+0x8>:      stw %r26,-0x164(%sp)
0x29f8 <foo+0xc>:      stw %r25,-0x68(%sp)
0x29fc <foo+0x10>:     stw %r24,-0x6c(%sp)
0x2a00 <foo+0x14>:     stw %r23,-0x70(%sp)
 Low level debugging…
PA32 binary function call stack

sp Current stack pointer

sp-0x4 Previous stack pointer
sp-0x14 RP (return pointer)
sp-0x18 External Stub RP (RP’)
sp-0x1C External LPT’ (SR4/Code)
sp-0x1C External Data LPT
sp-0x24 to arg0 to arg3
sp-0x30
sp- Nth arg (for N>=4) optional, allocated only if
(4*(N+9)) function takes more than 4 arguments.
 Low level debugging…
 Procedure calling in IPF 32 binary
• The first 8 arguments are passed through register stack.
• Arg8 and above are passed on the stack.
• Unlike PA, arguments are not stored back to stack.
• Leaf functions doesn’t allocate stack frame.
• The stack on Itanium systems grows the opposite
direction as compared to PA-RISC.
 Low level debugging…
(gdb) disass foo
<-- saves previous function state to r50.
<-- allocates 20 local registers (including its arguments)
<-- allocates 5 output registers.
0x4000a00:0 <foo>:       alloc            r50=ar.pfs,0,20,5,0
<-- saves return pointer to to r51.
0x4000a00:1 <foo+0x1>:  mov              r51=b0
<-- increment stack frame by size 64.
0x4000a00:2 <foo+0x2>:  adds             r12=-64,r12;;
<-- stores global pointer (r1) to r36.
0x4000a00:2 <foo+0x3>: addl r36=0,r1
<-- set return value (r8) to 0.
0x4000a00:42 <foo+0x42>: mov r8=0
 Low level debugging…
How functions in a shared library are invoked.
1. Program calls import stub.
2. Search PLT for the function. If not found, invoke dynamic
loader and update PLT.
3. The export stub is called.
4. The export stub calls the real function
5. The real function returns to the export stub
6. The export stub returns back to the original caller.
 Other Tools...
Change programs internal attributes
chatr +dbg enable|disable <executable_name>
controls the mapping of shared libraries mapping
privately.
and thus enable break point setting and debugging.

chatr +s enable|disable <executable_name>

controls whether the SHLIB path can be used to locate
shared libraries.

chatr –z <executable name>

Enable run time dereferencing of null pointer to produce
SIGSEGV

chatr <executable_name> : shows the attributes.

 Other Tools…
sh# adb <exe> <core>
adb > c
prints the backtrace
adb > m
initial and default maps for a valid core file with an
indication of which is currently active

Using crashinfo to display stack trace

sh# crashinfo -t -l pid=<PID>
 Other Tools…
P4/Q4/KWDB

p4> p s curr_file
p4> p int audit_state_flag
p4> trace pid 1
P4> p proc_t 0xe0000001806ba000 | grep
p_auditperproc
p4> p s $(( 0xe0000001_806bb1a0 + 32 ))
p4> p int cmpt_enabled
 Other Tools…
nm – lists the symbols in an object file.
• $ nm –A /usr/lib/hpux32/lib*.so | grep milli
• Symbols categorized: extern, global or undefined.
• Symbols origin: data or text (code).
• If the object file is stripped, nm can’t extract symbols.
 Use odump or elfdump in those cases

$ odump -slexport /usr/local/lib/libintl.a | grep “__umodsi3”

$ elfdump -n .dynsym -s /usr/local/lib/hpux32/libintl.a |

grep “__umodsi3”
 Other Tools…

lsof – lists open files and sockets

• $ lsof –l /usr/sbin/audomon

fuser - find processes which opened a given file

• fuser /var/spool/audfilterd/audfilterd.lock
 Other Tools…
tusc – system call tracer
• $ tusc –s bind 2134
• $ tusc –E /usr/bin/ioscan –funC disk
• $ tusc –a –p –f –n –k <deamon-pid> -o
/tmp/tusc.log

ltrace – libc tracer.

• Currently only for 32-bit IPF binaries.
• libc call tracing similar to tusc.
• Available with the linker patch :
 11.23: PHSS_38134
 11.31: PHSS_38135
 Remote Debugging...
 Normally used when gdb cannot be run on target process
or machine . For example
• Debugging Kernel boot process (KWDB)
• Embedded systems
 Target process is run/attached using gdbserver program
• Not available on HP-UX.
• For running kernel for remote debugging, specify at the ISL
prompt.
ISL> hpux <vmunix> kwdb_flags=lan[.<driver>[.nowait]]
ISL> boot <vmunix> kwdb_flags=lan kwdb_hw_path=1/1/1/1
HPUX> boot <vmunix> kwdb_flags=sub[.nowait]
HPUX> boot <vmunix> kwdb_flags=udp
kwdb_hw_path=1/1/1/1 kwdb_udp_ip_addr=15.76.98.42
 Remote Debugging…
 Use target command to connect to remote gdb process
 Tracepoints helps you to trace the program without
stopping it.
• trace command sets the tracepoint.
• actions + collect command sequence collects the data.
(gdb) trace tree.c:74
(gdb) actions
> collect *tree
> collect $locals
> while-stepping 3
> collect bst
>end
>end
• while-stepping command specifies the single stepping
behavior.
 Remote Debugging...
 Collect command collects data at tracepoints.
It can collect details like
• Registers – via $regs
• Function Args – via $args
• Local Variables – via $vars
• Individual variables – via variable name
(gdb) collect $locals
(gdb) collect max_index
 Passcount command can be used to set the count of data
collection
(gdb) passcount 8 2
Stop execution at 8th involcation of tracepoint 2
(gdb) passcount 2
Stop execution at 2nd involcation of most recent tracepoint.
 Remote Debugging…
 A trace event is generated whenever a tracepoint is
executed.

 Every trace event generates a data record called a trace

frame in a buffer called the trace buffer.

 If a tracepoint's actions include single-stepping, each

single-step also generates a trace event and a trace frame.

 Each trace frame has a sequential identifying number

starting from zero, plus the address of the event.
 Remote Debugging…
Following convenience variables can be used to collect data at
the tracepoints.
 $trace_frame - the ID of the currently selected frame or
-1 if no trace frame is selected.
 $tracepoint - the ID of the tracepoint corresponding to
$trace_frame
 $trace_line - the source line number corresponding to
$tracepoint
 $trace_file - the source filename corresponding to
$tracepoint
 $trace_func - the name of the function containing
$tracepoint
 Remote Debugging…
Tracepoint variables example:

(gdb) tfind start

(gdb) while ($trace_frame != -1)
> printf "Tracepoint %d, ", $tracepoint> printf "line %d, ", $trace_line
> if ($tracepoint == 1)
> printf ”node value = %d\n", data
> end
> if ($tracepoint == 2)
> printf ”DLT== %x, ", $r19
> printf ”bst node = 0x%x\n", *bst
> end
> tfind
> end
 Remote Debugging…
(gdb) tstart
Start trace data collection.
(gdb) tstop
Stop trace data collection.
(gdb) tstatus
Status of trace data collection.
(gdb) info tracepoints [number]
List of trace points.
(gdb) save-tracepoints <filename>
Save the current tracepoint definitions to a file.
 Remote Debugging…
(gdb) tdump
Print all of the data collected at the current trace frame.
(gdb) tfind [< event id > | start | none | -]
Select frame by id number.
(gdb) tfind tracepoint [< tracepoint id >]
Select the next trace for the given tracepoint.
(gdb) tfind pc [< address >]
Select the next trace frame for the given code address.
(gdb) tfind line [[filename:] linenum]
Select the next trace frame the given source line.
 Remote Debugging…
Tracepoint Breakpoint
(gdb) tfind start (gdb) continue
Tracepoint #12 at tree.c line 34 Breakpoint #12 at tree.c line 34
(gdb) print head (gbd) print head
$1 = 0x12345678 $1 = 0x12345678
(gdb) tfind next (gdb) step
tree.c line 45 tree.c line 45
(gdb) tfind line tree.c:53 (gdb) until tree.c:53
Tracepoint #3 at tree.c:53 tree.c line 53
(gdb) print head->data (gdb) head->data
$2 = 28 $2 = 28
(gdb) print head->left (gdb) head->left
Data not collected. $3 = 0x0