Goals

Your primary goal for this lab is to get familiar with using the GNU debugger (gdb) to explore and debug x86-64 programs, in which you may not be able to view the original C source. As a secondary goal, you’ll get some practice that may help with any sneaky escapades you may undertake.

Documentation and tutorials

You will find some potentially useful gdb resources posted on our Resources page. There is also a really handy summary sheet on page 280 of the textbook.

Getting started

In this lab, we’ll be working with a pre-compiled program named call_proc, which conveniently builds on the proc function (that really complicated one that takes eight parameters) that we’ve been discussing in class. Grab the program and what source code I’m giving you, and copy them to mantis.

Here are some steps to follow:

• SSH into mantis via VS Code and open a terminal in your cs208 folder.

• Make a new directory for this lab:

  mkdir lab6
  

• Change to your new directory:

  cd lab6
  

• Grab the compiled program and source code:

  wget https://cs.carleton.edu/faculty/tamert/courses/cs208-f23/resources/samples/call_proc.tar
  

• Un-tar it:

  tar xvf call_proc.tar
  

• Look at the source code in call_proc.c. You should see the familiar proc function, and some other stuff we’ll explore soon.

• You won’t be able to compile the program yourself, as you don’t have a definition for the function phase_0. You do have the executable, however. Try to run call_proc. Unless you get really lucky, you won’t win. :)

  ./call_proc
  

• Note: If you have any issues running call_proc, you can use the following command to make it executable:

  chmod +x call_proc
  

Step 1: Exploring proc with gdb

We’ll now explore the proc function we’ve been discussing in class.

• Open the file call_proc.c. You’ll find this function on lines 5–14.

• Open a Terminal window in VS Code.

• Run gdb, ready to step through call_proc:

  gdb call_proc
  

We want to explore the inputs to proc. We expect to find the first six arguments in registers, with arguments 7 and 8 (x4 and p4, respectively) instead on the stack. In this case, proc is called on line 23, as follows:

    proc(10, &a, 20, &b, 30, &c, 40, &d);

We drew the following picture in class, to represent the state of the stack at the start of a call to proc:

Let’s find the arguments!

• We’re using gdb, so set a breakpoint at the start of proc:

  (gdb) b proc
  

• Run the call_proc program:

  (gdb) r
  

• You’ll have to enter a number, and then you should hit the breakpoint. We expect the first argument in register %rdi; we should find the value 10 there:

  (gdb) print $rdi
  

(Note the very silly convention in gdb that you use $ for registers instead of %.)

• The next argument should be in register %rsi. This one is an address, and we don’t have a good sense of what the address is yet. We’ll explore that later, in Step 2:

  (gdb) print $rsi
  

• By default, this seems to display in decimal, which is kind of useless for addresses. Use /x to make it display in hex instead:

  (gdb) print /x $rsi
  

• Arguments 3–6 should be in registers %rdx, %rcx, %r8, and %r9, respectively. It’s tedious to view each individually, so let’s look at all registers instead:

  (gdb) i r
  

• You’ll likely notice that the contents of %rdx and %r8 don’t look like what we expect. Keep in mind that they’re in hex, not decimal! Conveniently, using i r in gdb displays both, one per column. Make sure you know which one you’re reading.

• Now let’s look at the stack to find our 7th and 8th arguments (when I ran this, %rsp contained the value 0x7fffffffe458):

  (gdb) x/3gx 0x7fffffffe458
  

The first 64 bits (a “giant” word in gdb, hence the g) contain the return address for when proc is done. The next 8 bytes are our 7th parameter (0x28 in hex is 2*16+8 = 40 in decimal), followed by an address that is our 8th parameter.

• If you type list, you’ll see the C code for the proc function:

  (gdb) list
  

• If instead you want to see the assembly, you can type layout asm. This gives us a split screen with the assembly code on the top and the (gdb) command prompt on the bottom. Note that it can be a bit erratic, so if it gets messed up visually, press the key combination Ctrl-L to redraw the screen.

  (gdb) layout asm
  

You should find very similar assembly to what we looked at in our classwork in Lesson 16. Read through it (it’s just seven lines) and make sure you can follow what it’s doing.

Next, we’ll look at a function that calls proc; this function is aptly named call_proc.

Step 2: Exploring call_proc with gdb

We’re going to step through call_proc to see how well it lines up with our understanding of its use of local variables and argument prep on the stack. We’re assuming the stack will look like this after the four local variables are given their values:

If you’re continuing from Step #1, type c to continue the program (this should complete its execution), then type d to delete all breakpoints. If you’re starting fresh, run gdb call_proc.

• Set a breakpoint in the function call_proc:

  (gdb) b call_proc
  

• Run the program, and enter a number:

  (gdb) r
  Guess a number: 123
  

• This should hit the breakpoint. We expect a to be stored in 24(%rsp). The first thing you should notice is that isn’t the case. Where is it storing $0x1 instead? Think about why this is okay (hint: look for later modifications to the stack before calling proc).

• Draw a picture for yourself that shows the state of the stack now that we’ve seen the actual assembly.

• Let’s move down to check the stack. Type ni for next instruction, and repeat it several times until you’re about to execute the first lea instruction.

• Inspect the stack. Let’s look at the 32 bytes starting at the top of the stack (for me, %rsp has value 0x7fffffffe7c0 as I run this):

  (gdb) x/32bx 0x7fffffffe7c0
  

You should see 0x01 followed by 7 0x00s. That’s a. On the line above, you’ll see each of b (4 bytes), c (2 bytes), and d (1 byte), all next to each other.

• Let’s keep going, and see the argument building to prep for calling proc. Notice the lea instructions; for example, address 0x10(%rsp) (that’s a) is put in %rsi, as &a is provided as our second argument on line 23 of the source code. Additionally, we see that %rax and $0x28 (that’s 40 in hex) are pushed to the stack; these are arguments 8 and 7, in that order.

• Finally, let’s print out the state of the stack right before the callq to jump to proc executes. Hit return several more times (and/or type ni again) to get there, then get the new value of the stack pointer:

  (gdb) print $rsp
  

• Finally, let’s look at the stack (my new value of %rsp is 0x7fffffffe7b0):

  (gdb) x/48bx 0x7fffffffe7b0
  

The stack should approximately match this state of the world:

(There are some extra push instructions up above, so we have some saved registers, too, but this should be close.)

Step 3: Solving a puzzle with gdb

Follow along with the video to see the steps you can take to work on your zoo escape.