Adding a system call to Linux

These instructions are valid for CS307 students during Carleton's fall term, 2005. We were using Red Hat Enterprise Linux 4 installed on VMware Workstation 5.0.0 (running in turn on Red Hat Enterprise Linux 4).

VMware

  1. Login to one of the four machines, using the account named after the machine (e.g. user backus on backus), using the password Mike gave us.
  2. Launch Applications->System Tools->VMware.
  3. Select the "Red Hat Enterprise Linux 4" virtual machine and then hit "Power On".
  4. Use the Snapshot Manager to select an appropriate snapshot for yourself. Don't delete the "Master Copy" snapshot, but do feel free to use it to create a new starting place for yourself.
  5. Login as root, using the same password as before.
  6. Open a Terminal in the VMware window, and off you go.

Linux source

The source tree is at /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9. Relative paths will refer to that base from here on.

Modify the system call table

  1. Edit arch/i386/kernel/entry.S. Add ".long sys_yourname" to the end of the table of sys-call function pointers.
  2. Edit include/asm/unistd.h. Add "#define __NR_yourname 285" to the bottom of the list of __NR constants. Change NR_syscalls to 286.
  3. Backup /usr/include/asm/unistd.h to /usr/include/asm/unistd.h.original, and then copy include/asm/unistd.h to /usr/include/asm.

Edit the Makefile structure

  1. Create a yourname/ directory.
  2. Edit Makefile (at the source root). The "core-y += kernel/..." line should become "core-y += yourname/ kernel/...".
  3. Copy mm/Makefile to yourname/.
  4. Edit yourname/Makefile to have only the line:
    obj-y := yourname.o
    plus an appropriate comment.

Add the source for your system call

  1. Create yourname/yourname.c with code for your system call. Note that the name of this function needs to be the same as the name ("sys_yourname", or whatever) that you used in the ".long" table in entry.S. Here's an example.

Compile and install the new kernel

  1. Make sure you're at the Linux source root, then execute "make". This should take four or five minutes. Since you put yourname/ at the beginning of the "core-y" line in the root Makefile, you should see evidence fairly quickly that your new code is compiling (or not).
  2. Everything okay? Does it say "bzImage is ready"? Then move to the next step, which is "make install".

Reboot

  1. You might find it helpful to edit /etc/grub.conf, and remove or comment out the "hiddenmenu" line and set timeout to 10 instead of 5. This will give you plenty of opportunity to choose which kernel you want to boot into (the original, labeled "2.6.9-11.EL", or the one you just compiled, labeled "2.6.9-11.prep").
  2. Run "reboot" in the VMware terminal window, and select "2.6.9-11.prep" when you get the chance.

Test your system call

  1. Login as root again, and create a working directory /root/yourname.
  2. In your working directory create a test file mytest.c or some such thing. Here's an example.
  3. Compile and run your test program. Note that the $PATH environment variable probably doesn't include ., so you'll need to execute "./mytest" or some such thing.
  4. If all goes well, you will see printf output, but not printk output. To see the printk output, do "tail /var/log/messages".

Wrap up

  1. You might want to save a snapshot at this point, or boot a floppy and save your sources.
  2. Questions? Let me know.