These instructions are correct during CS307, spring 2002, at Carleton, and may (or may not) be more generally applicable.
Adding a system call to Linux involves changing a couple of tables in the kernel, changing a couple Makefiles, and adding the source code for the new system call in a sensible location in the Linux source tree.
All of the path names in the following instructions are either full path names (starting with a /), or relative to the top of the linux source tree, which is in /usr/src/linux-2.4.
Never ever ever run "make clean" unless you are completely desperate. A complete recompile of Linux can take up to an hour, and "make clean" forces a complete recompile.
Log in to the root account using the password given in class. Alternatively, you can log in as cs307 with the same password, and then execute /bin/su and give the same password.
Make a directory /root/yourusername. Before you modify any Linux source file, first copy it into this directory so you can restore the original sources when you're done using the machine. Remember that you're sharing these machines with other groups.
Copy arch/i386/kernel/entry.S to /root/yourusername.
Copy include/linux/unistd.h to /root/yourusername.
Modify the original entry.S, adding a line of the form ".long SYMBOL_NAME(sys_yourusername_hello)" to the bottom of the system call table.
Add the constant __NR_yourusername_hello to the list of constants in include/asm/unistd.h.
Copy your modified version of unistd.h into /usr/include/asm/.
If there isn't one already, make a directory /usr/src/linux-2.4/cs307/.
Copy mm/Makefile into cs307/. Edit cs307/Makefile to set O_TARGET to cs307.o, export-objs to yourusername_hello.o, and obj-y to yourusername_hello.o. Get rid of obj-$(CONFIG_HIGHMEM), but keep the include line.
Check /usr/src/linux/Makefile to make sure that it contains the lines
CORE_FILES =cs307/cs307.o kernel/kernel.o mm/mm.o fs/fs.o ipc/ipc.o
SUBDIRS =cs307 kernel drivers mm fs net ipc lib
Note that you may modify the original without making a backup copy. For the remainder of the term, we want the compilation process to compile the sources in the cs307 directory and include the results in the compiled version of the kernel.
asmlinkage int sys_yourusername_hello()
{
}
The name "sys_yourusername_hello" must be the same as in entry.S. See jondich_hello.c for an example.
Change directories to /usr/src/linux-2.4/. If you have added new files or changed the #include statements in existing source files, run "make dep", which updates the dependencies between the source files. This takes about 2 minutes.
Run "make bzImage", which should take no more than a couple minutes, unless you or some evil other person have wilfully deleted lots of object files or run "make clean".
Assuming your code compiled successfully, copy /usr/src/linux/arch/i386/boot/bzImage to /boot/yourusername.
Edit /etc/lilo.conf, adding the following lines:
image=/boot/yourusername label=yourusername read-only root=/dev/hda5
Execute the command /sbin/lilo.
Restart the machine with the "reboot" command. At the Red Hat prompt screen, use the up/down arrow keys to select your kernel.
Log in as root and change directories to /root/yourusername/. Create a C program to test your system call. Your test code will need to create the wrapper function by invoking one of the _syscalln macros (depending on the number of parameters your system call takes). Take a look at hellotest.c for a zero-parameter example. Compile and run your test program to see whether your system call works.
Note that printk output does not go to stdout or stderr. It goes, rather, to the system console and to the end of the file /var/log/messages. After you test your hello world system call, do "tail /var/log/messages" to see whether your hello succeeded.
When you are done with your session, restore all original Linux sources and header files.