Systems Programming in Unix/Linux

This chapter presents an introduction to the book. It describes the book’s scope, intended audience and its suitability as a textbook in Computer Science/Engineering curriculums. It presents a brief history of Unix, which includes early versions of Unix at Bell Labs, AT&T System V and other developments of Unix, such as BSD, HP UX, IBM AIX and Sun/Solaris Unix. It describes the development of Linux and various Linux distributions, which include Debian, Ubuntu, Mint, Red Hat and Slackware. It lists both the hardware platforms and virtual machines for Linux. It shows how to install Ubuntu Linux to both VirtualBox and Vmware virtual machines inside the Microsoft Windows. It explains the startup sequence of Linux, from booting the Linux kernel to user login and command execution. It describes the Unix/Linux file system organization, file types and commonly used Unix/Linux commands. Lastly, it describes system administration tasks for users to manage and maintain their Linux systems.

This chapter covers the background information needed for systems programming. It introduces several GUI based text editors, such as vim, gedit and EMACS, to allow readers to edit files. It shows how to use the EMACS editor in both command and GUI mode to edit, compile and execute C programs. It explains program development steps. These include the compile-link steps of GCC, static and dynamic linking, format and contents of binary executable files, program execution and termination. It explains function call conventions and run-time stack usage in detail. These include parameter passing, local variables and stack frames. It also shows how to link C programs with assembly code. It covers the GNU make facility and shows how to write Makefiles by examples. It shows how to use the GDB debugger to debug C programs. It points out the common errors in C programs and suggests ways to prevent such errors during program development. Then it covers advanced programming techniques. It describes structures and pointer in C. It covers link lists and list processing by detailed examples. It covers binary trees and tree traversal algorithms. The chapter cumulates with a programming project, which is for the reader to implement a binary tree to simulate operations in the Unix/Linux file system tree. The project starts with a single root directory node. It supports mkdir, rmdir, creat, rm, cd, pwd, ls operations, saving the file system tree as a file and restoring the file system tree from saved file. The project allows the reader to apply and practice the programming techniques of tokenizing strings, parsing user commands and using function pointers to invoke functions for command processing.

This chapter covers process management in Unix/Linux. It explains the principle of multitasking and introduces the process concept. It uses a programming example to illustrate the principles and techniques of multitasking, context switching and processes. The multitasking system supports dynamic process creation, process termination, process synchronization by sleep and wakeup, process relations and implementation of process family tree as a binary tree, allowing parent process to wait for child process termination. It provides a concrete example of how process management functions work in an operating system kernel. Then it explains the origin of processes in Unix/Linux, from the initial process during booting to INIT process, daemon processes, login processes and sh process for user command execution. Next, it explains the execution modes of processes, transitions from User mode to Kernel mode by interrupts, exceptions and system calls. Then it describes Unix/Linux system calls for process management, which include fork, wait, exec and exit. It explains the relationship between parent and child processes, including a detailed description between process termination and the wait operation by parent process. It explains how to handle orphan processes by the INIT process, including subreaper processes in current Linux, and it demonstrates subreaper process by example. Then it explains changing process execution image by exec in detail, which includes the execve system call, command-line parameters and environment variables. It explains the principles and techniques of I/O redirections, pipes and shows pipe programming by examples. The programming project of this chapter is for the reader to integrate the concepts and techniques of process management to implement a sh simulator for command execution. The sh simulator works exactly the same as the standard sh. It supports executions of simple commands, commands with I/O redirections and multiple commands connected by pipes.

This chapter covers concurrent programming. It introduces the concept of parallel computing and points out its importance. It compares sequential algorithms with parallel algorithms, and parallelism vs. concurrency. It explains the principles of threads and their advantages over processes. It covers threads operations in Pthreads by examples. These include threads management functions, threads synchronization tools of mutex, join, condition variables and barriers. It demonstrates concurrent programming using threads by detailed examples. These include matrix computation, quicksort and solving systems of linear equations by concurrent threads. It explains the deadlock problem and shows how to prevent deadlocks in concurrent programs. It covers semaphores and demonstrates their advantages over condition variables. It also explains the unique way of supporting threads in Linux. The programming project is to implement user-level threads. It presents a base system to help the reader get started. The base system supports dynamic creation, execution and termination of concurrent tasks, which are equivalent to threads executing in the same address space of a process. The project is for the reader to implement threads join, mutex and semaphores for threads synchronization and demonstrate their usage in concurrent programs. The programming project should allow the reader to have a deeper understanding of the principles and techniques of multitasking, threads synchronization and concurrent programming.

This chapter covers timers and timer services. It explains the principle of hardware timers and the hardware timers in Intel x86 based PCs. it covers CPU operations and interrupts processing. It describes timer related system calls, library functions and commands for timer services in Linux. It discusses process interval timers, timer generated signals and demonstrates process interval timers by examples. The programming project is to implement timer, timer interrupts and interval timers in a multitasking system. The multitasking system runs as a Linux process, which acts as a virtual CPU for concurrent tasks inside the Linux process. The real-time mode interval timer of the Linux process is programmed to generate SIGALRM signals periodically, which acts as timer interrupts to the virtual CPU, which uses a SIGALRM signal catcher as the timer interrupt handler. The project is for the reader to implement interval timers for tasks by a timer queue. It also lets the reader use Linux signal masks to implement critical regions to prevent race conditions between tasks and interrupt handlers.

This chapter covers signals and signal processing. It presents a unified treatment of signals and interrupts, which helps put signals into a proper perspective. It treats signals as interrupts to processes, which diverts a process from its normal executions to do signal processing. It explains sources of signals, which include signals from hardware, exceptions and other processes. Then it uses examples to illustrate common usage of signals in Unix/Linux. It explains signal processing in Unix/Linux in detail. These include signal types, signal vector bits, signal mask bits, signal handlers in the process PROC structure and signal processing steps. It uses examples to show how to install signal catchers to handle program exceptions, such as segmentation faults, in user mode. It also discusses the suitability of using signals as a mechanism for Interprocess Communication (IPC). The programming project is for the reader to use signals and pipes to implement an IPC mechanism for processes to exchange message.

This chapter covers file systems. It explains the various levels of file operations in operating systems. These include preparing storage devices for file storage, file system support functions in kernel, system calls, library I/O functions on file streams, user commands and sh scripts for file operations. It presents a systematic overview of file operations, from read/write file streams in user space to system calls to kernel space down to the device I/O driver level. It describes low level file operations, which include disk partitions, example programs to display partition tables, format partitions for file systems and mount disk partitions. It presents an introduction to the EXT2 file system of Linux, which include the EXT2 file system data structures, example programs to display superblock, group descriptor, blocks and inodes bitmaps and directory contents. The programming project is to integrate the knowledge about EXT2/3 file systems and programming techniques presented in the chapter into a program which converts file pathnames into inodes and print their information.

This chapter covers system calls for file operations. It explains the role of system calls and the online manual pages of Linux. It shows how to use system calls for file operations. It lists and explains the most commonly used system calls for file operations. It explains hard link and symbolic link files. It explains the stat system call in detail. Based on the stat information, it develops a ls-like program to display directory contents and file information. Next, it explains the open-close-lseek system calls and file descriptors. Then it shows how to use read-write system calls to read-write file contents. Based on these, it shows how to use system calls to display and copy files. It shows how to develop a selective file copying program which behaves like a simplified dd utility program of Linux. The programming project is to use Linux system calls to implement a C program, which recursively copies a directory to a destination. The purpose of the project is for the reader to practice the design of hierarchical program structures and use stat(), open(), read(), write() system calls for file operations.

This chapter covers library I/O functions. It explains the roles of library I/O functions and their advantages over system calls. It uses sample programs to show the relationship between library I/O functions and system calls, and it explains their similarities and fundamental differences. It describes the algorithms of library I/O functions in detail. These include the algorithms of fread, fwrite and fclose, with an emphasis on their interactions with read, write and close system calls. It describes the different modes of library I/O functions, which include char mode, line mode, structured records mode and formatted I/O operations. It explains the buffering schemes of file streams and shows the effects of different buffering schemes by example programs. It explains functions with varying parameters and how to access parameters using stdarg macros.

The programming project is to integrate the principles and programming techniques of the chapter to implement a printf-like function for formatted printing of chars, strings and numbers in different number bases in accordance with a format string. The basis of the printf-like function is putchar() of Linux, but it works exactly the same as the library function printf(). The purpose of the project is to let the reader understand how library I/O functions are implemented.

This chapter covers sh programming. It explains sh scripts and different versions of sh. It compares sh scripts with C programs and points out the difference between interpreted and compiled languages. It shows how to write sh scripts in detail. These include sh variables, sh statements, sh built-in commands, regular system commands and command substitution. Then it explains sh control statements, which include test conditions, for loop, while loop, do-until loop, case statements, and it demonstrates their usage by examples. It shows how to write sh functions and invoke sh functions with parameters. It also shows the wide range of applications of sh scripts by examples. These include the installation, initialization and administration of the Linux system.

The programming project is for the reader to write a sh scripts which recursively copies files and directories. The project is organized in a hierarchy of three sh functions; cpf2f() which copies file to file, cpf2d() which copies file into a directory and cpd2d() which recursively copies directories.

This chapter covers EXT2 file system. The goal of this chapter is to lead the reader to implement a complete EXT2 file system that is totally Linux compatible. The premise is that if the reader understands one file system well, it should be easy to adapt to any other file systems. It first describes the historic role of EXT2 file system in Linux and the current status of EXT3/EXT4 file systems. It uses programming examples to show the various EXT2 data structures and how to traverse the EXT2 files system tree. Then it shows how to implement an EXT2 file system which supports all file operations as in the Linux kernel. It shows how to build a base file system by mount_root from a virtual disk. Then it divides the file system implementation into 3 levels. Level-1 expands the base file system to implement the file system tree. Level-2 implements read/write operations of file contents. Level-3 implements mount/umount of file systems and file protection. In each level, it describes the algorithms of the file system functions and demonstrates their implementations by programming examples. Each level is cumulated by a programming project. The final project is to integrate all the programming examples and exercises into a fully functional file system.

This chapter covers block device I/O and buffer management. It explains the principles of block device I/O and advantages of I/O buffering. It discusses the buffer management algorithm of Unix and points out its shortcomings. Then it uses semaphores to design new buffer management algorithms to improve the efficiency and performance of the I/O buffer cache. It is shown that the simple PV algorithm is easy to implement, has good cache effect and is free of deadlock and starvation. It presents a programming project to compare the performances of the Unix buffer management algorithm and the PV algorithm. The programming project should also help the reader to better understand I/O operations in file systems.

This chapter covers TCP/IP and network programming. The chapter consists of two parts. The first part covers the TCP/IP protocol and its applications. These include the TCP/IP stack, IP address, hostname, DNS, IP data packets and routers. It describes the UDP and TCP protocols, port number and data flow in TCP/IP networks. It explains the server-client computing model and the socket programming interface. It demonstrates network programming by examples using both UDP and TCP sockets. The first programming project is to implement a pair of TCP server-client to do file operations across the Internet. It allows the user to define additional communication protocols for reliable transfer of file contents.

The second part of the chapter covers Web and CGI programming. It explains the HTTP programming model, Web pages and Web browsers. It shows how to configure the Linux HTTPD server to support user Web pages, PHP and CGI programming. It explains both client side and server side dynamic Web pages. It shows how to create server side dynamic Web pages by both PHP and CGI. The second programming project is for the reader to implement server-side dynamic Web pages by CGI programming on a Linux HTTPD server machine.

This chapter covers the MySQL relational database system. It introduces MySQL and points out its importance. It shows how to install and run MySQL on Linux machines. It shows how to use MySQL to create and manage databases in both command mode and batch mode using SQL scripts. It shows how to interface MySQL with C programming. It also shows how to integrate MySQL with PHP to create and manage databases by dynamic Web pages.