UNIX System Programming

Introduction

1
 A History of computing
What is a computer?
Used to be a person whos job it was to
compute the results of a set of equations.
One of the original uses was to compute ballistic
trajectory tables for various artillery pieces.

Today, we understand the word computer to

mean a mechanical devices which can
follow a set of instructions to achieve a
specific goal.
2
 Phase 1: The Computer
Originally, the term was associated with
humans: Harvard Computers
(http://www.amnh.org/explore/news-blogs/news-posts/the-harvard-computers/)

Early computers were essentially just a

collection of separate mechanical devices.
One computer one program at a time.
One device for each purpose: Divide, multiply, add
Programming involved connecting the output of one device
to the input of others.
This was done by physically running wires from device to
device (e.g. http://www.columbia.edu/cu/computinghistory/eniac.html) 3
 The Computer
For now, lets just talk about a single
CPU/core and a single program/task.
The central main workhorse of a modern
computer is the CPU.
CPU Components:
Program Control Unit
Arithmetic Logic Unit
Busses:
Address
Data
Instructions
4
 5
(From: Stallings: Computer Organization and Architecture 10th ed)
 The Computer
How does it all work?
Where are the instructions read from?
Where does the data come from?
Where does the output go?

How dumb is your CPU?

What is it capable of doing?
Limited by its instruction set.

Limited by its physical architecture

Limited by the busses it is connected to.

6
 How do you print a character?
If you look at the instruction set for your
specific CPU, you wont find a PRINT
instruction.

If all you have is a CPU and some BUSes,

how do you get a character from the CPU to
the printer?

7
 The printer
Your printer:
You need it connected to a bus.
The CPU needs to know which bus to use.
You need to know how the printer recognizes
data that is meant for it to print. i.e. you need a
way of mechanically identifying the printer on
the bus.
You have to agree that it and only it should be
able to intercept that data (or other devices
could get confused).
How does the printer talk back to the CPU? 8
 The CPU and the character
The CPU has a character in a register that it
wants to print
The CPU must know that the printer is ready to receive
characters.
The appropriate BUS selection switches must be set by
the CPU.
The appropriate identifier (address) has to be set on the
bus to uniquely identify the printer.
The CPU then places the data on the data bus and
sends it to the printer.
The CPU must then wait for an acknowledgement from
the printer that the character was received.
9
 Coding the PRINT function
Can you imagine coding the PRINT function
every time you wanted to write a piece of
code?

Quickly people wrote standard decks of cards

what would have libraries of functions that
could be used to perform all of the tasks
similar to printing.

10
 A general purpose computer
Instructions for the CPU are stored in memory and not hard
wired

We have sets of fixed library routines which allows the CPU to

interact with the outside world.

We now have a general purpose computer that can run one

program at a time.
The program must run from start to finish.
When it is done, the system can be reset and memory loaded with
the next set of instructions.
New devices that become available, require their own
unique/specific set of driver libraries.
11
 Phase 2: Operating systems
Computers used to be very expensive. Good ones still are.

The push was on to allow for a single computer to be used as

efficiently as possible.

That meant running multiple jobs at closely to

simultaneously as possible.

As soon as you need to manage how the hardware will run

instructions from multiple sources at the same time, you
need your own program to manage the system.

Demand then drove the development of Operating Systems.

12
 Phase 2: Multi-
Multi-user systems
What happens when you try to accommodate
multiple users on a system?

Lets assume: BATCH: Only one program runs at a

time. It has FULL access to the system.

Need to manage resources

Who can access the system and when.
Resource limitations on a per user basis?
Need to manage running the next program when this
one finishes to avoid idle time.
The libraries for accessing devices/hardware become
system enforced to avoid ummm sneaky code. 13
 Phase 3: Time-
Time-Shared systems
Now we have multiple applications running:
at the same time
in the same memory space
Using the same devices
We need to restrict further:
Ownership becomes an issue
Access rights.
Playing NICE with the other users
Scheduling CPU (Yikes! multi-core !)
Managing memory resources 14
 Phase 3: The Kernel
Multi-user, multi-process environments require
one more level of system management.

The Kernel.
User applications can only run in a virtualized
environment.
The kernel manages access rights and resource
consumption for all processes.
They have zero access to the underlying hardware.
All access to recourses is strictly managed by the
kernel.
15
 The BOOT process
How does your computer go from being a dumb piece of silicon to being a
usable computer?

POST (Power on Self-Test)

Finding the boot device
Loading the O/S kernel
Starting kernel level routines
Starting background processes (databases, web
servers...)
Allowing interactive logins
16
 UNIX System Programming
All about making calls to the Kernel to get
access to resources.
The *NIX shell: What is it and how to use it.
Some C programming so were all on the same page.
Accessing files
Accessing Directories
Processes
Inter-Process communications
Signals
PIPES & FIFOs
Sockets

The UNIX Mantra: Write something ONCE. Write it well, make

it efficient and versatile. Never write another program to 17
do the same thing.
 UNIX History
Developed in the late 1960s and 1970s at Bell Labs ( the most versatile, powerful
and flexible OS in the word). K. Thomson, D. Ritchie, McIlroy, Ossanna (nroff) and
later Canaday
UNICS - a pun on MULTICSn time share system (Multiplexed Information and
Computer Service) which was supposed to support 1000 on line users but only
handled a few (barely 3). (MULTI-UNiplexed)
Thomson writes first version of UNICS in assembler for a PDP-7 in one MONTH
which contains a new type of file system (initial motivation was the game space
travel)
Kernel (notion of processes)
Shell
editor and the
assembler
1969 Thomson writes interpreter B based on BCPL Ritchie improves on B and
called it C (but first NB).
1972 UNIX is rewritten in C to facilitate porting

18
 UNIX History (contd)
1973 UNIX philosophy
developed:
Write programs that do one
thing and do it well:
e.g. only write one sort program!

Write programs that work

together (PIPEs, Filters)

Write programs that handle

text streams, because that is
the universal interface

KISS keep it simple stupid 19

 UNIX User interface
Originally, everything done at the command
line.
There were NO graphical displays, mice...
UNIX users typed in commands to do all
tasks.
Because of its underlying everything is a
file architecture, multiple commands could
be linked together to perform more complex
tasks.

20
 UNIX User interface
Eventually, graphical user interfaces were
developed.
Unlike most other operating systems
however, you can swap out one window
manager for a different one.
You can also, in some cases choose
different desktop managers as well.
GUI wars: which interface is prettiest, which
is more lightweight (which one looks more
like Windows???)
21
 UNIX Today
Supports many users running many programs
at the same time, all sharing the same
computer system
Information Sharing
Geared towards facilitating the job of creating
new programs
Sun: SunOS, Solaris; GNU: Linux; SGI: IRIX;
Free BSD; Hewlett Packard: HP-UX; Apple: OS
X (Darwin)

22
 UNIX
1) Everything in *NIX is CASE SENSITIVE !!!
Lab1 is different from lab1 which is different from
LAB1 etc.

2) You can create hidden files by using a

single period as a prefix. (e.g. .cshrc or
.bashrc)
You can see them by adding a a to the ls
command.
3) Theres no such concept as a file extension.
23
 The UNIX System
Kernel Heart of the OS
Process scheduling
Resource Sharing
I/O control (accesses)
Shell Interpreter between the user and the
computer
Tools and applications
Accessible from shell
Can be run independently of shell

24
 The UNIX System
Linux can come in a number of different
flavours:
Single user systems (GUI desktops)
Server class systems - allows simultaneous
multi-user access (no GUI)
Though the difference is basically just what
applications are installed/running

Two kinds of users / activity:

Those that can run in their own space
Those that need access to system resources
25
 UNIX System Programming
Programs make system calls (also called
supervisor calls to invoke kernel.
A system call is essentially a procedure call
into the operating system
The procedure call is protected
Types of system calls
File I/O
Process management
Inter-process communication (IPC),pipes, sockets, fifos
Signal handling

26
 System Calls (Library calls)
System calls
Interface to the kernel

Program
Code
Library fread

User Space read user

read
kernel
Kernel Space
27
 User UNIX Interface: SHELL
A shell is just a program that someone wrote.
Provides command line as an interface between
the user and the system
The shell will recognize the names and parameters
for specific commands and it will go run those
programs for you.
Is simply a program that starts automatically when
you login
Uses a command language
Allows programming (shell scripting) within the shell
environment
Uses variables, loops, conditionals, etc.

Accepts commands and often makes system calls to 28

carry them out
 Various UNIX shells
sh (Bourne shell)
ksh (Korn shell)
csh (C shell)
tcsh
bash
Differences mostly in scripting details

29
 Environment variables
A set of variables the shell uses for certain
operations
Variables have a name and a value
Current list can be displayed with the env
command
A particular variables value can be displayed
with echo $<var_name>
Some interesting variables: HOME, PATH,
PS1, USER, HOSTNAME, PWD

30
 Setting environment variables
Set a variable with
Ksh/bash: <name>=<value>
tcsh: setenv <name> <value>
Examples:
TERM=vt100
PS1=myprompt>
PS1=$USER@$HOSTNAME:
PS1=multiple word prompt>
PATH=$PATH:$HOME
DATE=`date`

31
 Aliases
Aliases are used as shorthand for frequently
used commands
Syntax:
ksh: alias <shortcut>=<command>
tcsh: alias <shortcut> <command>
Examples:
alias ll=ls -lF
alias la=ls -la
alias m=more
alias up=cd ..
alias prompt=echo $PS1

32
 Repeating Commands
Use history to list the last 100 commands

tcsh: traverse command history:

<CNTRL>-P previous history
<CNTRL>-N next history

ksh: ESC, then k (up), j (down) RETURN

!101 will execute command 101

33
 Login scripts
You dont want to enter aliases, set environment
variables, set up command line editing, etc. each
time you log in
All of these things can be done in a script that is
run each time the shell is started
For ksh:
~/.profile - is read for a login shell
~/.kshrc
For tcsh
~/.login
~/.cshrc
34
 Example .profile (partial)
# set ENV to a file invoked each time sh is
started for interactive use.
ENV=$HOME/.shrc; export ENV
HOSTNAME=`hostname`; export HOSTNAME
PS1="$USER@$HOSTNAME>"

alias 'll'='ls -l'

alias 'la'='ls -la'
alias 'ls'='ls -F'
alias 'rm'='rm -i'
alias 'm'='more'

set -o vi
echo ".profile was read"

35
 stdin, stdout, and stderr
Each shell (and in fact all programs) automatically
open three files when they start up
Standard input (stdin): Usually from the keyboard
Standard output (stdout): Usually to the terminal
Standard error (stderr): Usually to the terminal

Programs use these three files when reading (e.g.

cin), writing (e.g. cout), or reporting
errors/diagnostics

36
 Redirecting stdout
Instead of writing to the terminal, you can tell a
program to print its output to another file using the
> operator

>> operator is used to append to a file

Examples:
man ls > ls_help.txt
echo $PWD > current_directory
cat file1 >> file2

37
 Redirecting stderr
Instead of writing errors to the terminal, you can
tell a program to write them to another file using
the:
ksh: 2> operator
tcsh: >& operator

Examples (suppose j is a file that does not exist)

$ ls j
ls: j: No such file or directory
$ ls j >& hello.txt
$ cat hello.txt
ls: j: No such file or directory
38
 Redirecting stdin
Instead of reading from the terminal, you can
tell a program to read from another file using
the < operator

Examples:
Mail user@domain.com < message
interactive_program < command_list

39
 Pipes and filters
Pipe: a way to send the output of one
command to the input of another

Filter: a program that takes input and

transforms it in some way
wc - gives a count of words/lines/chars
grep - searches for lines with a given string
more - lists the contents of file
sort - sorts lines alphabetically or numerically

40
 Examples of filtering
ls -la | more
cat file | wc
man ksh | grep history
ls -l | grep dkl | wc
who | sort > current_users

41
 UNIX Filesystem
The filesystem is your interface to
physical storage (disks) on your machine
storage on other machines
output devices
etc.
Everything in UNIX is a file (programs, text,
peripheral devices, terminals, )
There are no drive letters in UNIX! The filesystem
provides a logical view of the storage devices

42
 Working directory
The current directory in which you are
working
pwd command: outputs the absolute path
(more on this later) of your working directory
Unless you specify another directory,
commands will assume you want to operate
on the working directory

43
 Home directory
A special place for each user to store
personal files
When you log in, your working directory will
be set to your home directory
Your home directory is represented by the
symbol ~ (tilde)
The home directory of user1 is represented
by ~user1

44
 UNIX file hierarchy
/

Directories may bin users tmp

contain plain files
or other dkl kangli
directories
Leads to a tree
foo.txt cois3380h
structure for the
file system
bar.c abcd
Root directory: /

45
 Path names /

Separate directories by / bin users tmp

Absolute path
start at root and follow the dkl rhurley
tree
e.g. /users/dkl/foo.txt
foo.txt cois3380h

Relative path
start at working directory bar.c abcd
.. refers to level above; . refers to working dir.
If /users/dkl/cois3380h is working dir, all these
refer to the same file
../foo.txt ~/foo.txt ~dkl/foo.txt
46
 Types of files
Plain (- in the first bit)
Most files
Includes binary and text files
Directory (d)
A directory is actually a file
Points to another set of files
Link (l): A pointer to another file or directory
Special: e.g. peripheral devices

47
 Changing directories
Change the working directory with the cd
command
cd <dir_name>
Use absolute or relative path names
cd by itself equivalent to cd ~

48
 Output of ls -lF
$ ls -lF
total 110
-rwx------ 1 rhurley staff 6732 Jul 3 11:52 a.out*
drwx------ 9 rhurley staff 512 May 18 2007 Library/
-rw------- 1 rhurley staff 239 May 2 12:55 execex.c

file name
modify date
group
owner size
number of links
permissions The F option appends a qualifier to the
listed file name. *=executable,
/=directory...
file type
49
 Creating links
ln s <curr_file> <link_name>

This command creates a symbolic link

The file link_name will be a pointer to the

curr_file which may be in another directory or
even on another physical machine

50
 File permissions
Permissions used to allow/disallow access
to file/directory contents
Read (r) 4, write (w) 2, and execute (x) 1
For owner, group, and world (everyone)
chmod <mode> <file(s)>
chmod 700 file.txt (only owner can
read, write, and execute)
chmod g+rw file.txt

51
 File Ownership
Each file has a single owner

chown command can be used to change the

owner (usually only root user can use this
command)

There are also various groups to which users

can belong

Groups may have different permissions than

everyone else

52
 File Modification Date
Last time the file was changed

Useful information when

There are many copies of a file
Many users are working on a file

touch command can be used to update

the modification date to the current date,
or to create a file if it doesnt exist

53
 Looking at file contents
cat <filename(s)>
concatenate
output the contents of the file all at once

more <filename(s)>
Output the contents of a file one screen at a time
Allows forward and backward scroll and search
You can use the less command as well

54
 Moving, renaming, copying, and
removing files
mv <file1> <file2> (rename)

mv <file1> <dir> (move)

mv <file1> <dir/file2> (move & rename)

cp <file1>
[<file2>|<dir>|<dir/file2>] (copy)

rm [-i] <file(s)> (remove)

55
 Creating and removing
directories
mkdir <dir_name>
Create a subdirectory of the current directory

rmdir <dir_name>
Remove a directory (only works for empty
directories)

rm r <dir_name>
Remove a directory and all of its contents, including
subdirectories

56
 Wildcards in file names
All of the commands covered here that
take file names as arguments can also
use wildcards
* for any string, e.g. *.txt, obj*, a*.*
? for any character, e.g. doc?
[ ] around a range of characters, e.g. [a-c]*

57
 Getting help on UNIX commands
man <command_name> shows you all the
documentation for a command
(https://www.kernel.org/doc/man-pages/)
1: User commands; man-pages includes a very few Section 1 pages that document
programs supplied by the GNU C library.
2: System calls documents the system calls provided by the Linux kernel.
3: Library functions documents the functions provided by the standard C library.
4: Devices documents details of various devices, most of which reside in /dev.
5: Files describes various file formats, and includes proc(5), which documents the /proc file
system.
7: Overviews, conventions, and miscellaneous.
8: Superuser and system administration commands; man-pages includes a very few Section
8 pages, mainly documenting programs supplied by the GNU C library.

apropos <keyword> shows you all the

commands with the keyword in their
description 58