In this chapter, we'll show you how to upgrade software on your system,
including rebuilding and installing a new operating system kernel.
Although most Linux distributions provide some automated means to install,
remove, and upgrade specific software packages on your system, it is
often necessary to install software by hand. The kernel is the operating
system itself. It is a set of routines and data that is loaded by the
system at boot time and controls everything on the system: software access
to hardware devices, scheduling of user processes, memory management, and
more. Building your own kernel is often beneficial, as you can select which
features you want included in the operating system.
Installing and upgrading free software is usually more complicated than
installing commercial products. Even when you have precompiled binaries
available, you may have to uncompress them and unpack them from an archive
file. You may also have to create symbolic
links or set environment variables so that the binaries know where to
look for the resources they use. In other cases, you'll need to compile
the software yourself from sources.
Another common Linux activity is building the kernel. This is an
important task for several reasons. First of all, you may find yourself
in a position where you need to upgrade your current kernel to a newer
version, to pick up new features or hardware support. Secondly building
the kernel yourself allows you to select which features you do (and do not)
want included in the compiled kernel.
Why is the ability to select features a win for you? All kernel code
and data is "locked down" in memory; that is, it cannot
be swapped out to disk.
For example, if you use a kernel image with drivers for hardware you do not
have or use, the memory consumed by those hardware drivers cannot be
reclaimed for use by user applications. Customizing the kernel allows you
to trim it down for your needs.
7.1. Archive and Compression Utilities
When installing or upgrading software on Unix systems, the first things
you need to be familiar with are the tools used for compressing and
archiving files. There are dozens of such utilities available.
Some of these (such as tar and compress)
date back
to the earliest days of Unix; others (such as gzip) are
relative newcomers. The main goal of these utilities is to
archive files (that is, to pack many files
together into a single file for easy transportation or backup) and
to compress files (to reduce the amount of disk space required
to store a particular file or set of files).
In this section, we're
going to discuss the most common file formats and utilities
you're likely to run into. For instance, a near-universal convention
in the Unix world is to transport files or software as a
tar archive, compressed using compress or gzip.
In order to create or unpack these files yourself, you'll need
to know the tools of the trade. The tools are most often used when
installing new software or creating backups--the subject of
the following two sections in this chapter.
7.1.1. Using gzip and bzip2
gzip is a fast and efficient compression program distributed by the
GNU project. The basic function of gzip is to take a file,
compress it, save the compressed version as filename.gz,
and remove the original, uncompressed file. The original file is
removed only if gzip is successful; it is very difficult to accidentally
delete a file in this manner. Of course, being GNU software, gzip has
more options than you want to think about, and many aspects of its
behavior can be modified using command-line options.
First, let's say that we have a large file named garbage.txt:
rutabaga% ls -l garbage.txt
-rw-r--r-- 1 mdw hack 312996 Nov 17 21:44 garbage.txt
To compress this file using
gzip, we simply use the command:
gzip garbage.txt
This replaces
garbage.txt with the compressed file
garbage.txt.gz. What we end up with is the following:
rutabaga% gzip garbage.txt
rutabaga% ls -l garbage.txt.gz
-rw-r--r-- 1 mdw hack 103441 Nov 17 21:44 garbage.txt.gz
Note that
garbage.txt is removed when
gzip completes.
You can give gzip a list of filenames; it compresses each file
in the list, storing each with a .gz extension. (Unlike the
zip program for Unix and MS-DOS systems, gzip will not,
by default, compress several files into a single .gz archive. That's
what tar is for; see the next section.)
How efficiently a file is compressed depends upon its format and contents.
For example, many graphics file formats (such as GIF and JPEG) are already
well compressed, and gzip will have little or no effect upon such
files. Files that compress well usually include plain-text files, and
binary files such as executables and libraries. You can get information on a gzipped file using gzip -l.
For example:
rutabaga% gzip -l garbage.txt.gz
compressed uncompr. ratio uncompressed_name
103115 312996 67.0% garbage.txt
To get our original file back from the compressed version, we use
gunzip, as in:
gunzip garbage.txt.gz
After doing this, we get:
rutabaga% gunzip garbage.txt.gz
rutabaga% ls -l garbage.txt
-rw-r--r-- 1 mdw hack 312996 Nov 17 21:44 garbage.txt
which is identical to the original file. Note that when you
gunzip a file, the compressed version is removed once the
uncompression is complete.
gzip stores the name of the original, uncompressed file
in the compressed version. This way, if the compressed filename
(including the .gz extension) is too long for the filesystem type
(say, you're compressing a file on an MS-DOS filesystem with 8.3 filenames),
the original filename can be restored using gunzip even if the
compressed file had a truncated name.
To uncompress a file to its original filename, use the -N option
with gunzip. To see the value of this option, consider
the following sequence of commands:
rutabaga% gzip garbage.txt
rutabaga% mv garbage.txt.gz rubbish.txt.gz
If we were to
gunzip rubbish.txt.gz at this point, the uncompressed
file would be named
rubbish.txt, after the new (compressed) filename.
However, with the
-N option, we get:
rutabaga% gunzip -N rubbish.txt.gz
rutabaga% ls -l garbage.txt
-rw-r--r-- 1 mdw hack 312996 Nov 17 21:44 garbage.txt
gzip and gunzip can also compress or uncompress data
from standard input and output. If gzip is given no filenames to
compress, it attempts to compress data read from standard input. Likewise,
if you use the -c option with gunzip, it writes uncompressed
data to standard output. For example, you could pipe the output of a
command to gzip to compress the output stream and save it to a file
in one step, as in:
rutabaga% ls -laR $HOME | gzip > filelist.gz
This will produce a recursive directory listing of your home directory
and save it in the compressed file
filelist.gz. You can display
the contents of this file with the command:
rutabaga% gunzip -c filelist.gz | more
This will uncompress
filelist.gz and pipe the output to the
more command. When you use
gunzip -c, the file on disk
remains compressed.
The zcat command is identical to gunzip -c.
You can think of this as a version of cat for
compressed files. Linux even has a version of
the pager less for compressed files,
called zless.
When compressing files, you can use one of the options -1,
-2, through -9 to specify the speed and quality
of the compression used. -1 (also --fast) specifies the fastest
method, which compresses the files less compactly, while -9
(also --best) uses the slowest, but best compression method.
If you don't specify one of these options the default is
-6. None of these options has any bearing on how you use
gunzip; gunzip will be able to uncompress the file no matter
what speed option you use.
gzip is relatively new in the Unix world. The compression programs
used on most Unix systems are compress
and uncompress, which were
included in the original Berkeley versions of Unix. compress and
uncompress are very much like gzip and gunzip,
respectively; compress saves compressed files as
filename.Z as opposed to filename.gz,
and uses a slightly less efficient compression algorithm.
However,
the free software community has been moving to gzip for several
reasons. First of all, gzip works better. Second there has
been a patent dispute over the compression algorithm used by
compress--the results of which could prevent third parties
from implementing the compress algorithm on their own. Because
of this, the Free Software Foundation urged a move to gzip, which
at least the Linux community has embraced. gzip has been ported to
many architectures, and many others are following suit. Happily, gunzip
is able to uncompress the .Z format files
produced by compress.
Another compression/decompression
program has also emerged to take the lead from
gzip. bzip2 is the new
kid on the block and sports even better compression (on the
average about 10-20 percent better than
gzip), at the expense of longer compression
times. You cannot use bunzip2 to uncompress
files compressed with gzip and vice versa,
and since you cannot expect everybody to have
bunzip2 installed on their machine, you
might want to confine yourself to gzip for
the time being if you want to send the compressed file to
somebody else. However, it pays to have
bzip2 installed, because more and more
FTP servers now provide
bzip2-compressed packages in order to
conserve disk space and bandwidth. You can recognize
bzip2-compressed files from their typical
.bz2 file name extension.
While the command-line options of
bzip2 are not exactly the same as those of
gzip, those that have been described in
this section are. For more information, see the
bzip2 manual page.
The bottom line is that you should use
gzip/gunzip or bzip2/bunzip2 for your
compression needs. If you encounter a file with the extension
.Z,
it was probably produced by compress, and
gunzip can
uncompress it for you.
Earlier versions of gzip used
.z (lowercase) instead of .gz as the compressed-filename
extension. Because of the potential confusion with .Z, this was
changed. At any rate, gunzip retains backwards-compatibility with
a number of filename extensions and file types.
7.1.2. Using tar
tar is a general-purpose archiving utility capable of packing many
files into a single archive file, retaining information, such as file permissions
and ownership. The name tar stands for tape archive, because
the tool was originally used to archive files as backups on tape. However,
use of tar is not at all restricted to making tape backups, as we'll see.
The format of the tar command is:
tar functionoptions files…
where
function is a single letter indicating the operation
to perform,
options is a list of (single-letter) options
to that function, and
files is the list of files to pack
or unpack in an archive. (Note that
function is not
separated from
options by any space.)
function can be one of:
- c
To create a new archive
- x
To extract files from an archive
- t
To list the contents of an archive
- r
To append files to the end of an archive
- u
To update files that are newer than those in the archive
- d
To compare files in the archive to those in the filesystem
You'll rarely use most of these functions; the more commonly used are
c, x, and t.
The most common options are:
- v
To print verbose information when packing or unpacking archives
- k
To keep any existing files when extracting--that
is, to not overwrite any existing files which are contained within the
tar file
- f filename
To specify that the tar file to
be read or written is filename
- z
To specify that the data to be written to the
tar file should be compressed or that the data in the tar file is
compressed with gzip
- v
To make tar show the files
it is archiving or restoring--it is good practice to use this so that
you can see what actually happens (unless, of course, you are writing
shell scripts)
There are others, which we will cover later in this section.
Although the tar syntax might appear complex at first, in practice
it's quite simple. For example, say we have a directory named
mt, containing these files:
rutabaga% ls -l mt
total 37
-rw-r--r-- 1 root root 24 Sep 21 1993 Makefile
-rw-r--r-- 1 root root 847 Sep 21 1993 README
-rwxr-xr-x 1 root root 9220 Nov 16 19:03 mt
-rw-r--r-- 1 root root 2775 Aug 7 1993 mt.1
-rw-r--r-- 1 root root 6421 Aug 7 1993 mt.c
-rw-r--r-- 1 root root 3948 Nov 16 19:02 mt.o
-rw-r--r-- 1 root root 11204 Sep 5 1993 st_info.txt
We wish to pack the contents of this directory into a single
tar
archive. To do this, we use the command:
tar cf mt.tar mt
The first argument to
tar is the
function (here,
c,
for create) followed by any
options. Here, we use the one
option
f mt.tar, to specify that the resulting tar archive
be named
mt.tar. The last argument is the name of the
file or files to
archive; in this case, we give the name of a directory, so
tar packs all files in that directory into the archive.
Note that the first argument to tar must be a function letter
followed by a list of options. Because of this, there's no reason
to use a hyphen (-) to precede the options as many Unix commands
require. tar allows you to use a hyphen, as in:
tar -cf mt.tar mt
but it's really not necessary. In some versions of
tar, the
first letter must be the
function,
as in
c,
t, or
x. In other versions, the order of letters does not matter.
The function letters as described here follow the so-called "old
option style." There is also a newer "short option
style" where you precede the function options with a hyphen, and
a "long option style," where you use long option names
with two hyphens. See the Info page for tar for
more details if you are interested.
It is often a good idea to use the v option with tar;
this lists each file as it is archived. For example:
rutabaga% tar cvf mt.tar mt
mt/
mt/st_info.txt
mt/README
mt/mt.1
mt/Makefile
mt/mt.c
mt/mt.o
mt/mt
If you use
v multiple times, additional information will
be printed, as in:
rutabaga% tar cvvf mt.tar mt
drwxr-xr-x root/root 0 Nov 16 19:03 1994 mt/
-rw-r--r-- root/root 11204 Sep 5 13:10 1993 mt/st_info.txt
-rw-r--r-- root/root 847 Sep 21 16:37 1993 mt/README
-rw-r--r-- root/root 2775 Aug 7 09:50 1993 mt/mt.1
-rw-r--r-- root/root 24 Sep 21 16:03 1993 mt/Makefile
-rw-r--r-- root/root 6421 Aug 7 09:50 1993 mt/mt.c
-rw-r--r-- root/root 3948 Nov 16 19:02 1994 mt/mt.o
-rwxr-xr-x root/root 9220 Nov 16 19:03 1994 mt/mt
This is especially useful as it lets you verify that
tar is doing the
right thing.
In some versions of tar, f must be the last letter in the list of options. This is because tar expects the f option to
be followed by a filename--the name of the tar file to read from
or write to. If you don't specify f filename at all,
tar assumes for historical reasons that it should use the
device /dev/rmt0 (that is, the first tape drive). In the
section "Section 8.1, "Making Backups","
in Chapter 8, "Other
Administrative
Tasks", we'll talk about using
tar in conjunction with a tape drive to make backups.
Now, we can give the file mt.tar to other people,
and they can extract it on their own system. To do this, they would
use the command:
tar xvf mt.tar
This creates the subdirectory
mt and places all the
original files into it, with the same permissions
as found on the original system.
The new files will be owned by the user running the
tar xvf (you) unless you are running as root, in
which case the original owner is preserved.
The
x option stands for
"extract."
The
v option is used again here
to list each file as it is extracted. This produces:
courgette% tar xvf mt.tar
mt/
mt/st_info.txt
mt/README
mt/mt.1
mt/Makefile
mt/mt.c
mt/mt.o
mt/mt
We can see that tar saves the pathname of each file relative to
the location where the tar file was originally created. That is,
when we created the archive using tar cf mt.tar mt, the only
input filename we specified was mt, the name of the
directory containing the files. Therefore, tar stores the directory
itself and all of the files below that directory in the tar file.
When we extract the tar file, the directory mt is created and
the files placed into it, which is the exact inverse of what was done to create
the archive.
By default, tar extracts all tar files relative to
the current directory where you execute tar. For example,
if you were to pack up the contents of your /bin directory
with the command:
tar cvf bin.tar /bin
tar would give the warning:
tar: Removing leading / from absolute path names in the archive.
What this means is that the files are stored in the archive within the
subdirectory
bin. When this tar file is extracted, the directory
bin is created in the working
directory of
tar--not as
/bin on the system where the extraction is being done.
This is very important and is meant to prevent terrible mistakes
when extracting tar files. Otherwise, extracting a tar file packed as,
say,
/bin, would trash the contents of your
/bin directory when
you extracted it. If you really wanted to extract such a tar file into
/bin, you would extract it from the root directory,
/.
You can override this behavior using the
P option when packing
tar files, but it's not recommended you do so.
Another way to create the tar file mt.tar would have been to cd
into the mt directory itself, and use a command such as:
tar cvf mt.tar *
This way the
mt subdirectory would not be stored in the
tar file; when extracted, the files would be placed directly in your
current working directory. One fine point of
tar etiquette is
to always pack tar files so that they contain a subdirectory, as we
did in the first example with
tar cvf mt.tar mt. Therefore,
when the archive is extracted, the subdirectory is also created and
any files placed there. This way you can ensure that the files
won't be placed directly in your current working directory; they
will be tucked out of the way and prevent confusion. This also
saves the person doing the extraction the trouble of having to
create a separate directory (should they wish to do so) to unpack
the tar file. Of course, there are plenty of situations where
you wouldn't want to do this. So much for etiquette.
When creating archives, you can, of course, give tar a list of
files or directories to pack into the archive. In the first
example, we have given tar the single directory mt,
but in the previous paragraph we used the wildcard *, which
the shell expands into the list of filenames in the current
directory.
Before extracting a tar file, it's usually a good idea to take a look
at its table of contents to determine how it was packed. This
way you can determine whether you do need to create a subdirectory yourself
where you can unpack the archive. A command such as:
tar tvf tarfile
lists the table of contents for the named
tarfile.
Note that when using the
t function, only one
v is
required to get the long file listing, as in this example:
courgette% tar tvf mt.tar
drwxr-xr-x root/root 0 Nov 16 19:03 1994 mt/
-rw-r--r-- root/root 11204 Sep 5 13:10 1993 mt/st_info.txt
-rw-r--r-- root/root 847 Sep 21 16:37 1993 mt/README
-rw-r--r-- root/root 2775 Aug 7 09:50 1993 mt/mt.1
-rw-r--r-- root/root 24 Sep 21 16:03 1993 mt/Makefile
-rw-r--r-- root/root 6421 Aug 7 09:50 1993 mt/mt.c
-rw-r--r-- root/root 3948 Nov 16 19:02 1994 mt/mt.o
-rwxr-xr-x root/root 9220 Nov 16 19:03 1994 mt/mt
No extraction is being done here; we're just displaying the archive's
table of contents. We can see from the filenames that this file
was packed with all files in the subdirectory
mt, so that when
we extract the tar file, the directory
mt will be created,
and the files placed there.
You can also extract individual files from a tar archive.
To do this, use the command:
tar xvf tarfile files
where
files is the list of files to extract. As we've
seen, if you don't specify any
files,
tar
extracts the entire archive.
When specifying individual files to extract, you must give the
full pathname as it is stored in the tar file. For example, if
we wanted to grab just the file mt.c from the previous
archive mt.tar, we'd use the command:
tar xvf mt.tar mt/mt.c
This would create the subdirectory
mt and place the file
mt.c within it.
tar has many more options than those mentioned here.
These are the features that you're likely to use most of the
time, but GNU tar, in particular,
has extensions that make
it ideal for creating backups and the like. See the tar
manual page and the following section
for more information.
7.1.3. Using tar with gzip
tar does not compress the data stored in its archives in
any way. If you are creating a tar file from three 200K files,
you'll end up with an archive of about 600K. It is common practice
to compress tar archives with gzip (or the older compress
program). You could create a gzipped tar file using the
commands:
tar cvf tarfile files…
gzip -9 tarfile
But that's so cumbersome, and requires you to have enough space to
store the uncompressed
tar file before you
gzip it.
A much trickier way to accomplish the same task is to use an interesting
feature of tar that allows you to write an archive to standard
output. If you specify - as the tar file to read or write, the
data will be read from or written to standard input or output. For
example, we can create a gzipped tar file using the command:
tar cvf - files… | gzip -9 > tarfile.tar.gz
Here,
tar creates an archive from the named
files and
writes it to standard output; next,
gzip reads the data from
standard input, compresses it, and writes the result to its own standard
output; finally, we redirect the gzipped tar file to
tarfile.tar.gz.
We could extract such a tar file using the command:
gunzip -9c tarfile.tar.gz | tar xvf -
gunzip uncompresses the named archive file, writes the result to
standard output, which is read by
tar on standard input and
extracted. Isn't Unix fun?
Of course, both of these commands are rather cumbersome to type.
Luckily, the GNU version of tar provides the z option which
automatically creates or extracts gzipped archives. (We saved
the discussion of this option until now, so you'd truly appreciate its
convenience.) For example, we could use the commands:
tar cvzf tarfile.tar.gz files…
and:
tar xvzf tarfile.tar.gz
to create and extract gzipped tar files. Note that you should
name the files created in this way with the
.tar.gz filename
extensions (or the equally often used
.tgz, which
also works on systems with limited filename capabilities), to
make their format obvious. The
z option works
just as well with other tar functions such as
t.
Only the GNU version of tar supports the z option; if you
are using tar on another Unix system, you may have to use one
of the longer commands to accomplish the same tasks.
Nearly all Linux systems use GNU tar.
When you want to use tar in
conjunction with bzip2, you need to tell
tar about your compression program preferences like
this:
tar cvf tarfile.tar.bz2 --use-compress-program=bzip2 files...
or, shorter:
tar cvf tarfile.tar.bz2 --use=bzip2 files...
or, shorter still:
tar cvIf tarfile.tar.bz2 files
The latter version only works with newer versions of GNU
tar that supports the I
option.
Keeping this in mind, you could write short shell scripts or
aliases to handle cookbook tar file creation and extraction for you.
Under bash, you could include the following functions in
your .bashrc:
tarc () { tar czvf $1.tar.gz $1 }
tarx () { tar xzvf $1 }
tart () { tar tzvf $1 }
With these functions, to create a gzipped tar file from a
single directory, you could use the command:
tarc directory
The resulting archive file would be named
directory.tar.gz.
(Be sure that there's no trailing slash on the directory name; otherwise
the archive will be created as
.tar.gz within the given
directory.)
To list the table of contents of a gzipped tar file, just use:
tart file.tar.gz
Or, to extract such an archive, use:
tarx file.tar.gz
7.1.4. tar Tricks
Because tar saves the ownership and permissions of files
in the archive and retains the full directory structure, as well
as symbolic and hard links, using tar is an excellent way to
copy or move an entire directory tree from one place to another
on the same system (or even between different systems, as we'll see).
Using the - syntax described earlier, you can write a tar file
to standard output, which is read and extracted on standard input
elsewhere.
For example, say that we have a directory containing two subdirectories:
from-stuff and to-stuff. from-stuff contains an entire
tree of files, symbolic links, and so forth--something that is
difficult to mirror precisely using a recursive
cp. In order
to mirror the entire tree beneath from-stuff to to-stuff,
we could use the commands:
cd from-stuff
tar cf - . | (cd ../to-stuff; tar xvf -)
Simple and elegant, right? We start in the directory
from-stuff
and create a tar file of the current directory, which is written to
standard output. This archive is read by a subshell (the commands
contained within parentheses); the subshell does a
cd to the target directory,
../to-stuff (relative to
from-stuff, that is), and then runs
tar xvf, reading
from standard input. No tar file is ever written to disk; the
data is sent entirely via pipe from one
tar process to another.
The second
tar process has the
v
option that prints
each file as it's extracted; in this way, we can verify that the command
is working as expected.
In fact, you could transfer directory trees from one machine to
another (via the network) using this trick; just include an appropriate
rsh command within the subshell on the right side of the pipe.
The remote shell would execute tar to read the archive on its
standard input. (Actually, GNU tar has facilities to read or write tar files automatically
from other machines over the network; see the
tar manual page for details.)
