Shell Scripting

Oh, a final word of warning. If you haven't noticed already, Unix and Linux people have a pretty wacky sense of humor. I'm no exception. If you find some of my jokes and quips in this course offensive, you're definitely taking this whole Linux thing WAY TOO SERIOUSLY. Take a chill pill and re-read it and relax!

The diagram below illustrates the structure of the info pages. Refer to it while reading this section

Figure 1.1. The structure of the info pages

Info pages are like man pages in many ways, except they provide a lot more information than man pages (well mostly anyway). Info pages are available on all the Linux distributions currently available, and they are similar to the man pages, although in some instances give you more information.

If we look at a typical way of invoking info, it would be the word info, followed by a space, followed by the name of the command that you want to invoke info on. For example, for the command ls:

info ls
---^-
                

	Note
	Type the commands just as you see them here. I have placed minus signs under the command and it's arguments, and a caret (^) under the space. This is to illustrate that the command should be typed EXACTLY as is.

This should give us an information page on the ls command. We could invoke info on a whole range of utilities by typing:

info coreutils 
---^------
                

where coreutils is just another group of info pages. Coreutils is a grouping of info pages, containing commands such as ls, cd, cp, mv or list directory contents (ls), change directory (cd), copy files and directories (cp) and move or rename files (mv). One could, for instance, type:

info mv
                

The way info is structured, is that when you first start it, it will display the node at which you are beginning your search.

In our coreutils example, on the top of the page (right at the top) the first line looks as follows:

File: coreutils.info,  Node: Top,  Next: Introduction,  Up: (dir)
                

Starting on the left-hand side, we see the file that we're "info'ing" the coreutils.info file.

The filename that contains information about the ls, cp, mv and cd commands amongst others is coreutils.info.

The current Node is Top, which indicates that we're at the top of the coreutils info page and cannot go higher within this group of pages.

From the top level of the info page on coreutils we can now do a couple of things:

We can go to the next node (by typing 'n'), which is the next topic, called Introduction. You will notice that there is no link to a Previous topic, since we are at the top of this group of pages.

We could scroll down to the menu and select a particular topic (node) we want displayed.

File: info.info,  Node: Top,  Next: Getting Started,  Up: (dir)

Info: An Introduction
*********************

   The GNU Project distributes most of its on-line manuals in the "Info
format", which you read using an "Info reader".  You are probably using
an Info reader to read this now.

   If you are new to the Info reader and want to learn how to use it,
type the command `h' now.  It brings you to a programmed instruction
sequence.

   To read about expert-level Info commands, type `n' twice.  This
brings you to `Info for Experts', skipping over the `Getting Started'
chapter.

* Menu:

* Getting Started::             Getting started using an Info reader.
* Expert Info::                 Info commands for experts.
* Creating an Info File::       How to make your own Info file.
* Index::                       An index of topics, commands, and variables.















--zz-Info: (info.info.gz)Top, 24 lines --All-------------------------------------------------------------------------


                

If you were to scroll down to the Directory listing line, you'll see that on the left-hand side there's an asterisk, followed by the topic, followed by a double colon and what is inside the info group:

* Directory listing:: ls dir vdir d v dircolors
                

These are the topics covered in this particular node.

If you hit enter at this stage. You should see that the node has changed. The top line of your page will look as follows:

           
File: coreutils.info, Node: Directory listing, Next: Basic operations,Prev: Operating on characters, Up: Top
                

This is similar to the top of the coreutils.info page as described above, but this example includes a previous node, which is "Operating on characters", with the next node being "Basic operations".

Once I've scrolled down (using my arrow keys) to * Directory listing, I may want to go and look at information about the ls command to see what I can do with ls. Again you use the up or down arrow key and scroll to "ls invocation" and hit Enter

Once there you can read the ls info page to see what it tells you about the ls command.

How do you go back to the Directory listing info page? Type u for UP, which should take you to the previous listing.

How do you go from "Directory listing" to "Basic operations", when you're currently at the "Directory listing" node? n will take you to the NEXT node (taking you from the "Directory listing" to "Basic operations").

If you want to go right to the top, in other words, back to the coreutils group, press t for TOP.

You can do searches within info by using the forward slash (/) and then include a pattern, which might be something like

           
/Directory
                

This tells info that you want to look for the pattern Directory. Bingo! We find Directory listing, as it is the first entry that matches the pattern. If you want to use the same pattern to search again, press forward slash followed by enter:

/<ENTER>
                

This will allow you to search for the pattern "Directory" once again. How do you quit info? q will quit info.

If you want to go one page up at a time, then your backspace key will take you one page up at a time.

Finally, to obtain help within info, type '?'. This will get you into the help page for info. To leave help, press CTRL-x-0.

That is essentially how info works. Part of the reason for moving to info rather than man pages is to put everything into texinfo format rather than gz-man format. In the future, much more software will include manual pages in texinfo format, so it's a good idea to learn how the info pages work.

Having covered the info pages, we need to look at man pages since man is the standard on most UNIX and Linux systems. 'man' is short for manual. This is not a sexist operating system. There are no woman pages but we can find out how to make some a little later (to keep man company).} Manual pages are available on every operating system. (If your system administrator hasn't installed them, ask him politely to do so, as no Linux system should be running without man pages.).

The man command is actually run through a program called less, which is like more except it offers more than the more command does.

	Note
	Mark Nudelman, the developer of less, couldn't call it more, since there was already a more command, so it became less. Linux people do have a sense of humor.

To invoke man pages type:

	
man <command>
                

For example, the ls command that we info'ed above,

$ man ls | less
                

Looking at our example above, the manual page on the ls command is run through the less command.

What can you do from within man?

Well, pretty much the things you can do with info. Instead of a menu system, and nodes, we're looking at a single page detailing all the options and switches of the ls command.

If we want to search for a particular pattern we would use forward slash (/) just like we did in info.

For example, we could type

/SEE ALSO
                

This pattern search would take us to the end of the man page to look at the SEE ALSO section.

We could type question mark with a pattern, which does a reverse search for the specified pattern. Remember forward slash does a forward search and question mark does a reverse search.

?NAME
                

This pattern search will reverse search up the man page to look for the pattern NAME.

	Note
	You will notice that I'm not saying look for the string NAME, rather I'm saying look for the pattern NAME. This is because pattern matching is a critically important part of UNIX and Linux and shell scripting. We'll learn more about patterns as we go through the course.

If we want to scroll down one page at a time within the man page (i.e. we've looked at page 1 and we've read and understood it, and we want to go to page 2), then the space bar takes us forward by a page.

Similarly if we want to reverse up the man page, we press b for back, which will scroll backwards through the man page.

How do we get back to our prompt? The 'q' key comes in handy again. 'q' for quit.

man pages are generally broken down into a host of different sections. There's a SYNOPSIS section, a DESCRIPTION section, and a SEE ALSO section. Read through the man page and you will understand the different sections.

If you need help on moving around through the man page, type 'h' for help, which will give you a listing of all the help commands. You will see that it has displayed the help commands NOT for man but for less. Why? Because the pager for man, (pager, the tool that gives you one page at a time instead of just scrolling the man page past you too fast to read), is the less command

We will cover the less command a bit later but you can look it up with the info pages as follows:

info less
                

So 'h' within the man page will show you help on the 'less' command at the same time as displaying the requested manual page.

Sometimes you need to read a man page three or four times before you completely understand it, and of course sometimes you may never understand it! Don't be deterred. That's what separates the kanga's from the roo's.

The whatis database is usually rebuilt on a Linux system at night. The job of the whatis database is to search through every manual page on your system looking for the NAME section within these man pages, classifying them and placing them into a database, to facilitate searching in the future.

The whatis database is useful in that it gives us the ability to quickly look up what a particular command does. So if I asked you to tell me what the nl command does, you could look it up in the man pages or you could look it up in the whatis database.

man nl
                

or

whatis nl
                

The latter method should return with the NAME section from the man page, showing you what the commands job is on the system. It should tell you that nl numbers the lines. Similarly wc counts words, lines and characters for you.

The whatis database is very useful because it allows you to quickly find out, what a particular command on the system does.

If the whatis database is not up-to-date, it is quite simple to update it. Generally though, the updating of the whatis database is a simple automated process. Once a night, the operating system should go about updating the whatis database. Even if the system administrator has installed a whole host of new software on the system, by virtue of the fact that the man pages for that software would be installed at the same time as your new application is installed, your whatis database should pick up those pages and add them to its database each night.

As a side note, updating the whatis database manually is simply a matter of

$ makewhatis -u -w
                

and the whatis database will be updated.

The who command is designed to tell you who's logged on to the system.

If we run the who command without any switches, the left hand column shows the user id. This the user currently logged on to the system. In your case, you might be logged on as root, or perhaps as your user. The second column indicates where you are logged in.

riaan@debian:~> who
riaan    :0           Apr 30 11:13 (console)
riaan    pts/0        Apr 30 11:13
riaan    pts/3        Apr 30 11:14
riaan    pts/4        Apr 30 11:30
riaan    pts/5        Apr 30 13:19
riaan    pts/6        Apr 30 12:07
riaan    pts/7        Apr 30 12:09
riaan@debian:~>
                

So if you look at the who command output, my user riaan is logged in from :0 which is the X console. He's also logged on to

pts/0 and 
pts/1
                

These are pseudo terminals, indicating he's also logged into two pseudo terminals.

The final, third column indicates what time the user logged on.

The who command tells us about users using our system. That's great!

What are the other switches that we can use with who.

who --help 
                

This will show you the various switches that we can use with the who; command. So if we use a:

who -H
                

it prints a heading line for us. The output should look as follows:

$ who -H
NAME             LINE     TIME         FROM
heidi            ttyp1    Nov 27 17:29 (168.210.56.177:S)
mwest            ttyp2    Nov 10 15:04 (apotheosis)
heidi            ttyp4    Nov 11 13:18 (168.210.56.177:S)
                

To view a short listing which is the default listing for the who command:

who -s 
                

Using the -u switch:

who -u
                

will show the users and their process id''s.

In scripts, one can use the same commands as on the command line, including all the switches those commands use. One can run any command and produce standard text output, which one can use. We'll talk about how you can use the output later.

Run the command

$ who -u
root     tty2         Aug  4 10:41   .          2839
riaan    :0           Aug  4 07:53  old         2836 (console)
                

to identify which users are logged into your system and from which processes they are logged on.

This will show how long a terminal has been idle. It will show not only which users are logged on and what process ids they are but also how long that user has been idle. Idle users might have gone out for lunch or they might have left for the day. In default mode, most of these systems don't log you out if you're idle for longer than 10 or 15 minutes. In the old days, most systems were configured to automatically log you out after 15 minutes.

	Note
	On Debian, the -i switch does not add any extra output, it simply prints a message suggesting that you not use -i as it will be removed in future releases. Use -u as above. However the -i switch may work with other brands of Linux.

Okay, so that's the who command. We're going to use these commands later to build a system to monitor our system automatically, because we want to be spending our time doing things we enjoy.

What does the w command do? You could run:

riaan@debian:~> whatis w
w (1)                - Show who is logged on and what they are doing.
riaan@debian:~>
                

riaan@linux:~> w
 21:40:17 up 11:03,  6 users,  load average: 0.30, 0.34, 0.30
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     tty2    	21:40    8.00s    0.06s  0.06s -bash
riaan    :0        	10:38    ?xdm?  58:31   0.04s -:0
riaan    pts/0     10:38   11:01m  0.00s  2.08s kdeinit: kwrited
riaan    pts/3     11:18   10:22m 14:37   2.63s /usr/lib/java/bin/java -mx32m -jar /home/riaan/jedit/4.2pre9/jedit.j
riaan    pts/4     11:28     1:07m  0.21s   0.21s /bin/bash
riaan    pts/5     11:28     0.00s   0.17s   0.03s w
                

which should print some information about the w command.

The w command not only tells us who are logged in, but also what they're doing. Are these users running applications? What actual processes are they running at this time? Perhaps someone's running an application like OpenOffice. w will tell us this.

If you look at the output of this command, it's got a list of headings that are fairly similar to the format of the who command.

Later we'll have a look at modifying the report columns, to get the output into a different format that may be more useful.

One of the reasons for taking you through these commands is that we're going to start writing our first shell scripts using these commands, so it is as well that we understand them now.

The date command is a useful command that can do all sorts of nifty things for us (apart from printing out the date).

It can convert between Unix time, (which is the number of seconds since 1/1/1970 - commonly known as the epoch) and a human readable (normal) date and time.

Conversely, it can also convert back from date time today to the number of seconds that have elapsed since the 1/1/1970. It can format output in a whole variety of ways. Let's look at some examples of the date command.

For that I'm going to do:

info date
                

If you scroll down, you will see a section with examples. Looking at the example:

date +"     "
                

We may now include a string describing the format inside these quotation marks.

In the shell there's a big distinction between double quotes, single quotes (which is another lesson altogether, see Chapter 5), and back quotes - let's not get them confused for now.

Within this double quoted string we can include any number of arguments. What arguments can we include? Each argument starts with a percentage sign.

To display the time, we could use:

%H - -will give us the hours in 24 hour format (0-23).

%M - -will give us the minutes (0-59) of the day
                

If we had the following string:

date +"%H:%M"
                

we will end up with hours and minutes of the day on our system. The result of the above command should be similar to:

15:04
                

But let's say that we want the hours in 12-hour format rather than 24-hour format. We could then replace the %H with a %l. The result would then be:

3:04
                

There's a host of other things that we could do. For example if we are in 12-hour format, 3:04 doesn't indicate whether it is morning or afternoon. Thus we could include %p:

date +"%l:%M %p"
                

This would show us that the time is actually:

3:04 PM
                

rather than 3:04 AM.

That's for time, but what about for the date? What happens if we want to show the date, which is:

24-Nov-2003
                

then, we should in theory be able to create a date string to reflect this format.

A way we can do is this is using the following:

date +"%d-%b-%Y"
                

where %b is a short format for month to produce Nov instead of November.

If we want to combine the date and time:

date +"%d-%b-%Y %l:%M %p"
                

This would give us the full date and time report:

24-Nov-2003  3:04 PM
                

There are a lot of other parameters that you can use within the date command. You can view these by looking at the relevant info page with :

info date
                

We're going to use this command in our script, because in almost every script that you will write you are going to want to know what time the script started, what time the script ended, when it did a particular job within the script, etc.

The final command I want to describe is a command used to send output to the screen: echo.

We've seen so far that we were able to run commands but, as yet, we don't know how to simply output some text to the screen. We may want to print a string to the screen, prior to printing the date.

Something such as:

Today's date is:
24-Nov-2003  3:04 PM
                

We would need some way of echoing that to the screen, wouldn't we?

In order to do this, there is the echo command. echo can be a bit of a nasty gremlin because there are at least two echo commands on your system. One is a shell built-in, and one is an external command and therefore it can be a bit tricky.

We're going to start off with a vanilla case. Later on we will see how we can choose which echo command we want to use.

So by way of an example, we'll use it to format the date command.

echo "Today's date is: "
date +"%d-%b-%Y %l:%M %p"
                

This would be a good time to show you how to create your first shell script. We're going to edit a file and for this you can use any editor^[1]

Open your editor (whatever editor you prefer) and put the following commands in the first lines of the file:

echo "Today's date is: "
date +"%d-%b-%Y %l:%M %p"
                

Save that file as myfirstscript and exit your editor.

You've just created your first shell script. Great! How easy was that? How do you run it to make it actually do its job?

chmod +x myfirstscript
                    

./myfirstscript
                    

Today's date is: 
24-Nov-2003  3:04 PM
                    

This section is going to cover file commands. File commands are commands such as ls (list).

Notice again, how the laziness of the Unix people comes to the fore. They could have had a command called list, but that would have required two more characters (and two more carpals - fingers!) and clearly that was a lot more effort, so we just have the ls command. The ls command shows us a listing of files in a particular directory.

This is an appropriate place to take a detour on our tour de scripting and have a look at file matching and wildcard matching. It may be something that you're familiar with, but let's have a look at it and come back to ls in a moment.

ls *
                        

ls -la ab?a
                        

abba
ab1a
ab_a
ab9a
abca
...	
                        

[abc] 	
                        

[ab]cde	
                        

ham01 	
                        

ham10
                        

ham[0-9][0-9]
                        

ls [abc123] 
                        

a
b
c
1
2
3
                        

[a-z] 
                        

ls [!a-c]*
                        

	
touch {planes,trains,boats,bikes}_{10,1,101,1001}.{bak,bat,zip,tar}
                        

ls p* 
                        

ls b* 
                        

ls bik*
                        

ls b*_??.* 
                        

boats_10.bak
boats_10.bat
boats_10.zip
boats_10.tar
bikes_10.bak
bikes_10.bat
bikes_10.zip
bikes_10.tar
                        

touch {fred,mary,joe,frank,penny}_{williams,wells,ferreira,gammon}.{1,2,3,4,5}
                        

touch {fred,mary,joe,frank,penny}_{williams,wells,ferreira,gammon}.{1,2,3,4,5}
  ------Name----------    ----------Surname-------    -Category-
                        

. 	this file is our current directory
.. 	this file is our previous or parent directory. 
                    

There are many system commands that we can use. We're going to start using these in our shell scripts.

Remember, a shell script is nothing more than a group of Linux commands working together to produce a new command.

In order to build our system that is going to manage our Linux machine, we're going to need to know a little bit about system commands. System commands such as:

df shows the disk free space

du shows the disk usage

fdisk shows the partitioning on our disk

iostat shows the input output status

vmstat shows the virtual memory status

free shows the amount of free memory

We will use these commands, but they are a subset of the commands available to us for monitoring our system.

df -h
                    

du - s 
                    

df - hT
                    

BlackBeauty:/install # fdisk -l

Disk /dev/hda: 10.0 GB, 10056130560 bytes
240 heads, 63 sectors/track, 1299 cylinders
Units = cylinders of 15120 * 512 = 7741440 bytes

   Device 	Boot      Start         End      Blocks   		Id  System
/dev/hda1   *           1          760     	5745568+ 	83 	Linux
/dev/hda2             761        1299     4074840    	5  	Extended
/dev/hda5             761         827      506488+  	82  	Linux swap
/dev/hda6             828        1299     3568288+  	83  	Linux

                    

baloo:/home/riaan# free
	     total       used       free     shared    buffers     cached
Mem:        498572     493308       5264          0      48700     230492
-/+ buffers/cache:     214116     284456
Swap:       706852       8196     698656
                    

baloo:/home/riaan# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0   3792   3508  14208  59348    0    0    32    72 1065   818 11  4 84  2

                    

the iostat command

Finally, the iostat command.

iostat Linux 2.6.4-52-default (debian) 09/02/04 avg-cpu: %user %nice %sys %iowait %idle 2.51 0.03 1.99 0.82 94.64 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn fd0 0.00 0.00 0.00 4 0 hda 3.93 304.75 23.83 3868650 302512 hdd 0.01 0.59 0.00 7524 0

This command gives you information about input and output on your system, and how well (or otherwise) it is performing.

We'll take a closer look at the system performance commands in more detail later.

	Note
	In order to provide you with further information on the performance of your system, you should install the sysstat package. rpm -ivh sysstat.x.y.z-r1.rpm (RedHat system) (see the appendix for other distributions) The iostat command is part of the sysstat package, so if you don't install sysstat, then skip the iostat stuff)

Generally standard input, referred to as "stdin", comes from the keyboard.

When you type stuff, you're typing it on stdin (a standard input terminal). A standard input device, which is usually the keyboard, but Linux also allows you take standard input from a file.

For example:

cat < myfirstscript 
                

This would tell cat to take input from the file myfirstscript instead of from the keyboard (This is of course the same as: cat myfirstscript).

Essentially what this boils down to is that the input for this command (cat) is no longer coming from where it was expecting to come from (the keyboard), but is now coming from a file.

Linux associates the file descriptor 0 with standard input. So, we could have said:

cat 0< myfirstscript
                

which would have produced the same as the previous command.

Why could we leave out the 0?

Since, at the time of creating a process, one standard input is associated with the process.

Standard output, as created at process creating time, goes to the console, your terminal or an X terminal. Exactly where output is sent clearly depends on where the process originated.

Our console or terminal should be the device that is accepting the output. Running the command:

cat file
                

would [con]catenate the file, by default, to our standard output i.e. our console or terminal screen. (Where the process originated.)

We can change this and redirect the output to a file.

Try the following:

ls -al myfirstscript > longlisting
                

This redirects the output not to the screen, but rather to a new file 'longlisting'. The process of redirection using the '>' will create the file 'longlisting' if it was not there. Alternately, if the file 'longlisting' existed, it would remove it, (removing the contents too of course) and put a new file there with the directory listing of "myfirstscript" within it.

How would we see the contents of the file?

cat longlisting
                

This will show the size, the owner, the group and the name of the file myfirstscript inside the file 'longlisting'.

In this example, the output of the ls command has not gone to the standard output (the screen by default), but rather into a file called 'longlisting'.

Linux has got a file descriptor for standard output, which is 1 (similar to the 0 for standard input file descriptor).

The above ls -la example can be rewritten as:

ls -al myfirstscript 1> longlisting
                

which, would do the same thing as leaving out the file descriptor identifier and just using the greater than sign.

In the same way we use our standard input as < (or a 0<), we use a > (or a 1> ) to mean standard output.

Returning to the cat example above, we could type data on the command line that gets sent directly to a file. If the file is not there it will create it and will insert the content we typed on the command line, into the file. This is illustrated below:

$ cat > test
This is the first line.
This is the second line.
This is the final line. < Ctrl-d pressed here >
$ cat test
This is the first line.
This is the second line.
This is the final line.
                

Doing this does not return us to a prompt. Why? What is it waiting for?

It's waiting for us to actually enter our string into a buffer. You should start typing a sentence then another sentence, and another, etc. Each time you type a character, it's getting appended to the file 'newfile'.

When you have finished typing in what you want, press Ctrl-d. The Ctrl-d (^D) character will send an end of file signal to cat thereby returning you to your prompt.

If you list your directory contents using:

ls -al 
                

you should see the file 'newfile'. This file is the one that you've just created on the command line.

cat newfile
                

will show you the contents of 'newfile' displayed onto the standard output.

Now why did all of this work? It worked because cat was taking its input from standard input and was putting its output not to standard out as normal, but was rather redirecting output to the file 'newfile'.

On typing the command and hitting enter, you are not returned to your prompt since the console is expecting input to come from stdin; you type line after line of standard input followed by ^D. The ^D stopped the input, by sending an end-of-file character to the file, hence the file 'newfile' is created.

Question: What do you think tac > newFile will do?

If you decide you want to copy the contents of two files to another file (instead of using the cp command - there is more than one way to skin a cat in Linux) you could do the following:

cat < myfirstscript > mynewscript 
                

Incidentally, this is equivalent to

cp myfirstscript mynewscript
                

Well that's fine and dandy, but what happens if we don't want to delete our longlisting script and want to rather append it to a file that's already there.

Initially we typed:

ls -al myfirstscript > longlisting
                

If you did this a second time, it would overwrite the first file longlisting. How could you append to it? Simply adding two greater than signs, immediately following one another as in the example below, would append output to the file 'longlisting'

ls -al myfirstscript >> longlisting
                

Each time you ran this command, it would not clobber (remove the contents of) the longlisting file but would rather append to it and as a result, the file 'longlisting' would grow in size.

A final note on standard output and standard input is that redirection must be the final command that you execute on the command line. In other words, you can't do any other command after the redirection. We will talk about this in a later section on pipes.

The final component in this dialog of file descriptors is standard error.

Every command could send it's output to one of two places: a) it could be valid output or b) it could be an error message.

It does the same with the errors as it does with the standard output; it sends them directly to your terminal screen.

If you run the command (as your user):

find / -name "*" -print
                

you would find that find would find a load of things, but it would also report a lot of errors in the form of 'Permissions denied. . .'.

Perhaps we're not interested in 'Permission denied...' - we may wish to discard these messages (as root, no error messages would be returned).

If we ran the command, we could put standard error into a file (remember standard error by default is going to the console; the same place as stdout). In this case I'm going to put it into a special file on the Linux system called /dev/null.

/dev/null is similar to the "Recycle Bin" under Windows except it's a waste paper basket with a point of no return - the Linux black hole! Once information has gone into /dev/null, it's gone forever.

Initially, I'm going to put any errors that the find command generates into /dev/null, because I'm not interested in them.

We saw that standard input was file descriptor 0, the standard output was file descriptor was 1, so no prizes for guessing that standard error has file descriptor 2.

Thus, the command

find / -name "*" -print 2> /dev/null
                

discards any errors that are generated by the find command. They're not going to pollute our console with all sorts of stuff that we're not interested in.

	Note
	Notice there is no space between the 2 and the >

We could do this with any command, we could for example say:

ls -al 2> myerror
                

Which would redirect all the error messages into a file called "myerror".

To recap we can use either:

<	OR	0<	for standard input
>	OR	1>	 for standard output
but for standard error I have to use 2> 
                

It's not optional to leave off the number two (2). Leaving it off would mean that the standard output would go to "myerror", including a 2 means standard error.

In the listing case of:

ls -al  2> myerror 
                

puts any errors into a file called 'myerror'.

If we wanted to keep all those error messages instead of using a single greater than sign, we would use double greater than signs.

By using a single greater than sign, it would clobber the file 'myerror' if it exists, no different to standard output. By using a double greater than sign, it will append to the contents of the file called myerror.

ls -al 2>> myerror 
                

Thus the contents of 'myerror' would not be lost.

With our new found knowledge, let's try and do a couple of things with the find command. Using the find command, I want to completely ignore all the error messages and I want to keep any valid output in a file. This could be done with:

find / -name "*" -print 2> /dev/null > MyValidOutput
                

This discards any errors, and retains the good output in the file "MyValidOutput". The order of the redirection signs is unimportant. Irrespective of whether standard output or standard error is written first, the command will produce the correct results.

Thus, we could've rewritten that command as:

find / -name "*" -print >MyValidOutput 2>/dev/null
                

Finally I could've appended the output to existing files with:

find / -name "*" -print >> output 2>> /dev/null
                

Clearly appending to /dev/null makes no sense, but this serves to illustrate the point that output and errors can both be appended to a file.

What happens if we want to take our standard output and put it into the same file as standard error? What we can do is this:

find / -name "*" -print 2> samefile 1>&2
                

This means that standard error goes into a file called samefile and standard output goes to the same place as file descriptor 2 which is the file called samefile.

Similarly we could've combined the output by doing:

find / -name "*" -print 1> file 2>&1
                

This captures the output of both standard output and standard error into the same file.

Clearly, we could've appended to a particular file instead of overwriting it.

find / -name "*" -print 1>> file 2>>&1
                

Up to now, we've seen that we can run any command and we can redirect its output into a particular file using our redirection characters (>, <, >>, 2> or 2>>).

It would be good if we could redirect the output of one command into the input of another. Potentially we may want the output of the next command to become the input of yet another command and so forth. We could repeat this process over and over until we achieve the desired output.

In Linux, this is affected using the pipe character, (which is a vertical bar '|'). An example is:

ls -la | less 	
                

This would pass the output of the ls -al command to the input of the less command.

The effect would be to page your output one page at a time, rather than scrolling it to the standard output all in one go - too fast for you to be able to read, unless of course you are Steve Austin!.

What makes the pipe command powerful in Linux, is that you can use it over and over again.

We could type:

ls -l | grep myfirstfile | less
                

Instead of grep's standard input coming from a file (or keyboard) it now comes from the ls command. The grep command is searching for a pattern (not a string) that matches myfirstfile. The output of the grep command becomes the input of the less command. Less could take its input from a keyboard, or from a file, but in this case it's taken its input from the command grep.

How many of these pipes can we have on a single line? As many as we want! We will see this and how it's used to good effect in our shell scripts from the next chapter onwards.

If pipes were not available, we may have to archive the above command in two or more steps. However, with pipes, this task is simplified (speeded up).

If we take a step back to run the who command:

who | grep <your user name> 
                

	Note
	The < and > here don't mean redirection!

We will see only the people that are logged on as your user (hopefully that is only you!!). Perhaps you want to only see people who are logged on to pseudo terminals, then:

who | grep pts
                

which would tell us only the usernames of the people logged on to pseudo terminals.

In these examples we are using the output of the who command as the input to the grep command.

Additionally we could redirect the output of this command to a file called outfile:

who | grep pts > outfile
                

This would produce the file 'outfile' containing only those users who have logged onto pseudo terminals. That's nifty.

We will see this put to good effect as we start scripting, building very complex filters where we use the output of one command to be the input of the next, and the output of that to be the input of the next.