Fancier Redirection

The shell lets you do a lot of fancy redirection without much work. I'll show many examples as we go along, but here are a couple you'll use frequently.

Q: How do I put stdout and stderr into the same file?

# first, let's make sure X exists and Y doesn't
> X
rm Y

# now some experiments
# try these on the command line
ls X Y
ls X Y > OUT
ls X Y 2> ERR
ls X Y &> BOTH
ls X Y > AMANDOI 2>&1    # Sometimes you see this form instead.  It's the same as &>.  
                         # We'll explain why another time.
                         # Any Romanians reading this?

If you didn't do the last commands by recalling and editing earlier commands,
I suggest going back and trying that out. Make using and editing the command history a habit now, not later.

Q: What if I don't want to overwrite files? What if I want to tack onto the end of a file instead?

#!/bin/bash
# > overwrites, >> appends

echo hello > /tmp/$$
echo world >> /tmp/$$
cat /tmp/$$

rm -f /tmp/$$

This script logs network activity, once an hour.

#!/bin/bash
while true
do
    date
    ruptime
    sleep $[ 60 * 60 ]      # an hour's worth of seconds
    echo                    # blank line between each timepoint
done >> /tmp/LOG

stderr

I've run programs before that huff and puff and produce ... an error message. If the output is going into some archive, or is input into another program, I usually don't find out about the error until too late, and I think, "Gee. I wish it had beeped or something."

In well-designed programs, errors are announced through a separate channel from normal output.

On Linux, default output goes to "standard output" (stdout), while errors go to "standard error" (stderr). These are two separate channels. Both go to the screen by default, but you can redirect them separately.

Try this. First, snarf-and-barf this script into the file hello:

#!/bin/sh

echo hello, world
ls A_NONEXISTENT_FILE

Run hello and you'll see this:

hello, world
ls: A_NONEXISTENT_FILE: No such file or directory

Next, use the command-line-editing keys to redirect the output into a file. What happens?

hello > OUT

You can capture the errors with the construction "2>"

hello 2> ERR

You can redirect both separately.

hello > OUT 2> ERR

Try this and think about what happens.

hello > OUT 2> OUT

Try redirecting either channel to the bit-bucket, /dev/null. Then make up experiments of your own.

Oh, Happy Thanksgiving!

Redirection Inside Your Scripts

Naturally, you can use redirection inside your scripts, too.

#!/bin/bash

# The shell lets you redirect output

echo hello, world > /tmp/$$         # "$$" is the process id of this script, just like Perl
cat /tmp/$$

rm -f /tmp/$$                       # when you make temporary files, clean up after yourself

Sometimes, when they're just starting out, people get confused about the difference between output produced by a script and output produced by commands inside a script. Snarf-n-barf this into a script:

#!/bin/bash
echo "the 'hello world' step comes next"
echo hello, world > X
echo "the 'hello world' step is now done"

Run it, then think about what happened for a second.

You can now stick debugging statements into your scripts. (I'm not just trying to show you features of the shell, I'm trying to help you learn how to program in it.)

But now that you're thinking carefully, think about this: there is a Catch-22 looming.

Suppose you're developing a script by letting the ouput go to your screen. Now you redirect output to a file. Something mysterious happens and the output file's suddenly full of garbage instead of what you expect. (This has happened to you with other programming languages; it will happen to you in the shell, too.)

How do you debug it? Where's your debugging output? In the same pile of garbage.

This sets me up for what I'll talk about tomorrow.

Learning by Playing Around, and a Psychological Time Bomb

I learn the shell by playing around.

Using the up-arrow key to recall commands, plus the backspace and left-and right-arrow keys to edit them, you can do a lot of quick, little experiments.

Try it now.

Start as usual:

./hello > 1
cat 1

What happens if the ">" symbol is in front of the command instead of after it?

> 2 ./hello
cat 2

What happens if there's no command at all?

> 3
cat 3

What happens if there's already something in the file you're writing to?

echo Your mother wears army boots. > 4
./hello > 4
cat 4

What happens if you accidentally write to yourself?

./hello > hello
cat hello

This last experiment shows you that the shell starts redirection before it runs the command. The shell opens up the output file, hello, cleans it out in preparation for filling it with the output of the the command "./hello," and then, uh, ... finds it's painted itself into a corner. The command you've told it to run is now an empty file.

You will do this to yourself eventually. Instead of being mystified, you'll remember this, smack your forehead, and say, "Dang!"

I'm setting a psychological time-bomb.

Redirection

You ran your "hello, world" script like this:

 ./hello 

Output to a file, instead of the screen, is trivial.

 ./hello > foo 

What could be easier? Think of ">" as an arrowhead that points the output into a file. If the file doesn't exist, it's created.

The easiest way to learn the shell is to bounce back and forth between interactive and non-interactive uses. We'll start doing that now. First, create a "hello, world" script and run it.

echo "echo hello, world" > hello
chmod +x hello 
./hello 

Press the up-arrow key. The ./hello command will reappear, with your cursor at the end of the line, ready for you to type. Append the characters " > foo" and press ENTER. Then look to see what was put in the file foo.

./hello > foo
cat foo

Next, recall that command (./hello > foo), and use the backspace key to change foo to bar. Press ENTER and presto! now the output goes to bar.

Sending output to a file instead of to the screen is called "redirection." (The key to stage magic is misdirection, the key to shell magic, redirection.)

This is a common, and fruitful, way to develop shell scripts. While you're developing, let output go to the screen. When it's finally the way you want it, recall the command and redirect to a file to capture the output.

November 21 is World Hello Day, so rerun your program a few times, redirecting or not as fits your fancy.

Quick tour: Output

Every language has strengths and weaknesses.

The shell's loops and conditionals, for constructing algorithms, are pretty
conventional. They'll do, but they're nothing special

The shell's data structures are just terrible. All it has are scalars and
arrays, just like FORTRAN. (Anyone out there remember FORTRAN?) Actually,
FORTRAN had multi-dimensional arrays. The shell doesn't even have that.

The shell does handle input and output of textfiles particularly well.

This week, I'll show you how to do basic output.