Shell Shul

Using the Return Value

2006-01-06T19:26:00.000-08:00

Functions, like programs, succeed or fail. That's what $? is about. (At the shell level, $? tells whether the last command succeeded or failed. Here, as far as the shell is concerned, the function is just like a command.)

This means that if you have a function to check out and build code

get-N-make() {
 svn co $*
 (
  cd $_
  make
 )
} &> /dev/null

You can call it like this:


get-N-make -Dyesterday mycode ||
 die "something in the checkout or build failed"

(Notice that you can redirect the output of a function as part of its definition.

The Return of the Value

2006-01-05T20:00:00.000-08:00

Okay, so the only way to return a value from a function is with a side effect. What the heck do you do to return values?

One option is to have the function create real output and capture it.

#!/bin/bash

date_diff() {
 t2=$(date -d "$1" +%s)
 t1=$(date -d "$2" +%s)
 echo $((t1-t2))
}

secs=$(date_diff "Dec 25" "now")
echo "$secs more shopping seconds until Christmas"

This works, but costs the execution of a subshell each time you call the function. A more efficient approach is to capture output in a global, and use a naming convention to keep from accidentally stepping on a some other variable with the same name.

I've seen a handful, but I like this:

date_diff() {
 t2=$(date -d "$1" +%s)
 t1=$(date -d "$2" +%s)
 _date_diff=$((t1-t2))
}

date_diff 
echo "$_date_diff more shopping seconds until Christmas"

Try using the name of the function, preceded by an underscore, to hold the return value of the latest call.

Functions Don't Return Values

2006-01-04T20:00:00.000-08:00

Functions really don't return values. Really. That's hard to get used to. The shell does have a return() call, but it doesn't do what you'd guess.

#!/bin/bash

jo_mamma() {
 return 69
}

fezmo=jo_mamma
echo fezmo is $fezmo

# No?  Well, what if we reason by analogy and use $() to call the function?

fezmo=$(jo_mamma)
echo fezmo is $fezmo

# Still no.  But look at this:

jo_mamma
echo after call, status is $?

return() returns from a function call and sets the predefined variable, $? -- the "exit status."

Even this isn't as useful as you'd hope. Substitute a string for the "69" in the example to see what happens.

Functions Can Set Globals

2006-01-03T20:00:00.000-08:00

The title says it all. Shell variables are global by default.

#!/bin/bash

fezmo=6
jo_mamma() {
 fezmo=9
}

echo before call fezmo is $fezmo
jo_mamma
echo after call fezmo is $fezmo

That means that you have to keep track of the names of every variable you set in a function. Bummer. Shell scripts aren't typically very long, so this usually isn't a big deal. Still, using local inside functions is a good habit to get into.


#!/bin/bash

fezmo=6
jo_mamma() {
 local fezmo=9
}

echo before call fezmo is $fezmo
jo_mamma
echo after call fezmo is $fezmo

(Don't just read these examples and nod your head. Use snarf-N-barf to try them.)

Design by Analogy

2006-01-02T20:00:00.000-08:00

Here's a function that's in almost every shell script I write:

die() {
 echo $* >&2 ; exit -1;
}

The syntax is straightforward and obvious. $* is the list of
arguments passed to die() by the caller. The call


die "usage: foobar mumble frabitz"

spits a usage message to stderr and then ends the program.

Why call it "die()"? Because that's what the corresponding Perl
function is called.

Designing things to mimic one another is a good habit. To link
bar to foo, do you say ln foo bar or
ln bar foo? It's easier to remember once you know that cp,
mv, and ln all use the same argument order. Similarly, once you
know rm -i, you're more likely to guess that cp -i,
mv -i, and ln -i all work, too.

The drawback is that you need to remember that the copy isn't exactly the same
as the original. For example, Perl's die() spits out additional
information unless the argument string ends with a carriage return. In
practice, I don't find this a disadvantage. If you do, you could name yours
s.die() or something.

For example, $* in a function is all the args to a function; in a
shell script, it's all the args to the script. $1, $2, etc.
are the individual args to a script. Guess what $1 is inside a
function.

But don't just guess. Try it. And try writing a warn() function to
mimic Perl's. If you don't know Perl, try writing a shell analogue to some
other useful thing you know from another language.

Functions

2006-01-01T20:00:00.000-08:00

Well, I'm back from the holidays. Hope yours were fun. This week: shell functions.

Macros, subroutines, procedures, functions ..., a programming language needs some way a way to encapsulate repeated actions.

The shell calls 'em functions. An academic might say a function should return a value, while a subroutine works through its side-effects. Unfortunately, shell functions mostly work through their side effects. Oh well.

They're still very useful little things, and you can use them a lot, to good effect.

Grow a Command, Then Execute It

2005-12-16T20:00:00.000-08:00

Let's make a copy of all the files under /etc/ that contain references to httpd, the Apache http daemon.

First, a directory to hold the copies:

mkdir /tmp/Apachefiles

Next, we'll construct the commands, on the fly, by growing a pipeline. Follow along with me by executing these in a terminal window. For each step, just recall the previous line and edit it.


find /etc/ # list all the files under /etc.
find /etc/ | xargs grep -l httpd # look for all those files that contain the string httpd
{ find /etc/ | xargs grep -l httpd; } 2>/dev/null # get rid of annoying warnings
# now transform the list into a series of commands
{ find /etc/ | xargs grep -l httpd; } 2>/dev/null |
 perl -lane 'print "cp $_ /tmp/Apachefiles/"'

Okay, these look good. (I've left out all the mistakes I made while developing the pipeline, because I know you would never make any. I do, so command history is my friend.)

Now, how about executing it? We could redirect the output into a file, mark the file as executable, and then execute it as a script.

Or we could just add one step to the pipeline, like this:

{ find /etc/ | xargs grep -l httpd; } 2>/dev/null |
 perl -lane 'print "cp $_ /tmp/Apachefiles/"' | bash

I frequently grow a set of commands like this, on the fly, then when I have them right, pipe them into a subshell, which will execute them, one by one.

Command Substitution for Counted Loops

2005-12-15T20:00:00.000-08:00

Here's a loop.


for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
do
    echo $i
done

Here's another way to do the same thing:


for i in $(seq 20)
do
    echo $i
done

Just the command seq 20 by itself spits out the sequence 1 .. 20 . (Try it.)

The expression $(command) says "Run command in a subshell,
substitute its output for the expression, and then run the resulting command in the current shell."

This is called command substitution

Since "$" marks variables, think of $(command) as "The value of the temporary variable containing the output of the subshell '(command)'"

You have a mystery version of foo.c and you want to know what archival version it's closest to?


for i in $(seq 10 -1 1)
do
    cvs co -p -r1.$i foo.c | diff - foo.c | wc -l
done

The version with the smallest diff is the version you're looking for.

If you have a very large number of versions, you can pipe the loop's output through less. Alternatively, if you're using a terminal emulator, you can use the scroll bar to scroll back up the screen, or, in kterm, the convenient Shift PgUp.

Get Scope With Subshells

2005-12-14T20:00:00.000-08:00

You can tell the shell to execute a few commands in a subshell by surrounding them with parens:

pwd; ( cd /tmp; pwd }; pwd

You change directories in the subshell, but once you exit that subshell, the original shell hasn't gone anywhere.

This is particularly useful for scripts where you want to go somewhere and do something, temporarily. My scripts are chock-a-block full of code like this:


cvs -Q co modulename
(
    cd modulename
    make
)

Don't confuse this with curly braces, which group commands in the current
shell:

pwd; { cd /tmp; pwd; }; pwd

The two kinds of grouping are for two different things. Curlies are good when you want to do I/O redirection of groups of commands. Parens are good when you want to insulate a parent shell from a bunch of temporary changes, without having to save and restore context by hand.

Use pstree to See Your Tree of Subshells

2005-12-13T20:00:00.000-08:00

Execute the command csh. The prompt changes! You are now running an older shell, csh, that has a different syntax. Try, for example, typing in a comment, like this

# this is a comment

This new shell is a subshell of the bash you were talking to a minute ago. Exit the subshell, and you'll see you're back in the parent shell. You can see
this with the ps command.


ps -o pid,ppid,args
csh
ps -o pid,ppid,args
exit
ps -o pid,ppid,args

Each running program is a process, with an identifying number, or Process ID. If that process is a subprocess of some other program, then the ps command shown above also shows the Parent Process ID.

You should be able to trace out what's a child of what in the listings above, pretty easily.

Actually, in Linux, every process is the child of another process, except for the very first process, which is called init. Look:


pstree -A -p | less

The numbers in parens are PIDs. Try invoking some subshells and watch how the process tree changes.

You Can Invoke the Shell By Hand

2005-12-12T12:57:00.000-08:00

You invoke a shell the same way you invoke any other program: type its name.


bash

You get back a prompt, but now this is a second shell, a subshell under the previous shell. The shell you were running a minute ago is now patiently waiting for this subshell to finish, so it can put up another prompt and let you issue some other command.

See for yourself? Type exit. You'll exit this subshell, and return to the shell a level up. You're back in the shell you started from.

Want more convincing? Type history. That's your current shell. Now execute a subshell, and look at the history again:


bash
history

Finally, exit the subshell, and look at the history again:


exit
history

You get two different histories because it's two different shells.

If you execute commands, then spawn a subshell, and then try to recall the commands you were just executing before you entered the subshell, you won't find them.

It's important that you have a firm grasp on the idea of subshells before you continue, so I'll return to this, tomorrow.

Subshells

2005-12-11T20:00:00.000-08:00

The shell is just a program. It's not wired into the kernel. Many programmers who've been in the business for a while have written a shell, for their own amusement. If you haven't, you can imagine the pseudocode.


while (1) {
    put up a prompt
    read the line the user types in
    parse the line
    execute any commands you find
}

Thinking of the shell as just another program, no different from ls or date or cc, is a big help in dealing with the shell.

This week, I'll explore that a little.

Ancient History? RTFM

2005-12-09T08:00:00.000-08:00

Just as our bodies have little, atavistic features that recapitulate our phylogeny, so does the shell. Back when dinosaurs roamed the earth, there was a shell called the "C shell" (cute name, eh?), that had a clumsy-but-complex history mechanism built around the ! key. Bash retains that mechanism, alongside its more modern one.

About the only useful thing left from it is !!, which means "redo the last command."

If you want to read about it, though, see the man page.

man bash

Impressive, huh?

I don't just mean the HISTORY EXPANSION section, which is 120 lines, starting around line 3172 on my machine. I mean the fact that the man page is over 3000 lines long. (Mine's close to 5000). There's a lot there.

It's usually easier to read a man page this big in a text editor, so stash a copy somewhere. Send it through col first, to remove funny characters.

man bash | col -b > /tmp/bash.man
vi /tmp/bash.man +/"^HISTORY EXPANSION"

On tomorrow, the seventh day, you should rest. The day after, I'll start to grade insensibly into more advanced topics: I'll talk a bit about processes, and how to handle them in the shell.

Command-line Comments

2005-12-08T08:00:00.000-08:00

Shell comments start with #. Important for scripts, but useless on the command-line, right? Wrong.

Suppose you've typed most of a line, but you need to stop and check on something -- how some file or command is spelled, for example.

Just go to the beginning of the line (Home), type a #, and press Enter. What happens? Nothing. It's a comment.

But it goes into the history. When you've finished checking on whatever it was, recall the command, make any needed modifications, delete the #, and press Enter again. Voila!

If you're using vi-style editing, use Esc to switch to command mode, then type the command #. Bash will insert a # at the beginning of the line and give you a new prompt for the next command.

In otherwords,

emacs: Home # Enter
vi: Escape #

History at Full Speed

2005-12-07T08:00:00.000-08:00

If the command-line editor is really a full-featured history editor, then history should be searchable.

Is this possible? Yes sirree.

In emacs mode, type ^R, then start typing the string you're looking for. You'll get the normal, emacs incremental search. If you find the string you want, but it's not the command you want, press ^R again to get the next instance.

In vi mode, use vi's search command, /. Find the next command containing that string with n.

If there's a command you issued a few minutes ago and you want to bring it back, you don't have to type up-arrow after up-arrow to find it. You can recall it by just typing in a few characters from the command and searching.

vi doesn't have incremental search, but it turns out that some of the emacs-mode commands, like ^R, still work in vi-mode, so it's often worth trying them.

No, this isn't logical. Oh, well.

Happy Pearl Harbor Day!

Editor Choice

2005-12-06T08:00:00.000-08:00

If you prefer vi to emacs, do this: set -o vi (If you're just experimenting, set -o emacs gets you back to emacs mode.)

Me, I use vi-mode in the shell. I've been using vi for so long, it's in my firmware. I don't have to think about how to edit files with vi, it just comes out of my fingers. Editing command lines comes just as naturally.

Do remember that vi is a modal editor. When you begin to type, you're in insert mode. To get to command mode, type Esc. Once you're in command mode, the familiar vi commands just work. End of line? $ Beginning of line? ^. Back two words? 2b. Change the word under the cursor? cw.

As soon as you type Enter, the current line, as you see it, is handed to the shell and executed. You're put back in insert-mode with a new, empty line.

If you prefer TECO or WordStar or some other editor, you're out of luck. emacs and vi come standard, but nothing else. Sorry.

On the other hand, if you want to learn vi or emacs (because, say, you're coming from another operating system), then the shell is an excellent teaching tool. Each history-editing command you learn will work in the editor, and each text-editing command you learn will work on the command-line.

As one last trivium, the old line editor, ex, feels just like vi-mode history editing. Some day, you'll be stuck using a line editor to recover a system. Instead of being nervous, you'll find yourself completely comfortable. It'll feel just like editing your history.

After you're comfortable with vi-mode history editing, go try ex; you'll see what I mean.

Editing History

2005-12-05T08:00:00.000-08:00

The Moving Finger writes; and, having writ,
Moves on: - Omar Khayyam.

You've used arrow keys and the backspace/delete keys to edit command lines. They're just the tip of the iceberg.

To give you access to the most powerful command-editing facilities, I have to give you a new viewpoint. Let me see if I can take you there.

Imagine all the commands you've issued as a file. history shows you this file, with line numbers.

Next, imagine that when you type, you're actually editing that file. There's an empty line at the bottom of the file; as you type, you're typing into that empty line. When you press the carriage return, you move on to the next empty line.

You're in a text editor with a screen so narrow it only displays a single line.

As you press the arrow keys, you're navigating through a file.

But since you are in an editor, you can edit the lines you see. Like most editors you've used, it understands the arrow keys, backspace, and delete, but it understands a lot more -- the Home and End keys for example.

How powerful an editor is it? I'll cut to the chase: it's emacs. Want to go to the beginning of the line? ^A (control-A). End of line? ^E. Delete to end of line? ^K.

Most familiar emacs commands will work.

One-line emacs.

History

2005-12-04T08:00:00.000-08:00

God alone knows the future, but only an historian can alter the past.
- Ambrose Bierce

This week, we're going to give an introduction to history. History is, in turn, the introduction to everything else.

Remember the Pipe

2005-12-02T08:00:00.000-08:00

Sometimes, you want to save output. For that, the shell gives you redirection.

Sometimes, though, you want to save the pipeline that you grew to get the
output.

You could recall it, write it down, and then type it into a file, but there's
an easier way.

After you run the pipeline, make your next command "fc."

Try this:

ls | wc -l
fc

(You've run other things since the pipeline? Recall it with command history, rerun it, then type fc)

Once fc puts the pipeline into a text editor, you can write it out to a file. Presto! A shell script.

And as long as you're in there, you could even take your time, add shebang lines and comments, and change the layout. The shell gives you command-line history and pipes, for quick-and-dirty script development, and then gives you fc, to capture what works and clean it up.

The default editor is vi, but you can make it whatever you want.

EDITOR=kate
ls | wc -l
fc

Notice that when you exit the editor, the command is rerun. fc stands for "fix command". Instead of editing a long command line with arrow keys, you can edit it with your favorite text editor.

Being able to write to a file and create a shell script, once you've edited it, is just a side-effect of the shell's clean design.

And now that we're talking about command history, we'll make that next week's introductory topic.

Pipelines

2005-12-01T08:00:00.000-08:00

You can add as many elements to a pipeline as you want.

ls -l /dev/
ls -l /dev/ | grep -v ,
ls -l /dev/ | grep -v , | grep -v -- '->'
ls -l /dev/ | grep -v , | grep -v -- '->' | grep -v ' 0'

This is an everyday, and brilliant, use of command history: exploratory
programming.

Issue a command, look at the output. Filter that output through something else to get closer to what you want. Look at the result of that, and filter it again. When you finally get what you want, redirect the output to save it.

The metaphor of pipes and filters suffuses shell programming. To make your own programs (including your shell scripts) fit in, all you have to do is design them to take input from standard in, and spit output to standard out.

Unless there are multiple inputs or outputs, don't write to files or read from them; use redirection for that.

Suppose, for example, you write a program that collects all the #ifdef variables into a file, one per line. One design choice is to have it called like this:

ifdefvars CFILE VARFILE

Worse, you could even design it to be called like this:

ifdefvars CFILE

and have it automatically generate CFILE.VARS.

If, instead, you have it read from stdin and write to stdout, so it's called like this

ifdefvars < CFILE > VARS

you can pop together a pipeline, at a moment's notice, to scan your code for mistyped variables:

find . -name '*.[ch]' | xargs cat | ifdefvars | sort | uniq -c | awk '$1 == 1'

(Don't know what all these commands do? Type the first one in and run it. Next, use command history to recall it, add the next stage in the pipeline and watch what changes.)

Pipes

2005-11-30T08:00:00.000-08:00

You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat. - Albert Einstein

The shell's biggest innovation is pipes.

To hook stdout of one program to stdin of another, you just connect the two programs with a pipe symbol: '|'.

How many files do I have?

ls | wc -l

You can think of it as shorthand for this:

ls > TMPFILE; wc -l TMPFILE; rm TMPFILE

The only thing is, there is no temporary file. In a pipeline, the output from one command really is hooked up to the input of the next, and both processes run simultaneously. It's magic.

If the first process is too slow, and the pipe gets empty, the second process just sits and waits for the first one to send it more stuff. If the first process is too fast, and the pipe starts to clog, it just blocks until the second sucks enough out of the pipe to make it worth starting up again.

For me, the ability to pipe into wc -l, alone, is invaluable.

How many processes are running?

ls -d /proc/[0-9]* | wc -l

How many .c and .h files are there in my source tree?

find . -name '*.[ch]' | wc -l

How many files have we had to change since the "FINAL" release?

cvs -nq update -rFINAL | wc -l

Next time you're asking "How many ...?", remember to answer it with a pipe.

Here Documents

2005-11-29T08:00:00.000-08:00

Ouput has '>' and '>>'. Now that you've seen '<', what about '<<'?

cat <<FOO
hello world
FOO

This is a here document, which says "Take input from from 'here to FOO'"

Actually, FOO can be a little visually confusing, because I often use FOO and foo for variable or file names, so I usually use a marker for "End of Input" or "End of File" instead, like __EOF__ or __EOI__. These stand out and I don't use them for other things.

If you want to imbed the contents of a file into a script, a here document is a good way to do it.

As usual, you should try out moving around the redirects, and use command history to do it.

There's even a '<<<'

H='hello, world'
cat <<< $H

It's sort of a "right here" document, which provides input from the current line.

Redirecting Standard Input

2005-11-28T08:00:00.000-08:00

Start like this:

```
cat
hello, world
hello, world
^D
```

cat

```
cat
cat > FOO
hello, world
^D
```

"hello, world"

FOO

```
cat FOO
hello, world
```

'>'

'<'

```
cat < FOO
```

'<'

FOO

cat

Part of learning the shell is losing your fear of making mistakes. The sooner you make your first 5000 mistakes, the better. What do these do?

cat < FOO > BAR
< FOO > BAR cat
< cat > BAR
BAR < > cat

Input

2005-11-27T08:00:00.000-08:00

We've talked about output, but without input, I/O would just be "O" -- a big
nothing of a subject. This week, I'll sketch some input redirection.

Fancier Redirection

2005-11-25T08:00:00.000-08:00

The shell lets you do a lot of fancy redirection without much work. I'll show many examples as we go along, but here are a couple you'll use frequently.

Q: How do I put stdout and stderr into the same file?

# first, let's make sure X exists and Y doesn't
> X
rm Y

# now some experiments
# try these on the command line
ls X Y
ls X Y > OUT
ls X Y 2> ERR
ls X Y &> BOTH
ls X Y > AMANDOI 2>&1    # Sometimes you see this form instead.  It's the same as &>.  
                         # We'll explain why another time.
                         # Any Romanians reading this?

If you didn't do the last commands by recalling and editing earlier commands,
I suggest going back and trying that out. Make using and editing the command history a habit now, not later.

Q: What if I don't want to overwrite files? What if I want to tack onto the end of a file instead?

#!/bin/bash
# > overwrites, >> appends

echo hello > /tmp/$$
echo world >> /tmp/$$
cat /tmp/$$

rm -f /tmp/$$

This script logs network activity, once an hour.

#!/bin/bash
while true
do
    date
    ruptime
    sleep $[ 60 * 60 ]      # an hour's worth of seconds
    echo                    # blank line between each timepoint
done >> /tmp/LOG

stderr

2005-11-24T08:00:00.000-08:00

I've run programs before that huff and puff and produce ... an error message. If the output is going into some archive, or is input into another program, I usually don't find out about the error until too late, and I think, "Gee. I wish it had beeped or something."

In well-designed programs, errors are announced through a separate channel from normal output.

On Linux, default output goes to "standard output" (stdout), while errors go to "standard error" (stderr). These are two separate channels. Both go to the screen by default, but you can redirect them separately.

Try this. First, snarf-and-barf this script into the file hello:

#!/bin/sh

echo hello, world
ls A_NONEXISTENT_FILE

Run hello and you'll see this:

hello, world
ls: A_NONEXISTENT_FILE: No such file or directory

Next, use the command-line-editing keys to redirect the output into a file. What happens?

hello > OUT

You can capture the errors with the construction "2>"

hello 2> ERR

You can redirect both separately.

hello > OUT 2> ERR

Try this and think about what happens.

hello > OUT 2> OUT

Try redirecting either channel to the bit-bucket, /dev/null. Then make up experiments of your own.

Oh, Happy Thanksgiving!

Redirection Inside Your Scripts

2005-11-23T08:00:00.000-08:00

Naturally, you can use redirection inside your scripts, too.

#!/bin/bash

# The shell lets you redirect output

echo hello, world > /tmp/$$         # "$$" is the process id of this script, just like Perl
cat /tmp/$$

rm -f /tmp/$$                       # when you make temporary files, clean up after yourself

Sometimes, when they're just starting out, people get confused about the difference between output produced by a script and output produced by commands inside a script. Snarf-n-barf this into a script:

#!/bin/bash
echo "the 'hello world' step comes next"
echo hello, world > X
echo "the 'hello world' step is now done"

Run it, then think about what happened for a second.

You can now stick debugging statements into your scripts. (I'm not just trying to show you features of the shell, I'm trying to help you learn how to program in it.)

But now that you're thinking carefully, think about this: there is a Catch-22 looming.

Suppose you're developing a script by letting the ouput go to your screen. Now you redirect output to a file. Something mysterious happens and the output file's suddenly full of garbage instead of what you expect. (This has happened to you with other programming languages; it will happen to you in the shell, too.)

How do you debug it? Where's your debugging output? In the same pile of garbage.

This sets me up for what I'll talk about tomorrow.

Learning by Playing Around, and a Psychological Time Bomb

2005-11-22T20:00:00.000-08:00

I learn the shell by playing around.

Using the up-arrow key to recall commands, plus the backspace and left-and right-arrow keys to edit them, you can do a lot of quick, little experiments.

Try it now.

Start as usual:

./hello > 1
cat 1

What happens if the ">" symbol is in front of the command instead of after it?

> 2 ./hello
cat 2

What happens if there's no command at all?

> 3
cat 3

What happens if there's already something in the file you're writing to?

echo Your mother wears army boots. > 4
./hello > 4
cat 4

What happens if you accidentally write to yourself?

./hello > hello
cat hello

This last experiment shows you that the shell starts redirection before it runs the command. The shell opens up the output file, hello, cleans it out in preparation for filling it with the output of the the command "./hello," and then, uh, ... finds it's painted itself into a corner. The command you've told it to run is now an empty file.

You will do this to yourself eventually. Instead of being mystified, you'll remember this, smack your forehead, and say, "Dang!"

I'm setting a psychological time-bomb.

Redirection

2005-11-21T08:00:00.000-08:00

You ran your "hello, world" script like this:

 ./hello

Output to a file, instead of the screen, is trivial.

 ./hello > foo

What could be easier? Think of ">" as an arrowhead that points the output into a file. If the file doesn't exist, it's created.

The easiest way to learn the shell is to bounce back and forth between interactive and non-interactive uses. We'll start doing that now. First, create a "hello, world" script and run it.


echo "echo hello, world" > hello
chmod +x hello 
./hello

Press the up-arrow key. The ./hello command will reappear, with your cursor at the end of the line, ready for you to type. Append the characters " > foo" and press ENTER. Then look to see what was put in the file foo.


./hello > foo
cat foo

Next, recall that command (./hello > foo), and use the backspace key to change foo to bar. Press ENTER and presto! now the output goes to bar.

Sending output to a file instead of to the screen is called "redirection." (The key to stage magic is misdirection, the key to shell magic, redirection.)

This is a common, and fruitful, way to develop shell scripts. While you're developing, let output go to the screen. When it's finally the way you want it, recall the command and redirect to a file to capture the output.

November 21 is World Hello Day, so rerun your program a few times, redirecting or not as fits your fancy.

Quick tour: Output

2005-11-20T08:00:00.000-08:00

Every language has strengths and weaknesses.

The shell's loops and conditionals, for constructing algorithms, are pretty
conventional. They'll do, but they're nothing special

The shell's data structures are just terrible. All it has are scalars and
arrays, just like FORTRAN. (Anyone out there remember FORTRAN?) Actually,
FORTRAN had multi-dimensional arrays. The shell doesn't even have that.

The shell does handle input and output of textfiles particularly well.

This week, I'll show you how to do basic output.

Control Flow: Loops

2005-11-17T14:27:00.000-08:00

Millions recognize the Shaker hymn, "Simple Gifts" from Aaron Copeland's "Appalachian Spring."

The words begin

'Tis the gift to be simple,
'tis the gift to be free,
'tis the gift to come down where you ought to be,
And when we find ourselves in the place just right,
It will be in the valley of love and delight.

Open Source programmers around the world today, like modern-day Shakers , practice freedom and simplicity. (The Shakers also practiced celibacy, however theirs was voluntary.)

In honor of the Shakers, you should try out the Linux, command-line version of their hymn, which is even simpler:


yes "'Tis the gift to be simple"

Unfortunately, no one can sing that fast.

The shell lets us bring this down to a reasonable pace with a loop:


while true
do
    echo "'Tis the gift to be simple"
    sleep 3
done

As usual, you should try this out, both in a file and on the command line. Try also substituting the lyrics to the K&R hymn, "Hello, world".

Once you know loops are available, it's surprising how useful they can be. This next loop tells us when the file "MY_LONG_AWAITED_OUTPUT" appears.


while true
do
    if test -e MY_LONG_AWAITED_OUPUT
    then
        printf 'DONE!!!\a\n'
        break
    fi
done

See if you can use a loop sometime today.

Control Flow: Conditionals

2005-11-17T08:00:00.000-08:00

Real programming languages also have control flow constructs, like conditionals and loops. Here's a conditional

#!/bin/bash

# Some variables are built in, and predefined.
# $0 is C's argv[0], or Perl's $0

# $1, $2, etc., are the rest of argv[]/$ARGV
# $* is the whole arg list

# You can even use "test" to test the value of variables

if test "$*" = ""
then
        HELLO="hello, world"
else
        HELLO="hello $*"
fi

echo $HELLO

As usual, snarf-n-barf this into a file -- call it hello -- make it executable, and run it. Try it both as hello and as hello $LOGNAME .

Try typing this program interactively to see how the interactive shell handles control-flow statements. Watch the prompt change. What's $* for your interactive shell? (Hint: echo $*).

But would anyone use an if ... then ... else ... or even an if ... then ... on the command line? Daily. I'll show practical examples, and some other useful forms for conditionals, too.

(The Boehm-Jacopini theorem says a programming language needs nothing but a "while" and an "if," but language designers know enough not to leave well enough alone. I could write this blog in Basic English, but why would I want to?)

Next up, loops.

Variables

2005-11-16T08:00:00.000-08:00

If it's a programming language, it needs variables. Here's how you assign a value to a shell variable, and how you get that value back (dereference it).


    #!/bin/bash

    # Here's how
    # you give variables values.

    HELLO="hello, world"

    # (No spaces allowed around the '=' sign:
    # a design botch in the shell.)

    # And here's how you use them.  Note the dollar sign.

    echo $HELLO

Convince yourself that this works interactively, too. Try putting spaces around the equals sign, to see what the error message says. That way, when you make this mistake, you'll recognize it.

I don't know anyone who's crazy about this design (no spaces, '$' to dereference, no '$' to set), but we can no more fix it now than we can fix English syntax. There may be shell scripts still running that were written before you were born. Hindsight's always 20-20.

Unsurprisingly, you can also assign variables the values of other variables. This is a programming language.


    GOODBYE=$HELLO
    echo $GOODBYE
    echo $HELLO

You can also unset a variable.

    
    unset HELLO
    echo $GOODBYE
    echo $HELLO

Shebang!

2005-11-15T08:00:00.000-08:00

The quickest way to make progress in bash is to bounce back and forth between using it interactivly and non-interactively.

Before we do that, though, it's worth our taking a quick tour of some things you'll see in shell scripts (which is what shell programs are called).

To cut down on confusion, most scripts we show on the tour will do the same thing as yesterday's: print "hello, world." (It was that or "Just Another Perl Hacker" :-)

Here's the second version of yesterday's script.


#!/bin/bash

# "#!" is prononced "shebang" (shell bang)
# 
# When a program starts like this,
# the Unix kernel treats it specially,
# and feeds it to a particular interpreter -- in this case, /bin/bash.
#
# When the kernel sends it to bash, bash treats that line as a comment,
# because it starts with a '#'

echo hello, world

# If you don't have bash, use ksh.  If you don't have either, us /bin/sh

Cut-and-paste this code into a file, make it executable, and run it. ("Cut-and-paste" is also colloquially known as "snarf-n-barf.")

Try modifying the shebang line to have the kernel send it to other interpreters, like /bin/ksh, /bin/sh, /bin/tcsh, or even /bin/true. What happens?

Claims that "shebang" was originally spelled "Sí bheang" are ill-founded, urban blarney.

The Canonical Program

2005-11-14T08:00:00.000-08:00

Perhaps the single most influential book on programming is The C Programming Language, by Brian Kernighan and Dennis Ritchie.

It begins like this:

1.1 Getting Started

The only way to learn a new programming language is by writing programs in it.The first program to write is the same for all languages: Print the words hello, world

This is the basic hurdle; to leap over it you have to be able to create the program text somewhere, compile it successfully, load it, run it, and find out where your output went. With these, mechanical details mastered, everything else is comparatively easy.

Since K&R, everyone's begun this way. No sense going against the crowd if you don't have to. Let's begin.


# The canonical program

echo hello, world   # note that '#' starts a comment

You can type these lines into a file, make the file executable, and run it as a command. You can also just type the lines at the command prompt. The result is the same. It's a programming language. It's a command-line language. It's new Shimmer.

(You can omit the comments, but it's more fun to type them in.)

Give it a try.

What's This Blog About, Anyway?

2005-11-13T08:00:00.000-08:00

This blog is about using the shell.

The shell is a full-featured command interpreter that's both a command-line and programming language. By "command-line," I mean you can type, interactively, at a prompt and get things done. By "programming language," I mean you can write programs in the same language.

Having something that does both well is a pleasant surprise.

I'll keep blog posts bite-sized, and I'll make the entries for a week relate to one another.

Topics will progress, but I'll try not to make them too interdependent. I'd like you to be able to jump in and out whenever you want, flitting from topic to topic like a butterfly going from flower to flower, sampling whatever attracts your fancy.

Because shells are just programs, there are a lot of shells for Unix, Linux, and other, similar operating systems. In this blog, when I say "shell," I mean a POSIX-conforming shell, which follows the specs laid out in ISO 9945-2.

In other words, not csh or tcsh, not zsh, not the Massey shell, and not even very old versions of /bin/sh. This still leaves a lot of possibilities.

Also, most commands you'll use in the shell are standalone applications. (Unlike other operating systems, most shell commands aren't built in to the command interpreter.) This means shell programs can depend on the underlying operating system.

You need to see real examples, so I'll focus on programming in bash ("the Bourne-again shell") running on Linux.

Ready?