Linux

Power Linux with pipes and redirection


Despite the fact that most people consider command-line computer use a thing of the past, there are some really interesting things you can do with the command-line. Despite pretty GUIs and eye candy being the order of the day, there is a definite advantage to being familiar and comfortable with the Linux command-line.

Two such advantages are the use of pipes and redirection. With pipes and redirection, you can "chain" multiple programs to become extremely powerful commands. Most programs on the command-line accept different modes of operation. Many can read and write to files for data, and most can accept standard input or output. This means that you can direct the output of one program as input to another program. You can then take the output of the second program and redirect it as input to yet another program, or redirect the output to a file.

In fact, most programs have two modes of output: standard output (stdout for short) and standard error (stderr). These are both different streams of output that are distinct from each other, which can allow you to redirect the stderr of a program to the same "stream" as the standard input (stdin), or redirect them to different places.

For example, to redirect the output of the ps program to a file, use:

$ ps ax > output.txt

If you wanted to merge two separate files together into a single file that already contains data, you could use the cat command and redirection:

$ cat 1.txt 2.txt >> 3.txt

A single > will redirect output to a file, overwriting the contents of the file if it exists. A double >> will redirect output to a file, but will append the output to the file if it exists, and create a new file if it doesn't.

By default, redirection works on stdout, while stderr will typically go to the screen. If you wish to redirect stderr to the same "stream" as stdout, use:

$ strace ps ax > output.txt 2>&1

strace is a unique program; the stdout of strace is the actual output of the program, while all the trace information goes to stderr. In the above command, strace is tracing the system calls and signals of the ps ax command and both stdout and stderr are being redirected to the file output.txt. This is accomplished by redirecting stderr (represented by the number 2) to wherever stdout (represented by &1) is directed. If you wanted to just have the trace output redirected to a file with the ps output on the screen, you would use:

$ strace ps ax 2>strace.txt

Pipes work a little differently. When you use a pipe, the output of the command on the left of the pipe is redirected as input to the command on the right of the pipe. For instance, to find all sshd processes in a ps listing, you would use:

$ ps ax | grep sshd | grep -v grep

This redirects the stdout of ps ax as stdin to grep sshd, which displays all of the processes that match sshd. This is then redirected as stdin to another grep command that filters out the previous grep's entry in the process table, resulting in a few lines that display only those processes that contain the word sshd in them.

This could be further tuned to use awk and kill each process:

$ for pid in `ps ax | grep sshd | grep -v grep | awk {'print $1'}`; do kill -9 $pid; done

The above command constructs a shell for-loop that iterates through the output of one ps command, two grep commands, and an awk command that return the process ID numbers of all sshd processes and then executes kill -9 on each one of them.

Despite how nice a good GUI looks, there is nothing like a good quick CLI command to make life more efficient and interesting.

Delivered each Tuesday, TechRepublic's free Linux NetNote provides tips, articles, and other resources to help you hone your Linux skills. Automatically sign up today!

About

Vincent Danen works on the Red Hat Security Response Team and lives in Canada. He has been writing about and developing on Linux for over 10 years and is a veteran Mac user.

4 comments
mikifin
mikifin

When I used Unix decades ago I found piping between threads, turning the work result from one thread over to another the most important aspect of these commands.

superbat
superbat

These are among the oldest and most useful of tools. Microsoft has even included them from their original copy of "CPM" they renamed "DOS" right up into XP in command Mode. The command line that is available in Win-2000 and XP is very impressive! I was nearly lost when I couldn't find them in Vista. Boggles the mind!

stomfi
stomfi

The order of redirection in the case of redirecting standard output and error is important depending what you want. If say you want to get rid of all messages from a command the correct way is: > /dev/null 2>&1 which send all output to the bottomless pit file /dev/null Try this yourself with ls non_existent_file_name 2>&1 /dev/null and you will get an error message whereas ls non_existent_file_name > /dev/null 2>&1 doesn't give one. This is because redirection operators may precede or appear anywhere within a simple command or may follow a command, and redirections are processed by the shell from left to right in the order they appear. Thus the first command directs only the standard output to /dev/null because the standard error was duplicated as standard output before the standard output was redirected. Hope that is clear. This is very useful in shell scripts when you want to output only what you need and no error messages confusing the issue. A shorthand way of saying >/dev/null 2>&1 is &>/dev/null Instead of /dev/null you can capture the output to a local file if you want to see what it says. Stomfi Brisbane

alaniane
alaniane

they did away with the command line in Vista or is it just the redirection and pipe characters?