What is 2>&1 (File Descriptor) ?

📅 January 4, 2024
“No, this is not a Q*bert expletive nor is it a mathematical expression.”

When perusing Bash scripts, you might encounter an odd-looking sequence of characters similar to this:

lolcat somefile.txt 2>> error.log

or this:

lolcat somefile.txt 2>&1

It might appear cryptic, but Linux, which is based upon UNIX, implements simple yet powerful concepts that perform predictably according to specific rules. One concept is the file descriptor, and it can be used to control the flow of data as you run and chain commands from the command line or a script.

File Descriptors

“A file…what?”

Data flows through your system like water flowing through pipes. With Bash, we can actually control the flow of data using techniques like piping (|) and redirection (>> or <). This allows us to tell Linux that error messages should go to a file while plain notifications are displayed in the terminal, for example.

Each file descriptor is represented by a unique number from 0 to 9. By default, whenever we open a process (a running program in Linux), three file descriptors are automatically created whether we specifically use them or not.

Each Linux process can use standard input, standard output, and standard error.

0 Standard Input stream. (stdin) Send data TO the process

1 Standard Output stream. (stdout) Normal message or output.

2 Standard Error stream. (stderr) Error messages.

The reason multiple descriptors exist is so we can separate input from the error messages and regular output instead of lumping all output together. When you run a regular process, both error messages (standard error) and normal output (standard output) are routed to the same terminal window, so it seems like there is no difference.

However, with a command like this,

lolcat somefile.txt 2>> error.log

we can redirect stderr (file descriptor 2) to a file named error.log located in the current directory. When a file produces and error message, that mesage will appear in a file instead of being displayed in the terminal.

lolcat is a program that colorizes text that it reads from the plain text file somefile.txt. If any errors occur — meaning error streams sent to stderr or file descriptor 2 — they will be appended to a file named error.log due to the 2>>. If we want to rewrite error.log each time, then we would use 2> error.log.

(You might also see 2>error.log squished together, but this makes no difference. Just make sure that the stream redirection appears last.)

Successful and Unsuccessful Runs

If the command runs properly, the text file contents will appear in the terminal as normal, but if we misspell lolcat to something nonexistent, such as lolca, then we will not see the error message in the terminal. It will appear in error.log instead because 2>>error.log redirects any errors to the file.

An error…but not quite.

Problems can still occur but will not be logged. For example, run this with a nonexistent text file that lolcat cannot find:

lolcat somefile2.txt 2>>error.log

An error message will appear in the terminal, not in error.log.

lolcat: somefile2.txt: No such file or directory

Why? Isn’t this an error too? Some messages are “errors,” but they are not sent to standard error. They act more like notification messages, and they are streamed to standard output, not standard error. There is nothing we can do about this in this example. Which file descriptor certain messages appear on depends upon the program itself, so do not expect all messages that we would consider errors to appear on the standard error stream.

What is 2>&1 ?

You might see this. It says, “Redirect stderr to the same place as stdout.”

In short, this is redundant by default because stderr is already redirected to the same stream as stdout, which is the terminal itself. There are times when you might need to use this to reset the file streams after modifying them previously in a script, for example.

lolca somefile.txt 2>&1

The above will result in the same default behavior as if we did not use 2>&1. The ampersand (&) denotes a file descriptor. If we omit it and use 2>1, then errors are sent to a file named 1. Make sure the stream redirection appears last.

These two redirects have completely different meanings, so pay attention!

lolca somefile.txt 2>1 # Redirect stderr to a file named 1 and overwrite that file each time

lolca somefile.txt 2>&1  # Redirect stderr to the same place as stdout

Send Output and Error to Different Files

This can be useful to separate output. Try this:

lolcat somefile.txt > goodoutput.txt 2>error.log

You will not see anything in the terminal window. If the program runs successfully, the contents of somefile.txt will be sent to goodoutput.txt (minus the color). Any errors that occur on the standard error stream are sent to error.log. This separates good output from error output.

However, like before, if we enter this,

lolcat somefile2.txt > goodoutput.txt 2>error.log

the file goodoutput.txt will contain an error message telling us that somefile2.txt cannot be found because this message is streamed to stdout, not stderr.

lolcat: somefile2.txt: No such file or directory

One point to keep in mind with the above stream redirection is that both files will be updated every time the command executes. If there are no errors, then goodoutput.txt will contain text and error.log will be empty. If an error is logged, error.log will contain the stderr error, but goodoutput.txt will be cleared.

Why Use File Descriptors?

By controlling stream redirection, a common use is to store all errors in an error log file to clean up the terminal output so we only see successful results. Suppose we want to computer md5 hashes of all directory contents but some items, like subdirectories, will not hash without recursion, which we might not want to use here.

md5sum -b * 2>error.log

By separating stderr from stdout, we can see which files were successfully hashed, and those items that could not be hashed are stored in a separate error.log file for further evaluation later.

If we do not care about errors all, then we can redirect them to the bit bucket like this:

md5sum -b * 2>/dev/null

No errors will be logged, and only successful hashes will be displayed.

Conclusion

There is more, much more involved with file descriptors than covered here, such as >&- (close the standard output stream) and >| (force output even if noclobbber is set). In fact, we can even create our own file descriptors from 3 to 9 for further customization.

However, this is something that is mostly useful to Linux system administrators for low-level stream dealings. For most of the time, the basic three stdin, stdout, and stderr are plenty.

Have fun!