Table of Contents
Introduction
In Unix, whenever a running program opens a file, the kernel stores a reference to it in the process’ memory. Those references to open files are integers starting from 0 for each process, and are called file descriptors or FDs. A process can only read and write files that it has opened, that is, for which it had obtained a file descriptor.
The kernel pre-initializes three file descriptors for every program that is run. So when the process
starts running, those three are already available without having to call open.
Because they are always available, and because everything defaults to using them, they are called standard.
They have the following names, abbreviations, assigned numbers, and purposes:
-
Standard input (stdin), fd 0, represents device for input into the program. By default, that input is set to the user’s keyboard
-
Standard output (stdout), fd 1, represents device for program’s standard (non-error) output. By default, that output is set to the computer screen
-
Standard error (stderr), fd 2, represents device for program’s error output. By default, that output is set to the computer screen
Other than being automatically present and standard, they are the same as any other file descriptor. File descriptors are explained in more detail in Unix File Descriptors.
Overlooked Ubiquity
Ubiquity 1. Existence everywhere, or in all places, at the same time; omnipresence; [1913 Webster]
A most useful characteristic of standard Unix input and output, which is taken for granted and overlooked, is that they are present everywhere and always automagically point to the correct location.
For example, if you are logged in to a Linux console, input and output automatically point to your keyboard and monitor attached to it.
If you are using SSH to access a remote computer, programs you run there are far away from you, but their standard input and output are automatically connected to your local keyboard and display.
If you are visiting a web page, the web server sees data sent by your browser as its standard input, and has your browser’s blank page as its standard output.
If two programs are communicating autonomously over the network, they have their standard inputs and outputs connected.
Redirection, Pipes
In addition to automatic behavior, users can manually control location of the processes’ standard input and output.
This is done by using redirection and piping. Redirection is used for files, and piping is used for programs. Their use can be combined.
Uses for pipes and redirection are numerous, and are all a consequence of powerful underlying concept.
Practical Examples
In this chapter we are going to show major types, most common patterns of use. They are representative and suitable for forming a good understanding of the concept.
You might notice slight variations in spacing, formatting, or method between examples. Those are not inconsistencies, but a demonstration of accepted variations in syntax. Variations shown are not limited to specific examples — they all apply to all examples.
Print “Hello, World!” to the default standard output:
echo "Hello, World!"
Hello, World!
Redirection
Using redirection we divert output to files.
Files can be local, or existing on network file partitions mounted from remote computers; it doesn’t make a difference.
Let’s “Hello, World!” and redirect output to file test and truncate (erase) the file before writing to it.
Two lines shown below are identical — fd 1 is the default and does not have to be specified.
We write to file twice, but because of truncation on every write, only the second line will be in it:
echo "Hello, World!" 1> test
echo 'Hello, World!!' > test
cat test
Hello, World!!
Then, let’s print “Hello, World!” and redirect output to file test, appending rather than truncating it:
echo "Hello, World!" >>test
>> test print "Hello, World!!"
cat test
Hello, World!! # (This line is a leftover in file `test` from previous example)
Hello, World!
Hello, World!!
Command find can produce both standard output and errors during run. Run it and send all errors to /dev/null.
/dev/null is a device that accepts any input and simply discards it:
echo "Hello, World!" 2> /dev/null
Run find and send standard output to file results, standard error to file errors:
echo "Hello, World!" >results 2> errors
1>results echo "Hello, World!" 2> errors
Run find and send standard output to results, standard error to errors:
echo "Hello, World!" >results 2> errors
1>results echo "Hello, World!" 2> errors
Run find and send both stdin and stdout to the same place:
echo "Hello, World!" 1>/dev/null 2>/dev/null
Run find and send all output (not limited to stdout and stderr) to the same file:
echo "Hello, World!" >& /dev/null
Run cat which will print content from standard input to the screen. Use content of /etc/passwd as input into cat.
Note that this example is identical in effect to cat /etc/password, but the reason is different. With “<”, file content is being passed as standard input
into the command. Without “<”, string “/etc/passwd” is being passed as the first argument to cat, which interprets it as the name of the file
it should open and display to screen.
cat < /etc/passwd
Pipes
Using pipes we pass input and output between programs. If programs are network-enabled, data can be passed between remote computers.
Print “Hello, World!” and pipe output to a program that cuts off the first 7 characters:
echo "Hello, World!" | cut -b8-
World!
Print contents of /etc/password and pipe output to a command that only prints lines containing a number between 1000 and 1999 in one of its fields:
cat /etc/passwd | grep -P ':1[0-9]{3}:'
user:x:1000:1000:,,,:/home/user:/bin/bash
Print “Hello, World!” and send it via TCP/IP to remote computer at address 10.0.0.1, port 8000:
echo "Hello, World!" |nc 10.0.0.1 8000