1. sed.js

GNU sed stream editor compiled to JavaScript

May be invoked with the following command-line options:

If no -e --expression options are given on the command-line, then the first non-option argument on the command line is taken to be the script to be executed.

2. Programs

A sed program consists of one or more sed commands, passed in by one or more of the -e --expression options, or the first non-option argument if zero of these options are used.

Commands within a script can be separated by semicolons (;) or newlines (ASCII 10). Some commands, due to their syntax, cannot be followed by semicolons working as command separators and thus should be terminated with newlines or be placed at the end of a script. Commands can also be preceded with optional non-significant whitespace characters.

Each sed command consists of an optional address or address range, followed by a one-character command name and any additional command-specific code.

sed maintains two data buffers: the active pattern space, and the auxiliary hold space. Both are initially empty.

sed operates by performing the following cycle on each line of input: first, sed reads one line from the input stream, removes any trailing newline, and places it in the pattern space. Then commands are executed; each command can have an address associated to it: addresses are a kind of condition code, and a command is only executed if the condition is verified before the command is to be executed.

When the end of the script is reached, unless the -n option is in use, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed. Then the next cycle starts for the next input line.

Unless special commands (like ‘D’) are used, the pattern space is deleted between two cycles. The hold space, on the other hand, keeps its data between cycles (see commands ‘h’, ‘H’, ‘x’, ‘g’, ‘G’ to move data between both buffers).

2.1. Selecting lines

Addresses in a sed script can be in any of the following forms:

If no addresses are given, then all lines are matched; if one address is given, then only lines matching that address are matched.

An address range can be specified by specifying two addresses separated by a comma (,). An address range matches lines starting from where the first address matches, and continues until the second address matches (inclusively).

If the second address is a regexp, then checking for the ending match will start with the line following the line which matched the first address: a range will always span at least two lines (except of course if the input stream ends).

If the second address is a number less than (or equal to) the line matching the first address, then only the one line is matched.

GNU sed also supports some special two-address forms; all these are GNU extensions:

Appending the ! character to the end of an address specification negates the sense of the match. That is, if the ! character follows an address range, then only lines which do not match the address range will be selected. This also works for singleton addresses, and, perhaps perversely, for the null address.

2.2. Regular Expression Syntax

To know how to use sed, people should understand regular expressions (regexp for short). A regular expression is a pattern that is matched against a subject string from left to right. Most characters are ordinary: they stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, the pattern

The quick brown fox

matches a portion of a subject string that is identical to itself. The power of regular expressions comes from the ability to include alternatives and repetitions in the pattern. These are encoded in the pattern by the use of special characters, which do not stand for themselves but instead are interpreted in some special way. Here is a brief description of regular expression syntax as used in sed.

Note that the regular expression matcher is greedy, i.e., matches are attempted from left to right and, if two or more matches are possible starting at the same character, it selects the longest.

Default Output

GIST | sed sends its results to the screen by default

Printing Lines

GIST | sed has printed each line twice now. This is because it automatically prints each line, and then we've told it to print explicitly with the p command.

GIST | We can clean up the results by passing the -n option to sed, which suppresses the automatic printing

Address Ranges

GIST | Let's modify the output by only having sed print the first line.

GIST | We've just given an address range to sed. If we give sed an address, it will only perform the commands that follow on those lines. In this example, we've told sed to print line 1 through line 5.

GIST | This will result in the same output, because we've told sed to start at line 1 and then operate on the next 4 lines as well.

GIST | If we want to print every other line, we can specify the interval after the ~ character. The following line will print every other line starting with line 1.

Deleting Text

GIST | We can easily perform text deletion where we previously were specifying text printing by changing the p command to the d command. We no longer need the -n command because with the delete command, sed will print everything that is not deleted, which will help us see what's going on. We can modify the last command from the previous section to make it delete every other line starting with the first. The result is that we should be given every line we were not given last time.

Substituting Text

GIST | Now let's substitute the expression "o" with "@"

GIST | To make sed replace every instance of "o" instead of just the first on each line, we can pass an optional g flag to the substitute command.

GIST | If we only wanted to change the second instance of "o" that sed finds on each line, then we could use the number 2 instead of the g.

GIST | If we only want to see which lines were substituted, we can use the -n option again to suppress automatic printing. We can then pass the p flag to the substitute command to print lines where substitution took place..

GIST | If we want the search process to ignore case, we can pass it the i flag.

Referencing Matched Text

GIST | If we wish to find more complex patterns with regular expressions, we have a number of different methods of referencing the matched pattern in the replacement text. For instance, if we want to match the from the beginning of the line to "on" we can use the expression.

GIST | Since you don't know the exact phrase that will match in the search string, you can use the "&" character to represent the matched text in the replacement string. This example shows how to put parentheses around the matched text.

GIST | A more flexible way of referencing matched text is to use escaped parentheses to group sections of matched text. Every group of search text marked with parentheses can be referenced by an escaped reference number. For instance, the first parentheses group can be referenced with "\1", the second with "\2" and so on. In this example, we'll switch the first two words of each line.

GIST | As you can see, previous results are not perfect. For instance, the second line skips the first word because it has a character not listed in our character set. Similarly, it treated "they'll" as two words in the fifth line. Let's improve the regular expression to be more accurate

Supplying Multiple Editing Sequences

GIST | We can string various commands to sed by using the "-e" option before each command.

GIST | Another approach to stringing commands together is using a semi-colon character (;) to separate distinct commands. This works the same as above, but the "-e" is not required.

Advanced Addressing

GIST | One of the advantages of sed's addressable commands is that regular expressions can be used as selection criteria. This means that we are not limited to operating on known line values, like we learned previously.

GIST | We can, instead, use regular expressions to match only lines that contain a certain pattern. We do this by placing our match pattern between two forward slashes (/) prior to giving the command strings.

GIST | This example demonstrates using regular expressions to generate addresses for other commands. This matches any blank lines (the start of a line followed immediately by the end of the line) and passes them to the delete command.

GIST | We can delete lines starting at a line that only contains the word "START" until a line reading "END" by issuing the following command.

Using the Hold Buffer

One piece of functionality that increases sed's ability perform multi-line aware edits is what is called the "hold buffer". The hold buffer is an area of temporary storage that can be modified by certain commands.

The presence of this extra buffer means that we can store lines while working on other lines, and then operate on each buffer as necessary.

The following are the commands that affect the holding buffer:

The contents of the holding buffer cannot be operated on until it is moved to the pattern buffer in one way or another.

GIST | This is a procedural example of how to join adjacent lines (sed actually has a built-in command that would take care of a lot of this for us. The "N" command appends the next line to the current line. We are going to do things the hard way though for the sake of practice)

The first thing to note is that the "-n" option is used to suppress automatic printing. Sed will only print when we specifically tell it too.

The first part of the script is "1~2h". The beginning is an address specification meaning to perform the subsequent operation on the first line, and then on every other line afterwards (each odd numbered line). The "h" part is the command to copy the matched line into the holding buffer.

The second half of the command is more complex. Again, it begins with an address specification. This time, it is referring to the even numbered lines (the opposite of the first command).

The rest of the command is enclosed in braces. This means that the rest of the commands will inherit the address that was just specified. Without the braces, only the "H" command would inherit the address, and the rest of the commands would be executed on every line.

The "H" command copies a new-line character, followed by the current pattern buffer, onto the end of the current holding pattern.

This holding pattern (an odd numbered line, followed by a new-line character, followed by an even numbered line) is then copied back into the pattern buffer (replacing the previous pattern buffer) with the "g" command.

Next, the new-line character is replaced with a space and the line is printed with the "p" command.

GIST | If you are curious, using the "N" command, as we described above, would shorten this considerably. This command will produce the same results that we've just seen.

Further Reading