A how-to manual: how awk reads fields, patterns and actions, numbers and strings, printf, arrays, functions, several files, separators and records, mawk against gawk, safe use in scripts and speed, with every example run for real.
▶Prefer to watch? This short video runs this page's main commands on a real Ubuntu 24.04 machine, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.
awk: reads text a line at a time, splits each line into fields, and runs your code on the lines you choose.
awk '{ print $1 }' file.txt
awk -F: '$3 >= 1000 { print $1 }' /etc/passwd
awk '{ n[$1]++ } END { for (k in n) print k, n[k] }' access.log
awk is for text that is a table: log lines, the output of ps and df, /etc/passwd, a report with columns. It can select lines, pick and rearrange columns, add numbers, count things and print a formatted report, all in a line you type once. It is also a complete small language (variables, arrays, functions, loops), so a script of twenty lines can do what would otherwise be a throwaway program.
This guide goes from the first print $1 to the parts that cause trouble: a number that prints as 1.23457e+07, a field separator set one line too late, a join that silently prints nothing when the first file is empty, a shell variable that becomes code, and the fact that the awk on your Ubuntu server (mawk) is not the awk in most tutorials (gawk). Where they differ, the lab ran both.
Every example below ran for real on Ubuntu 24.04 (mawk 1.3.4 20240123, gawk 5.2.1, the original BWK awk 20231127 and BusyBox awk 1.36.1), against the sample files shown in the next section. Where an example uses a gawk-only feature it says so. Things the lab could not run are marked as not run (macOS awk, for one).
What awk is, which one you have, and the sample data
awk reads its input one line (a record) at a time, splits it into fields, and for each record runs the rules you wrote. A rule is a pattern (which lines) and an action (what to do, in braces). Without -i or a redirection, awk never changes a file; it prints, so it is safe to try.
There are several awks, and the name awk is only a link to one of them:
mawk the awk on a fresh Ubuntu server (24.04: mawk 1.3.4 20240123). Small and the fastest of the four in the lab, but it has fewer features than gawk and it counts bytes, not characters.
gawk GNU awk, 5.2.1 here. What most tutorials assume: more functions, \s and \w in regexes, sorting, time functions, in-place editing, CSV helpers. Install it with: sudo apt install gawk
original-awk the "one true awk" by Brian Kernighan (20231127). It is also what macOS ships (an older build there; not run in the lab).
BusyBox awk 1.36.1 the awk in Alpine and many containers. Small, with some GNU features and no math functions in this build.
$ readlink -f /usr/bin/awk (in a fresh vid-base:24.04 container, before gawk was installed)
/usr/bin/mawk
$ awk -W version | head -1
mawk 1.3.4 20240123
Installing gawk changes the link (Debian alternatives), so afterwards plain awk means gawk:
$ ls -l /etc/alternatives/awk
lrwxrwxrwx 1 root root 13 Oct 18 00:54 /etc/alternatives/awk -> /usr/bin/gawk
The lab had all four installed side by side and the link pointed at gawk, so a command written as awk below ran on gawk 5.2.1. Where the result depends on the version, the guide runs mawk, gawk, original-awk and busybox awk by name.
How to ask each one for its version (they do not agree):
gawk gawk --version GNU Awk 5.2.1, API 3.2, ...
mawk mawk -W version mawk 1.3.4 20240123 (--version is an error in mawk)
original-awk original-awk --version awk version 20231127
BusyBox busybox | head -1 BusyBox v1.36.1 ... (awk --version is an error)
The examples use these files, which were made in ~/lab. sales.txt has a header line and six rows:
$ cat sales.txt
region product qty price
north apples 10 1.50
south pears 4 2.25
north pears 7 2.25
east apples 3 1.50
south apples 12 1.40
east plums 0 3.10
app.log (the same log as the sed guide) and access.log (a web server log):
$ cat app.log
2026-10-17 02:00:01 INFO api request ok user=ann ip=10.0.1.11
2026-10-17 02:00:02 INFO api request ok user=ed ip=10.0.1.12
2026-10-17 02:00:03 WARN api slow request 910ms user=ann ip=10.0.1.11
2026-10-17 02:00:04 ERROR pay card declined user=kim ip=10.0.2.31
2026-10-17 02:00:05 INFO web GET /index.html 200 ip=10.0.3.5
2026-10-17 02:00:06 ERROR pay upstream timeout user=ed ip=10.0.1.12
2026-10-17 02:00:07 INFO web GET /missing 404 ip=10.0.3.6
$ cat access.log
10.0.1.11 - - [17/Oct/2026:02:00:01 +0000] "GET /index.html HTTP/1.1" 200 5120
10.0.1.12 - - [17/Oct/2026:02:00:02 +0000] "GET /api/items HTTP/1.1" 200 842
10.0.3.5 - - [17/Oct/2026:02:00:03 +0000] "GET /missing HTTP/1.1" 404 196
10.0.1.11 - - [17/Oct/2026:02:00:04 +0000] "POST /api/login HTTP/1.1" 302 0
10.0.3.5 - - [17/Oct/2026:02:00:05 +0000] "GET /missing HTTP/1.1" 404 196
10.0.2.31 - - [17/Oct/2026:02:00:06 +0000] "GET /api/items HTTP/1.1" 500 1024
10.0.1.11 - - [17/Oct/2026:02:00:07 +0000] "GET /index.html HTTP/1.1" 200 5120
passwd.txt is five lines in /etc/passwd format, and data.csv is a small CSV with quoted fields:
$ cat passwd.txt
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
ann:x:1000:1000:Ann Lee:/home/ann:/bin/bash
ed:x:1001:1001:Ed Kay:/home/ed:/bin/zsh
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
$ cat data.csv
id,name,note,amount
1,ann,"likes tea, hot",10
2,ed,plain,20
3,kim,"says ""hi""",30
The examples that use only POSIX features were run on mawk as well as on gawk, and gave the same output, except for the order of for (k in a) and the space before a bracket in a function call (both described below). Two small files, ids.txt and names.txt, appear in the section on several files. A one-million-line log (63 MB) was made for the speed section.
How awk works: records, fields, patterns and actions
A program is a list of rules. For every input line awk does this:
1. read the line into $0 (the record),
2. split it into fields $1, $2, ... $NF (NF is how many there are),
3. for each rule in order: if the pattern is true for this line, run the action.
pattern { action }
The pieces are optional, and the defaults are what make awk short:
a rule with no pattern runs on every line: awk '{ print $1 }' file
a rule with no action prints the whole line: awk '$3 > 5' file (same as '$3 > 5 { print $0 }')
BEGIN { ... } runs once, before the first line (set variables, print a header)
END { ... } runs once, after the last line (print totals)
$ awk '{ print $1 }' sales.txt | head -3
region
north
south
$ awk '{ print $2, $3 }' sales.txt | head -3
product qty
apples 10
pears 4
$ awk 'NR > 1 && $3 > 5' sales.txt
north apples 10 1.50
north pears 7 2.25
south apples 12 1.40
$ awk '$2 == "apples"' sales.txt
north apples 10 1.50
east apples 3 1.50
south apples 12 1.40
$ awk 'BEGIN { print "start" } END { print NR, "lines" }' sales.txt
start
7 lines
Three things to remember from the start:
1. The program goes in single quotes, so the shell does not touch $1 or the braces. Fields are $1, $2 (with a dollar sign) inside awk, and the shell never sees them.
2. Fields count from 1. $0 is the whole line, $NF is the last field, $(NF-1) the one before it.
3. Comparing a field with a number compares numbers, and with a string compares strings. The section on numbers and strings has the rules.
The order of the rules matters, and all matching rules run for each line (unless one says next):
$ awk '1; NR == 2 { print "-- inserted --" }' sales.txt | head -4
region product qty price
north apples 10 1.50
-- inserted --
south pears 4 2.25
$ awk 'NR == 2 { print "-- before 2 --" } 1' sales.txt | head -3
region product qty price
-- before 2 --
north apples 10 1.50
The pattern 1 is always true, so it prints every line. It is an awk habit to put 1 last so that the line is printed after the edits made by the other rules.
The options you will use
awk [options] 'program' [file ...]
awk [options] -f program.awk [file ...]
-F fs set the field separator (FS). -F: -F, -F'\t' -F'[,;]'. A single character other than a space is taken literally; a longer one is a regex.
-v var=value set a variable before BEGIN runs. Escape sequences in the value are processed (\t becomes a tab).
-f file read the program from a file. Several -f are allowed, so a library file can be combined with a program.
-- end of options (the program text follows)
var=value between file names on the command line: set a variable when awk reaches that point
- read standard input as a file
gawk adds many long options. The ones worth knowing: --lint (warn about doubtful code), --posix, --traditional, --sandbox, --profile, --pretty-print, --dump-variables, -i inplace (edit files), -e 'program' (give the program as text and mix it with -f), -b (count bytes, not characters). mawk has -W options: -W version, -W interactive (unbuffered output), -W posix, -W usage, -W exec file.
$ awk -F: '{ print $1, $NF }' passwd.txt | head -3
root /bin/bash
daemon /usr/sbin/nologin
ann /bin/bash
$ awk -v min=5 'NR>1 && $3 >= min { print $1, $2 }' sales.txt
north apples
north pears
south apples
The -e option is a gawk (and BusyBox) option; mawk refuses it and the original awk ignores it:
$ mawk -e 'BEGIN { print "e-works" }'
mawk: not an option: -e
$ gawk -e 'BEGIN { print "e-works" }'
e-works
$ original-awk -e 'BEGIN { print "e-works" }'
original-awk: unknown option -e ignored
Assignments between file names take effect when awk reaches them, so a variable can differ per file:
$ awk '{ print v, $1 }' v=A passwd.txt v=B sales.txt | sed -n '1p;6p'
A root:x:0:0:root:/root:/bin/bash
B region
A program from a file can start with a shebang, and then it runs like a command (see the section on awk programs).
Using - as a file name reads standard input in the middle of a list of files:
$ echo 'ann' | awk '{ print $1 "|" }' -
ann|
Fields and records
Awk splits each line into fields on runs of spaces and tabs, ignoring leading and trailing blanks. The fields are $1 to $NF, and NF is a variable you can use anywhere:
$ echo ' a b c ' | awk '{ print NF; print $2 }'
3
b
$ awk '{ print NF, $NF, $(NF-1) }' sales.txt | head -2
4 price qty
4 1.50 10
A field can be an expression: $(NF-1), $NF, $(i+1). The $ binds tightly, so $NF-1 means ($NF)-1, not $(NF-1):
$ echo '5 b' | awk '{ print $NF-1; print $(NF-1) }'
-1
5
Print the fields you want. A comma puts the output separator (OFS, a space) between them; no comma joins them with nothing:
$ awk '{ print $1, $3 }' sales.txt | head -2
region qty
north 10
$ awk '{ print $1 $3 }' sales.txt | head -2
regionqty
north10
$ awk '{ print NR ": " $0 }' sales.txt | head -2
1: region product qty price
2: north apples 10 1.50
Fields can be assigned to. When you change a field (or NF), awk rebuilds $0 with OFS between the fields, which is how you squeeze spaces, drop a column or change the separator:
$ echo ' a b c ' | awk '{ print; $1=$1; print "[" $0 "]" }'
a b c
[a b c]
$ echo 'a b c' | awk '{ $2 = ""; print NF; print }'
3
a c
$ echo 'a b c' | awk '{ $1 = ""; sub(/^ /, ""); print }'
b c
$ echo 'a b c d' | awk '{ NF = 2; print; print NF }'
a b
2
$ echo 'a b c d' | awk '{ NF--; print }'
a b c
$ echo 'a b' | awk '{ $5 = "e"; print; print NF }' | od -c | head -3
0000000 a b e \n 5 \n
0000012
Assigning $2 = "" leaves the separator behind (a double space), so deleting a column needs the sub() above or a loop. Assigning to $5 when there are two fields creates $3 and $4 as empty fields, and NF becomes 5. NF = 2 and NF-- worked in gawk and in mawk.
Assigning to $0 splits the new text again:
$ echo 'a b c' | awk '{ $0 = "x y"; print NF, $2 }'
2 y
OFS is only used when awk joins fields. print $1, $2 uses it always, plain print uses it only if a field was changed:
$ echo 'a b c' | awk 'BEGIN { OFS="-" } { print $1, $2; print $1 $2; print; $3 = $3; print }'
a-b
ab
a b c
a-b-c
Print the fields in reverse order:
$ echo 'a b c' | awk '{ for (i = NF; i > 0; i--) printf "%s%s", $i, (i > 1 ? OFS : ORS) }'
c b a
In the END rule, $0 and the fields keep the last line that was read (in gawk and mawk):
$ awk 'END { print $0 }' sales.txt
east plums 0 3.10
$ printf 'a\nb\nc\n' | awk 'END { print $0, NR, NF }'
c 3 1
A negative field number is an error:
$ awk '{ print $-1 }' app.log
awk: cmd. line:1: (FILENAME=app.log FNR=1) fatal: attempt to access field -1
Field separators: FS, -F and OFS
The default separator is any run of blanks. Everything else is set with FS (the input) and OFS (the output). The -F option sets FS before the first line is read:
$ echo 'a:b:c' | awk -F: '{ print $2 }'
b
$ awk -F: '$3 >= 1000 { print $1 }' passwd.txt
ann
ed
A single character is taken literally, even if it is special in a regex, so -F. and -F'|' work:
$ printf 'a|b|c\n' | awk -F'|' '{ print $2 }'
b
$ printf 'a.b.c\n' | awk -F. '{ print $2 }'
b
Longer strings are regexes. This takes any run of digits, or a comma with spaces round it:
$ printf 'a1b22c\n' | awk -F'[0-9]+' '{ print $1 $2 $3, NF }'
abc 3
$ printf 'a, b,c ,d\n' | awk -F' *, *' '{ print $2 "|" $3 "|" $4 }'
b|c|d
A tab needs the escape, and the shortcut -Ft is not reliable:
$ printf 'a b\tc\n' | awk -F'\t' '{ print $1 "|" $2 }'
a b|c
$ printf 'atb\n' | gawk -Ft '{ print $2 }'
b
$ printf 'a\tb\n' | original-awk -Ft '{ print $2 }'
b
In gawk and mawk -Ft means the letter t (the first test split on it); the original awk treats -Ft as a tab. Write -F'\t' and the result is the same everywhere.
A single space is special: -F' ' (or the default) means any run of blanks. A single space in brackets means exactly one space, which makes empty fields:
$ printf 'a b\n' | awk -F' ' '{ print NF }'
2
$ printf 'a b\n' | awk -F'[ ]' '{ print NF }'
3
An empty FS splits into single characters. The -F'' form is not accepted (gawk prints its usage, mawk produces nothing), so set it in BEGIN:
$ printf 'abc\n' | awk 'BEGIN{FS=""} { print NF, $2 }'
3 b
(all four awks gave 3 b for the BEGIN form.)
The classic mistake is to set FS in the main rule. It takes effect on the next line, so the first line is split with the old separator:
$ printf 'a:b\nc:d\n' | awk '{ FS=":" } { print $1 }'
a:b
c
$ printf 'a:b\nc:d\n' | awk 'BEGIN { FS=":" } { print $1 }'
a
c
$ printf 'a:b\nc:d\n' | awk -F: '{ print $1 }'
a
c
To write with the same separator as you read, set OFS too, with -v or in BEGIN. Without OFS, an edited line is joined with spaces:
$ awk -F: -v OFS=: '{ $2 = "*" } 1' <<< 'a:b:c'
a:*:c
$ awk 'BEGIN { FS = OFS = ":" } { $2 = "*" } 1' <<< 'a:b:c'
a:*:c
$ awk -F: '{ $2 = "*" } 1' <<< 'a:b:c'
a * c
$ awk -F'\t' -v OFS='\t' '{ $2 = toupper($2) } 1' <<< $'a\tb\tc'
a B c
When fields are defined by what they look like (CSV with quoted commas), or by position, gawk has FPAT and FIELDWIDTHS (see the gawk section). Plain -F, breaks on a comma inside quotes, in every awk:
$ mawk -F, 'NR>1 { print $3 }' data.csv
"likes tea
plain
"says ""hi"""
The first row is cut in two at the comma inside the quotes. That is the point where to stop using FS and use gawk FPAT or a real CSV tool.
Patterns: choosing the lines
A pattern in front of an action decides which lines it runs on. The kinds:
/regex/ the line matches the regex
expression any expression that is true (non-zero, or a non-empty string)
$2 ~ /re/ !~ a field (or any string) matches or does not match a regex
pattern1, pattern2 a range: from the first line that matches pattern1 to the next that matches pattern2
BEGIN END before the first line, after the last
BEGINFILE ENDFILE before and after each file (gawk only)
$ awk '/ERROR/' app.log
2026-10-17 02:00:04 ERROR pay card declined user=kim ip=10.0.2.31
2026-10-17 02:00:06 ERROR pay upstream timeout user=ed ip=10.0.1.12
$ awk '!/INFO/' app.log
2026-10-17 02:00:03 WARN api slow request 910ms user=ann ip=10.0.1.11
2026-10-17 02:00:04 ERROR pay card declined user=kim ip=10.0.2.31
2026-10-17 02:00:06 ERROR pay upstream timeout user=ed ip=10.0.1.12
$ awk '$2 ~ /^p/' sales.txt
region product qty price
south pears 4 2.25
north pears 7 2.25
east plums 0 3.10
$ awk '/north/ && /pears/' sales.txt
north pears 7 2.25
$ awk '!/north/ && !/east/' sales.txt
region product qty price
south pears 4 2.25
south apples 12 1.40
Line numbers and arithmetic work as patterns:
$ awk 'NR % 2' sales.txt | head -3
region product qty price
south pears 4 2.25
east apples 3 1.50
$ awk 'length > 22' sales.txt
region product qty price
$ awk 'NF > 3 && $4 + 0 > 2' sales.txt
south pears 4 2.25
north pears 7 2.25
east plums 0 3.10
Ranges print from the line that starts it to the line that ends it, both included. They can use line numbers or regexes, and a range may start again later in the file:
$ awk 'NR==2,NR==4' sales.txt
north apples 10 1.50
south pears 4 2.25
north pears 7 2.25
$ awk '/WARN/,/web/' app.log | cut -c1-60
2026-10-17 02:00:03 WARN api slow request 910ms user=ann i
2026-10-17 02:00:04 ERROR pay card declined user=kim ip=10.
2026-10-17 02:00:05 INFO web GET /index.html 200 ip=10.0.3.
A range whose end never matches runs to the end of the file. If one line matches both the start and the end, the range is only that line (awk checks the end on the same line).
The header trap. A rule that works on numbers also reads the header, which is text. Here the minimum and maximum come out wrong because line 1 is the header:
$ awk 'NR==1 { min=max=$3 } $3 > max { max=$3 } $3 < min { min=$3 } END { print min, max }' sales.txt
0 qty
Skip the header with NR > 1 (or FNR > 1 with several files):
$ awk 'NR>1 && (min == "" || $3 < min) { min = $3 } NR>1 && $3 > max { max = $3 } END { print min, max }' sales.txt
0 12
The always-true trap. A bare string is true when it is not empty, so the second half of a || on a string is always true:
$ echo 'a' | awk '{ print ($1 == "z" || "b") }'
1
$ echo 'a' | awk '$1 == "z" || "b" { print "always true" }'
always true
Write each comparison in full: $1 == "z" || $1 == "b".
A string used as a regex is a regex, so the dot means any character, and a backslash has to be doubled in a string:
$ echo 'a.b' | awk '$0 ~ "." { print "dot matches anything" } $0 ~ "\\." { print "escaped" }'
dot matches anything
escaped
$ echo 'a.b' | awk -v re='.' '$0 ~ re { n++ } END { print n }'
1
The last command counted one match, but only because "." matches any character.
Variables and the built-in ones
A variable is created by using it. It is a number or a string depending on how it is used, it starts empty (which is 0 as a number and "" as a string), and it needs no declaration:
$ awk 'BEGIN { print x + 0, "[" x "]", length(x) }'
0 [] 0
$ awk '/nomatch/ { c++ } END { print c }' sales.txt
$ awk '/nomatch/ { c++ } END { print c + 0 }' sales.txt
0
$ awk '/apples/ { c++ } END { print c + 0 }' sales.txt
3
A counter that never ran prints an empty line, so add + 0 when the answer must be a number.
The built-in variables:
NR number of the current record, counted across all input
FNR record number within the current file
NF number of fields in the current record
FS OFS field separator for input / output (default: blanks / a space)
RS ORS record separator for input / output (default: newline)
FILENAME the current file name (empty in BEGIN)
ENVIRON array of the environment: ENVIRON["HOME"]
ARGC ARGV the command line arguments, as an array: ARGV[0] is awk
SUBSEP joins the parts of a multi-dimensional array index
RSTART RLENGTH set by match()
CONVFMT OFMT number to string formats (default %.6g)
(gawk adds many more: IGNORECASE, PROCINFO, ERRNO, RT, FPAT, FIELDWIDTHS, ARGIND and others.)
Setting variables from outside. -v sets one before BEGIN, and escape sequences in the value are processed. A name=value argument sets it later:
$ awk -v n=5 'BEGIN { print n + 1, n "" 1 }'
6 51
$ awk -v 's=a\tb' 'BEGIN { print s }'
a b
$ awk 'BEGIN { for (i = 0; i < ARGC; i++) print i, ARGV[i] }' a b=1 c
0 awk
1 a
2 b=1
3 c
$ X=hi awk 'BEGIN { print ENVIRON["X"] }'
hi
ENVIRON does not process escapes, which is why it is the safer way to pass text (see the section on shell scripts):
$ U='a\nb' awk 'BEGIN { print ENVIRON["U"] }'
a\nb
$ awk -v u='a\nb' 'BEGIN { print u }'
a
b
Numbers and strings
Awk has numbers (doubles) and strings, and it converts between them as needed. This is where most surprises come from, so the rules matter.
Fields, -v values, ENVIRON values, getline input and split() pieces that look like numbers are "numeric strings". A comparison between two numeric strings, or a numeric string and a number, is numeric. Otherwise it is a string comparison:
$ echo '10 9' | awk '{ print ($1 > $2), ("10" > "9"), ($1 > "9") }'
1 0 0
$ echo '10 abc' | awk '{ print ($1 > $2) }'
0
$ echo '10.0 ' | awk '{ print ($1 == 10), ($1 == "10"), ($1 "" == 10) }'
1 0 0
$ echo '5' | awk '{ print ($1 < 10), ($1 "" < 10) }'
1 0
$ awk 'BEGIN { x = "5"; y = "10"; print (x < y) }'
0
$ awk 'BEGIN { "echo 5" | getline x; print (x < 10) }'
1
Constants in the program, like "5", are always strings. Adding "" to a field turns it into a string, and adding + 0 turns it back into a number.
Conversions. A string becomes the number at its start, or 0:
$ echo '3 apples' | awk '{ print $0 + 0, $1 + 0 }'
3 3
$ awk 'BEGIN { a = "3x"; b = 4; print a + b, a b, a * 2 }'
7 3x4 6
Hexadecimal is not the same everywhere. mawk reads 0x1A as 26, gawk reads it as 0 (gawk's strtonum() does it, or the --non-decimal-data option):
$ echo '0x1A' | gawk '{ print $1 + 0 }'
0
$ echo '0x1A' | mawk '{ print $1 + 0 }'
26
$ gawk 'BEGIN { print strtonum("0x1A"), strtonum("017"), strtonum("12abc") }'
26 15 12
Number output. print uses OFMT (%.6g) for numbers that are not whole, so a total with decimals can come out in scientific notation. printf does not:
$ awk 'BEGIN { print 12345678.9, 1234567.891, 100000*100000, 0.1*3, 3.0 }'
1.23457e+07 1.23457e+06 10000000000 0.3 3
$ echo '1234567.891 2' | awk '{ s += $1 } END { print s; printf "%.2f\n", s }'
1.23457e+06
1234567.89
Whole numbers print in full, up to 2^53. Past that, doubles lose precision:
$ awk 'BEGIN { print 2^53, 2^53 + 1, 2^31, 123456789012 }'
9007199254740992 9007199254740992 2147483648 123456789012
$ awk 'BEGIN { print 0.1 + 0.2, (0.1+0.2 == 0.3), 1/3 }'
0.3 0 0.333333
OFMT and CONVFMT change the format of printed numbers and of number to string conversions:
$ awk 'BEGIN { OFMT="%.2f"; print 3.14159, 3.14159 ""; CONVFMT="%.2f"; print 3.14159 "" }'
3.14 3.14159
3.14
printf with %d truncates, and large values differ between awks:
$ awk 'BEGIN { printf "%.2f %d %.0f %.0f %.0f\n", 12345678.9, 3.99, 2.5, 3.5, 0.5 }'
12345678.90 3 2 4 0
$ awk 'BEGIN { printf "%d\n", 2^64 }'
18446744073709551616
$ mawk 'BEGIN { printf "%d\n", 2^64 }'
18446744073709551615
$ original-awk 'BEGIN { printf "%d\n", 2^64 }'
-9223372036854775808
(gawk, mawk and the original awk gave three different answers for the same line.) %.0f rounds halves to even: 2.5 prints 2 and 3.5 prints 4.
Arithmetic: + - * / % ^ (and ** in gawk, the original awk and not mawk). % keeps the sign of the left side, and int() cuts towards zero:
$ awk 'BEGIN { print 7%3, -7%3, int(-3.9), int(3.9), 7/2 }'
1 -1 -3 3 3.5
$ awk 'BEGIN { print -3 % 2, 2 ^ 3 ^ 2, 2 ** 3, -2 ^ 2, !0, !1, !"a", !"" }'
-1 512 8 -4 1 0 0 1
^ is right-associative (2^3^2 is 2^9), and binds tighter than a unary minus, so -2^2 is -4. The empty string and 0 are false, everything else is true. A field that holds a number is false when the number is 0 (even 0.0), but the constant string "0" in a program is a string, and a non-empty string is true:
$ echo '0.0' | awk '{ print ($1 == 0), ($1 == "0"), ($1 ? "true" : "false") }'
1 0 false
$ awk 'BEGIN { if ("0") print "const-true"; else print "const-false" }'
const-true
Division by zero is fatal in gawk and in the original awk. mawk prints inf:
$ awk 'BEGIN { x = 1/0 }'
awk: cmd. line:1: error: division by zero attempted
$ mawk 'BEGIN { x = 1/0; print x }'
inf
Concatenation has no operator, which sets a trap with a minus sign. These three lines show it:
$ awk 'BEGIN { print 1 " " -1; print 1 -1; print 1 " " (-1) }'
1-1
0
1 -1
The first line printed 1-1 (the minus bound as a subtraction on the string), the second is plain subtraction, and parentheses give what was meant.
BusyBox awk (this build) has no math support: the ^ operator and sin() fail with "Math support is not compiled in", while int() and length() work.
printf and output: print, files, pipes and system()
print adds the output separators (OFS between items, ORS at the end). printf prints exactly what the format says, with no newline added:
printf format, value, value ...
%s string %d integer %i integer %f decimal %e scientific %x hex %o octal %c character %% a percent sign
%-15s left-align in 15 columns %8.2f width 8, 2 decimals %05.1f zero-pad %*d width taken from an argument
$ awk 'NR>1 { printf "%-6s %-7s %3d %6.2f\n", $1, $2, $3, $3*$4 }' sales.txt
north apples 10 15.00
south pears 4 9.00
north pears 7 15.75
east apples 3 4.50
south apples 12 16.80
east plums 0 0.00
$ awk 'BEGIN { s = sprintf("%05.1f|%-5s|%5s|%x|%o|%c|%e", 3.14159, "ab", "cd", 255, 8, 65, 12345.678); print s }'
003.1|ab | cd|ff|10|A|1.234568e+04
$ awk 'BEGIN { printf "%5.1f%%\n", 45.678 }'
45.7%
$ awk 'BEGIN { printf "%*d|%-*d|\n", 5, 42, 5, 42 }'
42|42 |
%c prints the character with that number, or the first character of a string:
$ awk 'BEGIN { printf "%c%c|\n", "hello", 66 }'
hB|
Never use the data as the format. A % in the line is taken as a format, which breaks or prints the wrong thing:
$ echo '50% done' | awk '{ printf $0 "\n" }'
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: not enough arguments to satisfy format string
$ echo '50% done' | awk '{ printf "%s\n", $0 }'
50% done
Too few arguments is fatal in gawk, and a run-time error in mawk:
$ mawk 'BEGIN { printf "%s %s\n", "only-one" }'
mawk: run time error: not enough arguments passed to printf("%s %s
")
Redirection. print > "file" truncates the file once and then keeps it open, appending; >> appends to an existing file; | sends the output to a command. The target is an expression, so a file per group is easy:
$ awk 'NR>1 { print > ("/tmp/by-" $1 ".txt") }' sales.txt; cat /tmp/by-east.txt
east apples 3 1.50
east plums 0 3.10
$ awk 'NR>1 { print $2 | "sort -u" }' sales.txt
apples
pears
plums
$ awk 'BEGIN { print "to stderr" > "/dev/stderr" }' 2>/dev/null; echo '(hidden)'
(hidden)
close() ends a file or a command. It is needed to read a file back, to run the same command again, and to make a pipe finish (a sort prints only when its input is closed) before the next line of output:
$ awk 'NR==1 { print; next } { print | "sort -k2" } END { close("sort -k2"); print "-- total rows", NR-1 }' sales.txt
region product qty price
north apples 10 1.50
south apples 12 1.40
east apples 3 1.50
south pears 4 2.25
north pears 7 2.25
east plums 0 3.10
-- total rows 6
The > trap. In print a > b the > is a redirection, so this wrote a file called 2, not a comparison. Put the comparison in parentheses:
$ awk 'BEGIN { print 1 > 2 }'; ls 2 && cat 2; rm -f 2
2
1
$ awk 'BEGIN { print (1 > 2) }'
0
system() runs a shell command and returns its exit status. awk's own buffered output may come out after the command's, so flush first when the order matters:
$ awk 'BEGIN { r = system("exit 3"); print r }'
3
$ awk 'BEGIN { printf "x"; fflush(); system("echo from-shell") }' | cat
xfrom-shell
Buffering. When awk's output goes to a pipe (tail -f log | awk ... | next), it is held in a buffer, and the next program sees nothing until the buffer fills. The lab sent "a", waited two seconds, sent "b", and printed when each line arrived at the other end:
program a arrived b arrived
mawk '{ print }' +2s +2s (buffered)
gawk '{ print }' +2s +2s (buffered)
mawk -W interactive '{ print }' +0s +2s
mawk '{ print; fflush() }' +2s +3s (the first line still waited)
gawk '{ print; fflush() }' +0s +2s
original-awk '{ print }' +2s +2s (buffered)
busybox awk '{ print }' +0s +2s
Use fflush() after print in gawk, mawk -W interactive, or stdbuf-like tools when a stream must show up at once.
Regular expressions in awk
awk uses POSIX extended regular expressions (ERE): + ? | ( ) { } are special without a backslash, and a literal one needs \ or brackets. A regex goes between slashes, or in a string (a dynamic regex).
. any character * zero or more + one or more ? zero or one | or
^ start of the string $ end [abc] a set [^abc] not in the set ( ) group {n,m} repeat
$ echo 'aaa' | awk '/a{3}/ { print "yes" } !/a{4}/ { print "no4" }'
yes
no4
$ echo 'foo123' | awk '{ print match($0, /[0-9]+/), RSTART, RLENGTH, substr($0, RSTART, RLENGTH) }'
4 4 3 123
The {n} interval worked in all four awks of the lab. Character classes such as [[:digit:]], [[:space:]], [[:alpha:]], [[:upper:]] and [[:punct:]] work everywhere:
$ printf 'a b9\n' | mawk '{ print match($0, /[[:digit:]]/), match($0, /(a|b)+9/) }'
4 3
The shortcuts differ. The lab ran match() with each:
shortcut gawk mawk original-awk busybox
\s \w yes no no yes
\y (word edge) yes no no no
\d no no no no
gawk warns that \d is not a known operator. Use [[:space:]], [[:alnum:]_] and [[:digit:]] and the line runs on every awk. There are no backreferences in the pattern and no lookahead (ERE), in any of them:
$ printf 'abab\n' | gawk '{ print match($0, /(ab)\1/) }'
0
(0 means no match: \1 was not treated as a repeat.) The same line gave 0 in mawk, the original awk and busybox awk.
Case. Only gawk has IGNORECASE (and BusyBox, in the lab). The portable way is to lower-case the text:
$ echo 'Foo' | gawk 'BEGIN{IGNORECASE=1} /foo/ { print "ci" }'
ci
$ echo 'Foo' | mawk 'BEGIN{IGNORECASE=1} /foo/ { print "ci" } END{print "done"}'
done
$ echo 'Foo' | awk 'tolower($0) ~ /foo/ { print "tolower ok" }'
tolower ok
Dynamic regex. A string on the right of ~ is turned into a regex, so backslashes are processed twice: once as a string, once as a regex. Write \\. for a literal dot:
$ echo 'a.b' | awk '$0 ~ "\\." { print "escaped" }'
escaped
$ awk 'BEGIN { s = "a.b.c"; n = gsub(".", "-", s); print n, s }'
5 -----
$ awk 'BEGIN { s = "a.b.c"; n = gsub(/\./, "-", s); print n, s }'
2 a-b-c
$ awk 'BEGIN { s = "a.b.c"; n = gsub("[.]", "-", s); print n, s }'
2 a-b-c
In the first of the three, the dot in the string was a regex and replaced every character. A bracket, [.], is the easiest way to write a literal special character in a string, and index() finds plain text with no regex at all:
$ awk 'BEGIN { s = "a.b.c"; i = index(s, "."); print substr(s, 1, i-1), substr(s, i+1) }'
a b.c
match() returns the position (0 for none) and sets RSTART and RLENGTH. Only gawk can return the groups, with a third argument:
$ echo 'foo123bar45' | gawk '{ match($0, /([a-z]+)([0-9]+)/, m); print m[1], m[2] }'
foo 123
$ echo 'foo123bar45' | mawk '{ match($0, /([a-z]+)([0-9]+)/, m); print m[1] }'
mawk: line 1: syntax error at or near ,
Without gawk, take the text with substr(): substr($0, RSTART, RLENGTH).
sub() and gsub() change the first match or all matches, in $0 or in a variable, and return how many they changed. In the replacement, & stands for the matched text, and a literal ampersand is written \\& inside the awk string:
$ echo 'a b c' | awk '{ n = gsub(/ /, "_"); print n, $0 }'
2 a_b_c
$ echo 'hello' | awk '{ sub(/l+/, "[&]"); print }'
he[ll]o
$ echo 'hello' | awk '{ sub(/l+/, "[\\&]"); print }'
he[&]o
$ echo 'abc' | awk '{ gsub(//, "-"); print }'
-a-b-c-
$ echo 'a b' | awk '{ gsub(/[[:space:]]+/, ""); print }'
ab
Only gawk has gensub(), which can pick the nth match and use groups (\1) in the replacement:
$ echo 'a-b-c' | gawk '{ print gensub(/-/, "+", 2) }'
a-b+c
String functions
length(s) number of characters (of $0 if there is no argument; of an array in all four awks of the lab)
substr(s, start, n) the part of s from start (counting from 1), n characters long, or to the end
index(s, t) position of t in s, 0 if not found (plain text, no regex)
split(s, a, sep) split s into the array a on sep (a regex, or a single character); returns the count
sub(re, repl, s) gsub(...) replace the first / every match in s (default $0)
match(s, re) position of the first match, sets RSTART and RLENGTH
sprintf(fmt, ...) like printf, returns the string
tolower(s) toupper(s)
$ awk 'BEGIN { print index("hello", "ll"), index("hello", "z"), length("hello"), toupper("abc") tolower("DEF") }'
3 0 5 ABCdef
$ awk '{ print toupper(substr($1,1,1)) substr($1,2) }' <<< 'word'
Word
$ awk 'BEGIN { print length(12345), length(1/3), substr(123456, 2, 3), index(12345, 34) }'
5 8 234 3
substr(): the start counts from 1. A start below 1, or a length that runs past the end, is handled differently by each awk. Keep the start at 1 or more when the code has to be portable:
$ awk 'BEGIN { print substr("hello", 2, 3), substr("hello", 2), substr("hello", 2, 100) }'
ell ello ello
substr("hello", 0, 2) and substr("hello", -1, 3), the lab output of each awk:
awk substr("hello", 0, 2) substr("hello", -1, 3)
mawk he hell
gawk he hel
original-awk he hel
busybox awk he hel
split() fills an array (numbered from 1) and returns how many pieces. A single space as separator means any run of blanks, a regex is allowed, an empty string gives no pieces, and an empty separator splits into characters:
$ awk 'BEGIN { n = split("a,b,c", arr, ","); print n, arr[1], arr[3] }'
3 a c
$ awk 'BEGIN { n = split("a1b22c", arr, /[0-9]+/); print n, arr[1], arr[2], arr[3] }'
3 a b c
$ awk 'BEGIN { n = split(" a b ", arr); print n, arr[1] }'
2 a
$ awk 'BEGIN { n = split("", arr); print n, length(arr) }'
0 0
$ echo 'abc' | mawk '{ n = split($0, ch, ""); print n, ch[2] }'
3 b
The last form split abc into characters in mawk, gawk, the original awk and BusyBox.
Parsing a date or a key=value pair is split() plus array use:
$ echo 'k=v; x=y' | awk -F'[=;] *' '{ print $1, $2, $3, $4 }'
k v x y
Other functions: int(x), sqrt(x), exp(x), log(x), sin(x), cos(x), atan2(y, x), rand(), srand(). rand() is the same sequence on every run in gawk, and different on every run in mawk (it seeds itself from the clock):
$ gawk 'BEGIN { print rand(), rand() }'; gawk 'BEGIN { print rand() }'
0.924046 0.593909
0.924046
$ mawk 'BEGIN { print rand() }'; mawk 'BEGIN { print rand() }'
0.0146918
0.833972
$ awk 'BEGIN { srand(1); a = rand(); srand(1); print (a == rand()) }'
1
Call srand(seed) with a fixed seed for a repeatable sequence on every awk, and srand() (no argument) when you want a different one each run.
Time. gawk has systime(), strftime() and mktime(), and mawk 1.3.4 and BusyBox have them as well; the original awk does not. The result of strftime depends on the time zone unless you pass 1 as the third argument for UTC:
$ gawk 'BEGIN { print strftime("%Y-%m-%d %H:%M", 1760659200, 1) }'
2025-10-17 00:00
$ mawk 'BEGIN { print strftime("%Y-%m-%d", 1760659200, 1); print (systime() > 1700000000) }'
2025-10-17
1
Arrays
Arrays in awk are associative: the index is a string (a number is converted to its string), and the array grows as you use it. There is no declaration.
$ awk 'NR>1 { n[$2]++ } END { for (p in n) print p, n[p] }' sales.txt | sort
apples 3
pears 2
plums 1
$ awk 'NR>1 { sum[$1] += $3 * $4 } END { for (r in sum) print r, sum[r] }' sales.txt | sort
east 4.5
north 30.75
south 25.8
The for (key in array) order is not defined. gawk happened to give 1 2 3 4 5 and mawk gave 3 5 2 1 4 for the same array, so sort the output (above) or use gawk's sorting:
$ gawk 'BEGIN { for (i=1;i<=5;i++) a[i]=i; for (k in a) printf "%s ", k; print "" }'
1 2 3 4 5
$ mawk 'BEGIN { for (i=1;i<=5;i++) a[i]=i; for (k in a) printf "%s ", k; print "" }'
3 5 2 1 4
$ gawk 'BEGIN { a["b"]=1; a["a"]=2; a["c"]=3; PROCINFO["sorted_in"]="@ind_str_asc"; for (k in a) printf "%s ", k; print "" }'
a b c
$ gawk 'BEGIN { a["x"]=3; a["y"]=1; a["z"]=2; PROCINFO["sorted_in"]="@val_num_desc"; for (k in a) printf "%s=%s ", k, a[k]; print "" }'
x=3 z=2 y=1
$ gawk 'BEGIN { n=split("30 5 100", a); asort(a); for (i=1;i<=n;i++) printf "%s ", a[i]; print "" }'
5 30 100
$ mawk 'BEGIN { n=split("30 5 100", a); asort(a) }'
mawk: line 2: function asort never defined
On mawk, print the lines to a pipe and let sort do it: print k, a[k] | "sort -k2 -nr".
Testing for a key. The in operator tests without creating the key, but reading a missing element creates it (as an empty string), which changes length() and what a later loop sees:
$ awk 'BEGIN { a["x"]=1; print ("x" in a), ("y" in a), length(a); if (a["y"] == "") print "y read"; print ("y" in a), length(a) }'
1 0 1
y read
1 2
Use if (k in a) when you only want to ask. delete removes one element, delete a removes all of them (this form is in gawk, mawk, the original awk and BusyBox):
$ awk 'BEGIN { a[1]=1; a[2]=2; delete a[1]; print length(a); delete a; print length(a) }'
1
0
The index is a string, so "01" and 1 are different keys. A number is converted to a string before it is used, so the number 1.0 is the same key as 1, but the string "1.0" is not:
$ awk 'BEGIN { a["01"]=1; a[1]=2; print length(a) }'
2
$ awk 'BEGIN { a[1]="x"; print (1.0 in a), ("1.0" in a) }'
1 0
Multi-dimensional arrays join the parts with SUBSEP (the character \034). Test with (i,j) in a:
$ awk 'BEGIN { a[1,2]="x"; for (k in a) { split(k, p, SUBSEP); print p[1], p[2] }; if ((1,2) in a) print "in" }'
1 2
in
Useful patterns.
Remove duplicates, keeping the first and the order:
$ awk '!seen[$2]++' sales.txt
region product qty price
north apples 10 1.50
south pears 4 2.25
east plums 0 3.10
Reverse the lines (like tac):
$ awk '{ a[NR]=$0 } END { for (i=NR;i>0;i--) print a[i] }' sales.txt | head -2
east plums 0 3.10
south apples 12 1.40
Transpose rows and columns (ids.txt has three rows of two columns):
$ awk '{ for (i=1;i<=NF;i++) m[i,NR]=$i; if (NF>nf) nf=NF } END { for (i=1;i<=nf;i++) { row=m[i,1]; for (j=2;j<=NR;j++) row=row OFS m[i,j]; print row } }' ids.txt
ann ed zed
1000 1001 1
Count by key, then sort by the count:
$ awk '{ n[$1]++ } END { for (k in n) print n[k], k }' access.log | sort -rn | head -3
3 10.0.1.11
2 10.0.3.5
1 10.0.2.31
An array passed to a function is passed by reference, so the function can fill it (see the section on functions). Storing every line of a file in an array uses memory in proportion to the file (see speed).
Control flow, functions and getline
Statements are separated by a newline or a semicolon, and braces group them. The control statements are the same as in C:
if (cond) stmt else stmt while (cond) stmt do stmt while (cond)
for (init; cond; step) stmt for (key in array) stmt
break continue next (skip to the next record) nextfile (skip to the next file) exit [status] return value
$ awk '{ if ($3 > 5) print $1, "high"; else print $1, "low" }' sales.txt | head -3
region high
north high
south low
$ awk 'BEGIN { for (i = 1; i <= 5; i++) { if (i == 2) continue; if (i == 4) break; print i } }'
1
3
$ awk 'BEGIN { while (i++ < 3) print i; do { j++ } while (j < 5); print j; for (;;) { if (++k > 2) break }; print k }'
1
2
3
5
3
next ends the work on this record. It is how you write "skip header lines" or "handle this case and nothing else":
$ awk 'NR % 2 == 0 { next } 1' sales.txt | head -2
region product qty price
south pears 4 2.25
exit stops the program. END still runs, and exit inside END stops at once. The status is that of the last exit, so a script can test for a match:
$ awk '{ exit 2 } END { print "END still runs"; exit }' app.log; echo rc=$?
END still runs
rc=2
$ awk 'BEGIN { exit } END { print "END after BEGIN exit" }'
END after BEGIN exit
$ awk '/ERROR/ { found=1 } END { exit !found }' app.log; echo rc=$?
rc=0
$ awk '/NOPE/ { found=1 } END { exit !found }' app.log; echo rc=$?
rc=1
An exit status in an END with no value keeps the earlier one (the first example kept 2). Use it as grep does, to use awk in an if.
User-defined functions. A function can be defined anywhere in the program. The parameters are local and passed by value, except arrays, which are passed by reference. Extra parameters work as local variables, and the convention is to write them after a gap:
$ awk 'function max(a, b) { return a > b ? a : b } BEGIN { print max(3, 9), max("abc", "abd") }'
9 abd
$ awk 'function f(n, i, s) { for (i=1;i<=n;i++) s = s i; return s } BEGIN { i="outer"; print f(3), i }'
123 outer
$ awk 'function f(n, i) { for (i=1;i<=n;i++) ; return i } BEGIN { i="outer"; print f(3), i }'
4 outer
$ awk 'function fill(arr) { arr["k"]="v" } BEGIN { fill(z); print z["k"] }'
v
$ awk 'function f(x) { x = 5 } BEGIN { y = 1; f(y); print y }'
1
$ awk 'function fact(n) { return n<=1 ? 1 : n*fact(n-1) } BEGIN { print fact(10) }'
3628800
A variable used in a function without being a parameter is global, so it can overwrite the caller's. Declare the locals as extra parameters. Recursion works. func is a gawk, original-awk and BusyBox synonym for function (not mawk).
Calling a function: no space is allowed between the name and the bracket, and the awks disagree:
$ awk 'function f(x) { return x } BEGIN { print f (1) }'
awk: cmd. line:1: error: function `f' called with space between name and `(',
or used as a variable or an array
$ mawk 'function f(x) { return x } BEGIN { print f (1) }'
1
$ original-awk 'function f(x) { return x } BEGIN { print f (1) }'
original-awk: can't read value of f; it's a function.
getline reads input from somewhere other than the main loop. The forms:
getline the next record into $0 (updates NR, NF)
getline var the next record into var
getline < "file" the next line of a file into $0
getline var < "file" ... into var
"command" | getline the next line of a command's output into $0
"command" | getline var
$ awk 'NR==1 { getline; print "after getline:", $0, NR }' sales.txt
after getline: north apples 10 1.50 2
$ awk 'BEGIN { while ((getline line < "passwd.txt") > 0) n++; print n }'
5
$ awk 'BEGIN { "date +%Y" | getline y; close("date +%Y"); print (y > 2000) }'
1
$ awk 'BEGIN { while (("echo a; echo b" | getline x) > 0) print "got", x }'
got a
got b
getline returns 1 for a line, 0 at the end and -1 on an error. The test must be > 0. A file that does not exist returns -1 every time, so while (getline line < "file") never ends. The lab stopped it with timeout after 3 seconds:
$ awk 'BEGIN { r = (getline line < "nonexistent"); print r }'
-1
$ timeout 3 awk 'BEGIN { while (getline line < "nofile") n++; print n }' ; echo "rc=$?"
rc=124
$ timeout 3 awk 'BEGIN { while ((getline line < "nofile") > 0) n++; print n + 0 }'; echo rc=$?
0
rc=0
A command or file stays open after it is read, so a second read of the same command returns nothing until close() is called:
$ awk 'BEGIN { cmd = "echo hi"; cmd | getline a; cmd | getline b; print a "|" b "|" }'
hi||
$ awk 'BEGIN { cmd = "echo hi"; cmd | getline a; close(cmd); cmd | getline b; print a "|" b }'
hi|hi
Several files: NR, FNR, joins and ranges of input
NR keeps counting across files and FNR starts again for each one. FILENAME is the file being read. When there is more than one file, the two lets you tell them apart:
$ printf 'ann 1000\ned 1001\nzed 1\n' > ids.txt
$ printf 'ann Ann Lee\ned Ed Kay\nkim Kim Fox\n' > names.txt
$ awk 'FNR==1 { print FILENAME }' ids.txt names.txt
ids.txt
names.txt
The join idiom. NR==FNR is true only while the first file is being read, so the first file fills an array and next skips the rest of the rules; the second file then looks things up:
$ awk 'NR==FNR { id[$1]=$2; next } $1 in id { print $1, id[$1], $2 }' ids.txt names.txt
ann 1000 Ann
ed 1001 Ed
Lines in the second file with no match in the first (an anti-join):
$ awk 'NR==FNR { seen[$1]; next } !($1 in seen)' ids.txt names.txt
kim Kim Fox
The empty file trap. If the first file is empty, NR and FNR stay equal while the second file is read, so the second file is taken as the first one. The join then printed nothing at all, with no error:
$ : > empty.txt
$ awk 'NR==FNR { a[$1]; next } { print FILENAME, $1, ($1 in a) }' empty.txt ids.txt
$ awk 'FILENAME == ARGV[1] { a[$1]; next } { print FILENAME, $1, ($1 in a) }' empty.txt ids.txt
ids.txt ann 0
ids.txt ed 0
ids.txt zed 0
Comparing FILENAME with ARGV[1] is safe when the first file may be empty. gawk also has ARGIND (the index of the current file), and it gave the same result:
$ gawk 'ARGIND == 1 { a[$1]; next } { print FILENAME, $1, ($1 in a) }' empty.txt ids.txt
ids.txt ann 0
ids.txt ed 0
ids.txt zed 0
(the lines are not in the empty file, so each answer is 0.)
nextfile stops reading the current file and goes on with the next (gawk, mawk, the original awk and BusyBox all took it):
$ awk 'FNR==2 { nextfile } { print FILENAME, $1 }' ids.txt names.txt
ids.txt ann
names.txt ann
gawk has BEGINFILE and ENDFILE, which run before and after each file, and ERRNO to detect a file that cannot be opened. That is how to skip a missing file without an error:
$ gawk 'BEGINFILE { print "start", FILENAME } ENDFILE { print "end", FILENAME, FNR }' ids.txt names.txt
start ids.txt
end ids.txt 3
start names.txt
end names.txt 3
$ gawk 'BEGINFILE { if (ERRNO) { print "skip", FILENAME; nextfile } } END { print NR }' ids.txt nope.txt names.txt 2>&1
skip nope.txt
6
Without BEGINFILE a missing file is fatal in awk, with status 2 (not in BusyBox, which gives 1):
$ awk '{ print }' no-such-file; echo rc=$?
awk: fatal: cannot open file `no-such-file' for reading: No such file or directory
rc=2
A file name that contains an equal sign. A command line argument name=value is an assignment, not a file, so awk never opens a file called x=1 and, with no other file, reads standard input. Add a path to make it a file:
$ printf 'l1\n' > 'x=1'
$ awk '{ print "x is", x, $0 }' x=1 'x=1'
$ awk '{ print "x is", x, $0 }' ./x=1
x is l1
The first command printed nothing: both arguments were assignments, so awk waited for input. Use ./ in front of names that come from a variable or a glob.
Records, separators and real-world input: RS, CSV, CRLF, Unicode
Many guides use gawk features without saying so. On Ubuntu's default mawk they give an error or do nothing. What the lab found:
feature gawk mawk original-awk busybox
length(array), delete array yes yes yes yes
nextfile, fflush() yes yes yes yes
systime(), strftime(), mktime() yes yes no yes
toupper/tolower yes yes yes yes
\s and \w in a regex yes no no yes
gensub() yes no no yes
match(s, re, array) yes no no no
asort(), asorti(), PROCINFO yes no no no
IGNORECASE yes no no yes
switch / case yes no no no
BEGINFILE / ENDFILE yes no no no
strtonum(), typeof() yes no no no
and(), or(), xor(), lshift() yes no no yes (and)
RT, FPAT, FIELDWIDTHS yes no no RT yes
** (power) yes no yes no
printf %*d yes yes yes no
func (for function) yes no yes yes
RS as a regex yes yes yes yes
-e 'program', @include yes no no -e yes
Working examples of the gawk-only features:
$ gawk 'BEGIN { print and(12, 10), or(12, 10), xor(12, 10), lshift(1, 4), 2 ** 3 }'
8 14 6 16 8
$ gawk 'BEGIN { switch (3) { case 1: print "one"; break; case 3: print "three"; break; default: print "other" } }'
three
$ gawk 'BEGIN { x[1]; print typeof(x), typeof(1), typeof("a") }'
array number string
$ gawk 'BEGIN { print typeof(y) }'
untyped
$ gawk -f inc.awk (inc.awk is: @include "lib.awk" and a BEGIN rule that calls f(21))
42
$ mawk -f inc.awk
mawk: 1: unexpected character '@'
Edit a file in place (gawk 4.1 and later), with an optional backup. The in-place mode is a plug-in, loaded with -i:
$ printf 'one old\ntwo old\n' > ip.txt
$ gawk -i inplace '{ gsub(/old/, "new") } 1' ip.txt; cat ip.txt
one new
two new
$ gawk -i inplace -v inplace::suffix=.bak '{ gsub(/new/, "newer") } 1' ip.txt; cat ip.txt ip.txt.bak
one newer
two newer
one new
two new
Put the rule 1 at the end (it prints the line). Without it nothing is printed, and the file is rewritten empty, as with sed -n -i: the lab left a 0-byte file. A file that does not exist is an error (rc=2) and nothing is created. mawk has no -i, so with it use a temporary file: awk '...' file > file.tmp && mv file.tmp file (this replaces the file and its permissions; see the sed guide's section on -i).
Time and dates:
$ gawk 'BEGIN { t = mktime("2026 10 17 12 30 00"); print (t > 0) }'
1
$ mawk 'BEGIN { print mktime("2026 10 17 12 30 00") }'
1792240200
(mktime uses the local time zone of the machine; the lab was on UTC.)
Tools that help to see what a program does (gawk only):
--lint warns about code that is probably wrong
--profile writes a copy of the program with a run count by each line
--pretty-print writes the program neatly formatted
--dump-variables prints every variable at the end
--sandbox blocks system(), pipes, redirection and getline from files (see the shell scripts section)
--posix, --traditional switch off the gawk extras so that you can check that code is portable
$ gawk --posix 'BEGIN { print toupper("a"); print gensub(/a/, "b", 1, "a") }'
gawk: cmd. line:1: warning: regexp constant for parameter #1 yields boolean value
A
gawk: cmd. line:1: fatal: function `gensub' not defined
$ gawk --traditional 'BEGIN { print systime() }'
gawk: cmd. line:1: fatal: function `systime' not defined
--posix and --traditional are a quick way to find out whether a script uses an extension: they stop at the first one. A script that is meant to run on mawk can be tested by running it with mawk itself.
Awk programs in files, and scripts
When a program is longer than a line, put it in a file. Comments start with #, newlines separate statements (a trailing backslash continues a line, and so does a line ending in a comma, &&, || or {), and a program may start with a shebang:
$ cat report.awk
#!/usr/bin/awk -f
# sales report: totals per region, then a grand total
BEGIN {
printf "%-8s %5s %9s\n", "region", "units", "revenue"
}
NR > 1 {
units[$1] += $3
revenue[$1] += $3 * $4
total += $3 * $4
}
END {
for (r in units)
printf "%-8s %5d %9.2f\n", r, units[r], revenue[r] | "sort"
close("sort")
printf "%-8s %5s %9.2f\n", "total", "", total
}
$ chmod +x report.awk; ./report.awk sales.txt
region units revenue
east 3 4.50
north 17 30.75
south 16 25.80
total 61.05
$ awk -f report.awk sales.txt | tail -1
total 61.05
The header is printed by BEGIN, the rows are summed by one rule that skips line 1, and END prints the groups through sort (a pipe) and then the total. The close("sort") makes sort finish before the total line is printed, so the total comes last.
Run a program from a file against several inputs with the same command, or keep a library of functions and combine files with several -f options (the -f lib.awk -f prog.awk form worked; -e is the gawk way to mix a file with text):
$ awk -f lib.awk -f prog.awk x
from script x
$ gawk -f lib.awk -e 'BEGIN { print f(4) }'
8
Statements and line breaks. A newline ends a statement, and so does a semicolon. Awk allows a newline after else, after && and || and after a comma, but not in the middle of an expression, so a long condition continues with a backslash or after an operator:
$ awk 'BEGIN { print 1 +\
2 }'
3
$ awk 'BEGIN { if (1) print "a"
else print "b" }'
a
Writing a script is also a chance to use the syntax that does not fit on one line: a function with local variables, a multi-line pipeline into sort, arrays of arrays (gawk) and so on.
Passing the file name to the program, rather than the other way round, is useful in shell loops:
$ for f in sales.txt; do awk -v name="$f" 'END { print name, NR }' "$f"; done
sales.txt 7
Using awk safely in shell scripts
Most trouble comes from where the data goes: into the program text, into a command, or into a file name. The lab ran each of these.
1. Keep the program in single quotes, and pass values with -v or the environment. If you use double quotes the shell expands $1 and the program is not what you wrote:
$ echo 'ann' | awk '{ print $1 }'
ann
$ echo 'ann' | awk "{ print $1 }"
awk: cmd. line:1: { print echo 'ann' | awk "{ print $1 }" }
awk: cmd. line:1: ^ invalid char ''' in expression
(in that second command the shell replaced $1 with its own first argument, which was the command's text, so awk received a broken program.)
2. -v processes backslashes in the value, and ENVIRON does not. A value with \n in it is turned into a newline by -v. Use the environment for text you do not control:
$ U='a\nb'; awk -v u="$U" 'BEGIN { print u }'
a
b
$ U='a\nb' awk 'BEGIN { print ENVIRON["U"] }'
a\nb
$ U='ann'; awk -v u="$U" '$1 == u' ids.txt
ann 1000
3. Never put data into a command that awk runs. system() and "cmd" | getline and print | "cmd" all go through the shell. A line of a log or a file that contains a semicolon becomes code:
$ cat evil.txt
x; touch /tmp/pwned
ok
$ awk '{ system("echo " $0) }' evil.txt; ls /tmp/pwned
x
ok
/tmp/pwned
$ awk '{ print "echo", $0 | "sh" }' evil.txt >/dev/null; ls /tmp/pwned
/tmp/pwned
Both ran touch. Do not build commands from data. If you must, quote it (this is not shown in the lab and must be done with care), or use shell tools that take the data as arguments, such as xargs with -0 or -d '\n' and a quoted "$@".
4. A file name taken from data can point anywhere. The target of a redirection is whatever the field holds:
$ cat names2.txt
/tmp/owned-by-awk data
ok data
$ awk '{ print $2 > $1 }' names2.txt; ls -l /tmp/owned-by-awk ok
-rw-r--r-- 1 ubuntu ubuntu 5 Oct 18 00:58 /tmp/owned-by-awk
-rw-r--r-- 1 ubuntu ubuntu 5 Oct 18 00:58 ok
Build the name from a fixed directory and a checked part, for example print > ("out-" $1), and reject a name with a slash first: $1 !~ /\// . (With a slash in the data, awk looked for out-/tmp/owned-by-awk and stopped with an error, which is also what you want, but do not depend on it.)
5. gawk --sandbox blocks the dangerous parts, for programs you did not write. system(), redirection to files and pipes, and getline from a file or command are refused:
$ gawk --sandbox 'BEGIN { system("echo hi") }'
gawk: cmd. line:1: fatal: 'system' function not allowed in sandbox mode
$ gawk --sandbox 'BEGIN { print "x" > "/tmp/sb" }'
gawk: cmd. line:1: fatal: redirection not allowed in sandbox mode
$ gawk --sandbox 'BEGIN { "echo hi" | getline }'
gawk: cmd. line:1: fatal: redirection not allowed in sandbox mode
6. Quotes inside a program. A single quote cannot appear inside a single-quoted program. Use \047 (the octal code), a variable from -v, or close and reopen the shell quotes:
$ awk 'BEGIN { print "\047q\047" }'
'q'
$ awk -v q="'" 'BEGIN { print q "quoted" q }'
'quoted'
$ awk 'BEGIN { print "it'\''s" }'
it's
7. File names. A name with = in it is taken as an assignment (shown in the section on several files), so write ./name for names from a variable. A name that starts with - is read as an option by many tools; prefix it with ./ too.
8. Exit status. awk returns 0 for success. A syntax error or a missing file is 2 in mawk, the original awk and gawk's missing-file case; gawk gives 1 for a syntax error and for division by zero, BusyBox gives 1. Use exit in END for a yes/no test:
$ awk '/ERROR/ { found=1 } END { exit !found }' app.log; echo rc=$?
rc=0
A program that is syntactically valid but nonsense does nothing and exits with 0, which hides mistakes. This one has three words in a row; awk reads them as the concatenation of three empty variables, which is false:
$ awk 'bad syntax here' app.log; echo rc=$?
rc=0
Run --lint (gawk) over scripts that you keep.
Portability: mawk, gawk, the original awk and BusyBox
A one-liner that works on your laptop (gawk) may fail on a server (mawk) or in a container (BusyBox). The POSIX features work everywhere. The lab ran the same lines on all four, and the differences that mattered are these (the feature table in the gawk section has the rest):
1. The default awk. On Ubuntu a fresh server has mawk. On a desktop or after installing gawk it may be gawk (the alternatives link). In a script that needs gawk, call gawk by name, or check with: command -v gawk.
2. Unicode. mawk and BusyBox count bytes; gawk and the original awk count characters in a UTF-8 locale.
3. Regex shortcuts. \s and \w only exist in gawk and BusyBox. Use [[:space:]] and [[:alnum:]_] instead.
4. Sorting: gawk has asort and PROCINFO["sorted_in"]; elsewhere pipe to sort.
5. IGNORECASE, 3-argument match, gensub, strtonum, switch, BEGINFILE, -i inplace, @include, -e: gawk only. A script that uses one should start with a check, or say gawk in its first line.
6. Hex numbers: mawk reads 0x1A as 26, gawk as 0.
7. -v and ENVIRON differ over escapes (the same in every awk).
8. printf %d of a very large number and of 2^64: three different answers.
9. Division by zero: fatal in gawk, inf in mawk.
10. rand(): the same numbers each run in gawk, different each run in mawk.
11. substr() with a start below 1: the same for gawk, original awk and BusyBox; different in mawk.
12. A space between a function name and the bracket: an error in gawk and the original awk; fine in mawk and BusyBox.
13. -F'\t' is the portable tab. -Ft is not.
14. The order of for (k in a) is not defined, and differs between mawk and gawk.
15. BusyBox awk (this build) has no ^ or math functions; mawk has no ** or func.
mac OS and the BSDs ship the original awk (an older build than the lab's) and it behaves like the "original-awk" column. Not run in the lab: macOS awk itself, and Solaris or AIX awk.
How to write portable awk, in short: use POSIX features only, quote the program in single quotes, set separators in BEGIN or with -F, use [[:class:]] in regexes, test with mawk (the strictest in the lab) and with gawk --posix, and write gawk when you need gawk.
Errors and debugging
Each awk phrases errors differently. The lab gave a program with a missing closing brace to all four:
$ awk '{ print $1 ' app.log
awk: cmd. line:1: { print $1
awk: cmd. line:1: ^ unexpected newline or end of string
$ mawk '{ print $1 ' app.log
mawk: line 2: missing } near end of file
$ original-awk '{ print $1 ' app.log
original-awk: syntax error at source line 1
context is
{ print $1 >>> <<<
original-awk: illegal statement at source line 1
missing }
$ busybox awk '{ print $1 ' app.log
awk: cmd. line:1: Unexpected token
gawk shows the line with a caret under the problem; the others are shorter. A syntax error stops the program before it reads any input, with status 1 (gawk, BusyBox) or 2 (mawk, the original awk).
Other messages you will see:
$ awk 'BEGIN { print substr("abc") }'
awk: cmd. line:1: BEGIN { print substr("abc") }
awk: cmd. line:1: ^ 1 is invalid as number of arguments for substr
$ mawk 'BEGIN { print substr("abc") }'
mawk: line 1: wrong number of arguments in call to substr
$ awk '/[/ { print }' app.log
awk: cmd. line:1: /[/ { print }
awk: cmd. line:1: ^ unterminated regexp
$ mawk '/[/ { print }' app.log
mawk: line 1: runaway regular expression /[/ { print ...
$ awk 'BEGIN { print log(-1), sqrt(-1), int("abc") }'
awk: cmd. line:1: warning: log: received negative argument -1
awk: cmd. line:1: warning: sqrt: received negative argument -1
-nan -nan 0
$ mawk '{ print $-1 }' app.log
mawk: run time error: negative field index $-1
FILENAME="app.log" FNR=1 NR=1
Mistakes that give no error. These are valid awk, so nothing warns you: a variable with a typo (it is a new empty variable), = in a condition, a > in a print that writes a file, a header line that is read as data, FS set in the main rule, and a nonsense program of three words. gawk --lint finds some of them:
$ gawk --lint 'BEGIN { print undefinedvar; x = 1 } { print $1 }' /dev/null
gawk: cmd. line:1: warning: reference to uninitialized variable `undefinedvar'
$ gawk --lint 'BEGIN { if (x = 1) print "assign in cond" }'
gawk: cmd. line:1: warning: assignment used in conditional context
assign in cond
$ gawk --lint '{ print $1 > "/dev/null" } END { print total }' sales.txt
gawk: cmd. line:1: (FILENAME=sales.txt FNR=7) warning: reference to uninitialized variable `total'
gawk: (FILENAME=sales.txt FNR=7) warning: no explicit close of file `/dev/null' provided
Print to find out. print to standard error does not mix with the data: print "debug: " $1 > "/dev/stderr". Print the fields with a marker so that blanks are visible: print "[" $1 "][" $2 "]", and print NF.
gawk can show what it understood and where the time goes:
$ cat count.awk
{ n[$2]++ }
END { for (k in n) print k, n[k] }
$ gawk --profile=/dev/stdout -f count.awk sales.txt | tail -12
7 {
7 n[$2]++
}
# END rule(s)
END {
4 for (k in n) {
4 print k, n[k]
}
}
$ gawk --pretty-print=/dev/stdout -f count.awk
{
n[$2]++
}
END {
for (k in n) {
print k, n[k]
}
}
$ gawk --dump-variables=/dev/stdout 'BEGIN { a=1; b="x" }' | head -8
ARGC: 1
ARGIND: 0
ARGV: array, 1 elements
BINMODE: 0
CONVFMT: "%.6g"
ENVIRON: array, 10 elements
ERRNO: ""
FIELDWIDTHS: ""
The profile shows how many times each line ran: 7 records, and 4 distinct products in END. gawk has a debugger as well (gawk -D, with -f only), not run beyond starting it.
Exit status in the lab, for the same errors:
program mawk gawk original-awk busybox
syntax error 2 1 2 1
file not found 2 2 2 1
division by zero 0 1 2 1
Speed and big files
awk reads a line at a time, so memory does not grow with the file unless you store lines in an array. The lab used a log of 1,000,000 lines (63 MB), and timed several jobs. The times are indicative.
job mawk gawk original-awk busybox
count the values of field 3 0.23 s 0.28 s 0.77 s 0.96 s
count lines matching /ERROR/ 0.07 s 0.22 s 9.69 s 0.59 s
gsub on every line, add up length 0.17 s 0.65 s 11.48 s (not run)
For comparison in the same run: grep -c ERROR took 0.04 s, sed -n '/ERROR/p' 0.17 s and cut | sort | uniq -c on field 3 took 0.44 s. For a plain search grep is the fastest, so use awk when you need fields, arithmetic or a report.
mawk was the fastest awk in all three jobs, and the original awk was very slow on regex work (11.5 s against 0.17 s for the same gsub job). gawk is slower than mawk but still handles a million lines a second or less.
The locale changes gawk. The same gsub job on gawk took 0.65 s in the default locale, 0.80 s with LC_ALL=C.UTF-8 and 0.65 s with LC_ALL=C. A length() job took 0.21 s and 0.30 s. For ASCII logs LC_ALL=C is the faster setting.
Memory. Peak memory, measured by watching /proc while the job ran:
job mawk gawk
count lines, keep nothing 2.2 MB 3.6 MB
store every line in an array (63 MB) 139 MB 176 MB
Storing the file in an array costs two to three times the file size. When the answer needs only counts or sums, use an array of counters (the number of distinct keys, not the number of lines).
Speed tips:
1. Select early: put the cheap, restrictive pattern first, and use next so that later rules are skipped.
2. Avoid getline from a command and system() inside a loop; each starts a process. Run one command that does all the work, or read a file once and keep an array.
3. Close files you will not write again. awk keeps every file it writes open until you close it, and the system limits how many can be open. After a close(), print > truncates the file again, so use >> if you write to it after closing:
$ awk 'BEGIN { print "x" > "/tmp/cl.txt"; close("/tmp/cl.txt"); print "y" > "/tmp/cl.txt"; close("/tmp/cl.txt"); while ((getline l < "/tmp/cl.txt") > 0) print l }'
y
$ awk 'BEGIN { print "x" > "/tmp/cl.txt"; close("/tmp/cl.txt"); print "y" >> "/tmp/cl.txt"; close("/tmp/cl.txt"); while ((getline l < "/tmp/cl.txt") > 0) print l }'
x
y
4. Use mawk for plain field and number work when it is installed. Use gawk when you need its features.
5. Do a first pass with grep, and send only the lines you need to awk.
6. gawk: LC_ALL=C for ASCII data, or -b for bytes.
When not to use awk
awk is the right tool for text that is a table. It is the wrong one when the text has structure that is not columns, and the lab showed the failure each time:
CSV with quoted commas. -F, cuts a quoted field in two (shown above). gawk FPAT works for simple files only. Use a CSV tool or a language with a CSV library.
JSON. There are no columns, and a line break may be anywhere. Use jq (see the jq guide on this site).
YAML. Indentation is meaning, not columns. Use yq (see the yq guide).
XML and HTML. Use a parser (xmllint, xmlstarlet, python).
Editing a file in place. mawk has no in-place mode, and a file written by a redirect and renamed gets new permissions and ownership. Use sed -i for line edits, or gawk -i inplace if you have it.
A line edit that only needs a substitution. sed 's/old/new/' is shorter. awk is for when you also need a column or a sum.
Multi-line records that vary. Use RS="" for blank-line blocks; for anything more complicated, a script in Python or Perl is easier to read.
Floating point with money. awk uses doubles, so 0.1 + 0.2 is not exactly 0.3. For accounting use integers (cents), or bc / Python decimal.
Huge sorts and joins. Use sort and join (or a database); awk keeps arrays in memory.
For simple jobs the other tools are shorter and faster:
want use
pick columns by a single separator cut -d: -f1,3
search lines grep
count the distinct values sort | uniq -c
line edits sed
make columns line up column -t
sum a column awk (or paste -sd+ | bc)
The sed and awk guides on this site cover the pair: sed for editing lines, awk for columns and totals.
One-liners
All of these ran in the lab. They use the sample files from the start of the guide.
# Print the first and last fields
$ awk '{ print $1, $NF }' sales.txt | head -2
region price
north 1.50
# Add up a column and print the average (skipping the header)
$ awk 'NR>1 { s += $3 } END { printf "total %d, average %.2f\n", s, s/(NR-1) }' sales.txt
total 36, average 6.00
# Total of qty * price
$ awk 'NR>1 { total += $3 * $4 } END { print total }' sales.txt
61.05
# Count the lines (like wc -l)
$ awk 'END { print NR }' sales.txt
7
# Users with a login shell, from /etc/passwd format
$ awk -F: '$7 !~ /nologin|false/ { print $1 }' passwd.txt
root
ann
ed
# Count users by shell
$ awk -F: '{ shells[$NF]++ } END { for (s in shells) print shells[s], s }' passwd.txt | sort -rn | head -3
2 /usr/sbin/nologin
2 /bin/bash
1 /bin/zsh
# Top clients in a web log
$ awk '{ n[$1]++ } END { for (k in n) print n[k], k }' access.log | sort -rn | head -3
3 10.0.1.11
2 10.0.3.5
1 10.0.2.31
# Requests per status code
$ awk '{ c[$9]++ } END { for (k in c) print k, c[k] }' access.log | sort
200 3
302 1
404 2
500 1
# Show the failing requests (status 400 or more): the status is the second-to-last field
$ awk '$(NF-1) >= 400 { print $1, $7, $(NF-1) }' access.log
10.0.3.5 /missing 404
10.0.3.5 /missing 404
10.0.2.31 /api/items 500
# Bytes sent, in KB
$ awk '{ b += $NF } END { printf "%.1f KB\n", b/1024 }' access.log
12.2 KB
# The requested path, cut out with the quote as the separator
$ awk -F'"' '{ split($2, r, " "); print r[2] }' access.log | sort | uniq -c | sort -rn | head -3
2 /missing
2 /index.html
2 /api/items
# From a log line, the user: the field that starts with user=
$ awk '/ERROR/ { for (i = 1; i <= NF; i++) if ($i ~ /^user=/) print substr($i, 6) }' app.log
kim
ed
# Group by level (field 3) and print the count
$ awk '{ n[$3]++ } END { for (k in n) print k, n[k] }' app.log | sort
ERROR 2
INFO 4
WARN 1
# Print lines 3 to 5, or just line 4
$ awk 'NR>=3 && NR<=5' sales.txt
south pears 4 2.25
north pears 7 2.25
east apples 3 1.50
$ awk 'NR==4' sales.txt
north pears 7 2.25
# Print the last line, or the last two
$ awk 'END { print }' sales.txt
east plums 0 3.10
$ awk '{ a[NR] = $0 } END { print a[NR-1]; print a[NR] }' sales.txt
south apples 12 1.40
east plums 0 3.10
# Remove blank lines, and lines that are only blanks
$ printf 'a\n\n \nb\n' | awk 'NF'
a
b
# Trim leading and trailing blanks
$ printf ' a b \n' | awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print "[" $0 "]" }'
[a b]
# Join every three lines into one
$ awk 'NR % 3 == 1 { printf "%s", $1; next } { printf " %s", $1 } NR % 3 == 0 { print "" }' <<< $'a\nb\nc\nd\ne\nf'
a b c
d e f
# Uniq without sorting, keeping the first
$ awk '!seen[$2]++ { print $2 }' sales.txt
product
apples
pears
plums
# Sum per key and print a table
$ awk 'NR>1 { u[$1] += $3 } END { for (k in u) printf "%-6s %3d\n", k, u[k] }' sales.txt | sort
east 3
north 17
south 16
# Use a shell variable safely
$ U=ann; awk -v u="$U" '$1 == u' ids.txt
ann 1000
# Add a column from a calculation
$ awk 'NR>1 { print $0, $3 * $4 }' sales.txt | head -2
north apples 10 1.50 15
south pears 4 2.25 9
# Memory and load, from /proc
$ awk '/^(MemTotal|MemAvailable)/ { print $1, $2/1024 " MB" }' /proc/meminfo
MemTotal: 31778.5 MB
MemAvailable: 24478.9 MB
$ awk '{ print $1 }' /proc/loadavg
0.58
$ free -m | awk '/Mem:/ { printf "%.0f%% used\n", $3/$2*100 }'
23% used
# Sum the size column of ls -l (skip the "total" line)
$ ls -l /etc | awk 'NR>1 { s += $5 } END { print s " bytes" }'
268236 bytes
# Memory by user, from ps
$ ps -eo user=,rss= | awk '{ s[$1] += $2 } END { for (u in s) print u, s[u] }' | sort | head -3
root 1472
ubuntu 18388
# Disk use above zero, one per mount (df -P keeps each entry on one line)
$ df -P | awk 'NR>1 && $5+0 > 0 { print $6, $5 }' | head -3
/ 25%
/etc/hosts 25%
# Which network addresses are up
$ ip -brief addr | awk '{ print $1, $3 }' | head -3
lo 127.0.0.1/8
eth0@if62 172.17.0.5/16
Cheat sheet
Run awk 'program' file awk -f prog.awk file awk -F: -v n=5 'prog' file cmd | awk 'prog' awk 'prog' f1 f2
Rule pattern { action } no pattern = every line no action = print the line BEGIN { } END { }
Fields $0 whole line $1 $2 ... $NF last $(NF-1) NF count NR line number FNR line in this file FILENAME
Separators -F: -F'\t' -F'[,;]' BEGIN { FS=","; OFS="," } RS="" (paragraphs) ORS -v OFS='\t'
Patterns /re/ !/re/ $2 ~ /re/ $2 !~ /re/ $3 > 10 NR==5 NR>1 /a/,/b/ NF $1 == "x" && $2 > 3
Print print $1, $2 print $1 $2 printf "%-10s %5.2f\n", $1, $2 print > "file" print | "sort" close("sort")
Numbers + - * / % ^ int() sqrt() printf "%.2f" (print shows 6 digits: use printf) x + 0 forces a number
Strings length substr(s,start,n) index split(s,a,sep) sub gsub match sprintf tolower toupper s1 s2 joins
Regex (ERE) . * + ? | ( ) { } [abc] [^abc] ^ $ [[:digit:]] [[:space:]] [[:alpha:]] (\s \w only in gawk/BusyBox)
Arrays a[k]=v k in a delete a[k] delete a for (k in a) a[i,j] split(s,a) length(a) (order not defined)
Counting idioms { n[$1]++ } !seen[$0]++ (unique) { s[$1]+=$2 } END { for (k in n) print k, n[k] } | sort
Control if () else for (;;) for (k in a) while do while break continue next nextfile exit n
Functions function f(a, b, local) { return a+b } arrays are by reference no space before ( when calling
getline getline getline v getline v < file "cmd" | getline v close("cmd") > 0 test, never bare while (getline)
Several files NR==FNR { a[$1]=$2; next } $1 in a (empty first file: use FILENAME==ARGV[1]) FNR==1 nextfile
Variables in -v n=5 n=5 between files ENVIRON["N"] never "$var" inside the program text
Shell single quotes system() and "cmd"| use sh: no data inside print > $1 can write anywhere gawk --sandbox
gawk only IGNORECASE gensub match(s,re,m) asort PROCINFO["sorted_in"] strtonum switch BEGINFILE -i inplace FPAT FIELDWIDTHS @include --lint --profile
mawk fastest no gawk extras bytes not characters -W version -W interactive rand() differs per run
Exit status 0 ok 1 or 2 syntax error / missing file (differs by awk) exit !found in END
Common problems
output is a number like 1.23457e+07 print uses %.6g for non-whole numbers: use printf "%.2f"
a total or counter prints an empty line the variable was never set: print c + 0
min or max is wrong, includes the header the header is a line of text: use NR>1
every line is printed twice an action prints and the pattern 1 (or no-action rule) prints too: remove one
first line is split the wrong way FS set in the main rule: use BEGIN { FS=":" } or -F:
a column is lost or shifted with spaces in values the default separator is any blanks: set -F with the real separator
edited line has the wrong separator OFS was not set: use -v OFS=: or BEGIN { OFS=FS }
$2 = "" leaves a double space the separator is kept: use sub(/^ /, "") or rebuild the line in a loop
a file named 2 or 0 appeared print a > b is a redirect: write print (a > b), or use if
a file named x=1 is ignored, awk waits on stdin name=value is an assignment: use ./x=1
a join prints nothing the first file was empty: use FILENAME==ARGV[1] instead of NR==FNR
syntax error at the end of a long line a missing } or a quote: count them, and use a program file for long ones
the shell says unexpected token or awk gets a broken program the program has a single quote inside: use \047 or -v q="'"
sub or gsub with a dot changes everything a string is a regex: use "[.]" or /\./
a regex works in gawk, not in mawk \s \w \y \d, IGNORECASE: use [[:space:]] and tolower()
mawk: function never defined gensub, asort, strtonum, and so on are gawk only
mawk: syntax error at or near , match(s, re, arr) is gawk only
gawk: function called with space between name and ( remove the space between a function name and the bracket
0x1A is 26 in mawk and 0 in gawk hex strings are converted differently: use strtonum in gawk, or avoid them
printf: not enough arguments a % in your data used as the format: printf "%s\n", $0
length() or substr() cuts an accented letter mawk and the C locale count bytes: use gawk in a UTF-8 locale
last field has an invisible \r the file has CRLF line ends: sub(/\r$/, "") first, or RS="\r?\n" in gawk or mawk
a quoted CSV field is split at its comma FS cannot read CSV: use gawk FPAT, or a CSV tool
the order of for (k in a) changes the order is not defined: pipe to sort, or use gawk's PROCINFO["sorted_in"]
a pipe to sort prints at the wrong time the pipe is flushed at close(): call close("sort") before printing a total
the next command in a pipe sees nothing for a while awk is buffering: use fflush(), or mawk -W interactive
a pipeline with tail -f shows nothing same cause: gawk '{ print; fflush() }'
while (getline line < file) never ends the file does not exist (getline returns -1): test with > 0
getline from a command only works once the command is still open: close(cmd) first
a variable in a function changed the caller's it was not a parameter: add it as an extra parameter
data from a file ran a command system() or print | "sh" with data in the command: never build commands from data
awk -i inplace: command not found, or an unknown option mawk has no -i: use gawk -i inplace, or a temp file and mv
different random numbers each run on Ubuntu mawk seeds from the clock: call srand(1) for a repeatable sequence
syntax error exit status differs between servers 1 in gawk and BusyBox, 2 in mawk and the original awk: test for non-zero, not for a number
It works on my Mac and not on the server macOS awk is the original awk; the server has mawk: use only POSIX features, test with mawk
Tips
Quote the program with single quotes. Put values in with -v or the environment, never into the program text.
Start with a pattern and print, then add actions: awk '/ERROR/', then awk '/ERROR/ { print $1, $NF }'.
Look at your fields before you use them: awk '{ print NF; print "[" $1 "]" }' shows what awk thinks the line is.
Skip the header with NR > 1, and remember END sees the last line.
Use printf for anything that goes in a report, and %.2f for money. print is for quick output.
Set FS and OFS together: -F: -v OFS=: keeps the separator the same on the way out.
Write 1 at the end of a rule list to print every line after the edits.
Use the parentheses form for comparisons inside print: print (a > b).
Initialise counters with + 0 when you print them, or set them in BEGIN.
Use if (k in a) to test a key. Reading a[k] creates it.
Count and group with arrays, then pipe to sort: the for-in order is not defined.
Use close() on files and commands you finish with, and always test getline with > 0.
Never build a shell command or a file name from data without checking it. Use gawk --sandbox for a program you did not write.
Know which awk you have: awk -W version, awk --version. On Ubuntu it is usually mawk; call gawk by name when you need it.
Write portable awk (POSIX, [[:class:]], no gensub) when the script has to run on many machines, and test it with mawk.
Put a long program in a file with comments, run it with awk -f, and use gawk --lint on it while you write it.
Stop early with exit when you only want the first match, and use grep first to cut the input down on big files.
When the data is CSV with quotes, JSON, YAML or HTML, use a tool made for it, and keep awk for the columns.
