🧮 awk and sed in Scripts

awk and sed in scripts: log analysis, safe config edits and pipelines that stay correct
awk '{c[$9]++} END{for(s in c) print s, c[s]}' access.log

awk and sed are the two tools that turn a pile of text into answers (awk: "count, sum and report") and into changes (sed: "replace, delete, insert"). The Command Line pages for awk and for sed are the syntax reference. This page is about using them inside scripts that other people and cron will run: recipes that answer real questions about logs, reports and command output, safe ways to edit configuration files, and the traps that silently produce wrong numbers.
Every example was run for real on Ubuntu 24.04 with mawk 1.3.4 (the awk that Ubuntu and Debian install by default, and NOT gawk), GNU sed 4.9 and bash 5.2. Many examples on the internet assume gawk, so the differences are tested and shown. The sample data: a 40-line web access log (nginx "combined" format with the request time added at the end as rt=SECONDS), a small CSV and some config files; for timings, a generated log of 200,000 lines (17.9 MB). All output below is what the commands printed. Addresses are documentation addresses.

  grep           find lines (grep -c counts them, -F for fixed strings, -E for regular expressions)
  cut, sort, uniq, wc   fixed columns, ordering, counting: simple and very fast
  awk            anything with fields and arithmetic: sum, average, group by, filter on a column, reformat
  sed            change text: substitute, delete, insert, edit a file in place; works on lines, not on fields
  jq             JSON: never parse JSON with awk or sed (see the jq page)
  Python or similar   quoted CSV, nested data, anything over about 30 lines of awk, anything you need to test and maintain
A useful rule: if a line of awk needs more than one screen, or you copy it from a colleague without understanding it, write it as a small script file with comments (awk -f), or switch language.

"awk" is a name that points to a program. On Debian and Ubuntu it is mawk, which is small and fast; on Red Hat and Fedora it is gawk (GNU awk), which has many extra functions; on macOS it is the BSD awk. A script that works with gawk can fail, or worse run and give wrong results, with mawk. See which one you have:
$ awk --version 2>&1 | head -1
mawk 1.3.4 20240123
$ readlink -f $(command -v awk)
/usr/bin/mawk
Features that exist only in gawk, tried here with mawk:
$ awk 'BEGIN{print gensub(/a/,"b","g","banana")}'
awk: line 2: function gensub never defined
$ awk 'BEGIN{n=split("c a b",x); asort(x); print x[1]}'
awk: line 2: function asort never defined
$ awk 'BEGIN{print PROCINFO["version"]}'

$ echo 'a,"b,c",d' | awk 'BEGIN{FPAT="([^,]+)|(\"[^\"]+\")"} {print NF}'
1
$ awk 'BEGIN{IGNORECASE=1} /ABC/{print "gawk-style"}' <<< abc
The first two are loud errors (good: you notice). The last three are silent: PROCINFO printed an empty line, FPAT (gawk's way to split CSV) was simply ignored so the line has 1 field instead of 3, and IGNORECASE did nothing, so the pattern did not match. In mawk, unknown variables are just variables. That is the dangerous kind of difference: no error, no warning, wrong output.
What works the same in mawk 1.3.4: the standard functions (length, substr, index, split, sub, gsub, match, sprintf, tolower, toupper), regular-expression intervals such as /a{3}/, printf, getline, associative arrays, -v, -F, -f, and (as an extension) strftime and systime:
$ awk 'BEGIN{print toupper("abc"), length("abcd"), substr("hello",2,3), index("hello","ll")}'
ABC 4 ell 3
$ echo aaa | awk '/a{3}/{print "interval ok"}'
interval ok
Rules for portable scripts: write POSIX awk (no gensub, asort, asorti, PROCINFO, FPAT, FIELDWIDTHS, IGNORECASE, BEGINFILE, nextfile is fine in both, no strftime if you target BSD), sort with the sort command instead of asort, and if you truly need gawk, say so: check at the top of the script (command -v gawk >/dev/null || { echo "needs gawk" >&2; exit 1; }) and call it by name, gawk, not awk. Installing it is one command (sudo apt install gawk). Lowercasing and uppercasing use tolower and toupper, which are standard.

The first lines of the sample log:
$ head -4 access.log
192.0.2.44 - - [10/Oct/2026:08:01:36 +0000] "GET /index.html HTTP/1.1" 200 3992 "-" "curl/8.5" rt=0.062
203.0.113.5 - - [10/Oct/2026:08:02:07 +0000] "GET /index.html HTTP/1.1" 502 7701 "-" "curl/8.5" rt=0.139
203.0.113.5 - - [10/Oct/2026:08:04:58 +0000] "GET /health HTTP/1.1" 304 0 "-" "curl/8.5" rt=0.358
192.0.2.44 - - [10/Oct/2026:08:05:41 +0000] "GET /api/orders HTTP/1.1" 200 8560 "-" "curl/8.5" rt=0.837
By default awk splits each line on whitespace, so $1 is the client address, $4 the time (with a leading [), $7 the path, $9 the status, $10 the bytes and $NF (the last field) the request time. The pattern for almost every recipe is the same: build an associative array while reading the lines (c[key]++), then print it in an END block.
1. Top client addresses:
$ awk '{c[$1]++} END{for(ip in c) print c[ip], ip}' access.log | sort -rn | head -4
18 203.0.113.5
8 192.0.2.44
6 192.0.2.99
5 203.0.113.200
awk does not sort its arrays in any order (and mawk and gawk differ), so the sort is done by the sort command, which is exactly what it is for.
2. Status code counts:
$ awk '{c[$9]++} END{for(s in c) print s, c[s]}' access.log | sort
200 17
301 5
304 5
404 3
500 4
502 6
3. Error rate as a percentage. The ~ operator matches a regular expression:
$ awk '$9 ~ /^5/ {e++} END{printf "%d of %d requests are 5xx (%.1f%%)\n", e, NR, 100*e/NR}' access.log
10 of 40 requests are 5xx (25.0%)
NR is the number of lines read. Note the %% in printf to print a percent sign. If the log is empty, NR is 0 and you divide by zero. gawk stops with an error, but mawk does NOT: it carries on and prints nonsense, with exit status 0:
$ awk 'BEGIN{print 1/0}'
inf
$ awk 'BEGIN{print 0/0}'
-nan
$ : | awk 'END{printf "%.1f%%\n", 100*5/NR}'          (zero lines of input)
inf%
$ awk 'BEGIN{printf "%d\n", 1/0}'
18446744073709551615
A report that says "inf%" or 18446744073709551615 is easy to spot; the same value inside a larger calculation is not. Guard every division whose divisor can be zero:
$ : | awk 'END{printf "%.1f%%\n", NR ? 100*5/NR : 0}'
0.0%
(Also in the example above, e is unset when there were no errors, and awk treats an unset variable as 0, so printf prints 0 for it.)
4. Bytes per hour (split the timestamp on colons):
$ awk '{split($4,t,":"); h[t[2]]+=$10} END{for(k in h) printf "%s:00  %8d bytes\n", k, h[k]}' access.log | sort
08:00    123529 bytes
09:00     49353 bytes
5. The request line, via a different field separator. The quoted request is one piece when you split on the double quote:
$ awk -F'"' '{split($2,r," "); print r[1], r[2]}' access.log | sort | uniq -c | sort -rn | head -4
      7 GET /index.html
      7 GET /admin
      6 GET /health
      6 GET /api/orders
6. Count, average and maximum of the request time per URL (sub removes the "rt=" prefix from the last field, and the maximum is tracked by hand):
$ awk '{sub(/^rt=/,"",$NF); split($7,u,"?"); n[u[1]]++; s[u[1]]+=$NF; if($NF>m[u[1]]) m[u[1]]=$NF} END{for(p in n) printf "%-14s n=%-3d avg=%.3f max=%.3f\n", p, n[p], s[p]/n[p], m[p]}' access.log | sort -t= -k3 -rn | head -4
/api/orders    n=6   avg=1.808 max=2.412
/api/users     n=1   avg=0.405 max=0.405
/              n=6   avg=0.348 max=0.399
/static/app.js n=4   avg=0.275 max=0.375
(The split on ? removes the query string, so /a?x=1 and /a?x=2 count as one URL.) /api/orders is the slow one.
7. Percentiles: awk is not good at sorting, so let sort do it, then pick positions:
$ awk '{sub(/^rt=/,"",$NF); print $NF}' access.log | sort -n | awk '{v[NR]=$1} END{i=int(0.95*NR); if(i<1)i=1; printf "n=%d  p50=%s  p95=%s  max=%s\n", NR, v[int(NR/2)], v[i], v[NR]}'
n=40  p50=0.270  p95=2.156  max=2.412
The p95 is more than eight times the median: a few requests are very slow. Averages hide that; percentiles show it.
8. Filter on two conditions and print the line number (NR) with the match:
$ awk '$1=="203.0.113.5" && $9>=500 {print NR": "$7, $9}' access.log | head -4
2: /index.html 502
8: /static/app.js 502
34: /login 502
35: /api/orders 500
Comparison with $9>=500 works as a number here because the field looks like a number (see "awk traps").

A fixed-width report with printf. The format characters: %s string, %d integer, %f/%.3f decimals, %-6s left-aligned in 6 columns, %6d right-aligned:
$ awk '{sub(/^rt=/,"",$NF); c[$9]++; b[$9]+=$10; t[$9]+=$NF} END{printf "%-6s %6s %10s %8s\n","status","count","bytes","avg_rt"; for(s in c) printf "%-6s %6d %10d %8.3f\n", s, c[s], b[s], t[s]/c[s]}' access.log
status  count      bytes   avg_rt
301         5      24375    0.164
502         6      25681    0.562
200        17      87112    0.395
304         5          0    0.308
404         3      13306    0.934
500         4      22408    0.940
The rows are in no useful order: for (s in c) visits the keys in whatever order awk's hash table has them (mawk and gawk differ), and it can change from one version to the next. Never depend on it. Print the header yourself and send only the rows through sort:
$ awk '{sub(/^rt=/,"",$NF); c[$9]++; b[$9]+=$10; t[$9]+=$NF} END{for(s in c) printf "%-6s %6d %10d %8.3f\n", s, c[s], b[s], t[s]/c[s]}' access.log | sort | { printf "%-6s %6s %10s %8s\n" status count bytes avg_rt; cat; }
status  count      bytes   avg_rt
200        17      87112    0.395
301         5      24375    0.164
304         5          0    0.308
404         3      13306    0.934
500         4      22408    0.940
502         6      25681    0.562
The braces group the printf and the cat so both write into the same output, header first.
Percent of a total needs two passes over the data: one to get the total, one to compute the shares. Give the same file twice and use the NR==FNR idiom (NR counts lines over all files, FNR within the current file; they are equal only while reading the first file):
$ awk 'NR==FNR{tot+=$10; next} FNR==1{print "total bytes:", tot} {p[$9]+=$10} END{for(s in p) printf "%s %5.1f%%\n", s, 100*p[s]/tot}' access.log access.log | sort
200  50.4%
301  14.1%
304   0.0%
404   7.7%
500  13.0%
502  14.9%
total bytes: 172882
(The "total bytes" line comes out of the same pipeline, and sort put it last.) The same NR==FNR idiom joins two files: load the first into an array, then look up each line of the second. Two small files, a user table and an event list:
$ cat users.txt events.txt
u1 alice
u2 bob
u3 carol
u2 login
u1 logout
u2 upload
u9 login
$ awk 'NR==FNR{name[$1]=$2; next} {print ($1 in name ? name[$1] : "?unknown?"), $2}' users.txt events.txt
bob login
alice logout
bob upload
?unknown? login
The test ($1 in name) asks whether the key exists without creating it (name[$1] on a missing key would CREATE an empty entry). Remove duplicate lines while keeping the first occurrence and the original order, which sort -u cannot do:
$ printf 'b\na\nb\nc\na\n' | awk '!seen[$0]++'
b
a
c
Use awk to validate input: print the problem to stderr and set the exit status in END, so the calling script can react:
$ printf '1\n2\nx\n' | awk '$1 !~ /^[0-9]+$/ {print "bad line " NR ": " $0 > "/dev/stderr"; bad=1} END{exit bad}'; echo "[exit $?]"
bad line 3: x
[exit 1]
In a script: awk '...' file || die "file is invalid". Remember that with set -o pipefail a non-zero status from awk in a pipeline is a failure of the pipeline.

Use the machine-friendly form of each command: df -P (POSIX output, one line per filesystem, no wrapping), ps -eo with chosen columns and an empty header (stat=), free -m, /proc files, find -printf, lsblk -J and ip -j for JSON. Real examples:
$ df -P / | awk 'NR>1{gsub(/%/,"",$5); printf "%-16s %3d%% used, %6.1f GB free\n", $6, $5, $4/1048576}'
/                 18% used,  712.4 GB free
$ free -m | awk '/^Mem:/{printf "memory: %d%% available (%d of %d MB)\n", 100*$7/$2, $7, $2}'
memory: 85% available (27045 of 31778 MB)
$ ps -eo stat= | awk '{c[substr($1,1,1)]++} END{for(s in c) print s, c[s]}' | sort | tr '\n' ' '
I 98 R 1 S 246 Z 1
$ awk '/^(MemTotal|MemAvailable|SwapTotal):/{printf "%s %.1f GB\n", $1, $2/1048576}' /proc/meminfo
MemTotal: 31.0 GB
MemAvailable: 26.4 GB
SwapTotal: 8.0 GB
$ uptime | awk -F'load average: ' '{split($2,l,", "); print "load 1/5/15:", l[1], l[2], l[3]}'
load 1/5/15: 0.45 0.38 0.34
$ df -P | awk 'NR>1 && $5+0 > 10 {print $6, $5}'
/ 18%
/sys/firmware/efi/efivars 35%
The last example shows a filter you must adapt: df lists special filesystems (efivars, tmpfs and so on) too, so select real ones with df -P -x tmpfs -x devtmpfs or by the device name. $5+0 turns "18%" into the number 18 (awk reads the leading digits) so the comparison is numeric; a plain $5 > 10 would compare text.
Do not parse ls. Counting the files in /etc with ls -l and awk gave a different answer from the right tool:
$ ls -l /etc | awk 'NR>1 && $1 ~ /^-/ {s+=$5; n++} END{printf "%d files, %d bytes in /etc (top level)\n", n, s}'
91 files, 358957 bytes in /etc (top level)
$ find /etc -maxdepth 1 -type f -printf '%s\n' | awk '{s+=$1; n++} END{print n " files, " s " bytes"}'
94 files, 360027 bytes
91 against 94: ls -l shows symbolic links as type l and counts their own small size, while find -type f follows its own rules, and the output of ls is meant for people (file names with spaces shift the columns). find -printf, stat -c and the /proc files give stable fields.

The standard awk field separator breaks as soon as a field contains the separator inside quotes:
$ cat people.csv
name,city,age
"Smith, John","New York",34
Alice,Paris,28
"Doe, Jane",Berlin,41
$ awk -F, 'NR>1{print NF, $1}' people.csv
4 "Smith
3 Alice
4 "Doe
The first and the third rows were split into four fields, with the name in two pieces. gawk has FPAT for this; mawk ignores FPAT without a word (see above). A portable function that works in any awk walks the line one character at a time and tracks whether it is inside quotes (a doubled quote "" inside quotes means a literal quote):
$ cat csvsplit.awk
# csvsplit(line, out): split a CSV line into out[1..n], return n. Handles "quoted, fields" and ""escaped"" quotes.
function csvsplit(line, out,    n, f, c, inq, i) {
  n = 0; f = ""; inq = 0
  for (i = 1; i <= length(line); i++) {
    c = substr(line, i, 1)
    if (inq) {
      if (c == "\"") { if (substr(line, i+1, 1) == "\"") { f = f "\""; i++ } else inq = 0 }
      else f = f c
    } else if (c == "\"") inq = 1
    else if (c == ",") { out[++n] = f; f = "" }
    else f = f c
  }
  out[++n] = f; return n
}
NR > 1 { n = csvsplit($0, p); printf "%d fields: name=[%s] city=[%s] age=%s\n", n, p[1], p[2], p[3] }
$ awk -f csvsplit.awk people.csv
3 fields: name=[Smith, John] city=[New York] age=34
3 fields: name=[Alice] city=[Paris] age=28
3 fields: name=[Doe, Jane] city=[Berlin] age=41
All three rows have three fields now. (The extra variables after the blank in the function header are awk's way to declare local variables.) This version handles one record per line; CSV with line breaks inside quoted fields needs a real CSV parser: Python's csv module, or Miller (mlr) or csvkit if you can install them (neither was installed in the lab). For simple files without quotes, -F, is fine; for tab-separated files use -F'\t'.

1. print with > writes to a FILE. In a print statement, > is a redirection, not a comparison. This line meant to print whether the time is positive and instead created a file named 0:
$ awk 'BEGIN{print systime() > 0}'
$ ls -l 0
-rw-rw-r-- 1 demo demo 11 ... 0
Put the comparison in parentheses: print (systime() > 0). The same happens with print a > b anywhere in a script: it writes a to a file called by the value of b.
2. awk -v interprets backslash escapes, ENVIRON does not. Passing a shell variable that contains a backslash:
$ v='a\tb'; awk -v x="$v" 'BEGIN{print "-v     :", x}'; X="$v" awk 'BEGIN{print "ENVIRON:", ENVIRON["X"]}'
-v     : a	b
ENVIRON: a\tb
With -v the two characters \t became a real tab. If the value is a Windows path, a regular expression or user input, use ENVIRON (export the variable for the one command, as above) or read the value from a file. For plain safe values -v is fine.
3. Never paste a shell variable into the awk program. The program is in single quotes so the shell cannot expand it; breaking out of them ('"$var"') puts the value into the CODE:
$ q='x"y'; awk -v s="$q" 'BEGIN{print s}'; awk 'BEGIN{print "'"$q"'"}'
x"y
awk: line 1: runaway string constant "} ...
The first form (-v) is safe, the second is a syntax error at best and code injection at worst. Same rule as SQL: pass data as parameters, not by splicing it into the program.
4. Numbers and strings. Comparisons are numeric only when both sides look like numbers; otherwise they are strings. A field that is read from input that looks like a number counts as a number, a literal string constant does not:
$ cat nums.txt
9
10
100
abc
$ awk '$1 > 50' nums.txt
100
abc
$ awk '$1 > "50"' nums.txt
9
abc
$ printf '10 9\n' | awk '{print ($1 > $2)}'; printf 'a10 a9\n' | awk '{print ($1 > $2)}'
1
0
In the first command "abc" is greater than 50 because a non-number is compared as text ("abc" sorts after "50"); the 10 and 9 are numbers so 100 qualifies and 9 does not. In the second, the constant "50" turns the comparison into a string comparison: "9" is greater than "50" as text, and 10 and 100 are not. The last pair: 10 against 9 is true (numbers), while a10 against a9 is false (text: "a1" sorts before "a9"). To force a number write $1+0, to force text write $1 "". Always make sure the column holds what you think: a header row, a trailing "%" or a stray space makes a column text.
5. Uninitialised variables are 0 and "" at the same time. print x+0, "[" x "]" printed "0 []". It is handy for counters (c[k]++ needs no setup) and dangerous for typos: a misspelt variable name silently gives a total of zero. Run suspicious scripts with a typo check (gawk --lint, if you have gawk) or by reading the END output critically.
6. Assigning to a field rebuilds the line, and whitespace is squeezed:
$ echo 'a   b    c' | awk '{print $1, $2}'
a b
$ echo 'a   b    c' | awk '{$1=$1}1'
a b c
$ echo 'a b c' | awk '{$2=""; print; print NF}'
a  c
3
Assigning $1=$1 forces awk to rebuild $0 with OFS (a single space) between fields, which is the shortest way to normalise whitespace (and the 1 at the end is a pattern that is always true, so the line is printed). Setting a field to the empty string does NOT remove it: NF is still 3 and the line has two spaces in the middle.
7. printf with data as the format string. If the text can contain a percent sign, never use it as the format:
$ echo '50% done' | awk '{printf $0 "\n"}'
awk: run time error: not enough arguments passed to printf("50% done
")
$ echo '50% done' | awk '{printf "%s\n", $0}'
50% done
Always printf "%s\n", $0 (or use print).
8. Field splitting with a single character separator keeps empty fields; the default (whitespace) collapses them:
$ printf 'a\t\tc\n' | awk -F'\t' '{print "x=["$1"] y=["$2"] z=["$3"]"}'
x=[a] y=[] z=[c]
Compare the shell's read with IFS set to a tab: it collapses the two tabs and gets the fields wrong (x=[a] y=[c] z=[]); awk -F'\t' is the reliable way to read tab-separated data that may have empty columns.

sed reads a line, applies the commands, and prints it. The commands you use daily: s/old/new/ (substitute; add g for all matches on the line, I for case-insensitive, p with -n to print only changed lines), d (delete), p (print, with -n), a, i and c (append, insert, change text), and addresses (a line number, a /regex/, a range 3,7 or /start/,/end/) in front of any of them.
The delimiter problem. A slash inside the text breaks s/// because sed sees it as the end of a part. Use any other character as the delimiter, and if the text comes from a variable, expect it to contain the unexpected:
$ echo 'path=/old/dir' | sed 's/old/new/; s/dir/a/b/'
sed: -e expression #1, char 21: unknown option to `s'
$ echo 'path=/old/dir' | sed 's|/old/dir|/new/dir|'
path=/new/dir
$ val='x&y/z'; echo 'K=1' | sed "s/1/$val/"
sed: -e expression #1, char 9: unknown option to `s'
In the replacement text, & means "the whole match" and a backslash starts an escape, so these must be escaped too:
$ echo 'a b' | sed 's/a/[&]/; s/b/\&/'
[a] &
To use a shell variable safely as the REPLACEMENT, escape the three special characters (backslash, ampersand and your delimiter) first:
$ cat esc.sh
# escape a string for use as the REPLACEMENT in s|...|...|
sed_repl() { printf '%s' "$1" | sed -e 's/[\&|]/\\&/g'; }
val='x&y|z/w'
echo 'K=1' | sed "s|1|$(sed_repl "$val")|"
$ bash esc.sh
K=x&y|z/w
The value contained &, | and /, and arrived intact. (The PATTERN side has its own special characters: when the text to find comes from outside, escape it with sed 's/[][\.*^$/]/\\&/g', or use grep -F / awk's index() for literal matching.)
Greedy matching. .* matches as much as possible, so it eats everything up to the LAST comma:
$ echo 'a=1, b=2, c=3' | sed 's/=.*,//'
a c=3
$ echo 'a=1, b=2, c=3' | sed -E 's/=[0-9]+,//g'
a b c=3
The first deleted from the first = to the last comma; the second matched only digits, one pair at a time. Rule: describe what you want to remove ([0-9]+) rather than "anything" (.*), or use a negated class ([^,]*).
Basic and extended regular expressions. By default sed uses BASIC regular expressions, where + ? | ( ) { } are plain characters and need a backslash to be special (GNU extension); -E (or -r) switches to extended ones where it is the other way round:
$ echo 'aaa' | sed 's/a\+/X/'; echo 'aaa' | sed -E 's/a+/X/'; echo 'a+' | sed 's/a+/X/'
X
X
X
All three printed X, for different reasons: the first with \+ (GNU BRE), the second with + (ERE), and the third matched the literal text "a+" because in basic mode a plain + is just a plus sign. Use -E in scripts and you will not have to think about it (it is also what grep -E and awk use).
Ranges, insert, append and delete a block. Sample file blk.txt: one, BEGIN, two, three, END, four (one per line). Output joined into one line with tr:
$ sed -n '/BEGIN/,/END/p' blk.txt | tr '\n' ' '
BEGIN two three END
$ sed '/BEGIN/,/END/d' blk.txt | tr '\n' ' '
one four
$ sed '/BEGIN/,/END/{//!d}' blk.txt | tr '\n' ' '
one BEGIN END four
$ sed '/two/a\added after' blk.txt | tr '\n' ' '
one BEGIN two added after three END four
$ sed '/two/i\inserted before' blk.txt | tr '\n' ' '
one BEGIN inserted before two three END four
$ sed -n '2{p;q}' blk.txt
BEGIN
$ sed -n '$p' blk.txt
four
-n turns off the automatic printing so only p lines appear; /a/,/b/ is a range from a line matching a to the next line matching b; {//!d} inside a range deletes everything except the two marker lines (the empty regex // means "the last regex used"); 2{p;q} prints line 2 and quits, so on a huge file sed stops at once instead of reading to the end; $ is the last line.
Joining and multi-line work. sed works one line at a time, so for anything that spans lines you need N (append the next line), a loop label, or -z to treat the whole input as one line (NUL separated). Joining lines that end with a backslash (shell line continuations):
$ cat cont.txt
cmd --a \
  --b \
  --c
next
$ sed -e ':a' -e '/\\$/N; s/\\\n *//; ta' cont.txt
cmd --a --b --c
next
$ printf 'a\nb\nc\n' | sed -z 's/\n/,/g; s/,$/\n/'
a,b,c
-z is the simplest way to do a substitution across line breaks (GNU sed). Anything more complicated than this is a sign to use awk or Python.
In-place editing. sed -i edits the file itself. Three facts: -i.bak keeps a backup copy with that suffix (always use it when you edit something that matters, or use the safe pattern in the next section); -i on a symbolic link REPLACES the link with a regular file, unless you add --follow-symlinks; and the macOS/BSD sed needs -i '' (an empty suffix argument) where GNU sed takes -i alone, so scripts that must run on both need care (not tested in the lab, which has only GNU sed). The symlink case, run for real:
$ printf 'real file\n' > real.txt; ln -sf real.txt link.txt
$ sed -i 's/real/CHANGED/' link.txt
$ [ -L link.txt ] && echo "still a symlink" || echo "now a regular file"; cat real.txt
now a regular file
real file
$ rm -f link.txt; ln -sf real.txt link.txt
$ sed -i --follow-symlinks 's/real/CHANGED/' link.txt; [ -L link.txt ] && echo "still a symlink"; cat real.txt
still a symlink
CHANGED file
Without the option the link was replaced by a copy and the real file was never changed, so the service reading real.txt saw no edit; with it the link survived and the target was edited. Configuration files that are symlinks (managed by alternatives or a deployment tool) are common, so use --follow-symlinks whenever the path might be one.

Two properties make a configuration edit safe in a script: it is idempotent (running it twice changes nothing the second time) and it is checked (a broken result is never put in place).
Idempotent "set a value" for key-value files (such as sshd_config): if the key exists, active or commented out, replace that line; otherwise append it once. This is the function:
$ cat setopt.sh
#!/bin/bash
# setopt FILE KEY VALUE: replace an active or commented "KEY ..." line, or append it once
set -euo pipefail
f=$1 k=$2 v=$3
if grep -Eq "^[#[:space:]]*${k}[[:space:]]" "$f"; then
  sed -i -E "s|^[#[:space:]]*${k}[[:space:]].*|${k} ${v}|" "$f"
else
  printf '%s %s\n' "$k" "$v" >> "$f"
fi
Run on a small demo config (the original has a commented PermitRootLogin, an active PasswordAuthentication and no MaxAuthTries):
$ cp sshd_demo.conf c1.conf
$ bash setopt.sh c1.conf PermitRootLogin no; bash setopt.sh c1.conf PasswordAuthentication no; bash setopt.sh c1.conf MaxAuthTries 3
$ diff -u sshd_demo.conf c1.conf | tail -n +3
@@ -1,4 +1,5 @@
 # demo config
-#PermitRootLogin prohibit-password
-PasswordAuthentication yes
+PermitRootLogin no
+PasswordAuthentication no
 X11Forwarding yes
+MaxAuthTries 3
$ cp c1.conf c1.before; bash setopt.sh c1.conf PermitRootLogin no; bash setopt.sh c1.conf MaxAuthTries 3; cmp c1.before c1.conf && echo "identical after the second run"
identical after the second run
The commented line was uncommented and set, the active one was changed, and the missing key was appended exactly once; running the same commands again changed nothing. Always look at a diff before you trust an edit: diff -u OLD NEW shows the whole change in a form people can review, and it works on the real file after the edit as well as on a preview (diff -u file <(sed ... file) shows what an edit WOULD do without changing anything).
Checked, atomic replacement. Edit a COPY, validate the copy with the program's own checker (sshd -t -f FILE, nginx -t -c FILE, visudo -c -f FILE, python -m json.tool, jq . FILE, and so on), and only then move it over the original. A move inside one filesystem is atomic, so a reader sees the old file or the new one, never half of one:
$ cat safe_edit.sh
#!/bin/bash
# safe_edit FILE VALIDATOR SED_EXPR...: edit a copy, validate it, show the diff, then replace the original atomically
set -euo pipefail
safe_edit() {
  local file=$1 validator=$2; shift 2
  local dir tmp
  dir=$(dirname -- "$file")
  tmp=$(mktemp -- "$dir/.edit.XXXXXX")            # same folder = same filesystem, so mv is atomic
  trap 'rm -f -- "$tmp"' RETURN
  cp -p -- "$file" "$tmp"                          # keep owner and mode
  sed -i "$@" "$tmp"
  if cmp -s -- "$file" "$tmp"; then echo "no change needed: $file"; return 0; fi
  if ! "$validator" "$tmp"; then echo "validation FAILED, $file left untouched" >&2; return 1; fi
  diff -u -- "$file" "$tmp" | tail -n +3 || true
  cp -p -- "$file" "$file.bak"
  mv -- "$tmp" "$file"
  echo "updated $file (backup: $file.bak)"
}
# a stand-in validator: a config must not contain the word "BROKEN"
valid() { ! grep -q BROKEN "$1"; }
Run with a good edit, the same edit again, and an edit that produces an invalid file:
--- 1: a good edit
@@ -1,2 +1,2 @@
 Port 22
-PermitRootLogin yes
+PermitRootLogin no
updated app.conf (backup: app.conf.bak)
--- 2: the same edit again (nothing to do)
no change needed: app.conf
--- 3: an edit that produces an invalid file
validation FAILED, app.conf left untouched
[exit 1]
--- file after the failed edit, and leftovers:
Port 22
PermitRootLogin no
app.conf app.conf.bak
The good edit showed its diff and kept a backup; repeating it was a no-op; the bad one was refused and the file stayed as it was; and no temporary .edit.* file was left (the RETURN trap removed it). In a real script the validator is the service's own check, for example validator() { sshd -t -f "$1"; } (not run in the lab). The temp file is created in the SAME directory as the target on purpose: mv across filesystems is a copy, which is not atomic. And cp -p keeps the owner and mode of the original, so a root-owned 0600 file does not become world-readable.

Timed on the 200,000-line log (17.9 MB). The numbers are from this laptop and vary, but the ratios are the lesson. Counting requests per status code four ways gave the same answer (200: 167164, 404: 18952, 500: 13884) in very different times:
   62 ms   awk with an array
  173 ms   cut | sort | uniq -c
  115 ms   the same with LC_ALL=C
 2519 ms   bash while read loop
The shell loop was about forty times slower than awk. Reading the file with an EMPTY while read loop takes 1346 ms on its own: the cost is in bash's line-by-line reading, before it does any work. Rule: when a script handles more than a few hundred lines, hand the file to awk, sed, grep or sort in one pass, and do not loop over lines in bash.
The locale matters for sort. In a UTF-8 locale sort follows the language's collation rules, which is slow; the C locale compares bytes:
  429 ms   sort big.log (en_US.UTF-8)
  106 ms   sort with LC_ALL=C
Four times faster, and the output order is predictable (byte order: capitals before lower case, as shown in the Gotchas page). Use LC_ALL=C for any sort, uniq, comm, join or grep over big data where the language rules do not matter, and ALWAYS when two commands must agree on the order. The top three clients, three ways:
  143 ms   awk array + sort
  687 ms   awk | sort | uniq -c | sort -rn
  282 ms   the same pipeline with LC_ALL=C
Counting inside awk skipped two sorts of 200,000 lines. For a plain count of lines that match something, grep wins, because it is a single-purpose tool:
    6 ms   grep -c ' 500 '
    7 ms   grep -cF ' 500 '  (fixed string, no regex)
   11 ms   grep -cE '" (500|502) '
Three separate grep passes (12 ms) were even faster than one awk pass (72 ms) that counts all codes at once. So the honest rule is not "always use awk": for one simple count use grep -c, and use awk when you need to group, sum, compute or reformat, which grep cannot do. For regex-free matches use grep -F or awk's index(), which are faster than regular expressions.
sed on a big file:
  130 ms   sed -i 's/curl/CURL/'
  534 ms   sed -i -E with three capture groups
The more complicated the expression, the longer, and -i rewrites the whole file each time: change many things in ONE sed call with several -e options, not in many calls.
Other speed tips: filter early (grep or a pattern at the start of the awk program) so later steps see less; avoid cat file | awk (awk file is the same without the extra process); read compressed logs with zcat -f logs* | awk ... (-f passes plain files through unchanged); stop reading as soon as you have what you need (awk 'NR==10{print; exit}', sed -n '10{p;q}', head); never sort what you can count in an array.

  Quote the awk program in single quotes, pass data with -v or ENVIRON, and never splice shell variables into it (trap 3). If the program is long, put it in a file and run awk -f prog.awk file: it can be commented, indented and tested by itself, and a here-document keeps a short one next to the script that uses it.
  Set the locale you rely on: LC_ALL=C at the top of the script (or per command) so sort and character classes behave the same under cron as in your terminal.
  Empty or missing input is a normal case. awk '...' on an empty file prints nothing and exits 0; a division in END by NR fails on zero lines. Check [[ -s file ]] first, or make END handle NR==0, and decide what "no data" means (an alert, or a quiet success).
  Exit statuses: awk exits 0 unless the program says exit N or hits a runtime error (2); sed exits 0 even when nothing matched (use grep -q first, or count the changes with a comparison, as safe_edit does); grep exits 1 for "no match", which set -e and pipefail treat as a failure (add || true when "no match" is fine).
  With set -o pipefail, a pipeline like awk ... | head -1 can fail because awk gets SIGPIPE when head exits early: see the Error Handling page (status 141).
  Keep line endings in mind: files from Windows have a carriage return (\r) at the end of each line; awk and sed then see "200\r" and comparisons fail. Strip it first: sed 's/\r$//' file or tr -d '\r' < file.
  Test on a copy and on a small sample, show the diff, and keep a backup of anything you edit in place.

awk: function gensub never defined / asort never defined      you are on mawk and the code needs gawk: install gawk (sudo apt install gawk) and call gawk, or rewrite in POSIX awk
Wrong numbers but no error                                    a gawk-only variable (FPAT, IGNORECASE, PROCINFO) was ignored by mawk; a typo in a variable name; or a column that is text (a header, a % sign, a CR)
A file named 0 (or something odd) appeared                    print a > b inside awk is a redirection: use print (a > b)
awk -v mangled my backslashes                                 -v processes escapes: use ENVIRON["NAME"] for values you do not control
runaway string constant / syntax error after a variable       a shell variable was spliced into the awk program: pass it with -v or ENVIRON
Comparison gives the "wrong" order                            text versus number: force a number with $1+0 or text with $1 ""; check for a header row
Fields are wrong with quoted commas                           awk -F, cannot parse real CSV: use a CSV function (above), gawk's FPAT, or Python's csv module
sed: -e expression #1, char N: unknown option to `s'         the text contains the delimiter (usually a /): use another delimiter such as |, and escape the replacement (sed_repl above)
sed replaced too much                                         .* is greedy: use a negated class ([^,]*) or a more specific pattern, and test with sed -n 's/.../.../p'
The + ? | ( ) in my pattern do nothing in sed                 basic regular expressions: add -E (or backslash them in GNU sed)
sed -i broke a symlink / the target did not change            sed -i replaced the link with a file: use --follow-symlinks
sed -i works on Linux, fails on macOS                         BSD sed wants -i '' : write for GNU sed and say so, or use a temp file and mv (as safe_edit does)
The script is slow on a big log                               a bash while read loop, or many passes: use one awk pass; sort with LC_ALL=C
Matches fail on files from Windows                            a trailing \r: sed 's/\r$//' first
Output order differs between machines                         awk array order is undefined and sort follows the locale: sort the output explicitly with LC_ALL=C

Check which awk you have (awk --version): mawk on Debian and Ubuntu, gawk elsewhere. Write POSIX awk, or require gawk by name.
The pattern for most reports: count into an array while reading, print in END, sort with the sort command.
Let awk count and group; let grep -c do simple counts; never loop over lines in bash for more than a few hundred lines.
Pass values to awk with -v (safe values) or ENVIRON (anything else), never by pasting them into the program. Put comparisons in print inside parentheses.
Use sed with a delimiter that is not in your data, -E for readable regular expressions, and escape any variable you put in the replacement.
Make every edit idempotent and checked: edit a copy, validate it, show the diff, keep a backup, and mv it into place from the same folder. Use --follow-symlinks when the file might be a link.
Use machine-readable output (df -P, ps -o, find -printf, ip -j, lsblk -J), never ls -l, as input.
Set LC_ALL=C when speed or a stable order matters, and handle empty input and Windows line endings on purpose.
When the awk grows past a screen, or the data has real structure (CSV with quotes, JSON, nested text), switch tools: jq for JSON, Python for the rest.