bytebeast · Oct 10, 2026 · Oct 10, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,25 @@
+# Changelog
+
+## Unreleased (2026-10-10)
+
+### Bug Fixes
+
+- **streaming**: `tail -f | hl` showed nothing until a 4-64 KB buffer filled,
+  because `tr`, `sed` and `mawk` (the default awk on Debian/Ubuntu) all
+  block-buffer through pipes. Input scrubbing now happens inside `sed`
+  (`tr` has no line-buffered mode); `sed` runs with `-u`/`-l`, `mawk` with
+  `-W interactive`, and awk flushes each line. This is automatic when output
+  is a terminal, and available with `-u` for longer pipelines; bulk throughput
+  is unchanged.
+- **help**: `--help` printed a fixed line range of the script header; it now
+  prints the whole header, however long.
+- **README**: the "maximum 12 patterns" limit was removed from the code long
+  ago (only 15 distinct colors cap it); the docs now say so.
+
+### Features
+
+- `-u` / `--line-buffered`, and `--` to end options
+- warning when `-u` is requested but the system sed cannot line-buffer
+  (BusyBox sed)
+- test suite (`tests/test.sh`, plain Bash), verified with mawk, BusyBox awk,
+  GNU sed and BusyBox sed
diff --git a/README.md b/README.md
@@ -30,15 +30,16 @@ life.
 
 - **Literal or regex matching** - plain words by default, or wrap a pattern in
   `/.../` for `sed` BRE regex
-- **Multiple patterns per call**, applied left to right (up to 12)
+- **Multiple patterns per call**, applied left to right (any number, up to 15 distinct colors)
 - **Overlapping matches nest** - when one pattern highlights inside another, the
   outer color is restored afterwards instead of being cleared
 - **14 colors**, including bright variants with short aliases (`bred`, `bgreen`,
   ...)
 - **Zero dependencies** - pure Bash + `sed` + `awk` + `tput`/ANSI, nothing to
   install
-- **Streaming** - output appears line by line, so `tail -f | hl` and other live
-  pipelines work as you would expect
+- **Streaming** - when the output is your terminal, each line appears as soon
+  as it arrives, so `tail -f | hl` works as you would expect; add `-u` when hl
+  feeds another live command (`tail -f log | hl -u ERROR:red | grep ...`)
 - **Pipe-friendly** - reads stdin, writes stdout, plays nicely with `tail`,
   `grep`, `journalctl`, CI logs, anything text-based
 
@@ -51,6 +52,10 @@ life.
 
 No package manager, no runtime, no build step.
 
+> **Alpine / BusyBox note:** BusyBox `sed` cannot line-buffer, so live output
+> (`tail -f | hl`) may arrive in bursts; `hl -u` warns about it. Everything else
+> works. `apk add sed` installs GNU sed and fixes it.
+
 > **macOS note:** `hl` works with the system BSD `sed`, but a few regex
 > constructs (`\|`, `\+`, `\?`) are GNU extensions and will not match under it.
 > See [Regex support](#regex-support). `brew install gnu-sed` and putting
@@ -83,19 +88,30 @@ hl <<< "installed correctly"
 ## Usage
 
 ```
-echo "text" | hl pattern:color [pattern:color ...]
+echo "text" | hl [-u] pattern:color [pattern:color ...]
 hl -h | --help
 ```
 
+| Option                  | Meaning                                                                 |
+| ----------------------- | ----------------------------------------------------------------------- |
+| `-u`, `--line-buffered` | Flush every line. Automatic when output is a terminal; use it when hl feeds another command in a live pipeline |
+| `--`                    | End of options (for a pattern that starts with `-`)                     |
+| `-h`, `--help`          | Show help                                                               |
+
+Line buffering costs throughput (several times slower on large files), which is
+why it is off when the output is a file or a pipe unless you ask for it.
+
 - **Literal match** (default): `hl word:red`
 - **Regex match** (wrap the pattern in `/.../`, `sed` BRE syntax):
   `hl '/H[0-9]/':blue`
 
 The color is taken from the **last** colon in the argument, so patterns may
 contain colons themselves (`hl 'level:error:bred'` highlights `level:error`).
 
-Bad input is a hard error: an unknown color, a missing colon, or more than 12
-patterns exits with status `2` and a message on stderr.
+Bad input is a hard error: an unknown color, a missing colon or an empty pattern
+exits with status `2` and a message on stderr. There is no limit on the number
+of patterns; colors are shared, so the only ceiling is the 15 distinct colors in
+the palette.
 
 ### Examples
 
@@ -112,6 +128,9 @@ echo "The car is an H1 model" | hl car:red model:green '/H[0-9]/':blue
 # Tail a log and flag errors/warnings in bright colors
 tail -f app.log | hl ERROR:bred WARN:byellow INFO:cyan
 
+# ...and keep it live through a further filter
+tail -f app.log | hl -u ERROR:bred | grep -v healthcheck
+
 # Alternation (GNU sed)
 journalctl -f | hl '/error\|fatal\|panic/':bred
 
@@ -191,7 +210,15 @@ BRE - you escape them to make them special, which is the opposite of PCRE.
 - **`.` and negated classes like `[^x]` can match the internal marker bytes**
   `hl` uses to track highlights, which can produce odd output when combined with
   earlier patterns.
-- **Maximum 12 `pattern:color` pairs** per invocation.
+- **At most 15 distinct colors** per invocation (the whole palette); any number
+  of patterns can share them.
+
+## Tests
+
+```bash
+./tests/test.sh                 # plain Bash, no dependencies
+AWK=gawk ./tests/test.sh        # run hl's pipeline under a specific awk
+```
 
 ## Contributing
 

diff --git a/hl b/hl
@@ -14,7 +14,13 @@
 ##        hl '/H[0-9]/':blue
 ##
 ## USAGE:
-##   echo "text" | hl pattern:color [pattern:color ...]
+##   echo "text" | hl [-u] pattern:color [pattern:color ...]
+##
+## OPTIONS:
+##   -u, --line-buffered   flush every line, for live pipelines that continue
+##                         after hl (tail -f log | hl ERROR:red | grep x).
+##                         Automatic when output is a terminal.
+##   -h, --help            show this help
 ##
 ## EXAMPLES:
 ##   echo "The car is an H1 model" | hl car:red
@@ -87,7 +93,9 @@ STRIP='\004-\007\013\014\020-\032\034-\037'                               # scru
 MAX_COLORS=${#MARKS}
 
 usage() {
-    sed -n '3,50p' "$0" | sed 's/^#*//; s/^ //'
+    # Everything between the opening banner and the next one, so the help can
+    # grow without this line range going stale.
+    sed -n '3,/^#\{20,\}$/p' "$0" | sed '$d; s/^#*//; s/^ //'
     exit "${1:-0}"
 }
 
@@ -189,9 +197,29 @@ escape_bre() {
     exec cat
 }
 
-case "${1:-}" in
-    -h | --help) usage 0 ;;
-esac
+# Line buffering (see "Streaming" below): on by default when stdout is a
+# terminal, which covers `tail -f | hl`; -u / --line-buffered forces it for
+# pipelines such as `tail -f log | hl ERROR:red | grep ...`.
+LINE_BUFFERED=0
+LINE_BUFFERED_ASKED=0
+[[ -t 1 ]] && LINE_BUFFERED=1
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        -h | --help) usage 0 ;;
+        -u | --line-buffered)
+            LINE_BUFFERED=1
+            LINE_BUFFERED_ASKED=1
+            shift
+            ;;
+        --)
+            shift
+            break
+            ;;
+        *) break ;;
+    esac
+done
+[[ $# -gt 0 ]] || die "no pattern:color given"
 
 ###############################################################################
 ## Colors are interned: repeated colors reuse the same mark, so 40 patterns
@@ -254,11 +282,46 @@ done
 ## One streaming pipeline: scrub markers, run every substitution in a single
 ## sed pass, then resolve markers to ANSI codes with a color stack so nested
 ## highlights restore the enclosing color instead of clearing it.
+##
+## Streaming: every stage must pass each line on as soon as it has it, or
+## `tail -f | hl` shows nothing until a buffer (4-64 KB) fills. tr has no
+## line-buffered mode at all, so input scrubbing happens inside sed instead;
+## sed gets -u (GNU, FreeBSD/macOS) or -l (older BSD); mawk, Debian/Ubuntu's
+## default awk, also buffers INPUT from a pipe unless given -W interactive;
+## and every awk flushes after each line.
 ###############################################################################
-tr -d "$STRIP" |
-    sed "${sed_args[@]}" |
-    awk -v marks="$MARKS" -v cl="$CLOSE" -v sep="$SEP" \
-        -v cols="$colors" -v reset="$RESET" '
+## Flushing per line costs throughput (about 7x on a 200k-line file), so it
+## is only done when LINE_BUFFERED is set: by default when stdout is a
+## terminal, or with -u / --line-buffered.
+SED_STREAM=()
+AWK_STREAM=()
+if [[ $LINE_BUFFERED -eq 1 ]]; then
+    if sed -u -n '' </dev/null >/dev/null 2>&1; then
+        SED_STREAM=(-u)
+    elif sed -l -n '' </dev/null >/dev/null 2>&1; then
+        SED_STREAM=(-l)
+    elif [[ $LINE_BUFFERED_ASKED -eq 1 ]]; then
+        # BusyBox sed (Alpine, most minimal images) has neither flag, so its
+        # output stays block-buffered. Say so when -u was explicitly asked
+        # for, rather than silently not doing it.
+        printf '%s\n' "hl: this sed cannot line-buffer; output may lag" \
+            "    (install GNU sed, e.g. 'apk add sed' on Alpine)" >&2
+    fi
+    case "$(awk -W version </dev/null 2>&1 || true)" in
+        *mawk*) AWK_STREAM=(-W interactive) ;;
+    esac
+fi
+
+# Delete marker bytes from the input (same set as STRIP, listed byte by byte
+# so no locale-dependent range is involved).
+STRIP_EXPR=$'s/[\004\005\006\007\013\014\020\021\022\023\024\025\026\027\030\031\032\034\035\036\037]//g'
+
+# ${a[@]+"${a[@]}"} expands an empty array to nothing; plain "${a[@]}" is an
+# "unbound variable" error under set -u on bash < 4.4.
+sed ${SED_STREAM[@]+"${SED_STREAM[@]}"} -e "$STRIP_EXPR" "${sed_args[@]}" |
+    awk ${AWK_STREAM[@]+"${AWK_STREAM[@]}"} \
+        -v marks="$MARKS" -v cl="$CLOSE" -v sep="$SEP" \
+        -v cols="$colors" -v reset="$RESET" -v flush="$LINE_BUFFERED" '
     BEGIN { split(cols, C, sep); re = "[" marks cl "]" }
     {
         line = $0; out = ""; d = 0
@@ -275,4 +338,5 @@ tr -d "$STRIP" |
             line = substr(line, RSTART + 1)
         }
         print out line
+        if (flush) fflush()
     }'
diff --git a/tests/test.sh b/tests/test.sh
@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+###############################################################################
+## Tests for hl. No dependencies beyond what hl itself needs.
+##
+##   ./tests/test.sh                 # uses ./hl
+##   HL=/path/to/hl ./tests/test.sh  # test another copy
+##   AWK=gawk ./tests/test.sh        # run the pipeline under a specific awk
+###############################################################################
+set -uo pipefail
+
+HERE=$(cd "$(dirname "$0")" && pwd)
+HL=${HL:-$HERE/../hl}
+
+# Run hl under a chosen awk by putting a shim named "awk" first on PATH.
+if [[ -n "${AWK:-}" ]]; then
+    shim=$(mktemp -d)
+    ln -s "$(command -v "$AWK")" "$shim/awk"
+    PATH="$shim:$PATH"
+    export PATH
+    trap 'rm -rf "$shim"' EXIT
+fi
+
+# Fixed ANSI codes regardless of the terminal: TERM=dumb makes tput fail, so
+# hl falls back to its built-in escape sequences.
+export TERM=dumb
+R=$'\033[0;31m'   # red
+G=$'\033[0;32m'   # green
+B=$'\033[0;34m'   # blue
+BR=$'\033[1;31m'  # bright red
+X=$'\033[0m'      # reset
+
+pass=0
+fail=0
+
+check() { # name expected actual
+    if [[ "$2" == "$3" ]]; then
+        pass=$((pass + 1))
+    else
+        fail=$((fail + 1))
+        printf 'FAIL: %s\n  expected: %q\n  actual:   %q\n' "$1" "$2" "$3"
+    fi
+}
+
+hl() { "$HL" "$@"; }
+
+# --------------------------------------------------------------- matching
+
+check "literal" "a ${R}car${X} b" "$(printf 'a car b\n' | hl car:red)"
+check "every occurrence" "${R}x${X}-${R}x${X}" "$(printf 'x-x\n' | hl x:red)"
+check "regex" "H${B}1${X} model" "$(printf 'H1 model\n' | hl '/[0-9]/:blue')"
+check "regex interval" "at ${BR}09:30${X}" \
+    "$(printf 'at 09:30\n' | hl '/[0-9]\{2\}:[0-9]\{2\}/:bred')"
+check "literal dot is not a wildcard" "${R}a.b${X} axb" "$(printf 'a.b axb\n' | hl a.b:red)"
+check "literal star and brackets" "${R}[*]${X}" "$(printf '[*]\n' | hl '[*]:red')"
+check "literal caret and dollar" "${R}^\$${X}" "$(printf '^$\n' | hl '^$:red')"
+check "literal backslash" "${R}a\\b${X}" "$(printf 'a\\b\n' | hl 'a\b:red')"
+check "ampersand in pattern" "${G}a&b${X}" "$(printf 'a&b\n' | hl 'a&b:green')"
+check "slash in literal" "${B}path/to${X}" "$(printf 'path/to\n' | hl 'path/to:blue')"
+check "colon in pattern (last colon is the color)" "${R}12:30${X}" \
+    "$(printf '12:30\n' | hl '12:30:red')"
+check "alias color name" "${BR}x${X}" "$(printf 'x\n' | hl x:bright_red)"
+check "no match passes the line through" "plain" "$(printf 'plain\n' | hl zzz:red)"
+check "empty line" "" "$(printf '\n' | hl x:red)"
+check "multiple lines" "${R}a${X}"$'\n'"b"$'\n'"${R}a${X}" "$(printf 'a\nb\na\n' | hl a:red)"
+
+# ----------------------------------------------------- nesting and markers
+
+check "nested highlight restores outer color" \
+    "${R}ab${G}c${R}de${X}" "$(printf 'abcde\n' | hl abcde:red c:green)"
+check "later pattern cannot match escape-code digits" \
+    "${R}ERROR${X} ${B}3${X}" "$(printf 'ERROR 3\n' | hl ERROR:red '/[0-9]/:blue')"
+check "marker bytes in input are scrubbed" "${R}inject${X}mex" \
+    "$(printf 'inject\004me\020x\n' | hl inject:red)"
+check "marker bytes in a pattern are scrubbed" "${R}ab${X}" \
+    "$(printf 'ab\n' | hl $'a\004b:red')"
+check "same color reused across patterns" "${R}a${X} ${R}b${X}" \
+    "$(printf 'a b\n' | hl a:red b:red)"
+
+# ---------------------------------------------------------- errors / usage
+
+err=$(printf 'x\n' | hl x 2>&1)
+check "missing color exits 2" "2" "$(printf 'x\n' | hl x >/dev/null 2>&1; echo $?)"
+check "missing color message" "hl: missing color in 'x' (expected pattern:color)" "$err"
+check "unknown color exits 2" "2" "$(printf 'x\n' | hl x:mauve >/dev/null 2>&1; echo $?)"
+check "empty pattern exits 2" "2" "$(printf 'x\n' | hl :red >/dev/null 2>&1; echo $?)"
+check "option without pattern exits 2" "2" "$(printf 'x\n' | hl -u >/dev/null 2>&1; echo $?)"
+check "no args passes input through" "x" "$(printf 'x\n' | hl)"
+check "-- ends options" "${R}-u${X}" "$(printf -- '-u\n' | hl -- -u:red)"
+help=$(hl --help)
+check "help starts with title" "hl - simple text highlighter (literal + regex support)" \
+    "$(printf '%s\n' "$help" | sed -n '2p')"
+check "help includes the last section" "yes" \
+    "$(printf '%s\n' "$help" | grep -q '15 distinct colors' && echo yes || echo no)"
+
+# ------------------------------------------------------------- streaming
+#
+# Feed one line through a FIFO that stays open (like `tail -f`) and see
+# whether it comes out before hl is killed. With -u it must; without -u
+# into a pipe it is allowed to buffer (that is the fast path).
+
+streamed() { # extra hl args...
+    # Portable (no `timeout`, which macOS lacks): run hl in the background
+    # writing to a file, give it up to 2s to produce the line, then stop it.
+    local fifo out pid _
+    fifo=$(mktemp -u)
+    out=$(mktemp)
+    mkfifo "$fifo"
+    exec 3<>"$fifo"
+    printf 'live ERROR line\n' >&3
+    "$HL" "$@" ERROR:red <&3 >"$out" 2>/dev/null &
+    pid=$!
+    for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do
+        [[ -s "$out" ]] && break
+        sleep 0.1
+    done
+    [[ -s "$out" ]] && echo yes || echo no
+    kill "$pid" 2>/dev/null
+    wait "$pid" 2>/dev/null
+    exec 3>&-
+    rm -f "$fifo" "$out"
+}
+
+skipped=0
+if sed -u -n '' </dev/null >/dev/null 2>&1 || sed -l -n '' </dev/null >/dev/null 2>&1; then
+    can_stream=1
+else
+    # e.g. BusyBox sed: hl warns instead of streaming, by design.
+    can_stream=0
+    skipped=3
+    printf 'SKIP: streaming tests (this sed has no -u or -l)\n'
+    check "-u warns when sed cannot line-buffer" "yes" \
+        "$(printf 'x\n' | "$HL" -u x:red 2>&1 >/dev/null | grep -q 'cannot line-buffer' && echo yes || echo no)"
+fi
+
+if [[ $can_stream -eq 1 ]]; then
+    check "-u streams each line through a pipe" "yes" "$(streamed -u)"
+    check "--line-buffered streams too" "yes" "$(streamed --line-buffered)"
+fi
+
+if [[ $can_stream -eq 1 ]] && command -v python3 >/dev/null 2>&1; then
+    # stdout is a terminal: streaming must be automatic, no flag needed.
+    tty_result=$(python3 - "$HL" <<'EOF'
+import os, pty, select, subprocess, sys, time
+master, slave = pty.openpty()
+p = subprocess.Popen([sys.argv[1], "ERROR:red"], stdin=subprocess.PIPE,
+                     stdout=slave, stderr=slave)
+os.close(slave)
+p.stdin.write(b"live ERROR line\n"); p.stdin.flush()
+seen, start = b"", time.time()
+while time.time() - start < 2 and b"line" not in seen:
+    if select.select([master], [], [], 0.1)[0]:
+        seen += os.read(master, 4096)
+print("yes" if b"line" in seen else "no")
+p.stdin.close(); p.wait()
+EOF
+)
+    check "streams automatically on a terminal" "yes" "$tty_result"
+fi
+
+# ----------------------------------------------------------------- result
+
+printf '%d passed, %d failed, %d skipped (awk: %s)\n' "$pass" "$fail" "$skipped" \
+    "$(awk -W version </dev/null 2>&1 | head -n 1 || true)"
+[[ $fail -eq 0 ]]