diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..e2fa310 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,25 @@ +# Changelog + +## Unreleased (2026-10-10) + +### Bug Fixes + +- **streaming**: `tail -f | hl` showed nothing until a 4-64 KB buffer filled, + because `tr`, `sed` and `mawk` (the default awk on Debian/Ubuntu) all + block-buffer through pipes. Input scrubbing now happens inside `sed` + (`tr` has no line-buffered mode); `sed` runs with `-u`/`-l`, `mawk` with + `-W interactive`, and awk flushes each line. This is automatic when output + is a terminal, and available with `-u` for longer pipelines; bulk throughput + is unchanged. +- **help**: `--help` printed a fixed line range of the script header; it now + prints the whole header, however long. +- **README**: the "maximum 12 patterns" limit was removed from the code long + ago (only 15 distinct colors cap it); the docs now say so. + +### Features + +- `-u` / `--line-buffered`, and `--` to end options +- warning when `-u` is requested but the system sed cannot line-buffer + (BusyBox sed) +- test suite (`tests/test.sh`, plain Bash), verified with mawk, BusyBox awk, + GNU sed and BusyBox sed diff --git a/README.md b/README.md index 3e8c67b..f5827e5 100644 --- a/README.md +++ b/README.md @@ -30,15 +30,16 @@ life. - **Literal or regex matching** - plain words by default, or wrap a pattern in `/.../` for `sed` BRE regex -- **Multiple patterns per call**, applied left to right (up to 12) +- **Multiple patterns per call**, applied left to right (any number, up to 15 distinct colors) - **Overlapping matches nest** - when one pattern highlights inside another, the outer color is restored afterwards instead of being cleared - **14 colors**, including bright variants with short aliases (`bred`, `bgreen`, ...) - **Zero dependencies** - pure Bash + `sed` + `awk` + `tput`/ANSI, nothing to install -- **Streaming** - output appears line by line, so `tail -f | hl` and other live - pipelines work as you would expect +- **Streaming** - when the output is your terminal, each line appears as soon + as it arrives, so `tail -f | hl` works as you would expect; add `-u` when hl + feeds another live command (`tail -f log | hl -u ERROR:red | grep ...`) - **Pipe-friendly** - reads stdin, writes stdout, plays nicely with `tail`, `grep`, `journalctl`, CI logs, anything text-based @@ -51,6 +52,10 @@ life. No package manager, no runtime, no build step. +> **Alpine / BusyBox note:** BusyBox `sed` cannot line-buffer, so live output +> (`tail -f | hl`) may arrive in bursts; `hl -u` warns about it. Everything else +> works. `apk add sed` installs GNU sed and fixes it. + > **macOS note:** `hl` works with the system BSD `sed`, but a few regex > constructs (`\|`, `\+`, `\?`) are GNU extensions and will not match under it. > See [Regex support](#regex-support). `brew install gnu-sed` and putting @@ -83,10 +88,19 @@ hl <<< "installed correctly" ## Usage ``` -echo "text" | hl pattern:color [pattern:color ...] +echo "text" | hl [-u] pattern:color [pattern:color ...] hl -h | --help ``` +| Option | Meaning | +| ----------------------- | ----------------------------------------------------------------------- | +| `-u`, `--line-buffered` | Flush every line. Automatic when output is a terminal; use it when hl feeds another command in a live pipeline | +| `--` | End of options (for a pattern that starts with `-`) | +| `-h`, `--help` | Show help | + +Line buffering costs throughput (several times slower on large files), which is +why it is off when the output is a file or a pipe unless you ask for it. + - **Literal match** (default): `hl word:red` - **Regex match** (wrap the pattern in `/.../`, `sed` BRE syntax): `hl '/H[0-9]/':blue` @@ -94,8 +108,10 @@ hl -h | --help The color is taken from the **last** colon in the argument, so patterns may contain colons themselves (`hl 'level:error:bred'` highlights `level:error`). -Bad input is a hard error: an unknown color, a missing colon, or more than 12 -patterns exits with status `2` and a message on stderr. +Bad input is a hard error: an unknown color, a missing colon or an empty pattern +exits with status `2` and a message on stderr. There is no limit on the number +of patterns; colors are shared, so the only ceiling is the 15 distinct colors in +the palette. ### Examples @@ -112,6 +128,9 @@ echo "The car is an H1 model" | hl car:red model:green '/H[0-9]/':blue # Tail a log and flag errors/warnings in bright colors tail -f app.log | hl ERROR:bred WARN:byellow INFO:cyan +# ...and keep it live through a further filter +tail -f app.log | hl -u ERROR:bred | grep -v healthcheck + # Alternation (GNU sed) journalctl -f | hl '/error\|fatal\|panic/':bred @@ -191,7 +210,15 @@ BRE - you escape them to make them special, which is the opposite of PCRE. - **`.` and negated classes like `[^x]` can match the internal marker bytes** `hl` uses to track highlights, which can produce odd output when combined with earlier patterns. -- **Maximum 12 `pattern:color` pairs** per invocation. +- **At most 15 distinct colors** per invocation (the whole palette); any number + of patterns can share them. + +## Tests + +```bash +./tests/test.sh # plain Bash, no dependencies +AWK=gawk ./tests/test.sh # run hl's pipeline under a specific awk +``` ## Contributing diff --git a/hl b/hl index 816c98d..390ef41 100755 --- a/hl +++ b/hl @@ -14,7 +14,13 @@ ## hl '/H[0-9]/':blue ## ## USAGE: -## echo "text" | hl pattern:color [pattern:color ...] +## echo "text" | hl [-u] pattern:color [pattern:color ...] +## +## OPTIONS: +## -u, --line-buffered flush every line, for live pipelines that continue +## after hl (tail -f log | hl ERROR:red | grep x). +## Automatic when output is a terminal. +## -h, --help show this help ## ## EXAMPLES: ## echo "The car is an H1 model" | hl car:red @@ -87,7 +93,9 @@ STRIP='\004-\007\013\014\020-\032\034-\037' # scru MAX_COLORS=${#MARKS} usage() { - sed -n '3,50p' "$0" | sed 's/^#*//; s/^ //' + # Everything between the opening banner and the next one, so the help can + # grow without this line range going stale. + sed -n '3,/^#\{20,\}$/p' "$0" | sed '$d; s/^#*//; s/^ //' exit "${1:-0}" } @@ -189,9 +197,29 @@ escape_bre() { exec cat } -case "${1:-}" in - -h | --help) usage 0 ;; -esac +# Line buffering (see "Streaming" below): on by default when stdout is a +# terminal, which covers `tail -f | hl`; -u / --line-buffered forces it for +# pipelines such as `tail -f log | hl ERROR:red | grep ...`. +LINE_BUFFERED=0 +LINE_BUFFERED_ASKED=0 +[[ -t 1 ]] && LINE_BUFFERED=1 + +while [[ $# -gt 0 ]]; do + case "$1" in + -h | --help) usage 0 ;; + -u | --line-buffered) + LINE_BUFFERED=1 + LINE_BUFFERED_ASKED=1 + shift + ;; + --) + shift + break + ;; + *) break ;; + esac +done +[[ $# -gt 0 ]] || die "no pattern:color given" ############################################################################### ## Colors are interned: repeated colors reuse the same mark, so 40 patterns @@ -254,11 +282,46 @@ done ## One streaming pipeline: scrub markers, run every substitution in a single ## sed pass, then resolve markers to ANSI codes with a color stack so nested ## highlights restore the enclosing color instead of clearing it. +## +## Streaming: every stage must pass each line on as soon as it has it, or +## `tail -f | hl` shows nothing until a buffer (4-64 KB) fills. tr has no +## line-buffered mode at all, so input scrubbing happens inside sed instead; +## sed gets -u (GNU, FreeBSD/macOS) or -l (older BSD); mawk, Debian/Ubuntu's +## default awk, also buffers INPUT from a pipe unless given -W interactive; +## and every awk flushes after each line. ############################################################################### -tr -d "$STRIP" | - sed "${sed_args[@]}" | - awk -v marks="$MARKS" -v cl="$CLOSE" -v sep="$SEP" \ - -v cols="$colors" -v reset="$RESET" ' +## Flushing per line costs throughput (about 7x on a 200k-line file), so it +## is only done when LINE_BUFFERED is set: by default when stdout is a +## terminal, or with -u / --line-buffered. +SED_STREAM=() +AWK_STREAM=() +if [[ $LINE_BUFFERED -eq 1 ]]; then + if sed -u -n '' /dev/null 2>&1; then + SED_STREAM=(-u) + elif sed -l -n '' /dev/null 2>&1; then + SED_STREAM=(-l) + elif [[ $LINE_BUFFERED_ASKED -eq 1 ]]; then + # BusyBox sed (Alpine, most minimal images) has neither flag, so its + # output stays block-buffered. Say so when -u was explicitly asked + # for, rather than silently not doing it. + printf '%s\n' "hl: this sed cannot line-buffer; output may lag" \ + " (install GNU sed, e.g. 'apk add sed' on Alpine)" >&2 + fi + case "$(awk -W version &1 || true)" in + *mawk*) AWK_STREAM=(-W interactive) ;; + esac +fi + +# Delete marker bytes from the input (same set as STRIP, listed byte by byte +# so no locale-dependent range is involved). +STRIP_EXPR=$'s/[\004\005\006\007\013\014\020\021\022\023\024\025\026\027\030\031\032\034\035\036\037]//g' + +# ${a[@]+"${a[@]}"} expands an empty array to nothing; plain "${a[@]}" is an +# "unbound variable" error under set -u on bash < 4.4. +sed ${SED_STREAM[@]+"${SED_STREAM[@]}"} -e "$STRIP_EXPR" "${sed_args[@]}" | + awk ${AWK_STREAM[@]+"${AWK_STREAM[@]}"} \ + -v marks="$MARKS" -v cl="$CLOSE" -v sep="$SEP" \ + -v cols="$colors" -v reset="$RESET" -v flush="$LINE_BUFFERED" ' BEGIN { split(cols, C, sep); re = "[" marks cl "]" } { line = $0; out = ""; d = 0 @@ -275,4 +338,5 @@ tr -d "$STRIP" | line = substr(line, RSTART + 1) } print out line + if (flush) fflush() }' diff --git a/tests/test.sh b/tests/test.sh new file mode 100755 index 0000000..e0c95a5 --- /dev/null +++ b/tests/test.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +############################################################################### +## Tests for hl. No dependencies beyond what hl itself needs. +## +## ./tests/test.sh # uses ./hl +## HL=/path/to/hl ./tests/test.sh # test another copy +## AWK=gawk ./tests/test.sh # run the pipeline under a specific awk +############################################################################### +set -uo pipefail + +HERE=$(cd "$(dirname "$0")" && pwd) +HL=${HL:-$HERE/../hl} + +# Run hl under a chosen awk by putting a shim named "awk" first on PATH. +if [[ -n "${AWK:-}" ]]; then + shim=$(mktemp -d) + ln -s "$(command -v "$AWK")" "$shim/awk" + PATH="$shim:$PATH" + export PATH + trap 'rm -rf "$shim"' EXIT +fi + +# Fixed ANSI codes regardless of the terminal: TERM=dumb makes tput fail, so +# hl falls back to its built-in escape sequences. +export TERM=dumb +R=$'\033[0;31m' # red +G=$'\033[0;32m' # green +B=$'\033[0;34m' # blue +BR=$'\033[1;31m' # bright red +X=$'\033[0m' # reset + +pass=0 +fail=0 + +check() { # name expected actual + if [[ "$2" == "$3" ]]; then + pass=$((pass + 1)) + else + fail=$((fail + 1)) + printf 'FAIL: %s\n expected: %q\n actual: %q\n' "$1" "$2" "$3" + fi +} + +hl() { "$HL" "$@"; } + +# --------------------------------------------------------------- matching + +check "literal" "a ${R}car${X} b" "$(printf 'a car b\n' | hl car:red)" +check "every occurrence" "${R}x${X}-${R}x${X}" "$(printf 'x-x\n' | hl x:red)" +check "regex" "H${B}1${X} model" "$(printf 'H1 model\n' | hl '/[0-9]/:blue')" +check "regex interval" "at ${BR}09:30${X}" \ + "$(printf 'at 09:30\n' | hl '/[0-9]\{2\}:[0-9]\{2\}/:bred')" +check "literal dot is not a wildcard" "${R}a.b${X} axb" "$(printf 'a.b axb\n' | hl a.b:red)" +check "literal star and brackets" "${R}[*]${X}" "$(printf '[*]\n' | hl '[*]:red')" +check "literal caret and dollar" "${R}^\$${X}" "$(printf '^$\n' | hl '^$:red')" +check "literal backslash" "${R}a\\b${X}" "$(printf 'a\\b\n' | hl 'a\b:red')" +check "ampersand in pattern" "${G}a&b${X}" "$(printf 'a&b\n' | hl 'a&b:green')" +check "slash in literal" "${B}path/to${X}" "$(printf 'path/to\n' | hl 'path/to:blue')" +check "colon in pattern (last colon is the color)" "${R}12:30${X}" \ + "$(printf '12:30\n' | hl '12:30:red')" +check "alias color name" "${BR}x${X}" "$(printf 'x\n' | hl x:bright_red)" +check "no match passes the line through" "plain" "$(printf 'plain\n' | hl zzz:red)" +check "empty line" "" "$(printf '\n' | hl x:red)" +check "multiple lines" "${R}a${X}"$'\n'"b"$'\n'"${R}a${X}" "$(printf 'a\nb\na\n' | hl a:red)" + +# ----------------------------------------------------- nesting and markers + +check "nested highlight restores outer color" \ + "${R}ab${G}c${R}de${X}" "$(printf 'abcde\n' | hl abcde:red c:green)" +check "later pattern cannot match escape-code digits" \ + "${R}ERROR${X} ${B}3${X}" "$(printf 'ERROR 3\n' | hl ERROR:red '/[0-9]/:blue')" +check "marker bytes in input are scrubbed" "${R}inject${X}mex" \ + "$(printf 'inject\004me\020x\n' | hl inject:red)" +check "marker bytes in a pattern are scrubbed" "${R}ab${X}" \ + "$(printf 'ab\n' | hl $'a\004b:red')" +check "same color reused across patterns" "${R}a${X} ${R}b${X}" \ + "$(printf 'a b\n' | hl a:red b:red)" + +# ---------------------------------------------------------- errors / usage + +err=$(printf 'x\n' | hl x 2>&1) +check "missing color exits 2" "2" "$(printf 'x\n' | hl x >/dev/null 2>&1; echo $?)" +check "missing color message" "hl: missing color in 'x' (expected pattern:color)" "$err" +check "unknown color exits 2" "2" "$(printf 'x\n' | hl x:mauve >/dev/null 2>&1; echo $?)" +check "empty pattern exits 2" "2" "$(printf 'x\n' | hl :red >/dev/null 2>&1; echo $?)" +check "option without pattern exits 2" "2" "$(printf 'x\n' | hl -u >/dev/null 2>&1; echo $?)" +check "no args passes input through" "x" "$(printf 'x\n' | hl)" +check "-- ends options" "${R}-u${X}" "$(printf -- '-u\n' | hl -- -u:red)" +help=$(hl --help) +check "help starts with title" "hl - simple text highlighter (literal + regex support)" \ + "$(printf '%s\n' "$help" | sed -n '2p')" +check "help includes the last section" "yes" \ + "$(printf '%s\n' "$help" | grep -q '15 distinct colors' && echo yes || echo no)" + +# ------------------------------------------------------------- streaming +# +# Feed one line through a FIFO that stays open (like `tail -f`) and see +# whether it comes out before hl is killed. With -u it must; without -u +# into a pipe it is allowed to buffer (that is the fast path). + +streamed() { # extra hl args... + # Portable (no `timeout`, which macOS lacks): run hl in the background + # writing to a file, give it up to 2s to produce the line, then stop it. + local fifo out pid _ + fifo=$(mktemp -u) + out=$(mktemp) + mkfifo "$fifo" + exec 3<>"$fifo" + printf 'live ERROR line\n' >&3 + "$HL" "$@" ERROR:red <&3 >"$out" 2>/dev/null & + pid=$! + for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do + [[ -s "$out" ]] && break + sleep 0.1 + done + [[ -s "$out" ]] && echo yes || echo no + kill "$pid" 2>/dev/null + wait "$pid" 2>/dev/null + exec 3>&- + rm -f "$fifo" "$out" +} + +skipped=0 +if sed -u -n '' /dev/null 2>&1 || sed -l -n '' /dev/null 2>&1; then + can_stream=1 +else + # e.g. BusyBox sed: hl warns instead of streaming, by design. + can_stream=0 + skipped=3 + printf 'SKIP: streaming tests (this sed has no -u or -l)\n' + check "-u warns when sed cannot line-buffer" "yes" \ + "$(printf 'x\n' | "$HL" -u x:red 2>&1 >/dev/null | grep -q 'cannot line-buffer' && echo yes || echo no)" +fi + +if [[ $can_stream -eq 1 ]]; then + check "-u streams each line through a pipe" "yes" "$(streamed -u)" + check "--line-buffered streams too" "yes" "$(streamed --line-buffered)" +fi + +if [[ $can_stream -eq 1 ]] && command -v python3 >/dev/null 2>&1; then + # stdout is a terminal: streaming must be automatic, no flag needed. + tty_result=$(python3 - "$HL" <<'EOF' +import os, pty, select, subprocess, sys, time +master, slave = pty.openpty() +p = subprocess.Popen([sys.argv[1], "ERROR:red"], stdin=subprocess.PIPE, + stdout=slave, stderr=slave) +os.close(slave) +p.stdin.write(b"live ERROR line\n"); p.stdin.flush() +seen, start = b"", time.time() +while time.time() - start < 2 and b"line" not in seen: + if select.select([master], [], [], 0.1)[0]: + seen += os.read(master, 4096) +print("yes" if b"line" in seen else "no") +p.stdin.close(); p.wait() +EOF +) + check "streams automatically on a terminal" "yes" "$tty_result" +fi + +# ----------------------------------------------------------------- result + +printf '%d passed, %d failed, %d skipped (awk: %s)\n' "$pass" "$fail" "$skipped" \ + "$(awk -W version &1 | head -n 1 || true)" +[[ $fail -eq 0 ]]