diff --git a/change_log.md b/change_log.md index 2bcf8ba3..6e45fa75 100644 --- a/change_log.md +++ b/change_log.md @@ -3,6 +3,7 @@ ## Unreleased ### Added +* Records and JSON sugar (heldout-v3 records slice): `(rec k v …)` record literals with name keys, `(get m k …)` / `(put m k … v)` nested path read/write, `(read_json p)` / `(write_json p v)` one-call file JSON under the existing filesystem grant, `(pluck lst k)`, `(select lst k …)`, `(group_by x lst key)`, `(merge a b …)`. Desugar-only: no opcode or capability; arity entries; Go/JS reject with `UNSUPPORTED_ON_TARGET`; governed scan fails closed through every form. `parse_json` now runs under `-run`. `encode_json` already writes sorted keys, so output is stable. * Dogfood round 4: `apps/fetch_cli`, `apps/template_cli`, `apps/stamp_api`; `(map_has)`, `(when)`/`(unless)`, n-ary `and`/`or` desugar. Suite total HowlFrame 2331 vs Python 3025 / Go 3295 / Node 2927. Journal: `docs/journals/2026-10-10_dogfood_apps_round4.md`. * Dogfood round 3: `apps/wc_cli`, `apps/json_get`, `apps/counter_api`; density sugar `(str …)` / `(nz x d)`; `list_len` accepts string/bytes; `http_server` startup `store_open`/`store_put`/`do`/`let`. Real-app harness extended; totals HowlFrame 1907 vs Python 2395 / Go 2642 / Node 2297. Journal: `docs/journals/2026-10-10_dogfood_apps_round3.md`. * Dogfood round 2: `apps/notes_cli`, `apps/auth_api`, `apps/grep_cli` built on standalone bytecode; `store_open` accepts a runtime URI expression; AST density sugar `(switch …)` and `(inc x)`. Real-app token harness extended with Python/Go/Node ports and parity. Journal: `docs/journals/2026-10-10_dogfood_apps_round2.md`. diff --git a/docs/direct_execution_design.md b/docs/direct_execution_design.md index 4a7748eb..6d1a31da 100644 --- a/docs/direct_execution_design.md +++ b/docs/direct_execution_design.md @@ -33,7 +33,9 @@ Wired into `main()` via a new `-run` boolean flag: ### Explicitly out of scope for Phase 1 -`http_server`/`route`/`middleware`, `struct`/`parse_json`/`res`/`res_json`, `db_connect`/`sql_query`, `spawn`, `fetch`, `llm_generate`/`fuzzy_cast`/`assert_semantic`, `try_let`, `patch`, `with_context`, `test`, `match`, `exec`, `read_file`/`write_file`/`mkdir`, `web_app`/JS target. Attempting any of these under `-run` produces a clear "not supported under -run (Phase 1)" JSON error naming the specific node, not a silent no-op or crash. `read_file`/`write_file`/`mkdir` were deliberately deferred rather than included: the Go backend's usage pattern for these depends on `try_let` to catch the `(value, error)` tuple they return, and `try_let` itself is Phase 2 scope — including the I/O primitives without an error-handling primitive to pair them with would mean inventing an ad hoc error convention with no precedent, better decided alongside `try_let` itself. +`http_server`/`route`/`middleware`, `struct`/`res`/`res_json`, `db_connect`/`sql_query`, `spawn`, `fetch`, `llm_generate`/`fuzzy_cast`/`assert_semantic`, `try_let`, `patch`, `with_context`, `test`, `match`, `exec`, `read_file`/`write_file`/`mkdir`, `web_app`/JS target. Attempting any of these under `-run` produces a clear "not supported under -run (Phase 1)" JSON error naming the specific node, not a silent no-op or crash. `read_file`/`write_file`/`mkdir` were deliberately deferred rather than included: the Go backend's usage pattern for these depends on `try_let` to catch the `(value, error)` tuple they return, and `try_let` itself is Phase 2 scope — including the I/O primitives without an error-handling primitive to pair them with would mean inventing an ad hoc error convention with no precedent, better decided alongside `try_let` itself. + +`parse_json` has since been added to `-run` (heldout-v3 records slice): `(parse_json T text)` decodes an in-memory string, bytes or byte-number list with the same `decodeJSONValue` the bytecode `PARSE_JSON` uses, and needs no capability. ### Value representation diff --git a/docs/reference/construct_coverage.md b/docs/reference/construct_coverage.md index 41fc6237..7c97184b 100644 --- a/docs/reference/construct_coverage.md +++ b/docs/reference/construct_coverage.md @@ -50,7 +50,9 @@ | `filter` | No | None | Yes | No | - | AUTHORITATIVE | | `for` | Yes | None | No | No | - | AUTHORITATIVE | | `fuzzy_cast` | No | None | No | Yes | - | AUTHORITATIVE | +| `get` | No | None | Yes | No | - | AUTHORITATIVE | | `go_import` | No | None | No | Yes | - | AUTHORITATIVE | +| `group_by` | No | None | Yes | No | - | AUTHORITATIVE | | `html_escape` | Yes | None | No | No | - | AUTHORITATIVE | | `http_body` | No | None | Yes | No | - | AUTHORITATIVE | | `http_request` | Yes | network | No | No | - | AUTHORITATIVE | @@ -82,6 +84,7 @@ | `match` | No | None | No | Yes | - | AUTHORITATIVE | | `max` | No | None | Yes | No | - | AUTHORITATIVE | | `max_by` | No | None | Yes | No | - | AUTHORITATIVE | +| `merge` | No | None | Yes | No | - | AUTHORITATIVE | | `middleware` | No | None | No | Yes | - | AUTHORITATIVE | | `min` | No | None | Yes | No | - | AUTHORITATIVE | | `min_by` | No | None | Yes | No | - | AUTHORITATIVE | @@ -97,11 +100,15 @@ | `pad_right` | No | None | Yes | No | - | AUTHORITATIVE | | `parse_json` | Yes | None | No | No | - | AUTHORITATIVE | | `patch` | No | None | Yes | No | - | AUTHORITATIVE | +| `pluck` | No | None | Yes | No | - | AUTHORITATIVE | | `print` | Yes | None | No | No | - | AUTHORITATIVE | +| `put` | No | None | Yes | No | - | AUTHORITATIVE | | `range` | No | None | Yes | No | - | AUTHORITATIVE | | `read_file` | Yes | filesystem | No | No | - | AUTHORITATIVE | +| `read_json` | No | None | Yes | No | - | AUTHORITATIVE | | `read_line` | Yes | None | No | No | - | AUTHORITATIVE | | `read_text` | No | None | Yes | No | - | AUTHORITATIVE | +| `rec` | No | None | Yes | No | - | AUTHORITATIVE | | `reduce` | No | None | Yes | No | - | AUTHORITATIVE | | `regex_match` | Yes | None | No | No | - | AUTHORITATIVE | | `req_header` | Yes | None | No | No | - | AUTHORITATIVE | @@ -116,6 +123,7 @@ | `rtrim` | No | None | Yes | No | - | AUTHORITATIVE | | `schema` | No | None | No | Yes | - | AUTHORITATIVE | | `schema_bridge` | Yes | None | No | No | - | AUTHORITATIVE | +| `select` | No | None | Yes | No | - | AUTHORITATIVE | | `semantic_match` | No | None | No | Yes | - | AUTHORITATIVE | | `set` | Yes | None | No | No | - | AUTHORITATIVE | | `sget` | No | None | Yes | No | - | AUTHORITATIVE | @@ -171,3 +179,4 @@ | `while` | Yes | None | No | No | - | AUTHORITATIVE | | `with_context` | No | None | Yes | No | - | AUTHORITATIVE | | `write_file` | Yes | filesystem | No | No | - | AUTHORITATIVE | +| `write_json` | No | None | Yes | No | - | AUTHORITATIVE | diff --git a/experiments/token-efficiency-heldout-v3/REPORT.md b/experiments/token-efficiency-heldout-v3/REPORT.md new file mode 100644 index 00000000..725d2140 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/REPORT.md @@ -0,0 +1,82 @@ +# Held-out v3: records/JSON slice, measured blind + +Specs in `tasks/` are frozen (authored by bot-assurance-0001, base `d887b96b`, +pinned by `spec_freeze_test.go`) and were not edited. This directory adds the +harness, references and results only. + +## Protocol + +1. **Phase 1, blind.** The records/JSON forms were designed and committed + (`e18ad5d`) before anything in this directory was opened, guided only by + heldout-v2 evidence (records flat at 1.19x Python; cost in s-expression + structure and JSON I/O). +2. **Phase 2.** After that commit the specs were read and the references were + written: Python, JS and Go once (`solutions/`), HowlFrame twice: + - `solutions/howl/` baseline: only features on main `a6bb53b6`. Verified + 35/35 with a binary built from `a6bb53b6` and with the slice binary. + - `postrec/solutions/howl/` the same programs rewritten to use the slice + where it applies (`run_postrec.sh`). They do not build on `a6bb53b6`. +3. `run.sh` / `run_postrec.sh` replay every reference in a fresh sandbox + (`cmd/heldoutv3verify`, same rules as v2: governed build, `-run-bc`, + `filesystem:read=in`, `filesystem:write=out`), then count tokens with + tiktoken 0.8.0 (cl100k_base, o200k_base; `count_tokens.py` reuses the v0 + comment stripping). `TOKBENCH_PYTHON=/workspace/tokvenv/bin/python`. +4. Results are pinned by `results_freeze_test.go` and `postrec_test.go`; + `verification_test.go` checks coverage and source hashes. No gaps. + +## Headline (cl100k_base; o200k_base in `postrec/results/compare.md`) + +| HowlFrame vs | Median per-task ratio before → after | Total tokens before → after | W/T/L before → after | +|---|---|---|---| +| Python | 1.282 → **1.186** | 3783 → 3547 / 2840 (1.332 → **1.249**) | 3/0/32 → **8/0/27** | +| JS | 1.103 → **1.024** | 3783 → 3547 / 3304 (1.145 → **1.074**) | 13/0/22 → **16/0/19** | +| Go | 0.460 → **0.434** | 3783 → 3547 / 8095 (0.467 → **0.438**) | 35/0/0 → **35/0/0** | + +Median tokens per task: HowlFrame 105 → 99, Python 77, JS 94, Go 221. + +| Category | n | Median howl/py before → after | Total howl/py before → after | +|---|---|---|---| +| records | 17 | 1.195 → 1.113 | 1.244 → 1.118 | +| text | 3 | 1.282 → 1.282 | 1.377 → 1.377 | +| table | 3 | 1.411 → 1.400 | 1.431 → 1.427 | +| numbers | 3 | 1.260 → 1.260 | 1.411 → 1.370 | +| lists | 3 | 1.247 → 1.137 | 1.184 → 1.047 | +| strings | 3 | 1.458 → 1.458 | 1.436 → 1.436 | +| logic | 3 | 1.766 → 1.766 | 1.684 → 1.654 | + +## Reading it honestly + +- HowlFrame still **loses to Python on 27 of 35** tasks and on total tokens + (1.25x). The slice moved records from 1.24x to 1.12x of Python (total), not + to parity. Against JS it is now roughly even; against Go it wins every task. +- What paid: `read_json`/`write_json` (every JSON task), `get` (1 token + shorter per read, multi-step paths collapse), `rec` for literal records. +- What did not: `put`/`get` take a static path, so the three tasks with + runtime dotted paths (x01, x04, x17) still loop by hand and stay at + 1.1-1.5x. `merge` is shallow, so deep merge (x02) is still a recursive + defun. `pluck`/`select`/`group_by`/`merge` were not used by any v3 + reference: the blind guess at those shapes did not match this corpus. +- Remaining structural costs seen in the baseline: no type predicate (object + vs array is tested with `(starts_with (encode_json v) "{")`), recursion + needs a typed `defun`, `(call f …)` and explicit `return`, `append` is a + statement (no value), and list equality compares references. +- Non-record categories were untouched by design; their ratios moved only + where a JSON read/write appears. + +## Findings for follow-up (not fixed here) + +- HFIR bytecode: in `(print (encode_json (parse_json Any s)) (let (t (parse_json Any s)) …))` + the first argument prints the raw text `s` instead of the re-encoded value + (reproduces on `a6bb53b6`; AST bytecode and the interpreter are correct). + The parity test in `records_cli_test.go` avoids that shape. +- `encode_json` converts int64 to float64 (`normalizeForJSON`), so integers + above 2^53 lose precision on output. No v3 task hits it. + +## Notes on the frozen specs (from review of #152; specs not edited) + +- `tasks/README.md` says the input file names are new; five reuse earlier + names with new contents (`config.json`, `people.json`, `prices.json`, + `nums.txt`, `words.txt`). Contents and operations are new. +- x01 says "compact JSON" while the oracle uses Python's default separators; + no x01 output contains a separator, so it does not affect scoring. Tighten + the wording in v4. diff --git a/experiments/token-efficiency-heldout-v3/cmd/heldoutv3verify/main.go b/experiments/token-efficiency-heldout-v3/cmd/heldoutv3verify/main.go new file mode 100644 index 00000000..a2ad05c4 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/cmd/heldoutv3verify/main.go @@ -0,0 +1,229 @@ +// Command heldoutv3verify replays every held-out reference in a fresh sandbox +// and compares its single output with the frozen expected value. +package main + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/json" + "flag" + "fmt" + "io/fs" + "os" + "os/exec" + "path/filepath" + "reflect" + "sort" + "strings" + "time" +) + +type Task struct { + ID string `json:"id"` + Category string `json:"category"` + Inputs map[string]string `json:"inputs"` + Output string `json:"output"` + Kind string `json:"kind"` + Expected json.RawMessage `json:"expected"` +} + +type Gap struct { + Task string `json:"task"` + Lang string `json:"lang"` + Reason string `json:"reason"` +} + +type Result struct { + Task string `json:"task"` + Lang string `json:"lang"` + Pass bool `json:"pass"` + SHA256 string `json:"sha256"` + Detail string `json:"detail,omitempty"` +} + +var exts = map[string]string{"howl": "howl", "python": "py", "js": "js", "go": "go"} + +func main() { + root := flag.String("root", "experiments/token-efficiency-heldout-v3", "benchmark root") + howl := flag.String("howlframe", "/tmp/howlframe", "HowlFrame binary") + howlDir := flag.String("howl-dir", "", "HowlFrame solutions directory (default /solutions/howl)") + resultsDir := flag.String("results", "", "results directory (default /results)") + langs := flag.String("langs", "howl,python,js,go", "comma-separated languages to replay (results are only written for the full set)") + flag.Parse() + var manifest struct{ Tasks []Task } + must(readJSON(filepath.Join(*root, "tasks/manifest.json"), &manifest)) + var gaps []Gap + must(readJSON(filepath.Join(*root, "gaps.json"), &gaps)) + gapped := map[string]bool{} + for _, g := range gaps { + gapped[g.Task+"/"+g.Lang] = true + } + work, err := os.MkdirTemp("", "hf-heldout-v3-verify-") + must(err) + fmt.Println("Replay artifacts:", work) + var results []Result + failed := false + for _, lang := range strings.Split(*langs, ",") { + passed, total := 0, 0 + for _, t := range manifest.Tasks { + if gapped[t.ID+"/"+lang] { + continue + } + total++ + src, _ := filepath.Abs(filepath.Join(*root, "solutions", lang, t.ID+"."+exts[lang])) + if lang == "howl" && *howlDir != "" { + src, _ = filepath.Abs(filepath.Join(*howlDir, t.ID+"."+exts[lang])) + } + data, readErr := os.ReadFile(src) + r := Result{Task: t.ID, Lang: lang, SHA256: fmt.Sprintf("%x", sha256.Sum256(data))} + if readErr != nil { + r.Detail = readErr.Error() + } else { + r.Detail = verify(filepath.Join(work, lang+"-"+t.ID), src, lang, *howl, t) + r.Pass = r.Detail == "" + } + if r.Pass { + passed++ + } else { + failed = true + } + fmt.Printf("%s %s pass=%v %s\n", lang, t.ID, r.Pass, r.Detail) + results = append(results, r) + } + fmt.Printf("%s: %d/%d passed\n", lang, passed, total) + } + out, _ := json.MarshalIndent(results, "", " ") + res := filepath.Join(*root, "results") + if *resultsDir != "" { + res = *resultsDir + } + if *langs != "howl,python,js,go" { + if failed { + os.Exit(1) + } + return + } + must(os.MkdirAll(res, 0o755)) + must(os.WriteFile(filepath.Join(res, "verification.json"), append(out, '\n'), 0o644)) + if failed { + os.Exit(1) + } +} + +func verify(dir, src, lang, howl string, t Task) string { + if err := os.MkdirAll(filepath.Join(dir, "out"), 0o755); err != nil { + return err.Error() + } + for p, content := range t.Inputs { + full := filepath.Join(dir, p) + os.MkdirAll(filepath.Dir(full), 0o755) + if err := os.WriteFile(full, []byte(content), 0o644); err != nil { + return err.Error() + } + } + absDir, _ := filepath.Abs(dir) + var cmd []string + switch lang { + case "howl": + art := filepath.Join(filepath.Dir(absDir), filepath.Base(absDir)+".hfbc") + if o, err := run(absDir, 30*time.Second, howl, "build", "--profile", "governed", src, "-o", art); err != nil { + return "build: " + o + } + cmd = []string{howl, "-run-bc", "-allow-caps", "filesystem:read=" + filepath.Join(absDir, "in") + ",filesystem:write=" + filepath.Join(absDir, "out"), art} + case "python": + cmd = []string{"python3", src} + case "js": + cmd = []string{"node", src} + case "go": + bin := absDir + ".bin" + if o, err := run(filepath.Dir(absDir), 60*time.Second, "go", "build", "-o", bin, src); err != nil { + return "go build: " + o + } + cmd = []string{bin} + } + if o, err := run(absDir, 10*time.Second, cmd[0], cmd[1:]...); err != nil { + return "run: " + o + } + // Exactly the inputs (unchanged) plus the one output may exist. + var files []string + filepath.WalkDir(absDir, func(p string, d fs.DirEntry, err error) error { + if err == nil && !d.IsDir() { + rel, _ := filepath.Rel(absDir, p) + files = append(files, filepath.ToSlash(rel)) + } + return nil + }) + want := []string{t.Output} + for p := range t.Inputs { + want = append(want, p) + } + sort.Strings(files) + sort.Strings(want) + if !reflect.DeepEqual(files, want) { + return fmt.Sprintf("file set %v, want %v", files, want) + } + for p, content := range t.Inputs { + got, _ := os.ReadFile(filepath.Join(absDir, p)) + if string(got) != content { + return "input modified: " + p + } + } + got, err := os.ReadFile(filepath.Join(absDir, t.Output)) + if err != nil { + return err.Error() + } + return compare(got, t) +} + +func compare(got []byte, t Task) string { + if t.Kind == "text" { + var want string + json.Unmarshal(t.Expected, &want) + g := strings.TrimRight(string(got), "\n") + if g != strings.TrimRight(want, "\n") { + return fmt.Sprintf("text mismatch: got %q want %q", g, want) + } + return "" + } + var g, w any + if err := json.Unmarshal(got, &g); err != nil { + return "invalid JSON output: " + err.Error() + } + json.Unmarshal(t.Expected, &w) + if !reflect.DeepEqual(g, w) { + return fmt.Sprintf("json mismatch: got %s want %s", bytes.TrimSpace(got), t.Expected) + } + return "" +} + +func run(dir string, timeout time.Duration, name string, args ...string) (string, error) { + ctx, cancel := context.WithTimeout(context.Background(), timeout) + defer cancel() + c := exec.CommandContext(ctx, name, args...) + c.Dir = dir + out, err := c.CombinedOutput() + if ctx.Err() != nil { + return "timeout", ctx.Err() + } + s := string(out) + if len(s) > 400 { + s = s[:400] + } + return s, err +} + +func readJSON(path string, v any) error { + data, err := os.ReadFile(path) + if err != nil { + return err + } + return json.Unmarshal(data, v) +} + +func must(err error) { + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(2) + } +} diff --git a/experiments/token-efficiency-heldout-v3/compare_postrec.py b/experiments/token-efficiency-heldout-v3/compare_postrec.py new file mode 100644 index 00000000..06ea5402 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/compare_postrec.py @@ -0,0 +1,62 @@ +"""Before/after tables: frozen baseline (results/) vs post-records (postrec/results/).""" +import json, statistics +from pathlib import Path + +ROOT = Path(__file__).resolve().parent +LANGS = ("python", "js", "go") + + +def load(d): + rows = json.loads((ROOT / d / "tokens.json").read_text()) + out = {} + for r in rows: + out.setdefault(r["encoding"], {}).setdefault(r["lang"], {})[r["task"]] = r["tokens"] + return out + + +def main(): + before, after = load("results"), load("postrec/results") + cats = {t["id"]: t["category"] for t in json.loads((ROOT / "tasks/manifest.json").read_text())["tasks"]} + ids = sorted(cats) + md = [] + for enc in ("cl100k_base", "o200k_base"): + b, a = before[enc], after[enc] + md.append(f"### Headline ({enc})\n") + md.append("| Comparison | Median per-task ratio before → after | Total-token ratio before → after | W/T/L before → after |") + md.append("|---|---|---|---|") + for l in LANGS: + def stats(h): + rs = [h[t] / b[l][t] for t in ids] + w = sum(h[t] < b[l][t] for t in ids); ti = sum(h[t] == b[l][t] for t in ids) + return statistics.median(rs), sum(h.values()) / sum(b[l].values()), f"{w}/{ti}/{len(ids) - w - ti}" + m0, t0, w0 = stats(b["howl"]); m1, t1, w1 = stats(a["howl"]) + md.append(f"| HowlFrame vs {l} | {m0:.3f} → **{m1:.3f}** | {t0:.3f} → **{t1:.3f}** ({sum(b['howl'].values())} → {sum(a['howl'].values())} / {sum(b[l].values())}) | {w0} → **{w1}** |") + md.append("") + md.append(f"### Per category ({enc}, HowlFrame / Python)\n") + md.append("| Category | n | Median per-task before → after | Total-token before → after |") + md.append("|---|---|---|---|") + for cat in sorted(set(cats.values()), key=lambda c: ["records", "text", "table", "numbers", "lists", "strings", "logic"].index(c)): + cs = [t for t in ids if cats[t] == cat] + f = lambda h: (statistics.median(h[t] / b["python"][t] for t in cs), sum(h[t] for t in cs) / sum(b["python"][t] for t in cs)) + (m0, t0), (m1, t1) = f(b["howl"]), f(a["howl"]) + md.append(f"| {cat} | {len(cs)} | {m0:.3f} → {m1:.3f} | {t0:.3f} → {t1:.3f} |") + md.append("") + md.append("### Per task (HowlFrame tokens before → after; cl100k_base and o200k_base)\n") + md.append("| Task | Cat | cl100k howl | py | js | go | howl/py | o200k howl | py | js | go | howl/py |") + md.append("|---|---|---|---|---|---|---|---|---|---|---|---|") + for t in ids: + cells = [t, cats[t]] + for enc in ("cl100k_base", "o200k_base"): + b, a = before[enc], after[enc] + h0, h1 = b["howl"][t], a["howl"][t] + r0, r1 = h0 / b["python"][t], h1 / b["python"][t] + hv = f"{h0} → {h1}" if h0 != h1 else f"{h0} (=)" + rv = f"{r0:.2f} → **{r1:.2f}**" if r1 < 1 else f"{r0:.2f} → {r1:.2f}" + cells += [hv, str(b["python"][t]), str(b["js"][t]), str(b["go"][t]), rv] + md.append("| " + " | ".join(cells) + " |") + (ROOT / "postrec/results/compare.md").write_text("\n".join(md) + "\n") + print("\n".join(md)) + + +if __name__ == "__main__": + main() diff --git a/experiments/token-efficiency-heldout-v3/count_tokens.py b/experiments/token-efficiency-heldout-v3/count_tokens.py new file mode 100644 index 00000000..bdbe73c3 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/count_tokens.py @@ -0,0 +1,91 @@ +"""Count verified heldout-v3 sources with the v0 tokenizers and stripping rules.""" +import csv, hashlib, importlib.util, json, statistics +from pathlib import Path + +import tiktoken +import tiktoken.load + +ROOT = Path(__file__).resolve().parent +_spec = importlib.util.spec_from_file_location("v0count", ROOT.parent / "token-efficiency-v0/count_tokens.py") +v0 = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(v0) # reuse strip_comments and offline guard unchanged +LANGUAGES = v0.LANGUAGES + + +def main(): + # Defaults reproduce the frozen baseline; the post-records set passes + # --howl-dir postrec/solutions/howl --results postrec/results. + import argparse + ap = argparse.ArgumentParser() + ap.add_argument("--howl-dir", default="solutions/howl") + ap.add_argument("--results", default="results") + opts = ap.parse_args() + res = ROOT / opts.results + if tiktoken.__version__ != "0.8.0": + raise RuntimeError("Install requirements.txt: tiktoken==0.8.0 required") + tiktoken.load.read_file = v0.offline_read + encodings = {n: tiktoken.get_encoding(n) for n in ("cl100k_base", "o200k_base")} + tasks = json.loads((ROOT / "tasks/manifest.json").read_text())["tasks"] + cats = {t["id"]: t["category"] for t in tasks} + gaps = {(g["task"], g["lang"]) for g in json.loads((ROOT / "gaps.json").read_text())} + checks = {(r["task"], r["lang"]): r for r in json.loads((res / "verification.json").read_text())} + expected = {(t["id"], l) for t in tasks for l in LANGUAGES} - gaps + if set(checks) != expected: + raise RuntimeError("verification coverage must be every task x language minus gaps.json") + rows = [] + for t in tasks: + for lang, ext in LANGUAGES.items(): + if (t["id"], lang) in gaps: + continue + src_dir = ROOT / opts.howl_dir if lang == "howl" else ROOT / f"solutions/{lang}" + raw = (src_dir / f"{t['id']}.{ext}").read_bytes() + c = checks[t["id"], lang] + if hashlib.sha256(raw).hexdigest() != c["sha256"] or not c["pass"]: + raise RuntimeError(f"unverified or changed source {t['id']}/{lang}") + src = raw.decode() + stripped = v0.strip_comments(src, lang) + for name, enc in encodings.items(): + rows.append({"task": t["id"], "category": t["category"], "lang": lang, "encoding": name, + "tokens": len(enc.encode(src, disallowed_special=())), + "tokens_nocomments": len(enc.encode(stripped, disallowed_special=())), + "bytes": len(raw), "pass": c["pass"]}) + (res / "tokens.json").write_text(json.dumps(rows, indent=2) + "\n") + with (res / "tokens.csv").open("w", newline="") as f: + w = csv.DictWriter(f, fieldnames=list(rows[0])) + w.writeheader() + w.writerows(rows) + summary = {} + for enc in encodings: + summary[enc] = {} + for metric in ("tokens", "tokens_nocomments"): + by = {l: {r["task"]: r[metric] for r in rows if r["encoding"] == enc and r["lang"] == l} for l in LANGUAGES} + solved = sorted(by["howl"]) + s = {"languages": {l: {"median_all": statistics.median(v.values()), + "median_howl_solved": statistics.median(v[t] for t in solved), + "mean_howl_solved": statistics.mean(v[t] for t in solved)} for l, v in by.items()}, + "howl_vs": {}, "categories": {}} + for l in ("python", "js", "go"): + ratios = [by["howl"][t] / by[l][t] for t in solved] + s["howl_vs"][l] = { + "median_per_task_ratio": statistics.median(ratios), + "ratio_of_medians": statistics.median(by["howl"].values()) / statistics.median(by[l][t] for t in solved), + "wins": sum(by["howl"][t] < by[l][t] for t in solved), + "ties": sum(by["howl"][t] == by[l][t] for t in solved), + "losses": sum(by["howl"][t] > by[l][t] for t in solved), + "gaps": len(by[l]) - len(solved), + } + for cat in sorted(set(cats.values())): + ids = [t for t in by["python"] if cats[t] == cat] + hs = [t for t in ids if t in by["howl"]] + s["categories"][cat] = { + "n": len(ids), "howl_solved": len(hs), + **{f"mean_{l}": (statistics.mean(by[l][t] for t in hs) if hs else None) for l in LANGUAGES}, + "median_ratio_vs_python": statistics.median(by["howl"][t] / by["python"][t] for t in hs) if hs else None, + } + summary[enc][metric] = s + (res / "summary.json").write_text(json.dumps(summary, indent=2) + "\n") + print(json.dumps({e: summary[e]["tokens"]["howl_vs"] for e in summary}, indent=1)) + + +if __name__ == "__main__": + main() diff --git a/experiments/token-efficiency-heldout-v3/gaps.json b/experiments/token-efficiency-heldout-v3/gaps.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/gaps.json @@ -0,0 +1 @@ +[] diff --git a/experiments/token-efficiency-heldout-v3/postrec/results/compare.md b/experiments/token-efficiency-heldout-v3/postrec/results/compare.md new file mode 100644 index 00000000..4c88da71 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/results/compare.md @@ -0,0 +1,79 @@ +### Headline (cl100k_base) + +| Comparison | Median per-task ratio before → after | Total-token ratio before → after | W/T/L before → after | +|---|---|---|---| +| HowlFrame vs python | 1.282 → **1.186** | 1.332 → **1.249** (3783 → 3547 / 2840) | 3/0/32 → **8/0/27** | +| HowlFrame vs js | 1.103 → **1.024** | 1.145 → **1.074** (3783 → 3547 / 3304) | 13/0/22 → **16/0/19** | +| HowlFrame vs go | 0.460 → **0.434** | 0.467 → **0.438** (3783 → 3547 / 8095) | 35/0/0 → **35/0/0** | + +### Per category (cl100k_base, HowlFrame / Python) + +| Category | n | Median per-task before → after | Total-token before → after | +|---|---|---|---| +| records | 17 | 1.195 → 1.113 | 1.244 → 1.118 | +| text | 3 | 1.282 → 1.282 | 1.377 → 1.377 | +| table | 3 | 1.411 → 1.400 | 1.431 → 1.427 | +| numbers | 3 | 1.260 → 1.260 | 1.411 → 1.370 | +| lists | 3 | 1.247 → 1.137 | 1.184 → 1.047 | +| strings | 3 | 1.458 → 1.458 | 1.436 → 1.436 | +| logic | 3 | 1.766 → 1.766 | 1.684 → 1.654 | + +### Headline (o200k_base) + +| Comparison | Median per-task ratio before → after | Total-token ratio before → after | W/T/L before → after | +|---|---|---|---| +| HowlFrame vs python | 1.282 → **1.186** | 1.332 → **1.250** (3790 → 3556 / 2845) | 3/0/32 → **8/0/27** | +| HowlFrame vs js | 1.058 → **0.977** | 1.093 → **1.026** (3790 → 3556 / 3467) | 16/0/19 → **20/0/15** | +| HowlFrame vs go | 0.455 → **0.431** | 0.465 → **0.437** (3790 → 3556 / 8143) | 35/0/0 → **35/0/0** | + +### Per category (o200k_base, HowlFrame / Python) + +| Category | n | Median per-task before → after | Total-token before → after | +|---|---|---|---| +| records | 17 | 1.195 → 1.113 | 1.245 → 1.120 | +| text | 3 | 1.282 → 1.282 | 1.383 → 1.383 | +| table | 3 | 1.380 → 1.370 | 1.411 → 1.407 | +| numbers | 3 | 1.273 → 1.273 | 1.413 → 1.372 | +| lists | 3 | 1.264 → 1.147 | 1.189 → 1.056 | +| strings | 3 | 1.458 → 1.458 | 1.436 → 1.436 | +| logic | 3 | 1.766 → 1.766 | 1.688 → 1.658 | + +### Per task (HowlFrame tokens before → after; cl100k_base and o200k_base) + +| Task | Cat | cl100k howl | py | js | go | howl/py | o200k howl | py | js | go | howl/py | +|---|---|---|---|---|---|---|---|---|---|---|---| +| x01 | records | 127 → 122 | 110 | 139 | 314 | 1.15 → 1.11 | 129 → 124 | 112 | 146 | 316 | 1.15 → 1.11 | +| x02 | records | 131 → 121 | 106 | 132 | 268 | 1.24 → 1.14 | 131 → 121 | 106 | 139 | 269 | 1.24 → 1.14 | +| x03 | records | 127 → 119 | 102 | 137 | 253 | 1.25 → 1.17 | 127 → 119 | 101 | 142 | 254 | 1.26 → 1.18 | +| x04 | records | 140 → 130 | 86 | 101 | 266 | 1.63 → 1.51 | 140 → 130 | 86 | 105 | 267 | 1.63 → 1.51 | +| x05 | records | 125 → 108 | 97 | 93 | 248 | 1.29 → 1.11 | 125 → 108 | 97 | 97 | 249 | 1.29 → 1.11 | +| x06 | records | 163 → 143 | 111 | 107 | 312 | 1.47 → 1.29 | 163 → 143 | 111 | 111 | 314 | 1.47 → 1.29 | +| x07 | records | 89 → 76 | 79 | 90 | 249 | 1.13 → **0.96** | 89 → 76 | 79 | 96 | 250 | 1.13 → **0.96** | +| x08 | records | 25 → 20 | 46 | 106 | 149 | 0.54 → **0.43** | 25 → 21 | 46 | 111 | 150 | 0.54 → **0.46** | +| x09 | records | 129 → 109 | 113 | 134 | 313 | 1.14 → **0.96** | 129 → 109 | 113 | 140 | 316 | 1.14 → **0.96** | +| x10 | records | 71 → 62 | 66 | 63 | 201 | 1.08 → **0.94** | 71 → 62 | 66 | 67 | 202 | 1.08 → **0.94** | +| x11 | records | 89 → 79 | 81 | 59 | 221 | 1.10 → **0.98** | 89 → 79 | 81 | 63 | 222 | 1.10 → **0.98** | +| x12 | records | 72 → 62 | 66 | 64 | 200 | 1.09 → **0.94** | 73 → 63 | 67 | 69 | 203 | 1.09 → **0.94** | +| x13 | records | 92 → 86 | 77 | 84 | 229 | 1.19 → 1.12 | 92 → 86 | 77 | 88 | 230 | 1.19 → 1.12 | +| x14 | records | 91 → 83 | 70 | 89 | 200 | 1.30 → 1.19 | 91 → 83 | 70 | 93 | 201 | 1.30 → 1.19 | +| x15 | records | 129 → 124 | 82 | 71 | 220 | 1.57 → 1.51 | 129 → 124 | 82 | 75 | 221 | 1.57 → 1.51 | +| x16 | records | 58 → 45 | 60 | 68 | 230 | 0.97 → **0.75** | 58 → 45 | 60 | 72 | 233 | 0.97 → **0.75** | +| x17 | records | 170 → 154 | 117 | 122 | 315 | 1.45 → 1.32 | 170 → 154 | 117 | 128 | 316 | 1.45 → 1.32 | +| x18 | text | 91 (=) | 71 | 87 | 175 | 1.28 → 1.28 | 91 (=) | 71 | 92 | 176 | 1.28 → 1.28 | +| x19 | text | 80 (=) | 70 | 81 | 174 | 1.14 → 1.14 | 80 (=) | 70 | 86 | 176 | 1.14 → 1.14 | +| x20 | text | 125 (=) | 74 | 79 | 205 | 1.69 → 1.69 | 125 (=) | 73 | 83 | 207 | 1.71 → 1.71 | +| x21 | table | 74 (=) | 55 | 77 | 199 | 1.35 → 1.35 | 75 (=) | 55 | 81 | 200 | 1.36 → 1.36 | +| x22 | table | 131 (=) | 87 | 95 | 201 | 1.51 → 1.51 | 131 (=) | 89 | 99 | 203 | 1.47 → 1.47 | +| x23 | table | 127 → 126 | 90 | 106 | 212 | 1.41 → 1.40 | 127 → 126 | 92 | 111 | 213 | 1.38 → 1.37 | +| x24 | numbers | 165 → 155 | 94 | 105 | 246 | 1.76 → 1.65 | 166 → 156 | 95 | 110 | 247 | 1.75 → 1.64 | +| x25 | numbers | 85 (=) | 75 | 96 | 196 | 1.13 → 1.13 | 85 (=) | 75 | 100 | 197 | 1.13 → 1.13 | +| x26 | numbers | 97 (=) | 77 | 103 | 241 | 1.26 → 1.26 | 98 (=) | 77 | 107 | 242 | 1.27 → 1.27 | +| x27 | lists | 91 → 83 | 73 | 48 | 221 | 1.25 → 1.14 | 91 → 84 | 72 | 52 | 221 | 1.26 → 1.17 | +| x28 | lists | 126 → 109 | 95 | 100 | 253 | 1.33 → 1.15 | 126 → 109 | 95 | 106 | 254 | 1.33 → 1.15 | +| x29 | lists | 60 → 53 | 66 | 63 | 203 | 0.91 → **0.80** | 60 → 53 | 66 | 69 | 204 | 0.91 → **0.80** | +| x30 | strings | 82 (=) | 68 | 94 | 213 | 1.21 → 1.21 | 82 (=) | 68 | 98 | 214 | 1.21 → 1.21 | +| x31 | strings | 105 (=) | 72 | 85 | 212 | 1.46 → 1.46 | 105 (=) | 72 | 89 | 213 | 1.46 → 1.46 | +| x32 | strings | 73 (=) | 41 | 70 | 139 | 1.78 → 1.78 | 73 (=) | 41 | 75 | 141 | 1.78 → 1.78 | +| x33 | logic | 196 (=) | 111 | 152 | 338 | 1.77 → 1.77 | 196 (=) | 111 | 156 | 340 | 1.77 → 1.77 | +| x34 | logic | 140 (=) | 73 | 107 | 196 | 1.92 → 1.92 | 141 (=) | 73 | 110 | 197 | 1.93 → 1.93 | +| x35 | logic | 107 → 99 | 79 | 97 | 283 | 1.35 → 1.25 | 107 → 99 | 79 | 101 | 285 | 1.35 → 1.25 | diff --git a/experiments/token-efficiency-heldout-v3/postrec/results/summary.json b/experiments/token-efficiency-heldout-v3/postrec/results/summary.json new file mode 100644 index 00000000..80a169ac --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/results/summary.json @@ -0,0 +1,466 @@ +{ + "cl100k_base": { + "tokens": { + "languages": { + "howl": { + "median_all": 99, + "median_howl_solved": 99, + "mean_howl_solved": 101.34285714285714 + }, + "python": { + "median_all": 77, + "median_howl_solved": 77, + "mean_howl_solved": 81.14285714285714 + }, + "js": { + "median_all": 94, + "median_howl_solved": 94, + "mean_howl_solved": 94.4 + }, + "go": { + "median_all": 221, + "median_howl_solved": 221, + "mean_howl_solved": 231.28571428571428 + } + }, + "howl_vs": { + "python": { + "median_per_task_ratio": 1.1857142857142857, + "ratio_of_medians": 1.2857142857142858, + "wins": 8, + "ties": 0, + "losses": 27, + "gaps": 0 + }, + "js": { + "median_per_task_ratio": 1.0238095238095237, + "ratio_of_medians": 1.053191489361702, + "wins": 16, + "ties": 0, + "losses": 19, + "gaps": 0 + }, + "go": { + "median_per_task_ratio": 0.4336734693877551, + "ratio_of_medians": 0.4479638009049774, + "wins": 35, + "ties": 0, + "losses": 0, + "gaps": 0 + } + }, + "categories": { + "lists": { + "n": 3, + "howl_solved": 3, + "mean_howl": 81.66666666666667, + "mean_python": 78, + "mean_js": 70.33333333333333, + "mean_go": 225.66666666666666, + "median_ratio_vs_python": 1.1369863013698631 + }, + "logic": { + "n": 3, + "howl_solved": 3, + "mean_howl": 145, + "mean_python": 87.66666666666667, + "mean_js": 118.66666666666667, + "mean_go": 272.3333333333333, + "median_ratio_vs_python": 1.7657657657657657 + }, + "numbers": { + "n": 3, + "howl_solved": 3, + "mean_howl": 112.33333333333333, + "mean_python": 82, + "mean_js": 101.33333333333333, + "mean_go": 227.66666666666666, + "median_ratio_vs_python": 1.2597402597402598 + }, + "records": { + "n": 17, + "howl_solved": 17, + "mean_howl": 96.6470588235294, + "mean_python": 86.41176470588235, + "mean_js": 97.58823529411765, + "mean_go": 246.35294117647058, + "median_ratio_vs_python": 1.1134020618556701 + }, + "strings": { + "n": 3, + "howl_solved": 3, + "mean_howl": 86.66666666666667, + "mean_python": 60.333333333333336, + "mean_js": 83, + "mean_go": 188, + "median_ratio_vs_python": 1.4583333333333333 + }, + "table": { + "n": 3, + "howl_solved": 3, + "mean_howl": 110.33333333333333, + "mean_python": 77.33333333333333, + "mean_js": 92.66666666666667, + "mean_go": 204, + "median_ratio_vs_python": 1.4 + }, + "text": { + "n": 3, + "howl_solved": 3, + "mean_howl": 98.66666666666667, + "mean_python": 71.66666666666667, + "mean_js": 82.33333333333333, + "mean_go": 184.66666666666666, + "median_ratio_vs_python": 1.2816901408450705 + } + } + }, + "tokens_nocomments": { + "languages": { + "howl": { + "median_all": 99, + "median_howl_solved": 99, + "mean_howl_solved": 101.34285714285714 + }, + "python": { + "median_all": 77, + "median_howl_solved": 77, + "mean_howl_solved": 81.14285714285714 + }, + "js": { + "median_all": 94, + "median_howl_solved": 94, + "mean_howl_solved": 94.4 + }, + "go": { + "median_all": 215, + "median_howl_solved": 215, + "mean_howl_solved": 225.02857142857144 + } + }, + "howl_vs": { + "python": { + "median_per_task_ratio": 1.1857142857142857, + "ratio_of_medians": 1.2857142857142858, + "wins": 8, + "ties": 0, + "losses": 27, + "gaps": 0 + }, + "js": { + "median_per_task_ratio": 1.0238095238095237, + "ratio_of_medians": 1.053191489361702, + "wins": 16, + "ties": 0, + "losses": 19, + "gaps": 0 + }, + "go": { + "median_per_task_ratio": 0.4462809917355372, + "ratio_of_medians": 0.4604651162790698, + "wins": 35, + "ties": 0, + "losses": 0, + "gaps": 0 + } + }, + "categories": { + "lists": { + "n": 3, + "howl_solved": 3, + "mean_howl": 81.66666666666667, + "mean_python": 78, + "mean_js": 70.33333333333333, + "mean_go": 219.66666666666666, + "median_ratio_vs_python": 1.1369863013698631 + }, + "logic": { + "n": 3, + "howl_solved": 3, + "mean_howl": 145, + "mean_python": 87.66666666666667, + "mean_js": 118.66666666666667, + "mean_go": 266.3333333333333, + "median_ratio_vs_python": 1.7657657657657657 + }, + "numbers": { + "n": 3, + "howl_solved": 3, + "mean_howl": 112.33333333333333, + "mean_python": 82, + "mean_js": 101.33333333333333, + "mean_go": 221.66666666666666, + "median_ratio_vs_python": 1.2597402597402598 + }, + "records": { + "n": 17, + "howl_solved": 17, + "mean_howl": 96.6470588235294, + "mean_python": 86.41176470588235, + "mean_js": 97.58823529411765, + "mean_go": 239.8235294117647, + "median_ratio_vs_python": 1.1134020618556701 + }, + "strings": { + "n": 3, + "howl_solved": 3, + "mean_howl": 86.66666666666667, + "mean_python": 60.333333333333336, + "mean_js": 83, + "mean_go": 182, + "median_ratio_vs_python": 1.4583333333333333 + }, + "table": { + "n": 3, + "howl_solved": 3, + "mean_howl": 110.33333333333333, + "mean_python": 77.33333333333333, + "mean_js": 92.66666666666667, + "mean_go": 198, + "median_ratio_vs_python": 1.4 + }, + "text": { + "n": 3, + "howl_solved": 3, + "mean_howl": 98.66666666666667, + "mean_python": 71.66666666666667, + "mean_js": 82.33333333333333, + "mean_go": 178.66666666666666, + "median_ratio_vs_python": 1.2816901408450705 + } + } + } + }, + "o200k_base": { + "tokens": { + "languages": { + "howl": { + "median_all": 99, + "median_howl_solved": 99, + "mean_howl_solved": 101.6 + }, + "python": { + "median_all": 77, + "median_howl_solved": 77, + "mean_howl_solved": 81.28571428571429 + }, + "js": { + "median_all": 98, + "median_howl_solved": 98, + "mean_howl_solved": 99.05714285714286 + }, + "go": { + "median_all": 221, + "median_howl_solved": 221, + "mean_howl_solved": 232.65714285714284 + } + }, + "howl_vs": { + "python": { + "median_per_task_ratio": 1.1857142857142857, + "ratio_of_medians": 1.2857142857142858, + "wins": 8, + "ties": 0, + "losses": 27, + "gaps": 0 + }, + "js": { + "median_per_task_ratio": 0.9772727272727273, + "ratio_of_medians": 1.010204081632653, + "wins": 20, + "ties": 0, + "losses": 15, + "gaps": 0 + }, + "go": { + "median_per_task_ratio": 0.43147208121827413, + "ratio_of_medians": 0.4479638009049774, + "wins": 35, + "ties": 0, + "losses": 0, + "gaps": 0 + } + }, + "categories": { + "lists": { + "n": 3, + "howl_solved": 3, + "mean_howl": 82, + "mean_python": 77.66666666666667, + "mean_js": 75.66666666666667, + "mean_go": 226.33333333333334, + "median_ratio_vs_python": 1.1473684210526316 + }, + "logic": { + "n": 3, + "howl_solved": 3, + "mean_howl": 145.33333333333334, + "mean_python": 87.66666666666667, + "mean_js": 122.33333333333333, + "mean_go": 274, + "median_ratio_vs_python": 1.7657657657657657 + }, + "numbers": { + "n": 3, + "howl_solved": 3, + "mean_howl": 113, + "mean_python": 82.33333333333333, + "mean_js": 105.66666666666667, + "mean_go": 228.66666666666666, + "median_ratio_vs_python": 1.2727272727272727 + }, + "records": { + "n": 17, + "howl_solved": 17, + "mean_howl": 96.88235294117646, + "mean_python": 86.52941176470588, + "mean_js": 102.47058823529412, + "mean_go": 247.8235294117647, + "median_ratio_vs_python": 1.1134020618556701 + }, + "strings": { + "n": 3, + "howl_solved": 3, + "mean_howl": 86.66666666666667, + "mean_python": 60.333333333333336, + "mean_js": 87.33333333333333, + "mean_go": 189.33333333333334, + "median_ratio_vs_python": 1.4583333333333333 + }, + "table": { + "n": 3, + "howl_solved": 3, + "mean_howl": 110.66666666666667, + "mean_python": 78.66666666666667, + "mean_js": 97, + "mean_go": 205.33333333333334, + "median_ratio_vs_python": 1.3695652173913044 + }, + "text": { + "n": 3, + "howl_solved": 3, + "mean_howl": 98.66666666666667, + "mean_python": 71.33333333333333, + "mean_js": 87, + "mean_go": 186.33333333333334, + "median_ratio_vs_python": 1.2816901408450705 + } + } + }, + "tokens_nocomments": { + "languages": { + "howl": { + "median_all": 99, + "median_howl_solved": 99, + "mean_howl_solved": 101.6 + }, + "python": { + "median_all": 77, + "median_howl_solved": 77, + "mean_howl_solved": 81.28571428571429 + }, + "js": { + "median_all": 98, + "median_howl_solved": 98, + "mean_howl_solved": 99.05714285714286 + }, + "go": { + "median_all": 215, + "median_howl_solved": 215, + "mean_howl_solved": 226.4 + } + }, + "howl_vs": { + "python": { + "median_per_task_ratio": 1.1857142857142857, + "ratio_of_medians": 1.2857142857142858, + "wins": 8, + "ties": 0, + "losses": 27, + "gaps": 0 + }, + "js": { + "median_per_task_ratio": 0.9772727272727273, + "ratio_of_medians": 1.010204081632653, + "wins": 20, + "ties": 0, + "losses": 15, + "gaps": 0 + }, + "go": { + "median_per_task_ratio": 0.4444444444444444, + "ratio_of_medians": 0.4604651162790698, + "wins": 35, + "ties": 0, + "losses": 0, + "gaps": 0 + } + }, + "categories": { + "lists": { + "n": 3, + "howl_solved": 3, + "mean_howl": 82, + "mean_python": 77.66666666666667, + "mean_js": 75.66666666666667, + "mean_go": 220.33333333333334, + "median_ratio_vs_python": 1.1473684210526316 + }, + "logic": { + "n": 3, + "howl_solved": 3, + "mean_howl": 145.33333333333334, + "mean_python": 87.66666666666667, + "mean_js": 122.33333333333333, + "mean_go": 268, + "median_ratio_vs_python": 1.7657657657657657 + }, + "numbers": { + "n": 3, + "howl_solved": 3, + "mean_howl": 113, + "mean_python": 82.33333333333333, + "mean_js": 105.66666666666667, + "mean_go": 222.66666666666666, + "median_ratio_vs_python": 1.2727272727272727 + }, + "records": { + "n": 17, + "howl_solved": 17, + "mean_howl": 96.88235294117646, + "mean_python": 86.52941176470588, + "mean_js": 102.47058823529412, + "mean_go": 241.2941176470588, + "median_ratio_vs_python": 1.1134020618556701 + }, + "strings": { + "n": 3, + "howl_solved": 3, + "mean_howl": 86.66666666666667, + "mean_python": 60.333333333333336, + "mean_js": 87.33333333333333, + "mean_go": 183.33333333333334, + "median_ratio_vs_python": 1.4583333333333333 + }, + "table": { + "n": 3, + "howl_solved": 3, + "mean_howl": 110.66666666666667, + "mean_python": 78.66666666666667, + "mean_js": 97, + "mean_go": 199.33333333333334, + "median_ratio_vs_python": 1.3695652173913044 + }, + "text": { + "n": 3, + "howl_solved": 3, + "mean_howl": 98.66666666666667, + "mean_python": 71.33333333333333, + "mean_js": 87, + "mean_go": 180.33333333333334, + "median_ratio_vs_python": 1.2816901408450705 + } + } + } + } +} diff --git a/experiments/token-efficiency-heldout-v3/postrec/results/tokens.csv b/experiments/token-efficiency-heldout-v3/postrec/results/tokens.csv new file mode 100644 index 00000000..fb49588b --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/results/tokens.csv @@ -0,0 +1,281 @@ +task,category,lang,encoding,tokens,tokens_nocomments,bytes,pass +x01,records,howl,cl100k_base,122,122,379,True +x01,records,howl,o200k_base,124,124,379,True +x01,records,python,cl100k_base,110,110,402,True +x01,records,python,o200k_base,112,112,402,True +x01,records,js,cl100k_base,139,139,471,True +x01,records,js,o200k_base,146,146,471,True +x01,records,go,cl100k_base,314,308,1022,True +x01,records,go,o200k_base,316,310,1022,True +x02,records,howl,cl100k_base,121,121,359,True +x02,records,howl,o200k_base,121,121,359,True +x02,records,python,cl100k_base,106,106,379,True +x02,records,python,o200k_base,106,106,379,True +x02,records,js,cl100k_base,132,132,441,True +x02,records,js,o200k_base,139,139,441,True +x02,records,go,cl100k_base,268,262,812,True +x02,records,go,o200k_base,269,263,812,True +x03,records,howl,cl100k_base,119,119,348,True +x03,records,howl,o200k_base,119,119,348,True +x03,records,python,cl100k_base,102,102,358,True +x03,records,python,o200k_base,101,101,358,True +x03,records,js,cl100k_base,137,137,453,True +x03,records,js,o200k_base,142,142,453,True +x03,records,go,cl100k_base,253,247,774,True +x03,records,go,o200k_base,254,248,774,True +x04,records,howl,cl100k_base,130,130,373,True +x04,records,howl,o200k_base,130,130,373,True +x04,records,python,cl100k_base,86,86,306,True +x04,records,python,o200k_base,86,86,306,True +x04,records,js,cl100k_base,101,101,353,True +x04,records,js,o200k_base,105,105,353,True +x04,records,go,cl100k_base,266,260,841,True +x04,records,go,o200k_base,267,261,841,True +x05,records,howl,cl100k_base,108,108,314,True +x05,records,howl,o200k_base,108,108,314,True +x05,records,python,cl100k_base,97,97,331,True +x05,records,python,o200k_base,97,97,331,True +x05,records,js,cl100k_base,93,93,315,True +x05,records,js,o200k_base,97,97,315,True +x05,records,go,cl100k_base,248,242,804,True +x05,records,go,o200k_base,249,243,804,True +x06,records,howl,cl100k_base,143,143,443,True +x06,records,howl,o200k_base,143,143,443,True +x06,records,python,cl100k_base,111,111,400,True +x06,records,python,o200k_base,111,111,400,True +x06,records,js,cl100k_base,107,107,395,True +x06,records,js,o200k_base,111,111,395,True +x06,records,go,cl100k_base,312,306,1026,True +x06,records,go,o200k_base,314,308,1026,True +x07,records,howl,cl100k_base,76,76,211,True +x07,records,howl,o200k_base,76,76,211,True +x07,records,python,cl100k_base,79,79,275,True +x07,records,python,o200k_base,79,79,275,True +x07,records,js,cl100k_base,90,90,319,True +x07,records,js,o200k_base,96,96,319,True +x07,records,go,cl100k_base,249,243,780,True +x07,records,go,o200k_base,250,244,780,True +x08,records,howl,cl100k_base,20,20,75,True +x08,records,howl,o200k_base,21,21,75,True +x08,records,python,cl100k_base,46,46,185,True +x08,records,python,o200k_base,46,46,185,True +x08,records,js,cl100k_base,106,106,361,True +x08,records,js,o200k_base,111,111,361,True +x08,records,go,cl100k_base,149,134,502,True +x08,records,go,o200k_base,150,135,502,True +x09,records,howl,cl100k_base,109,109,331,True +x09,records,howl,o200k_base,109,109,331,True +x09,records,python,cl100k_base,113,113,397,True +x09,records,python,o200k_base,113,113,397,True +x09,records,js,cl100k_base,134,134,478,True +x09,records,js,o200k_base,140,140,478,True +x09,records,go,cl100k_base,313,307,988,True +x09,records,go,o200k_base,316,310,988,True +x10,records,howl,cl100k_base,62,62,186,True +x10,records,howl,o200k_base,62,62,186,True +x10,records,python,cl100k_base,66,66,227,True +x10,records,python,o200k_base,66,66,227,True +x10,records,js,cl100k_base,63,63,225,True +x10,records,js,o200k_base,67,67,225,True +x10,records,go,cl100k_base,201,195,685,True +x10,records,go,o200k_base,202,196,685,True +x11,records,howl,cl100k_base,79,79,231,True +x11,records,howl,o200k_base,79,79,231,True +x11,records,python,cl100k_base,81,81,296,True +x11,records,python,o200k_base,81,81,296,True +x11,records,js,cl100k_base,59,59,218,True +x11,records,js,o200k_base,63,63,218,True +x11,records,go,cl100k_base,221,215,755,True +x11,records,go,o200k_base,222,216,755,True +x12,records,howl,cl100k_base,62,62,180,True +x12,records,howl,o200k_base,63,63,180,True +x12,records,python,cl100k_base,66,66,226,True +x12,records,python,o200k_base,67,67,226,True +x12,records,js,cl100k_base,64,64,226,True +x12,records,js,o200k_base,69,69,226,True +x12,records,go,cl100k_base,200,194,666,True +x12,records,go,o200k_base,203,197,666,True +x13,records,howl,cl100k_base,86,86,248,True +x13,records,howl,o200k_base,86,86,248,True +x13,records,python,cl100k_base,77,77,270,True +x13,records,python,o200k_base,77,77,270,True +x13,records,js,cl100k_base,84,84,290,True +x13,records,js,o200k_base,88,88,290,True +x13,records,go,cl100k_base,229,223,739,True +x13,records,go,o200k_base,230,224,739,True +x14,records,howl,cl100k_base,83,83,217,True +x14,records,howl,o200k_base,83,83,217,True +x14,records,python,cl100k_base,70,70,230,True +x14,records,python,o200k_base,70,70,230,True +x14,records,js,cl100k_base,89,89,287,True +x14,records,js,o200k_base,93,93,287,True +x14,records,go,cl100k_base,200,194,652,True +x14,records,go,o200k_base,201,195,652,True +x15,records,howl,cl100k_base,124,124,350,True +x15,records,howl,o200k_base,124,124,350,True +x15,records,python,cl100k_base,82,82,307,True +x15,records,python,o200k_base,82,82,307,True +x15,records,js,cl100k_base,71,71,244,True +x15,records,js,o200k_base,75,75,244,True +x15,records,go,cl100k_base,220,214,686,True +x15,records,go,o200k_base,221,215,686,True +x16,records,howl,cl100k_base,45,45,136,True +x16,records,howl,o200k_base,45,45,136,True +x16,records,python,cl100k_base,60,60,214,True +x16,records,python,o200k_base,60,60,214,True +x16,records,js,cl100k_base,68,68,242,True +x16,records,js,o200k_base,72,72,242,True +x16,records,go,cl100k_base,230,224,771,True +x16,records,go,o200k_base,233,227,771,True +x17,records,howl,cl100k_base,154,154,451,True +x17,records,howl,o200k_base,154,154,451,True +x17,records,python,cl100k_base,117,117,424,True +x17,records,python,o200k_base,117,117,424,True +x17,records,js,cl100k_base,122,122,440,True +x17,records,js,o200k_base,128,128,440,True +x17,records,go,cl100k_base,315,309,987,True +x17,records,go,o200k_base,316,310,987,True +x18,text,howl,cl100k_base,91,91,264,True +x18,text,howl,o200k_base,91,91,264,True +x18,text,python,cl100k_base,71,71,272,True +x18,text,python,o200k_base,71,71,272,True +x18,text,js,cl100k_base,87,87,299,True +x18,text,js,o200k_base,92,92,299,True +x18,text,go,cl100k_base,175,169,591,True +x18,text,go,o200k_base,176,170,591,True +x19,text,howl,cl100k_base,80,80,231,True +x19,text,howl,o200k_base,80,80,231,True +x19,text,python,cl100k_base,70,70,277,True +x19,text,python,o200k_base,70,70,277,True +x19,text,js,cl100k_base,81,81,294,True +x19,text,js,o200k_base,86,86,294,True +x19,text,go,cl100k_base,174,168,604,True +x19,text,go,o200k_base,176,170,604,True +x20,text,howl,cl100k_base,125,125,333,True +x20,text,howl,o200k_base,125,125,333,True +x20,text,python,cl100k_base,74,74,231,True +x20,text,python,o200k_base,73,73,231,True +x20,text,js,cl100k_base,79,79,240,True +x20,text,js,o200k_base,83,83,240,True +x20,text,go,cl100k_base,205,199,694,True +x20,text,go,o200k_base,207,201,694,True +x21,table,howl,cl100k_base,74,74,233,True +x21,table,howl,o200k_base,75,75,233,True +x21,table,python,cl100k_base,55,55,187,True +x21,table,python,o200k_base,55,55,187,True +x21,table,js,cl100k_base,77,77,251,True +x21,table,js,o200k_base,81,81,251,True +x21,table,go,cl100k_base,199,193,683,True +x21,table,go,o200k_base,200,194,683,True +x22,table,howl,cl100k_base,131,131,375,True +x22,table,howl,o200k_base,131,131,375,True +x22,table,python,cl100k_base,87,87,306,True +x22,table,python,o200k_base,89,89,306,True +x22,table,js,cl100k_base,95,95,323,True +x22,table,js,o200k_base,99,99,323,True +x22,table,go,cl100k_base,201,195,680,True +x22,table,go,o200k_base,203,197,680,True +x23,table,howl,cl100k_base,126,126,348,True +x23,table,howl,o200k_base,126,126,348,True +x23,table,python,cl100k_base,90,90,303,True +x23,table,python,o200k_base,92,92,303,True +x23,table,js,cl100k_base,106,106,344,True +x23,table,js,o200k_base,111,111,344,True +x23,table,go,cl100k_base,212,206,705,True +x23,table,go,o200k_base,213,207,705,True +x24,numbers,howl,cl100k_base,155,155,369,True +x24,numbers,howl,o200k_base,156,156,369,True +x24,numbers,python,cl100k_base,94,94,315,True +x24,numbers,python,o200k_base,95,95,315,True +x24,numbers,js,cl100k_base,105,105,319,True +x24,numbers,js,o200k_base,110,110,319,True +x24,numbers,go,cl100k_base,246,240,731,True +x24,numbers,go,o200k_base,247,241,731,True +x25,numbers,howl,cl100k_base,85,85,232,True +x25,numbers,howl,o200k_base,85,85,232,True +x25,numbers,python,cl100k_base,75,75,244,True +x25,numbers,python,o200k_base,75,75,244,True +x25,numbers,js,cl100k_base,96,96,289,True +x25,numbers,js,o200k_base,100,100,289,True +x25,numbers,go,cl100k_base,196,190,609,True +x25,numbers,go,o200k_base,197,191,609,True +x26,numbers,howl,cl100k_base,97,97,272,True +x26,numbers,howl,o200k_base,98,98,272,True +x26,numbers,python,cl100k_base,77,77,239,True +x26,numbers,python,o200k_base,77,77,239,True +x26,numbers,js,cl100k_base,103,103,327,True +x26,numbers,js,o200k_base,107,107,327,True +x26,numbers,go,cl100k_base,241,235,807,True +x26,numbers,go,o200k_base,242,236,807,True +x27,lists,howl,cl100k_base,83,83,247,True +x27,lists,howl,o200k_base,84,84,247,True +x27,lists,python,cl100k_base,73,73,256,True +x27,lists,python,o200k_base,72,72,256,True +x27,lists,js,cl100k_base,48,48,181,True +x27,lists,js,o200k_base,52,52,181,True +x27,lists,go,cl100k_base,221,215,695,True +x27,lists,go,o200k_base,221,215,695,True +x28,lists,howl,cl100k_base,109,109,325,True +x28,lists,howl,o200k_base,109,109,325,True +x28,lists,python,cl100k_base,95,95,331,True +x28,lists,python,o200k_base,95,95,331,True +x28,lists,js,cl100k_base,100,100,347,True +x28,lists,js,o200k_base,106,106,347,True +x28,lists,go,cl100k_base,253,247,808,True +x28,lists,go,o200k_base,254,248,808,True +x29,lists,howl,cl100k_base,53,53,149,True +x29,lists,howl,o200k_base,53,53,149,True +x29,lists,python,cl100k_base,66,66,215,True +x29,lists,python,o200k_base,66,66,215,True +x29,lists,js,cl100k_base,63,63,222,True +x29,lists,js,o200k_base,69,69,222,True +x29,lists,go,cl100k_base,203,197,640,True +x29,lists,go,o200k_base,204,198,640,True +x30,strings,howl,cl100k_base,82,82,240,True +x30,strings,howl,o200k_base,82,82,240,True +x30,strings,python,cl100k_base,68,68,233,True +x30,strings,python,o200k_base,68,68,233,True +x30,strings,js,cl100k_base,94,94,291,True +x30,strings,js,o200k_base,98,98,291,True +x30,strings,go,cl100k_base,213,207,687,True +x30,strings,go,o200k_base,214,208,687,True +x31,strings,howl,cl100k_base,105,105,296,True +x31,strings,howl,o200k_base,105,105,296,True +x31,strings,python,cl100k_base,72,72,243,True +x31,strings,python,o200k_base,72,72,243,True +x31,strings,js,cl100k_base,85,85,278,True +x31,strings,js,o200k_base,89,89,278,True +x31,strings,go,cl100k_base,212,206,659,True +x31,strings,go,o200k_base,213,207,659,True +x32,strings,howl,cl100k_base,73,73,200,True +x32,strings,howl,o200k_base,73,73,200,True +x32,strings,python,cl100k_base,41,41,160,True +x32,strings,python,o200k_base,41,41,160,True +x32,strings,js,cl100k_base,70,70,233,True +x32,strings,js,o200k_base,75,75,233,True +x32,strings,go,cl100k_base,139,133,448,True +x32,strings,go,o200k_base,141,135,448,True +x33,logic,howl,cl100k_base,196,196,530,True +x33,logic,howl,o200k_base,196,196,530,True +x33,logic,python,cl100k_base,111,111,339,True +x33,logic,python,o200k_base,111,111,339,True +x33,logic,js,cl100k_base,152,152,458,True +x33,logic,js,o200k_base,156,156,458,True +x33,logic,go,cl100k_base,338,332,989,True +x33,logic,go,o200k_base,340,334,989,True +x34,logic,howl,cl100k_base,140,140,348,True +x34,logic,howl,o200k_base,141,141,348,True +x34,logic,python,cl100k_base,73,73,248,True +x34,logic,python,o200k_base,73,73,248,True +x34,logic,js,cl100k_base,107,107,328,True +x34,logic,js,o200k_base,110,110,328,True +x34,logic,go,cl100k_base,196,190,625,True +x34,logic,go,o200k_base,197,191,625,True +x35,logic,howl,cl100k_base,99,99,304,True +x35,logic,howl,o200k_base,99,99,304,True +x35,logic,python,cl100k_base,79,79,284,True +x35,logic,python,o200k_base,79,79,284,True +x35,logic,js,cl100k_base,97,97,347,True +x35,logic,js,o200k_base,101,101,347,True +x35,logic,go,cl100k_base,283,277,922,True +x35,logic,go,o200k_base,285,279,922,True diff --git a/experiments/token-efficiency-heldout-v3/postrec/results/tokens.json b/experiments/token-efficiency-heldout-v3/postrec/results/tokens.json new file mode 100644 index 00000000..c5dbaa31 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/results/tokens.json @@ -0,0 +1,2802 @@ +[ + { + "task": "x01", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 122, + "tokens_nocomments": 122, + "bytes": 379, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 124, + "tokens_nocomments": 124, + "bytes": 379, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 110, + "tokens_nocomments": 110, + "bytes": 402, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 112, + "tokens_nocomments": 112, + "bytes": 402, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 139, + "tokens_nocomments": 139, + "bytes": 471, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 146, + "tokens_nocomments": 146, + "bytes": 471, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 314, + "tokens_nocomments": 308, + "bytes": 1022, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 316, + "tokens_nocomments": 310, + "bytes": 1022, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 121, + "tokens_nocomments": 121, + "bytes": 359, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 121, + "tokens_nocomments": 121, + "bytes": 359, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 379, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 379, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 132, + "tokens_nocomments": 132, + "bytes": 441, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 139, + "tokens_nocomments": 139, + "bytes": 441, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 268, + "tokens_nocomments": 262, + "bytes": 812, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 269, + "tokens_nocomments": 263, + "bytes": 812, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 119, + "tokens_nocomments": 119, + "bytes": 348, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 119, + "tokens_nocomments": 119, + "bytes": 348, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 102, + "tokens_nocomments": 102, + "bytes": 358, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 101, + "tokens_nocomments": 101, + "bytes": 358, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 137, + "tokens_nocomments": 137, + "bytes": 453, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 142, + "tokens_nocomments": 142, + "bytes": 453, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 253, + "tokens_nocomments": 247, + "bytes": 774, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 254, + "tokens_nocomments": 248, + "bytes": 774, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 130, + "tokens_nocomments": 130, + "bytes": 373, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 130, + "tokens_nocomments": 130, + "bytes": 373, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 86, + "tokens_nocomments": 86, + "bytes": 306, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 86, + "tokens_nocomments": 86, + "bytes": 306, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 101, + "tokens_nocomments": 101, + "bytes": 353, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 105, + "tokens_nocomments": 105, + "bytes": 353, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 266, + "tokens_nocomments": 260, + "bytes": 841, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 267, + "tokens_nocomments": 261, + "bytes": 841, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 108, + "tokens_nocomments": 108, + "bytes": 314, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 108, + "tokens_nocomments": 108, + "bytes": 314, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 331, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 331, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 93, + "tokens_nocomments": 93, + "bytes": 315, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 315, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 248, + "tokens_nocomments": 242, + "bytes": 804, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 249, + "tokens_nocomments": 243, + "bytes": 804, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 143, + "tokens_nocomments": 143, + "bytes": 443, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 143, + "tokens_nocomments": 143, + "bytes": 443, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 400, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 400, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 107, + "tokens_nocomments": 107, + "bytes": 395, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 395, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 312, + "tokens_nocomments": 306, + "bytes": 1026, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 314, + "tokens_nocomments": 308, + "bytes": 1026, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 76, + "tokens_nocomments": 76, + "bytes": 211, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 76, + "tokens_nocomments": 76, + "bytes": 211, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 275, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 275, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 90, + "tokens_nocomments": 90, + "bytes": 319, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 96, + "tokens_nocomments": 96, + "bytes": 319, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 249, + "tokens_nocomments": 243, + "bytes": 780, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 250, + "tokens_nocomments": 244, + "bytes": 780, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 20, + "tokens_nocomments": 20, + "bytes": 75, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 21, + "tokens_nocomments": 21, + "bytes": 75, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 46, + "tokens_nocomments": 46, + "bytes": 185, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 46, + "tokens_nocomments": 46, + "bytes": 185, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 361, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 361, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 149, + "tokens_nocomments": 134, + "bytes": 502, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 150, + "tokens_nocomments": 135, + "bytes": 502, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 109, + "tokens_nocomments": 109, + "bytes": 331, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 109, + "tokens_nocomments": 109, + "bytes": 331, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 113, + "tokens_nocomments": 113, + "bytes": 397, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 113, + "tokens_nocomments": 113, + "bytes": 397, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 134, + "tokens_nocomments": 134, + "bytes": 478, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 140, + "tokens_nocomments": 140, + "bytes": 478, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 313, + "tokens_nocomments": 307, + "bytes": 988, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 316, + "tokens_nocomments": 310, + "bytes": 988, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 62, + "tokens_nocomments": 62, + "bytes": 186, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 62, + "tokens_nocomments": 62, + "bytes": 186, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 227, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 227, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 63, + "tokens_nocomments": 63, + "bytes": 225, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 67, + "tokens_nocomments": 67, + "bytes": 225, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 201, + "tokens_nocomments": 195, + "bytes": 685, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 202, + "tokens_nocomments": 196, + "bytes": 685, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 231, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 231, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 81, + "tokens_nocomments": 81, + "bytes": 296, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 81, + "tokens_nocomments": 81, + "bytes": 296, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 59, + "tokens_nocomments": 59, + "bytes": 218, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 63, + "tokens_nocomments": 63, + "bytes": 218, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 221, + "tokens_nocomments": 215, + "bytes": 755, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 222, + "tokens_nocomments": 216, + "bytes": 755, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 62, + "tokens_nocomments": 62, + "bytes": 180, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 63, + "tokens_nocomments": 63, + "bytes": 180, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 226, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 67, + "tokens_nocomments": 67, + "bytes": 226, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 64, + "tokens_nocomments": 64, + "bytes": 226, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 69, + "tokens_nocomments": 69, + "bytes": 226, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 200, + "tokens_nocomments": 194, + "bytes": 666, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 203, + "tokens_nocomments": 197, + "bytes": 666, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 86, + "tokens_nocomments": 86, + "bytes": 248, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 86, + "tokens_nocomments": 86, + "bytes": 248, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 270, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 270, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 84, + "tokens_nocomments": 84, + "bytes": 290, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 88, + "tokens_nocomments": 88, + "bytes": 290, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 229, + "tokens_nocomments": 223, + "bytes": 739, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 230, + "tokens_nocomments": 224, + "bytes": 739, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 83, + "tokens_nocomments": 83, + "bytes": 217, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 83, + "tokens_nocomments": 83, + "bytes": 217, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 230, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 230, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 287, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 93, + "tokens_nocomments": 93, + "bytes": 287, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 200, + "tokens_nocomments": 194, + "bytes": 652, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 201, + "tokens_nocomments": 195, + "bytes": 652, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 124, + "tokens_nocomments": 124, + "bytes": 350, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 124, + "tokens_nocomments": 124, + "bytes": 350, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 82, + "tokens_nocomments": 82, + "bytes": 307, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 82, + "tokens_nocomments": 82, + "bytes": 307, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 71, + "tokens_nocomments": 71, + "bytes": 244, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 244, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 220, + "tokens_nocomments": 214, + "bytes": 686, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 221, + "tokens_nocomments": 215, + "bytes": 686, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 45, + "tokens_nocomments": 45, + "bytes": 136, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 45, + "tokens_nocomments": 45, + "bytes": 136, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 60, + "tokens_nocomments": 60, + "bytes": 214, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 60, + "tokens_nocomments": 60, + "bytes": 214, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 68, + "tokens_nocomments": 68, + "bytes": 242, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 242, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 230, + "tokens_nocomments": 224, + "bytes": 771, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 233, + "tokens_nocomments": 227, + "bytes": 771, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 154, + "tokens_nocomments": 154, + "bytes": 451, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 154, + "tokens_nocomments": 154, + "bytes": 451, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 117, + "tokens_nocomments": 117, + "bytes": 424, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 117, + "tokens_nocomments": 117, + "bytes": 424, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 122, + "tokens_nocomments": 122, + "bytes": 440, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 128, + "tokens_nocomments": 128, + "bytes": 440, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 315, + "tokens_nocomments": 309, + "bytes": 987, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 316, + "tokens_nocomments": 310, + "bytes": 987, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 91, + "tokens_nocomments": 91, + "bytes": 264, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 91, + "tokens_nocomments": 91, + "bytes": 264, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 71, + "tokens_nocomments": 71, + "bytes": 272, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "python", + "encoding": "o200k_base", + "tokens": 71, + "tokens_nocomments": 71, + "bytes": 272, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 87, + "tokens_nocomments": 87, + "bytes": 299, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "js", + "encoding": "o200k_base", + "tokens": 92, + "tokens_nocomments": 92, + "bytes": 299, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 175, + "tokens_nocomments": 169, + "bytes": 591, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "go", + "encoding": "o200k_base", + "tokens": 176, + "tokens_nocomments": 170, + "bytes": 591, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 80, + "tokens_nocomments": 80, + "bytes": 231, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 80, + "tokens_nocomments": 80, + "bytes": 231, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 277, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "python", + "encoding": "o200k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 277, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 81, + "tokens_nocomments": 81, + "bytes": 294, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "js", + "encoding": "o200k_base", + "tokens": 86, + "tokens_nocomments": 86, + "bytes": 294, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 174, + "tokens_nocomments": 168, + "bytes": 604, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "go", + "encoding": "o200k_base", + "tokens": 176, + "tokens_nocomments": 170, + "bytes": 604, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 125, + "tokens_nocomments": 125, + "bytes": 333, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 125, + "tokens_nocomments": 125, + "bytes": 333, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 74, + "tokens_nocomments": 74, + "bytes": 231, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "python", + "encoding": "o200k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 231, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 240, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "js", + "encoding": "o200k_base", + "tokens": 83, + "tokens_nocomments": 83, + "bytes": 240, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 205, + "tokens_nocomments": 199, + "bytes": 694, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "go", + "encoding": "o200k_base", + "tokens": 207, + "tokens_nocomments": 201, + "bytes": 694, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 74, + "tokens_nocomments": 74, + "bytes": 233, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 233, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 55, + "tokens_nocomments": 55, + "bytes": 187, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "python", + "encoding": "o200k_base", + "tokens": 55, + "tokens_nocomments": 55, + "bytes": 187, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 251, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "js", + "encoding": "o200k_base", + "tokens": 81, + "tokens_nocomments": 81, + "bytes": 251, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 199, + "tokens_nocomments": 193, + "bytes": 683, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "go", + "encoding": "o200k_base", + "tokens": 200, + "tokens_nocomments": 194, + "bytes": 683, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 131, + "tokens_nocomments": 131, + "bytes": 375, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 131, + "tokens_nocomments": 131, + "bytes": 375, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 87, + "tokens_nocomments": 87, + "bytes": 306, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "python", + "encoding": "o200k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 306, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 95, + "tokens_nocomments": 95, + "bytes": 323, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "js", + "encoding": "o200k_base", + "tokens": 99, + "tokens_nocomments": 99, + "bytes": 323, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 201, + "tokens_nocomments": 195, + "bytes": 680, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "go", + "encoding": "o200k_base", + "tokens": 203, + "tokens_nocomments": 197, + "bytes": 680, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 126, + "tokens_nocomments": 126, + "bytes": 348, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 126, + "tokens_nocomments": 126, + "bytes": 348, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 90, + "tokens_nocomments": 90, + "bytes": 303, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "python", + "encoding": "o200k_base", + "tokens": 92, + "tokens_nocomments": 92, + "bytes": 303, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 344, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "js", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 344, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 212, + "tokens_nocomments": 206, + "bytes": 705, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "go", + "encoding": "o200k_base", + "tokens": 213, + "tokens_nocomments": 207, + "bytes": 705, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 155, + "tokens_nocomments": 155, + "bytes": 369, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 156, + "tokens_nocomments": 156, + "bytes": 369, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 94, + "tokens_nocomments": 94, + "bytes": 315, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "python", + "encoding": "o200k_base", + "tokens": 95, + "tokens_nocomments": 95, + "bytes": 315, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 105, + "tokens_nocomments": 105, + "bytes": 319, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "js", + "encoding": "o200k_base", + "tokens": 110, + "tokens_nocomments": 110, + "bytes": 319, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 246, + "tokens_nocomments": 240, + "bytes": 731, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "go", + "encoding": "o200k_base", + "tokens": 247, + "tokens_nocomments": 241, + "bytes": 731, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 85, + "tokens_nocomments": 85, + "bytes": 232, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 85, + "tokens_nocomments": 85, + "bytes": 232, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 244, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "python", + "encoding": "o200k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 244, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 96, + "tokens_nocomments": 96, + "bytes": 289, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "js", + "encoding": "o200k_base", + "tokens": 100, + "tokens_nocomments": 100, + "bytes": 289, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 196, + "tokens_nocomments": 190, + "bytes": 609, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "go", + "encoding": "o200k_base", + "tokens": 197, + "tokens_nocomments": 191, + "bytes": 609, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 272, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 98, + "tokens_nocomments": 98, + "bytes": 272, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 239, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "python", + "encoding": "o200k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 239, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 103, + "tokens_nocomments": 103, + "bytes": 327, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "js", + "encoding": "o200k_base", + "tokens": 107, + "tokens_nocomments": 107, + "bytes": 327, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 241, + "tokens_nocomments": 235, + "bytes": 807, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "go", + "encoding": "o200k_base", + "tokens": 242, + "tokens_nocomments": 236, + "bytes": 807, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 83, + "tokens_nocomments": 83, + "bytes": 247, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 84, + "tokens_nocomments": 84, + "bytes": 247, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 256, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "python", + "encoding": "o200k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 256, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 48, + "tokens_nocomments": 48, + "bytes": 181, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "js", + "encoding": "o200k_base", + "tokens": 52, + "tokens_nocomments": 52, + "bytes": 181, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 221, + "tokens_nocomments": 215, + "bytes": 695, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "go", + "encoding": "o200k_base", + "tokens": 221, + "tokens_nocomments": 215, + "bytes": 695, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 109, + "tokens_nocomments": 109, + "bytes": 325, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 109, + "tokens_nocomments": 109, + "bytes": 325, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 95, + "tokens_nocomments": 95, + "bytes": 331, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "python", + "encoding": "o200k_base", + "tokens": 95, + "tokens_nocomments": 95, + "bytes": 331, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 100, + "tokens_nocomments": 100, + "bytes": 347, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "js", + "encoding": "o200k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 347, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 253, + "tokens_nocomments": 247, + "bytes": 808, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "go", + "encoding": "o200k_base", + "tokens": 254, + "tokens_nocomments": 248, + "bytes": 808, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 53, + "tokens_nocomments": 53, + "bytes": 149, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 53, + "tokens_nocomments": 53, + "bytes": 149, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 215, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "python", + "encoding": "o200k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 215, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 63, + "tokens_nocomments": 63, + "bytes": 222, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "js", + "encoding": "o200k_base", + "tokens": 69, + "tokens_nocomments": 69, + "bytes": 222, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 203, + "tokens_nocomments": 197, + "bytes": 640, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "go", + "encoding": "o200k_base", + "tokens": 204, + "tokens_nocomments": 198, + "bytes": 640, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 82, + "tokens_nocomments": 82, + "bytes": 240, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 82, + "tokens_nocomments": 82, + "bytes": 240, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 68, + "tokens_nocomments": 68, + "bytes": 233, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "python", + "encoding": "o200k_base", + "tokens": 68, + "tokens_nocomments": 68, + "bytes": 233, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 94, + "tokens_nocomments": 94, + "bytes": 291, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "js", + "encoding": "o200k_base", + "tokens": 98, + "tokens_nocomments": 98, + "bytes": 291, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 213, + "tokens_nocomments": 207, + "bytes": 687, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "go", + "encoding": "o200k_base", + "tokens": 214, + "tokens_nocomments": 208, + "bytes": 687, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 105, + "tokens_nocomments": 105, + "bytes": 296, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 105, + "tokens_nocomments": 105, + "bytes": 296, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 243, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "python", + "encoding": "o200k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 243, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 85, + "tokens_nocomments": 85, + "bytes": 278, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "js", + "encoding": "o200k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 278, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 212, + "tokens_nocomments": 206, + "bytes": 659, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "go", + "encoding": "o200k_base", + "tokens": 213, + "tokens_nocomments": 207, + "bytes": 659, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 200, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 200, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 41, + "tokens_nocomments": 41, + "bytes": 160, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "python", + "encoding": "o200k_base", + "tokens": 41, + "tokens_nocomments": 41, + "bytes": 160, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 233, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "js", + "encoding": "o200k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 233, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 139, + "tokens_nocomments": 133, + "bytes": 448, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "go", + "encoding": "o200k_base", + "tokens": 141, + "tokens_nocomments": 135, + "bytes": 448, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 196, + "tokens_nocomments": 196, + "bytes": 530, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 196, + "tokens_nocomments": 196, + "bytes": 530, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 339, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "python", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 339, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 152, + "tokens_nocomments": 152, + "bytes": 458, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "js", + "encoding": "o200k_base", + "tokens": 156, + "tokens_nocomments": 156, + "bytes": 458, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 338, + "tokens_nocomments": 332, + "bytes": 989, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "go", + "encoding": "o200k_base", + "tokens": 340, + "tokens_nocomments": 334, + "bytes": 989, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 140, + "tokens_nocomments": 140, + "bytes": 348, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 141, + "tokens_nocomments": 141, + "bytes": 348, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 248, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "python", + "encoding": "o200k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 248, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 107, + "tokens_nocomments": 107, + "bytes": 328, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "js", + "encoding": "o200k_base", + "tokens": 110, + "tokens_nocomments": 110, + "bytes": 328, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 196, + "tokens_nocomments": 190, + "bytes": 625, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "go", + "encoding": "o200k_base", + "tokens": 197, + "tokens_nocomments": 191, + "bytes": 625, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 99, + "tokens_nocomments": 99, + "bytes": 304, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 99, + "tokens_nocomments": 99, + "bytes": 304, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 284, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "python", + "encoding": "o200k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 284, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 347, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "js", + "encoding": "o200k_base", + "tokens": 101, + "tokens_nocomments": 101, + "bytes": 347, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 283, + "tokens_nocomments": 277, + "bytes": 922, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "go", + "encoding": "o200k_base", + "tokens": 285, + "tokens_nocomments": 279, + "bytes": 922, + "pass": true + } +] diff --git a/experiments/token-efficiency-heldout-v3/postrec/results/verification.json b/experiments/token-efficiency-heldout-v3/postrec/results/verification.json new file mode 100644 index 00000000..179546a7 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/results/verification.json @@ -0,0 +1,842 @@ +[ + { + "task": "x01", + "lang": "howl", + "pass": true, + "sha256": "4d6170b9ae55dcb35f745e2d478b119787cea6eb99119bd94922ee5f26829dc1" + }, + { + "task": "x02", + "lang": "howl", + "pass": true, + "sha256": "715ac825a2e5b2d556e1cbc6b40131bb6df0f783fddfe1f6d6c47e8cf260afbf" + }, + { + "task": "x03", + "lang": "howl", + "pass": true, + "sha256": "aebe4fd99add62f0b4815a8e6456f59157b9645f499ce11bf599557af2e19f45" + }, + { + "task": "x04", + "lang": "howl", + "pass": true, + "sha256": "6705c1c57ed8c08ae57d544fcbc298e214a0a6360f0e169f2d2417d88cecfaf0" + }, + { + "task": "x05", + "lang": "howl", + "pass": true, + "sha256": "181b7b942fe39bb086ea3e21060a21addbbfae80a0a44f46e7efcc334235164f" + }, + { + "task": "x06", + "lang": "howl", + "pass": true, + "sha256": "e4bf00966815071b893fb01de88e064179645a1cf219fdcb3311794684f0f88e" + }, + { + "task": "x07", + "lang": "howl", + "pass": true, + "sha256": "2556e88c3a874e8ed674a40856f2a2b6c5aa711bfa16e4a3bcd99faa13ab5c8f" + }, + { + "task": "x08", + "lang": "howl", + "pass": true, + "sha256": "f9b4ff0bb9efe317da4e9027da34ccb48d5286e2ddc5ed9a838f5b44e8e62ac2" + }, + { + "task": "x09", + "lang": "howl", + "pass": true, + "sha256": "056383a7c938ea2870ba3cf3749199357d03c0e7bb682f726fe6990c46d283b9" + }, + { + "task": "x10", + "lang": "howl", + "pass": true, + "sha256": "3b40d7208213643dfca04d7295b3c86d2e0aa4c5e32aa706abcfaf2fae6dd423" + }, + { + "task": "x11", + "lang": "howl", + "pass": true, + "sha256": "23ec9126b76c7ed5fb6ef1698ebcd5575446a1cd116bed95881b5e91c76a66ac" + }, + { + "task": "x12", + "lang": "howl", + "pass": true, + "sha256": "488feb5e0d4e217df9f0509a8f00eaf9d1bb1a40f01cc6007d985a2dfc2117b1" + }, + { + "task": "x13", + "lang": "howl", + "pass": true, + "sha256": "eee3b431f2b65518429755468fd5329c1968ad0df76f3387fce6255108ccfc70" + }, + { + "task": "x14", + "lang": "howl", + "pass": true, + "sha256": "bbfb38683cc46c728a819feb544b8ecf6d521dec454d69395d4a186d5d4716bc" + }, + { + "task": "x15", + "lang": "howl", + "pass": true, + "sha256": "51117feae2697858f17bb86e81163885c6ab06607946629fde4b770a2607a612" + }, + { + "task": "x16", + "lang": "howl", + "pass": true, + "sha256": "af04bda275a73daef3f889f8a6b38c9ae01b6329b0fc6e511622aa9d49538add" + }, + { + "task": "x17", + "lang": "howl", + "pass": true, + "sha256": "02fd2bcb5f13e90d127d51eadd02babcfe100bf51a106cb97a511c28044c3320" + }, + { + "task": "x18", + "lang": "howl", + "pass": true, + "sha256": "8e492686094411300a951ac64e0f780b54d83989c25dedd040ce22387386b1f8" + }, + { + "task": "x19", + "lang": "howl", + "pass": true, + "sha256": "797a7bdfcc0b6e545f437b7d445ecd26fbd0184418482e836234ff4d3a56cb8f" + }, + { + "task": "x20", + "lang": "howl", + "pass": true, + "sha256": "5e9a366d4d4148917923244b40790b4e509d9f7f96f4c94f4683cf87143da193" + }, + { + "task": "x21", + "lang": "howl", + "pass": true, + "sha256": "83106acc2a0e8e786c416b003eb21c1eca0e75f76193c6a650e59cf0c96ecbfd" + }, + { + "task": "x22", + "lang": "howl", + "pass": true, + "sha256": "8459cb9edff5afcbfcfe75d23d95dbf9e975724138d24eb4c99e1c871f1eb89b" + }, + { + "task": "x23", + "lang": "howl", + "pass": true, + "sha256": "8c4ae0e3173b29e76fdc6000a3b146835d9f81701796b5144c72a7a54a437eb3" + }, + { + "task": "x24", + "lang": "howl", + "pass": true, + "sha256": "afc113e63cb924df37f48c224794e24cd888b538541b27a3e6c71480a201a8b3" + }, + { + "task": "x25", + "lang": "howl", + "pass": true, + "sha256": "6c8d3d87177312c832eabac4ef08b9b05588ebd81c73e6de34e67f72642786b6" + }, + { + "task": "x26", + "lang": "howl", + "pass": true, + "sha256": "f507ece1f166896d142b9e49a3ca85b8f620142cb5f08e3ab9f6fbd3a4d41d3f" + }, + { + "task": "x27", + "lang": "howl", + "pass": true, + "sha256": "bae1b2394c6e8cb097877a024db7cd9797f59a3b261f3aebf4e5dfcd06a090c6" + }, + { + "task": "x28", + "lang": "howl", + "pass": true, + "sha256": "7dcd5e30adf6096c0e3a863b1c5d27eeccacaddfb35514175681579885690527" + }, + { + "task": "x29", + "lang": "howl", + "pass": true, + "sha256": "e6f82d55d610b0e837e4cb72c5dde021b3f391666555f1fd1f49e129ec53249f" + }, + { + "task": "x30", + "lang": "howl", + "pass": true, + "sha256": "492d650079bdf7ae39c1a7a907df1f41018816df53806a0aea57a385c1ae003c" + }, + { + "task": "x31", + "lang": "howl", + "pass": true, + "sha256": "6831ab1b264b63584aa082331b7b237ebf2b2a0adb0f83cf46c92f170379d9b1" + }, + { + "task": "x32", + "lang": "howl", + "pass": true, + "sha256": "c4e32c54f02a5aa1ab3637be5921de3ff0a429ad0f280e86e7b8872f00901abe" + }, + { + "task": "x33", + "lang": "howl", + "pass": true, + "sha256": "7ce93978b39a55db22c0bda75f402d80a472c12120cb5f8b04a696cd06e8e6e4" + }, + { + "task": "x34", + "lang": "howl", + "pass": true, + "sha256": "7f4e4b1ed3b1744d45e5c236ecd15eb3085996b85dfa93b1c1180c7d9a5d37c4" + }, + { + "task": "x35", + "lang": "howl", + "pass": true, + "sha256": "6b996e50453812d82326c39bbb708c29c965b67fe0ee99de0b3efa334cf3653f" + }, + { + "task": "x01", + "lang": "python", + "pass": true, + "sha256": "ccd1d0519c928ea99a16be362296b9b350e0ec981a79991d2429a2565c97e404" + }, + { + "task": "x02", + "lang": "python", + "pass": true, + "sha256": "f61de199ada79c3d78b693b340d70f5d0850661013e824ae02b635416dfe8be2" + }, + { + "task": "x03", + "lang": "python", + "pass": true, + "sha256": "98432ebf8e928e7646dc02f23652a88f90c1314e2cfd991f78b1dfcd37b19555" + }, + { + "task": "x04", + "lang": "python", + "pass": true, + "sha256": "bcbbec67656bc8eec0ac9e95c8057368bd7a7df86914c92c52a20747f810c69a" + }, + { + "task": "x05", + "lang": "python", + "pass": true, + "sha256": "ee21c291704348289d666219e97be9c2b6749e74576d5c4693e66a9c55cb9f24" + }, + { + "task": "x06", + "lang": "python", + "pass": true, + "sha256": "569a5af7ef631e53ec3115c1d9fde860f4785ccb07966ede14f06e71006f1934" + }, + { + "task": "x07", + "lang": "python", + "pass": true, + "sha256": "5111d24b38ad58091b9909e56ee136e8c582d4c7ded227d3f46f0c50ffab407f" + }, + { + "task": "x08", + "lang": "python", + "pass": true, + "sha256": "78447469f75f555fbb3a75e3cf31ace274e10461bc783fa6d5b10ae093ff7280" + }, + { + "task": "x09", + "lang": "python", + "pass": true, + "sha256": "fc4ec9537bccb6356bb1c704df0b2cd7fdcd3ed43da8d7fcdea71c73c7928229" + }, + { + "task": "x10", + "lang": "python", + "pass": true, + "sha256": "8f20446c4ecec53e7a833636c6350f4061df7332c89750eba62ff5cfa608e3fb" + }, + { + "task": "x11", + "lang": "python", + "pass": true, + "sha256": "8935253e2185abfb488efe8cb166ae2f431835d52ccf85ec0e93884717617d3c" + }, + { + "task": "x12", + "lang": "python", + "pass": true, + "sha256": "2de65e25278c91ad606912145e35b2e423f6a23748493ef61ed41d86800cec6c" + }, + { + "task": "x13", + "lang": "python", + "pass": true, + "sha256": "b44a832324e7258b5f0c4764bbef299c211a0ef6f5f8af883399977138ecf8e2" + }, + { + "task": "x14", + "lang": "python", + "pass": true, + "sha256": "5c06b13449327018eb615b198e7b6266552cbdc2d5ded5d25e905ed8d1659914" + }, + { + "task": "x15", + "lang": "python", + "pass": true, + "sha256": "75fcee6da1bea919373ef64cbc42eabe9d00535b41b6849200da68a135fd16e8" + }, + { + "task": "x16", + "lang": "python", + "pass": true, + "sha256": "8eb2ec5b4eba480798e7e12da2f46186c5f3d6207224978c524996bb8821461a" + }, + { + "task": "x17", + "lang": "python", + "pass": true, + "sha256": "10db829777cda742b758f136d01a946313349d9256a26a9f3fb3e55ff77bbc6a" + }, + { + "task": "x18", + "lang": "python", + "pass": true, + "sha256": "3c158f59c2705cebaba14b72bd03844ebfbec59597046b23bbe23b05b5ca73cb" + }, + { + "task": "x19", + "lang": "python", + "pass": true, + "sha256": "222ae1940084838aed29875829a05680b2c9eb7c83852f6bb5458f62423c6ce2" + }, + { + "task": "x20", + "lang": "python", + "pass": true, + "sha256": "8ae5d273e50c34d208670886abc4cb9c079655e189665984c739d4bc52708232" + }, + { + "task": "x21", + "lang": "python", + "pass": true, + "sha256": "9418d868d415c97b092c813e33a08e5381ad32af217a366248090105ee264031" + }, + { + "task": "x22", + "lang": "python", + "pass": true, + "sha256": "6303c392e679e8f47a5ece5bd19d6b9df5bae23ae1a3429b59abde41b7781892" + }, + { + "task": "x23", + "lang": "python", + "pass": true, + "sha256": "b4368c9a96229c8d8a17b45525d8ee63f6b1528fdc9049b707169904735cb0ef" + }, + { + "task": "x24", + "lang": "python", + "pass": true, + "sha256": "a165b1f61b99d5991d340c5562d16e230792a5ff54a07b07ed258a2360c0f582" + }, + { + "task": "x25", + "lang": "python", + "pass": true, + "sha256": "4bf072b325eb139fe805f5dfe82e975e7779b1bc3cb132a28f2149967e47037a" + }, + { + "task": "x26", + "lang": "python", + "pass": true, + "sha256": "7cf7b19621179b98bce78cc196388d81fea9250f0c476ca055dabf65a1cc30ed" + }, + { + "task": "x27", + "lang": "python", + "pass": true, + "sha256": "7bad486d6f4e3188124ae2f7ef89dd3e415b14a7535fc943c4107c71d69206d8" + }, + { + "task": "x28", + "lang": "python", + "pass": true, + "sha256": "387f58955a58072056378c5bdc473d171792120a3b210dfc707a5ddebde0952a" + }, + { + "task": "x29", + "lang": "python", + "pass": true, + "sha256": "edb53b63f2d410706aac4993d676cc748befa6f0f1cf31c4f7da807842889aa5" + }, + { + "task": "x30", + "lang": "python", + "pass": true, + "sha256": "1c4a69290bfb11b954ad852077c0b9fe11f5f8bedb56ecc791e1b132f4d2dddd" + }, + { + "task": "x31", + "lang": "python", + "pass": true, + "sha256": "e017b810a85fb7c1026ed971a015ca2ead300cb5eced6a1f7dfdfd963d60f2aa" + }, + { + "task": "x32", + "lang": "python", + "pass": true, + "sha256": "949c8f57602fe3eb8b01f1a09db1fba8d451173ecb3669d3e2c5c65d181b26d2" + }, + { + "task": "x33", + "lang": "python", + "pass": true, + "sha256": "396a3ca6b2968657ce0b9d0be1d5d170f6b5d072985ab374ea50698df0a0512f" + }, + { + "task": "x34", + "lang": "python", + "pass": true, + "sha256": "e244ca0da1a888c05641fc328c9cc4f480780f36753751f790b40f81203fe7f7" + }, + { + "task": "x35", + "lang": "python", + "pass": true, + "sha256": "0d4ee7acf3c78e67b3901b760e69d32857370677b4723647e9f622c5bce741aa" + }, + { + "task": "x01", + "lang": "js", + "pass": true, + "sha256": "810cb930aba9ddf6dcdcfe6408876ecb61f6d07f7c86f13816515462ed54f84f" + }, + { + "task": "x02", + "lang": "js", + "pass": true, + "sha256": "d1afb52a7b6f54387c2e8bf070fae72901d0af0becfff018ff1bc241baf77669" + }, + { + "task": "x03", + "lang": "js", + "pass": true, + "sha256": "80861908391364abb51b7a9a1410cf5a6016b88cf60e759181fb22c197500579" + }, + { + "task": "x04", + "lang": "js", + "pass": true, + "sha256": "c54f79e0b77cda5fe65dba74e927b5a941d86499504e1dc9490b0678bd01023e" + }, + { + "task": "x05", + "lang": "js", + "pass": true, + "sha256": "64ce091ca0c891e742f67e073b8139eaf555ce81c0d4ccb58297cf9879997aad" + }, + { + "task": "x06", + "lang": "js", + "pass": true, + "sha256": "676fb98689b78e59fd89851e0e38febcb14455d73453d7ce7ee4fac92e67c24e" + }, + { + "task": "x07", + "lang": "js", + "pass": true, + "sha256": "ab2d00a382428b5ef67229c9c128ba502d8f189957335cb6644776c25cfb95a0" + }, + { + "task": "x08", + "lang": "js", + "pass": true, + "sha256": "9d7553aef97bb0e46e13a30e25e75465cdc0ff22f341a4e36bffc59ab3b5bdcf" + }, + { + "task": "x09", + "lang": "js", + "pass": true, + "sha256": "33a326473af951b90e699ed0f8ee43727fd50dd9ea71dde1e66c5eb009519fad" + }, + { + "task": "x10", + "lang": "js", + "pass": true, + "sha256": "1c30fef1559f824e90e59482d2a9addd7e77296dc6ce38b7a5998a39583a955a" + }, + { + "task": "x11", + "lang": "js", + "pass": true, + "sha256": "13d028aa8f691d684504e3d38680dc110ed2d30a58f92d130f71c389af6ae4ec" + }, + { + "task": "x12", + "lang": "js", + "pass": true, + "sha256": "5e1ada75df8a362f25fce523e783e3295c74756867964c333095960cc40983a5" + }, + { + "task": "x13", + "lang": "js", + "pass": true, + "sha256": "51946855c065070ee5b8e70704a469c75ffbb4f373a1ac3b419e7cfdc2df4422" + }, + { + "task": "x14", + "lang": "js", + "pass": true, + "sha256": "874214840db9f5e0501b387a3ccdf6433f0240958f4578aff201ef622d13183b" + }, + { + "task": "x15", + "lang": "js", + "pass": true, + "sha256": "f671cca8a42ff7a509d7da3d2a3bbc6a42de1e3af1a20962ab6b30b34d1368ce" + }, + { + "task": "x16", + "lang": "js", + "pass": true, + "sha256": "090b8df4bdc56bb266d74347c466ef22ed37608c4e3c37abdd7898ac2744889a" + }, + { + "task": "x17", + "lang": "js", + "pass": true, + "sha256": "d0d6357a4168a3badfa00035465ad36b8e3a22b0ed90fe8c38d8087004c2e848" + }, + { + "task": "x18", + "lang": "js", + "pass": true, + "sha256": "ba9d3b417b43705633d865e74772b188ffe8b61899cf0ecefdfe7eb928280dac" + }, + { + "task": "x19", + "lang": "js", + "pass": true, + "sha256": "022c89d96f330e9fdbab304810f6b9a4c8a00dfcf7fe4879e5a5b06a2d496cb9" + }, + { + "task": "x20", + "lang": "js", + "pass": true, + "sha256": "3502bde168ffc00ed111490587f1c3aa42b5e3b8aad247e65d81f8adbbe1aa49" + }, + { + "task": "x21", + "lang": "js", + "pass": true, + "sha256": "5e3b476fe4201867ad3bee01b9a43982088b8df5a9ce99082a3beb3d61495551" + }, + { + "task": "x22", + "lang": "js", + "pass": true, + "sha256": "532ebe15db0043824802797f2f5dcbc1de016e568a5d54286a9cfd775ab9a523" + }, + { + "task": "x23", + "lang": "js", + "pass": true, + "sha256": "67f085ba9a69bca5b1d6fe889950d145cb09ac8a163d000822b9e1427030878f" + }, + { + "task": "x24", + "lang": "js", + "pass": true, + "sha256": "1ca48bb55cd15bfd59058dac995ef30aaf3d0a573e48bb142e9f705178bf0e32" + }, + { + "task": "x25", + "lang": "js", + "pass": true, + "sha256": "34cf79d2da8f6310d0d499c119cce304b3e670827a2e8ad3f75d667aa6e2b566" + }, + { + "task": "x26", + "lang": "js", + "pass": true, + "sha256": "58e23dd101736fdc91e8edc3e4f192693bfc899e22697bb626eb2e1c466ff125" + }, + { + "task": "x27", + "lang": "js", + "pass": true, + "sha256": "1ce778fde075e46aa4867db9d7f045857566328f20be938a3973c043f2679d36" + }, + { + "task": "x28", + "lang": "js", + "pass": true, + "sha256": "13447bab585d93e84963d87cd80471c6938a70f359387f3baf847d224388c318" + }, + { + "task": "x29", + "lang": "js", + "pass": true, + "sha256": "70b485f9d6dc1df0ad81dd62a14ee49fd7748013cc0830e63278cd885cc52f15" + }, + { + "task": "x30", + "lang": "js", + "pass": true, + "sha256": "68edca153194d1d382af767e54faff0c34e228a339055c147e9f5463ce48a097" + }, + { + "task": "x31", + "lang": "js", + "pass": true, + "sha256": "f92896a9840257401f093eaa6e57de86fd798be228bbefe29635a6bea536b353" + }, + { + "task": "x32", + "lang": "js", + "pass": true, + "sha256": "c607224f1e2989710418985b889016e9a5adb5365b50db936b43d7b551bf998e" + }, + { + "task": "x33", + "lang": "js", + "pass": true, + "sha256": "e40f4384cec11f6b75412433b58dc663ff784e24ecec7803e21507b3ccc7b3b5" + }, + { + "task": "x34", + "lang": "js", + "pass": true, + "sha256": "6edf2a05d994f1bdc475853136656bd3e32c0710ae3cdfd38cc0d60ad85e1b3c" + }, + { + "task": "x35", + "lang": "js", + "pass": true, + "sha256": "72b60a642440ff4d67555c1597729d179cf305ab9ae7ec76a720d8ff47eaa23c" + }, + { + "task": "x01", + "lang": "go", + "pass": true, + "sha256": "6284cb689a56dabf5d022c4469a3b4c75a51dad767f054e1c064d925af7d0a1e" + }, + { + "task": "x02", + "lang": "go", + "pass": true, + "sha256": "b108237da81c2515bccf82226cc59041b22837e2cbd1d761c1d9abda1c749b28" + }, + { + "task": "x03", + "lang": "go", + "pass": true, + "sha256": "e77ff75e3310dd64061ab5f74e6eff51f6356cd056e7768110f72c6f5484a792" + }, + { + "task": "x04", + "lang": "go", + "pass": true, + "sha256": "361183ad1cb1834ef34b03e34e443bb69bb9b4ec1ba76b5d02c3cccda6e01354" + }, + { + "task": "x05", + "lang": "go", + "pass": true, + "sha256": "6e79ed287a2bddef12fdb78ae9cfd9813333d6acc8a37f02d9ebac4dbefd613b" + }, + { + "task": "x06", + "lang": "go", + "pass": true, + "sha256": "a16bb40277592823f6627507fc0d77a53f01aabeb41db1b4cb844d1aad60b248" + }, + { + "task": "x07", + "lang": "go", + "pass": true, + "sha256": "d9b156970e08efc73106fcf0fa66e1eaa8ab25d08686179b415301f7608f4aca" + }, + { + "task": "x08", + "lang": "go", + "pass": true, + "sha256": "f7fd216aee38dbcb83a15e3a236bdacd8df9eadd329eeb1852e8d7132165d51b" + }, + { + "task": "x09", + "lang": "go", + "pass": true, + "sha256": "a5c2519795dba802095583cad8ab2bbb03b10ce9b77ff4210d3352cc2300efed" + }, + { + "task": "x10", + "lang": "go", + "pass": true, + "sha256": "c64acf031639bd32a8a238100333da377877ad4187e21f049d61bd59faa808d5" + }, + { + "task": "x11", + "lang": "go", + "pass": true, + "sha256": "ce95a88cdb13a5def4e3ffebb7f4ee87e9e33e88307e04df6f0a515fcf7c3846" + }, + { + "task": "x12", + "lang": "go", + "pass": true, + "sha256": "253077c40025ff0f0e2da9256c26af0930839efea09ea69aed23ef268aede21b" + }, + { + "task": "x13", + "lang": "go", + "pass": true, + "sha256": "aab0e10848a8717c2ea39aa8da33b9a32067af352ca75a96d62ba3e994e5791d" + }, + { + "task": "x14", + "lang": "go", + "pass": true, + "sha256": "71aca530230938ac99975a655e63573f0cd6c55f50742a51a83632b7a86889a6" + }, + { + "task": "x15", + "lang": "go", + "pass": true, + "sha256": "4affe1a3021ed08836dce868c2dcb04cdd72b939c1a6a36b9414cb724c043aae" + }, + { + "task": "x16", + "lang": "go", + "pass": true, + "sha256": "f3e4a6882d6aa38a6d9ac7989e5231d8272eee0399894220ea544caf72144ab6" + }, + { + "task": "x17", + "lang": "go", + "pass": true, + "sha256": "9c07b793bbd7ff1fde0474dada31812f9ef068723cf585718755f4a56f6f06da" + }, + { + "task": "x18", + "lang": "go", + "pass": true, + "sha256": "ba64f2392d55677b78594f72e058d94d8cf94f26d1a2dd8ace4eeaf70e8ac618" + }, + { + "task": "x19", + "lang": "go", + "pass": true, + "sha256": "a6560101135b7c0245f647b1a176fb1b2e749aba789281f41ca971f9f385da97" + }, + { + "task": "x20", + "lang": "go", + "pass": true, + "sha256": "83d12841df9bbf2d26c208ec87882e78163f7f55ea114d11a2937cb2e567e5e3" + }, + { + "task": "x21", + "lang": "go", + "pass": true, + "sha256": "219ae7cc9429e3130125bc30144b1f3a4d549c08a3d6183341a371b7408cd4f1" + }, + { + "task": "x22", + "lang": "go", + "pass": true, + "sha256": "a90d9210696c1dd00de9e4545e65189dd6339cecb41e1f2d63f789e6dc948f9a" + }, + { + "task": "x23", + "lang": "go", + "pass": true, + "sha256": "52df17fcf7f25273cd9b5c142a3c96e213da611c3f120c79f8f98bb82e61471b" + }, + { + "task": "x24", + "lang": "go", + "pass": true, + "sha256": "81a6ac5a293e8d1f9d1482dead7de17c66c3d634628e0ae3e42384c3f59badab" + }, + { + "task": "x25", + "lang": "go", + "pass": true, + "sha256": "5c9c38fcdd02ee32eab70d2de6ff3837092208fbc931483ea4d0a5f25e4b5242" + }, + { + "task": "x26", + "lang": "go", + "pass": true, + "sha256": "5e4b9ad2d5dd164a319e3fe14e9df3eb0c29f5bf126bfb973c5db7c9e9157a66" + }, + { + "task": "x27", + "lang": "go", + "pass": true, + "sha256": "206cdc486dd3e1df770c9f83042bb68d2c3b4c39e12caf0551a28797d39c9667" + }, + { + "task": "x28", + "lang": "go", + "pass": true, + "sha256": "b28081cdbe6ceea24e53e5b64ce1f5274b200287dc032efd782b9a1df7a294fe" + }, + { + "task": "x29", + "lang": "go", + "pass": true, + "sha256": "b211ef65805a242a0cf3e1351fd82f58aa7e4dba3240f8ee6ca44a5edd6a88b1" + }, + { + "task": "x30", + "lang": "go", + "pass": true, + "sha256": "6cb62e3244df173f97285c09ac71adde19cc0a47732f4aa37788ccbe7903b24d" + }, + { + "task": "x31", + "lang": "go", + "pass": true, + "sha256": "1879945760488d3bbcecfa88f7638e148468a5545aea153ea11905a985fd6353" + }, + { + "task": "x32", + "lang": "go", + "pass": true, + "sha256": "9bbb572e9f5127e622dabc940d96056904e5a687796ba18cb3069f422fa4148f" + }, + { + "task": "x33", + "lang": "go", + "pass": true, + "sha256": "5e35144b9519728a74d9a118efbd0b3f52a8bfc43c268ff3e883bbfa94872fcb" + }, + { + "task": "x34", + "lang": "go", + "pass": true, + "sha256": "4c1b037d4491ef23ac9210a4637d34cf8a764c0e69d61604b9a785c9ec19d189" + }, + { + "task": "x35", + "lang": "go", + "pass": true, + "sha256": "db2125c2a9d22e97671c2c5c039f52258fe1aa424a8148b316f3f95c60ddd1a9" + } +] diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x01.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x01.howl new file mode 100644 index 00000000..a353dbe7 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x01.howl @@ -0,0 +1,11 @@ +(do + (let (cfg (read_json "in/config.json")) + (write_file "out/result.txt" + (str_join + (map_list p (lines (read_file "in/paths.txt")) + (let (v cfg ok true) + (do + (for k (str_split p ".") + (if (and (starts_with (encode_json v) "{") (map_has v k)) (set v (get v k)) (do (set ok false) (break)))) + (if ok (encode_json v) "missing")))) + "\n")))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x02.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x02.howl new file mode 100644 index 00000000..5d2dde27 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x02.howl @@ -0,0 +1,9 @@ +(do + (defun merge ((a any) (b any)) any + (do + (for k (map_keys b) + (let (x (get a k) y (get b k)) + (map_set a k (if (and (starts_with (encode_json x) "{") (starts_with (encode_json y) "{")) (call merge x y) y)))) + (return a))) + (write_file "out/result.json" + (encode_json (call merge (read_json "in/base.json") (read_json "in/override.json"))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x03.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x03.howl new file mode 100644 index 00000000..96786248 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x03.howl @@ -0,0 +1,8 @@ +(do + (defun flat ((d any) (pre string) (out any)) any + (do + (for k (map_keys d) + (let (v (get d k)) + (if (and (starts_with (encode_json v) "{") (!= (encode_json v) "{}")) (call flat v (str pre k ".") out) (map_set out (str pre k) v)))) + (return out))) + (write_json "out/result.json" (call flat (read_json "in/nested.json") "" (dict)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x04.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x04.howl new file mode 100644 index 00000000..ea5e966c --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x04.howl @@ -0,0 +1,10 @@ +(do + (let (f (read_json "in/flat.json") out (dict)) + (do + (for k (map_keys f) + (let (ps (str_split k ".") n (list_len ps) node out) + (do + (for p (list_slice ps 0 (- n 1)) + (do (when (is_nil (get node p)) (map_set node p (dict))) (set node (get node p)))) + (map_set node (list_get ps (- n 1)) (get f k))))) + (write_json "out/result.json" out)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x05.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x05.howl new file mode 100644 index 00000000..c4d6865c --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x05.howl @@ -0,0 +1,7 @@ +(do + (let (out (rec present (list) null (list) absent (list))) + (do + (for c (read_json "in/contacts.json") + (let (k (if (not (map_has c "email")) "absent" (is_nil (get c "email")) "null" "present") l (get out k)) + (do (append l (get c "id")) (map_set out k l)))) + (write_json "out/result.json" out)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x06.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x06.howl new file mode 100644 index 00000000..14c8d173 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x06.howl @@ -0,0 +1,11 @@ +(do + (let (out (dict)) + (do + (for s (read_json "in/sales.json") + (let (r (nz (get out (get s "region")) (rec units 0 products (list))) ps (get r "products")) + (do + (map_set r "units" (+ (get r "units") (get s "units"))) + (when (not (list_contains ps (get s "product"))) (append ps (get s "product"))) + (map_set r "products" (sort ps)) + (map_set out (get s "region") r)))) + (write_json "out/result.json" out)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x07.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x07.howl new file mode 100644 index 00000000..41a1edb3 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x07.howl @@ -0,0 +1,4 @@ +(do + (let (ren (read_json "in/rename.json")) + (write_json "out/result.json" (map_list r (read_json "in/rows.json") + (let (o (dict)) (do (for k (map_keys r) (map_set o (nz (get ren k) k) (get r k))) o)))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x08.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x08.howl new file mode 100644 index 00000000..f195d5ba --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x08.howl @@ -0,0 +1 @@ +(do (write_file "out/result.txt" (encode_json (read_json "in/doc.json")))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x09.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x09.howl new file mode 100644 index 00000000..2f904c55 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x09.howl @@ -0,0 +1,6 @@ +(do + (let (o (read_json "in/old.json") n (read_json "in/new.json")) + (write_json "out/result.json" (rec + added (filter k (map_keys n) (not (map_has o k))) + removed (filter k (map_keys o) (not (map_has n k))) + changed (filter k (map_keys o) (and (map_has n k) (!= (encode_json (get o k)) (encode_json (get n k))))))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x10.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x10.howl new file mode 100644 index 00000000..26016634 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x10.howl @@ -0,0 +1,5 @@ +(do + (write_file "out/result.txt" + (str_join + (map_list t (filter t (read_json "in/tickets.json") (list_contains (nz (get t "meta" "tags") (list)) "urgent")) (get t "id")) + "\n"))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x11.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x11.howl new file mode 100644 index 00000000..4b2658a3 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x11.howl @@ -0,0 +1,7 @@ +(do + (write_file "out/result.txt" + (str_join + (map_list a (read_json "in/accounts.json") + (let (p (get a "user" "profile")) + (str (get a "login") ": " (if (is_nil p) "unknown" (nz (get p "city") "unknown"))))) + "\n"))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x12.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x12.howl new file mode 100644 index 00000000..c89cf0c9 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x12.howl @@ -0,0 +1,5 @@ +(do + (let (out (dict)) + (do + (for p (read_json "in/people.json") (map_set out (str (get p "last") ", " (get p "first")) (get p "age"))) + (write_json "out/result.json" out)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x13.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x13.howl new file mode 100644 index 00000000..9a09866f --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x13.howl @@ -0,0 +1,7 @@ +(do + (let (out (dict)) + (do + (for l (lines (read_file "in/requests.jsonl")) + (let (r (parse_json Any l) lv (get r "level") ms (get r "ms")) + (map_set out lv (max (list ms (nz (get out lv) ms)))))) + (write_json "out/result.json" out)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x14.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x14.howl new file mode 100644 index 00000000..25d41689 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x14.howl @@ -0,0 +1,6 @@ +(do + (let (p (read_json "in/prices.json") out (dict)) + (do + (for k (map_keys p) + (let (v (get p k)) (map_set out k (str "$" (// v 100) "." (pad_left (% v 100) 2 "0"))))) + (write_json "out/result.json" out)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x15.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x15.howl new file mode 100644 index 00000000..ce371176 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x15.howl @@ -0,0 +1,9 @@ +(do + (defun dep ((v any)) int + (let (e (encode_json v) m 0) + (if (or (starts_with e "{") (starts_with e "[")) + (do + (for x (if (starts_with e "{") (map_list k (map_keys v) (get v k)) v) (set m (max (list m (call dep x))))) + (return (+ m 1))) + (return 0)))) + (write_file "out/result.txt" (str (call dep (read_json "in/tree.json"))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x16.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x16.howl new file mode 100644 index 00000000..7e5c50da --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x16.howl @@ -0,0 +1,3 @@ +(do + (let (e (read_json "in/env.json")) + (write_json "out/result.json" (map_list k (sort (map_keys e)) (rec key k value (get e k)))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x17.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x17.howl new file mode 100644 index 00000000..88786624 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x17.howl @@ -0,0 +1,10 @@ +(do + (let (doc (read_json "in/doc.json")) + (do + (for op (read_json "in/ops.json") + (let (ps (str_split (get op "path") ".") n (list_len ps) node doc k (list_get ps (- n 1))) + (do + (for p (list_slice ps 0 (- n 1)) + (do (when (is_nil (get node p)) (map_set node p (dict))) (set node (get node p)))) + (if (= (get op "op") "set") (map_set node k (get op "value")) (map_delete node k))))) + (write_json "out/result.json" doc)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x18.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x18.howl new file mode 100644 index 00000000..ef8b21a4 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x18.howl @@ -0,0 +1,6 @@ +(do + (let (out (list) cur "") + (do + (for l (lines (read_file "in/script.txt")) + (if (ends_with l "\\") (set cur (str cur (str_slice l 0 (- (str_len l) 1)))) (do (append out (str cur l)) (set cur "")))) + (write_file "out/result.txt" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x19.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x19.howl new file mode 100644 index 00000000..fdcc5600 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x19.howl @@ -0,0 +1,6 @@ +(do + (let (out (list) in false) + (do + (for l (lines (read_file "in/app.ini")) + (if (starts_with l "[") (set in (= l "[net]")) (when (and in (!= l "")) (append out l)))) + (write_file "out/result.txt" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x20.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x20.howl new file mode 100644 index 00000000..818e95c9 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x20.howl @@ -0,0 +1,8 @@ +(do + (let (out (list) w 0) + (do + (for l (str_split (str (read_text "in/doc.txt") "\n") "\n") + (if (= (str_trim l) "") + (when (> w 0) (do (append out (str (+ (list_len out) 1) ": " w)) (set w 0))) + (set w (+ w (list_len (filter x (str_split l " ") (!= x ""))))))) + (write_file "out/result.txt" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x21.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x21.howl new file mode 100644 index 00000000..52ddfb95 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x21.howl @@ -0,0 +1,4 @@ +(do + (let (rows (map_list l (lines (read_file "in/grid.csv")) (str_split l ","))) + (write_file "out/result.csv" + (str_join (map_list j (range (list_len (list_get rows 0))) (str_join (map_list r rows (list_get r j)) ",")) "\n")))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x22.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x22.howl new file mode 100644 index 00000000..d1f35298 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x22.howl @@ -0,0 +1,9 @@ +(do + (let (ls (lines (read_file "in/sites.csv")) prev (str_split (list_get ls 1) ",") out (list (list_get ls 0))) + (do + (for l (list_slice ls 1) + (let (c (str_split l ",") i 0) + (do + (set prev (map_list x c (do (inc i) (if (= x "") (list_get prev (- i 1)) x)))) + (append out (str_join prev ","))))) + (write_file "out/result.csv" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x23.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x23.howl new file mode 100644 index 00000000..2fa19fb9 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x23.howl @@ -0,0 +1,6 @@ +(do + (let (d (dict) out (list "id,name,dept")) + (do + (for l (list_slice (lines (read_file "in/dept.csv")) 1) (let ((i n) (str_split l ",")) (map_set d i n))) + (for l (list_slice (lines (read_file "in/emp.csv")) 1) (append out (str l "," (nz (get d (list_get (str_split l ",") 0)) "")))) + (write_file "out/result.csv" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x24.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x24.howl new file mode 100644 index 00000000..e38809ea --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x24.howl @@ -0,0 +1,10 @@ +(do + (let (g 0 m 1) + (do + (for s (str_split (read_text "in/vals.txt") ",") + (let (n (to_int s) a g b n) + (do + (while (!= b 0) (let (t (% a b)) (do (set a b) (set b t)))) + (set g a) + (let (x m y n) (do (while (!= y 0) (let (t (% x y)) (do (set x y) (set y t)))) (set m (* (// m x) n))))))) + (write_json "out/result.json" (rec gcd g lcm m))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x25.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x25.howl new file mode 100644 index 00000000..552e00e5 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x25.howl @@ -0,0 +1,6 @@ +(do + (write_file "out/result.txt" + (str_join + (map_list n (lines (read_file "in/nums.txt")) + (let (d n) (do (while (> (str_len d) 1) (set d (str (sum (map_list c (str_split d "") (to_int c)))))) (str n " -> " d)))) + "\n"))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x26.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x26.howl new file mode 100644 index 00000000..e51beefd --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x26.howl @@ -0,0 +1,4 @@ +(do + (let (rows (map_list l (lines (read_file "in/votes.txt")) (str_split l " ")) t (sum (map_list r rows (to_int (list_get r 1))))) + (write_file "out/result.txt" + (str_join (map_list r rows (str (list_get r 0) " " (// (* (to_int (list_get r 1)) 100) t) "%")) "\n")))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x27.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x27.howl new file mode 100644 index 00000000..f5e6ebbe --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x27.howl @@ -0,0 +1,6 @@ +(do + (defun flat ((v any)) any + (if (starts_with (encode_json v) "[") + (let (o (list)) (do (for e v (for x (call flat e) (append o x))) (return o))) + (return (list v)))) + (write_json "out/result.json" (call flat (read_json "in/deep.json")))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x28.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x28.howl new file mode 100644 index 00000000..5dde738c --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x28.howl @@ -0,0 +1,6 @@ +(do + (let (a (read_json "in/a.json") b (read_json "in/b.json") oa (list) ob (list)) + (do + (for x a (when (not (or (list_contains b x) (list_contains oa x))) (append oa x))) + (for x b (when (not (or (list_contains a x) (list_contains ob x))) (append ob x))) + (write_json "out/result.json" (rec only_a oa only_b ob))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x29.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x29.howl new file mode 100644 index 00000000..9dd1721b --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x29.howl @@ -0,0 +1,3 @@ +(do + (let (v (read_json "in/visits.json") i 0) + (write_json "out/result.json" (filter x v (do (inc i) (not (list_contains (list_slice v i) x))))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x30.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x30.howl new file mode 100644 index 00000000..ce94d28f --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x30.howl @@ -0,0 +1,6 @@ +(do + (write_file "out/result.txt" + (str_join + (map_list l (lines (read_file "in/pairs.txt")) + (let ((a b) (str_split l " ")) (if (= (str_join (sort (str_split a "")) "") (str_join (sort (str_split b "")) "")) "yes" "no"))) + "\n"))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x31.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x31.howl new file mode 100644 index 00000000..82ab5624 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x31.howl @@ -0,0 +1,6 @@ +(do + (let (out (list)) + (do + (for it (str_split (read_text "in/ranges.txt") ",") + (let (p (str_split it "-")) (for i (range (to_int (list_get p 0)) (+ (to_int (list_get p (- (list_len p) 1))) 1)) (append out i)))) + (write_file "out/result.txt" (str_join (map_list i out (str i)) ","))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x32.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x32.howl new file mode 100644 index 00000000..0da4d108 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x32.howl @@ -0,0 +1,5 @@ +(do + (let (ws (lines (read_file "in/words.txt")) p (list_get ws 0)) + (do + (for w ws (while (not (starts_with w p)) (set p (str_slice p 0 (- (str_len p) 1))))) + (write_file "out/result.txt" p)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x33.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x33.howl new file mode 100644 index 00000000..1f6fa783 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x33.howl @@ -0,0 +1,9 @@ +(do + (let (as (list) bs (list)) + (do + (for s (sort_by s (map_list l (lines (read_file "in/spans.txt")) (map_list x (str_split l "-") (to_int x))) (list_get s 0)) + (let (n (list_len bs) (a b) s) + (if (and (> n 0) (<= a (list_get bs (- n 1)))) + (let (e (max (list b (list_get bs (- n 1))))) (do (set bs (list_slice bs 0 (- n 1))) (append bs e))) + (do (append as a) (append bs b))))) + (write_file "out/result.txt" (str_join (map_list i (range (list_len as)) (str (list_get as i) "-" (list_get bs i))) "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x34.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x34.howl new file mode 100644 index 00000000..00b347dd --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x34.howl @@ -0,0 +1,10 @@ +(do + (let (ml (list 31 28 31 30 31 30 31 31 30 31 30 31)) + (write_file "out/result.txt" + (str_join + (map_list l (lines (read_file "in/days.txt")) + (let (d (to_int l) m 0) + (do + (while (> d (list_get ml m)) (do (set d (- d (list_get ml m))) (inc m))) + (str (pad_left (+ m 1) 2 "0") "-" (pad_left d 2 "0"))))) + "\n")))) diff --git a/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x35.howl b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x35.howl new file mode 100644 index 00000000..7befdb71 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec/solutions/howl/x35.howl @@ -0,0 +1,7 @@ +(do + (let (deps (read_json "in/deps.json") ks (sort (map_keys deps)) out (list)) + (do + (while (< (list_len out) (list_len ks)) + (for t ks + (when (and (not (list_contains out t)) (all d (get deps t) (list_contains out d))) (do (append out t) (break))))) + (write_json "out/result.json" out)))) diff --git a/experiments/token-efficiency-heldout-v3/postrec_test.go b/experiments/token-efficiency-heldout-v3/postrec_test.go new file mode 100644 index 00000000..ed2b1a00 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/postrec_test.go @@ -0,0 +1,71 @@ +package heldoutv3 + +import ( + "crypto/sha256" + "fmt" + "os" + "path/filepath" + "testing" +) + +// The post-records HowlFrame set (postrec/) is a separate results set: +// every task x language is verified with a current hash, the Python/JS/Go +// references are the baseline files, and the results are pinned below. +func TestHeldoutV3PostRecVerificationCoverage(t *testing.T) { + var manifest struct { + Tasks []struct { + ID string `json:"id"` + } `json:"tasks"` + } + readJSON(t, "tasks/manifest.json", &manifest) + var results []struct { + Task, Lang, SHA256 string + Pass bool + } + readJSON(t, "postrec/results/verification.json", &results) + seen := map[string]bool{} + for _, r := range results { + key := r.Task + "/" + r.Lang + if seen[key] { + t.Errorf("duplicate %s", key) + } + seen[key] = true + if !r.Pass { + t.Errorf("%s did not pass", key) + } + path := filepath.Join("solutions", r.Lang, r.Task+"."+exts[r.Lang]) + if r.Lang == "howl" { + path = filepath.Join("postrec", "solutions", "howl", r.Task+".howl") + } + src, err := os.ReadFile(path) + if err != nil { + t.Errorf("%s: %v", key, err) + continue + } + if got := fmt.Sprintf("%x", sha256.Sum256(src)); got != r.SHA256 { + t.Errorf("%s: stale hash", key) + } + } + if want := len(manifest.Tasks) * len(exts); len(results) != want { + t.Errorf("%d results, want %d (postrec has no gaps)", len(results), want) + } +} + +func TestHeldoutV3PostRecResultsFrozen(t *testing.T) { + frozen := map[string]string{ + "postrec/results/compare.md": "5f817fa42130ccabe0d71c00dc232c09dd83dfd104039ea0d3d595c797fad0e1", + "postrec/results/summary.json": "fa2d3ea8db027043b6bfd3ab1cb08c8f21070be9c091385094d37a3c91d11273", + "postrec/results/tokens.csv": "b11929b5ef4b19bf7a0fb6b2b9790d6e0b95d3aaf8f642a812df52dfc8dfc108", + "postrec/results/tokens.json": "6b5f82a76e65408281b2ebc6b2f4acbcb23bbfede37c646fac3d7d52ad334c6d", + "postrec/results/verification.json": "01de0278cd977f3cff115ac958a26104ceb3cf8c24cf9521aa9d5b04c545156c", + } + for path, want := range frozen { + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if got := fmt.Sprintf("%x", sha256.Sum256(data)); got != want { + t.Errorf("%s changed after post-records freeze: %s", path, got) + } + } +} diff --git a/experiments/token-efficiency-heldout-v3/requirements.txt b/experiments/token-efficiency-heldout-v3/requirements.txt new file mode 100644 index 00000000..135d4f58 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/requirements.txt @@ -0,0 +1 @@ +tiktoken==0.8.0 diff --git a/experiments/token-efficiency-heldout-v3/results/summary.json b/experiments/token-efficiency-heldout-v3/results/summary.json new file mode 100644 index 00000000..028b761d --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/results/summary.json @@ -0,0 +1,466 @@ +{ + "cl100k_base": { + "tokens": { + "languages": { + "howl": { + "median_all": 105, + "median_howl_solved": 105, + "mean_howl_solved": 108.08571428571429 + }, + "python": { + "median_all": 77, + "median_howl_solved": 77, + "mean_howl_solved": 81.14285714285714 + }, + "js": { + "median_all": 94, + "median_howl_solved": 94, + "mean_howl_solved": 94.4 + }, + "go": { + "median_all": 221, + "median_howl_solved": 221, + "mean_howl_solved": 231.28571428571428 + } + }, + "howl_vs": { + "python": { + "median_per_task_ratio": 1.2816901408450705, + "ratio_of_medians": 1.3636363636363635, + "wins": 3, + "ties": 0, + "losses": 32, + "gaps": 0 + }, + "js": { + "median_per_task_ratio": 1.1030927835051547, + "ratio_of_medians": 1.1170212765957446, + "wins": 13, + "ties": 0, + "losses": 22, + "gaps": 0 + }, + "go": { + "median_per_task_ratio": 0.45977011494252873, + "ratio_of_medians": 0.4751131221719457, + "wins": 35, + "ties": 0, + "losses": 0, + "gaps": 0 + } + }, + "categories": { + "lists": { + "n": 3, + "howl_solved": 3, + "mean_howl": 92.33333333333333, + "mean_python": 78, + "mean_js": 70.33333333333333, + "mean_go": 225.66666666666666, + "median_ratio_vs_python": 1.2465753424657535 + }, + "logic": { + "n": 3, + "howl_solved": 3, + "mean_howl": 147.66666666666666, + "mean_python": 87.66666666666667, + "mean_js": 118.66666666666667, + "mean_go": 272.3333333333333, + "median_ratio_vs_python": 1.7657657657657657 + }, + "numbers": { + "n": 3, + "howl_solved": 3, + "mean_howl": 115.66666666666667, + "mean_python": 82, + "mean_js": 101.33333333333333, + "mean_go": 227.66666666666666, + "median_ratio_vs_python": 1.2597402597402598 + }, + "records": { + "n": 17, + "howl_solved": 17, + "mean_howl": 107.52941176470588, + "mean_python": 86.41176470588235, + "mean_js": 97.58823529411765, + "mean_go": 246.35294117647058, + "median_ratio_vs_python": 1.1948051948051948 + }, + "strings": { + "n": 3, + "howl_solved": 3, + "mean_howl": 86.66666666666667, + "mean_python": 60.333333333333336, + "mean_js": 83, + "mean_go": 188, + "median_ratio_vs_python": 1.4583333333333333 + }, + "table": { + "n": 3, + "howl_solved": 3, + "mean_howl": 110.66666666666667, + "mean_python": 77.33333333333333, + "mean_js": 92.66666666666667, + "mean_go": 204, + "median_ratio_vs_python": 1.4111111111111112 + }, + "text": { + "n": 3, + "howl_solved": 3, + "mean_howl": 98.66666666666667, + "mean_python": 71.66666666666667, + "mean_js": 82.33333333333333, + "mean_go": 184.66666666666666, + "median_ratio_vs_python": 1.2816901408450705 + } + } + }, + "tokens_nocomments": { + "languages": { + "howl": { + "median_all": 105, + "median_howl_solved": 105, + "mean_howl_solved": 108.08571428571429 + }, + "python": { + "median_all": 77, + "median_howl_solved": 77, + "mean_howl_solved": 81.14285714285714 + }, + "js": { + "median_all": 94, + "median_howl_solved": 94, + "mean_howl_solved": 94.4 + }, + "go": { + "median_all": 215, + "median_howl_solved": 215, + "mean_howl_solved": 225.02857142857144 + } + }, + "howl_vs": { + "python": { + "median_per_task_ratio": 1.2816901408450705, + "ratio_of_medians": 1.3636363636363635, + "wins": 3, + "ties": 0, + "losses": 32, + "gaps": 0 + }, + "js": { + "median_per_task_ratio": 1.1030927835051547, + "ratio_of_medians": 1.1170212765957446, + "wins": 13, + "ties": 0, + "losses": 22, + "gaps": 0 + }, + "go": { + "median_per_task_ratio": 0.47619047619047616, + "ratio_of_medians": 0.4883720930232558, + "wins": 35, + "ties": 0, + "losses": 0, + "gaps": 0 + } + }, + "categories": { + "lists": { + "n": 3, + "howl_solved": 3, + "mean_howl": 92.33333333333333, + "mean_python": 78, + "mean_js": 70.33333333333333, + "mean_go": 219.66666666666666, + "median_ratio_vs_python": 1.2465753424657535 + }, + "logic": { + "n": 3, + "howl_solved": 3, + "mean_howl": 147.66666666666666, + "mean_python": 87.66666666666667, + "mean_js": 118.66666666666667, + "mean_go": 266.3333333333333, + "median_ratio_vs_python": 1.7657657657657657 + }, + "numbers": { + "n": 3, + "howl_solved": 3, + "mean_howl": 115.66666666666667, + "mean_python": 82, + "mean_js": 101.33333333333333, + "mean_go": 221.66666666666666, + "median_ratio_vs_python": 1.2597402597402598 + }, + "records": { + "n": 17, + "howl_solved": 17, + "mean_howl": 107.52941176470588, + "mean_python": 86.41176470588235, + "mean_js": 97.58823529411765, + "mean_go": 239.8235294117647, + "median_ratio_vs_python": 1.1948051948051948 + }, + "strings": { + "n": 3, + "howl_solved": 3, + "mean_howl": 86.66666666666667, + "mean_python": 60.333333333333336, + "mean_js": 83, + "mean_go": 182, + "median_ratio_vs_python": 1.4583333333333333 + }, + "table": { + "n": 3, + "howl_solved": 3, + "mean_howl": 110.66666666666667, + "mean_python": 77.33333333333333, + "mean_js": 92.66666666666667, + "mean_go": 198, + "median_ratio_vs_python": 1.4111111111111112 + }, + "text": { + "n": 3, + "howl_solved": 3, + "mean_howl": 98.66666666666667, + "mean_python": 71.66666666666667, + "mean_js": 82.33333333333333, + "mean_go": 178.66666666666666, + "median_ratio_vs_python": 1.2816901408450705 + } + } + } + }, + "o200k_base": { + "tokens": { + "languages": { + "howl": { + "median_all": 105, + "median_howl_solved": 105, + "mean_howl_solved": 108.28571428571429 + }, + "python": { + "median_all": 77, + "median_howl_solved": 77, + "mean_howl_solved": 81.28571428571429 + }, + "js": { + "median_all": 98, + "median_howl_solved": 98, + "mean_howl_solved": 99.05714285714286 + }, + "go": { + "median_all": 221, + "median_howl_solved": 221, + "mean_howl_solved": 232.65714285714284 + } + }, + "howl_vs": { + "python": { + "median_per_task_ratio": 1.2816901408450705, + "ratio_of_medians": 1.3636363636363635, + "wins": 3, + "ties": 0, + "losses": 32, + "gaps": 0 + }, + "js": { + "median_per_task_ratio": 1.0579710144927537, + "ratio_of_medians": 1.0714285714285714, + "wins": 16, + "ties": 0, + "losses": 19, + "gaps": 0 + }, + "go": { + "median_per_task_ratio": 0.45454545454545453, + "ratio_of_medians": 0.4751131221719457, + "wins": 35, + "ties": 0, + "losses": 0, + "gaps": 0 + } + }, + "categories": { + "lists": { + "n": 3, + "howl_solved": 3, + "mean_howl": 92.33333333333333, + "mean_python": 77.66666666666667, + "mean_js": 75.66666666666667, + "mean_go": 226.33333333333334, + "median_ratio_vs_python": 1.2638888888888888 + }, + "logic": { + "n": 3, + "howl_solved": 3, + "mean_howl": 148, + "mean_python": 87.66666666666667, + "mean_js": 122.33333333333333, + "mean_go": 274, + "median_ratio_vs_python": 1.7657657657657657 + }, + "numbers": { + "n": 3, + "howl_solved": 3, + "mean_howl": 116.33333333333333, + "mean_python": 82.33333333333333, + "mean_js": 105.66666666666667, + "mean_go": 228.66666666666666, + "median_ratio_vs_python": 1.2727272727272727 + }, + "records": { + "n": 17, + "howl_solved": 17, + "mean_howl": 107.70588235294117, + "mean_python": 86.52941176470588, + "mean_js": 102.47058823529412, + "mean_go": 247.8235294117647, + "median_ratio_vs_python": 1.1948051948051948 + }, + "strings": { + "n": 3, + "howl_solved": 3, + "mean_howl": 86.66666666666667, + "mean_python": 60.333333333333336, + "mean_js": 87.33333333333333, + "mean_go": 189.33333333333334, + "median_ratio_vs_python": 1.4583333333333333 + }, + "table": { + "n": 3, + "howl_solved": 3, + "mean_howl": 111, + "mean_python": 78.66666666666667, + "mean_js": 97, + "mean_go": 205.33333333333334, + "median_ratio_vs_python": 1.3804347826086956 + }, + "text": { + "n": 3, + "howl_solved": 3, + "mean_howl": 98.66666666666667, + "mean_python": 71.33333333333333, + "mean_js": 87, + "mean_go": 186.33333333333334, + "median_ratio_vs_python": 1.2816901408450705 + } + } + }, + "tokens_nocomments": { + "languages": { + "howl": { + "median_all": 105, + "median_howl_solved": 105, + "mean_howl_solved": 108.28571428571429 + }, + "python": { + "median_all": 77, + "median_howl_solved": 77, + "mean_howl_solved": 81.28571428571429 + }, + "js": { + "median_all": 98, + "median_howl_solved": 98, + "mean_howl_solved": 99.05714285714286 + }, + "go": { + "median_all": 215, + "median_howl_solved": 215, + "mean_howl_solved": 226.4 + } + }, + "howl_vs": { + "python": { + "median_per_task_ratio": 1.2816901408450705, + "ratio_of_medians": 1.3636363636363635, + "wins": 3, + "ties": 0, + "losses": 32, + "gaps": 0 + }, + "js": { + "median_per_task_ratio": 1.0579710144927537, + "ratio_of_medians": 1.0714285714285714, + "wins": 16, + "ties": 0, + "losses": 19, + "gaps": 0 + }, + "go": { + "median_per_task_ratio": 0.47058823529411764, + "ratio_of_medians": 0.4883720930232558, + "wins": 35, + "ties": 0, + "losses": 0, + "gaps": 0 + } + }, + "categories": { + "lists": { + "n": 3, + "howl_solved": 3, + "mean_howl": 92.33333333333333, + "mean_python": 77.66666666666667, + "mean_js": 75.66666666666667, + "mean_go": 220.33333333333334, + "median_ratio_vs_python": 1.2638888888888888 + }, + "logic": { + "n": 3, + "howl_solved": 3, + "mean_howl": 148, + "mean_python": 87.66666666666667, + "mean_js": 122.33333333333333, + "mean_go": 268, + "median_ratio_vs_python": 1.7657657657657657 + }, + "numbers": { + "n": 3, + "howl_solved": 3, + "mean_howl": 116.33333333333333, + "mean_python": 82.33333333333333, + "mean_js": 105.66666666666667, + "mean_go": 222.66666666666666, + "median_ratio_vs_python": 1.2727272727272727 + }, + "records": { + "n": 17, + "howl_solved": 17, + "mean_howl": 107.70588235294117, + "mean_python": 86.52941176470588, + "mean_js": 102.47058823529412, + "mean_go": 241.2941176470588, + "median_ratio_vs_python": 1.1948051948051948 + }, + "strings": { + "n": 3, + "howl_solved": 3, + "mean_howl": 86.66666666666667, + "mean_python": 60.333333333333336, + "mean_js": 87.33333333333333, + "mean_go": 183.33333333333334, + "median_ratio_vs_python": 1.4583333333333333 + }, + "table": { + "n": 3, + "howl_solved": 3, + "mean_howl": 111, + "mean_python": 78.66666666666667, + "mean_js": 97, + "mean_go": 199.33333333333334, + "median_ratio_vs_python": 1.3804347826086956 + }, + "text": { + "n": 3, + "howl_solved": 3, + "mean_howl": 98.66666666666667, + "mean_python": 71.33333333333333, + "mean_js": 87, + "mean_go": 180.33333333333334, + "median_ratio_vs_python": 1.2816901408450705 + } + } + } + } +} diff --git a/experiments/token-efficiency-heldout-v3/results/tokens.csv b/experiments/token-efficiency-heldout-v3/results/tokens.csv new file mode 100644 index 00000000..1a848fe5 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/results/tokens.csv @@ -0,0 +1,281 @@ +task,category,lang,encoding,tokens,tokens_nocomments,bytes,pass +x01,records,howl,cl100k_base,127,127,400,True +x01,records,howl,o200k_base,129,129,400,True +x01,records,python,cl100k_base,110,110,402,True +x01,records,python,o200k_base,112,112,402,True +x01,records,js,cl100k_base,139,139,471,True +x01,records,js,o200k_base,146,146,471,True +x01,records,go,cl100k_base,314,308,1022,True +x01,records,go,o200k_base,316,310,1022,True +x02,records,howl,cl100k_base,131,131,401,True +x02,records,howl,o200k_base,131,131,401,True +x02,records,python,cl100k_base,106,106,379,True +x02,records,python,o200k_base,106,106,379,True +x02,records,js,cl100k_base,132,132,441,True +x02,records,js,o200k_base,139,139,441,True +x02,records,go,cl100k_base,268,262,812,True +x02,records,go,o200k_base,269,263,812,True +x03,records,howl,cl100k_base,127,127,383,True +x03,records,howl,o200k_base,127,127,383,True +x03,records,python,cl100k_base,102,102,358,True +x03,records,python,o200k_base,101,101,358,True +x03,records,js,cl100k_base,137,137,453,True +x03,records,js,o200k_base,142,142,453,True +x03,records,go,cl100k_base,253,247,774,True +x03,records,go,o200k_base,254,248,774,True +x04,records,howl,cl100k_base,140,140,416,True +x04,records,howl,o200k_base,140,140,416,True +x04,records,python,cl100k_base,86,86,306,True +x04,records,python,o200k_base,86,86,306,True +x04,records,js,cl100k_base,101,101,353,True +x04,records,js,o200k_base,105,105,353,True +x04,records,go,cl100k_base,266,260,841,True +x04,records,go,o200k_base,267,261,841,True +x05,records,howl,cl100k_base,125,125,370,True +x05,records,howl,o200k_base,125,125,370,True +x05,records,python,cl100k_base,97,97,331,True +x05,records,python,o200k_base,97,97,331,True +x05,records,js,cl100k_base,93,93,315,True +x05,records,js,o200k_base,97,97,315,True +x05,records,go,cl100k_base,248,242,804,True +x05,records,go,o200k_base,249,243,804,True +x06,records,howl,cl100k_base,163,163,515,True +x06,records,howl,o200k_base,163,163,515,True +x06,records,python,cl100k_base,111,111,400,True +x06,records,python,o200k_base,111,111,400,True +x06,records,js,cl100k_base,107,107,395,True +x06,records,js,o200k_base,111,111,395,True +x06,records,go,cl100k_base,312,306,1026,True +x06,records,go,o200k_base,314,308,1026,True +x07,records,howl,cl100k_base,89,89,267,True +x07,records,howl,o200k_base,89,89,267,True +x07,records,python,cl100k_base,79,79,275,True +x07,records,python,o200k_base,79,79,275,True +x07,records,js,cl100k_base,90,90,319,True +x07,records,js,o200k_base,96,96,319,True +x07,records,go,cl100k_base,249,243,780,True +x07,records,go,o200k_base,250,244,780,True +x08,records,howl,cl100k_base,25,25,92,True +x08,records,howl,o200k_base,25,25,92,True +x08,records,python,cl100k_base,46,46,185,True +x08,records,python,o200k_base,46,46,185,True +x08,records,js,cl100k_base,106,106,361,True +x08,records,js,o200k_base,111,111,361,True +x08,records,go,cl100k_base,149,134,502,True +x08,records,go,o200k_base,150,135,502,True +x09,records,howl,cl100k_base,129,129,400,True +x09,records,howl,o200k_base,129,129,400,True +x09,records,python,cl100k_base,113,113,397,True +x09,records,python,o200k_base,113,113,397,True +x09,records,js,cl100k_base,134,134,478,True +x09,records,js,o200k_base,140,140,478,True +x09,records,go,cl100k_base,313,307,988,True +x09,records,go,o200k_base,316,310,988,True +x10,records,howl,cl100k_base,71,71,221,True +x10,records,howl,o200k_base,71,71,221,True +x10,records,python,cl100k_base,66,66,227,True +x10,records,python,o200k_base,66,66,227,True +x10,records,js,cl100k_base,63,63,225,True +x10,records,js,o200k_base,67,67,225,True +x10,records,go,cl100k_base,201,195,685,True +x10,records,go,o200k_base,202,196,685,True +x11,records,howl,cl100k_base,89,89,270,True +x11,records,howl,o200k_base,89,89,270,True +x11,records,python,cl100k_base,81,81,296,True +x11,records,python,o200k_base,81,81,296,True +x11,records,js,cl100k_base,59,59,218,True +x11,records,js,o200k_base,63,63,218,True +x11,records,go,cl100k_base,221,215,755,True +x11,records,go,o200k_base,222,216,755,True +x12,records,howl,cl100k_base,72,72,223,True +x12,records,howl,o200k_base,73,73,223,True +x12,records,python,cl100k_base,66,66,226,True +x12,records,python,o200k_base,67,67,226,True +x12,records,js,cl100k_base,64,64,226,True +x12,records,js,o200k_base,69,69,226,True +x12,records,go,cl100k_base,200,194,666,True +x12,records,go,o200k_base,203,197,666,True +x13,records,howl,cl100k_base,92,92,274,True +x13,records,howl,o200k_base,92,92,274,True +x13,records,python,cl100k_base,77,77,270,True +x13,records,python,o200k_base,77,77,270,True +x13,records,js,cl100k_base,84,84,290,True +x13,records,js,o200k_base,88,88,290,True +x13,records,go,cl100k_base,229,223,739,True +x13,records,go,o200k_base,230,224,739,True +x14,records,howl,cl100k_base,91,91,252,True +x14,records,howl,o200k_base,91,91,252,True +x14,records,python,cl100k_base,70,70,230,True +x14,records,python,o200k_base,70,70,230,True +x14,records,js,cl100k_base,89,89,287,True +x14,records,js,o200k_base,93,93,287,True +x14,records,go,cl100k_base,200,194,652,True +x14,records,go,o200k_base,201,195,652,True +x15,records,howl,cl100k_base,129,129,371,True +x15,records,howl,o200k_base,129,129,371,True +x15,records,python,cl100k_base,82,82,307,True +x15,records,python,o200k_base,82,82,307,True +x15,records,js,cl100k_base,71,71,244,True +x15,records,js,o200k_base,75,75,244,True +x15,records,go,cl100k_base,220,214,686,True +x15,records,go,o200k_base,221,215,686,True +x16,records,howl,cl100k_base,58,58,180,True +x16,records,howl,o200k_base,58,58,180,True +x16,records,python,cl100k_base,60,60,214,True +x16,records,python,o200k_base,60,60,214,True +x16,records,js,cl100k_base,68,68,242,True +x16,records,js,o200k_base,72,72,242,True +x16,records,go,cl100k_base,230,224,771,True +x16,records,go,o200k_base,233,227,771,True +x17,records,howl,cl100k_base,170,170,519,True +x17,records,howl,o200k_base,170,170,519,True +x17,records,python,cl100k_base,117,117,424,True +x17,records,python,o200k_base,117,117,424,True +x17,records,js,cl100k_base,122,122,440,True +x17,records,js,o200k_base,128,128,440,True +x17,records,go,cl100k_base,315,309,987,True +x17,records,go,o200k_base,316,310,987,True +x18,text,howl,cl100k_base,91,91,264,True +x18,text,howl,o200k_base,91,91,264,True +x18,text,python,cl100k_base,71,71,272,True +x18,text,python,o200k_base,71,71,272,True +x18,text,js,cl100k_base,87,87,299,True +x18,text,js,o200k_base,92,92,299,True +x18,text,go,cl100k_base,175,169,591,True +x18,text,go,o200k_base,176,170,591,True +x19,text,howl,cl100k_base,80,80,231,True +x19,text,howl,o200k_base,80,80,231,True +x19,text,python,cl100k_base,70,70,277,True +x19,text,python,o200k_base,70,70,277,True +x19,text,js,cl100k_base,81,81,294,True +x19,text,js,o200k_base,86,86,294,True +x19,text,go,cl100k_base,174,168,604,True +x19,text,go,o200k_base,176,170,604,True +x20,text,howl,cl100k_base,125,125,333,True +x20,text,howl,o200k_base,125,125,333,True +x20,text,python,cl100k_base,74,74,231,True +x20,text,python,o200k_base,73,73,231,True +x20,text,js,cl100k_base,79,79,240,True +x20,text,js,o200k_base,83,83,240,True +x20,text,go,cl100k_base,205,199,694,True +x20,text,go,o200k_base,207,201,694,True +x21,table,howl,cl100k_base,74,74,233,True +x21,table,howl,o200k_base,75,75,233,True +x21,table,python,cl100k_base,55,55,187,True +x21,table,python,o200k_base,55,55,187,True +x21,table,js,cl100k_base,77,77,251,True +x21,table,js,o200k_base,81,81,251,True +x21,table,go,cl100k_base,199,193,683,True +x21,table,go,o200k_base,200,194,683,True +x22,table,howl,cl100k_base,131,131,375,True +x22,table,howl,o200k_base,131,131,375,True +x22,table,python,cl100k_base,87,87,306,True +x22,table,python,o200k_base,89,89,306,True +x22,table,js,cl100k_base,95,95,323,True +x22,table,js,o200k_base,99,99,323,True +x22,table,go,cl100k_base,201,195,680,True +x22,table,go,o200k_base,203,197,680,True +x23,table,howl,cl100k_base,127,127,352,True +x23,table,howl,o200k_base,127,127,352,True +x23,table,python,cl100k_base,90,90,303,True +x23,table,python,o200k_base,92,92,303,True +x23,table,js,cl100k_base,106,106,344,True +x23,table,js,o200k_base,111,111,344,True +x23,table,go,cl100k_base,212,206,705,True +x23,table,go,o200k_base,213,207,705,True +x24,numbers,howl,cl100k_base,165,165,392,True +x24,numbers,howl,o200k_base,166,166,392,True +x24,numbers,python,cl100k_base,94,94,315,True +x24,numbers,python,o200k_base,95,95,315,True +x24,numbers,js,cl100k_base,105,105,319,True +x24,numbers,js,o200k_base,110,110,319,True +x24,numbers,go,cl100k_base,246,240,731,True +x24,numbers,go,o200k_base,247,241,731,True +x25,numbers,howl,cl100k_base,85,85,232,True +x25,numbers,howl,o200k_base,85,85,232,True +x25,numbers,python,cl100k_base,75,75,244,True +x25,numbers,python,o200k_base,75,75,244,True +x25,numbers,js,cl100k_base,96,96,289,True +x25,numbers,js,o200k_base,100,100,289,True +x25,numbers,go,cl100k_base,196,190,609,True +x25,numbers,go,o200k_base,197,191,609,True +x26,numbers,howl,cl100k_base,97,97,272,True +x26,numbers,howl,o200k_base,98,98,272,True +x26,numbers,python,cl100k_base,77,77,239,True +x26,numbers,python,o200k_base,77,77,239,True +x26,numbers,js,cl100k_base,103,103,327,True +x26,numbers,js,o200k_base,107,107,327,True +x26,numbers,go,cl100k_base,241,235,807,True +x26,numbers,go,o200k_base,242,236,807,True +x27,lists,howl,cl100k_base,91,91,278,True +x27,lists,howl,o200k_base,91,91,278,True +x27,lists,python,cl100k_base,73,73,256,True +x27,lists,python,o200k_base,72,72,256,True +x27,lists,js,cl100k_base,48,48,181,True +x27,lists,js,o200k_base,52,52,181,True +x27,lists,go,cl100k_base,221,215,695,True +x27,lists,go,o200k_base,221,215,695,True +x28,lists,howl,cl100k_base,126,126,382,True +x28,lists,howl,o200k_base,126,126,382,True +x28,lists,python,cl100k_base,95,95,331,True +x28,lists,python,o200k_base,95,95,331,True +x28,lists,js,cl100k_base,100,100,347,True +x28,lists,js,o200k_base,106,106,347,True +x28,lists,go,cl100k_base,253,247,808,True +x28,lists,go,o200k_base,254,248,808,True +x29,lists,howl,cl100k_base,60,60,180,True +x29,lists,howl,o200k_base,60,60,180,True +x29,lists,python,cl100k_base,66,66,215,True +x29,lists,python,o200k_base,66,66,215,True +x29,lists,js,cl100k_base,63,63,222,True +x29,lists,js,o200k_base,69,69,222,True +x29,lists,go,cl100k_base,203,197,640,True +x29,lists,go,o200k_base,204,198,640,True +x30,strings,howl,cl100k_base,82,82,240,True +x30,strings,howl,o200k_base,82,82,240,True +x30,strings,python,cl100k_base,68,68,233,True +x30,strings,python,o200k_base,68,68,233,True +x30,strings,js,cl100k_base,94,94,291,True +x30,strings,js,o200k_base,98,98,291,True +x30,strings,go,cl100k_base,213,207,687,True +x30,strings,go,o200k_base,214,208,687,True +x31,strings,howl,cl100k_base,105,105,296,True +x31,strings,howl,o200k_base,105,105,296,True +x31,strings,python,cl100k_base,72,72,243,True +x31,strings,python,o200k_base,72,72,243,True +x31,strings,js,cl100k_base,85,85,278,True +x31,strings,js,o200k_base,89,89,278,True +x31,strings,go,cl100k_base,212,206,659,True +x31,strings,go,o200k_base,213,207,659,True +x32,strings,howl,cl100k_base,73,73,200,True +x32,strings,howl,o200k_base,73,73,200,True +x32,strings,python,cl100k_base,41,41,160,True +x32,strings,python,o200k_base,41,41,160,True +x32,strings,js,cl100k_base,70,70,233,True +x32,strings,js,o200k_base,75,75,233,True +x32,strings,go,cl100k_base,139,133,448,True +x32,strings,go,o200k_base,141,135,448,True +x33,logic,howl,cl100k_base,196,196,530,True +x33,logic,howl,o200k_base,196,196,530,True +x33,logic,python,cl100k_base,111,111,339,True +x33,logic,python,o200k_base,111,111,339,True +x33,logic,js,cl100k_base,152,152,458,True +x33,logic,js,o200k_base,156,156,458,True +x33,logic,go,cl100k_base,338,332,989,True +x33,logic,go,o200k_base,340,334,989,True +x34,logic,howl,cl100k_base,140,140,348,True +x34,logic,howl,o200k_base,141,141,348,True +x34,logic,python,cl100k_base,73,73,248,True +x34,logic,python,o200k_base,73,73,248,True +x34,logic,js,cl100k_base,107,107,328,True +x34,logic,js,o200k_base,110,110,328,True +x34,logic,go,cl100k_base,196,190,625,True +x34,logic,go,o200k_base,197,191,625,True +x35,logic,howl,cl100k_base,107,107,339,True +x35,logic,howl,o200k_base,107,107,339,True +x35,logic,python,cl100k_base,79,79,284,True +x35,logic,python,o200k_base,79,79,284,True +x35,logic,js,cl100k_base,97,97,347,True +x35,logic,js,o200k_base,101,101,347,True +x35,logic,go,cl100k_base,283,277,922,True +x35,logic,go,o200k_base,285,279,922,True diff --git a/experiments/token-efficiency-heldout-v3/results/tokens.json b/experiments/token-efficiency-heldout-v3/results/tokens.json new file mode 100644 index 00000000..d33e4f61 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/results/tokens.json @@ -0,0 +1,2802 @@ +[ + { + "task": "x01", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 127, + "tokens_nocomments": 127, + "bytes": 400, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 129, + "tokens_nocomments": 129, + "bytes": 400, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 110, + "tokens_nocomments": 110, + "bytes": 402, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 112, + "tokens_nocomments": 112, + "bytes": 402, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 139, + "tokens_nocomments": 139, + "bytes": 471, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 146, + "tokens_nocomments": 146, + "bytes": 471, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 314, + "tokens_nocomments": 308, + "bytes": 1022, + "pass": true + }, + { + "task": "x01", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 316, + "tokens_nocomments": 310, + "bytes": 1022, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 131, + "tokens_nocomments": 131, + "bytes": 401, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 131, + "tokens_nocomments": 131, + "bytes": 401, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 379, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 379, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 132, + "tokens_nocomments": 132, + "bytes": 441, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 139, + "tokens_nocomments": 139, + "bytes": 441, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 268, + "tokens_nocomments": 262, + "bytes": 812, + "pass": true + }, + { + "task": "x02", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 269, + "tokens_nocomments": 263, + "bytes": 812, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 127, + "tokens_nocomments": 127, + "bytes": 383, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 127, + "tokens_nocomments": 127, + "bytes": 383, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 102, + "tokens_nocomments": 102, + "bytes": 358, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 101, + "tokens_nocomments": 101, + "bytes": 358, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 137, + "tokens_nocomments": 137, + "bytes": 453, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 142, + "tokens_nocomments": 142, + "bytes": 453, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 253, + "tokens_nocomments": 247, + "bytes": 774, + "pass": true + }, + { + "task": "x03", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 254, + "tokens_nocomments": 248, + "bytes": 774, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 140, + "tokens_nocomments": 140, + "bytes": 416, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 140, + "tokens_nocomments": 140, + "bytes": 416, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 86, + "tokens_nocomments": 86, + "bytes": 306, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 86, + "tokens_nocomments": 86, + "bytes": 306, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 101, + "tokens_nocomments": 101, + "bytes": 353, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 105, + "tokens_nocomments": 105, + "bytes": 353, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 266, + "tokens_nocomments": 260, + "bytes": 841, + "pass": true + }, + { + "task": "x04", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 267, + "tokens_nocomments": 261, + "bytes": 841, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 125, + "tokens_nocomments": 125, + "bytes": 370, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 125, + "tokens_nocomments": 125, + "bytes": 370, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 331, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 331, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 93, + "tokens_nocomments": 93, + "bytes": 315, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 315, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 248, + "tokens_nocomments": 242, + "bytes": 804, + "pass": true + }, + { + "task": "x05", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 249, + "tokens_nocomments": 243, + "bytes": 804, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 163, + "tokens_nocomments": 163, + "bytes": 515, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 163, + "tokens_nocomments": 163, + "bytes": 515, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 400, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 400, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 107, + "tokens_nocomments": 107, + "bytes": 395, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 395, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 312, + "tokens_nocomments": 306, + "bytes": 1026, + "pass": true + }, + { + "task": "x06", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 314, + "tokens_nocomments": 308, + "bytes": 1026, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 267, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 267, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 275, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 275, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 90, + "tokens_nocomments": 90, + "bytes": 319, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 96, + "tokens_nocomments": 96, + "bytes": 319, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 249, + "tokens_nocomments": 243, + "bytes": 780, + "pass": true + }, + { + "task": "x07", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 250, + "tokens_nocomments": 244, + "bytes": 780, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 25, + "tokens_nocomments": 25, + "bytes": 92, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 25, + "tokens_nocomments": 25, + "bytes": 92, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 46, + "tokens_nocomments": 46, + "bytes": 185, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 46, + "tokens_nocomments": 46, + "bytes": 185, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 361, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 361, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 149, + "tokens_nocomments": 134, + "bytes": 502, + "pass": true + }, + { + "task": "x08", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 150, + "tokens_nocomments": 135, + "bytes": 502, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 129, + "tokens_nocomments": 129, + "bytes": 400, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 129, + "tokens_nocomments": 129, + "bytes": 400, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 113, + "tokens_nocomments": 113, + "bytes": 397, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 113, + "tokens_nocomments": 113, + "bytes": 397, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 134, + "tokens_nocomments": 134, + "bytes": 478, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 140, + "tokens_nocomments": 140, + "bytes": 478, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 313, + "tokens_nocomments": 307, + "bytes": 988, + "pass": true + }, + { + "task": "x09", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 316, + "tokens_nocomments": 310, + "bytes": 988, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 71, + "tokens_nocomments": 71, + "bytes": 221, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 71, + "tokens_nocomments": 71, + "bytes": 221, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 227, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 227, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 63, + "tokens_nocomments": 63, + "bytes": 225, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 67, + "tokens_nocomments": 67, + "bytes": 225, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 201, + "tokens_nocomments": 195, + "bytes": 685, + "pass": true + }, + { + "task": "x10", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 202, + "tokens_nocomments": 196, + "bytes": 685, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 270, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 270, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 81, + "tokens_nocomments": 81, + "bytes": 296, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 81, + "tokens_nocomments": 81, + "bytes": 296, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 59, + "tokens_nocomments": 59, + "bytes": 218, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 63, + "tokens_nocomments": 63, + "bytes": 218, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 221, + "tokens_nocomments": 215, + "bytes": 755, + "pass": true + }, + { + "task": "x11", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 222, + "tokens_nocomments": 216, + "bytes": 755, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 223, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 223, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 226, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 67, + "tokens_nocomments": 67, + "bytes": 226, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 64, + "tokens_nocomments": 64, + "bytes": 226, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 69, + "tokens_nocomments": 69, + "bytes": 226, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 200, + "tokens_nocomments": 194, + "bytes": 666, + "pass": true + }, + { + "task": "x12", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 203, + "tokens_nocomments": 197, + "bytes": 666, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 92, + "tokens_nocomments": 92, + "bytes": 274, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 92, + "tokens_nocomments": 92, + "bytes": 274, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 270, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 270, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 84, + "tokens_nocomments": 84, + "bytes": 290, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 88, + "tokens_nocomments": 88, + "bytes": 290, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 229, + "tokens_nocomments": 223, + "bytes": 739, + "pass": true + }, + { + "task": "x13", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 230, + "tokens_nocomments": 224, + "bytes": 739, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 91, + "tokens_nocomments": 91, + "bytes": 252, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 91, + "tokens_nocomments": 91, + "bytes": 252, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 230, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 230, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 287, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 93, + "tokens_nocomments": 93, + "bytes": 287, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 200, + "tokens_nocomments": 194, + "bytes": 652, + "pass": true + }, + { + "task": "x14", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 201, + "tokens_nocomments": 195, + "bytes": 652, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 129, + "tokens_nocomments": 129, + "bytes": 371, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 129, + "tokens_nocomments": 129, + "bytes": 371, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 82, + "tokens_nocomments": 82, + "bytes": 307, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 82, + "tokens_nocomments": 82, + "bytes": 307, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 71, + "tokens_nocomments": 71, + "bytes": 244, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 244, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 220, + "tokens_nocomments": 214, + "bytes": 686, + "pass": true + }, + { + "task": "x15", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 221, + "tokens_nocomments": 215, + "bytes": 686, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 58, + "tokens_nocomments": 58, + "bytes": 180, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 58, + "tokens_nocomments": 58, + "bytes": 180, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 60, + "tokens_nocomments": 60, + "bytes": 214, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 60, + "tokens_nocomments": 60, + "bytes": 214, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 68, + "tokens_nocomments": 68, + "bytes": 242, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 242, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 230, + "tokens_nocomments": 224, + "bytes": 771, + "pass": true + }, + { + "task": "x16", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 233, + "tokens_nocomments": 227, + "bytes": 771, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 170, + "tokens_nocomments": 170, + "bytes": 519, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 170, + "tokens_nocomments": 170, + "bytes": 519, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 117, + "tokens_nocomments": 117, + "bytes": 424, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "python", + "encoding": "o200k_base", + "tokens": 117, + "tokens_nocomments": 117, + "bytes": 424, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 122, + "tokens_nocomments": 122, + "bytes": 440, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "js", + "encoding": "o200k_base", + "tokens": 128, + "tokens_nocomments": 128, + "bytes": 440, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 315, + "tokens_nocomments": 309, + "bytes": 987, + "pass": true + }, + { + "task": "x17", + "category": "records", + "lang": "go", + "encoding": "o200k_base", + "tokens": 316, + "tokens_nocomments": 310, + "bytes": 987, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 91, + "tokens_nocomments": 91, + "bytes": 264, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 91, + "tokens_nocomments": 91, + "bytes": 264, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 71, + "tokens_nocomments": 71, + "bytes": 272, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "python", + "encoding": "o200k_base", + "tokens": 71, + "tokens_nocomments": 71, + "bytes": 272, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 87, + "tokens_nocomments": 87, + "bytes": 299, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "js", + "encoding": "o200k_base", + "tokens": 92, + "tokens_nocomments": 92, + "bytes": 299, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 175, + "tokens_nocomments": 169, + "bytes": 591, + "pass": true + }, + { + "task": "x18", + "category": "text", + "lang": "go", + "encoding": "o200k_base", + "tokens": 176, + "tokens_nocomments": 170, + "bytes": 591, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 80, + "tokens_nocomments": 80, + "bytes": 231, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 80, + "tokens_nocomments": 80, + "bytes": 231, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 277, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "python", + "encoding": "o200k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 277, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 81, + "tokens_nocomments": 81, + "bytes": 294, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "js", + "encoding": "o200k_base", + "tokens": 86, + "tokens_nocomments": 86, + "bytes": 294, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 174, + "tokens_nocomments": 168, + "bytes": 604, + "pass": true + }, + { + "task": "x19", + "category": "text", + "lang": "go", + "encoding": "o200k_base", + "tokens": 176, + "tokens_nocomments": 170, + "bytes": 604, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 125, + "tokens_nocomments": 125, + "bytes": 333, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 125, + "tokens_nocomments": 125, + "bytes": 333, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 74, + "tokens_nocomments": 74, + "bytes": 231, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "python", + "encoding": "o200k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 231, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 240, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "js", + "encoding": "o200k_base", + "tokens": 83, + "tokens_nocomments": 83, + "bytes": 240, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 205, + "tokens_nocomments": 199, + "bytes": 694, + "pass": true + }, + { + "task": "x20", + "category": "text", + "lang": "go", + "encoding": "o200k_base", + "tokens": 207, + "tokens_nocomments": 201, + "bytes": 694, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 74, + "tokens_nocomments": 74, + "bytes": 233, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 233, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 55, + "tokens_nocomments": 55, + "bytes": 187, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "python", + "encoding": "o200k_base", + "tokens": 55, + "tokens_nocomments": 55, + "bytes": 187, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 251, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "js", + "encoding": "o200k_base", + "tokens": 81, + "tokens_nocomments": 81, + "bytes": 251, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 199, + "tokens_nocomments": 193, + "bytes": 683, + "pass": true + }, + { + "task": "x21", + "category": "table", + "lang": "go", + "encoding": "o200k_base", + "tokens": 200, + "tokens_nocomments": 194, + "bytes": 683, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 131, + "tokens_nocomments": 131, + "bytes": 375, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 131, + "tokens_nocomments": 131, + "bytes": 375, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 87, + "tokens_nocomments": 87, + "bytes": 306, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "python", + "encoding": "o200k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 306, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 95, + "tokens_nocomments": 95, + "bytes": 323, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "js", + "encoding": "o200k_base", + "tokens": 99, + "tokens_nocomments": 99, + "bytes": 323, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 201, + "tokens_nocomments": 195, + "bytes": 680, + "pass": true + }, + { + "task": "x22", + "category": "table", + "lang": "go", + "encoding": "o200k_base", + "tokens": 203, + "tokens_nocomments": 197, + "bytes": 680, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 127, + "tokens_nocomments": 127, + "bytes": 352, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 127, + "tokens_nocomments": 127, + "bytes": 352, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 90, + "tokens_nocomments": 90, + "bytes": 303, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "python", + "encoding": "o200k_base", + "tokens": 92, + "tokens_nocomments": 92, + "bytes": 303, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 344, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "js", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 344, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 212, + "tokens_nocomments": 206, + "bytes": 705, + "pass": true + }, + { + "task": "x23", + "category": "table", + "lang": "go", + "encoding": "o200k_base", + "tokens": 213, + "tokens_nocomments": 207, + "bytes": 705, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 165, + "tokens_nocomments": 165, + "bytes": 392, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 166, + "tokens_nocomments": 166, + "bytes": 392, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 94, + "tokens_nocomments": 94, + "bytes": 315, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "python", + "encoding": "o200k_base", + "tokens": 95, + "tokens_nocomments": 95, + "bytes": 315, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 105, + "tokens_nocomments": 105, + "bytes": 319, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "js", + "encoding": "o200k_base", + "tokens": 110, + "tokens_nocomments": 110, + "bytes": 319, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 246, + "tokens_nocomments": 240, + "bytes": 731, + "pass": true + }, + { + "task": "x24", + "category": "numbers", + "lang": "go", + "encoding": "o200k_base", + "tokens": 247, + "tokens_nocomments": 241, + "bytes": 731, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 85, + "tokens_nocomments": 85, + "bytes": 232, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 85, + "tokens_nocomments": 85, + "bytes": 232, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 244, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "python", + "encoding": "o200k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 244, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 96, + "tokens_nocomments": 96, + "bytes": 289, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "js", + "encoding": "o200k_base", + "tokens": 100, + "tokens_nocomments": 100, + "bytes": 289, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 196, + "tokens_nocomments": 190, + "bytes": 609, + "pass": true + }, + { + "task": "x25", + "category": "numbers", + "lang": "go", + "encoding": "o200k_base", + "tokens": 197, + "tokens_nocomments": 191, + "bytes": 609, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 272, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 98, + "tokens_nocomments": 98, + "bytes": 272, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 239, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "python", + "encoding": "o200k_base", + "tokens": 77, + "tokens_nocomments": 77, + "bytes": 239, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 103, + "tokens_nocomments": 103, + "bytes": 327, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "js", + "encoding": "o200k_base", + "tokens": 107, + "tokens_nocomments": 107, + "bytes": 327, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 241, + "tokens_nocomments": 235, + "bytes": 807, + "pass": true + }, + { + "task": "x26", + "category": "numbers", + "lang": "go", + "encoding": "o200k_base", + "tokens": 242, + "tokens_nocomments": 236, + "bytes": 807, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 91, + "tokens_nocomments": 91, + "bytes": 278, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 91, + "tokens_nocomments": 91, + "bytes": 278, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 256, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "python", + "encoding": "o200k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 256, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 48, + "tokens_nocomments": 48, + "bytes": 181, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "js", + "encoding": "o200k_base", + "tokens": 52, + "tokens_nocomments": 52, + "bytes": 181, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 221, + "tokens_nocomments": 215, + "bytes": 695, + "pass": true + }, + { + "task": "x27", + "category": "lists", + "lang": "go", + "encoding": "o200k_base", + "tokens": 221, + "tokens_nocomments": 215, + "bytes": 695, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 126, + "tokens_nocomments": 126, + "bytes": 382, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 126, + "tokens_nocomments": 126, + "bytes": 382, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 95, + "tokens_nocomments": 95, + "bytes": 331, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "python", + "encoding": "o200k_base", + "tokens": 95, + "tokens_nocomments": 95, + "bytes": 331, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 100, + "tokens_nocomments": 100, + "bytes": 347, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "js", + "encoding": "o200k_base", + "tokens": 106, + "tokens_nocomments": 106, + "bytes": 347, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 253, + "tokens_nocomments": 247, + "bytes": 808, + "pass": true + }, + { + "task": "x28", + "category": "lists", + "lang": "go", + "encoding": "o200k_base", + "tokens": 254, + "tokens_nocomments": 248, + "bytes": 808, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 60, + "tokens_nocomments": 60, + "bytes": 180, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 60, + "tokens_nocomments": 60, + "bytes": 180, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 215, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "python", + "encoding": "o200k_base", + "tokens": 66, + "tokens_nocomments": 66, + "bytes": 215, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 63, + "tokens_nocomments": 63, + "bytes": 222, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "js", + "encoding": "o200k_base", + "tokens": 69, + "tokens_nocomments": 69, + "bytes": 222, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 203, + "tokens_nocomments": 197, + "bytes": 640, + "pass": true + }, + { + "task": "x29", + "category": "lists", + "lang": "go", + "encoding": "o200k_base", + "tokens": 204, + "tokens_nocomments": 198, + "bytes": 640, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 82, + "tokens_nocomments": 82, + "bytes": 240, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 82, + "tokens_nocomments": 82, + "bytes": 240, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 68, + "tokens_nocomments": 68, + "bytes": 233, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "python", + "encoding": "o200k_base", + "tokens": 68, + "tokens_nocomments": 68, + "bytes": 233, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 94, + "tokens_nocomments": 94, + "bytes": 291, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "js", + "encoding": "o200k_base", + "tokens": 98, + "tokens_nocomments": 98, + "bytes": 291, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 213, + "tokens_nocomments": 207, + "bytes": 687, + "pass": true + }, + { + "task": "x30", + "category": "strings", + "lang": "go", + "encoding": "o200k_base", + "tokens": 214, + "tokens_nocomments": 208, + "bytes": 687, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 105, + "tokens_nocomments": 105, + "bytes": 296, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 105, + "tokens_nocomments": 105, + "bytes": 296, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 243, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "python", + "encoding": "o200k_base", + "tokens": 72, + "tokens_nocomments": 72, + "bytes": 243, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 85, + "tokens_nocomments": 85, + "bytes": 278, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "js", + "encoding": "o200k_base", + "tokens": 89, + "tokens_nocomments": 89, + "bytes": 278, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 212, + "tokens_nocomments": 206, + "bytes": 659, + "pass": true + }, + { + "task": "x31", + "category": "strings", + "lang": "go", + "encoding": "o200k_base", + "tokens": 213, + "tokens_nocomments": 207, + "bytes": 659, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 200, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 200, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 41, + "tokens_nocomments": 41, + "bytes": 160, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "python", + "encoding": "o200k_base", + "tokens": 41, + "tokens_nocomments": 41, + "bytes": 160, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 70, + "tokens_nocomments": 70, + "bytes": 233, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "js", + "encoding": "o200k_base", + "tokens": 75, + "tokens_nocomments": 75, + "bytes": 233, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 139, + "tokens_nocomments": 133, + "bytes": 448, + "pass": true + }, + { + "task": "x32", + "category": "strings", + "lang": "go", + "encoding": "o200k_base", + "tokens": 141, + "tokens_nocomments": 135, + "bytes": 448, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 196, + "tokens_nocomments": 196, + "bytes": 530, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 196, + "tokens_nocomments": 196, + "bytes": 530, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 339, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "python", + "encoding": "o200k_base", + "tokens": 111, + "tokens_nocomments": 111, + "bytes": 339, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 152, + "tokens_nocomments": 152, + "bytes": 458, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "js", + "encoding": "o200k_base", + "tokens": 156, + "tokens_nocomments": 156, + "bytes": 458, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 338, + "tokens_nocomments": 332, + "bytes": 989, + "pass": true + }, + { + "task": "x33", + "category": "logic", + "lang": "go", + "encoding": "o200k_base", + "tokens": 340, + "tokens_nocomments": 334, + "bytes": 989, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 140, + "tokens_nocomments": 140, + "bytes": 348, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 141, + "tokens_nocomments": 141, + "bytes": 348, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 248, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "python", + "encoding": "o200k_base", + "tokens": 73, + "tokens_nocomments": 73, + "bytes": 248, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 107, + "tokens_nocomments": 107, + "bytes": 328, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "js", + "encoding": "o200k_base", + "tokens": 110, + "tokens_nocomments": 110, + "bytes": 328, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 196, + "tokens_nocomments": 190, + "bytes": 625, + "pass": true + }, + { + "task": "x34", + "category": "logic", + "lang": "go", + "encoding": "o200k_base", + "tokens": 197, + "tokens_nocomments": 191, + "bytes": 625, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "howl", + "encoding": "cl100k_base", + "tokens": 107, + "tokens_nocomments": 107, + "bytes": 339, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "howl", + "encoding": "o200k_base", + "tokens": 107, + "tokens_nocomments": 107, + "bytes": 339, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "python", + "encoding": "cl100k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 284, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "python", + "encoding": "o200k_base", + "tokens": 79, + "tokens_nocomments": 79, + "bytes": 284, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "js", + "encoding": "cl100k_base", + "tokens": 97, + "tokens_nocomments": 97, + "bytes": 347, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "js", + "encoding": "o200k_base", + "tokens": 101, + "tokens_nocomments": 101, + "bytes": 347, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "go", + "encoding": "cl100k_base", + "tokens": 283, + "tokens_nocomments": 277, + "bytes": 922, + "pass": true + }, + { + "task": "x35", + "category": "logic", + "lang": "go", + "encoding": "o200k_base", + "tokens": 285, + "tokens_nocomments": 279, + "bytes": 922, + "pass": true + } +] diff --git a/experiments/token-efficiency-heldout-v3/results/verification.json b/experiments/token-efficiency-heldout-v3/results/verification.json new file mode 100644 index 00000000..52ff2a2a --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/results/verification.json @@ -0,0 +1,842 @@ +[ + { + "task": "x01", + "lang": "howl", + "pass": true, + "sha256": "bb062bb062082850c442547e8cca503eac4b2220c59a63a7044cde43c608bae6" + }, + { + "task": "x02", + "lang": "howl", + "pass": true, + "sha256": "cb0e48953617343f3ede9da9e5bd31a2ebce40d3c075c4242f2394cc0b720aab" + }, + { + "task": "x03", + "lang": "howl", + "pass": true, + "sha256": "1e2aab6e22cce7f9a3dbce259de1106239de97e092129b1c59fd3edb4608e3dd" + }, + { + "task": "x04", + "lang": "howl", + "pass": true, + "sha256": "014a5ab778bec264c876584596a2c39a4596a52888b6a43630a4831059ef457b" + }, + { + "task": "x05", + "lang": "howl", + "pass": true, + "sha256": "a8c0de09c5831e889830e265a2cd29991e46e6aa0586c8ab1ec41b24af92b89e" + }, + { + "task": "x06", + "lang": "howl", + "pass": true, + "sha256": "ca1541a5e9e7c92b09a6e737d6a6dac51c5b06ee29d9244ed2b49ffa877d1234" + }, + { + "task": "x07", + "lang": "howl", + "pass": true, + "sha256": "7d75abbf53b6fdea4829a487c328a07b2d80dc859389a121ce463968bf187d70" + }, + { + "task": "x08", + "lang": "howl", + "pass": true, + "sha256": "20f730a89069b710d6ca6f4fc6888b1dad1520d47cb9600d6306e1baaf727a19" + }, + { + "task": "x09", + "lang": "howl", + "pass": true, + "sha256": "abd3825daef88070a5cf16a03972969b42b7741f99bcc7d9b99b37748f6b936f" + }, + { + "task": "x10", + "lang": "howl", + "pass": true, + "sha256": "0a5c6f4f124a8823e3ea2d14cd7291acc5646327588b320ac4a308dc99ef68a5" + }, + { + "task": "x11", + "lang": "howl", + "pass": true, + "sha256": "b65236667fd82a29de123258348684c27e2efda649122e8774f360493c4e96a4" + }, + { + "task": "x12", + "lang": "howl", + "pass": true, + "sha256": "5ec25dca33eaf075b5b19d874a67d65cf7a9f973a4ddcf368a3f15374653274e" + }, + { + "task": "x13", + "lang": "howl", + "pass": true, + "sha256": "0c0da0a635b3a6971489b6c3059ed7c79687cf3ca178728b4ac610d973275eae" + }, + { + "task": "x14", + "lang": "howl", + "pass": true, + "sha256": "1c084b0180278ae924b1885171e4315bb64a173388bec24c7a0a4fa05c766e09" + }, + { + "task": "x15", + "lang": "howl", + "pass": true, + "sha256": "baa3ca340bdd350f3af33adc7d899068bcdfc94f72e6c23ed9dfb7eac1fbe2da" + }, + { + "task": "x16", + "lang": "howl", + "pass": true, + "sha256": "524b156f1c16a863a22dacea1cf31ab635cf2d710f96ada20240e3858fe576eb" + }, + { + "task": "x17", + "lang": "howl", + "pass": true, + "sha256": "12c2c859c551d6725b9f7f6c6040552bf7c83935d75057f74e8e0f315a110f1a" + }, + { + "task": "x18", + "lang": "howl", + "pass": true, + "sha256": "8e492686094411300a951ac64e0f780b54d83989c25dedd040ce22387386b1f8" + }, + { + "task": "x19", + "lang": "howl", + "pass": true, + "sha256": "797a7bdfcc0b6e545f437b7d445ecd26fbd0184418482e836234ff4d3a56cb8f" + }, + { + "task": "x20", + "lang": "howl", + "pass": true, + "sha256": "5e9a366d4d4148917923244b40790b4e509d9f7f96f4c94f4683cf87143da193" + }, + { + "task": "x21", + "lang": "howl", + "pass": true, + "sha256": "83106acc2a0e8e786c416b003eb21c1eca0e75f76193c6a650e59cf0c96ecbfd" + }, + { + "task": "x22", + "lang": "howl", + "pass": true, + "sha256": "8459cb9edff5afcbfcfe75d23d95dbf9e975724138d24eb4c99e1c871f1eb89b" + }, + { + "task": "x23", + "lang": "howl", + "pass": true, + "sha256": "34f4da905265c9d4a4ca12849dfb52bb6c650b527d38775e3f869f3211a615c9" + }, + { + "task": "x24", + "lang": "howl", + "pass": true, + "sha256": "336509b746bb677dc8e9f12f12a3b40fed6a91663447791841c80f0a8c45602b" + }, + { + "task": "x25", + "lang": "howl", + "pass": true, + "sha256": "6c8d3d87177312c832eabac4ef08b9b05588ebd81c73e6de34e67f72642786b6" + }, + { + "task": "x26", + "lang": "howl", + "pass": true, + "sha256": "f507ece1f166896d142b9e49a3ca85b8f620142cb5f08e3ab9f6fbd3a4d41d3f" + }, + { + "task": "x27", + "lang": "howl", + "pass": true, + "sha256": "d44ce2b8db1fba7749c957cccf50a70223a755a5a5f3fd0a2b2939a3ad40ee9b" + }, + { + "task": "x28", + "lang": "howl", + "pass": true, + "sha256": "40bf3760b6c253a067f7f5708f0af0dad23bf1a5cf8ca24c5418f4887e131331" + }, + { + "task": "x29", + "lang": "howl", + "pass": true, + "sha256": "a67cd229fa8296bdb2b2270c999cc83b6861a7a028aa23cbf7f6d30245ba7ebf" + }, + { + "task": "x30", + "lang": "howl", + "pass": true, + "sha256": "492d650079bdf7ae39c1a7a907df1f41018816df53806a0aea57a385c1ae003c" + }, + { + "task": "x31", + "lang": "howl", + "pass": true, + "sha256": "6831ab1b264b63584aa082331b7b237ebf2b2a0adb0f83cf46c92f170379d9b1" + }, + { + "task": "x32", + "lang": "howl", + "pass": true, + "sha256": "c4e32c54f02a5aa1ab3637be5921de3ff0a429ad0f280e86e7b8872f00901abe" + }, + { + "task": "x33", + "lang": "howl", + "pass": true, + "sha256": "7ce93978b39a55db22c0bda75f402d80a472c12120cb5f8b04a696cd06e8e6e4" + }, + { + "task": "x34", + "lang": "howl", + "pass": true, + "sha256": "7f4e4b1ed3b1744d45e5c236ecd15eb3085996b85dfa93b1c1180c7d9a5d37c4" + }, + { + "task": "x35", + "lang": "howl", + "pass": true, + "sha256": "a287aafe06a3e729889265bf24aef4ca10061b9768e43dc2321ec1cc805ff48f" + }, + { + "task": "x01", + "lang": "python", + "pass": true, + "sha256": "ccd1d0519c928ea99a16be362296b9b350e0ec981a79991d2429a2565c97e404" + }, + { + "task": "x02", + "lang": "python", + "pass": true, + "sha256": "f61de199ada79c3d78b693b340d70f5d0850661013e824ae02b635416dfe8be2" + }, + { + "task": "x03", + "lang": "python", + "pass": true, + "sha256": "98432ebf8e928e7646dc02f23652a88f90c1314e2cfd991f78b1dfcd37b19555" + }, + { + "task": "x04", + "lang": "python", + "pass": true, + "sha256": "bcbbec67656bc8eec0ac9e95c8057368bd7a7df86914c92c52a20747f810c69a" + }, + { + "task": "x05", + "lang": "python", + "pass": true, + "sha256": "ee21c291704348289d666219e97be9c2b6749e74576d5c4693e66a9c55cb9f24" + }, + { + "task": "x06", + "lang": "python", + "pass": true, + "sha256": "569a5af7ef631e53ec3115c1d9fde860f4785ccb07966ede14f06e71006f1934" + }, + { + "task": "x07", + "lang": "python", + "pass": true, + "sha256": "5111d24b38ad58091b9909e56ee136e8c582d4c7ded227d3f46f0c50ffab407f" + }, + { + "task": "x08", + "lang": "python", + "pass": true, + "sha256": "78447469f75f555fbb3a75e3cf31ace274e10461bc783fa6d5b10ae093ff7280" + }, + { + "task": "x09", + "lang": "python", + "pass": true, + "sha256": "fc4ec9537bccb6356bb1c704df0b2cd7fdcd3ed43da8d7fcdea71c73c7928229" + }, + { + "task": "x10", + "lang": "python", + "pass": true, + "sha256": "8f20446c4ecec53e7a833636c6350f4061df7332c89750eba62ff5cfa608e3fb" + }, + { + "task": "x11", + "lang": "python", + "pass": true, + "sha256": "8935253e2185abfb488efe8cb166ae2f431835d52ccf85ec0e93884717617d3c" + }, + { + "task": "x12", + "lang": "python", + "pass": true, + "sha256": "2de65e25278c91ad606912145e35b2e423f6a23748493ef61ed41d86800cec6c" + }, + { + "task": "x13", + "lang": "python", + "pass": true, + "sha256": "b44a832324e7258b5f0c4764bbef299c211a0ef6f5f8af883399977138ecf8e2" + }, + { + "task": "x14", + "lang": "python", + "pass": true, + "sha256": "5c06b13449327018eb615b198e7b6266552cbdc2d5ded5d25e905ed8d1659914" + }, + { + "task": "x15", + "lang": "python", + "pass": true, + "sha256": "75fcee6da1bea919373ef64cbc42eabe9d00535b41b6849200da68a135fd16e8" + }, + { + "task": "x16", + "lang": "python", + "pass": true, + "sha256": "8eb2ec5b4eba480798e7e12da2f46186c5f3d6207224978c524996bb8821461a" + }, + { + "task": "x17", + "lang": "python", + "pass": true, + "sha256": "10db829777cda742b758f136d01a946313349d9256a26a9f3fb3e55ff77bbc6a" + }, + { + "task": "x18", + "lang": "python", + "pass": true, + "sha256": "3c158f59c2705cebaba14b72bd03844ebfbec59597046b23bbe23b05b5ca73cb" + }, + { + "task": "x19", + "lang": "python", + "pass": true, + "sha256": "222ae1940084838aed29875829a05680b2c9eb7c83852f6bb5458f62423c6ce2" + }, + { + "task": "x20", + "lang": "python", + "pass": true, + "sha256": "8ae5d273e50c34d208670886abc4cb9c079655e189665984c739d4bc52708232" + }, + { + "task": "x21", + "lang": "python", + "pass": true, + "sha256": "9418d868d415c97b092c813e33a08e5381ad32af217a366248090105ee264031" + }, + { + "task": "x22", + "lang": "python", + "pass": true, + "sha256": "6303c392e679e8f47a5ece5bd19d6b9df5bae23ae1a3429b59abde41b7781892" + }, + { + "task": "x23", + "lang": "python", + "pass": true, + "sha256": "b4368c9a96229c8d8a17b45525d8ee63f6b1528fdc9049b707169904735cb0ef" + }, + { + "task": "x24", + "lang": "python", + "pass": true, + "sha256": "a165b1f61b99d5991d340c5562d16e230792a5ff54a07b07ed258a2360c0f582" + }, + { + "task": "x25", + "lang": "python", + "pass": true, + "sha256": "4bf072b325eb139fe805f5dfe82e975e7779b1bc3cb132a28f2149967e47037a" + }, + { + "task": "x26", + "lang": "python", + "pass": true, + "sha256": "7cf7b19621179b98bce78cc196388d81fea9250f0c476ca055dabf65a1cc30ed" + }, + { + "task": "x27", + "lang": "python", + "pass": true, + "sha256": "7bad486d6f4e3188124ae2f7ef89dd3e415b14a7535fc943c4107c71d69206d8" + }, + { + "task": "x28", + "lang": "python", + "pass": true, + "sha256": "387f58955a58072056378c5bdc473d171792120a3b210dfc707a5ddebde0952a" + }, + { + "task": "x29", + "lang": "python", + "pass": true, + "sha256": "edb53b63f2d410706aac4993d676cc748befa6f0f1cf31c4f7da807842889aa5" + }, + { + "task": "x30", + "lang": "python", + "pass": true, + "sha256": "1c4a69290bfb11b954ad852077c0b9fe11f5f8bedb56ecc791e1b132f4d2dddd" + }, + { + "task": "x31", + "lang": "python", + "pass": true, + "sha256": "e017b810a85fb7c1026ed971a015ca2ead300cb5eced6a1f7dfdfd963d60f2aa" + }, + { + "task": "x32", + "lang": "python", + "pass": true, + "sha256": "949c8f57602fe3eb8b01f1a09db1fba8d451173ecb3669d3e2c5c65d181b26d2" + }, + { + "task": "x33", + "lang": "python", + "pass": true, + "sha256": "396a3ca6b2968657ce0b9d0be1d5d170f6b5d072985ab374ea50698df0a0512f" + }, + { + "task": "x34", + "lang": "python", + "pass": true, + "sha256": "e244ca0da1a888c05641fc328c9cc4f480780f36753751f790b40f81203fe7f7" + }, + { + "task": "x35", + "lang": "python", + "pass": true, + "sha256": "0d4ee7acf3c78e67b3901b760e69d32857370677b4723647e9f622c5bce741aa" + }, + { + "task": "x01", + "lang": "js", + "pass": true, + "sha256": "810cb930aba9ddf6dcdcfe6408876ecb61f6d07f7c86f13816515462ed54f84f" + }, + { + "task": "x02", + "lang": "js", + "pass": true, + "sha256": "d1afb52a7b6f54387c2e8bf070fae72901d0af0becfff018ff1bc241baf77669" + }, + { + "task": "x03", + "lang": "js", + "pass": true, + "sha256": "80861908391364abb51b7a9a1410cf5a6016b88cf60e759181fb22c197500579" + }, + { + "task": "x04", + "lang": "js", + "pass": true, + "sha256": "c54f79e0b77cda5fe65dba74e927b5a941d86499504e1dc9490b0678bd01023e" + }, + { + "task": "x05", + "lang": "js", + "pass": true, + "sha256": "64ce091ca0c891e742f67e073b8139eaf555ce81c0d4ccb58297cf9879997aad" + }, + { + "task": "x06", + "lang": "js", + "pass": true, + "sha256": "676fb98689b78e59fd89851e0e38febcb14455d73453d7ce7ee4fac92e67c24e" + }, + { + "task": "x07", + "lang": "js", + "pass": true, + "sha256": "ab2d00a382428b5ef67229c9c128ba502d8f189957335cb6644776c25cfb95a0" + }, + { + "task": "x08", + "lang": "js", + "pass": true, + "sha256": "9d7553aef97bb0e46e13a30e25e75465cdc0ff22f341a4e36bffc59ab3b5bdcf" + }, + { + "task": "x09", + "lang": "js", + "pass": true, + "sha256": "33a326473af951b90e699ed0f8ee43727fd50dd9ea71dde1e66c5eb009519fad" + }, + { + "task": "x10", + "lang": "js", + "pass": true, + "sha256": "1c30fef1559f824e90e59482d2a9addd7e77296dc6ce38b7a5998a39583a955a" + }, + { + "task": "x11", + "lang": "js", + "pass": true, + "sha256": "13d028aa8f691d684504e3d38680dc110ed2d30a58f92d130f71c389af6ae4ec" + }, + { + "task": "x12", + "lang": "js", + "pass": true, + "sha256": "5e1ada75df8a362f25fce523e783e3295c74756867964c333095960cc40983a5" + }, + { + "task": "x13", + "lang": "js", + "pass": true, + "sha256": "51946855c065070ee5b8e70704a469c75ffbb4f373a1ac3b419e7cfdc2df4422" + }, + { + "task": "x14", + "lang": "js", + "pass": true, + "sha256": "874214840db9f5e0501b387a3ccdf6433f0240958f4578aff201ef622d13183b" + }, + { + "task": "x15", + "lang": "js", + "pass": true, + "sha256": "f671cca8a42ff7a509d7da3d2a3bbc6a42de1e3af1a20962ab6b30b34d1368ce" + }, + { + "task": "x16", + "lang": "js", + "pass": true, + "sha256": "090b8df4bdc56bb266d74347c466ef22ed37608c4e3c37abdd7898ac2744889a" + }, + { + "task": "x17", + "lang": "js", + "pass": true, + "sha256": "d0d6357a4168a3badfa00035465ad36b8e3a22b0ed90fe8c38d8087004c2e848" + }, + { + "task": "x18", + "lang": "js", + "pass": true, + "sha256": "ba9d3b417b43705633d865e74772b188ffe8b61899cf0ecefdfe7eb928280dac" + }, + { + "task": "x19", + "lang": "js", + "pass": true, + "sha256": "022c89d96f330e9fdbab304810f6b9a4c8a00dfcf7fe4879e5a5b06a2d496cb9" + }, + { + "task": "x20", + "lang": "js", + "pass": true, + "sha256": "3502bde168ffc00ed111490587f1c3aa42b5e3b8aad247e65d81f8adbbe1aa49" + }, + { + "task": "x21", + "lang": "js", + "pass": true, + "sha256": "5e3b476fe4201867ad3bee01b9a43982088b8df5a9ce99082a3beb3d61495551" + }, + { + "task": "x22", + "lang": "js", + "pass": true, + "sha256": "532ebe15db0043824802797f2f5dcbc1de016e568a5d54286a9cfd775ab9a523" + }, + { + "task": "x23", + "lang": "js", + "pass": true, + "sha256": "67f085ba9a69bca5b1d6fe889950d145cb09ac8a163d000822b9e1427030878f" + }, + { + "task": "x24", + "lang": "js", + "pass": true, + "sha256": "1ca48bb55cd15bfd59058dac995ef30aaf3d0a573e48bb142e9f705178bf0e32" + }, + { + "task": "x25", + "lang": "js", + "pass": true, + "sha256": "34cf79d2da8f6310d0d499c119cce304b3e670827a2e8ad3f75d667aa6e2b566" + }, + { + "task": "x26", + "lang": "js", + "pass": true, + "sha256": "58e23dd101736fdc91e8edc3e4f192693bfc899e22697bb626eb2e1c466ff125" + }, + { + "task": "x27", + "lang": "js", + "pass": true, + "sha256": "1ce778fde075e46aa4867db9d7f045857566328f20be938a3973c043f2679d36" + }, + { + "task": "x28", + "lang": "js", + "pass": true, + "sha256": "13447bab585d93e84963d87cd80471c6938a70f359387f3baf847d224388c318" + }, + { + "task": "x29", + "lang": "js", + "pass": true, + "sha256": "70b485f9d6dc1df0ad81dd62a14ee49fd7748013cc0830e63278cd885cc52f15" + }, + { + "task": "x30", + "lang": "js", + "pass": true, + "sha256": "68edca153194d1d382af767e54faff0c34e228a339055c147e9f5463ce48a097" + }, + { + "task": "x31", + "lang": "js", + "pass": true, + "sha256": "f92896a9840257401f093eaa6e57de86fd798be228bbefe29635a6bea536b353" + }, + { + "task": "x32", + "lang": "js", + "pass": true, + "sha256": "c607224f1e2989710418985b889016e9a5adb5365b50db936b43d7b551bf998e" + }, + { + "task": "x33", + "lang": "js", + "pass": true, + "sha256": "e40f4384cec11f6b75412433b58dc663ff784e24ecec7803e21507b3ccc7b3b5" + }, + { + "task": "x34", + "lang": "js", + "pass": true, + "sha256": "6edf2a05d994f1bdc475853136656bd3e32c0710ae3cdfd38cc0d60ad85e1b3c" + }, + { + "task": "x35", + "lang": "js", + "pass": true, + "sha256": "72b60a642440ff4d67555c1597729d179cf305ab9ae7ec76a720d8ff47eaa23c" + }, + { + "task": "x01", + "lang": "go", + "pass": true, + "sha256": "6284cb689a56dabf5d022c4469a3b4c75a51dad767f054e1c064d925af7d0a1e" + }, + { + "task": "x02", + "lang": "go", + "pass": true, + "sha256": "b108237da81c2515bccf82226cc59041b22837e2cbd1d761c1d9abda1c749b28" + }, + { + "task": "x03", + "lang": "go", + "pass": true, + "sha256": "e77ff75e3310dd64061ab5f74e6eff51f6356cd056e7768110f72c6f5484a792" + }, + { + "task": "x04", + "lang": "go", + "pass": true, + "sha256": "361183ad1cb1834ef34b03e34e443bb69bb9b4ec1ba76b5d02c3cccda6e01354" + }, + { + "task": "x05", + "lang": "go", + "pass": true, + "sha256": "6e79ed287a2bddef12fdb78ae9cfd9813333d6acc8a37f02d9ebac4dbefd613b" + }, + { + "task": "x06", + "lang": "go", + "pass": true, + "sha256": "a16bb40277592823f6627507fc0d77a53f01aabeb41db1b4cb844d1aad60b248" + }, + { + "task": "x07", + "lang": "go", + "pass": true, + "sha256": "d9b156970e08efc73106fcf0fa66e1eaa8ab25d08686179b415301f7608f4aca" + }, + { + "task": "x08", + "lang": "go", + "pass": true, + "sha256": "f7fd216aee38dbcb83a15e3a236bdacd8df9eadd329eeb1852e8d7132165d51b" + }, + { + "task": "x09", + "lang": "go", + "pass": true, + "sha256": "a5c2519795dba802095583cad8ab2bbb03b10ce9b77ff4210d3352cc2300efed" + }, + { + "task": "x10", + "lang": "go", + "pass": true, + "sha256": "c64acf031639bd32a8a238100333da377877ad4187e21f049d61bd59faa808d5" + }, + { + "task": "x11", + "lang": "go", + "pass": true, + "sha256": "ce95a88cdb13a5def4e3ffebb7f4ee87e9e33e88307e04df6f0a515fcf7c3846" + }, + { + "task": "x12", + "lang": "go", + "pass": true, + "sha256": "253077c40025ff0f0e2da9256c26af0930839efea09ea69aed23ef268aede21b" + }, + { + "task": "x13", + "lang": "go", + "pass": true, + "sha256": "aab0e10848a8717c2ea39aa8da33b9a32067af352ca75a96d62ba3e994e5791d" + }, + { + "task": "x14", + "lang": "go", + "pass": true, + "sha256": "71aca530230938ac99975a655e63573f0cd6c55f50742a51a83632b7a86889a6" + }, + { + "task": "x15", + "lang": "go", + "pass": true, + "sha256": "4affe1a3021ed08836dce868c2dcb04cdd72b939c1a6a36b9414cb724c043aae" + }, + { + "task": "x16", + "lang": "go", + "pass": true, + "sha256": "f3e4a6882d6aa38a6d9ac7989e5231d8272eee0399894220ea544caf72144ab6" + }, + { + "task": "x17", + "lang": "go", + "pass": true, + "sha256": "9c07b793bbd7ff1fde0474dada31812f9ef068723cf585718755f4a56f6f06da" + }, + { + "task": "x18", + "lang": "go", + "pass": true, + "sha256": "ba64f2392d55677b78594f72e058d94d8cf94f26d1a2dd8ace4eeaf70e8ac618" + }, + { + "task": "x19", + "lang": "go", + "pass": true, + "sha256": "a6560101135b7c0245f647b1a176fb1b2e749aba789281f41ca971f9f385da97" + }, + { + "task": "x20", + "lang": "go", + "pass": true, + "sha256": "83d12841df9bbf2d26c208ec87882e78163f7f55ea114d11a2937cb2e567e5e3" + }, + { + "task": "x21", + "lang": "go", + "pass": true, + "sha256": "219ae7cc9429e3130125bc30144b1f3a4d549c08a3d6183341a371b7408cd4f1" + }, + { + "task": "x22", + "lang": "go", + "pass": true, + "sha256": "a90d9210696c1dd00de9e4545e65189dd6339cecb41e1f2d63f789e6dc948f9a" + }, + { + "task": "x23", + "lang": "go", + "pass": true, + "sha256": "52df17fcf7f25273cd9b5c142a3c96e213da611c3f120c79f8f98bb82e61471b" + }, + { + "task": "x24", + "lang": "go", + "pass": true, + "sha256": "81a6ac5a293e8d1f9d1482dead7de17c66c3d634628e0ae3e42384c3f59badab" + }, + { + "task": "x25", + "lang": "go", + "pass": true, + "sha256": "5c9c38fcdd02ee32eab70d2de6ff3837092208fbc931483ea4d0a5f25e4b5242" + }, + { + "task": "x26", + "lang": "go", + "pass": true, + "sha256": "5e4b9ad2d5dd164a319e3fe14e9df3eb0c29f5bf126bfb973c5db7c9e9157a66" + }, + { + "task": "x27", + "lang": "go", + "pass": true, + "sha256": "206cdc486dd3e1df770c9f83042bb68d2c3b4c39e12caf0551a28797d39c9667" + }, + { + "task": "x28", + "lang": "go", + "pass": true, + "sha256": "b28081cdbe6ceea24e53e5b64ce1f5274b200287dc032efd782b9a1df7a294fe" + }, + { + "task": "x29", + "lang": "go", + "pass": true, + "sha256": "b211ef65805a242a0cf3e1351fd82f58aa7e4dba3240f8ee6ca44a5edd6a88b1" + }, + { + "task": "x30", + "lang": "go", + "pass": true, + "sha256": "6cb62e3244df173f97285c09ac71adde19cc0a47732f4aa37788ccbe7903b24d" + }, + { + "task": "x31", + "lang": "go", + "pass": true, + "sha256": "1879945760488d3bbcecfa88f7638e148468a5545aea153ea11905a985fd6353" + }, + { + "task": "x32", + "lang": "go", + "pass": true, + "sha256": "9bbb572e9f5127e622dabc940d96056904e5a687796ba18cb3069f422fa4148f" + }, + { + "task": "x33", + "lang": "go", + "pass": true, + "sha256": "5e35144b9519728a74d9a118efbd0b3f52a8bfc43c268ff3e883bbfa94872fcb" + }, + { + "task": "x34", + "lang": "go", + "pass": true, + "sha256": "4c1b037d4491ef23ac9210a4637d34cf8a764c0e69d61604b9a785c9ec19d189" + }, + { + "task": "x35", + "lang": "go", + "pass": true, + "sha256": "db2125c2a9d22e97671c2c5c039f52258fe1aa424a8148b316f3f95c60ddd1a9" + } +] diff --git a/experiments/token-efficiency-heldout-v3/results_freeze_test.go b/experiments/token-efficiency-heldout-v3/results_freeze_test.go new file mode 100644 index 00000000..4aba0325 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/results_freeze_test.go @@ -0,0 +1,31 @@ +package heldoutv3 + +import ( + "crypto/sha256" + "fmt" + "os" + "testing" +) + +// The heldout-v3 baseline (HowlFrame sources using only features on main +// a6bb53b6, before the records slice) is frozen: the recorded gaps and +// results must match the hashes pinned here. Re-running run.sh reproduces +// these bytes. +func TestHeldoutV3BaselineResultsFrozen(t *testing.T) { + frozen := map[string]string{ + "gaps.json": "37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570", + "results/summary.json": "a7cc74b4e1c454e66e3e0c24adc833599e0380d72cc9fc391253c14406f1950c", + "results/tokens.csv": "5204cbd4588c15aa2edd5ad484636bbbf3fff35c6bda44c5c07ba6e7cd6720e2", + "results/tokens.json": "507cd2686015598ec2578711292e07999b1af0ff59b06ef6a99ef7371c8fcd1e", + "results/verification.json": "b050ab05d4672623f279c7d0f0c9f61838f2be65883bc8867c680333e54248a1", + } + for path, want := range frozen { + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if got := fmt.Sprintf("%x", sha256.Sum256(data)); got != want { + t.Errorf("%s changed after baseline freeze: %s", path, got) + } + } +} diff --git a/experiments/token-efficiency-heldout-v3/run.sh b/experiments/token-efficiency-heldout-v3/run.sh new file mode 100755 index 00000000..8d7534e7 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/run.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")/../.." +export GOCACHE="${GOCACHE:-/tmp/hf-token-go-cache}" +go build -o /tmp/howlframe . +go build -o /tmp/heldoutv3verify ./experiments/token-efficiency-heldout-v3/cmd/heldoutv3verify +status=0 +/tmp/heldoutv3verify -howlframe /tmp/howlframe || status=$? +if [ "$status" -eq 0 ]; then + "${TOKBENCH_PYTHON:-python3}" experiments/token-efficiency-heldout-v3/count_tokens.py +fi +exit "$status" diff --git a/experiments/token-efficiency-heldout-v3/run_postrec.sh b/experiments/token-efficiency-heldout-v3/run_postrec.sh new file mode 100755 index 00000000..0502f41f --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/run_postrec.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# Post-records rerun: same frozen specs, same Python/JS/Go references and the +# same verifier/tokenizer as run.sh, HowlFrame sources from postrec/. The +# baseline (solutions/howl, results/) is not touched. +set -euo pipefail +cd "$(dirname "$0")/../.." +export GOCACHE="${GOCACHE:-/tmp/hf-token-go-cache}" +go build -o /tmp/howlframe . +go build -o /tmp/heldoutv3verify ./experiments/token-efficiency-heldout-v3/cmd/heldoutv3verify +B=experiments/token-efficiency-heldout-v3 +status=0 +/tmp/heldoutv3verify -howlframe /tmp/howlframe -howl-dir "$B/postrec/solutions/howl" -results "$B/postrec/results" || status=$? +if [ "$status" -eq 0 ]; then + "${TOKBENCH_PYTHON:-python3}" "$B/count_tokens.py" --howl-dir postrec/solutions/howl --results postrec/results + "${TOKBENCH_PYTHON:-python3}" "$B/compare_postrec.py" +fi +exit "$status" diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x01.go b/experiments/token-efficiency-heldout-v3/solutions/go/x01.go new file mode 100644 index 00000000..c22b6e39 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x01.go @@ -0,0 +1,59 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var cfg any + readJSON("in/config.json", &cfg) + var out []string + for _, line := range strings.Split(read("in/paths.txt"), "\n") { + v, ok := cfg, true + for _, k := range strings.Split(line, ".") { + m, isMap := v.(map[string]any) + if v, ok = m[k]; !isMap || !ok { + ok = false + break + } + } + if !ok { + out = append(out, "missing") + continue + } + enc, err := json.Marshal(v) + if err != nil { + panic(err) + } + out = append(out, string(enc)) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x02.go b/experiments/token-efficiency-heldout-v3/solutions/go/x02.go new file mode 100644 index 00000000..162e99a9 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x02.go @@ -0,0 +1,48 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func merge(a, b map[string]any) map[string]any { + for k, v := range b { + am, ok1 := a[k].(map[string]any) + bm, ok2 := v.(map[string]any) + if ok1 && ok2 { + a[k] = merge(am, bm) + } else { + a[k] = v + } + } + return a +} + +func main() { + var base, over map[string]any + readJSON("in/base.json", &base) + readJSON("in/override.json", &over) + writeJSON(merge(base, over)) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x03.go b/experiments/token-efficiency-heldout-v3/solutions/go/x03.go new file mode 100644 index 00000000..c37be356 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x03.go @@ -0,0 +1,46 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func flat(d map[string]any, pre string, out map[string]any) { + for k, v := range d { + if m, ok := v.(map[string]any); ok && len(m) > 0 { + flat(m, pre+k+".", out) + } else { + out[pre+k] = v + } + } +} + +func main() { + var d map[string]any + readJSON("in/nested.json", &d) + out := map[string]any{} + flat(d, "", out) + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x04.go b/experiments/token-efficiency-heldout-v3/solutions/go/x04.go new file mode 100644 index 00000000..07802643 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x04.go @@ -0,0 +1,49 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "strings" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var flat map[string]any + readJSON("in/flat.json", &flat) + out := map[string]any{} + for k, v := range flat { + parts := strings.Split(k, ".") + node := out + for _, p := range parts[:len(parts)-1] { + next, ok := node[p].(map[string]any) + if !ok { + next = map[string]any{} + node[p] = next + } + node = next + } + node[parts[len(parts)-1]] = v + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x05.go b/experiments/token-efficiency-heldout-v3/solutions/go/x05.go new file mode 100644 index 00000000..fe1d05b8 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x05.go @@ -0,0 +1,45 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var contacts []map[string]any + readJSON("in/contacts.json", &contacts) + out := map[string][]any{"present": {}, "null": {}, "absent": {}} + for _, c := range contacts { + email, ok := c["email"] + key := "present" + if !ok { + key = "absent" + } else if email == nil { + key = "null" + } + out[key] = append(out[key], c["id"]) + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x06.go b/experiments/token-efficiency-heldout-v3/solutions/go/x06.go new file mode 100644 index 00000000..d0dd9806 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x06.go @@ -0,0 +1,58 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "slices" + "sort" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var sales []struct { + Region, Product string + Units int + } + readJSON("in/sales.json", &sales) + type region struct { + Units int `json:"units"` + Products []string `json:"products"` + } + out := map[string]*region{} + for _, s := range sales { + r, ok := out[s.Region] + if !ok { + r = ®ion{Products: []string{}} + out[s.Region] = r + } + r.Units += s.Units + if !slices.Contains(r.Products, s.Product) { + r.Products = append(r.Products, s.Product) + } + } + for _, r := range out { + sort.Strings(r.Products) + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x07.go b/experiments/token-efficiency-heldout-v3/solutions/go/x07.go new file mode 100644 index 00000000..95eff119 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x07.go @@ -0,0 +1,47 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var rows []map[string]any + var ren map[string]string + readJSON("in/rows.json", &rows) + readJSON("in/rename.json", &ren) + out := []map[string]any{} + for _, r := range rows { + m := map[string]any{} + for k, v := range r { + if n, ok := ren[k]; ok { + k = n + } + m[k] = v + } + out = append(out, m) + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x08.go b/experiments/token-efficiency-heldout-v3/solutions/go/x08.go new file mode 100644 index 00000000..ce93f141 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x08.go @@ -0,0 +1,30 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func main() { + var doc any + readJSON("in/doc.json", &doc) + enc, err := json.Marshal(doc) // map keys are marshalled in sorted order + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.txt", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x09.go b/experiments/token-efficiency-heldout-v3/solutions/go/x09.go new file mode 100644 index 00000000..3260f092 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x09.go @@ -0,0 +1,53 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "reflect" + "sort" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var old, nw map[string]any + readJSON("in/old.json", &old) + readJSON("in/new.json", &nw) + out := map[string][]string{"added": {}, "removed": {}, "changed": {}} + for k, v := range nw { + if ov, ok := old[k]; !ok { + out["added"] = append(out["added"], k) + } else if !reflect.DeepEqual(ov, v) { + out["changed"] = append(out["changed"], k) + } + } + for k := range old { + if _, ok := nw[k]; !ok { + out["removed"] = append(out["removed"], k) + } + } + for _, v := range out { + sort.Strings(v) + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x10.go b/experiments/token-efficiency-heldout-v3/solutions/go/x10.go new file mode 100644 index 00000000..fdfd0e79 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x10.go @@ -0,0 +1,41 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "slices" + "strings" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var tickets []struct { + ID string + Meta struct{ Tags []string } + } + readJSON("in/tickets.json", &tickets) + var out []string + for _, t := range tickets { + if slices.Contains(t.Meta.Tags, "urgent") { + out = append(out, t.ID) + } + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x11.go b/experiments/token-efficiency-heldout-v3/solutions/go/x11.go new file mode 100644 index 00000000..4bd292b3 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x11.go @@ -0,0 +1,44 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "strings" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var accounts []struct { + Login string + User struct { + Profile struct{ City *string } + } + } + readJSON("in/accounts.json", &accounts) + var out []string + for _, a := range accounts { + city := "unknown" + if a.User.Profile.City != nil { + city = *a.User.Profile.City + } + out = append(out, a.Login+": "+city) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x12.go b/experiments/token-efficiency-heldout-v3/solutions/go/x12.go new file mode 100644 index 00000000..286296e0 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x12.go @@ -0,0 +1,41 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var people []struct { + First, Last string + Age int + } + readJSON("in/people.json", &people) + out := map[string]int{} + for _, p := range people { + out[p.Last+", "+p.First] = p.Age + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x13.go b/experiments/token-efficiency-heldout-v3/solutions/go/x13.go new file mode 100644 index 00000000..91f7b480 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x13.go @@ -0,0 +1,44 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + out := map[string]int{} + for _, line := range strings.Split(read("in/requests.jsonl"), "\n") { + var r struct { + Level string + Ms int + } + if err := json.Unmarshal([]byte(line), &r); err != nil { + panic(err) + } + if cur, ok := out[r.Level]; !ok || r.Ms > cur { + out[r.Level] = r.Ms + } + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x14.go b/experiments/token-efficiency-heldout-v3/solutions/go/x14.go new file mode 100644 index 00000000..5777c39f --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x14.go @@ -0,0 +1,39 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "fmt" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var prices map[string]int + readJSON("in/prices.json", &prices) + out := map[string]string{} + for k, v := range prices { + out[k] = fmt.Sprintf("$%d.%02d", v/100, v%100) + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x15.go b/experiments/token-efficiency-heldout-v3/solutions/go/x15.go new file mode 100644 index 00000000..91d56fc3 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x15.go @@ -0,0 +1,46 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "fmt" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func depth(v any) int { + var kids []any + switch t := v.(type) { + case map[string]any: + for _, x := range t { + kids = append(kids, x) + } + case []any: + kids = t + default: + return 0 + } + m := 0 + for _, x := range kids { + m = max(m, depth(x)) + } + return m + 1 +} + +func main() { + var tree any + readJSON("in/tree.json", &tree) + if err := os.WriteFile("out/result.txt", []byte(fmt.Sprintln(depth(tree))), 0o644); err != nil { + panic(err) + } +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x16.go b/experiments/token-efficiency-heldout-v3/solutions/go/x16.go new file mode 100644 index 00000000..44d0ca27 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x16.go @@ -0,0 +1,44 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "sort" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var env map[string]string + readJSON("in/env.json", &env) + keys := make([]string, 0, len(env)) + for k := range env { + keys = append(keys, k) + } + sort.Strings(keys) + out := []map[string]string{} + for _, k := range keys { + out = append(out, map[string]string{"key": k, "value": env[k]}) + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x17.go b/experiments/token-efficiency-heldout-v3/solutions/go/x17.go new file mode 100644 index 00000000..e7bdfaa8 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x17.go @@ -0,0 +1,57 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "strings" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var doc map[string]any + var ops []struct { + Op, Path string + Value any + } + readJSON("in/doc.json", &doc) + readJSON("in/ops.json", &ops) + for _, op := range ops { + parts := strings.Split(op.Path, ".") + node := doc + for _, p := range parts[:len(parts)-1] { + next, ok := node[p].(map[string]any) + if !ok { + next = map[string]any{} + node[p] = next + } + node = next + } + if last := parts[len(parts)-1]; op.Op == "set" { + node[last] = op.Value + } else { + delete(node, last) + } + } + writeJSON(doc) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x18.go b/experiments/token-efficiency-heldout-v3/solutions/go/x18.go new file mode 100644 index 00000000..0406dd51 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x18.go @@ -0,0 +1,36 @@ +//go:build ignore + +package main + +import ( + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var out []string + cur := "" + for _, line := range strings.Split(read("in/script.txt"), "\n") { + if s, ok := strings.CutSuffix(line, "\\"); ok { + cur += s + } else { + out = append(out, cur+line) + cur = "" + } + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x19.go b/experiments/token-efficiency-heldout-v3/solutions/go/x19.go new file mode 100644 index 00000000..d9db69b6 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x19.go @@ -0,0 +1,35 @@ +//go:build ignore + +package main + +import ( + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var out []string + inside := false + for _, line := range strings.Split(read("in/app.ini"), "\n") { + if strings.HasPrefix(line, "[") { + inside = line == "[net]" + } else if inside && line != "" { + out = append(out, line) + } + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x20.go b/experiments/token-efficiency-heldout-v3/solutions/go/x20.go new file mode 100644 index 00000000..53357c58 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x20.go @@ -0,0 +1,40 @@ +//go:build ignore + +package main + +import ( + "fmt" + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var out []string + words := 0 + lines := append(strings.Split(read("in/doc.txt"), "\n"), "") + for _, line := range lines { + if strings.TrimSpace(line) == "" { + if words > 0 { + out = append(out, fmt.Sprintf("%d: %d", len(out)+1, words)) + } + words = 0 + continue + } + words += len(strings.Fields(line)) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x21.go b/experiments/token-efficiency-heldout-v3/solutions/go/x21.go new file mode 100644 index 00000000..b4740582 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x21.go @@ -0,0 +1,38 @@ +//go:build ignore + +package main + +import ( + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.csv", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var rows [][]string + for _, line := range strings.Split(read("in/grid.csv"), "\n") { + rows = append(rows, strings.Split(line, ",")) + } + var out []string + for j := range rows[0] { + col := make([]string, len(rows)) + for i, r := range rows { + col[i] = r[j] + } + out = append(out, strings.Join(col, ",")) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x22.go b/experiments/token-efficiency-heldout-v3/solutions/go/x22.go new file mode 100644 index 00000000..6372f2cb --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x22.go @@ -0,0 +1,39 @@ +//go:build ignore + +package main + +import ( + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.csv", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + lines := strings.Split(read("in/sites.csv"), "\n") + out := []string{lines[0]} + var prev []string + for _, line := range lines[1:] { + cells := strings.Split(line, ",") + for j, c := range cells { + if c == "" { + cells[j] = prev[j] + } + } + prev = cells + out = append(out, strings.Join(cells, ",")) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x23.go b/experiments/token-efficiency-heldout-v3/solutions/go/x23.go new file mode 100644 index 00000000..e5fde59d --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x23.go @@ -0,0 +1,36 @@ +//go:build ignore + +package main + +import ( + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.csv", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + dept := map[string]string{} + for _, line := range strings.Split(read("in/dept.csv"), "\n")[1:] { + id, d, _ := strings.Cut(line, ",") + dept[id] = d + } + out := []string{"id,name,dept"} + for _, line := range strings.Split(read("in/emp.csv"), "\n")[1:] { + id, _, _ := strings.Cut(line, ",") + out = append(out, line+","+dept[id]) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x24.go b/experiments/token-efficiency-heldout-v3/solutions/go/x24.go new file mode 100644 index 00000000..eca69d1a --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x24.go @@ -0,0 +1,48 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "strconv" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func gcd(a, b int) int { + for b != 0 { + a, b = b, a%b + } + return a +} + +func main() { + g, l := 0, 1 + for _, s := range strings.Split(read("in/vals.txt"), ",") { + n, err := strconv.Atoi(s) + if err != nil { + panic(err) + } + g = gcd(g, n) + l = l / gcd(l, n) * n + } + writeJSON(map[string]int{"gcd": g, "lcm": l}) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x25.go b/experiments/token-efficiency-heldout-v3/solutions/go/x25.go new file mode 100644 index 00000000..99719f14 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x25.go @@ -0,0 +1,39 @@ +//go:build ignore + +package main + +import ( + "fmt" + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var out []string + for _, n := range strings.Split(read("in/nums.txt"), "\n") { + d := n + for len(d) > 1 { + s := 0 + for _, c := range d { + s += int(c - '0') + } + d = fmt.Sprint(s) + } + out = append(out, n+" -> "+d) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x26.go b/experiments/token-efficiency-heldout-v3/solutions/go/x26.go new file mode 100644 index 00000000..b5a5f893 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x26.go @@ -0,0 +1,43 @@ +//go:build ignore + +package main + +import ( + "fmt" + "os" + "strconv" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var names []string + var counts []int + total := 0 + for _, line := range strings.Split(read("in/votes.txt"), "\n") { + name, c, _ := strings.Cut(line, " ") + n, err := strconv.Atoi(c) + if err != nil { + panic(err) + } + names, counts, total = append(names, name), append(counts, n), total+n + } + var out []string + for i, name := range names { + out = append(out, fmt.Sprintf("%s %d%%", name, counts[i]*100/total)) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x27.go b/experiments/token-efficiency-heldout-v3/solutions/go/x27.go new file mode 100644 index 00000000..2cbf01fe --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x27.go @@ -0,0 +1,44 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func flat(v any, out []any) []any { + if l, ok := v.([]any); ok { + for _, e := range l { + out = flat(e, out) + } + return out + } + return append(out, v) +} + +func main() { + var deep []any + readJSON("in/deep.json", &deep) + writeJSON(flat(deep, []any{})) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x28.go b/experiments/token-efficiency-heldout-v3/solutions/go/x28.go new file mode 100644 index 00000000..a9ff1924 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x28.go @@ -0,0 +1,46 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "slices" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func only(xs, other []string) []string { + out := []string{} + for _, x := range xs { + if !slices.Contains(other, x) && !slices.Contains(out, x) { + out = append(out, x) + } + } + return out +} + +func main() { + var a, b []string + readJSON("in/a.json", &a) + readJSON("in/b.json", &b) + writeJSON(map[string][]string{"only_a": only(a, b), "only_b": only(b, a)}) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x29.go b/experiments/token-efficiency-heldout-v3/solutions/go/x29.go new file mode 100644 index 00000000..d8e057fb --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x29.go @@ -0,0 +1,41 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "slices" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var v []string + readJSON("in/visits.json", &v) + out := []string{} + for i, x := range v { + if !slices.Contains(v[i+1:], x) { + out = append(out, x) + } + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x30.go b/experiments/token-efficiency-heldout-v3/solutions/go/x30.go new file mode 100644 index 00000000..a3136bd8 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x30.go @@ -0,0 +1,41 @@ +//go:build ignore + +package main + +import ( + "os" + "slices" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + key := func(w string) string { + r := []rune(w) + slices.Sort(r) + return string(r) + } + var out []string + for _, line := range strings.Split(read("in/pairs.txt"), "\n") { + a, b, _ := strings.Cut(line, " ") + if key(a) == key(b) { + out = append(out, "yes") + } else { + out = append(out, "no") + } + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x31.go b/experiments/token-efficiency-heldout-v3/solutions/go/x31.go new file mode 100644 index 00000000..815f475d --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x31.go @@ -0,0 +1,38 @@ +//go:build ignore + +package main + +import ( + "os" + "strconv" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func main() { + var out []string + for _, item := range strings.Split(read("in/ranges.txt"), ",") { + a, b, ok := strings.Cut(item, "-") + if !ok { + b = a + } + lo, err1 := strconv.Atoi(a) + hi, err2 := strconv.Atoi(b) + if err1 != nil || err2 != nil { + panic(item) + } + for i := lo; i <= hi; i++ { + out = append(out, strconv.Itoa(i)) + } + } + if err := os.WriteFile("out/result.txt", []byte(strings.Join(out, ",")+"\n"), 0o644); err != nil { + panic(err) + } +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x32.go b/experiments/token-efficiency-heldout-v3/solutions/go/x32.go new file mode 100644 index 00000000..c60238ef --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x32.go @@ -0,0 +1,29 @@ +//go:build ignore + +package main + +import ( + "os" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func main() { + words := strings.Split(read("in/words.txt"), "\n") + p := words[0] + for _, w := range words { + for !strings.HasPrefix(w, p) { + p = p[:len(p)-1] + } + } + if err := os.WriteFile("out/result.txt", []byte(p+"\n"), 0o644); err != nil { + panic(err) + } +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x33.go b/experiments/token-efficiency-heldout-v3/solutions/go/x33.go new file mode 100644 index 00000000..2fb50607 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x33.go @@ -0,0 +1,49 @@ +//go:build ignore + +package main + +import ( + "fmt" + "os" + "sort" + "strings" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var spans [][2]int + for _, line := range strings.Split(read("in/spans.txt"), "\n") { + var a, b int + if _, err := fmt.Sscanf(line, "%d-%d", &a, &b); err != nil { + panic(err) + } + spans = append(spans, [2]int{a, b}) + } + sort.Slice(spans, func(i, j int) bool { return spans[i][0] < spans[j][0] }) + var merged [][2]int + for _, s := range spans { + if n := len(merged); n > 0 && s[0] <= merged[n-1][1] { + merged[n-1][1] = max(merged[n-1][1], s[1]) + } else { + merged = append(merged, s) + } + } + var out []string + for _, m := range merged { + out = append(out, fmt.Sprintf("%d-%d", m[0], m[1])) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x34.go b/experiments/token-efficiency-heldout-v3/solutions/go/x34.go new file mode 100644 index 00000000..c851c4b2 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x34.go @@ -0,0 +1,36 @@ +//go:build ignore + +package main + +import ( + "os" + "strconv" + "strings" + "time" +) + +func read(path string) string { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + return string(data) +} + +func writeLines(lines []string) { + if err := os.WriteFile("out/result.txt", []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + panic(err) + } +} + +func main() { + var out []string + for _, line := range strings.Split(read("in/days.txt"), "\n") { + d, err := strconv.Atoi(line) + if err != nil { + panic(err) + } + out = append(out, time.Date(2021, 1, d, 0, 0, 0, 0, time.UTC).Format("01-02")) + } + writeLines(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/go/x35.go b/experiments/token-efficiency-heldout-v3/solutions/go/x35.go new file mode 100644 index 00000000..fe9cfadf --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/go/x35.go @@ -0,0 +1,54 @@ +//go:build ignore + +package main + +import ( + "encoding/json" + "os" + "slices" + "sort" +) + +func readJSON(path string, v any) { + data, err := os.ReadFile(path) + if err != nil { + panic(err) + } + if err := json.Unmarshal(data, v); err != nil { + panic(err) + } +} + +func writeJSON(v any) { + enc, err := json.MarshalIndent(v, "", " ") + if err != nil { + panic(err) + } + if err := os.WriteFile("out/result.json", append(enc, '\n'), 0o644); err != nil { + panic(err) + } +} + +func main() { + var deps map[string][]string + readJSON("in/deps.json", &deps) + names := make([]string, 0, len(deps)) + for t := range deps { + names = append(names, t) + } + sort.Strings(names) + out := []string{} + for len(out) < len(names) { + for _, t := range names { + ready := !slices.Contains(out, t) + for _, d := range deps[t] { + ready = ready && slices.Contains(out, d) + } + if ready { + out = append(out, t) + break + } + } + } + writeJSON(out) +} diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x01.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x01.howl new file mode 100644 index 00000000..9a9fc74e --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x01.howl @@ -0,0 +1,11 @@ +(do + (let (cfg (parse_json Any (read_text "in/config.json"))) + (write_file "out/result.txt" + (str_join + (map_list p (lines (read_file "in/paths.txt")) + (let (v cfg ok true) + (do + (for k (str_split p ".") + (if (and (starts_with (encode_json v) "{") (map_has v k)) (set v (map_get v k)) (do (set ok false) (break)))) + (if ok (encode_json v) "missing")))) + "\n")))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x02.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x02.howl new file mode 100644 index 00000000..e8fb1430 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x02.howl @@ -0,0 +1,9 @@ +(do + (defun merge ((a any) (b any)) any + (do + (for k (map_keys b) + (let (x (map_get a k) y (map_get b k)) + (map_set a k (if (and (starts_with (encode_json x) "{") (starts_with (encode_json y) "{")) (call merge x y) y)))) + (return a))) + (write_file "out/result.json" + (encode_json (call merge (parse_json Any (read_text "in/base.json")) (parse_json Any (read_text "in/override.json")))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x03.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x03.howl new file mode 100644 index 00000000..ee218ed1 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x03.howl @@ -0,0 +1,8 @@ +(do + (defun flat ((d any) (pre string) (out any)) any + (do + (for k (map_keys d) + (let (v (map_get d k)) + (if (and (starts_with (encode_json v) "{") (!= (encode_json v) "{}")) (call flat v (str pre k ".") out) (map_set out (str pre k) v)))) + (return out))) + (write_file "out/result.json" (encode_json (call flat (parse_json Any (read_text "in/nested.json")) "" (dict))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x04.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x04.howl new file mode 100644 index 00000000..9eaa45c0 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x04.howl @@ -0,0 +1,10 @@ +(do + (let (f (parse_json Any (read_text "in/flat.json")) out (dict)) + (do + (for k (map_keys f) + (let (ps (str_split k ".") n (list_len ps) node out) + (do + (for p (list_slice ps 0 (- n 1)) + (do (when (is_nil (map_get node p)) (map_set node p (dict))) (set node (map_get node p)))) + (map_set node (list_get ps (- n 1)) (map_get f k))))) + (write_file "out/result.json" (encode_json out))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x05.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x05.howl new file mode 100644 index 00000000..1432b401 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x05.howl @@ -0,0 +1,7 @@ +(do + (let (out (dict ("present" (list)) ("null" (list)) ("absent" (list)))) + (do + (for c (parse_json Any (read_text "in/contacts.json")) + (let (k (if (not (map_has c "email")) "absent" (is_nil (map_get c "email")) "null" "present") l (map_get out k)) + (do (append l (map_get c "id")) (map_set out k l)))) + (write_file "out/result.json" (encode_json out))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x06.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x06.howl new file mode 100644 index 00000000..75b56874 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x06.howl @@ -0,0 +1,11 @@ +(do + (let (out (dict)) + (do + (for s (parse_json Any (read_text "in/sales.json")) + (let (r (nz (map_get out (map_get s "region")) (dict ("units" 0) ("products" (list)))) ps (map_get r "products")) + (do + (map_set r "units" (+ (map_get r "units") (map_get s "units"))) + (when (not (list_contains ps (map_get s "product"))) (append ps (map_get s "product"))) + (map_set r "products" (sort ps)) + (map_set out (map_get s "region") r)))) + (write_file "out/result.json" (encode_json out))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x07.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x07.howl new file mode 100644 index 00000000..cc32de58 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x07.howl @@ -0,0 +1,4 @@ +(do + (let (ren (parse_json Any (read_text "in/rename.json"))) + (write_file "out/result.json" (encode_json (map_list r (parse_json Any (read_text "in/rows.json")) + (let (o (dict)) (do (for k (map_keys r) (map_set o (nz (map_get ren k) k) (map_get r k))) o))))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x08.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x08.howl new file mode 100644 index 00000000..d081d01a --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x08.howl @@ -0,0 +1 @@ +(do (write_file "out/result.txt" (encode_json (parse_json Any (read_text "in/doc.json"))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x09.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x09.howl new file mode 100644 index 00000000..01df4bcd --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x09.howl @@ -0,0 +1,6 @@ +(do + (let (o (parse_json Any (read_text "in/old.json")) n (parse_json Any (read_text "in/new.json"))) + (write_file "out/result.json" (encode_json (dict + ("added" (filter k (map_keys n) (not (map_has o k)))) + ("removed" (filter k (map_keys o) (not (map_has n k)))) + ("changed" (filter k (map_keys o) (and (map_has n k) (!= (encode_json (map_get o k)) (encode_json (map_get n k))))))))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x10.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x10.howl new file mode 100644 index 00000000..21807bb8 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x10.howl @@ -0,0 +1,5 @@ +(do + (write_file "out/result.txt" + (str_join + (map_list t (filter t (parse_json Any (read_text "in/tickets.json")) (list_contains (nz (map_get (map_get t "meta") "tags") (list)) "urgent")) (map_get t "id")) + "\n"))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x11.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x11.howl new file mode 100644 index 00000000..3e32ba57 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x11.howl @@ -0,0 +1,7 @@ +(do + (write_file "out/result.txt" + (str_join + (map_list a (parse_json Any (read_text "in/accounts.json")) + (let (p (map_get (map_get a "user") "profile")) + (str (map_get a "login") ": " (if (is_nil p) "unknown" (nz (map_get p "city") "unknown"))))) + "\n"))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x12.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x12.howl new file mode 100644 index 00000000..ccc71522 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x12.howl @@ -0,0 +1,5 @@ +(do + (let (out (dict)) + (do + (for p (parse_json Any (read_text "in/people.json")) (map_set out (str (map_get p "last") ", " (map_get p "first")) (map_get p "age"))) + (write_file "out/result.json" (encode_json out))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x13.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x13.howl new file mode 100644 index 00000000..09c89eed --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x13.howl @@ -0,0 +1,7 @@ +(do + (let (out (dict)) + (do + (for l (lines (read_file "in/requests.jsonl")) + (let (r (parse_json Any l) lv (map_get r "level") ms (map_get r "ms")) + (map_set out lv (max (list ms (nz (map_get out lv) ms)))))) + (write_file "out/result.json" (encode_json out))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x14.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x14.howl new file mode 100644 index 00000000..ed245dcf --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x14.howl @@ -0,0 +1,6 @@ +(do + (let (p (parse_json Any (read_text "in/prices.json")) out (dict)) + (do + (for k (map_keys p) + (let (v (map_get p k)) (map_set out k (str "$" (// v 100) "." (pad_left (% v 100) 2 "0"))))) + (write_file "out/result.json" (encode_json out))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x15.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x15.howl new file mode 100644 index 00000000..2b7eb24a --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x15.howl @@ -0,0 +1,9 @@ +(do + (defun dep ((v any)) int + (let (e (encode_json v) m 0) + (if (or (starts_with e "{") (starts_with e "[")) + (do + (for x (if (starts_with e "{") (map_list k (map_keys v) (map_get v k)) v) (set m (max (list m (call dep x))))) + (return (+ m 1))) + (return 0)))) + (write_file "out/result.txt" (str (call dep (parse_json Any (read_text "in/tree.json")))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x16.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x16.howl new file mode 100644 index 00000000..1e9bbb8e --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x16.howl @@ -0,0 +1,3 @@ +(do + (let (e (parse_json Any (read_text "in/env.json"))) + (write_file "out/result.json" (encode_json (map_list k (sort (map_keys e)) (dict ("key" k) ("value" (map_get e k)))))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x17.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x17.howl new file mode 100644 index 00000000..d04a624f --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x17.howl @@ -0,0 +1,10 @@ +(do + (let (doc (parse_json Any (read_text "in/doc.json"))) + (do + (for op (parse_json Any (read_text "in/ops.json")) + (let (ps (str_split (map_get op "path") ".") n (list_len ps) node doc k (list_get ps (- n 1))) + (do + (for p (list_slice ps 0 (- n 1)) + (do (when (is_nil (map_get node p)) (map_set node p (dict))) (set node (map_get node p)))) + (if (= (map_get op "op") "set") (map_set node k (map_get op "value")) (map_delete node k))))) + (write_file "out/result.json" (encode_json doc))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x18.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x18.howl new file mode 100644 index 00000000..ef8b21a4 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x18.howl @@ -0,0 +1,6 @@ +(do + (let (out (list) cur "") + (do + (for l (lines (read_file "in/script.txt")) + (if (ends_with l "\\") (set cur (str cur (str_slice l 0 (- (str_len l) 1)))) (do (append out (str cur l)) (set cur "")))) + (write_file "out/result.txt" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x19.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x19.howl new file mode 100644 index 00000000..fdcc5600 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x19.howl @@ -0,0 +1,6 @@ +(do + (let (out (list) in false) + (do + (for l (lines (read_file "in/app.ini")) + (if (starts_with l "[") (set in (= l "[net]")) (when (and in (!= l "")) (append out l)))) + (write_file "out/result.txt" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x20.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x20.howl new file mode 100644 index 00000000..818e95c9 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x20.howl @@ -0,0 +1,8 @@ +(do + (let (out (list) w 0) + (do + (for l (str_split (str (read_text "in/doc.txt") "\n") "\n") + (if (= (str_trim l) "") + (when (> w 0) (do (append out (str (+ (list_len out) 1) ": " w)) (set w 0))) + (set w (+ w (list_len (filter x (str_split l " ") (!= x ""))))))) + (write_file "out/result.txt" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x21.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x21.howl new file mode 100644 index 00000000..52ddfb95 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x21.howl @@ -0,0 +1,4 @@ +(do + (let (rows (map_list l (lines (read_file "in/grid.csv")) (str_split l ","))) + (write_file "out/result.csv" + (str_join (map_list j (range (list_len (list_get rows 0))) (str_join (map_list r rows (list_get r j)) ",")) "\n")))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x22.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x22.howl new file mode 100644 index 00000000..d1f35298 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x22.howl @@ -0,0 +1,9 @@ +(do + (let (ls (lines (read_file "in/sites.csv")) prev (str_split (list_get ls 1) ",") out (list (list_get ls 0))) + (do + (for l (list_slice ls 1) + (let (c (str_split l ",") i 0) + (do + (set prev (map_list x c (do (inc i) (if (= x "") (list_get prev (- i 1)) x)))) + (append out (str_join prev ","))))) + (write_file "out/result.csv" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x23.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x23.howl new file mode 100644 index 00000000..3bd8b8a9 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x23.howl @@ -0,0 +1,6 @@ +(do + (let (d (dict) out (list "id,name,dept")) + (do + (for l (list_slice (lines (read_file "in/dept.csv")) 1) (let ((i n) (str_split l ",")) (map_set d i n))) + (for l (list_slice (lines (read_file "in/emp.csv")) 1) (append out (str l "," (nz (map_get d (list_get (str_split l ",") 0)) "")))) + (write_file "out/result.csv" (str_join out "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x24.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x24.howl new file mode 100644 index 00000000..3aa6bdca --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x24.howl @@ -0,0 +1,10 @@ +(do + (let (g 0 m 1) + (do + (for s (str_split (read_text "in/vals.txt") ",") + (let (n (to_int s) a g b n) + (do + (while (!= b 0) (let (t (% a b)) (do (set a b) (set b t)))) + (set g a) + (let (x m y n) (do (while (!= y 0) (let (t (% x y)) (do (set x y) (set y t)))) (set m (* (// m x) n))))))) + (write_file "out/result.json" (encode_json (dict ("gcd" g) ("lcm" m))))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x25.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x25.howl new file mode 100644 index 00000000..552e00e5 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x25.howl @@ -0,0 +1,6 @@ +(do + (write_file "out/result.txt" + (str_join + (map_list n (lines (read_file "in/nums.txt")) + (let (d n) (do (while (> (str_len d) 1) (set d (str (sum (map_list c (str_split d "") (to_int c)))))) (str n " -> " d)))) + "\n"))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x26.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x26.howl new file mode 100644 index 00000000..e51beefd --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x26.howl @@ -0,0 +1,4 @@ +(do + (let (rows (map_list l (lines (read_file "in/votes.txt")) (str_split l " ")) t (sum (map_list r rows (to_int (list_get r 1))))) + (write_file "out/result.txt" + (str_join (map_list r rows (str (list_get r 0) " " (// (* (to_int (list_get r 1)) 100) t) "%")) "\n")))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x27.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x27.howl new file mode 100644 index 00000000..4a962f4f --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x27.howl @@ -0,0 +1,6 @@ +(do + (defun flat ((v any)) any + (if (starts_with (encode_json v) "[") + (let (o (list)) (do (for e v (for x (call flat e) (append o x))) (return o))) + (return (list v)))) + (write_file "out/result.json" (encode_json (call flat (parse_json Any (read_text "in/deep.json")))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x28.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x28.howl new file mode 100644 index 00000000..31fba5cd --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x28.howl @@ -0,0 +1,6 @@ +(do + (let (a (parse_json Any (read_text "in/a.json")) b (parse_json Any (read_text "in/b.json")) oa (list) ob (list)) + (do + (for x a (when (not (or (list_contains b x) (list_contains oa x))) (append oa x))) + (for x b (when (not (or (list_contains a x) (list_contains ob x))) (append ob x))) + (write_file "out/result.json" (encode_json (dict ("only_a" oa) ("only_b" ob))))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x29.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x29.howl new file mode 100644 index 00000000..30414f66 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x29.howl @@ -0,0 +1,3 @@ +(do + (let (v (parse_json Any (read_text "in/visits.json")) i 0) + (write_file "out/result.json" (encode_json (filter x v (do (inc i) (not (list_contains (list_slice v i) x)))))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x30.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x30.howl new file mode 100644 index 00000000..ce94d28f --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x30.howl @@ -0,0 +1,6 @@ +(do + (write_file "out/result.txt" + (str_join + (map_list l (lines (read_file "in/pairs.txt")) + (let ((a b) (str_split l " ")) (if (= (str_join (sort (str_split a "")) "") (str_join (sort (str_split b "")) "")) "yes" "no"))) + "\n"))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x31.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x31.howl new file mode 100644 index 00000000..82ab5624 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x31.howl @@ -0,0 +1,6 @@ +(do + (let (out (list)) + (do + (for it (str_split (read_text "in/ranges.txt") ",") + (let (p (str_split it "-")) (for i (range (to_int (list_get p 0)) (+ (to_int (list_get p (- (list_len p) 1))) 1)) (append out i)))) + (write_file "out/result.txt" (str_join (map_list i out (str i)) ","))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x32.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x32.howl new file mode 100644 index 00000000..0da4d108 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x32.howl @@ -0,0 +1,5 @@ +(do + (let (ws (lines (read_file "in/words.txt")) p (list_get ws 0)) + (do + (for w ws (while (not (starts_with w p)) (set p (str_slice p 0 (- (str_len p) 1))))) + (write_file "out/result.txt" p)))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x33.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x33.howl new file mode 100644 index 00000000..1f6fa783 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x33.howl @@ -0,0 +1,9 @@ +(do + (let (as (list) bs (list)) + (do + (for s (sort_by s (map_list l (lines (read_file "in/spans.txt")) (map_list x (str_split l "-") (to_int x))) (list_get s 0)) + (let (n (list_len bs) (a b) s) + (if (and (> n 0) (<= a (list_get bs (- n 1)))) + (let (e (max (list b (list_get bs (- n 1))))) (do (set bs (list_slice bs 0 (- n 1))) (append bs e))) + (do (append as a) (append bs b))))) + (write_file "out/result.txt" (str_join (map_list i (range (list_len as)) (str (list_get as i) "-" (list_get bs i))) "\n"))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x34.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x34.howl new file mode 100644 index 00000000..00b347dd --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x34.howl @@ -0,0 +1,10 @@ +(do + (let (ml (list 31 28 31 30 31 30 31 31 30 31 30 31)) + (write_file "out/result.txt" + (str_join + (map_list l (lines (read_file "in/days.txt")) + (let (d (to_int l) m 0) + (do + (while (> d (list_get ml m)) (do (set d (- d (list_get ml m))) (inc m))) + (str (pad_left (+ m 1) 2 "0") "-" (pad_left d 2 "0"))))) + "\n")))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/howl/x35.howl b/experiments/token-efficiency-heldout-v3/solutions/howl/x35.howl new file mode 100644 index 00000000..2df18ce2 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/howl/x35.howl @@ -0,0 +1,7 @@ +(do + (let (deps (parse_json Any (read_text "in/deps.json")) ks (sort (map_keys deps)) out (list)) + (do + (while (< (list_len out) (list_len ks)) + (for t ks + (when (and (not (list_contains out t)) (all d (map_get deps t) (list_contains out d))) (do (append out t) (break))))) + (write_file "out/result.json" (encode_json out))))) diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x01.js b/experiments/token-efficiency-heldout-v3/solutions/js/x01.js new file mode 100644 index 00000000..ae29c859 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x01.js @@ -0,0 +1,13 @@ +const fs = require("fs"); + +const isObj = (v) => v !== null && typeof v === "object" && !Array.isArray(v); +const cfg = JSON.parse(fs.readFileSync("in/config.json", "utf8")); +const out = fs.readFileSync("in/paths.txt", "utf8").trim().split("\n").map((line) => { + let v = cfg; + for (const k of line.split(".")) { + if (!isObj(v) || !(k in v)) return "missing"; + v = v[k]; + } + return JSON.stringify(v); +}); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x02.js b/experiments/token-efficiency-heldout-v3/solutions/js/x02.js new file mode 100644 index 00000000..e4c23753 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x02.js @@ -0,0 +1,9 @@ +const fs = require("fs"); + +const isObj = (v) => v !== null && typeof v === "object" && !Array.isArray(v); +function merge(a, b) { + for (const [k, v] of Object.entries(b)) a[k] = isObj(a[k]) && isObj(v) ? merge(a[k], v) : v; + return a; +} +const out = merge(JSON.parse(fs.readFileSync("in/base.json", "utf8")), JSON.parse(fs.readFileSync("in/override.json", "utf8"))); +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x03.js b/experiments/token-efficiency-heldout-v3/solutions/js/x03.js new file mode 100644 index 00000000..fbda7bec --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x03.js @@ -0,0 +1,12 @@ +const fs = require("fs"); + +const isObj = (v) => v !== null && typeof v === "object" && !Array.isArray(v); +function flat(d, pre, out) { + for (const [k, v] of Object.entries(d)) { + if (isObj(v) && Object.keys(v).length) flat(v, pre + k + ".", out); + else out[pre + k] = v; + } + return out; +} +const out = flat(JSON.parse(fs.readFileSync("in/nested.json", "utf8")), "", {}); +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x04.js b/experiments/token-efficiency-heldout-v3/solutions/js/x04.js new file mode 100644 index 00000000..a3992d73 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x04.js @@ -0,0 +1,11 @@ +const fs = require("fs"); + +const out = {}; +for (const [k, v] of Object.entries(JSON.parse(fs.readFileSync("in/flat.json", "utf8")))) { + const parts = k.split("."); + const last = parts.pop(); + let node = out; + for (const p of parts) node = node[p] ||= {}; + node[last] = v; +} +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x05.js b/experiments/token-efficiency-heldout-v3/solutions/js/x05.js new file mode 100644 index 00000000..6b8a554d --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x05.js @@ -0,0 +1,7 @@ +const fs = require("fs"); + +const out = { present: [], null: [], absent: [] }; +for (const c of JSON.parse(fs.readFileSync("in/contacts.json", "utf8"))) { + out[!("email" in c) ? "absent" : c.email === null ? "null" : "present"].push(c.id); +} +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x06.js b/experiments/token-efficiency-heldout-v3/solutions/js/x06.js new file mode 100644 index 00000000..63b9735d --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x06.js @@ -0,0 +1,10 @@ +const fs = require("fs"); + +const out = {}; +for (const s of JSON.parse(fs.readFileSync("in/sales.json", "utf8"))) { + const r = (out[s.region] ||= { units: 0, products: [] }); + r.units += s.units; + if (!r.products.includes(s.product)) r.products.push(s.product); +} +for (const r of Object.values(out)) r.products.sort(); +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x07.js b/experiments/token-efficiency-heldout-v3/solutions/js/x07.js new file mode 100644 index 00000000..3c35de87 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x07.js @@ -0,0 +1,6 @@ +const fs = require("fs"); + +const ren = JSON.parse(fs.readFileSync("in/rename.json", "utf8")); +const out = JSON.parse(fs.readFileSync("in/rows.json", "utf8")).map((r) => + Object.fromEntries(Object.entries(r).map(([k, v]) => [ren[k] ?? k, v]))); +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x08.js b/experiments/token-efficiency-heldout-v3/solutions/js/x08.js new file mode 100644 index 00000000..1d1d9f6a --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x08.js @@ -0,0 +1,8 @@ +const fs = require("fs"); + +const canon = (v) => + Array.isArray(v) ? `[${v.map(canon).join(",")}]` + : v !== null && typeof v === "object" + ? `{${Object.keys(v).sort().map((k) => JSON.stringify(k) + ":" + canon(v[k])).join(",")}}` + : JSON.stringify(v); +fs.writeFileSync("out/result.txt", canon(JSON.parse(fs.readFileSync("in/doc.json", "utf8"))) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x09.js b/experiments/token-efficiency-heldout-v3/solutions/js/x09.js new file mode 100644 index 00000000..24db0865 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x09.js @@ -0,0 +1,10 @@ +const fs = require("fs"); + +const old = JSON.parse(fs.readFileSync("in/old.json", "utf8")); +const nw = JSON.parse(fs.readFileSync("in/new.json", "utf8")); +const out = { + added: Object.keys(nw).filter((k) => !(k in old)).sort(), + removed: Object.keys(old).filter((k) => !(k in nw)).sort(), + changed: Object.keys(old).filter((k) => k in nw && JSON.stringify(old[k]) !== JSON.stringify(nw[k])).sort(), +}; +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x10.js b/experiments/token-efficiency-heldout-v3/solutions/js/x10.js new file mode 100644 index 00000000..271e32ca --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x10.js @@ -0,0 +1,4 @@ +const fs = require("fs"); + +const out = JSON.parse(fs.readFileSync("in/tickets.json", "utf8")).filter((t) => (t.meta.tags || []).includes("urgent")).map((t) => t.id); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x11.js b/experiments/token-efficiency-heldout-v3/solutions/js/x11.js new file mode 100644 index 00000000..79b8a9f9 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x11.js @@ -0,0 +1,4 @@ +const fs = require("fs"); + +const out = JSON.parse(fs.readFileSync("in/accounts.json", "utf8")).map((a) => `${a.login}: ${a.user.profile?.city ?? "unknown"}`); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x12.js b/experiments/token-efficiency-heldout-v3/solutions/js/x12.js new file mode 100644 index 00000000..99c37d8c --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x12.js @@ -0,0 +1,5 @@ +const fs = require("fs"); + +const out = {}; +for (const p of JSON.parse(fs.readFileSync("in/people.json", "utf8"))) out[`${p.last}, ${p.first}`] = p.age; +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x13.js b/experiments/token-efficiency-heldout-v3/solutions/js/x13.js new file mode 100644 index 00000000..e414a7b3 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x13.js @@ -0,0 +1,8 @@ +const fs = require("fs"); + +const out = {}; +for (const line of fs.readFileSync("in/requests.jsonl", "utf8").trim().split("\n")) { + const r = JSON.parse(line); + out[r.level] = Math.max(out[r.level] ?? r.ms, r.ms); +} +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x14.js b/experiments/token-efficiency-heldout-v3/solutions/js/x14.js new file mode 100644 index 00000000..04bd6fff --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x14.js @@ -0,0 +1,7 @@ +const fs = require("fs"); + +const out = {}; +for (const [k, v] of Object.entries(JSON.parse(fs.readFileSync("in/prices.json", "utf8")))) { + out[k] = `$${Math.floor(v / 100)}.${String(v % 100).padStart(2, "0")}`; +} +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x15.js b/experiments/token-efficiency-heldout-v3/solutions/js/x15.js new file mode 100644 index 00000000..7e2a14c3 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x15.js @@ -0,0 +1,5 @@ +const fs = require("fs"); + +const depth = (v) => + v !== null && typeof v === "object" ? 1 + Math.max(0, ...Object.values(v).map(depth)) : 0; +fs.writeFileSync("out/result.txt", depth(JSON.parse(fs.readFileSync("in/tree.json", "utf8"))) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x16.js b/experiments/token-efficiency-heldout-v3/solutions/js/x16.js new file mode 100644 index 00000000..5f4f2396 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x16.js @@ -0,0 +1,5 @@ +const fs = require("fs"); + +const env = JSON.parse(fs.readFileSync("in/env.json", "utf8")); +const out = Object.keys(env).sort().map((k) => ({ key: k, value: env[k] })); +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x17.js b/experiments/token-efficiency-heldout-v3/solutions/js/x17.js new file mode 100644 index 00000000..d4ca6759 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x17.js @@ -0,0 +1,12 @@ +const fs = require("fs"); + +const out = JSON.parse(fs.readFileSync("in/doc.json", "utf8")); +for (const op of JSON.parse(fs.readFileSync("in/ops.json", "utf8"))) { + const parts = op.path.split("."); + const last = parts.pop(); + let node = out; + for (const p of parts) node = node[p] ||= {}; + if (op.op === "set") node[last] = op.value; + else delete node[last]; +} +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x18.js b/experiments/token-efficiency-heldout-v3/solutions/js/x18.js new file mode 100644 index 00000000..8d61f87e --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x18.js @@ -0,0 +1,9 @@ +const fs = require("fs"); + +const out = []; +let cur = ""; +for (const line of fs.readFileSync("in/script.txt", "utf8").trimEnd().split("\n")) { + if (line.endsWith("\\")) cur += line.slice(0, -1); + else { out.push(cur + line); cur = ""; } +} +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x19.js b/experiments/token-efficiency-heldout-v3/solutions/js/x19.js new file mode 100644 index 00000000..cad079fb --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x19.js @@ -0,0 +1,9 @@ +const fs = require("fs"); + +const out = []; +let inside = false; +for (const line of fs.readFileSync("in/app.ini", "utf8").split("\n")) { + if (line.startsWith("[")) inside = line === "[net]"; + else if (inside && line) out.push(line); +} +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x20.js b/experiments/token-efficiency-heldout-v3/solutions/js/x20.js new file mode 100644 index 00000000..a31deb6a --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x20.js @@ -0,0 +1,5 @@ +const fs = require("fs"); + +const out = fs.readFileSync("in/doc.txt", "utf8").split(/\n\s*\n/).filter((p) => p.trim()) + .map((p, i) => `${i + 1}: ${p.trim().split(/\s+/).length}`); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x21.js b/experiments/token-efficiency-heldout-v3/solutions/js/x21.js new file mode 100644 index 00000000..3a485794 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x21.js @@ -0,0 +1,5 @@ +const fs = require("fs"); + +const rows = fs.readFileSync("in/grid.csv", "utf8").trim().split("\n").map((l) => l.split(",")); +const out = rows[0].map((_, j) => rows.map((r) => r[j]).join(",")); +fs.writeFileSync("out/result.csv", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x22.js b/experiments/token-efficiency-heldout-v3/solutions/js/x22.js new file mode 100644 index 00000000..58583e03 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x22.js @@ -0,0 +1,10 @@ +const fs = require("fs"); + +const [head, ...lines] = fs.readFileSync("in/sites.csv", "utf8").trimEnd().split("\n"); +const out = [head]; +let prev = null; +for (const line of lines) { + prev = line.split(",").map((c, j) => c || prev[j]); + out.push(prev.join(",")); +} +fs.writeFileSync("out/result.csv", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x23.js b/experiments/token-efficiency-heldout-v3/solutions/js/x23.js new file mode 100644 index 00000000..3cb381a5 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x23.js @@ -0,0 +1,6 @@ +const fs = require("fs"); + +const rows = (p) => fs.readFileSync(p, "utf8").trim().split("\n").slice(1); +const dept = Object.fromEntries(rows("in/dept.csv").map((l) => l.split(","))); +const out = ["id,name,dept", ...rows("in/emp.csv").map((l) => `${l},${dept[l.split(",")[0]] ?? ""}`)]; +fs.writeFileSync("out/result.csv", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x24.js b/experiments/token-efficiency-heldout-v3/solutions/js/x24.js new file mode 100644 index 00000000..f1c360f6 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x24.js @@ -0,0 +1,6 @@ +const fs = require("fs"); + +const gcd = (a, b) => (b ? gcd(b, a % b) : a); +const vals = fs.readFileSync("in/vals.txt", "utf8").trim().split(",").map(Number); +const out = { gcd: vals.reduce(gcd), lcm: vals.reduce((a, b) => (a / gcd(a, b)) * b) }; +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x25.js b/experiments/token-efficiency-heldout-v3/solutions/js/x25.js new file mode 100644 index 00000000..61d1e784 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x25.js @@ -0,0 +1,8 @@ +const fs = require("fs"); + +const out = fs.readFileSync("in/nums.txt", "utf8").trim().split("\n").map((n) => { + let d = n; + while (d.length > 1) d = String([...d].reduce((s, c) => s + Number(c), 0)); + return `${n} -> ${d}`; +}); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x26.js b/experiments/token-efficiency-heldout-v3/solutions/js/x26.js new file mode 100644 index 00000000..26b83e69 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x26.js @@ -0,0 +1,6 @@ +const fs = require("fs"); + +const rows = fs.readFileSync("in/votes.txt", "utf8").trim().split("\n").map((l) => l.split(" ")); +const total = rows.reduce((s, [, c]) => s + Number(c), 0); +const out = rows.map(([n, c]) => `${n} ${Math.floor((Number(c) * 100) / total)}%`); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x27.js b/experiments/token-efficiency-heldout-v3/solutions/js/x27.js new file mode 100644 index 00000000..aa24cf95 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x27.js @@ -0,0 +1,4 @@ +const fs = require("fs"); + +const out = JSON.parse(fs.readFileSync("in/deep.json", "utf8")).flat(Infinity); +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x28.js b/experiments/token-efficiency-heldout-v3/solutions/js/x28.js new file mode 100644 index 00000000..01c8baea --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x28.js @@ -0,0 +1,6 @@ +const fs = require("fs"); + +const a = JSON.parse(fs.readFileSync("in/a.json", "utf8")); +const b = JSON.parse(fs.readFileSync("in/b.json", "utf8")); +const out = { only_a: [...new Set(a.filter((x) => !b.includes(x)))], only_b: [...new Set(b.filter((x) => !a.includes(x)))] }; +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x29.js b/experiments/token-efficiency-heldout-v3/solutions/js/x29.js new file mode 100644 index 00000000..3d6ee84b --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x29.js @@ -0,0 +1,5 @@ +const fs = require("fs"); + +const v = JSON.parse(fs.readFileSync("in/visits.json", "utf8")); +const out = v.filter((x, i) => v.lastIndexOf(x) === i); +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x30.js b/experiments/token-efficiency-heldout-v3/solutions/js/x30.js new file mode 100644 index 00000000..75c7374a --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x30.js @@ -0,0 +1,8 @@ +const fs = require("fs"); + +const key = (w) => [...w].sort().join(""); +const out = fs.readFileSync("in/pairs.txt", "utf8").trim().split("\n").map((l) => { + const [a, b] = l.split(" "); + return key(a) === key(b) ? "yes" : "no"; +}); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x31.js b/experiments/token-efficiency-heldout-v3/solutions/js/x31.js new file mode 100644 index 00000000..4764d7dd --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x31.js @@ -0,0 +1,8 @@ +const fs = require("fs"); + +const out = []; +for (const item of fs.readFileSync("in/ranges.txt", "utf8").trim().split(",")) { + const [a, b = a] = item.split("-").map(Number); + for (let i = a; i <= b; i++) out.push(i); +} +fs.writeFileSync("out/result.txt", out.join(",") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x32.js b/experiments/token-efficiency-heldout-v3/solutions/js/x32.js new file mode 100644 index 00000000..aeadc897 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x32.js @@ -0,0 +1,6 @@ +const fs = require("fs"); + +const words = fs.readFileSync("in/words.txt", "utf8").trim().split("\n"); +let p = words[0]; +for (const w of words) while (!w.startsWith(p)) p = p.slice(0, -1); +fs.writeFileSync("out/result.txt", p + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x33.js b/experiments/token-efficiency-heldout-v3/solutions/js/x33.js new file mode 100644 index 00000000..bee22169 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x33.js @@ -0,0 +1,12 @@ +const fs = require("fs"); + +const spans = fs.readFileSync("in/spans.txt", "utf8").trim().split("\n") + .map((l) => l.split("-").map(Number)).sort((x, y) => x[0] - y[0]); +const merged = []; +for (const [a, b] of spans) { + const last = merged[merged.length - 1]; + if (last && a <= last[1]) last[1] = Math.max(last[1], b); + else merged.push([a, b]); +} +const out = merged.map(([a, b]) => `${a}-${b}`); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x34.js b/experiments/token-efficiency-heldout-v3/solutions/js/x34.js new file mode 100644 index 00000000..42126d92 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x34.js @@ -0,0 +1,7 @@ +const fs = require("fs"); + +const out = fs.readFileSync("in/days.txt", "utf8").trim().split("\n").map((d) => { + const t = new Date(Date.UTC(2021, 0, Number(d))); + return `${String(t.getUTCMonth() + 1).padStart(2, "0")}-${String(t.getUTCDate()).padStart(2, "0")}`; +}); +fs.writeFileSync("out/result.txt", out.join("\n") + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/js/x35.js b/experiments/token-efficiency-heldout-v3/solutions/js/x35.js new file mode 100644 index 00000000..80cf850d --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/js/x35.js @@ -0,0 +1,8 @@ +const fs = require("fs"); + +const deps = JSON.parse(fs.readFileSync("in/deps.json", "utf8")); +const out = []; +while (out.length < Object.keys(deps).length) { + out.push(Object.keys(deps).filter((t) => !out.includes(t) && deps[t].every((d) => out.includes(d))).sort()[0]); +} +fs.writeFileSync("out/result.json", JSON.stringify(out, null, 2) + "\n"); diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x01.py b/experiments/token-efficiency-heldout-v3/solutions/python/x01.py new file mode 100644 index 00000000..a0e13bd7 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x01.py @@ -0,0 +1,12 @@ +import json +from pathlib import Path + +MISSING = object() +cfg = json.loads(Path("in/config.json").read_text()) +out = [] +for line in Path("in/paths.txt").read_text().splitlines(): + v = cfg + for k in line.split("."): + v = v[k] if isinstance(v, dict) and k in v else MISSING + out.append("missing" if v is MISSING else json.dumps(v)) +Path("out/result.txt").write_text("\n".join(out) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x02.py b/experiments/token-efficiency-heldout-v3/solutions/python/x02.py new file mode 100644 index 00000000..785326b3 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x02.py @@ -0,0 +1,12 @@ +import json +from pathlib import Path + +def merge(a, b): + for k, v in b.items(): + a[k] = merge(a[k], v) if isinstance(a.get(k), dict) and isinstance(v, dict) else v + return a + + +base = json.loads(Path("in/base.json").read_text()) +over = json.loads(Path("in/override.json").read_text()) +Path("out/result.json").write_text(json.dumps(merge(base, over), indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x03.py b/experiments/token-efficiency-heldout-v3/solutions/python/x03.py new file mode 100644 index 00000000..d55c1712 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x03.py @@ -0,0 +1,14 @@ +import json +from pathlib import Path + +def flat(d, pre, out): + for k, v in d.items(): + if isinstance(v, dict) and v: + flat(v, pre + k + ".", out) + else: + out[pre + k] = v + return out + + +d = json.loads(Path("in/nested.json").read_text()) +Path("out/result.json").write_text(json.dumps(flat(d, "", {}), indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x04.py b/experiments/token-efficiency-heldout-v3/solutions/python/x04.py new file mode 100644 index 00000000..e6b4e137 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x04.py @@ -0,0 +1,11 @@ +import json +from pathlib import Path + +out = {} +for k, v in json.loads(Path("in/flat.json").read_text()).items(): + *parts, last = k.split(".") + node = out + for p in parts: + node = node.setdefault(p, {}) + node[last] = v +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x05.py b/experiments/token-efficiency-heldout-v3/solutions/python/x05.py new file mode 100644 index 00000000..45491c37 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x05.py @@ -0,0 +1,8 @@ +import json +from pathlib import Path + +out = {"present": [], "null": [], "absent": []} +for c in json.loads(Path("in/contacts.json").read_text()): + key = "absent" if "email" not in c else "null" if c["email"] is None else "present" + out[key].append(c["id"]) +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x06.py b/experiments/token-efficiency-heldout-v3/solutions/python/x06.py new file mode 100644 index 00000000..68ea8dcb --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x06.py @@ -0,0 +1,12 @@ +import json +from pathlib import Path + +out = {} +for s in json.loads(Path("in/sales.json").read_text()): + r = out.setdefault(s["region"], {"units": 0, "products": []}) + r["units"] += s["units"] + if s["product"] not in r["products"]: + r["products"].append(s["product"]) +for r in out.values(): + r["products"].sort() +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x07.py b/experiments/token-efficiency-heldout-v3/solutions/python/x07.py new file mode 100644 index 00000000..71278f7d --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x07.py @@ -0,0 +1,7 @@ +import json +from pathlib import Path + +rows = json.loads(Path("in/rows.json").read_text()) +ren = json.loads(Path("in/rename.json").read_text()) +out = [{ren.get(k, k): v for k, v in r.items()} for r in rows] +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x08.py b/experiments/token-efficiency-heldout-v3/solutions/python/x08.py new file mode 100644 index 00000000..e194b51e --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x08.py @@ -0,0 +1,5 @@ +import json +from pathlib import Path + +doc = json.loads(Path("in/doc.json").read_text()) +Path("out/result.txt").write_text(json.dumps(doc, sort_keys=True, separators=(",", ":")) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x09.py b/experiments/token-efficiency-heldout-v3/solutions/python/x09.py new file mode 100644 index 00000000..72586daa --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x09.py @@ -0,0 +1,11 @@ +import json +from pathlib import Path + +old = json.loads(Path("in/old.json").read_text()) +new = json.loads(Path("in/new.json").read_text()) +out = { + "added": sorted(k for k in new if k not in old), + "removed": sorted(k for k in old if k not in new), + "changed": sorted(k for k in old if k in new and old[k] != new[k]), +} +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x10.py b/experiments/token-efficiency-heldout-v3/solutions/python/x10.py new file mode 100644 index 00000000..6006b2eb --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x10.py @@ -0,0 +1,6 @@ +import json +from pathlib import Path + +tickets = json.loads(Path("in/tickets.json").read_text()) +ids = [t["id"] for t in tickets if "urgent" in t["meta"].get("tags", [])] +Path("out/result.txt").write_text("\n".join(ids) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x11.py b/experiments/token-efficiency-heldout-v3/solutions/python/x11.py new file mode 100644 index 00000000..7597cfb4 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x11.py @@ -0,0 +1,8 @@ +import json +from pathlib import Path + +lines = [] +for a in json.loads(Path("in/accounts.json").read_text()): + city = a["user"].get("profile", {}).get("city") + lines.append(f"{a['login']}: {city if city is not None else 'unknown'}") +Path("out/result.txt").write_text("\n".join(lines) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x12.py b/experiments/token-efficiency-heldout-v3/solutions/python/x12.py new file mode 100644 index 00000000..3c800e25 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x12.py @@ -0,0 +1,6 @@ +import json +from pathlib import Path + +people = json.loads(Path("in/people.json").read_text()) +out = {f"{p['last']}, {p['first']}": p["age"] for p in people} +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x13.py b/experiments/token-efficiency-heldout-v3/solutions/python/x13.py new file mode 100644 index 00000000..23efa614 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x13.py @@ -0,0 +1,8 @@ +import json +from pathlib import Path + +out = {} +for line in Path("in/requests.jsonl").read_text().splitlines(): + r = json.loads(line) + out[r["level"]] = max(out.get(r["level"], r["ms"]), r["ms"]) +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x14.py b/experiments/token-efficiency-heldout-v3/solutions/python/x14.py new file mode 100644 index 00000000..c1a4f8a5 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x14.py @@ -0,0 +1,6 @@ +import json +from pathlib import Path + +prices = json.loads(Path("in/prices.json").read_text()) +out = {k: f"${v // 100}.{v % 100:02d}" for k, v in prices.items()} +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x15.py b/experiments/token-efficiency-heldout-v3/solutions/python/x15.py new file mode 100644 index 00000000..6fd41e45 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x15.py @@ -0,0 +1,13 @@ +import json +from pathlib import Path + +def depth(v): + if isinstance(v, dict): + v = list(v.values()) + if isinstance(v, list): + return 1 + max(map(depth, v), default=0) + return 0 + + +tree = json.loads(Path("in/tree.json").read_text()) +Path("out/result.txt").write_text(f"{depth(tree)}\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x16.py b/experiments/token-efficiency-heldout-v3/solutions/python/x16.py new file mode 100644 index 00000000..fdd213da --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x16.py @@ -0,0 +1,6 @@ +import json +from pathlib import Path + +env = json.loads(Path("in/env.json").read_text()) +out = [{"key": k, "value": env[k]} for k in sorted(env)] +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x17.py b/experiments/token-efficiency-heldout-v3/solutions/python/x17.py new file mode 100644 index 00000000..ce4faa7f --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x17.py @@ -0,0 +1,14 @@ +import json +from pathlib import Path + +doc = json.loads(Path("in/doc.json").read_text()) +for op in json.loads(Path("in/ops.json").read_text()): + *parts, last = op["path"].split(".") + node = doc + for p in parts: + node = node.setdefault(p, {}) + if op["op"] == "set": + node[last] = op["value"] + else: + node.pop(last, None) +Path("out/result.json").write_text(json.dumps(doc, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x18.py b/experiments/token-efficiency-heldout-v3/solutions/python/x18.py new file mode 100644 index 00000000..ef0b8fdb --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x18.py @@ -0,0 +1,10 @@ +from pathlib import Path + +out, cur = [], "" +for line in Path("in/script.txt").read_text().splitlines(): + if line.endswith("\\"): + cur += line[:-1] + else: + out.append(cur + line) + cur = "" +Path("out/result.txt").write_text("\n".join(out) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x19.py b/experiments/token-efficiency-heldout-v3/solutions/python/x19.py new file mode 100644 index 00000000..663b27a1 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x19.py @@ -0,0 +1,9 @@ +from pathlib import Path + +out, inside = [], False +for line in Path("in/app.ini").read_text().splitlines(): + if line.startswith("["): + inside = line == "[net]" + elif inside and line: + out.append(line) +Path("out/result.txt").write_text("\n".join(out) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x20.py b/experiments/token-efficiency-heldout-v3/solutions/python/x20.py new file mode 100644 index 00000000..9c3171b4 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x20.py @@ -0,0 +1,5 @@ +from pathlib import Path + +paras = [p for p in Path("in/doc.txt").read_text().split("\n\n") if p.strip()] +lines = [f"{i}: {len(p.split())}" for i, p in enumerate(paras, 1)] +Path("out/result.txt").write_text("\n".join(lines) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x21.py b/experiments/token-efficiency-heldout-v3/solutions/python/x21.py new file mode 100644 index 00000000..ae0750cd --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x21.py @@ -0,0 +1,4 @@ +from pathlib import Path + +rows = [l.split(",") for l in Path("in/grid.csv").read_text().splitlines()] +Path("out/result.csv").write_text("\n".join(",".join(c) for c in zip(*rows)) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x22.py b/experiments/token-efficiency-heldout-v3/solutions/python/x22.py new file mode 100644 index 00000000..7d0cf880 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x22.py @@ -0,0 +1,8 @@ +from pathlib import Path + +lines = Path("in/sites.csv").read_text().splitlines() +out, prev = [lines[0]], [] +for line in lines[1:]: + prev = [c or p for c, p in zip(line.split(","), prev)] if prev else line.split(",") + out.append(",".join(prev)) +Path("out/result.csv").write_text("\n".join(out) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x23.py b/experiments/token-efficiency-heldout-v3/solutions/python/x23.py new file mode 100644 index 00000000..525eb471 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x23.py @@ -0,0 +1,7 @@ +from pathlib import Path + +dept = dict(l.split(",") for l in Path("in/dept.csv").read_text().splitlines()[1:]) +out = ["id,name,dept"] +for l in Path("in/emp.csv").read_text().splitlines()[1:]: + out.append(f"{l},{dept.get(l.split(',')[0], '')}") +Path("out/result.csv").write_text("\n".join(out) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x24.py b/experiments/token-efficiency-heldout-v3/solutions/python/x24.py new file mode 100644 index 00000000..877969f8 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x24.py @@ -0,0 +1,7 @@ +import json, math +from functools import reduce +from pathlib import Path + +vals = [int(x) for x in Path("in/vals.txt").read_text().strip().split(",")] +out = {"gcd": reduce(math.gcd, vals), "lcm": reduce(lambda a, b: a * b // math.gcd(a, b), vals)} +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x25.py b/experiments/token-efficiency-heldout-v3/solutions/python/x25.py new file mode 100644 index 00000000..f2dc3f78 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x25.py @@ -0,0 +1,9 @@ +from pathlib import Path + +lines = [] +for n in Path("in/nums.txt").read_text().split(): + d = int(n) + while d > 9: + d = sum(map(int, str(d))) + lines.append(f"{n} -> {d}") +Path("out/result.txt").write_text("\n".join(lines) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x26.py b/experiments/token-efficiency-heldout-v3/solutions/python/x26.py new file mode 100644 index 00000000..c43170c9 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x26.py @@ -0,0 +1,5 @@ +from pathlib import Path + +rows = [l.split() for l in Path("in/votes.txt").read_text().splitlines()] +total = sum(int(c) for _, c in rows) +Path("out/result.txt").write_text("\n".join(f"{n} {int(c) * 100 // total}%" for n, c in rows) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x27.py b/experiments/token-efficiency-heldout-v3/solutions/python/x27.py new file mode 100644 index 00000000..37c5c555 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x27.py @@ -0,0 +1,9 @@ +import json +from pathlib import Path + +def flat(v): + return [x for e in v for x in flat(e)] if isinstance(v, list) else [v] + + +deep = json.loads(Path("in/deep.json").read_text()) +Path("out/result.json").write_text(json.dumps(flat(deep), indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x28.py b/experiments/token-efficiency-heldout-v3/solutions/python/x28.py new file mode 100644 index 00000000..ce17c7b6 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x28.py @@ -0,0 +1,8 @@ +import json +from pathlib import Path + +a = json.loads(Path("in/a.json").read_text()) +b = json.loads(Path("in/b.json").read_text()) +out = {"only_a": list(dict.fromkeys(x for x in a if x not in b)), + "only_b": list(dict.fromkeys(x for x in b if x not in a))} +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x29.py b/experiments/token-efficiency-heldout-v3/solutions/python/x29.py new file mode 100644 index 00000000..17adeb30 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x29.py @@ -0,0 +1,6 @@ +import json +from pathlib import Path + +v = json.loads(Path("in/visits.json").read_text()) +out = [x for i, x in enumerate(v) if x not in v[i + 1:]] +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x30.py b/experiments/token-efficiency-heldout-v3/solutions/python/x30.py new file mode 100644 index 00000000..0c1e2b94 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x30.py @@ -0,0 +1,7 @@ +from pathlib import Path + +lines = [] +for l in Path("in/pairs.txt").read_text().splitlines(): + a, b = l.split() + lines.append("yes" if sorted(a) == sorted(b) else "no") +Path("out/result.txt").write_text("\n".join(lines) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x31.py b/experiments/token-efficiency-heldout-v3/solutions/python/x31.py new file mode 100644 index 00000000..2b954122 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x31.py @@ -0,0 +1,7 @@ +from pathlib import Path + +out = [] +for item in Path("in/ranges.txt").read_text().strip().split(","): + a, _, b = item.partition("-") + out += range(int(a), int(b or a) + 1) +Path("out/result.txt").write_text(",".join(map(str, out)) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x32.py b/experiments/token-efficiency-heldout-v3/solutions/python/x32.py new file mode 100644 index 00000000..2e310406 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x32.py @@ -0,0 +1,5 @@ +import os +from pathlib import Path + +words = Path("in/words.txt").read_text().splitlines() +Path("out/result.txt").write_text(os.path.commonprefix(words) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x33.py b/experiments/token-efficiency-heldout-v3/solutions/python/x33.py new file mode 100644 index 00000000..19201453 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x33.py @@ -0,0 +1,10 @@ +from pathlib import Path + +spans = sorted(tuple(map(int, l.split("-"))) for l in Path("in/spans.txt").read_text().split()) +out = [] +for a, b in spans: + if out and a <= out[-1][1]: + out[-1][1] = max(out[-1][1], b) + else: + out.append([a, b]) +Path("out/result.txt").write_text("\n".join(f"{a}-{b}" for a, b in out) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x34.py b/experiments/token-efficiency-heldout-v3/solutions/python/x34.py new file mode 100644 index 00000000..b2995f5d --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x34.py @@ -0,0 +1,6 @@ +import datetime +from pathlib import Path + +lines = [(datetime.date(2021, 1, 1) + datetime.timedelta(int(d) - 1)).strftime("%m-%d") + for d in Path("in/days.txt").read_text().split()] +Path("out/result.txt").write_text("\n".join(lines) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/solutions/python/x35.py b/experiments/token-efficiency-heldout-v3/solutions/python/x35.py new file mode 100644 index 00000000..d3869570 --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/solutions/python/x35.py @@ -0,0 +1,8 @@ +import json +from pathlib import Path + +deps = json.loads(Path("in/deps.json").read_text()) +out = [] +while len(out) < len(deps): + out.append(min(t for t in deps if t not in out and all(d in out for d in deps[t]))) +Path("out/result.json").write_text(json.dumps(out, indent=2) + "\n") diff --git a/experiments/token-efficiency-heldout-v3/verification_test.go b/experiments/token-efficiency-heldout-v3/verification_test.go new file mode 100644 index 00000000..8290854c --- /dev/null +++ b/experiments/token-efficiency-heldout-v3/verification_test.go @@ -0,0 +1,84 @@ +package heldoutv3 + +import ( + "crypto/sha256" + "encoding/json" + "fmt" + "os" + "path/filepath" + "testing" +) + +var exts = map[string]string{"howl": "howl", "python": "py", "js": "js", "go": "go"} + +func readJSON(t *testing.T, path string, v any) { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(data, v); err != nil { + t.Fatal(err) + } +} + +// Every task × language is either verified (pass, matching hash) or an +// explicitly recorded HowlFrame gap with a reason and no solution file. +func TestHeldoutV3VerificationCoverage(t *testing.T) { + var manifest struct { + Tasks []struct { + ID string `json:"id"` + } `json:"tasks"` + } + readJSON(t, "tasks/manifest.json", &manifest) + var gaps []struct{ Task, Lang, Reason string } + readJSON(t, "gaps.json", &gaps) + var results []struct { + Task, Lang, SHA256 string + Pass bool + } + readJSON(t, "results/verification.json", &results) + if len(manifest.Tasks) < 30 { + t.Fatalf("held-out corpus has %d tasks, want >= 30", len(manifest.Tasks)) + } + gapped := map[string]bool{} + for _, g := range gaps { + if g.Lang != "howl" || g.Reason == "" { + t.Errorf("gap %s/%s must be a HowlFrame gap with a reason", g.Task, g.Lang) + } + gapped[g.Task+"/"+g.Lang] = true + if _, err := os.Stat(filepath.Join("solutions", g.Lang, g.Task+"."+exts[g.Lang])); err == nil { + t.Errorf("gap %s/%s has a solution file", g.Task, g.Lang) + } + } + seen := map[string]bool{} + for _, r := range results { + key := r.Task + "/" + r.Lang + if seen[key] || gapped[key] { + t.Errorf("duplicate or gapped entry %s", key) + } + seen[key] = true + if !r.Pass { + t.Errorf("%s did not pass", key) + } + src, err := os.ReadFile(filepath.Join("solutions", r.Lang, r.Task+"."+exts[r.Lang])) + if err != nil { + t.Errorf("%s: %v", key, err) + continue + } + if got := fmt.Sprintf("%x", sha256.Sum256(src)); got != r.SHA256 { + t.Errorf("%s: stale hash", key) + } + } + for _, task := range manifest.Tasks { + for lang := range exts { + key := task.ID + "/" + lang + if !seen[key] && !gapped[key] { + t.Errorf("missing %s", key) + } + } + } + if want := len(manifest.Tasks)*len(exts) - len(gaps); len(results) != want { + t.Errorf("%d results, want %d", len(results), want) + } +} diff --git a/howlframe.go b/howlframe.go index 363a1918..91283507 100644 --- a/howlframe.go +++ b/howlframe.go @@ -1245,7 +1245,9 @@ func rejectUnsupportedTargetValueForms(root *ast.Node, target string) { } // rejectUnsupportedTargetControlflow fails closed on --target=go|js for the -// heldout-v2 control-flow and core built-ins (%, //, break, range, max_by, +// records and JSON sugar (ast.RecordHeads: rec, get, put, read_json, +// write_json, pluck, select, group_by, merge) and the heldout-v2 control-flow +// and core built-ins (%, //, break, range, max_by, // min_by, sort_by, pad_left, pad_right, str_repeat, rtrim, ltrim, to_base), // n-ary if (elif chains) and let destructuring. They are bytecode-target sugar: // the expansions use let-expressions in value position or loop flags the Go @@ -1259,6 +1261,9 @@ func rejectUnsupportedTargetControlflow(root *ast.Node, target string) { for _, h := range ast.ControlflowHeads { heads[h] = true } + for _, h := range ast.RecordHeads { + heads[h] = true + } fail := func(what string, n *ast.Node) { ast.ReportError(fmt.Sprintf("UNSUPPORTED_ON_TARGET: %s is not supported on --target=%s (control-flow/core built-ins are bytecode-target sugar); build for the bytecode target instead", what, target), n.Line, n.Column) } diff --git a/internal/ast/desugar.go b/internal/ast/desugar.go index db254d77..0fbea466 100644 --- a/internal/ast/desugar.go +++ b/internal/ast/desugar.go @@ -231,6 +231,10 @@ func desugarInContext(node *Node, parentHead string, childIndex int) { if isSugarExprContext(parentHead, childIndex) && desugarControlflow(node) { desugarInContext(node, parentHead, childIndex) } + case "rec", "get", "put", "read_json", "write_json", "pluck", "select", "group_by", "merge": + if isSugarExprContext(parentHead, childIndex) && desugarRecords(node) { + desugarInContext(node, parentHead, childIndex) + } case "if": desugarNaryIf(node) case "for", "while": @@ -274,7 +278,9 @@ func isSugarExprContext(parentHead string, childIndex int) bool { // Core-library sugar (heldout-v1). "list_slice", "str_slice", "str_len", "index_of", "map_list", "filter", "reduce", "sum", "min", "max", "reverse", "count_by", "sort", // Control-flow and core built-ins (heldout-v2 follow-up). - "%", "//", "range", "max_by", "min_by", "sort_by", "pad_left", "pad_right", "str_repeat", "rtrim", "ltrim", "to_base": + "%", "//", "range", "max_by", "min_by", "sort_by", "pad_left", "pad_right", "str_repeat", "rtrim", "ltrim", "to_base", + // Records and JSON sugar (heldout-v3 records slice). + "rec", "get", "put", "read_json", "write_json", "pluck", "select", "group_by", "merge": return true case "for": return childIndex == 2 || childIndex == 3 @@ -377,6 +383,8 @@ var reservedCallHeads = map[string]bool{ "reduce": true, "sum": true, "min": true, "max": true, "reverse": true, "count_by": true, "sort": true, "%": true, "//": true, "range": true, "max_by": true, "min_by": true, "sort_by": true, "pad_left": true, "pad_right": true, "str_repeat": true, "rtrim": true, "ltrim": true, "to_base": true, "break": true, + "rec": true, "get": true, "put": true, "read_json": true, "write_json": true, "pluck": true, "select": true, + "group_by": true, "merge": true, "dict": true, "exit": true, "for": true, "if": true, "let": true, "list": true, "print": true, "return": true, "set": true, "stderr": true, "to_lower": true, "to_upper": true, "str_trim": true, diff --git a/internal/ast/desugar_records.go b/internal/ast/desugar_records.go new file mode 100644 index 00000000..1a24416c --- /dev/null +++ b/internal/ast/desugar_records.go @@ -0,0 +1,190 @@ +package ast + +// Records and JSON sugar (heldout-v3 records slice). Every form expands into +// existing, already lowered constructs (let, do, for, when, set, dict, +// map_get, map_set, map_keys, list_get, append, is_nil, nz, map_list, +// parse_json, encode_json, read_text, write_file), so it adds no opcode, +// capability or effect. read_json/write_json reach the filesystem only +// through the read_file/write_file they expand to, so they need exactly the +// filesystem:read / filesystem:write grant those need and fail closed +// without it. Go/JS targets reject every form with UNSUPPORTED_ON_TARGET. +// +// (rec k v ...) record literal: (dict (k v) ...). A key is a name +// (used as its own string: (rec id 1) has key "id") +// or a string literal; values are expressions. +// (get m k1 k2 ...) nested path read. An integer literal step indexes +// a list (list_get); every other step is a map key +// (map_get), so a missing key gives nil. +// (put m k1 ... kn v) nested path write in place: missing or nil +// intermediate map keys get a fresh (dict); integer +// literal intermediate steps index a list. The last +// key must be a map key. Statement position. +// (read_json path) (parse_json Any (read_text path)). +// (write_json path v) (write_file path (encode_json v)). encode_json +// writes object keys in sorted (UTF-8 byte) order, +// compact, so output is stable. +// (pluck lst k) list of (map_get r k) for each record r. +// (select lst k ...) list of records with only the keys k (names or +// strings, like rec); a missing key becomes null. +// (group_by x lst key) dict from (str key) to the elements with that key, +// in input order. +// (merge a b ...) new dict with the keys of a, then b, ... (later +// wins). The operands are not modified. + +// RecordHeads lists every head added by this file, for the Go/JS target +// rejection and the reserved-call list. +var RecordHeads = []string{"rec", "get", "put", "read_json", "write_json", "pluck", "select", "group_by", "merge"} + +// IsRecordSugar reports whether head is a form of this file. +func IsRecordSugar(head string) bool { + for _, h := range RecordHeads { + if h == head { + return true + } + } + return false +} + +func rnList(at *Node, kids ...*Node) *Node { + return &Node{Type: "List", Line: at.Line, Column: at.Column, Filename: at.Filename, Children: kids} +} + +func rnSym(name string, at *Node) *Node { return symNode(name, at) } + +func rnTemp(prefix string, at *Node) string { return compactTemp("rc" + prefix) } + +func rnIsIntLit(n *Node) bool { return n != nil && n.Type == "INT" } + +// rnKey turns a rec/select key (a name or a string literal) into a STRING. +func rnKey(head string, n *Node) (*Node, bool) { + switch n.Type { + case "SYMBOL": + return strNode(n.Value, n), true + case "STRING": + return strNode(n.Value, n), true + } + ReportError(head+" keys must be names or string literals", n.Line, n.Column) + return nil, false +} + +// rnStep reads one path step from the symbol t. +func rnStep(t string, key *Node, at *Node) *Node { + if rnIsIntLit(key) { + return rnList(at, rnSym("list_get", at), rnSym(t, at), key) + } + return rnList(at, rnSym("map_get", at), rnSym(t, at), key) +} + +func rnLet(at *Node, body *Node, pairs ...*Node) *Node { + return rnList(at, rnSym("let", at), rnList(at, pairs...), body) +} + +func desugarRecords(node *Node) bool { + head := node.Children[0].Value + args := node.Children[1:] + at := node + switch head { + case "rec": + if len(args)%2 != 0 { + ReportError("rec expects key/value pairs", node.Line, node.Column) + return false + } + kids := []*Node{rnSym("dict", at)} + for i := 0; i < len(args); i += 2 { + k, ok := rnKey(head, args[i]) + if !ok { + return false + } + kids = append(kids, rnList(at, k, args[i+1])) + } + node.Children = kids + case "get": + // (let (T0 m) (let (T1 step) ... stepN)) + t := rnTemp("G", at) + var build func(cur string, keys []*Node) *Node + build = func(cur string, keys []*Node) *Node { + if len(keys) == 1 { + return rnStep(cur, keys[0], at) + } + nt := rnTemp("G", at) + return rnLet(at, build(nt, keys[1:]), rnSym(nt, at), rnStep(cur, keys[0], at)) + } + node.Children = rnLet(at, build(t, args[1:]), rnSym(t, at), args[0]).Children + case "put": + if rnIsIntLit(args[len(args)-2]) { + ReportError("put: the last key must be a map key, not a list index", node.Line, node.Column) + return false + } + t := rnTemp("P", at) + var build func(cur string, keys []*Node) *Node + build = func(cur string, keys []*Node) *Node { + if len(keys) == 1 { + return rnList(at, rnSym("map_set", at), rnSym(cur, at), keys[0], args[len(args)-1]) + } + nt := rnTemp("P", at) + if rnIsIntLit(keys[0]) { + return rnLet(at, build(nt, keys[1:]), rnSym(nt, at), rnStep(cur, keys[0], at)) + } + kt := rnTemp("K", at) + // (let (K key) (do (when (is_nil (map_get cur K)) (map_set cur K (dict))) + // (let (N (map_get cur K)) rest))) + // No reassignment, so the checker never has to unify N's type. + fill := rnList(at, rnSym("when", at), rnList(at, rnSym("is_nil", at), rnList(at, rnSym("map_get", at), rnSym(cur, at), rnSym(kt, at))), + rnList(at, rnSym("map_set", at), rnSym(cur, at), rnSym(kt, at), rnList(at, rnSym("dict", at)))) + rest := rnLet(at, build(nt, keys[1:]), rnSym(nt, at), rnList(at, rnSym("map_get", at), rnSym(cur, at), rnSym(kt, at))) + return rnLet(at, rnList(at, rnSym("do", at), fill, rest), rnSym(kt, at), keys[0]) + } + node.Children = rnLet(at, build(t, args[1:len(args)-1]), rnSym(t, at), args[0]).Children + case "read_json": + node.Children = []*Node{rnSym("parse_json", at), rnSym("Any", at), rnList(at, rnSym("read_text", at), args[0])} + case "write_json": + node.Children = []*Node{rnSym("write_file", at), args[0], rnList(at, rnSym("encode_json", at), args[1])} + case "pluck": + x, k := rnTemp("X", at), rnTemp("K", at) + node.Children = rnLet(at, + rnList(at, rnSym("map_list", at), rnSym(x, at), args[0], rnList(at, rnSym("map_get", at), rnSym(x, at), rnSym(k, at))), + rnSym(k, at), args[1]).Children + case "select": + x := rnTemp("X", at) + d := []*Node{rnSym("dict", at)} + for _, a := range args[1:] { + k, ok := rnKey(head, a) + if !ok { + return false + } + d = append(d, rnList(at, k, rnList(at, rnSym("map_get", at), rnSym(x, at), strNode(k.Value, at)))) + } + node.Children = []*Node{rnSym("map_list", at), rnSym(x, at), args[0], rnList(at, d...)} + case "group_by": + if args[0].Type != "SYMBOL" { + ReportError("group_by expects a variable name as argument 1", node.Line, node.Column) + return false + } + r, k, g := rnTemp("R", at), rnTemp("K", at), rnTemp("G", at) + x := args[0] + inner := rnLet(at, + rnList(at, rnSym("do", at), + rnList(at, rnSym("append", at), rnSym(g, at), rnSym(x.Value, at)), + rnList(at, rnSym("map_set", at), rnSym(r, at), rnSym(k, at), rnSym(g, at))), + rnSym(k, at), rnList(at, rnSym("str", at), args[2]), + rnSym(g, at), rnList(at, rnSym("nz", at), rnList(at, rnSym("map_get", at), rnSym(r, at), rnSym(k, at)), rnList(at, rnSym("list", at)))) + node.Children = rnLet(at, + rnList(at, rnSym("do", at), rnList(at, rnSym("for", at), x, args[1], inner), rnSym(r, at)), + rnSym(r, at), rnList(at, rnSym("dict", at))).Children + case "merge": + r := rnTemp("R", at) + pairs := []*Node{rnSym(r, at), rnList(at, rnSym("dict", at))} + body := []*Node{rnSym("do", at)} + for _, a := range args { + s, k := rnTemp("S", at), rnTemp("K", at) + pairs = append(pairs, rnSym(s, at), a) + body = append(body, rnList(at, rnSym("for", at), rnSym(k, at), rnList(at, rnSym("map_keys", at), rnSym(s, at)), + rnList(at, rnSym("map_set", at), rnSym(r, at), rnSym(k, at), rnList(at, rnSym("map_get", at), rnSym(s, at), rnSym(k, at))))) + } + body = append(body, rnSym(r, at)) + node.Children = rnLet(at, rnList(at, body...), pairs...).Children + default: + return false + } + return true +} diff --git a/internal/bytecode/construct_drift_test.go b/internal/bytecode/construct_drift_test.go index c283bbdd..ed504bfc 100644 --- a/internal/bytecode/construct_drift_test.go +++ b/internal/bytecode/construct_drift_test.go @@ -152,6 +152,15 @@ func TestCompileTimeOnlyConstructsReachingCompileNodeHaveCases(t *testing.T) { "rtrim": true, // ast.Desugar core library "ltrim": true, // ast.Desugar core library "to_base": true, // ast.Desugar core library + "rec": true, // ast.Desugar records + "get": true, // ast.Desugar records + "put": true, // ast.Desugar records + "read_json": true, // ast.Desugar records + "write_json": true, // ast.Desugar records + "pluck": true, // ast.Desugar records + "select": true, // ast.Desugar records + "group_by": true, // ast.Desugar records + "merge": true, // ast.Desugar records "ver_ge": true, // ast.Desugar "nz": true, // ast.Desugar "sget": true, // ast.Desugar diff --git a/internal/construct/arity.go b/internal/construct/arity.go index f51b049d..dabed866 100644 --- a/internal/construct/arity.go +++ b/internal/construct/arity.go @@ -62,6 +62,9 @@ var arities = map[string]Arity{ "%": {2, 2}, "//": {2, 2}, "break": {0, 0}, "range": {1, 2}, "max_by": {3, 3}, "min_by": {3, 3}, "sort_by": {3, 3}, "pad_left": {2, 3}, "pad_right": {2, 3}, "str_repeat": {2, 2}, "rtrim": {1, 1}, "ltrim": {1, 1}, "to_base": {2, 2}, + // Records and JSON sugar (heldout-v3 records slice). + "rec": {0, Variadic}, "get": {2, Variadic}, "put": {3, Variadic}, "read_json": {1, 1}, "write_json": {2, 2}, + "pluck": {2, 2}, "select": {2, Variadic}, "group_by": {3, 3}, "merge": {1, Variadic}, } // arityExempt lists registry constructs with no arity entry, each with the diff --git a/internal/construct/construct.go b/internal/construct/construct.go index e822a561..f280644e 100644 --- a/internal/construct/construct.go +++ b/internal/construct/construct.go @@ -125,7 +125,9 @@ var table = []Entry{ {Name: "filter", Support: CompileTimeOnly, Note: "core-library sugar; (filter x lst pred) for/when/append by ast.Desugar (pure)"}, {Name: "for", Support: Supported, Note: "FOR_INIT / FOR_NEXT"}, {Name: "fuzzy_cast", Support: Unsupported, Note: "LLM struct coercion; no opcode, Go backend only"}, + {Name: "get", Support: CompileTimeOnly, Note: "records sugar; (get m k ...) nested path read via map_get/list_get by ast.Desugar (pure)"}, {Name: "go_import", Support: Unsupported, Note: "Go-backend-only directive; the standalone VM cannot honour a Go package dependency"}, + {Name: "group_by", Support: CompileTimeOnly, Note: "records sugar; (group_by x lst key) via for/nz/append/map_set by ast.Desugar (pure)"}, {Name: "html_escape", Support: Supported, Note: "HTML_ESCAPE; encodes &, <, >, \", and ' with no capability"}, {Name: "http_body", Support: CompileTimeOnly, Note: "density sugar; (http_body r) → bytes_to_string of map_get body by ast.Desugar"}, {Name: "http_request", Support: Supported, Note: "HTTP_REQUEST; returns dict (status int, body bytes); optional body sent"}, @@ -157,6 +159,7 @@ var table = []Entry{ {Name: "match", Support: Unsupported, Note: "no opcode; the interpreter also fails closed on it (internal/vm/vm.go); use switch"}, {Name: "max", Support: CompileTimeOnly, Note: "core-library sugar; (max lst) for/when/> by ast.Desugar (pure)"}, {Name: "max_by", Support: CompileTimeOnly, Note: "core-library sugar; (max_by x lst key) via map_list/for by ast.Desugar (pure)"}, + {Name: "merge", Support: CompileTimeOnly, Note: "records sugar; (merge a b ...) new dict via for/map_keys/map_set by ast.Desugar (pure)"}, {Name: "middleware", Support: Unsupported, Note: "HTTP middleware chain; no opcode, Go backend only"}, {Name: "min", Support: CompileTimeOnly, Note: "core-library sugar; (min lst) for/when/< by ast.Desugar (pure)"}, {Name: "min_by", Support: CompileTimeOnly, Note: "core-library sugar; (min_by x lst key) via map_list/for by ast.Desugar (pure)"}, @@ -172,11 +175,15 @@ var table = []Entry{ {Name: "pad_right", Support: CompileTimeOnly, Note: "core-library sugar; (pad_right s n [ch]) via while/str by ast.Desugar (pure)"}, {Name: "parse_json", Support: Supported, Opaque: true, Note: "PARSE_JSON; reads child .Value directly"}, {Name: "patch", Support: CompileTimeOnly, Note: "applied and removed by ast.ApplyPatches before lowering"}, + {Name: "pluck", Support: CompileTimeOnly, Note: "records sugar; (pluck lst k) via map_list/map_get by ast.Desugar (pure)"}, {Name: "print", Support: Supported, Note: "PRINT"}, + {Name: "put", Support: CompileTimeOnly, Note: "records sugar; (put m k ... v) nested in-place write via map_get/map_set/dict by ast.Desugar (pure)"}, {Name: "range", Support: CompileTimeOnly, Note: "control-flow sugar; (range [a] b) int list via while/append by ast.Desugar (pure)"}, {Name: "read_file", Support: Supported, Note: "READ_FILE"}, + {Name: "read_json", Support: CompileTimeOnly, Note: "records sugar; (read_json p) → parse_json of read_text by ast.Desugar (filesystem:read via read_file only)"}, {Name: "read_line", Support: Supported, Note: "READ_LINE"}, {Name: "read_text", Support: CompileTimeOnly, Note: "density sugar; expanded to bytes_to_string/read_file by ast.Desugar before lowering"}, + {Name: "rec", Support: CompileTimeOnly, Note: "records sugar; (rec k v ...) record literal → dict with name/string keys by ast.Desugar (pure)"}, {Name: "reduce", Support: CompileTimeOnly, Note: "core-library sugar; (reduce acc init x lst expr) for/set fold by ast.Desugar (pure)"}, {Name: "regex_match", Support: Supported, Note: "REGEX_MATCH"}, {Name: "req_header", Support: Supported, Note: "HTTP_REQ_HEADER; pure read of an already-accepted request"}, @@ -191,6 +198,7 @@ var table = []Entry{ {Name: "rtrim", Support: CompileTimeOnly, Note: "core-library sugar; (rtrim s) via str_split/while/list_slice/str_join by ast.Desugar (pure)"}, {Name: "schema", Support: Unsupported, Note: "declarative SQL migration; no opcode, Go backend only"}, {Name: "schema_bridge", Support: Supported, Note: "compiles the wrapped source expression"}, + {Name: "select", Support: CompileTimeOnly, Note: "records sugar; (select lst k ...) via map_list/dict/map_get by ast.Desugar (pure)"}, {Name: "semantic_match", Support: Unsupported, Note: "LLM intent dispatch; no opcode, Go backend only"}, {Name: "set", Support: Supported, Note: "SET_VAR"}, {Name: "sget", Support: CompileTimeOnly, Note: "density sugar; store string-val get via store_get/map_get by ast.Desugar"}, @@ -246,6 +254,7 @@ var table = []Entry{ {Name: "while", Support: Supported, Note: "JUMP_IF_FALSE / JUMP"}, {Name: "with_context", Support: CompileTimeOnly, Opaque: true, Note: "eliminated by ast.ApplyWithContext before lowering; its variable list is structural, not code"}, {Name: "write_file", Support: Supported, Note: "WRITE_FILE"}, + {Name: "write_json", Support: CompileTimeOnly, Note: "records sugar; (write_json p v) → write_file of encode_json by ast.Desugar (filesystem:write via write_file only)"}, } var index = func() map[string]Entry { diff --git a/internal/vm/vm.go b/internal/vm/vm.go index 97c6b66e..f34d9c85 100644 --- a/internal/vm/vm.go +++ b/internal/vm/vm.go @@ -679,6 +679,37 @@ func (interp *Interpreter) evalList(node *ast.Node, env *InterpEnv) any { InterpErr(fmt.Sprintf("encode_json: %v", err), node) } return string(b) + case "parse_json": + // (parse_json T text): ast.Desugar has bound a non-symbol text + // operand to a temporary, so this is the same variable read the + // bytecode PARSE_JSON does (string, bytes or a byte-number list), + // decoded with the same decodeJSONValue. No capability: the text + // is already in memory. + if len(node.Children) != 3 { + InterpErr("parse_json expects (parse_json T text)", node) + } + var data []byte + switch v := interp.eval(node.Children[2], env).(type) { + case string: + data = []byte(v) + case []byte: + data = v + case []any: + for _, e := range v { + n, ok := e.(int64) + if !ok || n < 0 || n > 255 { + InterpErr("parse_json failed: byte list must hold ints 0..255", node) + } + data = append(data, byte(n)) + } + default: + InterpErr(fmt.Sprintf("parse_json failed: expected text, got %T", v), node) + } + result, err := decodeJSONValue(data) + if err != nil { + InterpErr(fmt.Sprintf("parse_json failed: %v", err), node) + } + return result case "html_escape", "attr_escape": if len(node.Children) != 2 { InterpErr(fmt.Sprintf("%s expects (%s text)", head, head), node) diff --git a/records_cli_test.go b/records_cli_test.go new file mode 100644 index 00000000..d9a852a8 --- /dev/null +++ b/records_cli_test.go @@ -0,0 +1,234 @@ +package main + +import ( + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/howlcipher/howlframe/internal/bytecode" +) + +// Records and JSON sugar (heldout-v3 records slice): rec, get, put, pluck, +// select, group_by, merge are pure; parse_json also runs under -run. +const recordsProgram = `(do + (let (r (rec id 1 name "ann" "tags" (list "a" "b") addr (rec city "oslo"))) + (do + (print (encode_json r)) + (print (get r "name") (get r "addr" "city") (get r "tags" 1) (is_nil (get r "nope"))) + (put r "addr" "zip" "0150") + (put r "meta" "seen" "count" 3) + (print (encode_json r)) + (print (encode_json (merge r (rec id 2 extra true) (rec name "bo"))) (get r "id")))) + (let (rs (list (rec n "a" d "x" v 1) (rec n "b" d "y" v 2) (rec n "c" d "x" v 3))) + (do + (print (encode_json (pluck rs "n"))) + (print (encode_json (select rs n v))) + (print (encode_json (group_by r rs (get r "d")))) + (print (encode_json (group_by r rs (> (get r "v") 1)))))) + (let (s (str "{" "\"z\":1,\"a\":{\"y\":[1,2.5,null]}}")) + (do (print (encode_json (parse_json Any s))) (print (get (parse_json Any s) "a" "y" 1)))))` + +const recordsWant = `{"addr":{"city":"oslo"},"id":1,"name":"ann","tags":["a","b"]} +ann oslo b true +{"addr":{"city":"oslo","zip":"0150"},"id":1,"meta":{"seen":{"count":3}},"name":"ann","tags":["a","b"]} +{"addr":{"city":"oslo","zip":"0150"},"extra":true,"id":2,"meta":{"seen":{"count":3}},"name":"bo","tags":["a","b"]} 1 +["a","b","c"] +[{"n":"a","v":1},{"n":"b","v":2},{"n":"c","v":3}] +{"x":[{"d":"x","n":"a","v":1},{"d":"x","n":"c","v":3}],"y":[{"d":"y","n":"b","v":2}]} +{"false":[{"d":"x","n":"a","v":1}],"true":[{"d":"y","n":"b","v":2},{"d":"x","n":"c","v":3}]} +{"a":{"y":[1,2.5,null]},"z":1} +2.5 +` + +func recordsPaths(t *testing.T, bin, src, dir string) map[string][]string { + t.Helper() + art := filepath.Join(dir, "p.hfbc") + hart := filepath.Join(dir, "h.hfbc") + if out, err := exec.Command(bin, "build", "--profile", "governed", src, "-o", art).CombinedOutput(); err != nil { + t.Fatalf("build: %v %s", err, out) + } + if out, err := exec.Command(bin, "-compile-hfir-bc", src, "-o", hart).CombinedOutput(); err != nil { + t.Fatalf("hfir compile: %v %s", err, out) + } + return map[string][]string{ + "interpreter": {"-run", src}, + "bytecode_governed": {"-run-bc", art}, + "hfir_bytecode": {"-run-bc", hart}, + } +} + +func TestRecordsSugarIsPureOnEveryPath(t *testing.T) { + bin := c1c2Binary(t) + dir := t.TempDir() + src := writeTemp(t, dir, "p.howl", recordsProgram) + for _, profile := range []string{"default", "governed"} { + if out, err := exec.Command(bin, "check", "--profile", profile, src).CombinedOutput(); err != nil { + t.Fatalf("check %s: %v %s", profile, err, out) + } + } + runs := recordsPaths(t, bin, src, dir) + out, err := exec.Command(bin, "inspect", "--caps", runs["bytecode_governed"][1]).CombinedOutput() + if err != nil { + t.Fatalf("inspect: %v %s", err, out) + } + var report bytecode.Report + if err := json.Unmarshal(out, &report); err != nil { + t.Fatalf("%v: %s", err, out) + } + if len(report.Capabilities) != 0 { + t.Fatalf("pure records sugar required capabilities %v", report.Capabilities) + } + for name, args := range runs { + t.Run(name, func(t *testing.T) { + got, err := exec.Command(bin, args...).CombinedOutput() + if err != nil { + t.Fatalf("run: %v %s", err, got) + } + if string(got) != recordsWant { + t.Fatalf("got %q want %q", got, recordsWant) + } + }) + } +} + +// read_json/write_json need exactly filesystem (inferred from the +// read_file/write_file they expand to), run the same on every path with the +// grant, and fail closed without it or with only the read root. +func TestReadWriteJSONUnderFilesystemCapability(t *testing.T) { + bin := c1c2Binary(t) + dir := t.TempDir() + in, outDir := filepath.Join(dir, "in"), filepath.Join(dir, "out") + os.MkdirAll(in, 0o755) + os.MkdirAll(outDir, 0o755) + os.WriteFile(filepath.Join(in, "c.json"), []byte(`{"z":1,"a":{"y":[1,2]},"m":"q"}`), 0o644) + src := writeTemp(t, dir, "j.howl", `(do (let (c (read_json "in/c.json")) (do (put c "a" "k" 5) (write_json "out/o.json" c) (print (get c "m")))))`) + runs := recordsPaths(t, bin, src, dir) + out, err := exec.Command(bin, "inspect", "--caps", runs["bytecode_governed"][1]).CombinedOutput() + if err != nil { + t.Fatalf("inspect: %v %s", err, out) + } + var report bytecode.Report + json.Unmarshal(out, &report) + if strings.Join(report.Capabilities, ",") != "filesystem" { + t.Fatalf("capabilities %v, want [filesystem]", report.Capabilities) + } + grant := "filesystem:read=" + in + ",filesystem:write=" + outDir + for name, args := range runs { + t.Run(name, func(t *testing.T) { + os.Remove(filepath.Join(outDir, "o.json")) + full := append([]string{args[0], "-allow-caps", grant}, args[1:]...) + c := exec.Command(bin, full...) + c.Dir = dir + got, err := c.CombinedOutput() + if err != nil || string(got) != "q\n" { + t.Fatalf("run: %v %q", err, got) + } + data, _ := os.ReadFile(filepath.Join(outDir, "o.json")) + if string(data) != `{"a":{"k":5,"y":[1,2]},"m":"q","z":1}` { + t.Fatalf("wrote %q", data) + } + for _, deny := range [][]string{{args[0]}, {args[0], "-allow-caps", "filesystem:read=" + in}} { + os.Remove(filepath.Join(outDir, "o.json")) + c := exec.Command(bin, append(deny, args[1:]...)...) + c.Dir = dir + got, err := c.CombinedOutput() + if err == nil || !strings.Contains(string(got), "denied") { + t.Errorf("%v: err=%v out=%q", deny, err, got) + } + if _, statErr := os.Stat(filepath.Join(outDir, "o.json")); statErr == nil { + t.Errorf("%v wrote output without the write grant", deny) + } + } + }) + } +} + +func TestRecordsErrorsOnEveryPath(t *testing.T) { + bin := c1c2Binary(t) + for src, want := range map[string]string{ + `(do (let (j (parse_json Any (str "5"))) (print (get j "a"))))`: "map_get", + `(do (let (s "{bad") (print (parse_json Any s))))`: "parse_json failed", + `(do (let (m (rec a 1)) (do (put m "a" "b" 2) (print m))))`: "map_set", + } { + dir := t.TempDir() + p := writeTemp(t, dir, "e.howl", src) + for name, args := range recordsPaths(t, bin, p, dir) { + out, err := exec.Command(bin, args...).CombinedOutput() + if err == nil || !strings.Contains(string(out), want) { + t.Errorf("%s %s: err=%v out=%q want %q", src, name, err, out, want) + } + } + } +} + +func TestRecordsCompileErrors(t *testing.T) { + bin := c1c2Binary(t) + for src, want := range map[string]string{ + `(do (print (rec a)))`: "rec expects key/value pairs", + `(do (print (rec (+ 1 2) 3)))`: "rec keys must be names or string literals", + `(do (print (select (list) (+ 1 2))))`: "select keys must be names or string literals", + `(do (let (m (list 1)) (put m 0 2)))`: "put: the last key must be a map key", + `(do (print (group_by (list) (list) 1)))`: "group_by expects a variable name", + `(do (print (get (rec a 1))))`: "ARITY_ERROR: get expects at least 2 args, got 1", + `(do (print (read_json)))`: "ARITY_ERROR: read_json expects 1 arg", + `(do (print (pluck (list) "a" "b")))`: "ARITY_ERROR: pluck expects 2 args, got 3", + `(do (print (merge)))`: "ARITY_ERROR: merge expects at least 1 arg", + } { + dir := t.TempDir() + p := writeTemp(t, dir, "c.howl", src) + for _, args := range [][]string{{"-run", p}, {"build", "--profile", "governed", p, "-o", filepath.Join(dir, "c.hfbc")}, {"-compile-hfir-bc", p, "-o", filepath.Join(dir, "h.hfbc")}} { + out, err := exec.Command(bin, args...).CombinedOutput() + if err == nil || !strings.Contains(string(out), want) { + t.Errorf("%s %v: err=%v out=%q want %q", src, args[0], err, out, want) + } + } + } +} + +func TestRecordsRejectedOnGoJS(t *testing.T) { + bin := c1c2Binary(t) + forms := []string{ + `(print (rec a 1))`, `(let (m (dict)) (print (get m "a")))`, `(let (m (dict)) (put m "a" 1))`, + `(print (read_json "x.json"))`, `(write_json "x.json" (list))`, `(print (pluck (list) "a"))`, + `(print (select (list) a))`, `(print (group_by x (list) x))`, `(print (merge (dict)))`, + } + for _, f := range forms { + for _, target := range []string{"go", "js"} { + dir := t.TempDir() + p := writeTemp(t, dir, "g.howl", "(do "+f+")") + o := filepath.Join(dir, "out") + out, err := exec.Command(bin, "build", "--target", target, p, "-o", o).CombinedOutput() + if err == nil || !strings.Contains(string(out), "UNSUPPORTED_ON_TARGET") || strings.Contains(string(out), "Built") { + t.Errorf("%s on %s: err=%v out=%s", f, target, err, out) + } + if _, statErr := os.Stat(o); statErr == nil { + t.Errorf("%s on %s wrote output", f, target) + } + } + } +} + +func TestRecordsGovernedFailsClosed(t *testing.T) { + bin := c1c2Binary(t) + for _, src := range []string{ + `(do (print (rec a (exec "ls"))))`, + `(do (print (get (rec a 1) (exec "ls"))))`, + `(do (let (m (dict)) (put m "a" (exec "ls"))))`, + `(do (print (read_json (exec "ls"))))`, + `(do (write_json "o.json" (exec "ls")))`, + `(do (print (pluck (exec "ls") "a")))`, + `(do (print (select (exec "ls") a)))`, + `(do (print (group_by x (list 1) (exec "ls"))))`, + `(do (print (merge (dict) (exec "ls"))))`, + } { + dir := t.TempDir() + p := writeTemp(t, dir, "g.howl", src) + out, err := exec.Command(bin, "build", "--profile", "governed", p, "-o", filepath.Join(dir, "g.hfbc")).CombinedOutput() + if err == nil || !strings.Contains(string(out), "PROFILE_FORBIDDEN_CONSTRUCT") { + t.Errorf("%s: err=%v out=%s", src, err, out) + } + } +} diff --git a/tests/parity_stabilization/manifest.json b/tests/parity_stabilization/manifest.json index a31027d8..1965f4b2 100644 --- a/tests/parity_stabilization/manifest.json +++ b/tests/parity_stabilization/manifest.json @@ -112,12 +112,11 @@ "exit": 0, "targets": { "hfir_bytecode": "PASS", - "interpreter": "UNSUPPORTED", + "interpreter": "PASS", "go": "COMPILE_FAILURE", "javascript": "PASS" }, "notes": { - "interpreter": "parse_json is not supported under -run", "go": "generated Go decodes into a declared Go type; the undeclared Payload used by the VM is not a Go type" } },