Skip to content

docs: add technical documentation — overview, architecture, and module reference #46

Open
#47
Open
docs: add technical documentation — overview, architecture, and module reference#46
#47

Description

@KipTwitchell
KipTwitchell
opened on Sep 2, 2026

Summary

This issue tracks the creation of structured technical documentation for the GenevaERS Performance Engine. The codebase is a production-grade IBM z/OS High Level Assembler (HLASM) system implementing a high-throughput batch data-extraction and reporting engine (part of the broader GenevaERS / SAFR project). The source is richly commented internally, but has no developer- or operator-facing documentation beyond the README's Git setup instructions. This gap makes onboarding, maintenance, and contribution significantly harder.

Architecture overview: MR95/MR96 — a two-pass JIT compiler wrapped in a multi-threaded I/O harness

The fundamental design

GVBMR95 is not an interpreter. The Logic Table (LT) is a compiler IR, and the system JIT-compiles it into raw z/Architecture machine code before any event record is touched. This is the central fact everything else flows from.

Phase 0 — Program startup (GVBMR95 entry)

GVBMR95 enters in AMODE 31 and immediately executes SAM64 to switch to 64-bit addressing for the rest of its life. All 16 GPRs are zero-extended with LLGTR to eliminate high-order garbage before any pointer use.

Storage for the main thread work area (THRDLEN) is obtained with CHECKZERO=YES — the kernel is asked to confirm whether the storage is pre-zeroed so the expensive MVCL fill can be skipped.

GVBURALI is called immediately after startup. The returned 8-byte module alias name (stored in NAMEPGM) governs every subsequent file open — EXTRRPT/REFRRPT, EXTRLOG/REFRLOG, EXTRTRAC/REFRTRAC, EXTRDUMP/REFRDUMP. There is no separate PARM-driven mode switch; the alias name is the mode selector. MR95 then calls GVBMR96 via BASSM.

Phase 1 — GVBMR96 initialization (the compiler front-end)

MR96 runs entirely before the first event record is ever read. Its job is to take the Logic Table from disk and produce executable machine code in memory. The mainline sequences these steps:

Step 1 — VDP and Logic Table load (VDPLOAD → LTBLLOAD)

VDPLOAD reads the VDP file and chains all record types into linked lists by type (type 50 = control record, type 200B = file records, type 210 = exit program records, type 1000 = view records). Each record type gets its own first_NNN anchor pointer so MR96 can find records via LOCVIEW, LOCEXIT, LOCCID helpers.

LTBLLOAD reads the Logic Table sequentially into a contiguous in-memory array. The LOGICTBL DSECT header for each row carries: LTMAJFUN/LTSUBFUN (the 4-char function code), LTTRUE/LTFALSE (branch-row pointers, still as row numbers at this stage), LTCODSEG (will hold the generated code address after PASS1), and a flags halfword encoding optimisation decisions.

Step 2 — Thread area construction (THRDBLD)

Thread count = min(EXECDISK, actual_disk_LFs) + min(EXECTAPE, actual_tape_LFs) + pipe/other count. For each thread, THRDCOPY calls GETMAIN for a THRDLEN + thread_vars_sz block and chains the areas through THRDNEXT. Thread-level Logic Table variables (NV-type) are initialised here — packed-decimal fields via ZAP→P'0', binary fields via XC.

Step 3 — Clone RE/ES sets (CLONLTBL)

When multiple threads share a logical file, the RE/ES set is cloned. Each clone receives its own literal pool copy (MVCL) and its own lookup-buffer chain, with updated LTESLPAD/LTESLPSZ fields.

Step 4 — Memory-resident reference tables (LOADLKUP)

Reference files are read in their entirety and sorted into binary-searchable in-memory tables. Table sizes are derived from the LT view-column definitions.

Step 5 — Literal pool allocation (ALLOCLIT)

One 1 MB page-aligned pool is allocated per work unit (thread). This becomes the per-thread literal pool — a flat buffer that PASS1 fills with per-ES data: the LITP_HDR header, token-offset arrays, per-view accumulator areas, and serialised addresses of runtime variables. The base register (R2 at runtime) is set to pool_start + 512K so that all offsets fit within ±512K, enabling 20-bit signed displacements throughout the generated code.

Step 6 — PASS1: code generation (the JIT compiler)

PASS1 walks every LT row in sequence and for each row:

  1. Looks up the function table entry (LTFUNTBL) for the model code address (FCMODELA) and length (FCCODELN).
  2. Copies the model code verbatim into the code buffer with EX Rx,MVCMODEL.
  3. Saves the code buffer address back into LTCODSEG of the LT row — this is how PASS2 and the runtime find each row's code.
  4. For NV (new-view) rows: writes the NVPROLOG — a fixed-layout header holding NVNXVIEW (next-view prolog address), NVLAYR8 (a LAY R8,offset(R7) instruction whose displacement PASS2 will patch with the actual extract-record column offset), and NVNOP (a JLU *+offset whose target PASS2 patches for disabled-view skipping).
  5. For ES (end-of-set) rows: installs an EVNTPREV branch back to the event loop.
  6. For lookup-prefix rows (LTLKUPRE flag set): prepends a fixed-length prefix code block before the main code segment.
  7. CFCC verb (constant-field compare-compare): resolves the comparison at compile time using EX on a model CLC instruction, then substitutes either NOOP code or a false-branch stub. No runtime comparison ever runs for a CFCC row.
  8. Cookies (date/time tokens with negative length fields): resolves RunDay, RunMonth, RunYear, fiscal quarter boundaries, etc. to actual date values at compile time. No cookie expansion happens at event-record time.

Step 7 — Open extract files (OPENEXTF)

Walks the LT for WR_XT rows, finds each associated VDP0200B record, determines the file type (QSAM, VSAM, PIPE, NULL-PIPE, TOKEN), and opens DCBs in AMODE 31 (SAM31/OPEN/SAM64). Pipe/null-pipe cases patch the EXTPUTA/EXTCHKA/EXTPUT_6431/EXTCHK_6431 function vectors in the EXTFILE control block to the appropriate stubs. Page-fixing (EXECPAGF=Y) sets DCBEFLG3|=DCBEBFXU.

Phase 2 — Back in GVBMR95: PASS2 and execution

PASS2 — branch relocation

PASS1 left branch targets as raw row addresses. PASS2 converts them to relative halfword displacements by walking the relocation table (FCRELOCA) entry by entry:

  • Types 12/13 → true-branch displacement: load LTTRUE row address, find its LTCODSEG, subtract current position, halve, store via STH.
  • Types 14/15 → same via LTFALSE.
  • Type 29 → title-key column offset: compute EXSRTKEY-EXTREC + LTSORTLN + LTFLDPOS, patch the MVC displacement.
  • Branches targeting NV or ES rows: special-cased through NVNXVIEW to find the next-view code address before computing the displacement.

After PASS2, the generated code is fully self-contained z/Architecture machine code with real branch displacements. The SNAPped dump (DUMP_LT_AND_GENERATED_CODE=Y) writes this to EXTRDUMP/REFRDUMP.

Enclave and ZIIP setup

If APF-authorized: MODESET SUP → IWM4ECRE (create dependent enclave) → SYSEVENT ENCASSOC JOIN → IWMEJOIN → call ZIIP module with ZIIP_OCT to enable SRB offload. The enclave token is saved for teardown.

Thread dispatch

  • Single-thread (EXECUTE_IN_MAIN_TASK=1 or A): PICKEVNT selects the first event file, R13 switches to the thread work area, branches to MAIN.
  • Multi-thread: EVENTS ENTRIES=(N+1) allocates the ECB wait-list; ATTACHX EP=MR95THRD,SHSPV=15,SZERO=YES launches each subtask. One extra ECB slot is reserved for the ESTAE. The main task waits in an EVENTS WAIT loop, handling ESTAE-stop and daughter-task completion events.

MAIN — per-thread initialization

Each thread (main task or attached subtask):

  1. Saves FP8–FP15; loads FP8/10 = 0; loads FP9/11 = DFP_QUANTUM (these FPR pairs must never be clobbered by generated code — they hold the DFP zero and quantum values used by accumulator arithmetic throughout the run).
  2. Allocates Pause Element Tokens (IEA4APE) — TCB PETs and SRB PETs for the PAUSE/RELEASE write-serialisation protocol.
  3. Installs ESTAE (ESTAEX), sets PSW overflow mask.

RESTART → per-file setup

Resets per-file thread variables, copies DDNAMEs from the LT RE row, optionally calls DYNALLOC (SVC 99 via GVBUR35) to allocate the dataset, then dispatches to the I/O driver via a SELECT on the access-method ID:

Access method code Driver
SEQFILE GVBMRBS
KSDSFILE GVBMRVK
DB2SQL MRSQADDR → GVBMRSQ
DB2HPU MRSUADDR → GVBMRSU
CALLADA MRADADDR → GVBMRAD

Write exits (LTWRADDR) and lookup exits (LBSUBADR) are then called with their OP (initialize) function code by walking all NV rows in the ES.

EVNTLOOP — the hot path

R6 += GPRECLEN          advance to next record in buffer
if R6 >= EODADDR        buffer exhausted:
  ├── TCB→SRB mode switch (ZIIP) if needed
  ├── call EVNTREAD (GVBMRBS EXCP read)
  └── SRB→TCB mode switch after I/O completes

STG R6, RECADDR           save event record pointer
STY R0, GPEVENTA          write pointer into GENPARM

R7 = GPEXTRA              extract record work area
R8 = THRDES               ES row address
R2 = THRDLITP + 512K      literal pool base
R15 = LTESCODE            generated ES entry point
BR  R15                   → EXECUTE GENERATED CODE

The generated code runs the full view set for this ES. It calls back into GVBSRCHR for binary-search lookups and GVBDL96 for field formatting. When a WR row fires, the EXTPUTA / EXTPUT_6431 vector in the EXTFILE control block is invoked to write the extract record.

Teardown

After all threads complete: IWMEQTME gathers enclave CPU/ZIIP times → CLOSFILE → PREP_SORTS → report routines (ISRC, OWRT, LKUP, FINAL) → TERM_UEXITS → SYSEVENT ENCASSOC_LEAVE → IWMELEAV → IWM4EDEL → PR (pop linkage stack, return).

End-to-end flow summary

GVBMR95 entry (A31 → A64, BAKR)
  │
  ├─ GVBURALI       → 8-byte alias name → all DDNAMEs
  ├─ Open EXTRLOG/REFRLOG
  └─ BASSM → GVBMR96
               │
               ├─ VDPLOAD    → linked lists of VDP record types (50, 200B, 210, 1000…)
               ├─ LTBLLOAD   → in-memory LT array
               ├─ THRDBLD    → N thread work areas + thread-variable areas
               ├─ CLONLTBL   → copy RE/ES sets + literal pools per extra thread
               ├─ LOADLKUP   → reference files → sorted in-memory tables
               ├─ ALLOCLIT   → 1 MB literal pool per work unit
               ├─ PASS1      → JIT compile: model code → code buffer
               │               resolve cookies and CFCC at compile time
               │               set LTCODSEG per row
               └─ OPENEXTF   → open extract DCBs, patch write vectors
  │
  └─ back in GVBMR95
      ├─ PASS2       → patch true/false branch displacements, NV offsets
      ├─ Enclave     → IWM4ECRE + IWMEJOIN + ZIIP offload enable
      ├─ ATTACHX     → N−1 subtasks, each enters MAIN
      └─ MAIN (per thread)
           ├─ IEA4APE  → allocate Pause Element Tokens
           ├─ ESTAEX   → install error/abend recovery
           ├─ RESTART  → per-ES file open + exit OP calls
           └─ EVNTLOOP
                ├─ advance R6 through buffer
                ├─ EVNTREAD if buffer exhausted (TCB ↔ SRB flip for ZIIP)
                └─ BR LTESCODE  ← generated machine code executes
                     ├─ GVBSRCHR  (binary-search lookup)
                     ├─ GVBDL96   (field format/conversion)
                     └─ EXTPUTA   (write extract record)

Key insight: The Logic Table is never interpreted at event-record time. By the time MR95 reads the first event record, the LT has been fully compiled into native z/Architecture machine code with real branch displacements, accumulator offsets baked into LAY instructions, and all constant comparisons pre-resolved. The only runtime overhead per event record is the I/O and the execution of that generated code.

How a job actually runs: the three-step execution pattern

The architecture above describes what happens inside a single MR95 invocation. A real production job — such as the Demo RUNEXT1.JCL — orchestrates three distinct steps that must run in order, and understanding that sequence is essential context for anyone working with this codebase.

Step 1 — Control file generation (Java RCA, PSTEP200/JAVA)

Before MR95 is invoked at all, a Java program (org.genevaers.rcapps.Runner) reads a Workbench XML export (WBXMLI) containing the view definitions and compiles it into three binary control files:

DD Dataset suffix Content
VDPNEW .VDP View Definition Parameters — all view, file, column, and exit metadata (RECFM=VB, LRECL=32756)
JLTNEW .JLT Join Logic Table — logic for the reference phase (RECFM=VB, LRECL=820)
XLTNEW .XLT Extract Logic Table — logic for the extract phase (RECFM=VB, LRECL=820)

The DBVIEWS DD selects which view IDs to include (e.g. 10700, 10689, …). The RCAPARM DD controls the compiler: INPUT_TYPE=WBXML, GENERATE_RC_FILES=Y, and report flags (VDP_REPORT=Y, JLT_REPORT=Y, XLT_REPORT=Y). These three files are the sole input to every subsequent MR95 run. They are re-generated whenever view definitions change.

Step 2 — Reference phase (GVBMR95R, PSTEP505)

The first MR95 execution runs as the alias GVBMR95R. It uses the JLT (Join Logic Table) — not the XLT — and its job is exclusively to sort and pre-stage the reference (lookup) data into files that the extract phase can load directly into memory.

Key DD names for GVBMR95R:

DD Role
MR95VDP The .VDP file from step 1
REFRLTBL The .JLT (Join Logic Table — reference phase logic)
REFRPARM Execution parameters (see below)
REFRTPRM Trace filter keywords
REFRENVV Environment variable overrides (symbolic substitution via GVBUR33)
CUSTNAME, CUSTOMER, CUSTADDR, ORDER001, PRODDESC, STORE001 Raw reference input files — named by DDNAME to match LT file records
REFRREH Reference extract header file (RECFM=FB, LRECL=100) — indexes into the RED files
REFRRTH Reference token header file
REFR001–REFR006 Pre-sorted reference data files (.RED — Reference Extracted Data, RECFM=VB, LRECL=4144, up to 10 SYSDA units each)
REFRRPT / REFRLOG / REFRTRAC Diagnostic output — report, log, trace
SNAPDATA DUMP_LT_AND_GENERATED_CODE=Y output
SYSMDUMP Abend dump dataset

The output REFR001–REFR006 files are sorted, indexed reference datasets — one per distinct reference logical file. They are not human-readable extract output; they are an optimised on-disk representation of the in-memory lookup tables that LOADLKUP will reconstruct in the extract phase. The REFRREH header file carries metadata (record counts, key lengths, sort order) that LOADLKUP uses to rebuild the binary-search tables without re-reading the source files.

REFRPARM parameter keywords (all optional, shown with defaults):

DISK_THREAD_LIMIT=10          default: 9999
TAPE_THREAD_LIMIT=10          default: 9999
IO_BUFFER_LEVEL=8             default: 4
OPTIMIZE_PACKED_OUTPUT=N      default: Y
PAGE_FIX_IO_BUFFERS=N         default: Y
TREAT_MISSING_VIEW_OUTPUTS_AS_DUMMY=Y   default: N
ABEND_ON_CALCULATION_OVERFLOW=N         default: Y
TRACE=Y                       default: N
DUMP_LT_AND_GENERATED_CODE=Y  default: N
SOURCE_RECORD_LIMIT=100       default: (no limit)
ABEND_ON_LOGIC_TABLE_ROW_NBR=57         default: (no abend)
ABEND_ON_MESSAGE_NBR=149      default: (no abend)
EXECUTE_IN_MAIN_TASK=1        default: N  (1=first unit, A=all units)

Return code ≥ 1 triggers GVBUT99 (step PSTEP506) with abend code 1099, stopping the job.

Step 3 — Extract phase (GVBMR95E, PSTEP705)

The second MR95 execution runs as the alias GVBMR95E. It uses the XLT (Extract Logic Table) and reads both the pre-staged reference data from step 2 and the actual event (source transaction) files.

Key DD names for GVBMR95E:

DD Role
MR95VDP Same .VDP file — shared between both phases
EXTRLTBL The .XLT (Extract Logic Table — extract phase logic)
EXTRPARM Execution parameters (see below)
EXTRTPRM Trace filter keywords
EXTRENVV Environment variable overrides ($SUBSYS=DM12, $QUAL=SDATRT01 etc.) passed to symbolic variable substitution (GVBUR33) at runtime
EXTRREH The reference header file written by GVBMR95R — tells LOADLKUP how to rebuild the tables
REFR001–REFR006 The pre-sorted .RED files from step 2 — loaded into memory by LOADLKUP
ORDIT001–ORDIT003 Event (source/driver) files — named by DDNAME to match LT RE rows
EXTR001–EXTR003 Extract output files destined for the format phase (.EXT, RECFM=VB, LRECL=8192, 10 SYSDA units)
SORT001–SORT003 Sort control records for each extract file (.SXT, RECFM=FB, LRECL=80) — consumed by the subsequent format job
OUTPUT01–OUTPUT03 Direct extract-only view outputs (tape, VT3592, RECFM=VB, LRECL=1004) — bypasses format phase entirely
EXTRRPT / EXTRLOG / EXTRTRAC Diagnostic output
MERGRPT Multi-thread merge report
IDIOFF DUMMY — suppresses IBM Fault Analyzer interception
SYSMDUMP / CEEDUMP Abend dump datasets

EXTRPARM parameter keywords (superset of reference phase, with extract-specific additions):

RUN_DATE=20170105             default: current date
FISCAL_DATE_DEFAULT=20161231  default: RUN_DATE
FISCAL_DATE_OVERRIDE=1:20160731   default: FISCAL_DATE_DEFAULT (per control record ID)
DISK_THREAD_LIMIT=2           default: 9999
...all other parameters same as reference phase...
EXECUTE_IN_PARENT_THREAD=A    default: N  (note: different keyword name from reference phase)

RUN_DATE and FISCAL_DATE_* feed the cookie resolution at PASS1 time — they become the concrete date values substituted for RunDay, RunMonth, FiscalDay etc. tokens in the LT. EXTRENVV supplies $-prefixed symbolic variables resolved by GVBUR33 for any LT fields that reference them.

Step 4 — Chained format jobs (JSTEPNX1–JSTEPNX3)

The final steps in RUNEXT1.JCL use IEBGENER to submit up to three format jobs (RUNFMT1–RUNFMT3) to the internal reader. Each format job runs GVBMR87+GVBMR88 against one of the EXTR00x/SORT00x extract file pairs. The job chain is implicit: RUNEXT1 submits the format jobs before it ends, ensuring they queue behind it without a scheduler dependency.

The complete job-to-job data flow

WBXMLI (Workbench XML)
    │
    └─ Java RCA ──────────────────────────────────┐
                                                   │
                         ┌─────────────────────────┤
                         │   .VDP   .JLT   .XLT    │
                         └──────┬──────────────────┘
                                │
            ┌───────────────────┼──────────────────────┐
            │                   │                      │
     GVBMR95R (alias)     GVBMR95E (alias)        (shared VDP)
     reads: .JLT              reads: .XLT
     reads: CUSTNAME etc       reads: REFRREH
     writes: REFRREH           reads: REFR001-006
             REFRRTH           reads: ORDIT001-003
             REFR001-006       writes: EXTR001-003
                                       SORT001-003
                                       OUTPUT01-03
                                           │
                               ┌───────────┴────────────┐
                           RUNFMT1               RUNFMT2/3
                        GVBMR87+MR88           GVBMR87+MR88
                        reads: EXTR001          reads: EXTR002/3
                               SORT001                 SORT002/3
                        writes: reports         writes: reports

Note for ARCHITECTURE.md: The .JLT and .XLT are separate compilations of the same view set, optimised for their respective phases. The reference phase JLT contains only lookup-table build logic; the extract phase XLT contains the full event-record processing logic including join references back into the tables that GVBMR95R pre-staged. A single .VDP file is shared by both — it is the complete metadata catalogue for the run.

Proposed documentation structure

1. Architecture overview — ARCHITECTURE.md (repo root)

File: ARCHITECTURE.md at the repository root, linked from README.md.

Rationale: The repo currently has only README.md at the root — there is no docs/ directory. Because the architecture document is the single most important piece of documentation for anyone who clones the repo, burying it in a subdirectory, or leaving it unlisted from the root, would make it invisible. Placing it at the root as ARCHITECTURE.md means:

  • It is immediately visible in the GitHub repository file listing, one click from the landing page.
  • It follows the same convention used by major open-source projects (Linux kernel Documentation/, LLVM, GCC) where the top-level architecture doc lives at or one level below the root.
  • A direct link in README.md ([Architecture & internals](ARCHITECTURE.md)) ensures no contributor can miss it.
  • The remaining documents (docs/modules.md, docs/dsects.md, etc.) can live under docs/ once that directory is created, with ARCHITECTURE.md serving as the entry point that links to them.

Content should be the full architecture overview above — the JIT compiler design, the two-phase MR95/MR96 relationship, the three-step job execution pattern (reference phase → extract phase → format phase), the DD interface, PARM keywords, and the job-to-job data flow diagram.

2. Module reference — docs/modules.md

Per-module table covering name, function, AMODE/RMODE, key register conventions, and major callers/callees. Suggested initial grouping:

Extract Phase

Module Role
GVBMR96 Init: loads Logic Table, opens extract files, loads reference tables, builds binary-search paths, generates skeleton code per LT row
GVBMR95 Runtime engine: reads event records, executes compiled Logic Table machine code, performs lookups, writes extract records
GVBMRBS BSAM sequential file initialisation for MR95
GVBMRVK VSAM keyed I/O handler for MR95 (extended-dataset compatible)
GVBMRSQ DB2 dynamic SQL initialisation — PREPARE + FETCH loop feeding MR95
GVBMRSU DB2 HPU (High Performance Unload) initialisation
GVBMRHPU DB2 HPU exit — FUNCTION 0/1/2 row-processing/init/term
GVBMRAD Adabas I/O initialisation (parses DBID/FNR/FB fields from VDP0200B_DBMS_SQL)
GVBSRCHR Binary-search lookup routines for key lengths 1–156; shares register frame with MR95

Format Phase

Module Role
GVBMR87 Init: loads view-definition parameter tables, opens input files, loads title lookup tables, initialises binary search
GVBMR88 Runtime: outputs sorted extract data, accumulates column totals, prints accumulators at control breaks (detail + summary)

Utility / Support

Module Role
GVBTP90 General VSAM/QSAM I/O handler (OP/CL/RD/LO/SB/BR/WR/UP/DL/IN/RI) — runtime-dynamic DDNAMEs
GVBUR20 BSAM sequential read initialisation (disk + tape, EXCP-based multi-buffer)
GVBUR33 Symbolic variable substitution (env-variable table walk — resolves $-prefixed tokens in EXTRENVV/REFRENVV)
GVBUR35 SVC 99 (dynamic allocation) interface
GVBUR39 GENWRITE address resolver (RMODE 24 stub)
GVBURALI Alias detection — resolves which 8-byte load-module alias (GVBMR95E, GVBMR95R) was used to invoke MR95
GVBURZTM CPU/ZIIP time formatter — accepts TOD-clock values, writes to DCB or WTO
GVBDL96 Field formatter — converts any source format (packed, binary, etc.) to any output format/mask
GVBDAYS Julian-date arithmetic — days between two CCYYDDD dates
GVBUT99 User-abend utility — issues a PARM-specified abend code from JCL (used as job-step guard: COND=(1,GE), PARM='1099')
GVBUTHDR Report header formatter (shared by ASM and COBOL modules)
GVBUTMSG SAFR message-build service — substitutes up to 3 insert strings into a message template
GVBUTMUE Mixed-case message table builder (invokes GVBMSGGE macro)

3. Data structures / DSECTs reference — docs/dsects.md

Narrative + field tables for the primary shared data structures defined in the macro library:

  • EXECDATA (MAC/EXECDATA.mac) — EXEC PARM area passed to MR95/MR96: thread counts, trace/snap switches, ZIIPs, page-fix, DB2 plan names, date overrides, hash-table tuning.
  • LOGICTBL / LTGN_REC (GVBMR95L/GVBLTGEN) — Logic Table row prefix and GEN header record: function codes, true/false branch addresses, generated-code segment pointer, optimisation flags.
  • EXTFILE (GVBMR95C) — extract file control block: DCB/DECB addresses, write/check routine vectors, record/byte counts, AMODE 64 write variant pointers, pipe counts.
  • VDP0200B_FILE_RECORD (GVB0200B) — per-file VDP record: DDNAME, access method ID, record format/length, DB2/Adabas SQL text (VDP0200B_DBMS_SQL), input exit reference.
  • DL96AREA (MAC/DL96AREA.mac) — field-specification entry for GVBDL96: source format/content/mask, output format/decimals/rounding.
  • Logic Table type-specific records (HD, NV, F0, F1, F2, RE, WR, CC, VN, VV — from the GVBLT*A macros) — counts and structures generated/consumed during extract.

4. Build and deployment guide — docs/build.md

Expand the README with:

  • HLASM Toolkit version requirements
  • zapp.yaml explained (SYSLIB macro resolution path, language hlasm)
  • How to assemble and link-edit each module (referencing LINKPARM/*.ftl binder control files)
  • DB2 BIND procedure (walk-through of JCL/BIND.JCL, required variables, plan/package naming)
  • SMP/E vs. manual installation paths

5. EXEC PARM guide — docs/parm.md

Operator-facing reference documenting every keyword accepted by REFRPARM/EXTRPARM, its default, valid values, and the EXECDATA DSECT field it maps to — currently undocumented outside inline comments in the source.

Why now

The project has active development (tags through WB_4.21.1_RC15), multiple source branches, and no contribution guide. Documentation at this level will:

  1. Lower the barrier for new contributors unfamiliar with z/OS assembler conventions.
  2. Provide operators clear guidance on tuning parameters (threads, ZIIPs, hash-table multipliers, page-fixing).
  3. Give a map of module interdependencies needed before refactoring or adding new I/O adapters.

Suggested first PR

Create ARCHITECTURE.md at the repo root (content from the full architecture overview and job execution pattern above) and add a single line to README.md linking to it. This one file gives every contributor and operator immediate orientation to the codebase — from JCL job structure and DD interface, through the control file pipeline, to the JIT compiler internals — and is a prerequisite for everything else in this issue.

Postscript: the people who built this engine

The code in this repository did not arrive fully formed. It is the product of more than four decades of continuous engineering by a named, traceable group of people — and ARCHITECTURE.md should acknowledge that.

Origins: Alaska, 1982

The ideas at the heart of the Performance Engine were first worked out in 1981–1985, when Rick Roth worked on the Alaska Statewide Accounting System. Doug Kunkel joined in 1982, and together with the early team they developed the large-scale, single-pass reporting concepts that everything since has been built upon. The Federal Government's enterprise accounting mandates for state systems created the forcing function for the scale of thinking that made GenevaERS possible.

From there the same team — Rick Roth, Rip Sanders, Doug Kunkel, Kirk Lowery, William "Bill" Bengtson, Mukesh Patel, Dave "Murph" Murphy, Randall Ness, Julia Braun Roth, Jay Poulos — carried the architecture through the Oregon Department of Transportation (1987–1990) and then the Wyoming Department of Transportation (1989–1992), where the extract engine was converted to assembler and enhanced to include parallel processing, and the product was given its first name: GENEVA.

Sacramento, 1992–1996: productisation

The GENEVA Development Center in Sacramento centralised product development and converted more of the codebase to assembler. The four 1994 diagrams linked from the GenevaERS history site — drawn by Doug Kunkel to explain the internal memory structures and pointer chains of MR95 to the maintenance team — are the earliest surviving technical documentation of the structures this architecture document describes. The Sacramento team included Brandy Smith, Clyde Simmons, Rob Clark, Monica Logan, Chris Stallman, Mona Breed, Cheryl Gentsch, Dave Haws, Spiros Velianitis, Barry Arabi, Mark Ashton, David Colpitts, and Kip Twitchell, among others.

Financial services and IBM, 1996–2010

Beginning in 1996, GenevaERS was deployed at major financial services firms. This decade hardened the engine, introduced the Workbench GUI and view compiler, and eventually led to IBM acquiring the tool and renaming it SAFR (Scalable Architecture for Financial Reporting). During this period the batch process was updated to use reentrant code (reducing memory footprint), the Windows Workbench was converted to the Eclipse framework, and the extract process was progressively modernised.

The team that carried GenevaERS through this era — in alphabetical order as of June 2009 — included: Abhay Kumar, Al Sung, Amit Trehan, Amy Y Huang, Angela Liang, Ann Randall, Anoop K Sharma, Anthony "Tony" Minetti Jr., Archana Dasar, Bharat Shah, Bill Lewis, Bill Yankowiak, Burt Phannemiller, Chandra Devarkonda, Chris Stallman, Daniel Cho, David "Murph" Murphy, Debarati Parui, Devin Brand, Devyani Sahasrabudhe, Dilip Sanchora, Doreen Rose, Doug Kunkel, Elana R Dunn, Eric Levy, Fred Horwitz, Greg Forsythe, Greg Halper, Greg Shi, Jim Hladyshewsky, Jeff Horner, Kip Twitchell, Lynnette Groves Zuccala, Mark Cederberg, Mark Kimmell, Mark Thyen, Michael Shapiro, Mike Schroeck, Peter Corbett, Randall Ness, Rakesh Kant, Sam Kahng, Seth Weis, Steve Graham, Todd Topolski, Tony Boles, and many others.

Refinement, 2010–2019

The support organisation continued improving the engine: updating the extract process to use 64-bit code (freeing memory for I/O buffers), consolidating six legacy UNIX programs into one, adding enhanced TRACE facilities and the VDP Analyser, and building full regression coverage of all Logic Table function codes. Peter Cook, Girish Koppikar, Craig Brookes, Gillian Hannington, Neil Beesley, John "Dravo" Dravnieks, and others drove the Performance Engine work during this era. Workbench development was led by Ian Cunningham and Venkatesh Kaikala. The consistent backbone of the support organisation through this period was Jim Hladyshewsky (manager), Elana Dunn (assistant), Randall Ness (lead architect), Michael Shapiro, Steve Brown, and Kip Twitchell.

Open source, 2020–present

In April 2020, IBM donated the codebase and related assets to the Linux Foundation's Open Mainframe Project, returning it to the name GenevaERS. Sandy Peresie became the first open source developer. Andrea Orth joined the project at the outset. The open source era brought new contributors while honouring the engineering decisions — and the register conventions — established three decades earlier.

In memoriam

Doug Kunkel — the principal inventor of the extract engine's memory architecture and the author of the 1994 internal diagrams — died while still a member of the team. Michael Berger, a GenevaERS consultant, also passed away in service to the project. Their work is present in every LTCODSEG pointer, every NVPROLOG field layout, and every binary-search path in this repository.

Further reading: GenevaERS History Overview and Participants — Randall Ness, compiled for a team reunion, April 2019. Kip Twitchell, Balancing Act: A Practical Approach to Business Event Based Insights, Chapter 38.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions