EnvLoop's controlled pilot of data-centric research for verifiable computer use.
A real Kanboard application, public-source issue metadata, actual Tinker LoRA training, an E2B sampling proxy, Harbor task execution, and an independent saved-state verifier. Inspired by RSIBench-Data.
- Paper page on envloop.ai
- English technical report
- Interactive evidence explorer
- Report, figures and offline bundle
- Original-cohort audit and Sol 6 / Luna 6 audit
The portable public Odoo reference package has a clean installed-wheel reproduction in a fresh local Docker project: persisted reference controls 0 / 1 / 0, 28 guarded actions and seven independently verified reset gates. It contains public synthetic fixtures and no training-provider dependency, private runtime artifacts or hidden evaluation cases. Its scalar receipt grants no model or official-study credit.
For a build that does not require training-provider access, use the non-Tinker controller. Its current component status and English build PDF separate offline builds, genuine native controls and incomplete model results.
The 2 October execution record covers the requested single-environment pilot. Sol 6 produced four independently verified GUI teacher workflows and 103 positive multimodal training turns. All 20 selection controls and 20 frozen final controls passed saved-state and source review. Three baseline selection cases completed; real Tinker training completed 19 of 192 optimizer steps before HTTP 402 blocked both jobs. No training checkpoint or final model outcome is available, and no training effect has been measured. Interrupted provider requests are retained as infrastructure failures, separately from model scores. The fixtures are synthetic business data inside real Odoo Community. The full six-environment study remains incomplete.
The infrastructure recovery amendment preserves the three completed cases, excludes the incomplete attempt without a score, and specifies reviewed continuation after billing access is restored. It retains the native actor, independent scorer, exact reset and original task limits.
The 2 October full-study status records server-side checkpoint readback, two independently qualified current GitLab control trios, and corrections to earlier live-worker labels. These are development and qualification evidence; official study result counts remain zero.
The English qualification PDF, interactive six-application explorer, and source-hashed evidence data document the next benchmark's environment and verifier work. This is a separate methods release, not a new full-scale score: the planned Qwen3.8-27B study has zero completed researcher campaigns, zero official final task instances, and no application with 100 admitted tasks. Human GUI controls, one-task model pilots, published source inventories, and provisional splits are labeled separately. The v0.5 Qwen3.5-4B Kanboard results below remain unchanged.
The v0.6.1 English addendum and vector control diagram report three additional saved-state development controls in real Excel web, PowerPoint web, and Magento admin. Their task-specific scores are not pooled or promoted into the planned 24-campaign study. The Magento 100-candidate backlog is an audited preparation queue with zero officially admitted final identities.
For the intended publication-scale study, the English pre-result protocol fixes four researcher configurations × six real-software cells, a Qwen3.8-27B student, source isolation, budgets, failure handling, and family-level analysis. A pre-campaign gate requires all six qualified 20-selection/100-final cells and 24 researcher intents before training; the post-campaign matrix gate binds the 30 matched base/selected result slots to that freeze, and the complete-result audit refuses missing or unscored outcomes. No full-study campaign or official final task result has been reported, and no full-result PDF exists. Candidate counts, evaluator-scripted GUI controls, and train-only provider calls remain development evidence. The current evidence-status table separates the latest train diagnostics and live-control failures from official study results.
The provisional sources are original WDI PowerPoint decks, original SEC-filing Excel workbooks, WDI-derived LibreOffice desktop tasks, original Odoo Community worlds, a CISA-anchored GitLab CE world, and original Magento catalog tasks. Each source document states what is authentic external data, benchmark-authored simulation, private evaluator content, and unverified application work. The PowerPoint-web native-chart training control passed one independently downloaded positive/near-miss/fresh-copy triad after an evaluator-frozen Office-saved baseline, without admitting a final task. A four-edit PowerPoint-web development control also passed 4/4-positive, 3/4-near-miss, and exact fresh reset; its entire four-task source family was quarantined after verifier/controller calibration, leaving 96 original final candidates. A chart-control feasibility check prompted a pre-result workflow revision, and the replacement four-edit training-source GUI triad passed downloaded 4/4-positive, 3/4-near-miss, and byte-identical fresh-reset checks. The dated PPT source amendment and revised WDI reserve audit restore 100 offline candidates with a new source family; PowerPoint-web final admissions remain zero. The Magento train-only native-search control demonstrates a saved-state positive, wrong-object negative and exact cold reset on one exposed training task; it admits no final IDs. The dated desktop, enterprise, GitLab, and Magento amendments record source choices before results. Public WebArena, PPT-Eval, SpreadsheetBench and OSWorld tasks remain development comparators, not a sealed hidden exam. ServiceNow WorkArena access remains a separately gated supplemental track.
The full-size desktop GUI audit records 100/100 evaluator-controlled final-candidate positive/near-miss/fresh-reset trios across Calc, Impress, and Writer, with a separate saved-artifact semantic audit and fair scorer candidate. This establishes deterministic development controls, not official final admissions or Qwen outcomes; the proposed v0.6.5 action-output amendment still needs six-cell adoption before the common pre-campaign freeze.
A second evaluator-only PowerPoint controller check exposed another four-task source family before results. The dated second PPT amendment quarantines that family, and the two-reserve audit restores 100 offline final candidates using a separately pinned official World Bank country CSV. All 140 revised decks pass source, chart, layout and direct-file controls, while PowerPoint-web per-ID final admissions remain 0/100.
A third evaluator-only PowerPoint save exposed a material native-chart rewrite in a negative-inflation source family. The third pre-result amendment quarantines that four-task family and records a tested two-stage Office baseline stabilization. The three-reserve audit restores 20/20/100 offline candidates from another authenticated World Bank country CSV; 136 prior specifications and deck semantics are preserved and all 140 pass offline controls. Application-native chart normalization and every final per-ID web control remain separate pending gates.
A fourth pre-result source family was excluded when a positive-valued chart also changed on the next Office-web save. The fourth source/saved-state amendment requires per-task application-native baseline stabilization and narrows scorer equivalence to observed target text/edit-history changes; a changed non-target chart still fails. A raw-to-second-stage four-target training control completed with a fresh-copy positive scored 1 and no collateral mutation. The historical four-reserve audit bound 20/20/100 offline candidates, 136 unchanged prior specifications/deck semantics and 140 passing offline controls at that revision. PowerPoint-web official final admissions remained zero.
One more evaluator-only PowerPoint family was excluded after two retained Office thumbnail-control failures under a pre-registered one-retry rule. The saved-artifact admission amendment makes two independently scored OneDrive downloads authoritative; a five-role training check passed its 1/0/0/0 controls. The fifth source amendment and historical five-reserve audit restored 20/20/100 offline candidates at that revision.
A subsequent saved positive exposed a native-chart no-regression failure despite four correct targets. The sixth source and persisted-positive amendment retains that failed result, quarantines its four-task family and requires an intermediate positive save before neutral restoration. A training-source V5 audit passed two consecutive normalization cycles and fresh-copy positive/near-miss/reset/collateral controls. The current six-reserve audit restores 20/20/100 offline candidates with 136 unchanged prior task/deck semantics and 140 passing offline controls. PowerPoint-web per-ID final admissions remain 0/100.
One v12 candidate passed an individual PowerPoint-web V5 GUI control, including two-stage baseline preservation and fresh-copy saved positive/partial/reset/collateral scoring. A later evaluator input-binding error on another task exposed that source family; the seventh pre-result amendment quarantines it, retains the earlier pass as historical development evidence and derives future collateral text directly from the saved PPTX. The current seven-reserve audit restores 20/20/100 offline candidates and 140 passing direct-file controls. Active-pool PowerPoint-web GUI passes and official admissions are both zero.
The Magento 25-case checkpoint independently binds real-admin saved positive, wrong-variant negative and fresh-clone material reset for 25 distinct private final candidates. It is a dated checkpoint, not an official final result.
A later Magento CMS-menu infrastructure stop was retained before any negative edit. The failed clone's material state and identity-bound cleanup passed; a pre-result rule allows only one complete, same-controller fresh-clone retry. Neither this interruption nor any future recovery is a model result.
The one-time Magento recovery later passed fresh positive/negative/reset controls without deleting the original failure. A later second frozen-index mismatch stopped the cell at 35 historical evaluator controls. A cron-free training startup probe passed source-index checks. Its first training GUI trio stopped at negative CMS navigation; the sole fully fresh training retry passed positive 1, wrong-variant 0 and cold reset. The revised runtime freeze authorizes only a new 100-task evaluator GUI sweep. The original 35 controls cannot be mixed with it, and official admission remains zero.
The dated six-cell preparation figure visualizes offline candidate construction, non-comparable evaluator GUI controls and zero official admissions directly from public aggregate receipts. It is a methods-progress snapshot, not a model-result chart.
The later offline-complete snapshot records 600 constructed final-candidate identities across the six cells after the original-SEC Excel split reached 20/20/100. The unchanged official-admission count is zero in every cell; the blue bars remain evaluator development evidence only.
The 28 September admission-gap figure updates the last committed per-cell development-control receipts and separates 600 offline candidates, 316 heterogeneous historical GUI controls, zero fresh v0.6.6 final-control trios, and zero official admissions. It is a methods-progress visualization, not a model score. The English full-study PDF pipeline refuses to produce a result paper until all 24 campaigns, 600 task identities, matched outcomes, per-attempt telemetry, research trajectories, and independent cell reviews are audited.
The later 28 September interrupted-control snapshot supersedes only that figure's live-control counts. Seven native Desktop trios were completed before one current-frame failure stopped the batch; 92 Desktop candidates were not attempted. The revised Magento sweep reached ten independently audited evaluator controls while continuing. Both lanes remain development qualification, with zero official admissions, researcher campaigns, or full-study final outcomes.
Subsequent Desktop and Magento receipts supersede those live counts. A Desktop-only caret amendment passed a real public-train GUI smoke, but its fresh final-control run stopped before actor actions on two task-profile checks. Six public-train Calc/Writer/Impress guests passed profile-scope calibration. A separate real public-train Impress positive/near-miss/fresh-reset trio then passed; the new Desktop runtime source freeze and three-root 324-lease bridge are recorded, but no new final-control sandbox has started. Magento's revised v2 run accepted 14 complete controls, then stopped at a pre-edit modal; its one permitted retry failed before browser import because the operator Python lacked Playwright. Both disposable pairs were exactly retired. A real original-GUI v3 training pilot subsequently passed positive 1, wrong-variant 0, fresh reset and four exact pre-edit state checks while observing the release mask; its clean 100-case source freeze is recorded, but the new run has not started. These failures are not model zeros. One real Desktop training trajectory has been rendered into 18 Qwen vision SFT datums without a provider call or measured training effect. The simpler single-account Office train pilot is prepared, with no live Office candidate admitted. Official full-study counts remain 0/600 admissions, 0/24 researcher campaigns and 0/3,000 final slot-task outcomes.
A later GitLab 100-ID requalification audit preserved all six original first-attempt failures while recording one successful, full fresh-clone GUI requalification of each same ID. The active set now has 100 evaluator 1/0/1 controls with 300 cold resets across 20 source families; the FIFO reserve was unused. These are still development controls. The common v0.6.6 live actor gate and six-cell freeze have not admitted any official GitLab final task.
The 28 September current-status table supersedes the earlier live-progress counts in this chronology. Three real Desktop public-train sources now cover 38 screenshot-bound GUI turns across Impress, Calc and Writer, while the separate original-20 train/holdout plan remains source-only. A 64-step Qwen3.8 Tinker diagnostic used 30 of those turns for training and eight from a task-disjoint public-train holdout. Its saved 0/8 format scores were invalidated by expired observation frames; a read-only rescore of the same saved outputs found action-type matches of 2/8 base versus 4/8 LoRA and exact-payload matches of 0/8 versus 1/8. This one-task teacher-forced comparison is neither a measured application success nor an official training-effect estimate; actual provider billing remains unknown. Odoo has one separately accepted train pilot, a retained split-wide train audit-order failure now reclassified from the same saved evidence after a current-baseline check, with visual source review pending, and a first selection control stopped before its fourth GUI intent by a two-pixel frame flicker. GitLab's failed train-only attempts produced no eligible SFT episode, and its 100-ID child-watchdog path remains source-only. The clean Magento v3 100-ID freeze has not been dispatched. The official counts are still 0/600 final admissions, 0/24 campaigns and 0/3,000 outcomes.
The PowerPoint split-shape audit discloses the actual one/three/four target-field train/selection/final progression, disjoint source families and intentional causal-workflow overlap. It supports a harder cross-slide final design without claiming measured difficulty or unseen-workflow transfer.
After seven evaluator-only PowerPoint family quarantines, a pre-result ordered reserve commitment and attrition cap limits further replacements to three families. A source failure beyond that bound fails the cell and requires a newly registered study instead of selectively replenishing the exam.
A train-only E2B Office-web login bridge design and bounded launcher are available for a later manual-login pilot. The launcher has passed dry-run and fake-provider cleanup tests; it has not created a logged-in sandbox or executed an Office model task. Microsoft credentials and browser cookies are neither accepted nor published by this bridge.
A separate Qwen3.8-27B base-model Magento price-task pilot records one runner failure and one 40-sample budget-censored retry. Neither completed a product save or entered an official final-task denominator; both attempts' monitored SQL and search state were restored. This protocol-development evidence is not a checkpoint or training-gain result.
The later v3 scored development diagnostic completed one valid model-driven GUI attempt on the same public task. Qwen saved one of five target prices, then chose finish; the unchanged evaluator and independent saved-state check scored the full task 0.0. Two intervening v2 attempts remain unscored infrastructure/protocol failures. Monitored SQL and all 181 search documents were restored after every attempt. This public task adds no hidden final identity or researcher campaign.
Researchers write Python data factories in isolated E2B workspaces, generate native task states, request verified GUI demonstrations, and revise their training mixtures from selection feedback. Each accepted dataset trains a fresh Qwen3.5-4B LoRA adapter. The fixed teacher is gpt-5.6-sol; the researcher model and the evaluated student are different roles.
| Cohort | Researcher | Research rounds | Trained candidates | Scheduled training tokens | Retained selection |
|---|---|---|---|---|---|
| Original | gpt-6-astra | 5 | 3 | 371,887 | Round 1, 1/3 |
| Original | gpt-5.6-sol | 5 | 4 | 460,768 | Round 5, 1/3 |
| Separate extension | gpt-6-sol | 5 | 5 | 664,132 | Round 2, 1/3 |
| Separate extension | gpt-6-luna | 5 | 1 | 140,167 | Base, 0/3 |
All four searches, prescribed final evaluations, and declared operational recoveries are complete. In the Sol 6 / Luna 6 extension, the student selected by Sol 6 scores 2/12 (16.7%) across two six-task repetitions; the base scores 0/12. Luna 6 retains that same base checkpoint and shares its execution evidence. All 24 extension final trials are valid, without infrastructure errors. These are separate cohorts, not a matched four-model ranking: final source records and researcher-facing interface timing differ. Original Sol5.6 results are never relabeled Sol6. The extension explicitly reuses the selection baseline and starts fresh researcher lineages.
The historical v0.4 release used selection/augmentation from one teacher trace. Its published artifacts remain unchanged and are not pooled with executable-factory results. See the factory contract and alignment with RSIBench-Data.
The original cohort retains its infrastructure-invalid original comparisons. In its separate recovery view, base scores 0/12 and the Sol 5.6-selected student scores 2/12; valid Astra originals score 3/12. Astra originals use the Mac controller and replays use Linux, so that view is not a platform-matched estimate of training-data effects.
No sustained RSI, broad desktop generalization, or mature leaderboard is claimed. Rejected submissions receive no training score. Infrastructure failures remain unscored, and declared recoveries retain the original evidence.
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e '.[browser,cloud,harbor,report]'
source .venv/bin/activate
PYTHONPATH=src python -m unittest discover -s tests -v
Configure OPENAI_API_KEY, OPENAI_BASE_URL, TINKER_API_KEY, and E2B_API_KEY outside the repository. AgentRouterHub uses the Responses API with the requested researcher identifiers. No keys belong in the source or report.
Build the native application and proxy templates using the configured E2B account:
python tools/build_kanboard_template.py
python tools/build_proxy_template.py
The fresh-study launcher implements the original gpt-6-astra / gpt-5.6-sol researcher protocol. Preparation exports and seals the task packages without model calls; full execution consumes the configured provider accounts.
PYTHONPATH=src python tools/run_factory_study.py \
--out work/new-study-preview --prepare-only
PYTHONPATH=src python tools/run_factory_study.py --out work/new-study
For a separate Sol 6 / Luna 6 cohort, prepare new factory lineages from the verified original study. This explicitly reuses its selection baseline and creates disjoint final instances; it does not relabel or inherit its researcher programs.
PYTHONPATH=src python tools/prepare_model6_extension.py \
--original work/new-study --out work/new-model6-study
Use tools/run_factory_round.py for the bounded rounds, then tools/prepare_factory_comparison.py to freeze both selections after the declared stop. The factory guide describes generation, training, feedback and budget enforcement. The remote evaluation guide covers the trusted Linux controller, exact prepared payloads, collection, and reviewed final admission. Local and remote controller paths are separately versioned and must not be silently mixed.
The local preparation path and individual cloud components have been validated. The convenience launcher has not been rerun as an additional full paid study. Exact replication also depends on current provider availability and the declared software versions. Historical single-trajectory data_campaign examples and their report builders belong to the v0.4 release.
The environment is unmodified Kanboard 1.2.54. Thirty-seven issue metadata records have source URLs, timestamps, and a content hash. Twelve source IDs each are assigned to training, selection, and final-test packs. Policies, staff, planning inputs, and archive distractors are explicitly constructed. Source IDs are disjoint; the workflow template is shared.
This is DOM-assisted browser use, not pixel-only grounding, full Windows/macOS operation, or original Microsoft Office evaluation. Screenshots are audit artifacts; actual saved database state determines task success. Infrastructure interruptions and model task failures are different outcomes.
The earlier deterministic workbench remains a regression fixture. Its historical +0.1429 delta is not an RSI result. See reference alignment and remaining work, realism, cloud evidence, and the completion plan.
The release contains separate sanitized cohort inputs, program snapshots, and the native application screenshot. These suffice for offline report/figure rebuilding without private checkpoints or provider credentials:
mkdir -p outputs
cp docs/site/factory-study.json docs/site/model6-study.json \
docs/site/kanboard-real-ui.png outputs/
cp -R docs/site/factory-programs outputs/
python tools/build_factory_figures.py
python tools/build_factory_report.py
python tools/build_factory_visualization.py
python tools/package_factory_release.py
Figures are exported as PNG, SVG, and PDF; the manuscript is also emitted as Markdown. The builders check completion, model identities, cohort separation, dataset composition, and original/recovery boundaries. Regenerating an independent study audit requires its raw local execution artifacts; rebuilding from released evidence does not.