feat(dea): add autopush option to model_env and update documentation - #1

Open

graceqi-g wants to merge 364 commits into

mainfrom

feat/dea-model-env-autopush

This pull request is big! We're only showing the most recent 250 commits

Commits on Jun 4, 2026

chore: increase persistent volume claim storage request to 1000Gi
Ismail Mehdi
committed
228bf47
Merge branch 'main' of github.com:GoogleCloudPlatform/evalbench
Ismail Mehdi
committed
56c3c10
Merge branch 'main' into support-agy
omkargaikwad23
committed
9e0a7c9
refactor: move agy binary installation to per-session sandbox and remove host PATH dependency
omkargaikwad23
committed
5de2fac
refactor: remove Secret Manager support for agy authentication and update subprocess calls to use stdin=DEVNULL
omkargaikwad23
committed
c20f942
refactor: validate MCP tool schema files by content to ignore non-tool JSON artifacts
omkargaikwad23
committed
e82ccb9
chore: comment out token_consumption in agy-cli-tools config files due to missing usage data
omkargaikwad23
committed
c58575d
refactor: update agy CLI harness to use --model flag for model selection and support v1.0.5+ binary changes
omkargaikwad23
committed
1a4aaf2
feat: recover resolved model label from agy cli logs for statistics bucket tagging
omkargaikwad23
committed
ee7635f
refactor: modularize agy-cli transcript parsing and improve tool call pairing and latency tracking logic
omkargaikwad23
committed
a85ee63

Commits on Jun 5, 2026

Commits on Jun 8, 2026

Commits on Jun 10, 2026

fix(dea): resolve concurrency deadlock in GcpAdcCredentialService (#422)

minzznguyen
and
James Nguyen
authored
6619ae5
Spanner GoogleSQL fixes (#418)

authored
deba199

Commits on Jun 11, 2026

Commits on Jun 12, 2026

Implement Data Engineering Agent generator, orchestrator, Evaluator, and robustness (#426)

authored
fc80051

Commits on Jun 16, 2026

chore(deps): bump pyjwt from 2.12.0 to 2.13.0
dependabot[bot]
authored
60b2ee3
Add tool's responses to conversation history (#423)

authored
edf613f
chore(deps): bump aiohttp from 3.13.4 to 3.14.1
dependabot[bot]
authored
485ce88
Merge pull request #430 from GoogleCloudPlatform/dependabot/uv/aiohttp-3.14.1
omkargaikwad23
authored
4d823d9
chore(deps): bump python-multipart from 0.0.26 to 0.0.31
dependabot[bot]
authored
03db0be
chore(deps): bump cryptography from 46.0.7 to 48.0.1
dependabot[bot]
authored
ee7717f
Package DEA scenarios in one request for sequential execution (#428)

minzznguyen
and
James Nguyen
authored
71c9651

Commits on Jun 17, 2026

Merge branch 'main' into dependabot/uv/cryptography-48.0.1
IsmailMehdi
authored
e911f09

Commits on Jun 18, 2026

Commits on Jun 19, 2026

Commits on Jun 20, 2026

chore(deps): bump pydantic-settings from 2.13.1 to 2.14.2
dependabot[bot]
authored
db46f08

Commits on Jun 22, 2026

Commits on Jun 23, 2026

Commits on Jun 24, 2026

fix(dea): accumulate all agent text parts and increase timeout (#456)

minzznguyen
and
James Nguyen
authored
cf3f758
Merge branch 'main' into dependabot/uv/pydantic-settings-2.14.2
omkargaikwad23
authored
043f63c
Merge pull request #451 from GoogleCloudPlatform/dependabot/uv/pydantic-settings-2.14.2
omkargaikwad23
authored
6dff91e
Change Config to upload conversation history to BigQuery (#457)

authored
0505fcb

Commits on Jun 26, 2026

feat(util): add DataformHelper utility for managing cloud sandboxes and mock-based unit tests (#458)

minzznguyen
and
James Nguyen
authored
aa1642a

Commits on Jun 27, 2026

## Fix Sandbox Isolation Violation (cwd omission) in Codex CLI Generator (#461)

silviojr
and
Silvio Junior
authored
de397c0

Commits on Jun 29, 2026

feat(mcp-readability): add MCP tool man-page formatter
Akangsha Goel
committed
3e4c16e
feat(mcp-readability): add McpToolsGenerator with pluggable tools_source
Akangsha Goel
committed
66b67d5
fix: tight copy of claude settings from local environment (#467)
helloeve
authored
474a9be
Merge branch 'main' into feat/mcp-readability
akangsha7
authored
aa3b9e7
Merge branch 'main' into feat/mcp-readability-generator
akangsha7
authored
b3f35b2
Fixing issues in BQ eval

Hailong Li
authored and
totoleon
committed
c1cc66b
Merge pull request #468 from GoogleCloudPlatform/hailong-wip-bq
totoleon
authored
5f2b0ed
Merge pull request #463 from akangsha7/feat/mcp-readability
akangsha7
authored
620cc0b
Merge branch 'main' into feat/mcp-readability-generator
akangsha7
authored
aebf5b1
feat(mcp-readability): add sample run config, endpoints, style guide, exceptions
Akangsha Goel
committed
a21ec5e
refactor(mcp-readability): fold endpoint_url into tools_source
Akangsha Goel
committed
f84de29

Commits on Jun 30, 2026

Commits on Jul 1, 2026

Commits on Jul 2, 2026

feat(util): update DataformHelper utility with high-level workspace lifecycle methods (#466)

authored
9a75a7a

Commits on Jul 7, 2026

Commits on Jul 8, 2026

add extension support
Silvio Junior
committed
4858b08
feat(dea): implement GCS archival for DataEngineeringAgent (#476)

minzznguyen
and
James Nguyen
authored
89c24e4
feat(dea): adjust conversation turns and plan for programmatic scenarios (#479)

minzznguyen
and
James Nguyen
authored
85b90ec
feat(query_data_api): support api_endpoint override in model config (#480)
wangauone
authored
0d59e90

Commits on Jul 9, 2026

fix(dataform): synchronize GCP project env vars in dataform scorer
graceqi-g
committed
be31ac6
Merge pull request #481 from graceqi-g/fix-dataform-scorer-gke-env
prernakakkar-google
authored
de7dc22
Merge pull request #472 from akangsha7/feat/mcp-readability-compliance-eval
prernakakkar-google
authored
fefb034
Fix Claude Code + Google MCP auth (DCR error) and tool-result parsing
prernakakkar-google
committed
904c5d6
feat(dea): support dynamic workspace lifecycle and parameter passing in evaluator, generator, and scorers (#474)

authored and
prernakakkar-google
committed
b5e97f0
feat: Support cross-database evaluation with SQLite ground truth (#465)

authored and
prernakakkar-google
committed
e71692c
feat(dea): implement GCS archival for DataEngineeringAgent (#476)

authored and
prernakakkar-google
committed
5e36350
feat(dea): adjust conversation turns and plan for programmatic scenarios (#479)

authored and
prernakakkar-google
committed
7434a91
feat(query_data_api): support api_endpoint override in model config (#480)

wangauone
authored and
prernakakkar-google
committed
3f1868c
fix(dataform): synchronize GCP project env vars in dataform scorer

graceqi-g
authored and
prernakakkar-google
committed
a15a54f
feat(mcp-readability): compliance orchestrator, LLM judge, and metrics scorer

Akangsha Goel
authored and
prernakakkar-google
committed
52ffade
Merge branch 'main' into fix-claude-auth
prernakakkar-google
committed
d7fa036
Add Haoyu and Prerna as Code Owners
prernakakkar-google
committed
7d3564a
Merge pull request #478 from GoogleCloudPlatform/fix-claude-auth
prernakakkar-google
authored
5178bb4
Merge pull request #482 from GoogleCloudPlatform/owners
prernakakkar-google
authored
79d31a2
fix(deps): add mcp to pyproject.toml dependencies
Akangsha Goel
committed
5928137
Merge branch 'main' into fix/add-mcp-dependency
akangsha7
authored
f9bfa27
Add effective billed tokens and total tokens processed scorers.
prernakakkar-google
committed
ed1d27c
Merge branch 'main' into scorers
prernakakkar-google
committed
3345481
Merge pull request #483 from GoogleCloudPlatform/fix/add-mcp-dependency
prernakakkar-google
authored
62c7436
Potential fix for pull request finding 'Empty except'

prernakakkar-google
and
github-code-quality[bot]
authored
0d70827
Potential fix for pull request finding 'Empty except'

prernakakkar-google
and
github-code-quality[bot]
authored
8d11552
Merge branch 'main' into scorers
prernakakkar-google
authored
f184864
Merge pull request #484 from GoogleCloudPlatform/scorers
prernakakkar-google
authored
1929a27
Add ADC support for onemcp+codex
prernakakkar-google
committed
ff0b9bf
Merge branch 'main' into codex-adc
prernakakkar-google
committed
d4cf28d
Merge pull request #485 from GoogleCloudPlatform/codex-adc
prernakakkar-google
authored
6d179bb
chore(deps): bump starlette from 0.52.1 to 1.3.1
dependabot[bot]
authored
cd8528e

Commits on Jul 10, 2026

feat(dea): support static workspaces (Mode 2) and update README (#486)

minzznguyen
and
James Nguyen
authored
39cb2b8
fix(reporting): Add retry logic and configurable chunk size for BigQuery writes
erinlimbogoogle
committed
8774d02
Merge branch 'main' into fix-evaluator-timeout
erinlimbogoogle
authored
4960b41

Commits on Jul 12, 2026

feat: Add hybrid evaluation Python notebook (#487)
arieljassan
authored
ace0149
docs(dea): add automated suite run section to README (#489)

minzznguyen
and
James Nguyen
authored
15881cf

Commits on Jul 13, 2026

Merge branch 'main' into dependabot/uv/starlette-1.3.1
prernakakkar-google
authored
34b151d
Merge pull request #488 from erinlimbogoogle/fix-evaluator-timeout
erinlimbogoogle
authored
6b5f0a6
addresses reviewer's comments
Silvio Junior
committed
0dccc86
fix lint
Silvio Junior
committed
f84ffec
feat(dea): support dynamic workspace environment files mapping and injection (#491)

minzznguyen
and
James Nguyen
authored
a409608

Commits on Jul 14, 2026

chore(main): release 1.10.0 (#440)
release-please[bot]
authored
2e71a59

Commits on Jul 16, 2026

Commits on Jul 17, 2026

feat(dea): parameterize url_agent_name and agent_type_uri options (#497)

minzznguyen
and
James Nguyen
authored
30a065f

Commits on Jul 20, 2026

Add new scorer to count steps agent took to answer
prernakakkar-google
committed
df9142c
Merge branch 'main' into improve-turn-scorer
prernakakkar-google
committed
4662eb9
feat(mcp_readability): human-readable LLM feedback without score, with waivers
Akangsha Goel
committed
ffa4d1f
fea(dea): Add support for tools metrics for DEA (#494)

minzznguyen
and
James Nguyen
authored
381a3b0
Merge pull request #506 from GoogleCloudPlatform/evalbench/mcp-readability-feedback-format
akangsha7
authored
c49f45f

Commits on Jul 21, 2026

Commits on Jul 22, 2026

feat(query_data_api): add option to use REST API fallback for unreleased proto fields (#503)
g-lynnzee
authored
b29d37f
chore(main): release 1.12.0
release-please[bot]
authored
89ff052
docs: update agy-cli documentation and comments regarding token usage and error surfaces
omkargaikwad23
committed
a0bc8b8
feat(query_data_api): add option to use REST API fallback for unreleased proto fields (#503)

g-lynnzee
authored and
omkargaikwad23
committed
3e3a867
feat: add support for --print-timeout and pin --log-file to a deterministic path in AgyCliGenerator
omkargaikwad23
committed
b63bab8
Merge remote-tracking branch 'origin/main' into agy_output_format
omkargaikwad23
committed
ab8497f
feat: enable token_consumption tracking in example run configuration
omkargaikwad23
committed
086935a
docs: remove token count mention from agent testing documentation
omkargaikwad23
committed
be1c9eb
Merge pull request #511 from GoogleCloudPlatform/release-please--branches--main--components--evalbench
prernakakkar-google
authored
2aaae61
chore(deps): bump gitpython from 3.1.50 to 3.1.52
dependabot[bot]
authored
fe16b94
chore(deps): bump pyasn1 from 0.6.3 to 0.6.4
dependabot[bot]
authored
df9557b
Merge branch 'main' into evalbench/mcp-readability-gemini-3.1
prernakakkar-google
authored
35c2379
Merge pull request #513 from GoogleCloudPlatform/dependabot/uv/gitpython-3.1.52
prernakakkar-google
authored
63bb7a7
Merge pull request #514 from GoogleCloudPlatform/dependabot/uv/pyasn1-0.6.4
prernakakkar-google
authored
8ef9bc5
fix: skip non-object lines in agy stream and report model errors based on result status
omkargaikwad23
committed
e3ac999
Merge remote-tracking branch 'origin/main' into agy_output_format
omkargaikwad23
committed
b14a01f
refactor: remove print_timeout configuration support from agy_cli generator
omkargaikwad23
committed
4ba9e2f
fix: add type checking to stream event parsing to safely handle non-dict payloads
omkargaikwad23
committed
1003e08
Merge pull request #512 from GoogleCloudPlatform/agy_output_format
prernakakkar-google
authored
d403cca
Add missing parameter
erinlimbogoogle
committed
33d0005
fix gcloud credential issues
Silvio Junior
committed
a7d9c75
extract function
Silvio Junior
committed
42f7fc4
fix codestyle
Silvio Junior
committed
ff3489f

Commits on Jul 23, 2026

Commits on Jul 24, 2026

antigravity timeout
Silvio Junior
committed
09f9ec1
address comments
Silvio Junior
committed
8d5c33e
feat: make the BigQuery dataset name configurable via reporting.table_name
omkargaikwad23
committed
c0830de
Merge pull request #519 from GoogleCloudPlatform/fix_timeout
prernakakkar-google
authored
e674a8f
chore(main): release 1.13.0
release-please[bot]
authored
af3f0b8
Merge branch 'main' into feat/bq-configurable-dataset
prernakakkar-google
authored
40a8775
Merge pull request #521 from GoogleCloudPlatform/release-please--branches--main--components--evalbench
prernakakkar-google
authored
ee0801c
Merge pull request #518 from GoogleCloudPlatform/fix_credentials
silviojr
authored
968c46b
feat: allow customization of BigQuery dataset name via run configuration
omkargaikwad23
committed
daa0426
Merge branch 'main' into feat/bq-configurable-dataset
omkargaikwad23
authored
1357867
Merge pull request #522 from GoogleCloudPlatform/feat/bq-configurable-dataset
omkargaikwad23
authored
ee85f96
feat(dea): add autopush option to model_env and update documentation
graceqi-g
committed
143bd7a