You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature Request] Ideas for Improving ARTEX Reliability, Security, and Observability #310
After reviewing the project's documented architecture and capabilities, I would like to propose 10 ideas that could further improve ARTEX's reliability, security governance, and overall observability.
Suggested Features
Deterministic Task Replay — Allow archived tasks to be replayed in a controlled environment to investigate behavior and reproduce results.
Security Benchmark Lab — Introduce a standardized benchmark suite to evaluate system reliability, security controls, and performance across versions.
Policy Simulation Engine — Preview how scope rules, permissions, and execution policies affect a task before it starts.
Tamper-Evident Evidence Ledger — Maintain verifiable integrity records for collected evidence and important task events.
Cross-Version Regression Testing — Compare task outcomes and system behavior across releases to detect unintended regressions.
Finding Intelligence Deduplication — Identify and group duplicate or closely related findings across tasks while preserving their original evidence.
Explicit Resumability and Recovery Observability — Make recovery state, resumability decisions, and interrupted-task recovery outcomes easier to inspect.
Outcome-Aware LLM Routing — Evaluate model selection using task outcomes, reliability, latency, and cost rather than availability alone.
Authorization Evidence Binding — Associate task execution records with the relevant authorization and scope evidence for stronger auditability.
Sensitive Evidence Governance — Provide configurable retention, redaction, access control, and lifecycle policies for sensitive evidence.
Suggested Priorities
Deterministic Task Replay
Policy Simulation Engine
Security Benchmark Lab
These are exploratory proposals rather than claims that the current implementation lacks these capabilities. Some may already exist partially or be planned.
Thank you for developing ARTEX. I hope these suggestions contribute to its future roadmap.
This is a feature request proposing a roadmap of ten enhancements — spanning deterministic task replay, a security benchmark lab, policy simulation, tamper-evident evidence ledgers, cross-version regression testing, finding deduplication, recovery observability, outcome-aware LLM routing, authorization evidence binding, and sensitive evidence governance — with replay, policy simulation, and benchmarking suggested as top priorities. As the report notes, several of these may already be partially implemented (for example, the repository appears to include benchmark tooling under scripts/benchmark and evidence-related packages, though I'm not certain how far they cover the proposals), so it would help maintainers if the request were split into individual issues, each naming the specific workflow that would benefit and the observable outcome expected, so they can be triaged and prioritized separately.
🤖 automated reply from the repo bot. A human maintainer reads everything. Ping me in any comment with @artex-bot.
Thank you for reviewing my suggestions. I agree that splitting them into separate, focused feature requests would make them easier to evaluate and prioritize.
I would suggest starting with these three priorities:
Deterministic Task Replay
Policy Simulation Engine
Security Benchmark Lab
Each issue could describe a specific use case, the proposed workflow, and the expected observable outcome.
I also understand that some of these capabilities may already exist partially. Reviewing the current implementation first would help avoid duplication and identify meaningful improvements.
Thank you for your consideration and for maintaining ARTEX.
Thanks for writing this up, and for agreeing to split it. Before the three issues, one thing would save us both time: a few of these already exist in some form, so a proposal should start from the current baseline rather than from zero.
Security Benchmark Lab: scripts/benchmark and BENCHMARKS.md already run a reproducible corpus and publish results, so the question is what your lab adds to them.
Replay: POST /api/traffic/replay replays a recorded HTTP exchange, and task archives plus session resume restore a task. Neither replays an agent run, so deterministic task replay is the genuinely new part.
Finding deduplication: findings groups already group related findings in the UI; deduplication across tasks is the new part.
Outcome-aware LLM routing: multi-LLM rotation exists, but it routes on availability and health, not on outcomes.
Recovery observability: task archives, session resume and the drain path exist; an explicit recovery state and decision view is the new part.
Please open three focused issues, one per priority, each with the workflow that benefits, what should be observable, and how we would know it works. For deterministic replay, say what deterministic means for an agent that calls external models: replaying the model responses and replaying the HTTP exchanges are different designs with different guarantees. That one needs a definition before code.
We will triage each issue on its own. This is not a roadmap commitment.
Activity
github-actions commentedon Nov 17, 2026
This is a feature request proposing a roadmap of ten enhancements — spanning deterministic task replay, a security benchmark lab, policy simulation, tamper-evident evidence ledgers, cross-version regression testing, finding deduplication, recovery observability, outcome-aware LLM routing, authorization evidence binding, and sensitive evidence governance — with replay, policy simulation, and benchmarking suggested as top priorities. As the report notes, several of these may already be partially implemented (for example, the repository appears to include benchmark tooling under scripts/benchmark and evidence-related packages, though I'm not certain how far they cover the proposals), so it would help maintainers if the request were split into individual issues, each naming the specific workflow that would benefit and the observable outcome expected, so they can be triaged and prioritized separately.
🤖 automated reply from the repo bot. A human maintainer reads everything. Ping me in any comment with
@artex-bot.M34nX commentedon Nov 17, 2026
Thank you for reviewing my suggestions. I agree that splitting them into separate, focused feature requests would make them easier to evaluate and prioritize.
I would suggest starting with these three priorities:
Each issue could describe a specific use case, the proposed workflow, and the expected observable outcome.
I also understand that some of these capabilities may already exist partially. Reviewing the current implementation first would help avoid duplication and identify meaningful improvements.
Thank you for your consideration and for maintaining ARTEX.
BaaRai commentedon Nov 17, 2026
Thanks for writing this up, and for agreeing to split it. Before the three issues, one thing would save us both time: a few of these already exist in some form, so a proposal should start from the current baseline rather than from zero.
scripts/benchmarkandBENCHMARKS.mdalready run a reproducible corpus and publish results, so the question is what your lab adds to them.POST /api/traffic/replayreplays a recorded HTTP exchange, and task archives plus session resume restore a task. Neither replays an agent run, so deterministic task replay is the genuinely new part.Please open three focused issues, one per priority, each with the workflow that benefits, what should be observable, and how we would know it works. For deterministic replay, say what deterministic means for an agent that calls external models: replaying the model responses and replaying the HTTP exchanges are different designs with different guarantees. That one needs a definition before code.
We will triage each issue on its own. This is not a roadmap commitment.