Skip to content

[Feature Request] Ideas for Improving ARTEX Reliability, Security, and Observability #310

Open

Description

@M34nX

Hello ARTEX team,

After reviewing the project's documented architecture and capabilities, I would like to propose 10 ideas that could further improve ARTEX's reliability, security governance, and overall observability.

Suggested Features

  1. Deterministic Task Replay — Allow archived tasks to be replayed in a controlled environment to investigate behavior and reproduce results.

  2. Security Benchmark Lab — Introduce a standardized benchmark suite to evaluate system reliability, security controls, and performance across versions.

  3. Policy Simulation Engine — Preview how scope rules, permissions, and execution policies affect a task before it starts.

  4. Tamper-Evident Evidence Ledger — Maintain verifiable integrity records for collected evidence and important task events.

  5. Cross-Version Regression Testing — Compare task outcomes and system behavior across releases to detect unintended regressions.

  6. Finding Intelligence Deduplication — Identify and group duplicate or closely related findings across tasks while preserving their original evidence.

  7. Explicit Resumability and Recovery Observability — Make recovery state, resumability decisions, and interrupted-task recovery outcomes easier to inspect.

  8. Outcome-Aware LLM Routing — Evaluate model selection using task outcomes, reliability, latency, and cost rather than availability alone.

  9. Authorization Evidence Binding — Associate task execution records with the relevant authorization and scope evidence for stronger auditability.

  10. Sensitive Evidence Governance — Provide configurable retention, redaction, access control, and lifecycle policies for sensitive evidence.

Suggested Priorities

  • Deterministic Task Replay
  • Policy Simulation Engine
  • Security Benchmark Lab

These are exploratory proposals rather than claims that the current implementation lacks these capabilities. Some may already exist partially or be planned.

Thank you for developing ARTEX. I hope these suggestions contribute to its future roadmap.

Activity

  1. github-actions commented on Nov 17, 2026

    @github-actions

    This is a feature request proposing a roadmap of ten enhancements — spanning deterministic task replay, a security benchmark lab, policy simulation, tamper-evident evidence ledgers, cross-version regression testing, finding deduplication, recovery observability, outcome-aware LLM routing, authorization evidence binding, and sensitive evidence governance — with replay, policy simulation, and benchmarking suggested as top priorities. As the report notes, several of these may already be partially implemented (for example, the repository appears to include benchmark tooling under scripts/benchmark and evidence-related packages, though I'm not certain how far they cover the proposals), so it would help maintainers if the request were split into individual issues, each naming the specific workflow that would benefit and the observable outcome expected, so they can be triaged and prioritized separately.


    🤖 automated reply from the repo bot. A human maintainer reads everything. Ping me in any comment with @artex-bot.

  2. M34nX commented on Nov 17, 2026

    @M34nX
    Author

    Thank you for reviewing my suggestions. I agree that splitting them into separate, focused feature requests would make them easier to evaluate and prioritize.

    I would suggest starting with these three priorities:

    1. Deterministic Task Replay
    2. Policy Simulation Engine
    3. Security Benchmark Lab

    Each issue could describe a specific use case, the proposed workflow, and the expected observable outcome.

    I also understand that some of these capabilities may already exist partially. Reviewing the current implementation first would help avoid duplication and identify meaningful improvements.

    Thank you for your consideration and for maintaining ARTEX.

  3. BaaRai commented on Nov 17, 2026

    @BaaRai
    Owner

    Thanks for writing this up, and for agreeing to split it. Before the three issues, one thing would save us both time: a few of these already exist in some form, so a proposal should start from the current baseline rather than from zero.

    • Security Benchmark Lab: scripts/benchmark and BENCHMARKS.md already run a reproducible corpus and publish results, so the question is what your lab adds to them.
    • Replay: POST /api/traffic/replay replays a recorded HTTP exchange, and task archives plus session resume restore a task. Neither replays an agent run, so deterministic task replay is the genuinely new part.
    • Finding deduplication: findings groups already group related findings in the UI; deduplication across tasks is the new part.
    • Outcome-aware LLM routing: multi-LLM rotation exists, but it routes on availability and health, not on outcomes.
    • Recovery observability: task archives, session resume and the drain path exist; an explicit recovery state and decision view is the new part.

    Please open three focused issues, one per priority, each with the workflow that benefits, what should be observable, and how we would know it works. For deterministic replay, say what deterministic means for an agent that calls external models: replaying the model responses and replaying the HTTP exchanges are different designs with different guarantees. That one needs a definition before code.

    We will triage each issue on its own. This is not a roadmap commitment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions