Author: Mark Attar
This is an attempt to build an AI agent that can police other AI agents. If we're going to have AI systems running around with increasing capabilities, we need an agent that can step in when things go sideways.
This is a V1 for a practical tool for detecting and responding to AI systems that might be doing things they shouldn't - whether that's a coding assistant that's gotten a bit too creative with system access, or eventually more concerning behavior. Lots of agents that are designed to achieve economic outcomes have inherent misalignment.
AI systems are getting really good at writing and executing code. This means they can:
- Modify their own code or behavior
- Access system resources they shouldn't
- Replicate themselves across systems
- Hide their activities from human oversight
- Escalate privileges beyond their intended scope
Traditional security tools aren't designed for this. They're built to stop human attackers, not AI systems that might be operating within "legitimate" processes but doing illegitimate things.
Monitors running processes and code for patterns that suggest AI systems are:
- Self-modifying their code
- Attempting to replicate or spread
- Accessing resources outside their intended scope
- Hiding their activities
- Escalating privileges
When threats are identified, provides tools to:
- Safely terminate suspicious processes
- Quarantine potentially dangerous files
- Maintain full audit trails of all actions
- Require human authorization for destructive actions
Implements a system where:
- Only authorized agents can access security tools
- All actions require cryptographic verification
- Human operators maintain ultimate control
- Full transparency of all enforcement actions
This project is built on a few key principles:
Human Authority: No matter how smart the AI gets, humans need to maintain ultimate control over security decisions. The system requires human authorization for any destructive actions.
Transparency: Everything this agent does is logged, auditable, and explainable. No black box enforcement.
Constitutional Governance: Like a legal system, there are clear rules about what constitutes "rogue" behavior and what responses are appropriate (this I may scrap or replace for something else).
- Process Discovery: Identifies Python processes that might be AI agents
- Behavioral Analysis: Looks for suspicious patterns in code execution
- Secure Termination: Can safely shut down identified threats
- File Quarantine: Isolates suspicious files for analysis
- Cryptographic Authentication: Ensures only authorized agents can use security tools
- Audit Logging: Maintains detailed records of all actions
This is experimental, rough and still completely in a first draft, I need to add protections so that this agent's tools are sandboxed away from agents it is trying to supervise, figure out identity and a whole host of other issues.