She’ll module improvements

CODEBASE.md

@@ -331,7 +331,21 @@ Cursors (`dm:<uid>`, `group:<cid>`, `thread:<tid>`) are persisted in
 
 ### `system_prompt` — injects SOUL.md, AGENTS.md,, TOOLS.md into every system prompt via `register_prompt` providers.
 
-### `rag` — indexes `workspace/memory/*.md` files; auto-injects relevant chunks each turn (BM25 or embedding cosine similarity); provides `memory_search` tool; triggers background memory consolidation when context budget is near.
+### `rag` — indexes named databank folders under `workspace/rag/` (BM25 or embedding cosine similarity via `rag_search`/`set_auto_rag_databanks` tools). Any `*.md` file may open with a YAML frontmatter header (`modules/rag/lorefile.py`: `name`, `mode`, `keys`, `secondary_keys`, `constant`, `selective`, `selective_logic`, `case_sensitive`, `whole_words`, `disabled`) declaring it a keyword-triggered lore entry — files with no frontmatter are indexed as plain text, default to `mode: hybrid`, and never keyword-trigger (no keys), unchanged from before. Frontmatter is stripped before chunking so only the body is embedded/BM25-indexed (prefixed with name+keys for recall). Legacy SillyTavern lorebook/worldinfo JSON dropped at the root of a databank dir is auto-converted the first time it's discovered — `discover_databanks()` writes one native lore `.md` doc per active entry into a same-named folder (`lorefile.convert_lorebook_json`), then renames the original to `<name>.json.bak` (never deletes) so it's never re-converted; from then on it's an ordinary folder databank, no JSON support at runtime. `FilesDataBank` (`modules/rag/databanks.py`) is the only databank kind — `LoreBookDataBank` no longer exists, and there's no separate `DataBank` protocol either; callers type against `FilesDataBank` directly.
+
+**Auto-inject is both keyword-triggered and semantic, gated per-entry by `mode`.** `FilesDataBank.auto_inject(text, store=None, embedder=None, top_k=0, bm25_weight=0.3)` is `async`: it runs ST-style keyword/regex matching (ported selectiveLogic AND_ANY/NOT_ALL/NOT_ANY/AND_ALL) over frontmatter entries deterministically, and when a `store` is passed it additionally runs the same hybrid BM25+vector search `rag_search` uses over the whole databank index, merging in new hits deduplicated by resolved file path. Each entry's frontmatter `mode` (`lorefile.LoreEntry.mode`, default `"hybrid"`) decides which half applies to it:
+  - `hybrid` (default) — deterministic firing AND eligible for the passive semantic merge.
+  - `vector` — semantic-only; skips keyword/regex matching entirely regardless of `keys`/`constant`.
+  - `keyword` — deterministic-only via literal substring/whole-word `keys`; excluded from the semantic merge so it never surfaces passively without a keyword hit.
+  - `regex` — like `keyword`, but `keys` (and `secondary_keys`) are compiled as regular expressions (`lorefile.compile_regex_key` — accepts SillyTavern's `/pattern/flags` syntax, or a bare pattern) instead of literal substrings; also excluded from the semantic merge.
+
+  `rag_search` (the explicit tool) is unaffected by `mode` — it always searches the full hybrid index regardless. Legacy lorebook conversion (`convert_lorebook_json`) auto-assigns `mode: regex` when any of an entry's keys use ST's `/pattern/flags` syntax, else `mode: keyword` (converted entries never default to `hybrid`/`vector` since legacy lorebooks have no concept of embeddings — edit the converted `.md` by hand to opt in). The pre-assemble hook in `modules/rag/__main__.py` syncs each auto-rag target's indexer before calling `auto_inject`, same as `rag_search` does.
+
+**`FilesDataBank` caches parsed entries by `(path, mtime)`** (`_entry_cache` in `databanks.py`) so `iter_files()`/`auto_inject()` — the latter invoked once per user turn per auto-rag target — don't re-read and re-parse every file in the folder when nothing on disk changed; stale cache entries are dropped when a file disappears or falls outside the indexed extensions.
+
+**Auto-rag targets have a configurable default.** `default_auto_targets` (list of databank names, default `[]`) in `EXTENSION_META`'s `default_config` is used by the pre-assemble hook when a branch's `rag_auto_targets` session-state key was never written (`get_state` returns `None`) — once `set_auto_rag_databanks` is called on a branch, even with `[]` to explicitly clear it, that stored value wins over the config default from then on.
+
+**Module state in `modules/rag/__main__.py`** is one `_RagState` dataclass instance (`_state`) instead of eight separate module globals (`initialized`, `stores`, `indexers`, `databanks`, `embedder`, `cfg`, `workspace`, `strategy`, `model_name`).
 
 ### `memory` (v2) — scoped LadybugDB property-graph knowledge store at `<instance>/data/memory/memory.lbug` (not workspace/). See `modules/memory/PLAN.md` for the full design.
 
@@ -357,7 +371,22 @@ Superseded modules `decay.py` / `dedup_agents.py` / `librarian_agents.py` are in
 
 ### `filesystem` — `view`, `write_file`, `edit_file`, `grep`, `glob_search` tools. Write tools require prior `view()` to prevent blind overwrites. `write_file`/`edit_file` are sandboxed to `workspace/` only. `view`/`grep`/`glob_search` can additionally reach any directory listed in `filesystem.read_only_paths` in config.yaml (e.g. `/app` for the agent's own source in the container) — read-only, never writable; nothing outside workspace/ + that whitelist is reachable at all. `view()` detects image files by extension (`.jpg/.jpeg/.png/.gif/.webp`) and returns them as `IMAGE_BLOCK_PREFIX` sentinels; `agent._execute_tool` unwraps these, converts to PNG via Pillow, and injects a follow-up user turn with an `image_url` block. If PNG conversion fails (Pillow unavailable), the image path falls back to a plain text description rather than sending an unconverted format to the backend (which would cause a misleading "mmproj is missing" backend error).
 
-### `shell` — `shell` tool. Runs in workspace directory. Maintains a blacklist of dangerous commands.
+### `shell` — `shell` tool. Runs in workspace directory. Linux only (PowerShell support and `_normalize_windows` were removed with the container refactor).
+
+**Command policy is AST-based (v2).** `validate.py` parses the command with `tree-sitter-bash` and checks each *resolved command* in it separately; `policy.py` compiles the rules from YAML. This replaced substring-glob `blacklist.txt` / `whitelist.txt`, which could not distinguish a command from a quoted argument containing the same text — `echo "i"; echo "am"; echo "harmless"` was blocked, and `git commit -m "don't rm -rf your repo"` was too. Design of record: `modules/shell/PLAN.md`.
+
+- Two policy files, both shipped in the module, overridable via `extra.shell.policy.{deny,allow}` (typically pointed at the instance dir, mounted read-only): `deny.yaml` (`default_action: allow` — deny-list, for callers at/above `neutral`) and `allow.yaml` (`default_action: deny` — allow-list with per-command `subcommand`/`allowed_flags`/`max_args`/`arg_matches` constraints, for callers below `neutral`). Deny beats allow. Loaded once, cached by `(path, workspace)`.
+- **Policies compose via `extends:`** — `builtin:allow`, `builtin:deny`, or a path (relative to the extending file). An instance adds one rule without vendoring the whole file. Base rules come first, then the child's; a matching `id` replaces a base rule, `disable: [id, ...]` drops one (unknown ids raise, so a typo can't silently leave a rule in force). `default_action` and `constructs` are inherited, declared constructs merge over the base, and a child cannot flip `default_action` (the two postures use different rule schemas). Cycles raise. Worked example: `modules/shell/example.instance-allow.yaml` (re-enables `ps` with its `-o` columns constrained to non-argv fields).
+- `allow.yaml`'s bar is "a stranger on Discord could run this": nothing on it can read a file, the network, or the environment, so the only data enterable at that tier is text the caller typed. That excludes `cat`/`grep`/`ls`/`ps`/`env`/`whoami`. Enforced by `max_args: 0` on the stdin filters (they cannot name a file) and explicit `allowed_flags` everywhere (omitting the key permits no flags — which is what keeps `sort -o FILE`, `date -s`, and `wc --files0-from=F` out). Deliberately stricter than the sandbox's actual threat model; loosen per-instance via `extends`, not by editing it.
+- Rules match on parsed structure: `command` (basename), `subcommand`, `any_flag`/`all_flags`, `no_operands`, `path_under`/`path_outside`/`redirect_under`. Flags are normalized, so `-rf`, `-fr` and `-r -f` are one rule. `action: warn` replaced the old `_DESTRUCTIVE` regex list — warnings run and prefix the output.
+- Each YAML also carries a `constructs:` map of permitted bash node types. **Anything not listed is denied**, so a `tree-sitter-bash` grammar upgrade fails closed; hence the version pin in `pyproject.toml`. `function_definition` is denied at every tier (rebinding `rm()` would defeat a name-based rule). At the allow-list tier substitutions, expansions, redirects and backgrounding are all denied, which is what makes injection structurally impossible and retires the old `{arg}` character-class hack — quoted text is a parser leaf, so callers get their metacharacters back as data.
+- **Fails closed:** a missing/malformed policy blocks every command (the old `blacklist.txt` did the opposite — missing file meant unrestricted). Not configurable. Also denied: commands whose name isn't a literal word (`$CMD`, `$(echo rm)`), and any input with a parse error.
+- **Explicitly out of scope:** obfuscation and interpreter escape. No recursive re-parsing of `bash -c` payloads, no base64 decoding. Flat rules catch the unsubtle cases (`bash -c`, `python -c`, pipe-to-shell via `no_operands`); nothing more is claimed. Backgrounding (`&`) is allowed at `neutral` so the agent can start long jobs. The container is the security boundary.
+- **Tiers are config, not code.** `extra.shell.min_permission` (default 30) is the level below which the tool isn't offered. `extra.shell.policies` is a flat list where each entry carries the level at which a caller *outgrows* it: `[{policy: builtin:allow, applies_below: 45}, {policy: builtin:deny, applies_below: 90}]`. A caller is subject to every entry whose `applies_below` they're under — so level 20 gets both, 50 gets the deny-list only, 95 gets nothing. Omit `applies_below` for a policy that binds everyone (no unrestricted tier at all). `policy:` accepts `builtin:allow`, `builtin:deny`, a name relative to the config dir, or an absolute path. `_dispatch()` does not know which policy is an allow-list and which is a deny-list — each file declares that via `default_action` — so tier selection is one comparison per policy.
+  - Note what the shape *can't* express: a policy binding a higher-privileged caller while sparing a lower one. That's incoherent; a flat threshold list makes it unrepresentable. An earlier band-table design (`tiers: [{min_level, apply: [...]}]`) could express it, which is why it was replaced. Before that it was a hardcoded if-chain plus `use_whitelist`/`neutral`/`bypass_blacklist` — three fixed tiers, unextendable. **Those three keys are now an error**, not ignored: silently dropping a `use_whitelist: 90` someone set to lock the shell down would loosen access.
+- `extra.shell.permissions.access_backend` (default 80) stays a scalar — it gates *where* a command runs (main container vs sandbox), orthogonal to which policy applies. Levels resolve from the actual caller (`agent.caller.permission_level`, snapshotted once per cycle like `modules/sysops` does) rather than a static config value — fixes a bug where `backend_access=True`'s guard read `agent.config.permissions.level`, an attribute `PermissionsConfig` never defines (it only has `minimal_tokens: bool`).
+- A policy that won't load blocks **every** caller, including one who'd otherwise be unrestricted — a broken config can't tell us whether the failed entry was the one that would have bound them.
+- Tests: `tests/test_shell_policy.py` is a must-deny/must-allow corpus run against the **real shipped YAML**, including a check that every shipped rule has a test case proving it fires. `tests/test_shell.py` covers tier routing and fail-closed behaviour against throwaway fixture policies.
 
 ### `web` — `web_search` (DuckDuckGo via `ddgs`) and `open_url` (Playwright, headless by default; `headless=False` for captchas).
 
@@ -399,6 +428,16 @@ An *instance* is a self-contained directory holding one agent's config, workspac
 ```
 <instance>/                 e.g. ~/.tinyctx, or anywhere via --dir
 ├── config.yaml             Loaded by default from here (workspace.path / data.path default relative to this file)
+├── config/                  Extra config too big or structured to inline into config.yaml — currently
+│                            the shell module's policy YAML. Bound READ-ONLY at /app/config in the
+│                            container (`compose_env` exports the host path as TINYCTX_CONFIG_DIR;
+│                            compose.yaml sets TINYCTX_CONFIG_DIR_PATH=/app/config for the code side).
+│                            Deliberately NOT the instance dir itself, which holds .env and data/.
+│                            Created by `tinyctx onboard` and `tinyctx start` — Docker would otherwise
+│                            auto-create the missing bind source root-owned. Resolve it at runtime via
+│                            `utils/instance.py::runtime_config_dir()`, which is what lets a config
+│                            reference a file by bare name and have it work on host *and* in Docker,
+│                            where the same directory has a different absolute path.
 ├── .env                     Optional. KEY=VALUE per line (e.g. DISCORD_BOT_TOKEN=...). Loaded via
 │                            `utils/instance.py`'s `load_instance_env()` — with override=True, so
 │                            values here win over anything already exported in the shell/global env.

README.md

@@ -31,7 +31,7 @@ A context-efficient agentic assistant framework. Connect it to your LLM, configu
 > - **`prefix_required: true`** - in group channels, only respond when @mentioned or prefixed. This reduces noise but is not a security boundary on its own.
 > - **Gateway `api_key`** - always set a strong, random key if the gateway is enabled. Never expose the gateway port to the public internet without authentication.
 >
-> The filesystem module sandboxes `write_file`/`edit_file` to the workspace directory only. `view`/`grep`/`glob_search` can additionally see any directory listed in `filesystem.read_only_paths` in config.yaml (read-only, never writable - e.g. `/app` for the agent's own source code in the container). Nothing outside workspace + that whitelist is reachable at all. The module also maintains a shell command blacklist, but these are last-resort guardrails, not a substitute for access control.
+> The filesystem module sandboxes `write_file`/`edit_file` to the workspace directory only. `view`/`grep`/`glob_search` can additionally see any directory listed in `filesystem.read_only_paths` in config.yaml (read-only, never writable - e.g. `/app` for the agent's own source code in the container). Nothing outside workspace + that whitelist is reachable at all. The shell module parses commands with tree-sitter-bash and checks each one against a YAML policy, but these are last-resort guardrails, not a substitute for access control - the sandbox container is the real security boundary.
 >
 > **The right mental model: treat TinyCTX like an SSH session. Only give access to people you'd give a shell to.**