feat: Memory system enhancements - dedup, TTL, context window, passport

CHANGELOG.md

@@ -0,0 +1,110 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+
+#### Memory System Enhancements
+- **Content Hash Deduplication** - SHA-256 based server-side deduplication with unique constraint per namespace
+- **TTL Support** - Time-to-live for temporary memories with automatic expiration
+- **Client-side Deduplication** - `skip_duplicate_threshold` parameter in `remember` tool (0.95 exact, 0.85 semantic)
+- **Session Namespace Auto-TTL** - `session:*` namespaces automatically expire after 24 hours
+
+#### Advanced Retrieval
+- **Context Window Ranking** - Weighted scoring algorithm for LLM context optimization:
+  - Relevance: 50%
+  - Recency: 30% (24h half-life)
+  - Type boost: 20% (decision > bug > outcome > pattern)
+- **Context Passport Pattern** - Cross-system context transfer for Linear/GitHub/agent handoff
+
+#### Observability
+- **Structured Logging** - JSON-formatted logs with configurable levels (debug/info/warn/error)
+- **Prometheus Metrics** - `/metrics` endpoint with HTTP, database, and MCP operation metrics
+- **Request Tracing** - Automatic request ID generation and propagation
+
+#### Documentation
+- **OpenAPI Specification** - Complete REST API documentation (`docs/openapi.yaml`)
+- **Architecture Decision Records** - 6 ADRs documenting key architectural decisions
+- **Contributing Guide** - Comprehensive guide for contributors
+- **API Reference** - Detailed API documentation (`docs/api.md`)
+
+### Changed
+
+- **Database Schema v2** - Added `content_hash`, `ttl_seconds`, `expires_at` columns
+- **Recall Behavior** - Automatically filters expired memories from search results
+- **Namespace Convention** - Documented hierarchical convention (`global > project > agent > session`)
+
+### Deprecated
+
+- None
+
+### Removed
+
+- None
+
+### Fixed
+
+- None
+
+### Security
+
+- None
+
+## [1.0.8] - 2026-03-07
+
+### Added
+- MCP tool descriptions improved with vector/fts weight parameters
+
+## [1.0.7] - 2026-03-06
+
+### Added
+- Release workflow automation
+
+## [1.0.6] - 2026-03-05
+
+### Added
+- Claude plugin marketplace integration
+- GoReleaser CI/CD pipeline
+
+## [1.0.0] - 2026-03-01
+
+### Added
+- Initial release
+- Core MCP tools (remember, recall, forget, list, stats)
+- Hybrid search with RRF
+- Web UI with Vue.js
+- REST API
+- Namespace support
+
+---
+
+## Release Notes Template
+
+When creating a new release, use this template:
+
+```markdown
+## [X.Y.Z] - YYYY-MM-DD
+
+### Added
+- New features
+
+### Changed
+- Changes in existing functionality
+
+### Deprecated
+- Soon-to-be removed features
+
+### Removed
+- Now removed features
+
+### Fixed
+- Bug fixes
+
+### Security
+- Security improvements
+```

CLAUDE.md

@@ -2,13 +2,63 @@
 
 > Persistent semantic memory for AI agents via MCP (Model Context Protocol).
 
-## Session Start Protocol
-
-At the beginning of each session:
-
-1. Call `stats` (no namespace) to see what namespaces exist and how much is stored.
-2. Call `recall` 2–3 times with task-relevant queries (e.g. "trindex architecture decisions", "recent bugs fixed", "user preferences for this project") before starting work.
-3. This orients you before you write a single line of code.
+## Session Start Protocol (REQUIRED)
+
+At the beginning of **every** session, you MUST call `recall` to orient yourself. This is non-optional.
+
+### Standard Session Initialization Pattern
+
+```go
+func InitializeSession(agentName, projectName string) SessionContext {
+    // 1. Get global user context (always searched automatically)
+    globalFacts := Recall("user identity preferences work style", nil, 0.3, 10)
+
+    // 2. Get project-specific context
+    projectFacts := Recall("project architecture decisions patterns", ["project:"+projectName], 0.3, 10)
+
+    // 3. Get recent activity for continuity
+    recentActivity := Recall("recent work current tasks", ["project:"+projectName], 0.1, 5, since=24h)
+
+    return SessionContext{
+        GlobalFacts:    globalFacts,
+        ProjectFacts:   projectFacts,
+        RecentActivity: recentActivity,
+    }
+}
+```
+
+### Required Queries (Minimum)
+
+Call `recall` at least 2–3 times with these query types:
+
+1. **User Context**: `recall("user preferences identity work style", [], 0.3, 10)`
+   - Threshold: 0.3 (moderate precision for user facts)
+   - Returns: How user likes to work, their name, preferences
+
+2. **Project Context**: `recall("project architecture decisions", ["project:trindex"], 0.3, 10)`
+   - Threshold: 0.3 (moderate precision for architecture)
+   - Returns: Key decisions, patterns, codebase structure
+
+3. **Recent Activity**: `recall("recent work bugs fixed", ["project:trindex"], 0.1, 5)` with since=24h
+   - Threshold: 0.1 (wide net for recency-filtered results)
+   - Returns: What you were working on recently
+
+### Why This Matters
+
+- Prevents repeating the same questions to the user
+- Maintains continuity between sessions
+- Surfaces relevant context before you start coding
+- Reduces "context loss" errors
+
+### Threshold Guidelines
+
+| Query Type | Threshold | Rationale |
+|------------|-----------|-----------|
+| User facts | 0.3–0.5 | Precise matches for preferences |
+| Architecture | 0.3 | Balance precision/recall |
+| Recent work | 0.1 | Wide net filtered by time |
+| Exact lookup | 0.7+ | Specific function names, error codes |
+| Exploration | 0.0001 | Broad discovery mode |
 
 ## When to Remember
 
@@ -26,14 +76,40 @@ Write content as 1–3 concise sentences stating the fact directly. Example:
 
 ## Namespace Conventions
 
-| Namespace | Purpose |
-|-----------|---------|
-| `global` | Cross-agent user facts: preferences, identity, persistent personal context. Always searched automatically on every recall — you never need to include it explicitly. |
-| `default` | Fallback when no project context is clear. |
-| `trindex` | Memories specific to the Trindex project itself. |
-| `personal` | Personal context about the user that is not project-specific. |
+Namespaces follow a **hierarchical convention** for clear scoping and automatic inheritance:
+
+```
+global > project:{name} > agent:{name} > session:{id}
+```
+
+| Namespace | Purpose | Auto-searched |
+|-----------|---------|---------------|
+| `global` | Cross-agent user facts: preferences, identity, persistent personal context. | Always |
+| `project:{name}` | Project-specific memories: architecture, decisions, patterns (e.g., `project:trindex`). | No |
+| `agent:{name}` | Agent-specific learnings and optimizations (e.g., `agent:claude-code`). | No |
+| `session:{id}` | Ephemeral session context (auto-expires after 24h). | No |
+| `default` | Fallback when no project context is clear. | No |
+
+### Namespace Selection Rules
+
+1. **Use `global`** for cross-cutting user facts that any agent should know:
+   - User preferences ("User prefers dark mode")
+   - Identity facts ("User's name is Dave")
+   - Persistent personal context
 
-Use `global` for facts that should be available to any agent in any session. Use `trindex` for anything specific to this codebase or its development.
+2. **Use `project:{name}`** for project-specific knowledge:
+   - Architecture decisions
+   - Code patterns discovered
+   - Bug root causes
+   - Implementation notes
+
+3. **Use `session:{id}`** for temporary context:
+   - Current debugging session
+   - Temporary file paths
+   - Transient errors
+   - These auto-expire (TTL = 24h by default)
+
+4. **Avoid `default`** — be explicit about scope.
 
 ## Metadata Conventions
 
@@ -53,13 +129,27 @@ Use `global` for facts that should be available to any agent in any session. Use
 - **fts_weight high (e.g. 0.9)** — better for exact terms: function names, error codes, identifiers.
 - Always search `["trindex"]` namespace for project-specific memories; `global` is included automatically.
 - Pass multiple namespaces (`["trindex", "personal"]`) to cast a wider net across contexts.
+- Use `skip_duplicate_threshold` on `remember` to avoid storing duplicates (0.95 for exact, 0.85 for semantic).
 
 ## Memory Hygiene
 
 - Before ending a long session, check for duplicate memories with `recall` and use `forget` to prune stale or incorrect ones.
 - If you store a memory and later find it was wrong, `forget` it by ID and store a corrected version.
 - Use `list` with a namespace to audit what exists before bulk operations.
 
+## Memory Lifecycle (TTL)
+
+Memories can have automatic expiration to prevent storage bloat:
+
+| Namespace Pattern | Default TTL | Rationale |
+|------------------|-------------|-----------|
+| `session:*` | 24 hours | Ephemeral session context |
+| `global` | Never | User preferences persist |
+| `project:*` | Never | Project knowledge persists |
+| `default` | 30 days | Fallback cleanup |
+
+Use the `ttl_seconds` parameter on `remember` to set explicit expiration for temporary context.
+
 ## Key Architectural Facts
 
 | Fact | Detail |
@@ -72,3 +162,7 @@ Use `global` for facts that should be available to any agent in any session. Use
 | Main tools file | `internal/mcp/tools.go` |
 | Memory layer | `internal/memory/recall.go` (hybrid search), `internal/memory/store.go` (CRUD) |
 | Config | `internal/config/config.go` — all env-based with defaults |
+| Deduplication | Content hash (SHA-256) + semantic similarity threshold |
+| TTL | `ttl_seconds` param, `session:*` defaults to 24h |
+| Context window | `internal/memory/context_window.go` — relevance/recency/type weighted ranking |
+| Passport | `internal/memory/passport.go` — cross-system context transfer |