Skip to content

feat: content deduplication on ingest via BLAKE3 hashing #52

@salishforge

Description

@salishforge

Summary

Hash-check memory content before hot-tier insertion to prevent storing duplicate information that inflates consolidation batches.

Design

  • Compute BLAKE3 (or SHA-256) hash of content on add()
  • Check hash against recent hot-tier entries for the same agent
  • If duplicate detected: update metadata/timestamp of existing entry, skip insert
  • Add content_hash column to hot_tier for fast lookup

Inspiration

Inspired by CCRider (MIT) BLAKE3 content-based deduplication strategy.

Closes #52

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformancePerformance improvements

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions