ST-Mem · Linguistic Trajectory Encoding

Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents

Accepted at NeurIPS 2026

Paper  ·  Project page  ·  SMB benchmark  ·  Code & demo  ·  Citation

Linguistic Trajectory Encoding (LTE) builds object-centric spatiotemporal memory for long-horizon embodied agents. It compresses object motion histories into natural-language descriptions, sparse spatial anchors, and visual evidence. This makes what happened, where, and when queryable across hours to days.

From long observations to queryable object histories. Original teaser from the paper. Numbers are paper-reported results. View full resolution.

Highlights

• Compact memory. Language, sparse geometry, and visual anchors preserve complementary evidence in a per-object trajectory representation.
• Long-horizon retrieval. Query object state history and motion in natural language, or retrieve an object's last observed occurrence.
• Spatial Memory Benchmark (SMB). 600 queries from EgoLife recordings for Semantic Trajectory Retrieval (STR) and Long-Horizon Object Retrieval (LOR), alongside evaluation on Ego4D NLQ and VQ2D.

Explore the results → · Benchmark tasks and protocol →

Method

ST-Mem is the LTE-based memory system. Its five linked views are Object, Scene, Text, Event, and Image. They connect shared object records to language, spatial, and visual evidence.

Full research architecture from the paper, including components outside the core demo. View full resolution.

Code

This repository contains the ST-Mem project website and demo. CPU demo · Real-video demo.

Citation

@misc{xie2026linguistictrajectory,
  title = {Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents},
  author = {Tianyidan Xie and Shenyi Wang and Qiang Tang and Mingjie Wang and Zhicheng Qiu and Xuanfu Li and Zhan Xu and Jian Yang and Lanjun Wang and Zili Yi},
  year = {2026},
  note = {Accepted at NeurIPS 2026},
  eprint = {2609.04802},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  doi = {10.48550/arXiv.2609.04802},
  url = {https://arxiv.org/abs/2609.04802}
}

---

Website and demo code: MIT · Model dependencies

Original paper figures: CC BY 4.0 (attribution).