TrajLoc localizes a trajectory instead of a single ground image: given a street-view video sequence, a natural-language route description, or both, it retrieves the matching geo-tagged satellite tile. The repository also introduces SeqGeo-VL, the first cross-view benchmark that is sequential and linguistic — 38,863 aligned video–text–satellite triplets — and ships an agent-ready tool interface so multimodal agents can call TrajLoc for outdoor spatial reasoning.
🎬 The task in action — click to expand the demo
The task in action: watch a route at street level, read its narrative, and pick the satellite tile it traverses — a strong MLLM picks the wrong tile, TrajLoc gets it right. Try it yourself →
- 2026-07 — 🚗 Code, model weights, and the SeqGeo-VL dataset are released.
- 2026-07 — 🎉 TrajLoc is accepted to ECCV 2026. See project page.
- News
- Highlights
- Results
- Installation
- Repository Structure
- SeqGeo-VL Dataset
- Training
- Evaluation
- Agent Tool
- Citation
- Acknowledgements
- License
- A setting prior datasets missed. Existing cross-view datasets are single-image, video-only, or text-only — never sequential and linguistic together. SeqGeo-VL pairs route videos, route narratives, and satellite imagery in 38,863 triplets with a 91.8% human verification pass rate.
- One model, three query modes. TrajLoc adapts a single pretrained CLIP ViT-L/14 into video, text, and satellite encoders through a co-training curriculum — no bespoke temporal architecture. Query with video frames, plain language, or a blend of both.
- Spatially-grounded Embeddings. TrajMod maps route geometry
τ = {(Δxᵢ, Δyᵢ, θᵢ)}— waypoint offsets and headings, no map or POI metadata — to FiLM scale/shift parameters that modulate the query embedding (f′ = γ ⊙ f + β) — plug-and-play on top of frozen encoders. - An agentic tool, not just a benchmark.
trajloc_tool.pypackages the model as a function-calling tool with a persistent Python API and a JSON CLI, so multimodal agents can localize routes as part of broader reasoning.
TrajLoc sets the state of the art on SeqGeo-VL for both query types, using the same ViT-L/14 backbone as its closest competitors.
Video → Satellite
| Method | Backbone | R@1 | R@5 | R@10 | R@1% |
|---|---|---|---|---|---|
| SeqGeo | VGG16 | 1.80 | 6.45 | 10.36 | 34.38 |
| SeqGeo† | ViT-L/14 | 8.14 | 24.42 | 34.02 | 66.17 |
| GARet | DeiT-m | 3.34 | 11.19 | 17.18 | 44.39 |
| FlexGeo | ConvNeXt-B | 3.51 | 14.22 | 20.77 | 48.53 |
| Qwen3-VL-Embedding* | Qwen3-VL-2B | 5.44 | 19.21 | 28.15 | 61.16 |
| TrajLoc (ours) | ViT-L/14 | 12.09 | 35.77 | 47.68 | 80.21 |
Text → Satellite
| Method | Backbone | R@1 | R@5 | R@10 | R@1% |
|---|---|---|---|---|---|
| CLIP | ViT-L/14 | 0.80 | 3.32 | 6.01 | 22.99 |
| SigLIP | ViT-L/16@384 | 0.85 | 3.13 | 5.24 | 21.33 |
| EVA2-CLIP | ViT-L/14@336 | 0.89 | 3.78 | 6.70 | 26.99 |
| Qwen3-VL-Embedding* | Qwen3-VL-2B | 0.93 | 3.38 | 5.55 | 20.91 |
| CrossText2Loc | ViT-L/14@336 | 0.98 | 4.08 | 7.08 | 25.45 |
| TrajLoc (ours) | ViT-L/14 | 2.52 | 9.82 | 16.11 | 45.48 |
* LoRA fine-tuned. † Reimplemented with a ViT-L/14 backbone.
Query with video, language, or both. At inference, a parameter-free blend f = (1−α)·f_video + α·f_text (α = 0.4) consistently beats the stronger single modality — up to +9.49 R@10 when only one video frame is available.
Geometry makes the modalities cooperate. With TrajMod, adding text co-training lifts video R@1 from 9.69 to 12.09, and video co-training lifts text R@1 from 1.90 to 2.52.
conda create -n traj python=3.10 -y
conda activate traj
pip install -r requirements.txtAll commands must run from the repository root: config.yaml is loaded relative to the current directory.
TrajLoc
├── finetune.py # Three-phase training entry point
├── trainer.py # Contrastive, distillation, and TrajMod objectives
├── model_loader.py # TrajLoc and TrajMod (FiLM) model definitions
├── my_dataset.py # SeqGeo-VL dataset loader and frame sampling
├── zeroshot.py # Retrieval evaluation entry point
├── evaluate.py # Recall metrics and similarity computation
├── trajloc_tool.py # Agent-facing tool: Python API + JSON CLI
├── utils.py # Shared helpers
├── config.yaml # Dataset and output paths
└── run_trajloc.sh # One-command three-phase reproduction
SeqGeo-VL extends SeqGeo with route-level natural-language narratives generated by a four-stage VLM+LLM pipeline (per-frame captioning → motion integration → LLM summarization → human quality control).
| Triplets | Frames / route | Avg. words / description | Human QC pass rate |
|---|---|---|---|
| 38,863 | ~7 | 101.3 | 91.8% |
Download the dataset from 🤗 MVRL/SeqGeo-VL, then point config.yaml at your local copies:
paths:
sat_root_dir: "/path/to/SeqGeo_dataset"
video_root_dir: "/path/to/SeqGeo_dataset"
trainset_path: "/path/to/train_dataset_full_v2.json"
testset_path: "/path/to/val_dataset_full_v2.json"
checkpoint_dir: "./checkpoints"
result_dir: "./results"
case_dir: "./cases"
log_dir: "./logs"Annotation format
Each annotation item holds one video–text–satellite trajectory. Seven video frames are sampled per trajectory; Phase 3 (TrajMod) additionally requires traj_coords and traj_headings.
{
"trajectory_id": "000001",
"grd_images": ["streetview/frame_000.jpg", "streetview/frame_001.jpg"],
"sat_image": "satellite/tile_000001.png",
"text": "The route proceeds east along a tree-lined street ...",
"traj_coords": [[38.0, -90.0], [38.0, -89.999]],
"traj_headings": [91.0, 92.5]
}The full three-phase curriculum (video–satellite contrastive training → text–satellite adaptation with distillation → TrajMod fine-tuning) runs on a single GPU:
bash run_trajloc.shPretrained checkpoints are available at 🤗 MVRL/TrajLoc.
python zeroshot.py \
--version trajloc_trajmod_eval \
--model CLIP-L/14 \
--expand \
--use_trajmod \
--checkpoint /path/to/trajmod_checkpoint.pthEvaluation reports Video→Satellite and Text→Satellite R@1, R@5, R@10, and R@1%. Checkpoints, logs, and metrics are written under checkpoints/, logs/, and results/.
trajloc_tool.py exposes TrajLoc as a function-calling tool for multimodal agents: it keeps the model and satellite gallery in memory, accepts structured trajectory queries, and returns ranked satellite matches as JSON. In the example below, an LLM agent sketches trajectory hypotheses from a vague route memory and calls TrajLoc to pin them on the map — jump to the demo.
Python API — persistent across calls, ideal for in-process agents:
from trajloc_tool import TrajLocTool
tool = TrajLocTool(
checkpoint_path="checkpoints/trajloc_full.pth",
gallery_manifest="gallery.json", # satellite tiles to search over
gallery_cache="gallery_features.pt", # features cached here after the first run
)
result = tool.localize_trajectory(
query_mode="text", # or "video" with video_paths=[...]
text="The route heads east along a tree-lined street, then turns right at the church.",
traj_coords=[[38.6488, -90.3108], [38.6495, -90.3089]],
traj_headings=[91.0, 92.5],
top_k=5,
)JSON CLI — reads a request from a file or stdin and emits JSON only on stdout, ready for subprocess-based tool adapters:
# Inspect the OpenAI-style function-calling schema
python trajloc_tool.py --schema
# Run one request against a satellite gallery
python trajloc_tool.py \
--checkpoint checkpoints/trajloc_full.pth \
--gallery-manifest gallery.json \
--gallery-cache gallery_features.pt \
--request request.json # or "-" to read from stdinThe checkpoint and gallery can also be configured through the TRAJLOC_CHECKPOINT, TRAJLOC_GALLERY_MANIFEST, and TRAJLOC_GALLERY_CACHE environment variables.
If you find TrajLoc or SeqGeo-VL useful, please cite:
@inproceedings{gao2026trajloc,
title = {Trajectory-aware Cross-view Geo-localization with Sequential Observations},
author = {Gao, Tianyi and Lin, Jiayu and Beaulieu, Danielle and Jacobs, Nathan},
booktitle = {European Conference on Computer Vision},
year = {2026}
}This project builds on OpenAI CLIP, the SeqGeo dataset, and the CrossText2Loc codebase. Please also follow the licenses of the pretrained models and datasets you use with this code.
This repository is released under the MIT License. Questions and issues are welcome on the issue tracker.