X Digest

X Digest keeps a private, local copy of your X bookmarks. It reads data from the official X API, stores the raw responses and media, and builds a searchable SQLite catalog.

The current version does not write posts, generate summaries, or use an LLM.

Key Capabilities

• Private archive in the local data/ directory.
• Incremental bookmark sync.
• Bookmark folder archive with an ignore list.
• One Markdown file per archived post.
• Search, export, verify, and rebuild commands.

Requirements

• macOS
• Python 3.11 or newer
• uv
• An X Developer application with OAuth 2.0 PKCE enabled

Install

git clone https://github.com/joaomj/x-digest.git
cd x-digest
uv sync

Register this redirect URI in the X Developer application:

http://localhost:8080/callback

Copy the template and set the client credentials:

cp .env.example .env

XDIGEST_X_CLIENT_ID=your-client-id
XDIGEST_X_CLIENT_SECRET=your-client-secret

Set XDIGEST_X_CLIENT_SECRET only when the X application requires one. The .env file stays local and is ignored by Git.

Enable the bookmark, post, and user read permissions in the X Developer Console.

Authorize X

uv run x-digest auth

Open the printed URL and authorize the application. Copy the complete callback URL from the browser address bar and run:

uv run x-digest auth --callback-url 'http://localhost:8080/callback?code=...&state=...'

The OAuth token is stored in the macOS Keychain. Authorization runs once; later commands reuse the token.

Core Usage

Sync bookmarks

uv run x-digest sync

The sync is incremental: it stops as soon as a page contains only already-archived posts. See tech-context.md, section 13.5 for the details.

Force a complete re-read:

uv run x-digest sync --full

Skip folders by name or ID. Their posts are never fetched, archived, or indexed:

uv run x-digest sync --ignore-folder spam

Set the same list in .env:

XDIGEST_IGNORE_FOLDERS=spam

Browse and export

uv run x-digest status
uv run x-digest search "local archive"
uv run x-digest show 1234567890
uv run x-digest export --format markdown
uv run x-digest export-post 1234567890 --output ./post.md

Verify and rebuild

uv run x-digest verify --full
uv run x-digest rebuild-silver

verify checks the archive; rebuild-silver rebuilds the searchable database from the raw archive and applies the ignore list.

Inspect API samples

uv run x-digest probe-bookmarks --max-results 20
uv run x-digest probe-post 'https://x.com/user/status/1234567890'

The probe commands fetch bounded samples and never paginate the full bookmark collection.

Generate missing Markdown

Every sync writes one Markdown file per newly archived post. A file is written once and never regenerated, so hand-made edits are safe. Generate files for all posts that still lack one, without any sync:

uv run x-digest markdown

Configuration

The complete settings list is in tech-context.md, section 7. Common .env settings:

Setting	Purpose	Default
XDIGEST_X_CLIENT_ID	X Developer application client ID	None
XDIGEST_X_CLIENT_SECRET	Optional X client secret	None
XDIGEST_X_REDIRECT_URI	OAuth callback URI	http://localhost:8080/callback
XDIGEST_IGNORE_FOLDERS	Comma-separated folder names or IDs to skip	empty
XDIGEST_FOLDER_SYNC_DAYS	Minimum days between folder reads	7
XDIGEST_VAULT_PATH	Vault location	<project-root>/data
XDIGEST_LOG_LEVEL	Log level	info

Local Storage

<project-root>/data/
├── bronze/              # immutable API responses and media
├── silver.sqlite        # normalized records and search index
├── markdown/            # one Markdown file per archived post
└── logs/                # aggregate and per-run logs

The project is self-contained. Move the entire data/ directory to relocate everything.

Automated Weekly Sync

Install the launchd agents, which run x-digest sync every Sunday at 06:00 and the GCS backup at 06:15:

./scripts/install-scheduler.sh
./scripts/install-backup-scheduler.sh

Macs that are off or locked at 06:00 miss the calendar run. The shell trigger covers that case: the first interactive shell each ISO week starts the same two agents in the background after a 30-minute delay, once per week. It is already hooked into ~/.zshrc via scripts/zshrc-init.sh, which calls scripts/weekly-shell-trigger.sh. Progress lands in data/logs/weekly-trigger.log with a once-per-week stamp at data/logs/weekly-shell-trigger.stamp.

Remove the agent:

./scripts/install-scheduler.sh --remove

Trigger the first run immediately:

launchctl kickstart "gui/$(id -u)/com.x-digest.sync"

See tech-context.md, section 17 for the agent behavior.

Weekly Telegram digest

The Sunday sync also sends one private Telegram digest after Markdown files are written. OpenRouter summarizes the oldest undelivered posts as themed key points with source links. Delivery is best-effort: archive success never depends on the digest, and failed batches stay pending for the next run.

Configure the bot and OpenRouter key in .env:

TELEGRAM_BOT_TOKEN=your-bot-token
TELEGRAM_USER_ID=your-chat-id
XDIGEST_LLM_API_KEY=your-openrouter-key

XDIGEST_TELEGRAM_BOT_TOKEN and XDIGEST_TELEGRAM_CHAT_ID work as prefixed alternatives. Create the bot with @BotFather, send it /start, then read your chat ID from getUpdates:

curl "https://api.telegram.org/botYOUR_TOKEN/getUpdates"

Preview the next batch without network calls:

uv run x-digest digest --dry-run

The preview lists selected sources and layout only. Send the next pending batch manually:

uv run x-digest digest --send

Large weeks stay pending across runs. Remove digest credentials to disable notifications without changing the archive schedule.

Backup to Google Cloud Storage (Free Tier)

A weekly backup copies the data/ directory to a private GCS bucket with rclone. Files are never deleted on the bucket; the backup only grows. Use a STANDARD bucket in an Always Free region (us-central1, us-west1, us-east1) with uniform bucket-level access, public access prevention enforced, and 7 day soft delete.

Requirements:

• rclone installed (for example via Homebrew).

• A GCS bucket and a rclone remote of type google cloud storage. Grant the backup service account roles/storage.objectUser on only the backup bucket. Configure the bucket name and remote through .env:

  XDIGEST_BACKUP_BUCKET=your-gcs-bucket-name
  XDIGEST_BACKUP_REMOTE=gcs
  

  The backup reads credentials from the macOS login Keychain. Store the service account JSON under service x-digest and account gcs-backup-credentials:

  chmod 600 "$HOME/.config/gcloud/your-key.json"
  security unlock-keychain "$HOME/Library/Keychains/login.keychain-db"
  security add-generic-password \
    -U \
    -s x-digest \
    -a gcs-backup-credentials \
    -w "$(tr -d '\n' < "$HOME/.config/gcloud/your-key.json")" \
    "$HOME/Library/Keychains/login.keychain-db"

  An SSH session can use the read-only System keychain by default. The explicit login-keychain path prevents -61 Write permissions error during setup.

  Verify the stored JSON without printing it:

  security find-generic-password \
    -s x-digest \
    -a gcs-backup-credentials \
    -w \
    "$HOME/Library/Keychains/login.keychain-db" \
    | uv run python -c "import json,sys; d=json.load(sys.stdin); print('Keychain OK:', d.get('type') == 'service_account')"

  The expected result is Keychain OK: True. Run the backup before you remove the source JSON file. After a successful backup, remove the file:

  ./scripts/backup-to-drive.sh
  tail -n 8 data/logs/backup.log
  rm "$HOME/.config/gcloud/your-key.json"

  The log must end with backup end. The script exports RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS from Keychain for rclone.

  As a file-based fallback, configure the rclone remote with the JSON file:

  rclone config create gcs googlecloudstorage service_account_file "$HOME/.config/gcloud/your-key.json" bucket_policy_only true

Run the backup once:

./scripts/backup-to-drive.sh

Install the launchd agent, which runs the backup every Sunday at 06:15, after the weekly sync:

./scripts/install-backup-scheduler.sh

Remove the agent:

./scripts/install-backup-scheduler.sh --remove

To restore, copy back from the bucket with rclone copy:

rclone copy gcs:your-gcs-bucket-name ./data/ --fast-list
# or, with env vars set:
rclone copy "$XDIGEST_BACKUP_REMOTE:$XDIGEST_BACKUP_BUCKET" ./data/ --fast-list

See tech-context.md, section 14.2 for the backup details.

Scope Boundaries

The current version does not include:

• X write operations.
• Summaries or LLM processing.
• X data-export archive import.
• A web interface.
• Multiple X accounts.

Develop

uv run pytest -q
uv run ruff check src tests

Further Reading

• tech-context.md: architecture, configuration reference, X API cost record, log details.