Recovery: ~200 images missing from S3 due to failed Refile migration (Jan 4 2026)

[holden]

Background

During the Refile → ActiveStorage migration on January 4, 2026, some Sidekiq jobs failed mid-transfer. The ActiveStorage blob DB records were committed for all files, but a small percentage of S3 uploads to the new bucket (cdn4.recordtemple.com) never completed. This causes ActiveStorage::FileNotFoundError (500) when those images are requested.

These are not corrupted files — the original files are intact in the old Refile bucket (cdn.recordtemple.com) and can be fully recovered.

Scope

Metric	Value
Total blobs	147,617
Blobs from Refile migration (have refile_id in metadata)	39,461
Estimated missing from new bucket (~0.5% sample rate)	~200 files
Still recoverable from old bucket	100%
Direct uploads affected (no refile_id)	0

How to identify missing files

The migration job stored the original Refile ID in the blob's metadata column:

{"refile_id": "697dc01c5ae8ba03105deafe...", "identified": true}

A missing file is one where:

1. metadata->>'refile_id' is present (it was part of the migration)
2. ActiveStorage::Blob.service.exist?(blob.key) returns false (not in new bucket)

This naturally scopes the recovery to only migration blobs — direct uploads have no refile_id and are untouched.

Recovery plan

1. Query all refile blobs in batches using find_each
2. Check existence in new bucket via ActiveStorage::Blob.service.exist?(blob.key)
3. For missing ones only: fetch the file from cdn.recordtemple.com/store/{refile_id} and re-upload it to the new bucket using the same key the blob already expects (blob.key)
4. Skip everything else — blobs already in the new bucket are never touched

Uploading to the existing key (rather than creating a new blob) means no DB changes are needed and all existing attachments, variant records, and signed URLs continue to work.

Safety guarantees

• Only touches blobs with refile_id metadata — direct uploads are never checked
• Only acts on blobs confirmed missing from new bucket — present files are skipped
• No DB writes — re-uploads to the same S3 key the blob already references
• Idempotent — can be run multiple times safely; already-recovered files will be skipped on re-run
• Old bucket is read-only — nothing is deleted from cdn.recordtemple.com

Implementation sketch

Run as a rake task or one-off Rails runner. Key logic:

missing_blobs = []

ActiveStorage::Blob
  .where("metadata::json->>'refile_id' IS NOT NULL")
  .find_each(batch_size: 100) do |blob|
    unless ActiveStorage::Blob.service.exist?(blob.key)
      missing_blobs << blob
    end
  end

# Then for each missing blob:
# 1. download from cdn.recordtemple.com/store/{refile_id}
# 2. upload to cdn4.recordtemple.com/{blob.key}  (same key blob already expects)
# 3. log result

The existing MigrateRefileJob already has the S3 client setup and download logic — the recovery task can reuse that pattern with minimal new code.

Before implementing

• Confirm old bucket (cdn.recordtemple.com) still has IAM access
• Run a full count (not sample) to get exact number of missing blobs — this will take a few minutes due to S3 HEAD requests
• Decide: rake task vs Sidekiq job (Sidekiq preferred for parallelism and retry)