Site Rescue

Site Rescue is a platform-neutral, CLI-first, local-first toolkit for rescuing websites from hosted platforms that are shutting down or becoming inaccessible. Weebly is the first source adapter; it is not the architecture of the product.

Site Rescue v0.1.0 is the first public release and is licensed under Apache-2.0.

Current capabilities

• Versioned normalized content schema with runtime validation
• Platform-neutral source-adapter contract and deterministic registry selection
• Bounded, deterministic, same-origin HTML crawler
• Shared bounded HTTP retry, timeout, and response-body policy for HTML and assets
• Fresh-output-only lifecycle for capture, export, report, and end-to-end rescue commands
• Self-contained Node.js CLI release package with SHA-256 inventory and smoke verification
• Apache-2.0 project licensing with bundled third-party license and NOTICE preservation
• site-rescue --version release-version command
• Weebly detection with confidence and human-readable evidence
• Generic HTML fallback detection and standards-based content normalization
• site-rescue diagnose <url> CLI command
• site-rescue capture <url> normalized JSON capture command
• site-rescue export <site.json> platform-neutral static HTML export command
• site-rescue report <site.json> migration readiness and manual-review report command
• site-rescue verify <site-directory> offline static link and asset verification command
• site-rescue manifest <rescue-directory> deterministic whole-bundle SHA-256 manifest command
• site-rescue verify-bundle <rescue-directory> offline rescue bundle integrity command
• site-rescue preview <site-directory> verified local HTTP preview command
• site-rescue rescue <url> end-to-end local capture, static export, report, verification, and bundle manifest command
• Content-addressed asset downloads with SHA-256 checksums and duplicate-file reuse
• Bounded CSS @import and url() dependency discovery with local reference rewriting
• Local path rewriting for normalized image, gallery, and video blocks
• Static pages with local navigation, responsive baseline CSS, and verified asset copies
• Explicit baseline or source static export style modes with safe baseline fallback
• Same-origin redirect preservation with normalized 301 mappings and duplicate-page prevention
• Deterministic redirects.json handoff manifests and offline-verifiable HTML fallback pages
• Deterministic rescue-manifest.json inventories for capture, site, and report artifacts
• Offline fixture tests for schema, crawling, adapter detection, and CLI orchestration

Requirements

• Node.js 20.19 or newer (the CI baseline is recorded in .node-version)
• pnpm 10.34.5 (recorded in packageManager)

Release package

Download site-rescue-v0.1.0.zip and site-rescue-v0.1.0.zip.sha256 from the v0.1.0 GitHub Release. The archive contains a bundled CLI and does not require pnpm or a dependency installation, but it still requires Node.js 20.19 or newer.

Verify and extract on PowerShell:

Get-FileHash .\site-rescue-v0.1.0.zip -Algorithm SHA256
Get-Content .\site-rescue-v0.1.0.zip.sha256
Expand-Archive .\site-rescue-v0.1.0.zip
node .\site-rescue-v0.1.0\bin\site-rescue.js --version
node .\site-rescue-v0.1.0\bin\site-rescue.js --help

Verify and extract on macOS or Linux:

shasum -a 256 -c site-rescue-v0.1.0.zip.sha256
unzip site-rescue-v0.1.0.zip
node site-rescue-v0.1.0/bin/site-rescue.js --version
node site-rescue-v0.1.0/bin/site-rescue.js --help

The source archive generated automatically by GitHub is not the bundled CLI release package.

Quick start

pnpm install
pnpm site-rescue --version
pnpm site-rescue diagnose https://example.weebly.com
pnpm site-rescue diagnose https://example.com
pnpm site-rescue rescue https://example.weebly.com --output ./rescued-site
pnpm site-rescue rescue https://example.weebly.com --output ./styled-rescue --style-mode source
pnpm site-rescue capture https://example.weebly.com --output ./capture
pnpm site-rescue export ./capture/site.json --output ./site
pnpm site-rescue export ./capture/site.json --output ./styled-site --style-mode source
pnpm site-rescue report ./capture/site.json --output ./report
pnpm site-rescue verify ./site
pnpm site-rescue manifest ./rescued-site
pnpm site-rescue verify-bundle ./rescued-site
pnpm site-rescue preview ./site

The one-step command keeps reusable capture data separate from the generated website:

rescued-site/
├─ rescue-manifest.json
├─ capture/
│  ├─ site.json
│  └─ assets/
├─ site/
│  ├─ index.html
│  ├─ redirects.json
│  ├─ assets/
│  └─ ...
└─ report/
   ├─ migration-report.json
   └─ migration-report.md

Skip binary asset downloads when only a normalized capture document is needed:

pnpm site-rescue capture https://example.weebly.com --output ./capture --skip-assets

Limit discovery when testing a large site:

pnpm site-rescue diagnose https://example.weebly.com --max-pages 20

Tune bounded request reliability when a source server is slow or unstable:

pnpm site-rescue diagnose https://example.weebly.com --max-retries 3 --request-timeout-ms 20000 --max-html-bytes 8388608

capture, export, report, and rescue require a new output directory. If the path already exists, the command stops before changing it. A failed run removes only the new directory created for that run.

After building, the CLI entry point can also be run directly:

pnpm build
node apps/cli/dist/index.js diagnose https://example.weebly.com
node apps/cli/dist/index.js rescue https://example.weebly.com --output ./rescued-site
node apps/cli/dist/index.js rescue https://example.weebly.com --output ./styled-rescue --style-mode source
node apps/cli/dist/index.js capture https://example.weebly.com --output ./capture
node apps/cli/dist/index.js export ./capture/site.json --output ./site
node apps/cli/dist/index.js export ./capture/site.json --output ./styled-site --style-mode source
node apps/cli/dist/index.js report ./capture/site.json --output ./report
node apps/cli/dist/index.js verify ./site
node apps/cli/dist/index.js manifest ./rescued-site
node apps/cli/dist/index.js verify-bundle ./rescued-site
node apps/cli/dist/index.js preview ./site

Prepare and verify the self-contained release candidate package:

pnpm build
pnpm release:check
node release/package/bin/site-rescue.js --version
node release/package/bin/site-rescue.js --help

The generated release/package directory is ignored by Git. It contains one bundled CLI entry point, Apache-2.0 project licensing, third-party runtime notices, release metadata and documentation, a source map, and SHA256SUMS.txt.

capture/site.json contains site metadata, navigation, normalized pages and blocks, an asset manifest, redirects, and warnings. Successfully downloaded files are stored under capture/assets/; each manifest entry retains its original sourceUrl and adds localPath, mediaType, and a SHA-256 checksum. Failed downloads remain as source URLs and produce asset-download-failed warnings without aborting the rest of the capture.

Example output:

Site Rescue diagnosis
URL: https://example.weebly.com/
Platform: Weebly (100% confidence)
  evidence: Hostname is on weebly.com
  evidence: Generator metadata names Weebly
Discovered pages (3):
  - /
  - /about.html
  - /contact.html

Repository layout

site-rescue/
├─ apps/
│  └─ cli/                    # Thin command-line application
├─ packages/
│  ├─ content-schema/         # Platform-neutral normalized site model
│  ├─ core/                   # Source-adapter contracts and registry selection
│  ├─ http-policy/            # Bounded retry, timeout, and response-body reads
│  ├─ output-safety/          # Fresh-output-only command lifecycle
│  ├─ crawler/                # Bounded same-origin page discovery
│  ├─ css-references/         # Shared CSS AST parsing and reference rewriting
│  ├─ redirects/              # Redirect evidence mapping and conflict handling
│  ├─ asset-pipeline/          # Downloading, hashing, deduplication, and path rewriting
│  ├─ report/                  # Platform-neutral migration analysis and reports
│  ├─ rescue-manifest/         # Whole-bundle inventory and integrity verification
│  ├─ static-verifier/         # Offline generated-site integrity checks
│  ├─ preview-server/          # Loopback-only static site preview
│  ├─ release-tools/           # Release package build, checksum, and smoke verification
│  ├─ exporters/
│  │  └─ static-html/          # Platform-neutral static website generation
│  └─ adapters/
│     ├─ generic-html/        # Conservative standards-based fallback adapter
│     └─ weebly/              # Weebly-only detection rules
├─ fixtures/
│  └─ weebly/                 # Sanitized, deterministic test pages
└─ docs/
   ├─ RELEASE_CHECKLIST.md    # Exact-commit public release procedure
   └─ adr/                    # ADR-001 through ADR-018

The intended long-term boundary is:

source website -> source adapter -> normalized site schema -> exporter -> target website

Development checks

pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm release:check

GitHub Actions runs these checks plus built and packaged CLI smoke tests on every pull request and every push to main. The workflow uses a frozen lockfile, pinned action commits, read-only repository permissions, and a 10-minute job timeout.

Default tests do not access live websites. Live compatibility checks should remain separate and explicit.

Crawl behavior and limitations

• Crawls only HTTP(S) links on the exact origin of the starting page.
• Removes URL fragments and ignores common non-document extensions.
• Fetches HTML only, follows links breadth-first, and stops at 50 pages by default.
• Limits each HTML response to 5 MiB and each attempt to 15 seconds by default.
• Retries transient request failures and HTTP 408, 429, 500, 502, 503, and 504 responses twice by default.
• Does not retry permanent HTTP responses such as 404.
• Records same-origin requested-to-final URL aliases and normalizes each final page only once.
• Does not execute JavaScript, sign in, bypass access controls, or process private content.
• Client-rendered pages and interaction-only navigation may be missed.

Only crawl and migrate websites you own or are authorized to process. Review the source site's terms and applicable policies before running the tool.

Generic HTML fallback behavior

• Runs after platform-specific adapters with intentionally low detection confidence.
• Extracts server-rendered metadata, navigation, common content blocks, and referenced assets.
• Preserves forms, tables, and objects as inert source for manual review.
• Adds a generic-html-fallback warning to every capture, so reports never claim automatic readiness.
• Does not execute JavaScript or recover client-rendered state, application behavior, or platform-specific features.

Asset behavior and limitations

• Downloads HTTP(S) assets already discovered by the source adapter, including images, video sources, posters, and linked stylesheets.
• Recursively discovers CSS @import and url() dependencies, including fonts and background images.
• Rewrites successfully downloaded CSS references to local relative paths; failed references remain absolute and produce warnings.
• Stores binary assets using content-derived filenames and reuses one file when multiple URLs return identical bytes.
• Stores rewritten stylesheets under stable source-URL hash paths so circular imports can be resolved safely.
• Limits each asset to 50 MiB by default and records individual failures as warnings.
• Limits each asset request attempt to 30 seconds and uses the shared transient retry policy.
• Limits stylesheet import depth to 5 and CSS-discovered assets to 500 by default.
• Does not execute downloaded content.
• Preserves source stylesheets as migration assets and applies them only when --style-mode source is explicitly selected.

Static export behavior and limitations

• Consumes only a validated site.json and its local assets; it does not call the crawler or Weebly adapter.
• Writes one HTML file per normalized page and rewrites known internal page and asset references as relative links.
• Copies local assets only after validating their recorded SHA-256 checksums.
• Escapes text content and displays preserved embed or raw HTML as inert source instead of executing it.
• Adds a small responsive baseline stylesheet; it does not reproduce the original platform theme pixel-for-pixel.
• Defaults to --style-mode baseline; source mode loads the baseline first and then verified top-level source stylesheets.
• Source mode refuses captures with missing or unverified top-level stylesheets instead of linking remote CSS.
• Re-running export in baseline mode provides a deterministic fallback if source CSS conflicts with normalized markup.

Redirect behavior and limitations

• Converts followed same-origin path aliases into normalized 301 migration mappings.
• Writes a deterministic redirects.json manifest for production-host configuration.
• Writes a canonical, meta-refresh, and clickable-link HTML fallback for each non-colliding redirect source path.
• Rejects redirect output collisions and redirects whose targets are not exported pages.
• Treats generated 301 status as migration intent; it does not reproduce the source server's exact status or intermediate chain.
• Skips query-bearing, cross-origin, missing-target, and conflicting redirect evidence with explicit warnings.
• HTML fallback pages do not deliver HTTP 301 responses; configure the manifest in the production host.

Migration report behavior

• Consumes only a validated site.json; it does not call the crawler or a source adapter.
• Writes deterministic migration-report.json and migration-report.md artifacts.
• Summarizes pages, nested content blocks, verified, unverified, and remote assets, warnings, and manual-review items.
• Summarizes preserved redirects and requires review for production HTTP redirect configuration.
• Marks a capture as review-required when warnings, remote assets, preserved embeds, or raw HTML remain.
• The report is guidance for migration review; it does not claim visual or functional parity with the source site.

Static verification behavior

• Scans generated HTML and CSS files and validates local HTML attributes, @import, and url() references.
• Checks page fragments against HTML id and legacy anchor name values.
• Rejects unsafe URL schemes, invalid encodings, paths outside the site root, and unverified symbolic-link targets.
• Counts external references but never requests them over the network.
• Returns exit code 2 when broken local references are found, making the command suitable for CI.
• A successful verification proves local file integrity, not migration completeness or external-link availability.

Local preview behavior

• Runs static verification first and refuses to serve a site with broken local references.
• Serves only on 127.0.0.1, using port 4173 unless --port is supplied.
• Supports normal nested index.html routes and common static asset content types.
• Prevents path traversal and symbolic-link escapes outside the selected site directory.
• Uses no-cache response headers so repeated manual reviews read current local files.
• Stops cleanly with Ctrl+C; it does not open a browser automatically.

Rescue bundle integrity behavior

• Treats capture/, site/, and report/ as one portable rescue bundle.
• Writes a canonically sorted rescue-manifest.json containing relative POSIX paths, roles, byte sizes, and SHA-256 checksums.
• Validates capture/site.json, requires reports to match that capture, and runs static verification before writing a manifest.
• Detects missing, unexpected, size-mismatched, and checksum-mismatched files without network access.
• Rejects symbolic links, special filesystem entries, unsafe manifest paths, unexpected top-level entries during manifest creation, and bundles above 100,000 files.
• Returns exit code 2 from verify-bundle for integrity mismatches, making it suitable for backup and handoff checks.
• Excludes the manifest itself from its file inventory so identical bundle contents produce identical manifest bytes.
• SHA-256 proves content consistency against the recorded manifest; it is not a digital signature and does not prove who created the bundle.

Safe output behavior

• capture, export, report, and rescue accept only a non-existing output directory.
• Existing files and directories are refused before command work begins and are never modified.
• A failed operation removes only the exact output directory created by that operation.
• manifest and verify-bundle intentionally inspect an existing rescue bundle.
• Phase 16 does not provide overwrite, resume, or incremental capture modes.

Release readiness

• All workspace packages must use the root semantic version and remain private before the release policy changes.
• release:check rebuilds release/package, verifies every checksum, executes bundled --version and --help, and produces a real migration report from an offline fixture.
• The release package bundles runtime dependencies but still requires Node.js 20.19 or newer.
• Package metadata remains private: true to prevent accidental npm publication; the planned first distribution channel is GitHub Releases.
• Root, workspace, and generated package metadata use the Apache-2.0 SPDX identifier; bundled dependencies remain under their own terms recorded in THIRD_PARTY_NOTICES.txt. Exact-version audited license overrides cover published packages that omit their license file.
• Follow the release checklist before changing visibility, tagging, or publishing.
• See the public repository publication record for the completed visibility, history-audit, and GitHub protection checks.
• See CHANGELOG.md, CONTRIBUTING.md, and SECURITY.md for project policy.

Community reports

• Use the structured bug report for reproducible defects with sanitized fixtures and logs.
• Use the feature request for focused migration capabilities.
• Follow SECURITY.md and use a private security advisory for vulnerabilities. Never disclose credentials, private website content, personal data, or customer data in an issue.

Explicitly out of scope before the public v0.1.0 release

• GUI or hosted SaaS
• Cloudflare infrastructure or domain automation
• Payments, CMS, page builder, membership, or e-commerce
• Pixel-perfect cloning or AI redesign
• Astro or framework-specific exporters
• Automatic deployment or hosting
• Host-specific redirect configuration, cross-origin redirects, or query-based redirect rules
• LAN or public-network serving, HTTPS, authentication, or live reload
• Automatically applying source stylesheets by default, CSS optimization, or visual parity analysis
• CSS sanitization, selector translation, theme editing, or an interactive style toggle
• Inline styles and JavaScript-loaded CSS
• JavaScript rendering, browser automation, authenticated crawling, or form behavior recreation
• ZIP/TAR creation, encryption, digital signatures, cloud upload, or incremental backups

Decisions

See Architecture Decision Records for ADR-001 through ADR-019.

License

Site Rescue is licensed under the Apache License 2.0. Bundled third-party runtime components retain their own licenses and notices.