Implement SEO and discoverability features

[will-lamerton]

Summary

The documentation site currently lacks essential SEO and discoverability features. This issue requests implementing sitemap.xml, robots.txt, OpenGraph tags, and other metadata to improve search engine visibility.

Problem Description

• No sitemap.xml for search engines to crawl
• No robots.txt to guide search engine behavior
• Missing OpenGraph tags for social sharing
• No canonical URLs
• Page descriptions not properly set
• No structured data (JSON-LD)

Current State

Pages currently render but lack proper SEO metadata. The DocsWrapper component has PagefindMetadata for search but not for SEO.

Important: Static Export Constraint

This site uses output: "export" in next.config.ts, meaning it's built as a fully static site. This has implications:

• app/sitemap.ts and app/robots.ts work — Next.js generates these at build time during next build
• No runtime/dynamic generation — There is no server at runtime, so sitemaps cannot be generated on-the-fly
• All URLs must be known at build time — This is already the case since generateStaticParams() enumerates all project/version/slug combinations
• The sitemap can reuse the same data source as generateStaticParams() to enumerate all pages

Proposed Solution

1. Generate sitemap.xml (Build-Time)

Create app/sitemap.ts using Next.js's built-in sitemap support. This runs at build time and outputs a static sitemap.xml:

// app/sitemap.ts
import type { MetadataRoute } from 'next';
import { getAllProjects } from '@/lib/projects';
import { getVersions } from '@/lib/versions';
import { getDocPathsForVersion } from '@/lib/page-map-builder';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = 'https://docs.nanocollective.org';
  const urls: MetadataRoute.Sitemap = [
    { url: baseUrl, changeFrequency: 'daily', priority: 1.0 },
  ];

  // Reuse the same project/version enumeration as generateStaticParams()
  const projects = getAllProjects();
  for (const project of projects) {
    const versions = await getVersions(project.id, project.repo);
    for (const version of versions) {
      const paths = await getDocPathsForVersion(project.id, version, project.repo);
      for (const slugPath of paths) {
        const slug = slugPath.length > 0 ? `/${slugPath.join('/')}` : '';
        urls.push({
          url: `${baseUrl}/${project.id}/docs/${version}${slug}`,
          changeFrequency: 'weekly',
          priority: 0.8,
        });
      }
    }
  }

  return urls;
}

2. robots.txt (Build-Time)

Use app/robots.ts for build-time generation:

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/' },
    sitemap: 'https://docs.nanocollective.org/sitemap.xml',
  };
}

3. OpenGraph Meta Tags

Add to page metadata:

export const metadata = {
  metadataBase: new URL('https://docs.nanocollective.org'),
  openGraph: {
    title: 'Nanocoder Documentation',
    description: 'A beautiful privacy-first coding agent running in your terminal',
    url: 'https://docs.nanocollective.org/nanocoder',
    siteName: 'Nano Collective',
    locale: 'en_US',
    type: 'website',
  },
  twitter: {
    card: 'summary_large_image',
    title: 'Nanocoder Documentation',
    description: 'A beautiful privacy-first coding agent running in your terminal',
  },
};

4. Canonical URLs

Each page should have a canonical URL to prevent duplicate content issues.

5. JSON-LD Structured Data

Add Organization and WebSite schema:

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Nano Collective",
  "url": "https://nanocollective.org",
  "logo": "https://nanocollective.org/logo.png"
}

Requirements

1. sitemap.xml - Auto-generated with all documentation pages
2. robots.txt - Allow crawling, point to sitemap
3. OpenGraph tags - Title, description, image, URL for each page
4. Canonical URLs - Proper canonical tag on each page
5. Twitter cards - Summary card with image
6. JSON-LD - Organization and website structured data

Files Likely Affected

• app/sitemap.ts - New file for build-time sitemap generation (reuses getDocPathsForVersion())
• app/robots.ts - New file for build-time robots.txt
• app/[project]/docs/[version]/[[...slug]]/page.tsx - Extend generateMetadata() with OpenGraph (already has basic title/description)
• app/[project]/docs/[version]/layout.tsx - Version-specific metadata
• lib/page-map-builder.ts - May need to expose URL lists for sitemap

Success Criteria

• Sitemap.xml is accessible and includes all pages
• Robots.txt exists and allows crawling
• OpenGraph tags appear in page source
• Canonical URLs are correct
• Twitter card images work
• Structured data validates in testing tools