diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..841c4be --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,49 @@ +# CLAUDE.md + +Personal site & blog for Yonatan Lourie — `https://yonatanlou.github.io`. + +## Stack & deploy + +- **Eleventy (11ty) v3** static site generator, ES modules (`"type": "module"`). +- Deployed to **GitHub Pages** via `.github/workflows/gh-pages.yml` (`peaceiris/actions-gh-pages`), which runs `npm run build-ghpages` on push to `main`. PRs build but do not deploy. +- **Static hosting only**: no custom HTTP headers, no request-time content negotiation, no server/edge. Anything requiring those is out of scope here. +- `netlify.toml` / `vercel.json` exist but are not the live pipeline. + +## Commands + +```bash +npm install +npx @11ty/eleventy --serve # local dev at http://localhost:8080 +npm run build # build to _site/ (add ELEVENTY_RUN_MODE=build to drop drafts) +``` + +## Layout + +- `content/` — all pages & posts (Eleventy `input` dir). Posts live in `content/blog/**`, each a Markdown file with `title`/`description`/`date` frontmatter; `content/blog/blog.11tydata.js` tags them `posts` and applies `layouts/post.njk`. +- `_includes/layouts/` — `base.njk` (the HTML shell / ``), `home.njk`, `post.njk`. +- `_includes/` — shared partials (e.g. `structured-data.njk`, `postslist.njk`). +- `_data/metadata.js` — site-wide metadata (title, url, author, social `sameAs`, OG image, twitter handle). `_config/filters.js` — custom filters (`readableDate`, `htmlDateString`, `filterTagList`, …). +- `public/` — static assets copied to site root (CSS, `robots.txt`); `content/img/` → `/img`. +- Output goes to `_site/` (gitignored). Never commit `_site/` or `.cache/`. + +## Conventions + +- Match existing style; keep files small. Comments only when necessary. +- Nunjucks auto-escapes `{{ }}` — use `| safe` when emitting raw Markdown/HTML (e.g. `post.rawInput`). +- Build absolute URLs with `{{ page.url | htmlBaseUrl(metadata.url) }}`. +- Templates that should not appear in collections/sitemap set `eleventyExcludeFromCollections: true`. + +## Agent-facing / AI-crawler features + +This site is deliberately optimized for AI agents and LLM crawlers (design doc: `docs/superpowers/specs/2026-07-15-agent-ready-design.md`). When adding or changing content, keep these in sync: + +- **`public/robots.txt`** — allows named AI agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …) with a `Content-Signal: ai-train=yes, search=yes, ai-input=yes` per block, plus the `Sitemap:` line. Named-bot blocks must repeat their own rules (no inheritance from `*`). +- **`content/llms.txt.njk` → `/llms.txt`** — llmstxt.org index: one bullet per post (title + description + absolute URL) and key pages. Auto-generated from `collections.posts`, so new posts appear automatically as long as they have a `description`. +- **`content/llms-full.txt.njk` → `/llms-full.txt`** — every post's raw Markdown (`post.rawInput`) concatenated, newest first. +- **`content/blog/post-md.njk`** — paginates `collections.posts` to emit a clean `index.md` for every post, advertised on the HTML page via ``. +- **`_includes/structured-data.njk`** — JSON-LD included from `base.njk`: `BlogPosting` on post pages, `WebSite` + `Person` elsewhere. Values are emitted with `| dump` for valid JSON. +- **`base.njk` ``** — page-aware ``, `author`, canonical, Open Graph, Twitter Card, and sitewide `<link rel="describedby" href="/llms.txt">`. + +**Every new post needs a `description` in its frontmatter** — it feeds `llms.txt`, the meta description, OG/Twitter, and JSON-LD. Post source filenames must be clean ASCII (a stray control char once produced a broken `%03` URL). + +After changing any of the above, rebuild and sanity-check that `/robots.txt`, `/llms.txt`, `/llms-full.txt`, and the per-post `index.md` files are generated and that JSON-LD parses as valid JSON. diff --git a/_data/metadata.js b/_data/metadata.js index ca23d7f..2700dea 100644 --- a/_data/metadata.js +++ b/_data/metadata.js @@ -6,8 +6,19 @@ export default { author: { name: "Yonatan Lourie", email: "yonatanlou@gmail.com", - url: "https://yonatanlou.github.io/about-me/" + url: "https://yonatanlou.github.io/about/" }, + // Default image used for Open Graph / Twitter cards. + image: "/img/favicon/android-chrome-512x512.png", + // Twitter/X handle used for card attribution. + twitter: "@yonatanlou", + // Canonical profiles, reused by JSON-LD `sameAs` and the footer. + sameAs: [ + "https://github.com/yonatanlou", + "https://www.linkedin.com/in/yonatanlourie/", + "https://x.com/yonatanlou", + "https://www.goodreads.com/user/show/103722180-yonatan-lourie" + ], gtag: "G-C6PG57BBFC" diff --git a/_includes/layouts/base.njk b/_includes/layouts/base.njk index a0a3d38..b35df6a 100644 --- a/_includes/layouts/base.njk +++ b/_includes/layouts/base.njk @@ -13,9 +13,35 @@ <link rel="icon" type="image/png" sizes="512x512" href="/android-chrome-512x512.png"> <link rel="manifest" href="/site.webmanifest"> - <title>{{ metadata.title }} + {% if title %}{{ title }} | {{ metadata.title }}{% else %}{{ metadata.title }}{% endif %} + + + {#- Canonical + agent-readable resources -#} + + + {%- if tags and "posts" in tags %} + + {%- endif %} + + {#- Open Graph -#} + + + + + + + + {#- Twitter Card -#} + + + + + + + {#- Structured data (JSON-LD) -#} + {% include "structured-data.njk" %} diff --git a/_includes/structured-data.njk b/_includes/structured-data.njk new file mode 100644 index 0000000..960496f --- /dev/null +++ b/_includes/structured-data.njk @@ -0,0 +1,58 @@ +{%- if tags and "posts" in tags -%} +{#- Blog post pages -> BlogPosting -#} + +{%- else -%} +{#- All other pages -> WebSite + Person -#} + +{%- endif -%} diff --git "a/content/blog/how-you-can-run-the-same-python-distribution-on-eks-and-aws-lambda/\003how_you_can_run_the_same_python_distribution_on_eks_and_aws_lambda.md" b/content/blog/how-you-can-run-the-same-python-distribution-on-eks-and-aws-lambda/how_you_can_run_the_same_python_distribution_on_eks_and_aws_lambda.md similarity index 100% rename from "content/blog/how-you-can-run-the-same-python-distribution-on-eks-and-aws-lambda/\003how_you_can_run_the_same_python_distribution_on_eks_and_aws_lambda.md" rename to content/blog/how-you-can-run-the-same-python-distribution-on-eks-and-aws-lambda/how_you_can_run_the_same_python_distribution_on_eks_and_aws_lambda.md diff --git a/content/blog/post-md.njk b/content/blog/post-md.njk new file mode 100644 index 0000000..5373785 --- /dev/null +++ b/content/blog/post-md.njk @@ -0,0 +1,22 @@ +--- +pagination: + data: collections.posts + size: 1 + alias: post +permalink: "{{ post.url }}index.md" +layout: false +eleventyExcludeFromCollections: true +--- +# {{ post.data.title }} + +- URL: {{ post.url | htmlBaseUrl(metadata.url) }} +- Published: {{ post.date | htmlDateString }} +{%- if post.data.description %} +- Summary: {{ post.data.description }} +{%- endif %} +{%- set filteredTags = post.data.tags | filterTagList %} +{%- if filteredTags | length %} +- Tags: {% for tag in filteredTags %}{{ tag }}{% if not loop.last %}, {% endif %}{% endfor %} +{%- endif %} + +{{ post.rawInput | safe }} diff --git a/content/llms-full.txt.njk b/content/llms-full.txt.njk new file mode 100644 index 0000000..1f1abaf --- /dev/null +++ b/content/llms-full.txt.njk @@ -0,0 +1,22 @@ +--- +permalink: /llms-full.txt +layout: false +eleventyExcludeFromCollections: true +--- +# Yonatan Lourie — Full Blog Text + +> Every post on yonatanlou.github.io concatenated as raw Markdown, newest first. +{% for post in collections.posts | reverse %} + +--- + +# {{ post.data.title }} + +- URL: {{ post.url | htmlBaseUrl(metadata.url) }} +- Published: {{ post.date | htmlDateString }} +{%- if post.data.description %} +- Summary: {{ post.data.description }} +{%- endif %} + +{{ post.rawInput | safe }} +{%- endfor %} diff --git a/content/llms.txt.njk b/content/llms.txt.njk new file mode 100644 index 0000000..5dc5e40 --- /dev/null +++ b/content/llms.txt.njk @@ -0,0 +1,24 @@ +--- +permalink: /llms.txt +layout: false +eleventyExcludeFromCollections: true +--- +# Yonatan Lourie + +> AI Researcher at Tavily building web infrastructure for agents. MSc in Statistics & Data Science from Tel Aviv University (Dead Sea Scrolls authorship attribution with NLP and Graph Neural Networks). This site collects writing on machine learning, clustering, NLP, and ML engineering. + +## Blog +{% for post in collections.posts | reverse %} +- [{{ post.data.title }}]({{ post.url | htmlBaseUrl(metadata.url) }}): {{ post.data.description }} +{%- endfor %} + +## Pages + +- [About]({{ "/about/" | htmlBaseUrl(metadata.url) }}): who I am and what I work on. +- [Now]({{ "/now.html" | htmlBaseUrl(metadata.url) }}): what I'm focused on at the moment. +- [Projects]({{ "/projects/" | htmlBaseUrl(metadata.url) }}): selected projects and code. +- [Links]({{ "/Links/" | htmlBaseUrl(metadata.url) }}): things worth reading and following. + +## Optional + +- [Full text of all posts]({{ "/llms-full.txt" | htmlBaseUrl(metadata.url) }}): every post concatenated as raw Markdown. diff --git a/docs/superpowers/specs/2026-07-15-agent-ready-design.md b/docs/superpowers/specs/2026-07-15-agent-ready-design.md new file mode 100644 index 0000000..8dcf157 --- /dev/null +++ b/docs/superpowers/specs/2026-07-15-agent-ready-design.md @@ -0,0 +1,55 @@ +# Making the site agent-ready — design + +Goal: make `yonatanlou.github.io` discoverable and consumable by AI agents and +LLM crawlers (ChatGPT, Claude, Perplexity, Gemini, etc.), following the +practices in [jlhernando.com](https://jlhernando.com/blog/make-static-site-agent-ready/) +and [joost.blog](https://joost.blog/agent-ready/). + +## Constraint + +The site is a static Eleventy build deployed to **GitHub Pages**. GitHub Pages +serves static files only — no custom HTTP headers and no request-time content +negotiation. So techniques that require an edge (HTTP `Link` headers, +`Accept: text/markdown` negotiation, MCP/WebMCP endpoints) are **out of scope**. +Everything below is achievable with static output. + +## What we ship + +1. **`/robots.txt`** (`public/robots.txt`) — explicit `Allow` for named AI agents + (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …), a + `Content-Signal: ai-train=yes, search=yes, ai-input=yes` line, and a + `Sitemap:` reference. Named-bot blocks repeat their own rules (they do not + inherit from `User-agent: *`). + +2. **`/llms.txt`** (`content/llms.txt.njk`) — llmstxt.org index: site summary, + a bullet per post (title + description + absolute URL), key pages, and a link + to `/llms-full.txt`. + +3. **`/llms-full.txt`** (`content/llms-full.txt.njk`) — every post's raw + Markdown (`page.rawInput`) concatenated newest-first. + +4. **Per-post Markdown** (`content/blog/post-md.njk`) — for each post, a static + `…/index.md` with clean, chrome-free source, advertised on the HTML page via + ``. + +5. **JSON-LD structured data** (`_includes/structured-data.njk`, included in + `base.njk`) — `BlogPosting` on posts; `WebSite` + `Person` (with `sameAs`, + `jobTitle`, `worksFor`, `alumniOf`) elsewhere. + +6. **Meta tags** (`base.njk`) — page-aware ``, `author`, canonical URL, + Open Graph, Twitter Card, and a sitewide `<link rel="describedby" href="/llms.txt">`. + +7. **`_data/metadata.js`** — adds default OG `image`, `twitter` handle, and a + `sameAs` social list reused by the JSON-LD. + +## Incidental fix + +The newest post's source file had a stray control character (`\003`) at the +start of its filename, producing a broken `%03` URL on the live site. Renamed to +a clean filename. + +## Explicitly out of scope + +HTTP `Link` headers, `Accept`-based content negotiation, `.well-known/skills.json`, +MCP/WebMCP, `entitymap.json` — either impossible on GitHub Pages or overkill for +a personal blog. diff --git a/public/robots.txt b/public/robots.txt new file mode 100644 index 0000000..c68a32c --- /dev/null +++ b/public/robots.txt @@ -0,0 +1,67 @@ +# robots.txt — yonatanlou.github.io +# This site welcomes AI crawlers, search-grounding agents, and assistants. +# Content-Signal follows the Cloudflare proposal (https://developers.cloudflare.com/bots/concepts/content-signals-policy/). + +User-agent: * +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +# --- Named AI / search agents (rules do not inherit from "*", so repeat them) --- + +User-agent: GPTBot +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: ChatGPT-User +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: OAI-SearchBot +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: ClaudeBot +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: Claude-User +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: anthropic-ai +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: PerplexityBot +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: Perplexity-User +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: Google-Extended +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: Applebot-Extended +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: CCBot +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: Meta-ExternalAgent +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: cohere-ai +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +User-agent: Bytespider +Allow: / +Content-Signal: ai-train=yes, search=yes, ai-input=yes + +Sitemap: https://yonatanlou.github.io/sitemap.xml