Skip to content

v0.2.0: senses as meanings, provenance, public API, and quality gates on the Convex stack - #227

Merged
amanthanvi merged 12 commits into
mainfrom
audit/convex-port
Sep 12, 2026
Merged

amanthanvi merged 12 commits into
mainfrom
audit/convex-port

Conversation

@amanthanvi

Copy link
Copy Markdown
Owner

Summary

Ports the audit roadmap to the Convex and GitOps content architecture and ships it as v0.2.0. The earlier draft PR #226 targets the March Postgres codebase and stays as a reference only.

  • Senses are meanings. The compiler groups near-identical definitions from different sources into one sense (Dice similarity, attach at 0.8 or higher, flag between 0.5 and 0.8) and carries the other sources' wording as attestations. Ambiguous groups raise a needsLabel warning in content:check; editors resolve them with the new labelSenses, disambiguationNotes, groupSenses, and splitSenses override fields. Every sense has a heading (label, expansion, or the citing source) and the lede no longer duplicates the first sense. 25 entries currently need labels.
  • Sense navigation works. The table of contents double-prefixed its anchors and every link was dead; it is fixed, shown from two senses up, and a chip row replaces the hidden rail below 1280px. Each sense has a copy-link control, a "Cite" disclosure with BibTeX, a "Report" link with the sense fragment, and a "Compare wording" disclosure when other sources define the same meaning.
  • Provenance. Each source declares a content mode (quoted, summarized, paraphrased) and a one-line public license statement; every citation carries them plus the document hash, citation text, and locator, rendered at readable size. JSON-LD DefinedTermSet on entry and tag pages. A citation record endpoint returns the sense's sources with hashes.
  • Search. Typed prefix scan, full text from two characters, tag filtering before the candidate cap, an expansion and alias bucket, totals, snippets from the entry summary instead of the concatenated search document, meaning-level search (?scope=senses) on a definition search index, sense counts for terms as well as acronyms.
  • Ingest. NIST emits one sense per definition with the per-definition source as its label (the live "domain" page has eight); NIST aliases no longer pull in unrelated cross-references; RFC 4949 "See" and "Compare" lines become relationships; the RFC parser stops at the end of the glossary (the "zone-of-control" definition had swallowed sections 5 to 7); the classifier is shared; crawling honors robots.txt with per-host pacing and conditional requests. Bundles regenerate on the next scheduled ingest run and go through the documented tag-assignment procedure, so content/generated is unchanged here.
  • Tags. Assignment provenance (editorial or model, with score) reaches the serving layer and the tag directory.
  • Web. Cached Convex reads with deploy-time revalidation (POST /api/v1/internal/revalidate), no anonymous cookie and no view tracking, metadata and canonicals on every route, /tags/[slug] 404s, accessible relation previews and palette (inert background, live region, Tab restored), contrast and target-size fixes, a public read API with OpenAPI, ETags, CORS, and a paged dataset export, health check, form-action and CSP reporting, IP-first rate limiting behind trusted proxies with a required production salt, one BrowsePage for terms and acronyms, dead primitives and legacy aliases removed.
  • CI and repo. ci.yml matches pnpm gate (format, strict lint, convex lint and typecheck, tagging check); noUncheckedIndexedAccess and type-aware lint on; SHA-pinned actions; the ingest workflow no longer interpolates input into shell; a quality workflow runs Playwright with axe and Lighthouse budgets against an anonymous Convex deployment; CodeQL security-extended; grouped Dependabot; .github/CODEOWNERS no longer shadows the current boundary; issue templates capture sense, wording origin, and license; runbooks and governance describe the Convex system; SPEC infra sections marked superseded; pnpm.overrides moved to pnpm-workspace.yaml (pnpm 10 ignored them, so the frozen install was already failing on main).

Verification

Check Result
lint, strict lint, convex lint, format, typecheck, convex typecheck, versions pass
tagging check, content check, content history pass
unit and Convex tests 182 passed
web build pass
Playwright with axe (WCAG 2.2 AA) against the local stack 17 of 17 passed

Deploy notes

  • Set SYNAC_RATE_LIMIT_SALT and SYNAC_REVALIDATE_SECRET (Vercel) and SYNAC_REVALIDATE_SECRET (repo secret for the Deploy workflow) before merging. SYNAC_SESSION_HASH_SALT is no longer read.
  • The Convex schema changed (new fields, entryViews dropped); the Deploy workflow's convex deploy plus sync handles it, and the deploy fails if the revalidate call fails.
  • .env.example still lists SYNAC_SESSION_HASH_SALT; it is protected in the author's tooling and needs a manual edit.
  • After merge, tag v0.2.0 on the merge commit.

- Compile groups near-duplicate definitions across sources into one sense
  (Dice similarity: attach at 0.8, flag 0.5 to 0.8 as needsLabel) and keeps
  the other sources' wording as attestations; overrides gain labelSenses,
  disambiguationNotes, groupSenses, splitSenses.
- Sources declare contentMode and a public license statement; citations
  carry them plus the document hash, citation text, and locator.
- Tag assignment provenance (editorial or model, with score) reaches Convex.
- Search: typed prefix scan, full text from two characters, tag-filtered
  candidates, expansion and alias bucket, totals, clean snippets, and a
  meaning-level index with search.senses.
- View tracking removed; service key compared by digest; rate-limit keys
  validated; entry batches sized under the exec-argument limit.
- Summary omitted when it duplicates the first sense; sense summaries for
  all entry types; label fallback to the citing source.
…awling

- NIST emits a sense per term-def-text-N with the per-definition source as
  its label and locator; aliases keep only expansions and initialisms and
  other cross-references become SEE_ALSO.
- RFC 4949: See and Compare lines become SEE_ALSO and CONTRAST; the glossary
  stops at the end of section 4 (zone-of-control had swallowed sections 5-7).
- Shared classifier; robots.txt and per-host pacing; conditional requests
  with etag and lastModified recorded on bundle documents; fetch seam for
  tests instead of module mocking.
… API

- Sense TOC anchors fixed (they were double-prefixed and dead), rail from
  two senses, chip row under 1280px, per-sense copy link, cite, and report.
- Attestations with word-level comparison; content mode, citation text,
  locator, and license statement on every source; CONTRAST relations.
- Accessible relation previews and palette (inert background, live region,
  Tab restored); contrast, focus ring, and target-size fixes; metadata,
  canonicals, and DefinedTerm JSON-LD; tag 404s; search totals and
  meaning-level scope; page rate limiting.
- Cached Convex reads with a revalidate endpoint, health check, no
  anonymous cookie, IP-first rate limiting with a required production salt,
  CSP form-action and reporting, public read API with OpenAPI, ETags, CORS,
  citation records, and a paged dataset export.
- One BrowsePage for terms and acronyms; dead primitives, legacy aliases,
  loading skeletons, and duplicate helpers removed.
…ouse

- ci.yml runs format, strict lint, convex lint and typecheck, and the
  tagging check; quality.yml runs Playwright with axe and Lighthouse budgets
  against an anonymous Convex deployment; CodeQL security-extended; cdxgen
  SBOM; SHA-pinned actions; grouped Dependabot.
- ingest.yml passes its input through env and splits write permissions to
  the PR job; Deploy calls the revalidate endpoint after sync.
- noUncheckedIndexedAccess and type-aware lint on; pnpm overrides moved to
  pnpm-workspace.yaml (pnpm 10 ignored the package.json field).
- .github/CODEOWNERS reflects the current boundary; issue templates capture
  sense, wording origin, and license; version check script.
- Runbooks, governance, triage, contributing, releasing, backups, and the
  architecture overview describe the Convex and GitOps system; SPEC infra
  sections marked superseded; PRODUCT.md corrected.
- New docs for sense grouping and editorial labels, the public API, and
  content modes in the licensing policy; versions bumped to 0.2.0.
@vercel

vercel Bot commented Sep 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
synac-web Ready Ready Preview Sep 12, 2026 5:11am UTC

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: c770b1ea-2e0e-46a0-9423-0934b41e7523


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, we are unable to review this pull request

The GitHub API does not allow us to fetch diffs exceeding 20000 lines

Comment thread apps/web/src/components/PublicEntryPage.tsx Fixed
CI's test step and Vercel's web-only build could not find the package
because its entry pointed at an unbuilt dist directory.
…ects it

The project builds from apps/web; a root vercel.json doubled the output
path to apps/web/apps/web/.next and failed after a successful build.
…vice key

The backend serves /version before the first push commits, so env set hit
an optimistic-concurrency 503.
The first deploy after this branch merges validates every existing
production document against the new schema. Fields added since the last
generation (content mode, document hash, tag provenance, sense
attestations, match terms) are now optional so that validation passes;
the sync always writes them and the public queries fall back to defaults
until the next generation replaces the rows.

Also ignore the local .vercel link directory in the format check.
CodeQL flagged the two-step escaper as incomplete sanitization because
the second replace does not handle backslashes on its own. A single
character class that includes the backslash produces the same output
and is recognizably complete.
@amanthanvi

Copy link
Copy Markdown
Owner Author

@greptileai review

@amanthanvi

Copy link
Copy Markdown
Owner Author

@CodeRabbit review

@coderabbitai

coderabbitai Bot commented Sep 12, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review skipped: 254 files exceed the limit of 100.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@greptile-apps

greptile-apps Bot commented Sep 12, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no blocking or actionable new issue remains after the follow-up fixes.

Summary

  • Models meanings as grouped senses with attestations, labels, navigation, citations, and provenance.
  • Adds public read APIs, OpenAPI documentation, exports, caching, rate limiting, health checks, and deployment revalidation.
  • Improves source ingestion, robots handling, search behavior, tag provenance, accessibility, CI, and security workflows.
  • The changes since the previous review address all six prior findings without introducing a confirmed new defect.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Sources[External sources] --> Ingest[Ingest adapters]
  Ingest --> Bundles[Generated content bundles]
  Overrides[Editorial overrides] --> Compiler[Content compiler]
  Bundles --> Compiler
  Compiler --> Dataset[Compiled senses and provenance]
  Dataset --> Sync[Convex sync]
  Sync --> Convex[Convex serving layer]
  Convex --> Web[Next.js pages]
  Convex --> API[Public read API]
  Deploy[Deploy workflow] --> Sync
  Deploy --> Revalidate[Cache revalidation]
  Revalidate --> Web
Loading

Reviews (2) · Last reviewed commit: "fix: address review findings on grouping..."

Comment thread tools/content/src/compile.ts
Comment thread apps/web/src/lib/openapi.ts
Comment thread .github/workflows/ingest.yml
Comment thread apps/web/src/app/api/v1/csp-report/route.ts Outdated
Comment thread tools/ingest/src/net/crawl.ts Outdated
Comment thread content/sources/owasp-vulnerabilities.json Outdated
…bots, and OWASP mode

- compile: a reviewed groupSenses primary stays a leader instead of being
  attached to an earlier near-duplicate, which orphaned its members.
- compile: bundle senses from a source that declares SUMMARIZED or
  PARAPHRASED are rejected; adapters can only copy source wording.
- OWASP declares QUOTED, the mode its adapter performs. The share-alike
  rule in the licensing policy is corrected: quoting keeps the text under
  its own license, while summaries and paraphrases are derived works.
- openapi: search results are a oneOf of the entry and sense shapes.
- ingest workflow: intent-to-add content/ so a new source's first bundle
  reaches the pull request job.
- csp-report: the body is read with a byte cap instead of buffered first.
- crawler: an unreachable robots.txt (server error, timeout) halts the
  host per RFC 9309; a 404 or an HTML page still means no rules.
@amanthanvi

Copy link
Copy Markdown
Owner Author

@greptileai review

…ane on Linux

- compileContent exposes the ungrouped classification view; prepare,
  local review, and the emitter hash it instead of the grouped dataset,
  so their corpus and entry hashes match the ones the content check
  verifies. They diverged once sense grouping landed.
- The emitter accepts a reviewed removal for an entry that left the
  corpus when its current hash is the documented all-zero sentinel.
- codex-cli passes HOME through so codex finds its login on Linux and
  macOS, and a child that exits before reading its prompt reports the
  exit instead of crashing on EPIPE.
@amanthanvi
amanthanvi merged commit 9324c4e into main Sep 12, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants