Skip to content

docs(deploy): connect the evaluation guide to the Policies & Decisions docs - #41

Merged
ghchinoy merged 1 commit into
mainfrom
docs/evaluate-policy-links
Sep 30, 2026
Merged

ghchinoy merged 1 commit into
mainfrom
docs/evaluate-policy-links

Conversation

@ghchinoy

Copy link
Copy Markdown
Owner

Follow-up to #40: the evaluation guide only linked the policy docs in one sentence.

docs/deploy/evaluate.md

  • §5.1 Design your policy (was "Write a policy"): a "what to read" table:

    • applications + decision matrix (does the use case fit?)
    • template catalog + creator's guide (start from an existing policy)
    • first policy (step by step)
    • authoring §2 (fast and reliable)
    • datasets step 2 (co-design with MCP decide_custom_questions)

    It also lists the policy rules that matter for an evaluation: fixed schema with per-row state, samples: 1, the 26-option limit, depends_on/ask_if costing a second pass, and evaluating the policy you will ship.

  • §5.4: maps each error pattern to its fix:

    • errors cluster on one question → reword the options (authoring, first policy §3)
    • inputs fit no option → taxonomy discovery
    • too many hesitant answers → Stage 2 cascade. Noted that it calls Gemini on Vertex AI, so a self-hosted evaluation needs a GCP project (cmd/cascade_gemini.go).
    • need an error bar per decision → samples: 4

    Run the test set once with the final policy and any cascade enabled.

  • Step 5 label in the overview table updated; Next links the full Policies & Decisions set.

Link-backs

  • policies/first-policy.md "Where to go next" → evaluation guide.
  • policies/datasets.md intro: callout for self-hosters to verify the install first, and to use a local gateway with the runners.

Checks

make docs-sync-check (44/44), docs_link_check.py (0 broken) and make check-public pass. The docs-site build passes (45 pages), and all 11 in-page anchors on the three changed pages resolve in the built HTML.

…s docs

evaluate.md §5.1 becomes "Design your policy": a what-to-read table (applications
+ decision matrix, template catalog, first policy, authoring, MCP co-design) and
the policy rules that matter for an evaluation (fixed schema, samples, 26-option
limit, depends_on costs a second pass). §5.4 maps error patterns to fixes
(option rewording, taxonomy discovery, Stage 2 cascade - needs a GCP project for
Gemini on Vertex AI, samples). Next links the full policy docs.

first-policy.md and datasets.md link back to the evaluation guide.
@ghchinoy
ghchinoy merged commit 4c18684 into main Sep 30, 2026
3 checks passed
@ghchinoy
ghchinoy deleted the docs/evaluate-policy-links branch October 3, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant