diff --git a/README.md b/README.md index f1da791..38738e5 100644 --- a/README.md +++ b/README.md @@ -188,6 +188,7 @@ Want to add your own? See the [open call for provider examples](https://github.c * **[USECASES.md](USECASES.md)** — Detailed guide on where BOOTH fits in your architecture and where it doesn't. * **[CHANGELOG.md](CHANGELOG.md)** — Release notes and history of features, bug fixes, and API evolution. * **[examples/](https://github.com/Vedantgitbot/booth/tree/main/examples)** — Real, runnable integration examples across providers and models, with documented results. +* **[TROUBLESHOOTING.md](TROUBLESHOOTING.md)** — Known gotchas from real integration runs, with fixes. --- diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md new file mode 100644 index 0000000..f0c801c --- /dev/null +++ b/TROUBLESHOOTING.md @@ -0,0 +1,96 @@ +# Troubleshooting + +Known gotchas collected from real integration runs in `examples/`. Each entry follows the same shape: **symptom → cause → what to do**, with a link to where it was observed. + +Facts below were checked against BOOTH v0.5.3. If a later release changes behavior, the [CHANGELOG](CHANGELOG.md) is the source of truth. + +## Contents + +1. [`check()` returns `UNCERTAIN` with `answer=None` and no error](#1-check-returns-uncertain-with-answernone-and-no-error) +2. [Wrong answers come back `ACCEPTED` with confidence 1.0](#2-wrong-answers-come-back-accepted-with-confidence-10) +3. [404 / "model not found" from the provider](#3-404--model-not-found-from-the-provider) +4. [401 with a new-style API key](#4-401-with-a-new-style-api-key) +5. [An ambiguous question was not flagged `AMBIGUOUS`](#5-an-ambiguous-question-was-not-flagged-ambiguous) +6. [`check_with_evidence()` accepts answers it should block](#6-check_with_evidence-accepts-answers-it-should-block) + +--- + +## 1. `check()` returns `UNCERTAIN` with `answer=None` and no error + +**Symptom.** Every call returns `status="UNCERTAIN"`, `answer=None`, `confidence=None`, usually with `attempts=2`. Nothing is raised or printed, so a real failure looks the same as the model genuinely being unsure. + +**Cause.** If `call_fn` raises (bad API key, retired model name, network error, quota), `check()` / `acheck()` catch the exception, treat it as a failed attempt, retry, and finally return a normal-looking `UNCERTAIN`. As of v0.5.3 the exception is not surfaced on the result (see [#9](https://github.com/Vedantgitbot/booth/issues/9)). + +**What to do.** Pass `on_attempt=` and print the attempt's error: + +```python +def show_error(i, attempt): + if attempt.error: + print(f"[attempt {i + 1}] {attempt.error}") + +result = booth.check(call_llm, "What is the capital of France?", on_attempt=show_error) +``` + +Rule of thumb: if you see `UNCERTAIN` with `answer=None` and `attempts > 1`, look for an exception inside `call_fn` before blaming the model. You can also call your `call_fn` once directly, outside BOOTH, to see the raw error. + +**Seen in:** [`examples/AkashGoswamy_gemini`](examples/AkashGoswamy_gemini) (401 for a bad key, 404 for a retired model; see also [#8](https://github.com/Vedantgitbot/booth/pull/8)). + +--- + +## 2. Wrong answers come back `ACCEPTED` with confidence 1.0 + +**Symptom.** The model gives a factually wrong answer, and `check()` returns `ACCEPTED` with `confidence=1.0`. + +**Cause.** The confidence value is self-reported by the model; BOOTH does not verify it. `check()` checks format and internal consistency (parseable output, confidence above the threshold, no ambiguity flag), not facts. This is expected behavior, not a bug. In one run on a local 7B model, all three wrong answers still came back with `confidence: 1.0`. + +**What to do.** Do not treat `ok=True` from `check()` as "the answer is true". If you have ground truth, use `check_with_evidence()` to compare the answer against it. Smaller models tend to over-report confidence. + +**Seen in:** [`examples/Maintythehuman_ollama`](examples/Maintythehuman_ollama). + +--- + +## 3. 404 / "model not found" from the provider + +**Symptom.** An example copied from another folder or an older tutorial fails with a 404 or "model not found", often hidden behind entry 1 above. + +**Cause.** Provider model names get retired. During the Gemini example, `gemini-2.5-flash-lite` stopped being available to new users, and the API returned a 404 pointing to `gemini-3.5-flash-lite` instead. + +**What to do.** Do not assume a model name in an older example is still valid. Check the provider's current model list, and make the model configurable (for example through an environment variable) so a rename is a one-line change. + +**Seen in:** [`examples/AkashGoswamy_gemini`](examples/AkashGoswamy_gemini). + +--- + +## 4. 401 with a new-style API key + +**Symptom.** `401 UNAUTHENTICATED` (for example `ACCESS_TOKEN_TYPE_UNSUPPORTED`) even though the key was copied correctly. + +**Cause.** Newer provider key formats may not be understood by older SDK versions. In the Gemini example, a new-style key (prefix `AQ.` instead of `AIza`) worked only after upgrading the SDK. A deleted or wrong key also returns a 401, so rule that out first. + +**What to do.** Upgrade the provider SDK (`pip install -U `), confirm the key is the one currently loaded in your environment, and run it once outside BOOTH to separate a key problem from a BOOTH problem. Remember that `check()` hides the error unless you use `on_attempt=` (entry 1). + +**Seen in:** [`examples/AkashGoswamy_gemini`](examples/AkashGoswamy_gemini). + +--- + +## 5. An ambiguous question was not flagged `AMBIGUOUS` + +**Symptom.** The same kind of ambiguous question is flagged `AMBIGUOUS` in one setup and passes through unflagged in another. For example, "What is the capital of Georgia?" (the country or the US state) was not flagged against a local 7B model, which simply picked one interpretation, while a vague question in the Gemini example was flagged correctly. + +**Cause.** Ambiguity detection relies on the model noticing the ambiguity and saying so in its response. It is model-dependent and not guaranteed. This is a limitation of the approach, not inconsistent behavior from BOOTH itself. + +**What to do.** Do not rely on `AMBIGUOUS` as a complete safety net for vague input. Check `result.status` or `result.ambiguous`, not `confidence`, when detecting it: in one run the ambiguous case came back with `confidence=1.0` and the explanation sitting in `.answer`. Where it matters, disambiguate the question before sending it. + +**Seen in:** [`examples/Maintythehuman_ollama`](examples/Maintythehuman_ollama) (Georgia question not flagged), [`examples/AkashGoswamy_gemini`](examples/AkashGoswamy_gemini) (a different vague question flagged `AMBIGUOUS`). + +--- + +## 6. `check_with_evidence()` accepts answers it should block + +**Symptom.** An answer that contradicts your evidence is `ACCEPTED`. + +**Cause.** `check_with_evidence()` only runs the `compare_fn` you give it. A weak comparator gives a weak checkpoint. BOOTH can detect a `compare_fn` that crashes or returns something unusable (`result.checker_failed`), but not one that runs fine with bad logic. See [USECASES.md §27](USECASES.md). + +**What to do.** Test your `compare_fn` on its own with answers you know should pass and fail, including a deliberately wrong one, before wiring it in. Simple keyword or stance checks are fine for demos; use an embedding, entailment, or LLM-judge comparison for real use. + +**Seen in:** nearly every example, for instance [`examples/Vedantgitbot_groq`](examples/Vedantgitbot_groq) and the Anthropic example ([`examples/opencode13241-eng_anthropic`](examples/opencode13241-eng_anthropic)), whose RESULTS.md lists lexical phrase matching as a limitation.