feat: aicertify demo command + Markdown renderer bugfix (v0.7.1)#63
Merged
Conversation
The pre-existing CLI was a flat one-shot that required both --contract and --policy, with no bundled sample. After a fresh `pip install aicertify`, a user could not run anything without cloning the repo for examples/sample_contract.json. The README's "Quick Start" line was therefore not actually executable. This change adds: - `aicertify demo` — a self-contained subcommand. Loads a bundled sample contract (aicertify/_demo/sample_contract.json), runs the canonical evaluation pipeline against the EU AI Act policy set, and writes a Markdown report to the current working directory. Requires only the `opa` binary on PATH (no contract file, no API keys for the OPA path). - `aicertify evaluate` — the previous flat CLI behaviour, now under an explicit subcommand. - Backwards-compat shim: `aicertify --contract X --policy Y …` (the pre-0.7.1 invocation) is silently rewritten to `aicertify evaluate --contract X --policy Y …`, so old scripts keep working. - Friendly platform-aware install message when the `opa` binary is not on PATH — replaces the previous stack trace. The demo uses the same code path as `aicertify evaluate` and as `python examples/quickstart.py`; it is not a hand-crafted approximation. What a user sees in the demo report is what the canonical pipeline produces.
Two pre-existing bugs in the Markdown report renderer prevented any
EvaluationReport from being formatted whenever metric groups were
present. The same path is shared by the PDF report (which converts
Markdown internally), so PDF reports were affected too.
1. `MetricGroup.metrics` is declared as `List[Dict[str, Any]]` in
aicertify/models/report.py and `create_metric_group()` populates
it with dicts. The renderer was reaching into each metric with
attribute access (`metric.display_name`, `metric.value`), raising
`AttributeError: 'dict' object has no attribute 'display_name'`
the moment a policy returned any metric.
2. `EvaluationReport.summary` is declared as `Dict[str, Any]` and is
populated as such by create_evaluation_report(). The renderer was
appending it directly to a list of markdown lines that gets `"\n".
join`-ed at the end, causing `TypeError: sequence item N: expected
str instance, dict found`. Now the summary dict is rendered as one
bullet per key, with display-cased labels.
Reproducible with:
aicertify demo # before: crashed
aicertify evaluate --contract X --policy eu_ai_act \\
--report-format markdown # before: crashed
After this fix, both paths produce a clean Markdown audit report. The
JSON path was unaffected (it never went through this renderer) and
keeps working as before.
- README.md and 4 translated READMEs (zh-CN, ja-JP, ko-KR, hi-IN):
collapse the Quick Start to three honest, executable commands:
pip install aicertify # ~3-5 min first install
curl -L … opa … && chmod +x … # one-time OPA binary
aicertify demo # ~10 sec, no API keys
Removes the previous "git clone the repo + python examples/quickstart.py"
workaround. Adds an explicit pointer to the heavier examples/* path for
users who want LangFair / DeepEval / PDF output.
- CHANGELOG: 0.7.1 entry documenting the demo subcommand, the back-compat
shim, the OPA-binary detection UX, and the Markdown renderer bug fixes.
- AGENTS.md: "Useful commands" updated — `aicertify demo` is now the
fastest sanity check; `aicertify evaluate` example replaces the path-y
`python -m aicertify.cli --contract … --policy …` legacy invocation.
- Version bump to 0.7.1 in pyproject.toml + aicertify/__init__.py.
github-actions
Bot
added
📝 documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make
pip install aicertify && aicertify demoactually work as a self-contained Quick Start. Three logical commits in one PR.1.
feat(cli): aicertify demo subcommandThe pre-existing CLI required
--contractand--policywith no bundled sample, so the README's Quick Start line was not literally executable afterpip install aicertify. This adds:aicertify demo— loads a bundled sample contract (aicertify/_demo/sample_contract.json), runs the canonical evaluation pipeline against the EU AI Act policy set, and writes a Markdown report to the cwd. No contract file, no API keys required for the OPA path.aicertify evaluate— the previous flat CLI, now under an explicit subcommand.aicertify --contract X --policy Y …(pre-0.7.1 invocation) is silently rewritten toaicertify evaluate …so old scripts keep working.The demo uses the same code path as
aicertify evaluateandpython examples/quickstart.py. Not a hand-crafted approximation — what users see in the demo report is what the canonical pipeline produces for any contract.2.
fix(report-generation): Markdown renderer bugsTwo pre-existing bugs blocked Markdown (and therefore PDF) reports from being generated whenever an EvaluationReport had any metric groups:
MetricGroup.metricsisList[Dict[str, Any]]per the model schema, populated as dicts bycreate_metric_group(). The renderer was using attribute access (metric.display_name,metric.value) →AttributeError.EvaluationReport.summaryisDict[str, Any]. The renderer was appending it directly to a list of markdown lines that gets"\n".join-ed at the end →TypeError: sequence item N: expected str instance, dict found.Affects every user who ran any
--report-format markdownor--report-format pdfevaluation that produced metrics, not just the demo.3.
docs+release: v0.7.1Test plan
aicertify demo(with OPENAI_API_KEY in env) → ~30s run with full ML evaluator pipeline, clean Markdown report writtenOPENAI_API_KEYunset → ~10s OPA-only run, same canonical Markdown reportaicertify demo --format json→ JSON path still worksv0.7.1, build wheel,twine upload dist/*to PyPINotes for the maintainer
00000000-0000-0000-0000-000000000d3e) and the requiredmodel_infofield — verified againstAiCertifyContractPydantic schema.pip install aicertify && aicertify demowork for end users. v0.7.0 users will not have thedemosubcommand until they upgrade.