From 7d6e9d8b13831872a29595b10a4086a41eca40d5 Mon Sep 17 00:00:00 2001
From: Damien Vande Kerckhove <damien.vandekerckhove@gmail.com>
Date: Fri, 17 Apr 2026 15:44:41 +0200
Subject: [PATCH] doc : How to write evals for a skill

---
 README.md | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)

diff --git a/README.md b/README.md
index 7651a29..8f8afbd 100644
--- a/README.md
+++ b/README.md
@@ -115,6 +115,87 @@ The UI runs on `http://localhost:3000` and proxies `/api` to the backend.
 | Database | SQLite |
 | Deployment | Docker Compose |
 
+## Writing Evals for a Skill
+
+Arctiq discovers evals from a file named `evals/evals.json` inside each skill's directory in the Git repository.
+
+### File location
+
+```
+your-skill-repo/
+└── your-skill/
+    ├── skill.md          # skill instructions
+    └── evals/
+        └── evals.json    # eval test cases
+```
+
+### Format
+
+```json
+{
+  "evals": [
+    {
+      "id": 1,
+      "name": "optional case name",
+      "prompt": "The input prompt sent to the model",
+      "assertions": [
+        { "id": "a1", "text": "Response is concise and under 100 words", "weight": 1.0 },
+        { "id": "a2", "text": "Answer mentions the correct library name", "weight": 2.0 }
+      ]
+    }
+  ]
+}
+```
+
+### Fields
+
+| Field | Required | Description |
+|---|---|---|
+| `evals[].id` | Yes | Integer — unique identifier for the test case |
+| `evals[].prompt` | Yes | The prompt sent to the model under test |
+| `evals[].name` | No | Human-readable label shown in the UI |
+| `evals[].assertions` | No | List of criteria judged by the LLM judge |
+| `assertions[].id` | Yes | String — unique identifier for the assertion |
+| `assertions[].text` | Yes | The criterion evaluated by the judge model (YES/NO) |
+| `assertions[].weight` | No | Scoring weight (default `1.0`). Higher = more impact on the final score |
+
+### Scoring
+
+Each assertion is evaluated independently by a judge model using a strict YES/NO prompt. The final score is a weighted percentage:
+
+```
+score = sum(weight of passed assertions) / sum(weight of all assertions) × 100
+```
+
+An eval case with no assertions is recorded but produces no score.
+
+### Example
+
+```json
+{
+  "evals": [
+    {
+      "id": 1,
+      "name": "Summarise a short article",
+      "prompt": "Summarise the following article in 2–3 sentences:\n\nThe James Webb Space Telescope...",
+      "assertions": [
+        { "id": "s1", "text": "Summary is 2 to 3 sentences long", "weight": 1.0 },
+        { "id": "s2", "text": "Summary does not introduce facts not present in the article", "weight": 2.0 },
+        { "id": "s3", "text": "Summary is written in plain English", "weight": 1.0 }
+      ]
+    },
+    {
+      "id": 2,
+      "name": "Empty input handling",
+      "prompt": "Summarise the following article:\n\n",
+      "assertions": [
+        { "id": "e1", "text": "Model asks for or acknowledges missing input rather than hallucinating a summary", "weight": 1.0 }
+      ]
+    }
+  ]
+}
+```
+
 ## License
 
 [MIT](LICENSE)