skills(aibi-dashboards): backport widget index + dataset measures + new widgets from a-d-k #558

[QuentinAmbard]

Why this PR exists

Backports databricks-solutions/ai-dev-kit#558 ("Dashboard skill: widget index + dataset measures + new widgets + forecast partial-bucket fix") plus four follow-up commits on the same files to the DAS databricks-aibi-dashboards experimental skill:

• da45f3a — queryLines newline rule + borderless theme trick (widgetBorderColor).
• 38c4491 — explain why fewer datasets = better cross-filtering UX.
• 084db5f — pure-white light-mode canvas in the default theme (#FCFCFC → #FFFFFF).
• 76103a0 — drop the format block from MEASURE()-backed counters in the worked example (triggers the browser's "unsupported widget definition was automatically fixed" warning).

Same content as upstream; this is a straight backport.

What this PR improves

Widget index — one-stop lookup. SKILL.md now opens with a table mapping every supported widget type (counter, table, line/bar, pie, area, scatter, combo, forecast-line, pivot, histogram, heatmap, funnel, sankey, box, waterfall, choropleth-map, symbol-map, all filter widgets) to its spec version and the exact reference file + section + line number where it's documented. Agents skimming for "do I have a sankey?" or "is forecast supported?" no longer need to read three files to find out.

Dataset measures + MEASURE(). New top-level pattern for defining reusable aggregations on the dataset rather than per-widget. Reduces query bloat when many widgets share the same aggregation and lets MEASURE() invocations stay readable.

Cross-filtering UX rule (from 38c4491). Clicking a value in a chart filters that chart's dataset and cascades to every widget sharing it. Splitting widgets across many narrow datasets silently breaks the cross-filter experience. Skill now leads with this so the agent designs around one wide dataset per domain instead of one-dataset-per-widget.

Advanced widgets backfilled. Forecast-line (with the AI_FORECAST SQL pattern), pivot with conditional cell rules, sankey, choropleth-map, symbol-map, histogram with bin(col, binWidth=N), box, waterfall — all gain dedicated sections in 2-advanced-widget-specifications.md.

Forecast partial-bucket fix. The trap where the last forecast bucket renders as a half-height bar (because the prediction window straddles the current period boundary) is now called out with the fix in 5-troubleshooting.md.

queryLines verbatim-join warning (from da45f3a). queryLines array elements are joined character-for-character at runtime — a line ending in -- comment with no trailing newline swallows the next line. Now explicit, plus the forecast example that originally hit this is fixed.

Borderless theme trick (from da45f3a). Document widgetBorderColor and recommend setting it equal to widgetBackgroundColor to hide the busy default border (matches the dbdemos customer-support reference dashboard).

Light-mode canvas tweak (from 084db5f). Default theme's light canvas switches from #FCFCFC (reads as off-white/beige next to true-white widget cards) to #FFFFFF so the example renders as one flat modern surface instead of a faint beige page. Dark-mode pair unchanged.

MEASURE-counter format-block warning (from 76103a0). A counter whose value is a MEASURE() plus a format: {type: \"number\", ...} block triggers the browser's "unsupported widget definition was automatically fixed" warning. Omitting the format renders cleanly as an integer. The three KPI counters in the worked example (Total Cases, Avg Resolution Hours, Reopen Rate) drop the offending block.

Cohort retention = pivot with color-scale. Explicit note that there is no cohort widget type — agents would otherwise either invent one or get stuck.

Reference template in 4-examples.md. Single complete working dashboard exercising every construct (dataset measures + MEASURE(), sparkline counters, forecast-line with annotations, pivot with conditional cells, symbol-map, histogram, range-slider filter, theme). Replaces the prior fragmented examples.

Summary of changes

File	What changed
SKILL.md	New widget-index table at the top with spec version + reference file + line number per widget. Cross-filtering rationale up front. widgetBorderColor theme trick. Light-mode canvas #FCFCFC → #FFFFFF. Cohort-retention-via-pivot clarification. Internal links updated to DAS's references/ layout.
references/1-widget-specifications.md	Counter / table / line / bar / pie widgets refreshed; added text widget structure.
references/2-advanced-widget-specifications.md	Added/expanded forecast-line, pivot (conditional cells), histogram, heatmap, funnel, sankey, box, waterfall, choropleth-map, symbol-map. Area and scatter sections expanded.
references/3-filters.md	Filter-widget structure section added.
references/4-examples.md	Replaced prior fragmented examples with a single complete reference dashboard exercising every construct. `queryLines` forecast example fixed for the verbatim-join rule. Theme block matches the new `#FFFFFF` light-mode canvas. `MEASURE()`-backed KPI counters drop the `format` block to avoid the browser auto-fix warning.
references/5-troubleshooting.md	Added forecast partial-bucket trap.

Path remapping

The upstream files live at databricks-skills/databricks-aibi-dashboards/N-widget-foo.md (flat in databricks-skills/); DAS uses experimental/databricks-aibi-dashboards/references/N-widget-foo.md. Content is identical; only the file location differs. SKILL.md's sibling-style links to reference files were rewritten to references/-prefixed to match DAS's layout. One cross-skill link to databricks-spark-declarative-pipelines (the a-d-k name) was renamed to databricks-pipelines (DAS's name for the same skill) so it resolves once installed.

Reviewer aid

Upstream PR: ai-dev-kit#558. Follow-up commits: da45f3a, 38c4491, 084db5f, 76103a0. Diff is a straight content backport.

This pull request and its description were written by Isaac.

[simonfaltum]

Reviewed as a skill gate -- verified the load-bearing claims against Databricks docs and a local scripts/skills.py validate run, not just a read. Net: in good shape, fix-then-merge. Two items worth fixing before merge, the rest are polish -- all left inline.

Fix before merge

1. The frontmatter drops parent / compatibility / metadata.version, which rolls the published manifest version backward (0.1.0 -> 0.0.1) and breaks consistency with every other experimental skill. (inline on SKILL.md)
2. BIN_FLOOR is presented as raw SQL but isn't in the SQL function reference -- needs confirmation it's a valid dashboard expression, since the skill tells agents to CLI-test all SQL. (inline on 2-advanced)

Polish (inline): hard-coded (L###) line numbers in the widget index (one already wrong), unverified symbol-map / range-slider widgetType strings, one cloud-specific docs URL.

Verified clean

• scripts/skills.py validate passes; manifest / Codex metadata / icons / plugin manifests all in sync (the version flip is a regen side effect, not a hand-edit).
• ai_forecast signature + output columns (*_forecast / _upper / _lower, horizon / time_col / value_col) match the docs.
• Dataset columns[] + MEASURE() is a real feature (dataset-scoped custom calculations).
• The new widget types (pivot, histogram, heatmap, sankey, funnel, box, waterfall, choropleth, combo, area, scatter) are all real per the visualization-types doc.
• All widget-index anchors and cross-file links resolve.
• The databricks-spark-declarative-pipelines -> databricks-pipelines rename is correct.
• Deleting the old "do NOT use mappings / colorRamp on bar/line" rule looks right -- per-value categorical color mapping is a documented feature (2024 AI/BI release notes). Worth a quick render check of the JSON shape.

Useful backport overall -- the widget index and the single reference dashboard are genuinely good additions.

[simonfaltum]

Frontmatter dropped (fix before merge). This change removes three frontmatter fields that exist on main: parent: databricks-core, compatibility: Requires databricks CLI (>= v1.0.0), and metadata.version: "0.1.0". The PR description doesn't mention it, so it looks like the upstream ai-dev-kit frontmatter was copied over the destination's.

Two consequences:

• Version regression. scripts/skills.py derives the manifest version from metadata.version and falls back to a 0.0.1 floor when it's absent, which is why manifest.json flips 0.1.0 -> 0.0.1 (a backward version for consumers).
• Consistency. Every other experimental skill carries parent + compatibility (backfilled in experimental: backfill metadata.version + parent + compatibility frontmatter #105 six days ago); this drops them for one skill and removes the databricks-core hierarchy link.

Suggest restoring the three lines, then python3 scripts/skills.py generate so the manifest version returns to 0.1.0 (or bump to 0.2.0 for this feature release).

[simonfaltum]

This 0.1.0 -> 0.0.1 downgrade is a side effect of dropping metadata.version from the SKILL.md frontmatter (see my comment there). validate passes because the manifest faithfully regenerates from the now-missing field, but the published version still moves backward. Restoring the frontmatter and re-running scripts/skills.py generate fixes it.

[simonfaltum]

BIN_FLOOR unverified (fix before merge). It isn't in the Databricks SQL function reference (only bin, floor, histogram_numeric), so calling it "the actual SQL expression" is misleading -- and this skill's headline rule is "test ALL SQL queries via CLI before deploying," which would fail on BIN_FLOOR(...). It's plausibly a valid dashboard field-expression function (it lives in a widget expression, not dataset SQL). Could you confirm it renders (a working dashboard from the upstream PR is enough) and, if it's dashboard-expression-only, reword this so it doesn't claim to be raw SQL? Same expression is used at 4-examples.md:342.

[simonfaltum]

This replaces the previous marker-map entry. The visualization-types doc calls it a "point map," and I couldn't confirm the exact serialized widgetType string from public docs. Worth double-checking symbol-map is what the dashboard JSON actually uses (vs marker-map / point-map), since a wrong widgetType silently breaks the widget.

[simonfaltum]

Every other filter in this skill is filter--prefixed (filter-single-select, filter-multi-select, filter-date-range-picker), but this one is bare range-slider. The range-slider filter is real (confirmed in the 2024 AI/BI release notes); please just confirm the serialized widgetType is actually un-prefixed and the version is 2.

[simonfaltum]

The hard-coded (L###) hints in this index will drift on the next edit to any reference file -- and this one is already wrong: the Range Slider heading is at line 282 of 3-filters.md, not 239 (L239 is the Multi-Select Parameters section). The anchor itself resolves, so the link works; consider dropping the (L###) hints and relying on the anchors, which are stable.

[simonfaltum]

House style is cloud-neutral docs URLs -- strip the /aws/en/ prefix: https://docs.databricks.com/dashboards/manage/visualizations/types. (The line this replaced was already neutral.)