Skip to content

[RFC] Vega Visualization Integration in the OpenSearch Discover Experience #22829

Description

@KarstenSchnitter

Is your feature request related to a problem? Please describe

Summary

This RFC proposes extending the OpenSearch Dashboards Discover experience to surface Vega as a first-class visualization option. OpenSearch 3.8 significantly advanced the built-in discovery and analytics experience with a PPL visual query builder, SQL support, new PPL commands, and richer log exploration tooling. However, a set of analytically valuable use cases remains out of reach for users who rely only on the standard chart types and query builders. This RFC identifies those use cases and proposes how Vega integration can close the gap without replacing what the new Discover experience already does well.

Goals

  • Identify concrete analytical use cases that the standard Discover chart types cannot serve, and that Vega can.
  • Propose a pathway from a PPL/SQL query in Discover to a Vega visualization, so users do not have to re-author query and filter context by hand.
  • Define a convention for connecting Vega panels to the existing Dashboard Variables feature, in both read and write directions.

Non-Goals

  • Replacing the standard Discover charting experience. The built-in chart types remain the right tool for the majority of workflows. Vega is an escape hatch for cases they cannot express, not a default.
  • Building a general-purpose visualization designer. This RFC does not propose a drag-and-drop Vega authoring surface; specs are authored as JSON.
  • Changing Vega or Vega-Lite themselves. The proposal is about integration glue within OpenSearch Dashboards, not modifications to the upstream grammar.

What we are asking

This RFC seeks community and TSC feedback on two questions: (1) whether the use cases below justify investing in tighter Vega integration, and (2) which of the three proposed integration points (P1–P3) should be prioritized. P1 delivers value independently; P2 and P3 are additive.


Motivation

OpenSearch 3.8 delivered meaningful improvements to the discovery and analytics experience:

  • A visual PPL query builder lets users construct queries without writing syntax.
  • Experimental SQL support in Discover provides a familiar tabular interface.
  • New PPL commands (timewrap, xyseries, foreach, makeresults) enable richer data shaping before visualization.
  • Lazy panel loading and alert creation from the metrics page improve dashboard ergonomics.

These features lower the barrier to entry and cover the majority of log analytics and metric exploration workflows. However, they share a common constraint: the output visualizations are rendered by a fixed set of standard chart types (bar, line, area, pie, metric, etc.). When a user's analytical question does not map cleanly to one of those chart types — or when a visualization requires behavior that the chart builder does not expose — there is currently no escape hatch within the Discover flow.

Vega is already present in OpenSearch Dashboards as a panel type, but it is essentially a blank canvas disconnected from the Discover experience. Users must manually wire up the OpenSearch data source in a Vega/Vega-Lite JSON spec, and there is no pathway from the PPL/SQL query builder to a Vega output. This RFC asks: what would it look like to close that gap?

There is also a data model affinity that makes PPL a particularly natural partner for Vega. PPL query results are tabular data frames — rows of objects with named fields — which map directly to Vega's data arrays without transformation. Classic OpenSearch DSL responses, by contrast, return deeply nested aggregation bucket structures that must be flattened and reshaped before Vega can consume them, adding error-prone boilerplate to every spec. The shift to PPL in the Discover experience therefore reduces the integration cost for Vega significantly, making the pathway described in P1 more practical than it would have been in a DSL-centric workflow.


Use Cases

The following use cases are not achievable — or are severely degraded — using only the built-in chart types and the current Discover flow.

1. Chart Types That Do Not Exist in the Standard Library

Standard panels are limited to bar, line, area, pie, metric, and a handful of variants. These answer how much and over time, but not structural or relational questions. Some analytical questions simply require a different visual form, and no amount of PPL reshaping can produce them from a bar chart.

Log severity by service — Sankey diagram: An index contains OTel fields service.name and severity_number. A Sankey diagram with services as source nodes and severity levels as target nodes, with flow width proportional to log volume, immediately answers "which services produce the most warnings and errors?" A grouped bar chart can carry the same data, but the viewer must compare bar heights across groups; in a Sankey the relative flow width is the answer, and outliers are visible without scanning.

Customized service dependency graph — force-directed network: OpenSearch provides a built-in service map derived from trace data. Where the standard service map is sufficient, no custom visualization is needed. Vega becomes relevant when additional dimensions must be encoded on the graph. A multi-tenant platform is a practical example: nodes can be colored by tenant.id, edge width scaled by request volume, and the visible graph filtered to a selected tenant via an osd_tenant signal — making it immediately clear which services a given tenant's traffic touches and where error rates are elevated. These encodings require direct control over mark properties that the built-in service map does not expose.

Resource hierarchy — treemap or sunburst: Log volume grouped by service, host, and log_level forms a three-level hierarchy. A treemap encodes the full hierarchy and relative sizes in a single panel; nested bar charts approximate this but require separate panels per level.

2. Visualization Across Multiple Dimensions Simultaneously

Standard chart types expose at most two or three visual encodings simultaneously — typically x-axis, y-axis, and color. Vega allows a single mark to encode four or five independent data dimensions at once, and crucially, to make those encodings interactive so the user can select from the actual data rather than from pre-enumerated lists.

Kubernetes workload identification is a representative observability scenario. OpenTelemetry and Kubernetes resource attributes produce fields such as k8s.namespace.name, k8s.pod.name, k8s.container.name, and an owner reference that may be a k8s.deployment.name, k8s.statefulset.name, k8s.job.name, or k8s.replicaset.name — with no guarantee which of these is populated for any given record. With Istio sidecar injection, each pod carries at least two containers (app and istio-proxy), but the full space of namespace × owner × pod × container combinations is sparse: most combinations do not exist.

This creates two problems for standard visualizations:

  • Structural heterogeneity: The owner dimension has no fixed field name. A standard chart cannot group by "whichever of deployment, statefulset, job, or replicaset is present" without preprocessing. PPL can compute a derived owner field with a conditional expression, but the chart editor still has no way to represent the resulting hierarchy visually.
  • Sparse selection: A dropdown control for pod_name lists all known pods regardless of what exists in the current time window or given the selected namespace. The user selects from a list; the visualization does not reveal which combinations are meaningful.

A Vega panel addresses both. A faceted scatter plot — one facet per namespace, each mark a pod sized by log volume and colored by error rate — renders only the combinations that exist in the query result. The owner type can be encoded as mark shape (circle for deployment pods, square for statefulset pods, diamond for job pods). Clicking a mark sets osd_pod and osd_container simultaneously via the signal system, driving filters on every other panel in one gesture rather than requiring sequential dropdown selections.

The interaction model is the key distinction: the user selects from the data as it actually exists rather than from a pre-enumerated attribute list.

3. Interactive Visualizations That Write Back to Dashboard State

Standard panels are read-only participants in dashboard state: they consume the global time filter and filter bar, but they cannot modify them. Vega's signal system, combined with the osd_ variable integration described in P2, enables panels that drive the dashboard rather than just responding to it.

Time range selection via brush: A Vega panel renders a high-level overview of event volume across a long time window. The user drags a brush selection over a region of interest; on release, the osd_timeRange signal fires and updates the global time filter, zooming all other panels into the selected window. This pattern — overview-plus-detail navigation — is well established in visual analytics but is not achievable with any standard panel type.

Drill-down via node click: A force-directed service graph panel has a dashboard variable osd_selectedService. Clicking a node in the graph sets osd_selectedService to that service's name, which drives a filter on every other panel on the dashboard — log panels, metric panels, and alert history — without the user touching the filter bar. The interaction is direct manipulation of the visualization rather than a separate UI control.

Both patterns depend on the Dashboard Variables feature introduced in OpenSearch Dashboards 3.7 (opensearch-dashboards#11550) and the osd_ signal convention described in P2 of this RFC.


Proposal

The use cases above suggest three distinct integration points. P1 is self-contained and can be implemented independently. P2 builds on Dashboard Variables, which shipped in OpenSearch Dashboards 3.7, and requires integration work to bridge the existing text-substitution system with Vega's signal model. P3 is optional tooling that can follow at any point.

P1 — PPL/SQL Query Result as a Vega Data Source

When a user executes a PPL or SQL query in the Discover logs or metrics view, they can currently send the result to a standard chart. The proposal is to add a "Visualize with Vega" export path that:

  1. Wraps the current query (PPL or SQL) in an OpenSearch Vega data source block automatically.
  2. Forwards the active time filter and all dashboard filters into the generated data block, exactly as they would apply to any standard panel — users should not have to re-express filter context.
  3. Opens a Vega/Vega-Lite spec editor pre-populated with the data binding and a field inventory derived from the result schema.
  4. Allows the user to author the visual encoding from there, then save the panel to a dashboard.

This removes the single biggest friction point: users currently have to re-express their query in Vega's %context% / %timefield% syntax from scratch, duplicating work already done in the query builder. Filter and time context forwarding ensures the Vega panel behaves consistently with every other panel on the dashboard — changing the global time picker or applying a filter bar entry affects Vega panels without any spec changes.

P2 — Dashboard Variables Integration

Dashboard variables were introduced in OpenSearch Dashboards 3.7 with a $name / ${name} text substitution syntax that rewrites query strings before execution. Vega panels should be able to consume and emit these variables, but the integration requires bridging two structurally different mechanisms: the existing system substitutes text into query strings before they run, while Vega's signal system operates on live runtime values inside an executing spec. The proposal is to have the integration layer resolve current variable values and inject them as named osd_ signals before the Vega spec is evaluated, and to propagate osd_ signal writes back as variable updates after each settled interaction.

Signal naming convention

Dashboard variables are exposed as Vega signals with the prefix osd_. A dashboard variable named region arrives in the spec as the signal osd_region and can be referenced anywhere a signal expression is valid:

{ "signal": "osd_region" }

The osd_ prefix keeps the signal namespace clean — spec authors can freely define signals such as brushExtent or selectedBar without risk of colliding with an injected dashboard variable.

Reading variables

The integration layer injects each dashboard variable as a top-level signal before the spec is evaluated. No explicit declaration is required in the spec; referencing osd_<variableName> is sufficient. When the dashboard variable changes (e.g., the user selects a new value from a variable dropdown), the integration updates the corresponding signal, triggering any dependent marks or data transforms automatically.

Writing variables — settled event pattern

When a Vega signal named osd_<variableName> changes value, the integration layer propagates that value back to the corresponding dashboard variable, which may trigger re-queries in other panels. To avoid query storms during continuous interactions such as dragging a brush, spec authors should follow a two-signal pattern: a local signal that streams freely for smooth visual feedback, and a separate osd_ signal that commits only on a settled event.

[
  {
    "name": "brushExtent",
    "on": [{
      "events": "[@rect:mousedown, window:mouseup] > window:mousemove",
      "update": "invert('xscale', [down, x()])"
    }]
  },
  {
    "name": "osd_timeRange",
    "on": [{ "events": "window:mouseup", "update": "brushExtent" }]
  }
]

brushExtent updates on every mousemove, keeping the visual selection responsive. osd_timeRange — the signal the integration layer watches — updates only on mouseup, so the dashboard variable (and any dependent re-query) fires once per completed interaction rather than on every intermediate drag position. This pattern should be documented as the idiomatic approach in the developer documentation.

P3 — Vega Spec Editor with OpenSearch-Aware Tooling (Optional)

The existing Vega editor in Dashboards is a plain textarea with syntax highlighting. The following improvements would reduce friction for authors who work directly in the spec editor. Given that Vega specs are increasingly authored with the assistance of coding agents, these enhancements are useful but not critical to the core integration.

  • Index pattern picker and field autocomplete within the spec editor, consistent with the PPL builder's field selector.
  • Inline data preview: Render the result of each data block in a collapsible table below the editor, so users can verify their query and transformation without switching tabs.
  • Error surfacing: Parse Vega runtime errors and map them back to spec lines, rather than surfacing only a blank panel.

Alternatives Considered

Vega-Lite only: Vega-Lite is a higher-level grammar built on top of Vega and covers the majority of standard chart types more concisely. However, Vega-Lite cannot express all of the custom chart types in the use cases above (Sankey, force-directed network, treemap). Supporting Vega-Lite as a shorthand while keeping full Vega available preserves simplicity for common cases without cutting off advanced ones. OpenSearch Dashboards already supports both.

Custom visualization plugins per chart type: Rather than exposing Vega generically, one could add Sankey, force-graph, and treemap panels as separate built-in types. This approach produces more polished individual chart types but does not address multi-dimensional sparse encodings or interactive write-back to dashboard state — the use cases that are structurally unique to Vega.

Notebook-based visualization (e.g., Observability Notebooks): OpenSearch Notebooks allow Vega specs to be embedded alongside prose. However, notebooks are a different workflow from the Discover/Dashboard experience and are not accessible to users who work primarily in the visual PPL builder. The two should be complementary rather than alternatives.

Additional context

  1. Security model for external data sources: Vega specs can issue HTTP requests to arbitrary URLs. The existing panel already supports a config.kibana.allowedHosts restriction. Should the integration make this constraint more discoverable, or should it be tightened further for the Discover-embedded flow?

  2. Variable substitution vs. signal semantics: Dashboard variables use $name text substitution applied before a query runs. Vega signals are live runtime values inside an executing spec. The integration layer must bridge these: resolving variable values at spec evaluation time for reads, and deciding how signal writes propagate back (immediately, on settled event, or only on explicit user confirmation). Are there variable types — for example, multi-value variables or variables that affect query structure rather than just field values — that cannot be cleanly represented as Vega signals?

  3. Mobile and embed rendering: Vega uses Canvas or SVG rendering. How should Vega panels behave on small viewports or in the OpenSearch Dashboards embed (iframe) mode?


References


Request for Comments

Feedback is welcome on any part of this proposal. In particular:

  • Do the use cases resonate? Are there analytical needs you have hit that the standard chart types could not serve, that Vega would? Concrete examples strengthen the case.
  • Prioritization. Is P1 (query-to-Vega export) the right first step, or would P2 (Dashboard Variables integration) deliver more value sooner for your workflows?
  • The osd_ signal convention. Does the naming scheme and the read/write semantics fit how you would expect variables and Vega to interact?
  • Concerns not captured in Open Questions. Security, performance on large result sets, or maintenance burden.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions