Skip to content

Feat: scheduler support for device groups#1934

Merged
Flix6x merged 68 commits into
mainfrom
feat/switching-between-gas-and-electricity
Jun 3, 2026
Merged

Feat: scheduler support for device groups#1934
Flix6x merged 68 commits into
mainfrom
feat/switching-between-gas-and-electricity

Conversation

@Ahmad-Wahid

@Ahmad-Wahid Ahmad-Wahid commented Jan 27, 2026

Copy link
Copy Markdown
Contributor

This PR extends the Commitment model and device scheduler to support grouped devices within a single commitment. This work is the basis for quite the list of follow-up features:

Why

Until now, a commitment could apply to:

  • a single device, or
  • the entire EMS.

This was insufficient for real‑world setups such as:

  • multiple producers feeding a shared thermal buffer,
  • several devices jointly constrained by a single stock target,
  • aggregated soft limits that should be penalized once per group, not per device.

What changed

  • Commitments now support custom device groups, allowing a single

  • FlowCommitment or StockCommitment to aggregate over multiple devices.

  • The scheduler aggregates flows or stocks per device group and applies deviation penalties once per group per timestep.

  • Commitments may reference:

    • a single device,
    • multiple devices (via list‑valued device entries),
    • or multiple disjoint groups within the same commitment.
  • New Pyomo sets (cg, cjg) and constraints enforce grouped commitment
    equalities consistently for both flow and stock commitments.

  • Cost reporting correctly aggregates costs per logical commitment instead of
    duplicating them per device.

Result

This enables correct modelling of shared buffers and feeder‑level soft
constraints without artificial cost inflation or duplicated baselines.

How to test

1️⃣ Run the new unit tests

pytest -k test_multi_feed_device_scheduler_shared_buffer
pytest -k test_device_group_shared_buffer

These tests verify:

  • two devices jointly satisfy a single stock target,
  • deviation costs are applied once per group, not per device,
  • gas and electricity commitments are costed separately and consistently.

2️⃣ Sanity‑check behaviour (what to look for)

  • Only one commitment group exists for the shared thermal buffer
  • Battery remains in its own independent group
  • Total commitment costs do not scale with number of devices
  • Electricity and gas costs match expected values

Flix6x and others added 19 commits December 5, 2025 16:50
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…ontext

Signed-off-by: F.N. Claessen <felix@seita.nl>
…formulation

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
…and map them to the respective group id

Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
@read-the-docs-community

read-the-docs-community Bot commented Jan 27, 2026

Copy link
Copy Markdown

Documentation build overview

📚 flexmeasures | 🛠️ Build #32423445 | 📁 Comparing a9174a5 against latest (5d0d44f)

  🔍 Preview build  

4 files changed
± changelog.html
± genindex.html
± _autosummary/flexmeasures.data.models.planning.html
± api/v3_0.html

@Flix6x
Flix6x changed the base branch from main to docs/commitments-section January 27, 2026 19:43

@Flix6x Flix6x left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great draft! Let's start collecting todos in the PR description. We can decide what is in scope and what is out. One idea would be to split it up by feature:

Feature: multi-commodity

  • spec how we would encode the commodity per device in the flex-model
  • extend flex-model schema, limiting the commodity field to electricity and gas
  • exclude gas-fired devices from all current EMS commitments
  • add gas-price field to flex-context
  • set up gas commitment using gas-powered devices and gas-price
  • extend scheduling.rst
  • extend tutorial

Feature: beyond electricity and gas

  • add commodity field to commitment schema
  • document how to set up other commodities

Feature: multi-feed stocks

  • spec how we would encode the shared storage per device in the flex-model
  • extend flex-model schema
  • extend scheduling.rst
  • extend tutorial

Feature: multi-feed flows

  • spec how we would encode the shared capacity per device in the flex-model
  • extend flex-model schema
  • extend scheduling.rst
  • extend tutorial

Alternatively, we do one PR first to create support for all 3 features in the device_scheduler only, and touch the StorageScheduler just enough so we can safely experiment with the first two features in simulation. Then, once we're experienced we tackle the flex-model schema, and split up the documentation work for the 3 features.

NB for the schema speccing, we already have some results from a previous session regarding feature 3, in terms of defining a grid topology. I believe the design that came out of that should also fit the other two features, so we end up with a similar UX for what would at first glance appear to be three very different features.

Comment thread flexmeasures/data/models/planning/tests/test_commitments.py Outdated
Comment thread flexmeasures/data/models/planning/tests/test_commitments.py Outdated
Comment thread flexmeasures/data/models/planning/tests/test_commitments.py
Comment thread flexmeasures/data/models/planning/tests/test_commitments.py
Comment thread flexmeasures/data/models/planning/tests/test_commitments.py Outdated
Comment thread flexmeasures/data/models/planning/tests/test_commitments.py Outdated
@Ahmad-Wahid Ahmad-Wahid self-assigned this Jan 31, 2026
Ahmad-Wahid and others added 5 commits January 31, 2026 17:23
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
Signed-off-by: Ahmad-Wahid <ahmedwahid16101@gmail.com>
…n-gas-and-electricity

# Conflicts:
#	flexmeasures/data/models/planning/tests/test_commitments.py
@Flix6x Flix6x mentioned this pull request Feb 13, 2026
2 tasks
Base automatically changed from docs/commitments-section to main February 18, 2026 12:48
@Flix6x Flix6x linked an issue Apr 24, 2026 that may be closed by this pull request
@Flix6x Flix6x added this to the 0.33.0 milestone Apr 24, 2026
Flix6x added 13 commits April 24, 2026 17:26
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…mmitment does the optimizer treat their stocks as a combined resource

Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
… preference for full soc sooner

Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
The two changes together are the backwards-compatible default that was missing: when no device is specified (device=None), to_frame() must emit NaN in the device_group column (not crash), so the optimizer routes the commitment through ems_flow_commitment_equalities exactly as it did before device_group was introduced.

Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x
Flix6x marked this pull request as ready for review April 24, 2026 22:13
Flix6x added 2 commits April 25, 2026 10:56
…the pre-existing behaviour — each device is its own group

Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
expected_ev_costs = 2.2375
expected_battery_costs = -5.515
expected_total_cost = -3.2775
expected_ev_costs = 2.3125

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one thing I'm still unsure about. The total costs stayed the same, which is good, but the distribution between EV and battery shifted in the battery's favour. I don't have an explanation for it. I do know that there is currently no tie-breaking rule that would favour one over the other, which makes me willing to accept this.

@nhoening nhoening modified the milestones: 0.33.0, 0.34.0 May 23, 2026
…n-gas-and-electricity

# Conflicts:
#	documentation/changelog.rst
if isinstance(v, (list, tuple, set)):
groups = {device_group_map[d] for d in v}
if len(groups) != 1:
raise ValueError(f"Devices {v} map to multiple device groups: {groups}")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will likely become problematic once we need to model a device that belongs to both a shared power constraint and a shared storage, but let's deal with it once the problem actually presents itself.

@Flix6x
Flix6x merged commit 98d5522 into main Jun 3, 2026
12 checks passed
@Flix6x
Flix6x deleted the feat/switching-between-gas-and-electricity branch June 3, 2026 08:50
Flix6x added a commit that referenced this pull request Jul 10, 2026
Add a new `group` field to the storage flex-model, letting device entries
reference a power sensor that represents a group of devices (e.g. a hybrid
inverter shared by a battery and PV). The group sensor's own flex-model entry
constrains the group's aggregate power:

- power-capacity: hard constraint (both directions)
- consumption-capacity / production-capacity: soft constraints, enforced via
  breach commitments with default breach prices (10000 <currency>/kW)

The group's scheduled aggregate power is saved to the group sensor. Nested
groups are supported (cycles rejected); groups require multi-device
flex-models and members must share a commodity.

Implemented on top of the existing device-group solver machinery
(ems_constraint_groups from the multi-commodity work, and grouped
FlowCommitments from PR #1934); no changes to the optimizer itself.

Also fixes a latent device-index misalignment in MetaStorageScheduler._prepare,
where per-device lists were built from the full flex-model list (including
stock-only entries) instead of the filtered device models.

Closes #2092

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAad46Ayg86DHpL6nY54sX
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Flix6x added a commit that referenced this pull request Jul 18, 2026
…el group field) (#2276)

* feat: support intermediate power constraints on groups of devices

Add a new `group` field to the storage flex-model, letting device entries
reference a power sensor that represents a group of devices (e.g. a hybrid
inverter shared by a battery and PV). The group sensor's own flex-model entry
constrains the group's aggregate power:

- power-capacity: hard constraint (both directions)
- consumption-capacity / production-capacity: soft constraints, enforced via
  breach commitments with default breach prices (10000 <currency>/kW)

The group's scheduled aggregate power is saved to the group sensor. Nested
groups are supported (cycles rejected); groups require multi-device
flex-models and members must share a commodity.

Implemented on top of the existing device-group solver machinery
(ems_constraint_groups from the multi-commodity work, and grouped
FlowCommitments from PR #1934); no changes to the optimizer itself.

Also fixes a latent device-index misalignment in MetaStorageScheduler._prepare,
where per-device lists were built from the full flex-model list (including
stock-only entries) instead of the filtered device models.

Closes #2092

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAad46Ayg86DHpL6nY54sX

* docs: cite PR #2276 in changelog entry

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAad46Ayg86DHpL6nY54sX

* feat: support intermediate power constraints on groups of devices

Add a new `group` field to the storage flex-model, letting device entries
reference a power sensor that represents a group of devices (e.g. a hybrid
inverter shared by a battery and PV). The group sensor's own flex-model entry
constrains the group's aggregate power:

- power-capacity: hard constraint (both directions)
- consumption-capacity / production-capacity: soft constraints, enforced via
  breach commitments with default breach prices (10000 <currency>/kW)

The group's scheduled aggregate power is saved to the group sensor. Nested
groups are supported (cycles rejected); groups require multi-device
flex-models and members must share a commodity.

Implemented on top of the existing device-group solver machinery
(ems_constraint_groups from the multi-commodity work, and grouped
FlowCommitments from PR #1934); no changes to the optimizer itself.

Also fixes a latent device-index misalignment in MetaStorageScheduler._prepare,
where per-device lists were built from the full flex-model list (including
stock-only entries) instead of the filtered device models.

Closes #2092

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAad46Ayg86DHpL6nY54sX
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* docs: cite PR #2276 in changelog entry

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAad46Ayg86DHpL6nY54sX
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* fix: add group field to UI flex-model schema

The UI-DB flex-model schema parity test requires every DBStorageFlexModelSchema
field to have UI support.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAad46Ayg86DHpL6nY54sX
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* feat: define intermediate power constraints via DB flex-models on the asset tree (#2277)

Extend the storage flex-model's group field to also accept {"asset": <id>},
so a group can be identified by an asset-keyed flex-model entry - the form
that DB-stored flex-models naturally produce. This makes intermediate power
constraints fully definable via an asset tree with stored flex-models, with
an empty flex-model in the scheduling trigger.

- Asset-referenced group entries define no power sensor of their own; the
  group's scheduled aggregate power is saved via the entry's consumption
  and/or production output sensors, following the usual output-sensor
  conventions (full profile on a single sensor; clip-split when both given).
- Sensor-referenced groups keep saving the aggregate to the group power
  sensor, and may now also define output sensors.
- The UI asset flex-model editor recommends the parent asset when editing a
  child's group field (one-click suggestion), and hints in both the parent's
  and children's editors when the parent's flex-model defines power-capacity.
- New tutorial: toy example for intermediate power constraints, driven
  entirely by DB-stored flex-models.

Bug fixes uncovered along the way:
- AssetTriggerSchema.check_flex_model_sensors raised KeyError on asset-only
  flex-model entries.
- The freeze_server_now test fixture leaked its server_now monkeypatch into
  all subsequently run tests in the same process, causing order-dependent
  duplicate-key failures when scheduling jobs saved beliefs with identical
  (frozen) belief times.


Claude-Session: https://claude.ai/code/session_01TAad46Ayg86DHpL6nY54sX

Signed-off-by: F.N. Claessen <claessen@seita.nl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* docs: keep the group-constraints tutorial audience-appropriate

Remove references to code modules, tests and GitHub issues from the tutorial;
tutorials address FlexMeasures users, not developers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAad46Ayg86DHpL6nY54sX
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* fix: address review on the group field (XSS, error path, schema base)

- asset_properties.html: HTML-escape the parent asset name before it is
  inserted into innerHTML in the group hint (it is user-controlled -> XSS).
- storage.py: attach the "group must reference a power sensor" ValidationError
  to the `group` field, not `sensor`.
- GroupReferenceSchema now inherits from SharedSensorReferenceSchema instead of
  SensorReferenceSchema, so it accepts only `sensor`/`asset` (a group is a
  device-group identifier, not a belief-query reference) and no longer exposes
  the source-* filter fields; regenerate openapi-specs.json accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016MLCUiSdXDqDBmg8GbYp1B
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* docs: the model computes group schedules, it does not save them

Reword _add_group_schedules' docstring: the scheduling model assembles and
returns each group's aggregate power schedule; persistence is the data
service's responsibility.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016MLCUiSdXDqDBmg8GbYp1B
Signed-off-by: F.N. Claessen <claessen@seita.nl>

---------

Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support grouping devices

3 participants