Automations - first roundtrip for forecasts

[nhoening]

As hosts, we want to automate recurring tasks of the three main features (schedules, forecasts, reports) in a way that allows to manage them across larger numbers of sites.

Current state

recurring tasks can be run by cron jobs (e.g. the job runs flexmeasures add schedule ...) if the gateway/client does not automate through calling the API. More likely on the FlexMeasures side would be flexmeasures add forecasts, flexmeasures add report or even custom commands.

Overall vision

We will define automations as a generic concept, create API & UI so hosts/users can define & edit them, and use rq-scheduler to run them (create & queue the jobs on a recurring basis).

Data sources will play an important role, as we will store the needed config (e.g. some flex-config for schedules) there.

Goal of this PR is Milestone 1: automations for forecasts

Why forecasts? Forecasts are well-defined, so a good starting point. They are also less likely to be triggered from the outside (often, they are simply needed so schedules can be computed, but users don't really want to be involved in that layer). Also, reports still cannot be run as jobs, as of today.

The result of this PR is a CLI to create automations and a simple UI to view them.

TODO

Automations data model

The automations data model has these fields:

• id: int
• created_at: datetime
• asset_id: int
• type: str ("forecasts", "schedules", "reports" - for now only the former)
• name: str
• cronstr: str (define the recurrence)
• active: bool
• generator_id: int (links to a data source)
• parameters: JSONB

The parameters are free, in this case we will validate them by the forecast parameter schema.

Add the model class for this. Access control list (__acl__) should allow account_admins and consultants to create, update and delete.

Very basic roundtrip

• CLI commands add automation and delete automation. Make sure to record these events in the asset's audit log. add automation uses the forecasting schema to validate parameters
• a GET endpoint GET /api/v3_0/assets/ID/automations/ (make sure it's documented in OpenAPI). Each entry in the response has all fields aside from generator_id and parameters.
• a UI page for automations (e.g. https://ems.seita.energy/assets/1/automations, added to the breadcrumbs dropdown) which for now just lists the automations in a simple table. Already wrap that list in a tabbed view (schedules and reports are deactivated tabs, shwowing our intent for later). Columns: Name, Created At (human readable, isostr in title tag), Active, Recurrence (human readable, cronstr in title tag).

Actually running automations

• Logic to activate or deactivate an automation. The add automation CLI activates per default.
• CLI command edit automation which for now can only edit name, cronstr and activation status. Also record to audit log of the asset.
• Use RQ to actually pick up automations that should run. This might have to be a CLI command which runs once per minute or so. The result is queued (forecasting) jobs. Document this.
• UI:
  • The automations page links to the status page where recent jobs are listed.
  • The status page's job table should also get a column showing which automation (if any) was used. Actually, when we create a job (in this case, forecasting jobs) we should record on the job how it was created (API, CLI, automation) and use that information in this table.
  • Each automation on the automation page could show some more information: JSONB parameters and the data source info (not sure which) appear in modal when clicking "Details" button. And we could actually show the number of jobs this automation has queued/failed/succeeded. For this, the endpoint GET /api/v3_0/assets/ID/automations/ID could be made, which provides more information. Load it asynchronously with Ajax, and add the information to the table row when it comes in.

Related / future stories

Schedules as automations

Add schedules to the above. They are well-described with schema's which we should use. Maybe some practical parameterisation issues will come up.

Reports as automations

• API endpoint to trigger reports as jobs
• Prepared & documented report templates (e.g. self-consumption, energy costs, etc etc)
• Add reports to the automation flow

CRUD for automations in the UI

Make the automations page more powerful: Create, (de)activate and delete automations there.
For this, API endpoints need to be made (POST, PATCH, DELETE)