Configuration and API Simplification

[gupichon]

Overview

The goal of this discussion is to consolidate the ongoing improvements around configuration and API design in order to maintain overall consistency. It also aims to validate key API choices early, in particular to avoid redundancy or overlap between existing concepts such as YellowPages and ElementHolder, and to ensure a coherent direction across all related developments.

This discussion focuses on improving both the configuration model and the API usability in PyAML. The goal is to simplify workflows, increase flexibility, and enable more dynamic behavior while keeping the system consistent and maintainable.

---

Configuration Model

Backend Linking

The configuration should rely on a key/value manager to link objects to the backend:

• A static YAML catalog provides the base configuration.
• Dynamic catalogs, specific to each control system (CS), allow contextual adaptation.

This approach is aligned with the catalog work introduced in PR #192 .

---

Workflow

The expected workflow is:

1. Load a static configuration

2. Optionally modify it dynamically:

   • Fix errors
   • Adapt to a specific execution context (e.g., different tools or environments)

This implies that configurations should not be considered immutable after loading.

An open question is whether dedicated tools should be introduced to help build configurations. If so, several aspects need to be clarified:

• What form should these tools take (Python API, CLI, external scripts, UI, etc.)
• What should be their source of truth (existing control system, databases, simulation models, catalogs, lattice files, etc.)
• How they should integrate with the current configuration workflow

---

Configuration Composition and Modification

Current State

Configuration composition is already supported through multiple YAML files. However:

• It is not fully user-friendly
• It requires building a main aggregated YAML file manually

---

Proposed Improvements

1. Direct Multi-file Loading

Current state:

Configuration composition is already possible by splitting the configuration into multiple YAML files and referencing them from a main file. However, this approach has several limitations:

• It requires creating and maintaining an explicit main configuration file
• The composition logic is externalized and not handled by the builder itself
• It prevents directly passing multiple independent configuration fragments to the loader
• It makes dynamic or context-dependent composition more difficult

In particular, it is currently not possible to write something like:

Accelerator.load([
    "base.yaml",
    "magnets.yaml",
    "diagnostics.yaml",
])

or:

Accelerator.load(
    "base.yaml",
    additional_configs=["magnets.yaml", "diagnostics.yaml"]
)

It is also not possible to express composition directly within YAML itself. For example, the following is not supported:

devices:
  - bpms.yaml
  - diag_tools.yaml
  - tuning_tools.yaml
  - magnets.yaml

Proposed improvement:

Allow the PyAML builder to accept multiple configuration fragments directly, without requiring a pre-built main file.

This would:

• Simplify user workflows
• Avoid redundant configuration generation steps
• Enable more flexible and dynamic configuration assembly
• Better support programmatic use cases (e.g., tests, simulations, environment-specific overrides)

---

2. Dynamic Post-load Modification

After loading a configuration, it should be possible to:

• Update object references
• Rename objects and tools
• Fix inconsistencies dynamically

---

3. Validation and Mutability Support

To support safe dynamic updates:

• Introduce a validation decorator
• Introduce a "changeable" decorator for mutable elements

This approach has already been explored in PR #214.

Benefits:

• Cleaner and more maintainable code
• Explicit control over what can be modified at runtime
• Safer dynamic reconfiguration

---

API: Access to Tools and Objects

Improving the API for accessing high-level objects is already being discussed in:

• Issue #199

The objective is to provide:

• A more intuitive access pattern
• Reduced boilerplate
• Better discoverability of available objects

---

High-Level Object Management

Introducing a manager for high-level objects could significantly improve consistency across:

• Multiple control systems
• Simulators
• Digital twins

Expected Benefits

• Centralized update propagation
• Consistent object identity across contexts
• Simplified synchronization between environments

---

Relation with Yellow Pages

The Yellow Pages concept could be extended to support this:

• Act as a central registry of high-level objects
• Provide a unified view across control modes
• Facilitate dynamic lookup and updates

[JeanLucPons]

devices:
  - bpms.yaml
  - diag_tools.yaml
  - tuning_tools.yaml
  - magnets.yaml

The above has to be supported and it should be easy to fixed during list expansion:

    # Recursively expand a list
    def expand_list(self, l: list):
        idx = 0
        while idx < len(l):
            value = l[idx]
            if hasToLoad(value):
                obj = load(value, self.files_stack)
                if isinstance(obj,list):
                    l[idx:idx+1] = obj
                    idx += len(obj)
                else:
                    l[idx] = obj
                    idx += 1
            else:
                self.expand(value)
                idx += 1

[JeanLucPons]

I did a PR #226 fix it