Converting core (resp. edge) to plasma_profiles/transport/sources

[olivhoenen]

While discussing DD4 and the use of plasma_profiles with @MireilleSchneider @SimonPinches @prasad-sawantdesai and others, it became evident that a tool to convert the core_profiles and/or edge_profiles (resp. sources/transport) into the new plasma IDS would be of great importance.

It appears that imas-python is ideal for this kind of manipulation (cf. iterorganization/IMAS-Data-Dictionary#60). It may be added into the DD3-->DD4 specific conversion rules, or into a different functionality. @maarten-ic any thoughts?

[maarten-ic]

Hi Olivier, all,

A couple of questions and remarks to guide where we should put this functionality (I'm using the _profiles here, but all questions apply to each flavor of IDS 🙂):

1. Do you want to merge an existing core_profiles with an existing edge_profiles into a single plasma_profiles?
   1. If yes: how do you foresee to merge common elements (e.g. profiles_1d(itime)) with differently filled data? Are timebases the same? What to do with different coordinates (e.g. rho_tor_norm <= 1 for core and >=1 for edge), and what if coordinates overlap?
   2. If no: what output do you expect when converting a data entry with both core_profiles and edge_profiles? Create multiple occurrences of plasma_profiles?
2. Should the conversion be uni-directional (e.g. only core/edge -> plasma) or bi-directional (core/edge <-> plasma)?
3. Should this conversion be done always when going from DD3 to DD4?
4. I expect that this conversion should be possible outside of a DD version conversion as well, right?

This will be easy to implement when it's just copying data from one IDS to another, but may become difficult when having to merge two IDSs into one (or split one into two).

[MireilleSchneider]

To answer your questions:

1. In my own test cases, I needed to convert core_profiles <---> plasma_profiles, and core_sources <---> plasma_sources, I was not using the edge data at all. It is just that since we are at a crossroads, we have actors using core_profiles and core_sources, and others using plasma_profiles and plasma_sources, so it would be great to automatically convert from one to another
2. The conversion needs to be bi-directional (I needed the two directions for the METIS-TORBEAM coupling)
3. I think we agreed that in ddv4 we should systematically use plasma_profiles and plasma_sources, and NOT core_profiles and core_sources anymore. The plan is to even convert the scenario database to ddv4. However in the intermediate situation where we have still many actors using ddv3, we need to manipulate core_profiles and core_sources, still.
4. I don't understand the question, I think what is globally confusing is that ddv4 has all IDSs declared, i.e. core_profiles, plasma_profiles, core_sources, plasma_sources, and it seemed to me that exporting both core_sources and plasma_sources to the same output dataset, the AL was unhappy, but it needs further testing.

I had the complication that I also had to do some merges indeed, but I'd say that these merges can always be done manually. I had to merge some content of plasma_sources (from METIS) with some content of core_sources (from hcd2core-sources), to fill a global plasma_sources which I gave back to METIS.

[olivhoenen]

1. We want to try merging core and edge into the same plasma (if we have existing cases where they are consistent, cf. jintrac-coconut or jintrac-dina maybe). For cases where they are obviously not consistently defined, we should store them into two occurrences (and label the ids_properties/occurrence/type with the proper identifier)
2. The conversion should not be bi-directional, only core/edge --> plasma. As plasma is overlapping core/edge, they are not expected to stay both on the long term.
3. In principle, it should be optional (as core/edge are still there and active at the moment in DD4), but if it is done for each DD3->4 conversion it may just speed up the handover
4. Ideally yes.

[maarten-ic]

Decided in the progress meeting today to work this out in more detail.

[maarten-ic]

Hi @olivhoenen, @MireilleSchneider,

I've had a look at this ticket in more detail today. My proposal is to implement the following:

1. Create new functions imas.ids_convert.convert_to_plasma_profiles/_transport/_sources, with an alias in the imas namespace (so users can do e.g. imas.convert_to_plasma_profiles(edge_profiles)).

2. These functions accept a corresponding core_ or edge_ IDS. They copy all data into the plasma_ IDS (with the same DD version).

   A warning will be logged for data that cannot be copied. For example in DD 4.1.0, profiles_1d/ion/state/ionization_potential exists in the core_profiles IDS but not in the plasma_profiles IDS.

3. The command line conversion utility (imas convert <input> <target_dd_version> <output>) will get a new flag --convert-to-plasma-ids. When this flag is added to the command line, any core_* and edge_* is converted to the corresponding plasma_ IDS. If both core and edge IDSs exist in the input Data Entry, each will be stored as a different occurrence.

   Example: the input data entry contains three core_profiles (occurrences 0, 1 and 2) and two edge_profiles (occurrences 0 and 1). The output data entry will get 5 plasma_profiles IDSs: 0, 1 and 2 are the converted core_profiles IDSs. 3 and 4 are the converted edge_profiles IDSs.

For IO to decide:
A. The proposal only covers conversion in one direction (core/edge -> plasma), not the bidirectional conversion requested by Mireille. Is this okay for now?
B. This proposal does not cover merging edge&core IDSs to keep the scope small and feasible. Is this okay for now?
C. Do you have any other comments or remarks/suggestions on the proposed solution?

[olivhoenen]

Sounds good to me as a starting point. @imbeauf would this be a good case for introducing an occurrence_type identifier (core, edge, core-edge)?

[imbeauf]

On point 2 from Marteen: profiles_1d/ion/state/ionization_potential is there in the plasma_profiles ... but with an "s" instead of a "z".
It has escaped the s->z transformations due to parallel branch developments ...
I will correct this

On the occurence_type from Olivier: agree I will create this