Support In-work loads

[taldcroft]

Description

This adds two key new features:

1. A new Chandra Command event type Load in backstop that allows including in-work backstop commands.

• These are taken from https://occweb.cfa.harvard.edu/occweb/FOT/mission_planning/Backstop if the Load in backtop parameter is a load name like SEP0825A.
• A full path on OCCweb (e.g. FOT/mission_planning/Backstop/Archive/2024/02_feb/FEB0524A) can also be supplied. This is primarily useful for testing but might find other applications.

2. New special scenario names flight+custom and custom that allow easily using a custom Google sheet.

• The custom sheet ID is specified in the new configuration option cmd_events_custom_id.
• The scenario flight+custom includes both the flight command events and the custom command events.
• The scenario custom includes only the custom command events.

These changes are being introduced to support FOT MP MATLAB tools initial configuration, and the final version of this PR will include a custom sheet ID that is owned and maintained by FOT MP.

However, other teams could certainly create their own custom Google sheet and override the cmd_events_custom_id configuration option by default.

To do

• Update the user documentation
• On deployment, update the Command Events documentation

Requires

None

Interface impacts

None

Testing

Unit tests

• Mac

(ska3) ➜  kadi git:(allow-in-work-loads) git rev-parse --short HEAD                           
1e5f0ea
(ska3) ➜  kadi git:(allow-in-work-loads) pytest --pdb
=============================================== test session starts ===============================================
platform darwin -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
rootdir: /Users/aldcroft/git
configfile: pytest.ini
plugins: anyio-4.7.0, timeout-2.3.1
collected 216 items                                                                                               

kadi/commands/tests/test_commands.py ...................................................................... [ 32%]
................                                                                                            [ 39%]
kadi/commands/tests/test_filter_events.py ..                                                                [ 40%]
kadi/commands/tests/test_states.py ...............................................x........................ [ 74%]
..                                                                                                          [ 75%]
kadi/commands/tests/test_validate.py ......................                                                 [ 85%]
kadi/tests/test_events.py ..........                                                                        [ 89%]
kadi/tests/test_occweb.py ......................                                                            [100%]

=================================== 215 passed, 1 xfailed in 120.87s (0:02:00) ====================================

Independent check of unit tests by Jean

• linux

(ska3-latest) jeanconn-kady> pytest
================================================= test session starts ==================================================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
rootdir: /proj/sot/ska/jeanproj/git
configfile: pytest.ini
plugins: anyio-4.7.0, timeout-2.3.1
collected 216 items                                                                                                    

kadi/commands/tests/test_commands.py ........................................................................... [ 34%]
...........                                                                                                      [ 39%]
kadi/commands/tests/test_filter_events.py ..                                                                     [ 40%]
kadi/commands/tests/test_states.py ...............................................x..........................    [ 75%]
kadi/commands/tests/test_validate.py ......................                                                      [ 85%]
kadi/tests/test_events.py ..........                                                                             [ 89%]
kadi/tests/test_occweb.py ......................                                                                 [100%]

=================================================== warnings summary ===================================================
kadi/kadi/commands/tests/test_commands.py: 113 warnings
kadi/kadi/commands/tests/test_filter_events.py: 27 warnings
kadi/kadi/commands/tests/test_states.py: 25 warnings
kadi/kadi/commands/tests/test_validate.py: 9 warnings
kadi/kadi/tests/test_occweb.py: 19 warnings
  /export/jeanconn/miniforge3/envs/ska3-latest/lib/python3.12/site-packages/bs4/builder/_lxml.py:124: DeprecationWarning: The 'strip_cdata' option of HTMLParser() has never done anything and will eventually be removed.
    parser = parser(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================== 215 passed, 1 xfailed, 193 warnings in 294.89s (0:04:54) ===============================
(ska3-latest) jeanconn-kady> git rev-parse HEAD
b4a1e50e7a5cf35949d4804a7030202e07157ec9

Functional tests

I replicated the unit test in a notebook but without any function mocking so that it actually reads the custom sheet https://docs.google.com/spreadsheets/d/1dGEf5FvD43JrsVy41LvCN3PNKKsbHIgYfx5Z7D5oXZk/edit?gid=0#gid=0.

This gave the expected results.

[taldcroft]

Questions:

• Should the Params for Load in work be:
  • Load name only like NOV1425A
  • Path like FOT/mission_planning/Backstop/NOV1425A
• Should the command source for in-work loads be distinguishable from approved loads?
  • A simple/easy idea is making it lower-case. Not sure if this might have unexpected consequences.

[jzuhone]

@taldcroft I looked it over and looks good. will there also be docs to show an end-user how to implement it for themselves?

[jeanconn]

@jzuhone - we should talk a little bit more about use cases with regard to docs. For non-fot-mp users, I was thinking that you'd likely continue to either override the current cmd_events_flight_id or use a local scenario. I'm not sure if ACIS would end up with situations that would demand the "chained" flight+custom with a customized cmd_events_custom_id? Did you have any situation/scenario in mind that has been tricky to model thus far?

[jeanconn]

Needs docs on the intent of the new archive param here.

[jeanconn]

I think this still needs an update.

[jeanconn]

When I try to make a copy of the Command Events Sheet or the test sheets, there's an Apps Script attached. Is that thing doing anything useful?

[jeanconn]

Now that we're supporting multiple ids, It would be useful to have the error" raise ValueError(f"Failed to get cmd events sheet: {req.status_code}")" mention which sheet failed and/or give the id.

[jeanconn]

Looks good to me except still needs updated user docs.

Requested change has been implemented.

[taldcroft]

OK. I've update the user documentation.

[taldcroft]

@jeanconn - with regards to your comment to @jzuhone about "continue to either override the current cmd_events_flight_id or use a local scenario".

I think that the new machinery is much more convenient and would be appropriate for ACIS to quickly evaluate the impact of adding an ECS measurement. The key point is that using the flight+custom scenario automatically includes whatever has happened in flight (where this is always kept up to date). The the ACIS custom sheet only needs to include the ECS measurement as a delta update. Either by way of a custom per-user ~/.kadi/config/kadi.cfg file or programmatically in acis_thermal_check, the ACIS custom sheet can be hardwired within their process it just works. 🤞

[taldcroft]

When I try to make a copy of the Command Events Sheet or the test sheets, there's an Apps Script attached. Is that thing doing anything useful?

I have no idea what that is (or even that such a thing existed). @johnny1up - were you playing with an Apps Script? In any case, Gemini tells me that it is benign.

[taldcroft]

Now that we're supporting multiple ids, It would be useful to have the error" raise ValueError(f"Failed to get cmd events sheet: {req.status_code}")" mention which sheet failed and/or give the id.

Fair point.

[jeanconn]

"I think that the new machinery is much more convenient and would be appropriate for ACIS to quickly evaluate the impact of adding an ECS measurement." That's fair. At the time I was just thinking it was still pretty easy to copy the flight sheet and set the kadi flight sheet id to that and I was worried about maintenance and possible confusion on a new "persistent" ACIS sheet. But if that's the preferred strategy that's fine and I see the new docs support it well.

[jeanconn]

"I have no idea what that is (or even that such a thing existed). @johnny1up - were you playing with an Apps Script? In any case, Gemini tells me that it is benign." When I looked at the apps script that was attached it seemed to be perhaps doing nothing "function myFunction() {}" so if we can delete it from the flight sheet we'll stop getting it in all the copies...

[taldcroft]

@johnny1up @julia-zachary @jeanconn @jayhead13 - I updated the Command events documentation to include the Load in backstop event type.

While I was there, I did some tidying and organizing (with help from copilot):

• Event type sections are alphabetized.
• Added a list with links and a summary for each event type.

I also put this whole document into version control in the command-events-docs branch as command-events-docs.twiki.

[jeanconn]

Do we need to asterisk the "Load in backstop" event type for now? I don't know when this code to use it is actually going to go live.

[johnny1up]

@johnny1up @julia-zachary @jeanconn @jayhead13 - I updated the Command events documentation to include the Load in backstop event type.

While I was there, I did some tidying and organizing (with help from copilot):

• Event type sections are alphabetized.

• Added a list with links and a summary for each event type.

I also put this whole document into version control in the command-events-docs branch as command-events-docs.twiki.

I noticed the twiki file is being GitHub versioned. Im assuming versioning isn't a requirement for page updates but rather booking at some interval or trigger?

Otherwise, looks great. Thanks for the update and notification!

[taldcroft]

Do we need to asterisk the "Load in backstop" event type for now? I don't know when this code to use it is actually going to go live.

Is the red Available with MATLAB tools 2026010 and Ska3 2026.1. not sufficient?

About the version, I'm hoping that we can get it into ska3 2026.1 and the MATLAB release that includes 2026.1.

[taldcroft]

Im assuming versioning isn't a requirement for page updates but rather booking at some interval or trigger?

You can update the TWiki page as you like. My plan is to copy/paste the raw TWiki content back into the git-tracked branch as needed. This time around I wanted to use CoPilot for help and having things in version control gave me confidence to be able to see the diffs.

[jeanconn]

"Is the red Available with MATLAB tools 2026010 and Ska3 2026.1. not sufficient?" I mean, I'm the kind of user who would probably use the docs page just for the little bit of extra text you get in the list of available commands? But since users shouldn't be able to use the new command until you add it to the options in the command event sheet, hopefully there won't be a mismatch of expectations.