agasc-supplement-tasks disposition failed

[taldcroft]

After running agasc-supplement-tasks disposition for this week (https://cxc.cfa.harvard.edu/mta/ASPECT/agasc/supplement_reports/weekly/2023:093/)

WARNING - 'agasc-update-magnitudes --log-level debug --output-dir /proj/sot/ska3/flight/data/agasc/rc --report --obs-status-file /proj/sot/ska3/flight/data/agasc/rc/obs_status.yml --args-file /proj/sot/ska3/flight/data/agasc/rc/supplement_reports/weekly/latest/call_args.yml' returned non-zero status: 256
/proj/sot/ska3/flight/lib/python3.10/site-packages/kadi/commands/commands_v1.py:11: FutureWarning: kadi commands v1 is deprecated, use v2 instead
  warnings.warn("kadi commands v1 is deprecated, use v2 instead", FutureWarning)
Traceback (most recent call last):
  File "/proj/sot/ska3/flight/bin/agasc-update-magnitudes", line 33, in <module>
    sys.exit(load_entry_point('agasc==4.14.1', 'console_scripts', 'agasc-update-magnitudes')())
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/agasc/scripts/update_mag_supplement.py", line 129, in main
    agasc_ids += update_supplement.update(args)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/agasc/scripts/update_supplement.py", line 228, in update
    status = parse_args(
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/agasc/scripts/update_supplement.py", line 163, in parse_args
    status = _sanitize_args(status)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/agasc/scripts/update_supplement.py", line 58, in _sanitize_args
    raise Exception(f"Can not guess last_obs_time for agasc_id={star['agasc_id']}")
Exception: Can not guess last_obs_time for agasc_id=648684104

The obs_status.yml file was:

ska3-aca-kady% cat obs_status.yml
# See https://sot.github.io/agasc/supplement.html for detailed guidance.
#
# After reviewing and updating this file, run as `aca`:
#   agasc-supplement-tasks disposition
# Check the diffs at:
#   https://cxc.cfa.harvard.edu/mta/ASPECT/agasc/supplement/agasc_supplement_diff.html
# After reviewing the post-disposition email with diffs, run as `aca`:
#   agasc-supplement-tasks schedule-promotion
#
# Replace BAD_STAR_CODE with one of the following bad star codes:
#  9: Bad star added based on automated magnitude processing. Magnitude is a lower bound, set to MAXMAG.
# 10: Bad star added manually due to bad position (bad catalog position or high or missing proper motion)
# 11: Bad star added based on automated magnitude processing. Magnitude is an upper bound.
# 12: Bad star added based on automated magnitude processing. Magnitude set manually.
# 13: Bad star added based on automated magnitude processing. Star shows variability not captured in VAR.
# See also: https://github.com/sot/agasc/wiki/Add-bad-star-to-AGASC-supplement-manually
bad:
  648684104: 9
#
# Mags below assume that the star is not detected and uses the star max mag.
# Edit accordingly if this is not the case.
mags:
  - agasc_id: 648684104
    mag_aca: 11.203
    mag_aca_err: 0.1
#
# status: 0 = star-obs is OK and mag data should be used => REMOVE the mags entry
#         1 = star-obs is bad/unreliable => use mags values above
# comments: see the Rubric in https://sot.github.io/agasc/supplement.html
#   Most commonly:
#     "Never acquired" (status=1)
#     "Almost never acquired" (status=0 or 1, depending on data quality)
#     "Faint star" (status=0)
obs:
  - mp_starcat_time: 2023:083:04:49:53.926
    obsid: 44715
    status: 1
    agasc_id: [648684104]
    comments: Never acquired

[javierggt]

This error was caused by a wrong "latest" symlink in the output directory, caused by an unknown script failure. The exact sequence of events is not evident as first, so I will explain it below.

This error happens after the dispositions are applied, when updating the magnitudes including the dispositions. The crucial aspect is that this magnitude update should run with the same arguments as the weekly update (which happens on Sundays).

This specific error happens in update_supplement.py#L58:

        if 'last_obs_time' in star:
            star['last_obs_time'] = CxoTime(star['last_obs_time']).cxcsec
        else:
            obs = cat.STARS_OBS[cat.STARS_OBS['agasc_id'] == star['agasc_id']]
            if len(obs) == 0:
                raise Exception(f"Can not guess last_obs_time for agasc_id={star['agasc_id']}")
            star['last_obs_time'] = CxoTime(obs['mp_starcat_time']).max().cxcsec

so the error actually means that the star was not found in cat.STARS_OBS.

The part that is not evident is that, in this case, agasc-update-magnitudes was called with the --args-file argument. What this does is to modify the default values in the argument parser (in this case that is --stop). That argument is then passed to initialize cat.STARS_OBS in update_mag_supplement.py#L120:

    star_obs_catalogs.load(args.stop)

The final piece is that the args-file used is /proj/sot/ska3/flight/data/agasc/rc/supplement_reports/weekly/latest/call_args.yml, and the key is that latest is a symlink to the latest output directory.

In this case, the processing script failed, for a yet unknown reason. This means that the symlink was not updated, and it was pointing to an earlier week, prior to the observation of the problematic star (agasc_id=648684104). This caused cat.STARS_OBS to be initialized without all observations after this time, and that excluded this star.

A few things could be fixed:

• maybe give a better error. Something like "Star agasc_id=648684104 is not in cat.STARS_OBS between start and stop"
• somehow change it so it does not depend on a symlink being updated in a previous step. This might require some change in processing (the user provides the date of the latest weekly processing, or the date is inferred based on the current date). I am not very inclined to fix this one.