From acbf12fff09c82df21d58348cf1cc8f5a27def25 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 19 Nov 2025 16:08:15 -0500 Subject: [PATCH 001/221] Added the getting started with XRISM Xtend notebook, currently just the template, as well as the necessary index entries. For issue #128 --- index.md | 1 + .../xrism/getting-started-xrism-xtend.md | 268 ++++++++++++++++++ .../xrism/xrism_index.md | 13 + 3 files changed, 282 insertions(+) create mode 100644 tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md create mode 100644 tutorials/mission_specific_analyses/xrism/xrism_index.md diff --git a/index.md b/index.md index a965db66..89659af1 100644 --- a/index.md +++ b/index.md @@ -35,6 +35,7 @@ caption: Mission specific tutorials --- tutorials/mission_specific_analyses/nustar/nustar_index tutorials/mission_specific_analyses/rxte/rxte_index +tutorials/mission_specific_analyses/xrism/xrism_index ``` ## Demonstrations of useful high-energy tools diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md new file mode 100644 index 00000000..37a5aec7 --- /dev/null +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -0,0 +1,268 @@ +--- +authors: +- name: David Turner + affiliations: ['University of Maryland, Baltimore County', 'HEASARC, NASA Goddard'] + email: djturner@umbc.edu + orcid: 0000-0001-9658-1396 + website: https://davidt3.github.io/ +- name: Kenji Hamaguchi + affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] + website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 + orcid: 0000-0001-7515-2779 +date: '2025-11-19' +file_format: mystnb +jupytext: + text_representation: + extension: .md + format_name: myst + format_version: 1.3 + jupytext_version: 1.17.3 +kernelspec: + display_name: heasoft + language: python + name: heasoft +title: Getting started with XRISM-Xtend +--- + +# Getting started with XRISM-Xtend + +## Learning Goals + +By the end of this tutorial, you will be able to: + +- Find... +- + +## Introduction + +Alter this file according to your use case but retain the basic structure and try to use the same syntax for things like section headings, numbering schemes, and bullet points. +Specifically the headings in this Intro section should not be edited to maintain consistency between notebooks. + +All contributed notebooks should be in [MyST markdown](https://mystmd.org) format. +See the [Fornax documentation](https://docs.fornax.sciencecloud.nasa.gov/markdown-and-code-dev) for more info about this. + +The Introduction should provide context and motivation. +Why should someone use this notebook? +Give background on the science or technical problem. +Point out the parts that are particularly challenging and what solutions we chose for what reasons. + +### Inputs + +- List the data, catalogs, or files needed, and where they come from. + If there are data that get downloaded to Fornax as part of this notebook, place those in a `data` directory. + Please do not change the name of this directory for consistency with other notebooks. + Do not add the contents of `data` to the repo, just the empty directory. + +### Outputs + +- List the products the notebook generates (plots, tables, derived data, etc.) +- If there are intermediate products produced by your notebook, generate an `output` directory for those data. + Please do not change the name of this directory for consistency with other notebooks. + Do not add the contents of `output` to the repo, just the empty directory. + +### Runtime + +Please report actual numbers and machine details for your notebook if it is expected to run longer or requires specific machines, for example, on Fornax. +Also, if querying archives, please include a statement like: +"This runtime is heavily dependent on archive servers which means runtime will vary for users". + +Here is a template runtime statement: +As of {Date}, this notebook takes ~{N}s to run to completion on Fornax using the ‘Default Astrophysics' image and the ‘{name: size}’ server with NGB RAM/ NCPU. + +## Imports + +This should be a list of the modules that are required to run this code. +Importantly, even those that are already installed in Fornax should be listed here so users wanting to run this locally on their own machines have the information they need to do this. + +Make sure that you have built a "requirements_notebook_name.txt" file with the modules to be imported. +The name of the notebook should be present in the name of the requirements file, as in our example "requirements_notebook_template.txt" + +```{code-cell} python +# This cell should not be edited below this line except for the name of +# the requirements_notebook_name.txt + +# Uncomment the next line to install dependencies if needed. +# %pip install -r requirements_notebook_name.txt +``` + +```{code-cell} python +import numpy as np +``` + +## Global Setup + +### Functions + +Please avoid writing functions where possible, but if they are necessary, then place them in the following +code cell - it will be minimized unless the user decides to expand it. **Please replace this text with concise +explanations of your functions or remove it if there are no functions.** + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + +# This cell will be automatically collapsed when the notebook is rendered, which helps +# to hide large and distracting functions while keeping the notebook self-contained +# and leaving them easily accessible to the user +``` + +### Constants + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + +``` + +### Configuration + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + +``` + +*** + +## 1. Data Access + +The name of this, and all future sections can change. +In general, it probably is a good idea to start with something like "Data Access". +Please note, and stick to, the existing numbering scheme. + +```{code-cell} python +# Create some example data. +data = np.random.randint(0, 100, size=100) +``` + +## 2. Data Exploration + +Describe what the data look like. +Add summary statistics, initial plots, sanity checks. + +For cuts or other data filtering and cleaning steps, explain the scientific reasoning behind them. +This helps people understand both the notebook and the data so that they're more equipped to use the data appropriately in other contexts. + ++++ + +:::{tip} +Please include a narrative for *all* your code cells to help the reader figure out what you are doing and why you chose that path. + +Using [MyST admonitions](https://mystmd.org/guide/admonitions) such as this `tip` are encouraged +::: + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + +hist, bin_edges = np.histogram(data, bins=10) +hist +``` + +:::{important} +The HEASARC-tutorials style guide requires that cells using matplotlib (or similar) to produce figures +should be isolated (i.e., only contain plotting code), and must include the following metadata to hide the +code from view (see the source of this cell for the unrendered text: + +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +::: + +For any Figures, please add a few sentences about what the users should be noticing. + ++++ + +## 3. Analysis + +The working part of the notebook. +Lay out the step-by-step analysis workflow. +Each subsection should describe what is being done and why. +These can be sections or subsections. + ++++ + +### 3.1 Design Principles + +- Make no assumptions: define terms, common acronyms, link to things you reference. +- Keep in mind who your audience is. +- Design for portability - will this notebook work on both Fornax and someone's individual laptop. +- Cells capture logical units of work. +- Use markdown before or after cells to describe what is happening in the notebook. + ++++ + +### 3.2 Style Principles + +- Follow suggestions of The Turing Way community [markdown style](https://book.the-turing-way.org/community-handbook/style) +- Write each sentence in a new line (line breaks) to make changes easier to read in PRs. +- Avoid latin abbreviation to avoid failing CI. + +#### 3.2.1 Best Practice Guidelines +It would be nice if all contributed codes did the following, but these guidelines will not be checked in a code review + +- Section titles should not end with ":". +- List items should start at the beginning of the line, no spaces first. Exception is nested lists. +- One empty line between section header and text. +- One empty line before a list and after. +- No more than one empty line between any two non-empty lines. + +```{code-cell} python + +``` + +## 4. PR Review + +Notebooks go through a two step process: first step is getting into the repo, and the second step gets it into the [published tutorials](https://nasa-fornax.github.io/fornax-demo-notebooks/). +Final notebooks are expected to go through both a science and tech review checklist. +Checklists are [here](https://github.com/nasa-fornax/fornax-demo-notebooks/tree/main/template/notebook_review_checklists.md). +Please consider these checklist requirements as you are writing your code. + +The first PR of a notebook does not need to have everything from the checklists completed, but should have all the pieces there, and the authors should be aware of the requirements. + +To complete the second step of this process and be both rendered and included in users Fornax home directories, both a science and technical reviewer will be looking at [this checklist](https://github.com/nasa-fornax/fornax-demo-notebooks/tree/main/template/notebook_review_checklists.md) to see if the new tutorial notebook meets all of the requirements, or has a reasonable excuse not to. + +Any PRs can be opened as drafts, which is in fact preferred, if authors are still working on them. + ++++ + +## About this notebook + +- **Authors:** Specific author and/or team names, plus "and the Fornax team". +- **Contact:** For help with this notebook, please open a topic in the [Fornax Community Forum](https://discourse.fornax.sciencecloud.nasa.gov/) "Support" category. +- Please edit and keep the above 2 bullet points, and remove this last line. + ++++ + +### Additional Resources + + +### Acknowledgements + +Did anyone help you? +Probably these teams did, so include them: MAST, HEASARC, & IRSA Fornax teams. + +Did you use AI for any part of this tutorial, if so please include a statement such as: +"AI: This notebook was created with assistance from OpenAI’s ChatGPT 5 model.", which is a good time to mention that this template notebook was created with assistance from OpenAI’s ChatGPT 5 model. + +### References + +This work made use of: + +- STScI style guide: https://github.com/spacetelescope/style-guides/blob/master/guides/jupyter-notebooks.md +- Fornax tech and science review guidelines: https://github.com/nasa-fornax/fornax-demo-notebooks/blob/main/template/notebook_review_checklists.md +- The Turing Way Style Guide: https://book.the-turing-way.org/community-handbook/style diff --git a/tutorials/mission_specific_analyses/xrism/xrism_index.md b/tutorials/mission_specific_analyses/xrism/xrism_index.md new file mode 100644 index 00000000..65b32608 --- /dev/null +++ b/tutorials/mission_specific_analyses/xrism/xrism_index.md @@ -0,0 +1,13 @@ +# XRISM + +Here we include notebooks that demonstrate how to use data taken by the XRISM mission. + + +```{toctree} +--- +maxdepth: 2 +caption: XRISM tutorials +--- +getting-started-xrism-xtend + +``` From 09d7488892541308acf05830edad76a331496467 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 19 Nov 2025 18:11:02 -0500 Subject: [PATCH 002/221] Started adding things to the global configuration section of the getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 229 ++++++------------ 1 file changed, 68 insertions(+), 161 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 37a5aec7..04a3d274 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -15,7 +15,7 @@ jupytext: text_representation: extension: .md format_name: myst - format_version: 1.3 + format_version: 0.13 jupytext_version: 1.17.3 kernelspec: display_name: heasoft @@ -35,68 +35,46 @@ By the end of this tutorial, you will be able to: ## Introduction -Alter this file according to your use case but retain the basic structure and try to use the same syntax for things like section headings, numbering schemes, and bullet points. -Specifically the headings in this Intro section should not be edited to maintain consistency between notebooks. - -All contributed notebooks should be in [MyST markdown](https://mystmd.org) format. -See the [Fornax documentation](https://docs.fornax.sciencecloud.nasa.gov/markdown-and-code-dev) for more info about this. - -The Introduction should provide context and motivation. -Why should someone use this notebook? -Give background on the science or technical problem. -Point out the parts that are particularly challenging and what solutions we chose for what reasons. ### Inputs -- List the data, catalogs, or files needed, and where they come from. - If there are data that get downloaded to Fornax as part of this notebook, place those in a `data` directory. - Please do not change the name of this directory for consistency with other notebooks. - Do not add the contents of `data` to the repo, just the empty directory. +- ### Outputs -- List the products the notebook generates (plots, tables, derived data, etc.) -- If there are intermediate products produced by your notebook, generate an `output` directory for those data. - Please do not change the name of this directory for consistency with other notebooks. - Do not add the contents of `output` to the repo, just the empty directory. +- ### Runtime -Please report actual numbers and machine details for your notebook if it is expected to run longer or requires specific machines, for example, on Fornax. -Also, if querying archives, please include a statement like: -"This runtime is heavily dependent on archive servers which means runtime will vary for users". - -Here is a template runtime statement: -As of {Date}, this notebook takes ~{N}s to run to completion on Fornax using the ‘Default Astrophysics' image and the ‘{name: size}’ server with NGB RAM/ NCPU. +As of 22nd November 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports -This should be a list of the modules that are required to run this code. -Importantly, even those that are already installed in Fornax should be listed here so users wanting to run this locally on their own machines have the information they need to do this. +```{code-cell} python +# import contextlib +import multiprocessing as mp +import os -Make sure that you have built a "requirements_notebook_name.txt" file with the modules to be imported. -The name of the notebook should be present in the name of the requirements file, as in our example "requirements_notebook_template.txt" +import heasoftpy as hsp -```{code-cell} python -# This cell should not be edited below this line except for the name of -# the requirements_notebook_name.txt +# from typing import Tuple, Union +# from warnings import warn -# Uncomment the next line to install dependencies if needed. -# %pip install -r requirements_notebook_name.txt -``` -```{code-cell} python -import numpy as np +# import matplotlib.pyplot as plt +# import numpy as np +# import xspec as xs +# from astropy.coordinates import SkyCoord +# from astropy.units import Quantity +# from astroquery.heasarc import Heasarc +# from matplotlib.ticker import FuncFormatter +# from tqdm import tqdm ``` ## Global Setup ### Functions -Please avoid writing functions where possible, but if they are necessary, then place them in the following -code cell - it will be minimized unless the user decides to expand it. **Please replace this text with concise -explanations of your functions or remove it if there are no functions.** - ```{code-cell} python --- tags: [hide-input] @@ -104,9 +82,6 @@ jupyter: source_hidden: true --- -# This cell will be automatically collapsed when the notebook is rendered, which helps -# to hide large and distracting functions while keeping the notebook self-contained -# and leaving them easily accessible to the user ``` ### Constants @@ -117,7 +92,11 @@ tags: [hide-input] jupyter: source_hidden: true --- +# The name of the source we're examining in this demonstration +SRC_NAME = "AX J1910.7+0917" +# Controls the verbosity of all HEASoftPy tasks +TASK_CHATTER = 3 ``` ### Configuration @@ -128,123 +107,62 @@ tags: [hide-input] jupyter: source_hidden: true --- - +# ------------- Configure global package settings -------------- +# Raise Python exceptions if a heasoftpy task fails +# TODO Remove once this becomes a default in heasoftpy +hsp.Config.allow_failure = False + +# Set up the method for spawning processes. +mp.set_start_method("fork", force=True) +# -------------------------------------------------------------- + +# ------------- Setting how many cores we can use -------------- +NUM_CORES = None +total_cores = os.cpu_count() + +if NUM_CORES is None: + NUM_CORES = total_cores +elif not isinstance(NUM_CORES, int): + raise TypeError( + "If manually overriding 'NUM_CORES', you must set it to an integer value." + ) +elif isinstance(NUM_CORES, int) and NUM_CORES > total_cores: + raise ValueError( + f"If manually overriding 'NUM_CORES', the value must be less than or " + f"equal to the total available cores ({total_cores})." + ) +# -------------------------------------------------------------- + +# -------------- Set paths and create directories -------------- +if os.path.exists("../../../_data"): + ROOT_DATA_DIR = "../../../_data/XRISM/" +else: + ROOT_DATA_DIR = "XRISM/" + +ROOT_DATA_DIR = os.path.abspath(ROOT_DATA_DIR) + +# Make sure the download directory exists. +os.makedirs(ROOT_DATA_DIR, exist_ok=True) + +# Setup path and directory into which we save output files from this example. +OUT_PATH = os.path.abspath("XRISM_output") +os.makedirs(OUT_PATH, exist_ok=True) +# -------------------------------------------------------------- ``` *** ## 1. Data Access -The name of this, and all future sections can change. -In general, it probably is a good idea to start with something like "Data Access". -Please note, and stick to, the existing numbering scheme. - -```{code-cell} python -# Create some example data. -data = np.random.randint(0, 100, size=100) -``` - -## 2. Data Exploration - -Describe what the data look like. -Add summary statistics, initial plots, sanity checks. - -For cuts or other data filtering and cleaning steps, explain the scientific reasoning behind them. -This helps people understand both the notebook and the data so that they're more equipped to use the data appropriately in other contexts. - +++ -:::{tip} -Please include a narrative for *all* your code cells to help the reader figure out what you are doing and why you chose that path. - -Using [MyST admonitions](https://mystmd.org/guide/admonitions) such as this `tip` are encouraged -::: - -```{code-cell} python ---- -tags: [hide-input] -jupyter: - source_hidden: true ---- - -hist, bin_edges = np.histogram(data, bins=10) -hist -``` - -:::{important} -The HEASARC-tutorials style guide requires that cells using matplotlib (or similar) to produce figures -should be isolated (i.e., only contain plotting code), and must include the following metadata to hide the -code from view (see the source of this cell for the unrendered text: - ---- -tags: [hide-input] -jupyter: - source_hidden: true ---- -::: - -For any Figures, please add a few sentences about what the users should be noticing. - -+++ - -## 3. Analysis - -The working part of the notebook. -Lay out the step-by-step analysis workflow. -Each subsection should describe what is being done and why. -These can be sections or subsections. - -+++ - -### 3.1 Design Principles - -- Make no assumptions: define terms, common acronyms, link to things you reference. -- Keep in mind who your audience is. -- Design for portability - will this notebook work on both Fornax and someone's individual laptop. -- Cells capture logical units of work. -- Use markdown before or after cells to describe what is happening in the notebook. - -+++ - -### 3.2 Style Principles - -- Follow suggestions of The Turing Way community [markdown style](https://book.the-turing-way.org/community-handbook/style) -- Write each sentence in a new line (line breaks) to make changes easier to read in PRs. -- Avoid latin abbreviation to avoid failing CI. - -#### 3.2.1 Best Practice Guidelines -It would be nice if all contributed codes did the following, but these guidelines will not be checked in a code review - -- Section titles should not end with ":". -- List items should start at the beginning of the line, no spaces first. Exception is nested lists. -- One empty line between section header and text. -- One empty line before a list and after. -- No more than one empty line between any two non-empty lines. - -```{code-cell} python - -``` - -## 4. PR Review - -Notebooks go through a two step process: first step is getting into the repo, and the second step gets it into the [published tutorials](https://nasa-fornax.github.io/fornax-demo-notebooks/). -Final notebooks are expected to go through both a science and tech review checklist. -Checklists are [here](https://github.com/nasa-fornax/fornax-demo-notebooks/tree/main/template/notebook_review_checklists.md). -Please consider these checklist requirements as you are writing your code. - -The first PR of a notebook does not need to have everything from the checklists completed, but should have all the pieces there, and the authors should be aware of the requirements. - -To complete the second step of this process and be both rendered and included in users Fornax home directories, both a science and technical reviewer will be looking at [this checklist](https://github.com/nasa-fornax/fornax-demo-notebooks/tree/main/template/notebook_review_checklists.md) to see if the new tutorial notebook meets all of the requirements, or has a reasonable excuse not to. - -Any PRs can be opened as drafts, which is in fact preferred, if authors are still working on them. +## About this notebook -+++ +Author: David J Turner, HEASARC Staff Scientist. -## About this notebook +Author: Kenji Hamaguchi, XRISM GOF Scientist. -- **Authors:** Specific author and/or team names, plus "and the Fornax team". -- **Contact:** For help with this notebook, please open a topic in the [Fornax Community Forum](https://discourse.fornax.sciencecloud.nasa.gov/) "Support" category. -- Please edit and keep the above 2 bullet points, and remove this last line. +Updated On: 2025-11-19 +++ @@ -253,16 +171,5 @@ Any PRs can be opened as drafts, which is in fact preferred, if authors are stil ### Acknowledgements -Did anyone help you? -Probably these teams did, so include them: MAST, HEASARC, & IRSA Fornax teams. - -Did you use AI for any part of this tutorial, if so please include a statement such as: -"AI: This notebook was created with assistance from OpenAI’s ChatGPT 5 model.", which is a good time to mention that this template notebook was created with assistance from OpenAI’s ChatGPT 5 model. ### References - -This work made use of: - -- STScI style guide: https://github.com/spacetelescope/style-guides/blob/master/guides/jupyter-notebooks.md -- Fornax tech and science review guidelines: https://github.com/nasa-fornax/fornax-demo-notebooks/blob/main/template/notebook_review_checklists.md -- The Turing Way Style Guide: https://book.the-turing-way.org/community-handbook/style From 42285669fc867d921c888ea907478b2dd2b2ae01 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 20 Nov 2025 13:43:01 -0500 Subject: [PATCH 003/221] Added name of possible target to xrism xtend getting started notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 04a3d274..6b5dc592 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -93,7 +93,8 @@ jupyter: source_hidden: true --- # The name of the source we're examining in this demonstration -SRC_NAME = "AX J1910.7+0917" +SRC_NAME = "NGC4151" +# SRC_NAME = "AX J1910.7+0917" # Controls the verbosity of all HEASoftPy tasks TASK_CHATTER = 3 From cf3d81b597cc25e808e99cabb3f612129faf028d Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 10:36:29 -0500 Subject: [PATCH 004/221] Started writing the text of the XRISM-Xtend tutorial, including a brief introduction to the xtdpipeline, and a note about the relatively immature nature of XRISM's software and best practises. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 159 ++++++++++++++++-- 1 file changed, 149 insertions(+), 10 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 6b5dc592..959be482 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-11-19' +date: '2025-11-24' file_format: mystnb jupytext: text_representation: @@ -35,6 +35,9 @@ By the end of this tutorial, you will be able to: ## Introduction +XRISM is... + + ### Inputs @@ -52,21 +55,24 @@ As of 22nd November 2025, this notebook takes ~{N}m to run to completion on Forn ```{code-cell} python # import contextlib +import glob import multiprocessing as mp import os import heasoftpy as hsp +# import matplotlib.pyplot as plt +# import numpy as np +# import xspec as xs +from astropy.coordinates import SkyCoord +from astropy.time import Time +from astropy.units import Quantity +from astroquery.heasarc import Heasarc + # from typing import Tuple, Union # from warnings import warn -# import matplotlib.pyplot as plt -# import numpy as np -# import xspec as xs -# from astropy.coordinates import SkyCoord -# from astropy.units import Quantity -# from astroquery.heasarc import Heasarc # from matplotlib.ticker import FuncFormatter # from tqdm import tqdm ``` @@ -93,7 +99,8 @@ jupyter: source_hidden: true --- # The name of the source we're examining in this demonstration -SRC_NAME = "NGC4151" +SRC_NAME = "LMC N132D" +# SRC_NAME = "NGC4151" # SRC_NAME = "AX J1910.7+0917" # Controls the verbosity of all HEASoftPy tasks @@ -153,7 +160,134 @@ os.makedirs(OUT_PATH, exist_ok=True) *** -## 1. Data Access +## 1. Finding and downloading XRISM observations of **NAMEHERE** + + +### Determining the name of the XRISM observation summary table + +```{code-cell} python +catalog_name = Heasarc.list_catalogs(master=True, keywords="xrism")[0]["name"] +catalog_name +``` + +### What are the coordinates of **NAMEHERE**? + +To search for relevant observations, we have to know the coordinates of our +source. The astropy module allows us to look up a source name in CDS' Sesame name + resolver and retrieve its coordinates. + +```{hint} +You could also set up a SkyCoord object directly, if you already know the coordinates. +``` + +```{code-cell} python +src_coord = SkyCoord.from_name(SRC_NAME) +# This will be useful later on in the notebook, for functions that take +# coordinates as an astropy Quantity. +src_coord_quant = Quantity([src_coord.ra, src_coord.dec]) +src_coord +``` + +### Searching for relevant observations + +```{code-cell} python +all_xrism_obs = Heasarc.query_region(src_coord, catalog_name) +all_xrism_obs +``` + +For an active mission (i.e. actively collecting data and adding to the archive)... + +```{code-cell} python +public_times = Time(all_xrism_obs["public_date"], format="mjd") +avail_xrism_obs = all_xrism_obs[public_times <= Time.now()] +rel_obsids = avail_xrism_obs["obsid"].value.data + +avail_xrism_obs +``` + +### Downloading the selected XRISM observations + +```{code-cell} python +data_links = Heasarc.locate_data(avail_xrism_obs) +data_links +``` + +```{code-cell} python +Heasarc.download_data(data_links, "aws", ROOT_DATA_DIR) +``` + +```{note} +We choose to download the data from the HEASARC AWS S3 bucket, but you could +pass 'heasarc' to acquire data from the FTP server. Additionally, if you are working +on SciServer, you may pass 'sciserver' to use the pre-mounted HEASARC dataset. +``` + +### What do the downloaded data directories contain? + +```{code-cell} python +glob.glob(os.path.join(ROOT_DATA_DIR, rel_obsids[0], "") + "*") +``` + +```{code-cell} python +glob.glob(os.path.join(ROOT_DATA_DIR, rel_obsids[0], "xtend", "") + "**/*") +``` + +## 2. Processing XRISM-Xtend data + +There are multiple steps involved in processing XRISM-Xtend data into a +science-ready state. As with many NASA-affiliated high-energy missions, HEASoft +includes a beginning-to-end pipeline(s) to streamline this process for XRISM data - the +XRISM-Xtend and Resolve instruments both have their own pipelines. + +In this tutorial we are focused only on preparing and using data from XRISM's Xtend +instrument and will not discuss how to handle XRISM-Resolve data; we note however that +there is a third XRISM pipeline task in HEASoft called `xapipeline`, which can be used +to run either or both the Xtend and Resolve pipelines. It contains some convenient +functionality that can identify and automatically pass the attitude, housekeeping, etc. files. + +We will show you how to run the Xtend-specific pipeline, `xtdpipeline`, but the +use of `xapipeline` is nearly functionally identical. + +The Python interface to HEASoft, HEASoftPy, is used throughout this tutorial, and we +will implement parallel observation processing wherever possible. + +### HEASoft and HEASoftPy versions + +```{warning} +XRISM is a relatively new mission, and as such the analysis software and recommended +best practises are still immature and evolving. We are checking and updating this tutorial +on a regular basis, but please report any issues or suggestions to the HEASARC Help Desk. +``` + +Both the HEASoft and HEASoftPy package versions can be retrieved from the +HEASoftPy module. + +The HEASoft version: +```{code-cell} python +hsp.fversion() +``` + +The HEASoftPy version: +```{code-cell} python +hsp.__version__ +``` + +### Running the XRISM-Xtend pipeline + +`xtdpipeline` will take us from a brand-new set of raw XRISM-Xtend data files, all the way +through to generating the 'quick-look' data products (images, spectra, and light curves) +included in HEASARC's XRISM archive 'products' directories. + +The pipeline has three stages and provides the option to start and stop the processing +at any of those stages; this can be useful if you wish to re-run a stage with slightly +different configuration without repeating the entire pipeline run. + +A different set of tasks is encapsulated by each stage, and they have the following general goals: +- **Stage 1** - Calibration and preparation of raw Xtend data. +- **Stage 2** - Screening and filtering of the prepared Xtend event lists. +- **Stage 3** - Generation of quick-look data products. + + +++ @@ -163,12 +297,17 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-11-19 +Updated On: 2025-11-24 +++ ### Additional Resources +HEASoftPy GitHub Repository: https://github.com/HEASARC/heasoftpy + +HEASoftPy HEASARC Page: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/heasoftpy.html + +HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html ### Acknowledgements From df7bb378c79aee7dc5c47f12680b37ee2cf49b34 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 11:26:21 -0500 Subject: [PATCH 005/221] Added the beginnings of the wrapper for XTEND processing, as well as the call to it, and the setting up of template file paths. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 102 +++++++++++++++++- 1 file changed, 98 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 959be482..0756961e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -54,7 +54,7 @@ As of 22nd November 2025, this notebook takes ~{N}m to run to completion on Forn ## Imports ```{code-cell} python -# import contextlib +import contextlib import glob import multiprocessing as mp import os @@ -87,7 +87,46 @@ tags: [hide-input] jupyter: source_hidden: true --- - +def process_xrism_xtend(cur_obs_id: str, out_dir: str): + """ + A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process + XRISM-Xtend observation data. This wrapper function is primarily to enable the + use of multiprocessing. + + This function is set to run xtdpipeline until the end of stage 2, excluding the + final stage that generates the 'quick-look' data products. + + :param str cur_obs_id: The ObsID of the XRISM observation to be processed. + :param str out_dir: The directory where output files should be written. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + + # Makes sure the specified output directory exists. + os.makedirs(out_dir, exist_ok=True) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(out_dir), hsp.utils.local_pfiles_context(): + + # The processing/preparation stage of any X-ray telescope's data is the most + # likely to go wrong, and we use a Python try-except as an automated way to + # collect ObsIDs that had an issue during processing. + try: + out = hsp.xtdpipeline( + indir=os.path.join(ROOT_DATA_DIR, cur_obs_id), + outdir=out_dir, + instrument="XTEND", + ) + task_success = True + + except hsp.HSPTaskException as err: + task_success = False + out = str(err) + + return cur_obs_id, out, task_success ``` ### Constants @@ -195,7 +234,7 @@ all_xrism_obs = Heasarc.query_region(src_coord, catalog_name) all_xrism_obs ``` -For an active mission (i.e. actively collecting data and adding to the archive)... +For an active mission (i.e., actively collecting data and adding to the archive)... ```{code-cell} python public_times = Time(all_xrism_obs["public_date"], format="mjd") @@ -263,15 +302,37 @@ Both the HEASoft and HEASoftPy package versions can be retrieved from the HEASoftPy module. The HEASoft version: + ```{code-cell} python hsp.fversion() ``` The HEASoftPy version: + ```{code-cell} python hsp.__version__ ``` +### Setting up file paths to pass to the XRISM-Xtend pipeline + +```{code-cell} python +att_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.att.gz") + +orbit_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.orb.gz") + +obs_gti_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}_gen.gti.gz") + +xtd_hk_path_temp = os.path.join( + ROOT_DATA_DIR, "{oi}", "xtend", "hk", "xa{oi}xtd_a0.hk.gz" +) + +mkf_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.mkf.gz") + +ehk_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.ehk.gz") + +file_stem_temp = "xa{oi}" +``` + ### Running the XRISM-Xtend pipeline `xtdpipeline` will take us from a brand-new set of raw XRISM-Xtend data files, all the way @@ -288,8 +349,41 @@ A different set of tasks is encapsulated by each stage, and they have the follow - **Stage 3** - Generation of quick-look data products. +***MUCH MORE SPECIFIC INFORMATION SHOULD GO HERE*** -+++ + +```{note} +We will stop the execution of `xtdpipeline` at **Stage 2**, as the latter part of this +demonstration will show you how to make more customised data products than are output +by default. +``` + +Though we are using the HEASoftPy `xtdpipeline` function, called +as `hsp.xtdpipeline(indir=...)`, it is called within a wrapper function we have +written in the 'Global Setup: Functions' section of this notebook. The `process_xrism_xtend` +wrapper function exists primarily to let us run the processing of different XRISM-Xtend +observations in parallel. + +We can use Python's multiprocessing module to call the wrapper function for each +of our XRISM observations, passing the relevant arguments. + +The multiprocessing pool will then execute the processing of observations +simultaneously, if there are more cores available than there are observations. + +If there are fewer cores than observations, the pool will handle the allocation of +resources to each observation's processing run, and they will be processed in parallel +until all are complete. + +```{code-cell} python +with mp.Pool(NUM_CORES) as p: + arg_combs = [[oi, os.path.join(OUT_PATH, oi), src_coord] for oi in rel_obsids] + pipe_result = p.starmap(process_xrism_xtend, arg_combs) + +xtd_pipe_problem_ois = [all_out[0] for all_out in pipe_result if not all_out[2]] +rel_obsids = [oi for oi in rel_obsids if oi not in xtd_pipe_problem_ois] + +xtd_pipe_problem_ois +``` ## About this notebook From 3dcd9e3bbf3f5a5ea69844e7897cce679d1ec0b6 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 12:48:37 -0500 Subject: [PATCH 006/221] Added explanation of the supporting files required by xtdpipeline to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 42 +++++++++++++++++-- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0756961e..b8cd16a5 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -315,6 +315,26 @@ hsp.__version__ ### Setting up file paths to pass to the XRISM-Xtend pipeline +In order to properly prepare and calibrate XRISM-Xtend data, `xtdpipeline` must +make use of a number of housekeeping files that describe the observatory's status. + +Here we set up template file path variables to the required files so that we can +more easily pass observation-specific file paths to the XRISM-Xtend processing +function in the next section. + +The only expected difference in file name between the equivalent files of different +observations is the included ObsID string, represented by the `{oi}` placeholder. This +placeholder will be replaced by the relevant ObsID for each observation being processed. + +In summary, the supporting files required by `xtdpipeline` are: +- **Attitude file** - Describes the pointing of XRISM in many short time steps throughout the observation. +- **Orbit file** - Orbital telemetry of the XRISM spacecraft during the observation. +- **Observation good-time-intervals (GTI) file** - Contains base GTIs for the observation; used to exclude times when the spacecraft was slewing, or its attitude was inconsistent with that required to observe the target. +- **Filter file (MKF)** - The base filters used to exclude times when the instruments or spacecraft were not operating normally. +- **Extended housekeeping (EHK) file** - Contains extra information about the observation derived from attitude and orbit files, used to screen events. Much of the data relates to attitude, the South Atlantic Anomaly (SAA), and cut-off rigidity (COR). +- **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarises the electrical and thermal state of Xtend in small time steps throughout the observation. + + ```{code-cell} python att_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.att.gz") @@ -322,14 +342,23 @@ orbit_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.orb.gz") obs_gti_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}_gen.gti.gz") +# The overall XRISM observation filter file +mkf_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.mkf.gz") + +# The XRISM extended housekeeping file +ehk_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.ehk.gz") + +# The Xtend housekeeping file xtd_hk_path_temp = os.path.join( ROOT_DATA_DIR, "{oi}", "xtend", "hk", "xa{oi}xtd_a0.hk.gz" ) +``` -mkf_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.mkf.gz") - -ehk_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.ehk.gz") +`xtdpipeline` also needs the 'stem' of the input file names to be defined, so that it +can identify the relevant event list files. The way we call the pipeline, the input +stem will also be used to format output file names. +```{code-cell} python file_stem_temp = "xa{oi}" ``` @@ -385,6 +414,13 @@ rel_obsids = [oi for oi in rel_obsids if oi not in xtd_pipe_problem_ois] xtd_pipe_problem_ois ``` +```{warning} +Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. +``` + +## 3. + + ## About this notebook Author: David J Turner, HEASARC Staff Scientist. From 9e47c43e5aabda95fd1f1a3ce3e35ba3f4e056c7 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 13:01:41 -0500 Subject: [PATCH 007/221] Filled out the process_xrism_xtend docstring and its call to hsp.xtendpipeline in the getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 39 +++++++++++++++++-- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b8cd16a5..4df8f5d0 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -87,7 +87,18 @@ tags: [hide-input] jupyter: source_hidden: true --- -def process_xrism_xtend(cur_obs_id: str, out_dir: str): +def process_xrism_xtend( + cur_obs_id: str, + evt_dir: str, + out_dir: str, + attitude: str, + orbit: str, + obs_gti: str, + mkf_filter: str, + file_stem: str, + extended_housekeeping: str, + xtend_housekeeping: str, +): """ A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process XRISM-Xtend observation data. This wrapper function is primarily to enable the @@ -97,7 +108,19 @@ def process_xrism_xtend(cur_obs_id: str, out_dir: str): final stage that generates the 'quick-look' data products. :param str cur_obs_id: The ObsID of the XRISM observation to be processed. + :param str evt_dir: The directory containing the raw, unfiltered, event list + files for the observation. :param str out_dir: The directory where output files should be written. + :param str attitude: XRISM attitude file for the observation. + :param str orbit: XRISM orbit file for the observation. + :param str obs_gti: XRISM base good-time-invterval file for the observation. + :param str mkf_filter: XRISM overall filter file for the observation. + :param str file_stem: The stem of the input event list files (also used for + output file names). + :param str extended_housekeeping: Extended housekeeping file for the + XRISM observation. + :param str xtend_housekeeping: Instrument-specific Xtend housekeeping file + for the observation. :return: A tuple containing the processed ObsID, the log output of the pipeline, and a boolean flag indicating success (True) or failure (False). :rtype: Tuple[str, hsp.core.HSPResult, bool] @@ -116,9 +139,18 @@ def process_xrism_xtend(cur_obs_id: str, out_dir: str): # collect ObsIDs that had an issue during processing. try: out = hsp.xtdpipeline( - indir=os.path.join(ROOT_DATA_DIR, cur_obs_id), + entry_stage=1, + exit_stage=2, + steminputs=file_stem, + stemoutputs=file_stem, + indir=evt_dir, outdir=out_dir, - instrument="XTEND", + attitude=attitude, + orbit=orbit, + obsgti=obs_gti, + makefilter=mkf_filter, + extended_housekeeping=extended_housekeeping, + housekeeping=xtend_housekeeping, ) task_success = True @@ -334,7 +366,6 @@ In summary, the supporting files required by `xtdpipeline` are: - **Extended housekeeping (EHK) file** - Contains extra information about the observation derived from attitude and orbit files, used to screen events. Much of the data relates to attitude, the South Atlantic Anomaly (SAA), and cut-off rigidity (COR). - **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarises the electrical and thermal state of Xtend in small time steps throughout the observation. - ```{code-cell} python att_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.att.gz") From 721de5430cd7fe8f296e7c9e4c10be090dbe93bb Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 13:27:42 -0500 Subject: [PATCH 008/221] Built the list of arguments that process_xrism_xtend needs in the getting-started-xrism-xtend.md. Also put in subsection heading for the 'generating new products' part. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 44 +++++++++++++++++-- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 4df8f5d0..198693e6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -89,8 +89,8 @@ jupyter: --- def process_xrism_xtend( cur_obs_id: str, - evt_dir: str, out_dir: str, + evt_dir: str, attitude: str, orbit: str, obs_gti: str, @@ -108,9 +108,9 @@ def process_xrism_xtend( final stage that generates the 'quick-look' data products. :param str cur_obs_id: The ObsID of the XRISM observation to be processed. + :param str out_dir: The directory where output files should be written. :param str evt_dir: The directory containing the raw, unfiltered, event list files for the observation. - :param str out_dir: The directory where output files should be written. :param str attitude: XRISM attitude file for the observation. :param str orbit: XRISM orbit file for the observation. :param str obs_gti: XRISM base good-time-invterval file for the observation. @@ -367,10 +367,13 @@ In summary, the supporting files required by `xtdpipeline` are: - **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarises the electrical and thermal state of Xtend in small time steps throughout the observation. ```{code-cell} python +# File containing XRISM pointing information att_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.att.gz") +# File containing XRISM orbital telemetry orbit_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.orb.gz") +# The base XRISM observation GTI file obs_gti_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}_gen.gti.gz") # The overall XRISM observation filter file @@ -393,6 +396,14 @@ stem will also be used to format output file names. file_stem_temp = "xa{oi}" ``` +Finally, we set up a template variable for the directory containing the raw +Xtend event information for each observation. It contains several files, and +`xtdpipeline` will identify the ones it needs to use: + +```{code-cell} python +raw_evt_dir_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "xtend", "event_uf") +``` + ### Running the XRISM-Xtend pipeline `xtdpipeline` will take us from a brand-new set of raw XRISM-Xtend data files, all the way @@ -436,7 +447,22 @@ until all are complete. ```{code-cell} python with mp.Pool(NUM_CORES) as p: - arg_combs = [[oi, os.path.join(OUT_PATH, oi), src_coord] for oi in rel_obsids] + arg_combs = [ + [ + oi, + os.path.join(OUT_PATH, oi), + raw_evt_dir_temp.format(oi=oi), + att_path_temp.format(oi=oi), + orbit_path_temp.format(oi=oi), + obs_gti_path_temp.format(oi=oi), + mkf_path_temp.format(oi=oi), + file_stem_temp.format(oi=oi), + ehk_path_temp.format(oi=oi), + xtd_hk_path_temp.format(oi=oi), + ] + for oi in rel_obsids + ] + pipe_result = p.starmap(process_xrism_xtend, arg_combs) xtd_pipe_problem_ois = [all_out[0] for all_out in pipe_result if not all_out[2]] @@ -449,8 +475,18 @@ xtd_pipe_problem_ois Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` -## 3. +## 3. Generating new XRISM-Xtend images, exposure maps, spectra, and light curves + +### New XRISM-Xtend images + +### New XRISM-Xtend exposure maps + +### New XRISM-Xtend spectra and supporting files + +### New XRISM-Xtend light curves + +## 4. Fitting ## About this notebook From 749378f3c4cd74dbfaf1e0dddb377df96d2b198e Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 14:31:12 -0500 Subject: [PATCH 009/221] Made some changes to the process_xrism_xtend function, as xtdpipeline gets angry if the outdir already exists, and I like to make sure my demos move INTO the out directory to work. As such there is a lower layer temporary outdir created which is where xtdpipeline can write files. All in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 198693e6..839737f9 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -127,7 +127,8 @@ def process_xrism_xtend( """ # Makes sure the specified output directory exists. - os.makedirs(out_dir, exist_ok=True) + temp_outdir = os.path.join(out_dir, "temp") + os.makedirs(temp_outdir, exist_ok=True) # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so @@ -144,13 +145,14 @@ def process_xrism_xtend( steminputs=file_stem, stemoutputs=file_stem, indir=evt_dir, - outdir=out_dir, + outdir=temp_outdir, attitude=attitude, orbit=orbit, obsgti=obs_gti, makefilter=mkf_filter, extended_housekeeping=extended_housekeeping, housekeeping=xtend_housekeeping, + clobber=True, ) task_success = True @@ -158,6 +160,12 @@ def process_xrism_xtend( task_success = False out = str(err) + # Moves files from the temporary output directory into the + # final output directory + if os.path.exists(temp_outdir) and len(os.listdir(temp_outdir)) != 0: + for f in os.listdir(temp_outdir): + os.rename(os.path.join(temp_outdir, f), os.path.join(out_dir, f)) + return cur_obs_id, out, task_success ``` From 2d33eeace5b250bedeca85d6e83ccf3205f836a4 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 10:53:39 -0500 Subject: [PATCH 010/221] Added discussion of the mapping between XRISM-Xtend channel and energy, as well as a demonstration of how to validate our assumed relation through looking at a pre-generated RMF. All in getting-started-xrism-xtend.md, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 131 +++++++++++++++++- 1 file changed, 125 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 839737f9..b363268a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -60,11 +60,13 @@ import multiprocessing as mp import os import heasoftpy as hsp +import matplotlib.pyplot as plt +import numpy as np -# import matplotlib.pyplot as plt -# import numpy as np # import xspec as xs from astropy.coordinates import SkyCoord +from astropy.io import fits +from astropy.table import Table from astropy.time import Time from astropy.units import Quantity from astroquery.heasarc import Heasarc @@ -184,6 +186,9 @@ SRC_NAME = "LMC N132D" # Controls the verbosity of all HEASoftPy tasks TASK_CHATTER = 3 + +# The approximate linear relationship between Xtend PI and event energy +XTD_EV_PER_CHAN = (1 / Quantity(166.7, "chan/keV")).to("eV/chan") ``` ### Configuration @@ -483,18 +488,132 @@ xtd_pipe_problem_ois Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` -## 3. Generating new XRISM-Xtend images, exposure maps, spectra, and light curves +### Identifying problem pixels + +```{code-cell} python + +``` + +## 3. Generating new XRISM-Xtend images, exposure maps, and light curves + +### Converting energy bounds to channel bounds + +```{code-cell} python +XTD_EV_PER_CHAN +``` + +Alternatively, we can figure out this relationship between PI and energy by looking at +a XRISM-Xtend Redistribution Matrix File (RMF), which exists to describe this +mapping. + +We will be creating new RMFs as part of the generation of XRISM-Xtend spectra in the +next section. For our current purpose, however, it is acceptable to use the RMFs that +were included in the XRISM-Xtend archive we downloaded earlier. + +The archived RMFs are generated for the entire Xtend FoV, rather than for the CCDs +our particular target fall on, but practically speaking, that doesn't make a significant +difference. + +Using observation 000128000 as an example, we determine the path to the relevant +pre-generated RMF. We only expect a single file, and include a validity check to +ensure that this does not change in future versions of the archive: + +```{code-cell} python +chosen_demo_obsid = "000128000" + +pregen_rmf_wildcard = os.path.join( + ROOT_DATA_DIR, "{oi}", "xtend", "products", "xa{oi}xtd_p*.rmf*" +) +poss_rmfs = glob.glob(pregen_rmf_wildcard.format(oi=chosen_demo_obsid)) +print(poss_rmfs) + +# Check how many RMF files we found - there should only be one +if len(poss_rmfs) != 1: + raise ValueError(f"Expected exactly one RMF file, but found {len(poss_rmfs)}.") +else: + pregen_rmf_path = poss_rmfs[0] +``` + +XRISM-Xtend RMFs are written in the FITS file format, and so can be read into +Python using the `astropy.io.fits` module: + +```{code-cell} python +# Loading the fits file using astropy +with fits.open(pregen_rmf_path) as rmfo: + # Iterate through the tables in the RMF, printing their names + for tab in rmfo: + print(tab.name) + + # Associate the EBOUNDS table with a variable, so it can be used outside + # the fits.open context + e_bounds = rmfo["EBOUNDS"].data + +# Convert the read-out energy bound information to an astropy Table, mainly +# because it will look nicer whe we show it below +e_bounds = Table(e_bounds) +# Display a subset of the table +e_bounds[90:110] +``` + +We can use this file to visualize the basic linear mapping between energy and +channel - it will not be the most interesting figure you've ever seen: + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +plt.figure(figsize=(5.5, 5.5)) + +plt.minorticks_on() +plt.tick_params(which="both", direction="in", top=True, right=True) + +mid_ens = (e_bounds["E_MIN"] + e_bounds["E_MAX"]) / 2 + +plt.plot(e_bounds["CHANNEL"], mid_ens, color="navy", alpha=0.9, label="XRISM-Xtend") + +plt.xlim(0) +plt.ylim(0) + +plt.xlabel("Channel [PI]", fontsize=15) +plt.ylabel("Central Energy [keV]", fontsize=15) + +plt.legend(fontsize=14) + +plt.tight_layout() +plt.show() +``` + +Finally, we can validate our assumed relationship between energy and channel by +calculating the mean change in minimum energy between adjacent channels: + +```{code-cell} python +# +rmf_ev_per_chan = Quantity(np.diff(e_bounds["E_MIN"].data).mean(), "keV/chan").to( + "eV/chan" +) +rmf_ev_per_chan +``` + +Clearly, our assumed relationship is valid: + +```{code-cell} python +rmf_ev_per_chan / XTD_EV_PER_CHAN +``` ### New XRISM-Xtend images -### New XRISM-Xtend exposure maps -### New XRISM-Xtend spectra and supporting files + +### New XRISM-Xtend exposure maps ### New XRISM-Xtend light curves -## 4. Fitting +## 4. Generating new XRISM-Xtend spectra and supporting files + + ## About this notebook From 9ddfa9004cebc24df8e8ef9910b49adc053805ff Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 11:21:58 -0500 Subject: [PATCH 011/221] Added more explanation of why a mapping between channel and energy is required for generating energy bound data products for XRISM-Xtend to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 33 ++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b363268a..b70d3b28 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -74,7 +74,6 @@ from astroquery.heasarc import Heasarc # from typing import Tuple, Union # from warnings import warn - # from matplotlib.ticker import FuncFormatter # from tqdm import tqdm ``` @@ -498,6 +497,34 @@ Processing XRISM-Xtend data can take a long time, up to several hours for a sing ### Converting energy bounds to channel bounds +The data products we generate in this section can all benefit from selecting events +from within a specific energy range. This might be because your source of interest only +emits in a narrow energy range, and you don't care about the rest, or because different +mechanisms emit at different energies, and you wish to separate them. + +Such filtering needs to be performed at the event list level so that the resulting +subset of events can be binned in spatial and temporal dimensions to produce +images and light curves. + +The event lists of most high-energy missions (including XRISM) do not directly store +event energies - instead they contain the pulse-height-amplitude (PHA), and/or the +pulse-invariant (PI) channel (calculated from PHA and instrument gain tables) information. + +This is because the calibration of detector-channel to energy, the understanding of the +behaviors of the instrument and its electronics, and the performance of the detectors +can all change dramatically over time. + +All that said, the tools we will use to generate our energy-bounded images and light +curves do not take _energy_ bounds as an input, but rather _channel_ bounds. + +Thus, we have the responsibility of determining equivalent channel bounds for our +hopefully-physics-driven energy-bound choices. For images and light curves, we can +safely assume a perfect linear relationship between energy and channel. + +The XRISM ABC guide provides the following mapping +([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/Xtend_Data_Analysis.html#SECTION001043000000000000000)) +for Xtend: + ```{code-cell} python XTD_EV_PER_CHAN ``` @@ -627,6 +654,8 @@ Updated On: 2025-11-24 ### Additional Resources +XRISM Data Reduction (ABC) Guide - https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide + HEASoftPy GitHub Repository: https://github.com/HEASARC/heasoftpy HEASoftPy HEASARC Page: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/heasoftpy.html @@ -637,3 +666,5 @@ HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/softwa ### References + +[XRISM GOF & SDC (2024) - _XRISM ABC GUIDE XTEND ENERGY-CHANNEL MAPPING_ [ACCESSED 25-NOV-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/Xtend_Data_Analysis.html#SECTION001043000000000000000) From 32e7de8c052c5b495308ccea4c063f9506b9e4f5 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 11:55:07 -0500 Subject: [PATCH 012/221] Added the beginnings of a wrapper for extractor to generate XRISM-Xtend images (gen_xrism_xtend_image) to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b70d3b28..70dcd517 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -168,6 +168,74 @@ def process_xrism_xtend( os.rename(os.path.join(temp_outdir, f), os.path.join(out_dir, f)) return cur_obs_id, out, task_success + + +def gen_xrism_xtend_image( + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + lo_en: Quantity, + hi_en: Quantity, +): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend image. + :param str cur_xtend_data_class: + :param str event_file: + :param str out_dir: The directory where output files should be written. + :param Quantity lo_en: Lower bound of the energy band within which we will + generate the image. + :param Quantity hi_en: Upper bound of the energy band within which we will + generate the image. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + if cur_xtend_data_class[0] != "3": + raise ValueError( + f"The first digit of the Xtend data class ({cur_xtend_data_class}) " + "must be 3 for in-flight data." + ) + + # Make sure the lower and upper energy limits make sense + if lo_en > hi_en: + raise ValueError( + "The lower energy limit must be less than or equal to the upper " + "energy limit." + ) + else: + lo_en_val = lo_en.to("keV").value + hi_en_val = hi_en.to("keV").value + + # Convert the energy limits to channel limits, rounding down and up to the nearest + # integer channel for the lower and upper bounds respectively. + lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + + # Create modified input event list file path, where we use the just-calculated + # PI channel limits to subset the events + evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" + + # Set up the output file name for the image we're about to generate. + im_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"en{lo_en_val}_{hi_en_val}keV-image.fits" + ) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(out_dir), hsp.utils.local_pfiles_context(): + + # The processing/preparation stage of any X-ray telescope's data is the most + # likely to go wrong, and we use a Python try-except as an automated way to + # collect ObsIDs that had an issue during processing. + out = hsp.extractor( + filename=evt_file_chan_sel, imgfile=im_out, noprompt=True, clobber=True + ) + + return out ``` ### Constants @@ -631,7 +699,22 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ### New XRISM-Xtend images +```{code-cell} python +# Defining the energy bounds we want images within +xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.3, 10.0]], "keV") + +# Convert energy bounds to channel bounds +xtd_im_ch_bounds = (xtd_im_en_bounds / XTD_EV_PER_CHAN).to("chan") +xtd_im_ch_bounds[:, 0] = np.floor(xtd_im_ch_bounds[:, 0]) +xtd_im_ch_bounds[:, 1] = np.ceil(xtd_im_ch_bounds[:, 1]) +xtd_im_ch_bounds = xtd_im_ch_bounds.astype(int) +xtd_im_ch_bounds +``` +```{code-cell} python +# hsp.extractor(filename=evt_file_chan_sel, imgfile='testo_img.fits', noprompt=True, +# clobber=True) +``` ### New XRISM-Xtend exposure maps From f11f5d315fff903b26ad86b147c5cc9468386033 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 13:16:04 -0500 Subject: [PATCH 013/221] Added bits and bobs to the getting-started-xrism-xtend.md notebook to make the image generation work. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 88 +++++++++++++++++-- 1 file changed, 80 insertions(+), 8 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 70dcd517..cc7a9120 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -58,6 +58,8 @@ import contextlib import glob import multiprocessing as mp import os +from random import randint +from shutil import rmtree import heasoftpy as hsp import matplotlib.pyplot as plt @@ -223,18 +225,29 @@ def gen_xrism_xtend_image( f"en{lo_en_val}_{hi_en_val}keV-image.fits" ) + # Create a temporary working directory + temp_work_dir = os.path.join( + out_dir, "im_extractor_{}".format(randint(0, int(1e8))) + ) + os.makedirs(temp_work_dir) + # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). - with contextlib.chdir(out_dir), hsp.utils.local_pfiles_context(): - + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): # The processing/preparation stage of any X-ray telescope's data is the most # likely to go wrong, and we use a Python try-except as an automated way to # collect ObsIDs that had an issue during processing. out = hsp.extractor( - filename=evt_file_chan_sel, imgfile=im_out, noprompt=True, clobber=True + filename=evt_file_chan_sel, + imgfile=os.path.join(out_dir, im_out), + noprompt=True, + clobber=True, ) + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + return out ``` @@ -342,7 +355,10 @@ src_coord ### Searching for relevant observations ```{code-cell} python -all_xrism_obs = Heasarc.query_region(src_coord, catalog_name) +col_str = ( + "obsid,name,ra,dec,time,exposure,status,public_date,xtd_dataclas1,xtd_dataclas2" +) +all_xrism_obs = Heasarc.query_region(src_coord, catalog_name, columns=col_str) all_xrism_obs ``` @@ -351,7 +367,20 @@ For an active mission (i.e., actively collecting data and adding to the archive) ```{code-cell} python public_times = Time(all_xrism_obs["public_date"], format="mjd") avail_xrism_obs = all_xrism_obs[public_times <= Time.now()] + +# Define a couple of useful variables that make accessing information in the +# table a little easier later on in the notebook +# Create an array of the relevant ObsIDs rel_obsids = avail_xrism_obs["obsid"].value.data +# Create a dictionary connecting ObsIDs to their associated Xtend data classes +rel_dataclasses = { + oi: [ + dc + for dc in avail_xrism_obs[oi_ind][["xtd_dataclas1", "xtd_dataclas2"]].values() + if dc != "" + ] + for oi_ind, oi in enumerate(rel_obsids) +} avail_xrism_obs ``` @@ -444,7 +473,7 @@ In summary, the supporting files required by `xtdpipeline` are: - **Observation good-time-intervals (GTI) file** - Contains base GTIs for the observation; used to exclude times when the spacecraft was slewing, or its attitude was inconsistent with that required to observe the target. - **Filter file (MKF)** - The base filters used to exclude times when the instruments or spacecraft were not operating normally. - **Extended housekeeping (EHK) file** - Contains extra information about the observation derived from attitude and orbit files, used to screen events. Much of the data relates to attitude, the South Atlantic Anomaly (SAA), and cut-off rigidity (COR). -- **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarises the electrical and thermal state of Xtend in small time steps throughout the observation. +- **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarizes the electrical and thermal state of Xtend in small time steps throughout the observation. ```{code-cell} python # File containing XRISM pointing information @@ -502,6 +531,9 @@ A different set of tasks is encapsulated by each stage, and they have the follow ***MUCH MORE SPECIFIC INFORMATION SHOULD GO HERE*** +***INCLUDE XTEND DATA CLASS SUMMARY, AND HOW THERE MAY BE MULTIPLE EVENT LISTS PER OBSID*** + + ```{note} We will stop the execution of `xtdpipeline` at **Stage 2**, as the latter part of this @@ -555,6 +587,19 @@ xtd_pipe_problem_ois Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` +Finally, we set up a template variable for the cleaned event lists we just created: + +***CONSIDER MORE COMMENTARY ON DATACLASS ETC. HERE*** + + +***Need to consider the 'p0' bit - apparently the '0' is a counter for splitting large datasets, but I don't know when it is used*** + +***sc == 'split counter'*** + +```{code-cell} python +evt_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") +``` + ### Identifying problem pixels ```{code-cell} python @@ -701,8 +746,16 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ```{code-cell} python # Defining the energy bounds we want images within -xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.3, 10.0]], "keV") +xtd_im_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") +``` + +Converting those energy bounds to channel bounds is straightforward, we simply divide +the energy values by our assumed mapping between energy and channel. + +The resulting lower and upper bound channel values are rounded down and up to the +nearest integer channel respectively. +```{code-cell} python # Convert energy bounds to channel bounds xtd_im_ch_bounds = (xtd_im_en_bounds / XTD_EV_PER_CHAN).to("chan") xtd_im_ch_bounds[:, 0] = np.floor(xtd_im_ch_bounds[:, 0]) @@ -711,9 +764,28 @@ xtd_im_ch_bounds = xtd_im_ch_bounds.astype(int) xtd_im_ch_bounds ``` +```{note} +Though we demonstrate how to convert energy to channel bounds above, the wrapper +function for image generation will repeat this exercise, as it will write +energy bounds into output file names. +``` + ```{code-cell} python -# hsp.extractor(filename=evt_file_chan_sel, imgfile='testo_img.fits', noprompt=True, -# clobber=True) +arg_combs = [ + [ + oi, + dc, + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + os.path.join(OUT_PATH, oi), + *cur_bnds, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs + for cur_bnds in xtd_im_en_bounds +] + +with mp.Pool(NUM_CORES) as p: + im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` ### New XRISM-Xtend exposure maps From ad722e42f5f00c692f28def4d985459a4075c87a Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 14:20:50 -0500 Subject: [PATCH 014/221] Added an image binning factor subsection to the getting-started-xrism-xtend.md notebook, and included some commentary. May add a bit about how to use an event list to find the size of the event pixels. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 36 +++++++++++++++++-- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index cc7a9120..0145cc85 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-11-24' +date: '2025-11-25' file_format: mystnb jupytext: text_representation: @@ -179,6 +179,7 @@ def gen_xrism_xtend_image( out_dir: str, lo_en: Quantity, hi_en: Quantity, + im_bin: int = 1, ): """ @@ -222,7 +223,7 @@ def gen_xrism_xtend_image( # Set up the output file name for the image we're about to generate. im_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"en{lo_en_val}_{hi_en_val}keV-image.fits" + f"imbinfactor{im_bin}-en{lo_en_val}_{hi_en_val}keV-image.fits" ) # Create a temporary working directory @@ -243,6 +244,7 @@ def gen_xrism_xtend_image( imgfile=os.path.join(out_dir, im_out), noprompt=True, clobber=True, + binf=im_bin, ) # Make sure to remove the temporary directory @@ -744,6 +746,8 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ### New XRISM-Xtend images +#### Image energy bounds + ```{code-cell} python # Defining the energy bounds we want images within xtd_im_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") @@ -770,6 +774,27 @@ function for image generation will repeat this exercise, as it will write energy bounds into output file names. ``` +#### Image binning factor + +When generating images, you might wish to bin the event X-Y sky coordinate system so +that one pixel of the output image represents a grouping of 'event pixels'. + +This binning could be motivated by increasing the signal-to-noise of each pixel or +reducing the size of the output image file, or your own scientific purpose. + +It is worth noting that the Xtend **event pixel** size dramatically subsamples the +point-spread-function (PSF) size induced by the X-ray optics, so an extreme binning +factor would be required to minimize cross-talk between image pixels. As such, this +should not be the primary motivation for your choice of image binning factor. + +```{code-cell} python +bin_factor = 4 +``` + +#### Running image generation + +We use... + ```{code-cell} python arg_combs = [ [ @@ -778,6 +803,7 @@ arg_combs = [ evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), *cur_bnds, + bin_factor, ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -790,6 +816,10 @@ with mp.Pool(NUM_CORES) as p: ### New XRISM-Xtend exposure maps +```{code-cell} python + +``` + ### New XRISM-Xtend light curves @@ -803,7 +833,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-11-24 +Updated On: 2025-11-25 +++ From 7c1d26347244b133729c106225c625754da3c9fe Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 15:42:35 -0500 Subject: [PATCH 015/221] Added first go at a wrapper function for XRISM Xtend exposure map generation to the getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 118 +++++++++++++++++- 1 file changed, 115 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0145cc85..0171b5da 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -236,9 +236,6 @@ def gen_xrism_xtend_image( # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): - # The processing/preparation stage of any X-ray telescope's data is the most - # likely to go wrong, and we use a Python try-except as an automated way to - # collect ObsIDs that had an issue during processing. out = hsp.extractor( filename=evt_file_chan_sel, imgfile=os.path.join(out_dir, im_out), @@ -251,6 +248,111 @@ def gen_xrism_xtend_image( rmtree(temp_work_dir) return out + + +def gen_xrism_xtend_expmap( + cur_obs_id: str, + cur_xtend_data_class: str, + out_dir: str, + gti_file: str, + extend_hk_file: str, + bad_pix_file: str, + pix_gti_file: str = "NONE", + im_bin: int = 1, + out_map_type: str = "EXPOSURE", + radial_delta: float = Quantity(20.0, "arcmin"), + num_phi_bin: int = 1, +): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend exposure map. + :param str cur_xtend_data_class: + :param str event_file: + :param str out_dir: The directory where output files should be written. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + + # Validity check on the passed data class + if cur_xtend_data_class[0] != "3": + raise ValueError( + f"The first digit of the Xtend data class ({cur_xtend_data_class}) " + "must be 3 for in-flight data." + ) + + # Make sure the radial_delta value is in arcminutes/is convertible to arcmins + # Also will assume that radial_delta is in arcmin if it is not a Quantity object + if not isinstance(radial_delta, Quantity): + radial_delta = Quantity(radial_delta, "arcmin") + elif radial_delta.unit.is_equivalent("arcmin"): + radial_delta = radial_delta.to("arcmin") + else: + raise ValueError( + f"The 'radial_delta' argument must be in arcmin or convertible to " + f"arcmin, not {radial_delta.unit}." + ) + + # Now we're certain of 'radial_delta's unit, we read out the value + radial_delta = radial_delta.value.astype(float) + + # Two variants of exposure map can be generated by the function we're about to + # call; the default is a map of the integrated exposure time for each pixel, and + # the second (not recommended by the documentation) is a flat-fielding map + ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" + + # Set up the output file name for the exposure map we're about to generate. + ex_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"attraddelta{radial_delta}arcmin-attphibin{num_phi_bin}-" + f"imbinfactor1-enALL-{ex_type}.fits" + ) + + # If the user wants to bin up the exposure map, we'll need to set up another + # output file name with the bin factor set to the input value (this variable + # is not used if the user does not want to bin the map) + binned_ex_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"attraddelta{radial_delta}arcmin-attphibin{num_phi_bin}-" + f"imbinfactor{im_bin}-enALL-{ex_type}.fits" + ) + + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + out = hsp.xaexpmap( + instrume="XTEND", + ehkfile=extend_hk_file, + gtifile=gti_file, + pixgtifile=pix_gti_file, + delta=radial_delta, + numphi=num_phi_bin, + outfile=os.path.join(out_dir, ex_out), + badimgfile=bad_pix_file, + noprompt=True, + clobber=True, + ) + + # If the user wants a spatially binned exposure map, we run the fimgbin task + if im_bin != 1: + rebin_out = hsp.fimgbin( + infile=os.path.join(out_dir, ex_out), + outfile=os.path.join(out_dir, binned_ex_out), + xbinsize=im_bin, + noprompt=True, + clobber=True, + ) + out = [out, rebin_out] + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return out ``` ### Constants @@ -816,6 +918,16 @@ with mp.Pool(NUM_CORES) as p: ### New XRISM-Xtend exposure maps +xaexpmap \ + instrume=XTEND \ + outfile="n132d_r2m_src.expo" \ + ehkfile="xa000126000.ehk" \ + gtifile="xa000126000xtd_p030000010_cl.evt" \ + badimgfile="xa000126000xtd_p030000010.bimg" \ + pixgtifile=NONE \ + outmaptype=EXPOSURE delta=20.0 numphi=1 stopsys=SKY \ + clobber=yes mode=hl + ```{code-cell} python ``` From 0490c22425d755a5fd2830a5d843f62391068bcc Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 16:13:41 -0500 Subject: [PATCH 016/221] Think I've worked out the kinks in the 'gen_xrism_xtend_expmap' function in the getting-started-xrism-xtend.md notebook. Seems to work fine, including spatial re-binning and creating different types of map. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0171b5da..47fea099 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -334,6 +334,7 @@ def gen_xrism_xtend_expmap( numphi=num_phi_bin, outfile=os.path.join(out_dir, ex_out), badimgfile=bad_pix_file, + outmaptype=out_map_type, noprompt=True, clobber=True, ) @@ -341,8 +342,8 @@ def gen_xrism_xtend_expmap( # If the user wants a spatially binned exposure map, we run the fimgbin task if im_bin != 1: rebin_out = hsp.fimgbin( - infile=os.path.join(out_dir, ex_out), - outfile=os.path.join(out_dir, binned_ex_out), + infile=os.path.join("..", ex_out), + outfile=os.path.join("..", binned_ex_out), xbinsize=im_bin, noprompt=True, clobber=True, @@ -928,6 +929,10 @@ xaexpmap \ outmaptype=EXPOSURE delta=20.0 numphi=1 stopsys=SKY \ clobber=yes mode=hl +```{code-cell} python +badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") +``` + ```{code-cell} python ``` From 642d9f93387988b6eb88305cd971aa12b78b59f3 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 16:53:15 -0500 Subject: [PATCH 017/221] Added the code that calls the exposure map wrapper function to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 76 ++++++++++++++++--- 1 file changed, 64 insertions(+), 12 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 47fea099..4de1228d 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -259,7 +259,6 @@ def gen_xrism_xtend_expmap( bad_pix_file: str, pix_gti_file: str = "NONE", im_bin: int = 1, - out_map_type: str = "EXPOSURE", radial_delta: float = Quantity(20.0, "arcmin"), num_phi_bin: int = 1, ): @@ -299,6 +298,8 @@ def gen_xrism_xtend_expmap( # Two variants of exposure map can be generated by the function we're about to # call; the default is a map of the integrated exposure time for each pixel, and # the second (not recommended by the documentation) is a flat-fielding map + # TODO REINSTATE WHEN WE HAVE A BETTER UNDERSTANDING OF POTENTIAL USER USES + out_map_type = "EXPOSURE" ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" # Set up the output file name for the exposure map we're about to generate. @@ -692,7 +693,10 @@ xtd_pipe_problem_ois Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` -Finally, we set up a template variable for the cleaned event lists we just created: +Finally, we set up some template variables for the various useful files output by the +XRISM-Xtend processing pipeline. These include the cleaned event lists we just created, + +cleaned event lists we just created: ***CONSIDER MORE COMMENTARY ON DATACLASS ETC. HERE*** @@ -702,7 +706,11 @@ Finally, we set up a template variable for the cleaned event lists we just creat ***sc == 'split counter'*** ```{code-cell} python +# Cleaned event list path template - obviously going to be useful later evt_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") + +# The path to the bad pixel map, useful for excluding dodgy pixels from data products +badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") ``` ### Identifying problem pixels @@ -917,24 +925,66 @@ with mp.Pool(NUM_CORES) as p: im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` +Once again we set up a template variable for output image file names: + +```{code-cell} python +im_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-imbinfactor{ibf}-en{lo}_{hi}keV-image.fits", +) +``` + ### New XRISM-Xtend exposure maps -xaexpmap \ - instrume=XTEND \ - outfile="n132d_r2m_src.expo" \ - ehkfile="xa000126000.ehk" \ - gtifile="xa000126000xtd_p030000010_cl.evt" \ - badimgfile="xa000126000xtd_p030000010.bimg" \ - pixgtifile=NONE \ - outmaptype=EXPOSURE delta=20.0 numphi=1 stopsys=SKY \ - clobber=yes mode=hl +Exposure maps... ```{code-cell} python -badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") +expmap_rad_delta = Quantity(20, "arcmin") +expmap_phi_bins = 1 ``` +gen_xrism_xtend_expmap(chosen_demo_obsid, + "32000010", + os.path.join(OUT_PATH, chosen_demo_obsid), + evt_path_temp.format(oi=chosen_demo_obsid, xdc="32000010", sc=0), + ehk_path_temp.format(oi=chosen_demo_obsid), + badpix_path_temp.format(oi=chosen_demo_obsid, xdc="32000010", sc=0), + im_bin=4, out_map_type='EFFICIENCY') + +***WHAT ABOUT THIS GTI?? - xa000128000xtd_mode.gti*** + ```{code-cell} python +arg_combs = [ + [ + oi, + dc, + os.path.join(OUT_PATH, oi), + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + ehk_path_temp.format(oi=oi), + badpix_path_temp.format(oi=oi, xdc=dc, sc=0), + "NONE", + expmap_rad_delta, + expmap_phi_bins, + bin_factor, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs +] +with mp.Pool(NUM_CORES) as p: + ex_result = p.starmap(gen_xrism_xtend_expmap, arg_combs) +``` + +Set up a template variable for output exposure map file names: + +```{code-cell} python +ex_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-attraddelta{rd}arcmin-" + "attphibin{npb}-imbinfactor{ibf}-enALL-expmap.fits", +) ``` ### New XRISM-Xtend light curves @@ -970,3 +1020,5 @@ HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/softwa ### References [XRISM GOF & SDC (2024) - _XRISM ABC GUIDE XTEND ENERGY-CHANNEL MAPPING_ [ACCESSED 25-NOV-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/Xtend_Data_Analysis.html#SECTION001043000000000000000) + +[](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html) From d023d3da93feeacc86caa8c6b69357c688d481ce Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 17:05:31 -0500 Subject: [PATCH 018/221] Added first pass at a wrapper function for XRISM light curve generation to the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 96 ++++++++++++++++++- 1 file changed, 95 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 4de1228d..422641f3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -70,7 +70,7 @@ from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.table import Table from astropy.time import Time -from astropy.units import Quantity +from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc # from typing import Tuple, Union @@ -355,6 +355,100 @@ def gen_xrism_xtend_expmap( rmtree(temp_work_dir) return out + + +def gen_xrism_xtend_lightcurve( + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + lo_en: Quantity, + hi_en: Quantity, + time_bin_size: Quantity = Quantity(100, "s"), +): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend light curve. + :param str cur_xtend_data_class: + :param str event_file: + :param str out_dir: The directory where output files should be written. + :param Quantity lo_en: Lower bound of the energy band within which we will + generate the light curve. + :param Quantity hi_en: Upper bound of the energy band within which we will + generate the light curve. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + + # Validity check on the passed data class + if cur_xtend_data_class[0] != "3": + raise ValueError( + f"The first digit of the Xtend data class ({cur_xtend_data_class}) " + "must be 3 for in-flight data." + ) + + # Check the units of the passed time bin size - also if the passed value is + # a float or integer, we'll assume it is in seconds + if not isinstance(time_bin_size, Quantity): + time_bin_size = Quantity(time_bin_size, "s") + elif not time_bin_size.unit.is_equivalent("s"): + raise UnitConversionError( + f"The 'time_bin_size' argument ({time_bin_size}) " + "must be an astropy Quantity that is convertible " + "to seconds." + ) + + # Convert the time bin size to seconds and convert it to a simple integer/float + time_bin_size = time_bin_size.to("s").value + + # Make sure the lower and upper energy limits make sense + if lo_en > hi_en: + raise ValueError( + "The lower energy limit must be less than or equal to the upper " + "energy limit." + ) + else: + lo_en_val = lo_en.to("keV").value + hi_en_val = hi_en.to("keV").value + + # Convert the energy limits to channel limits, rounding down and up to the nearest + # integer channel for the lower and upper bounds respectively. + lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + + # Create modified input event list file path, where we use the just-calculated + # PI channel limits to subset the events + evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" + + # Set up the output file name for the image we're about to generate. + lc_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"en{lo_en_val}_{hi_en_val}keV-tb{time_bin_size}s-lightcurve.fits" + ) + + # Create a temporary working directory + temp_work_dir = os.path.join( + out_dir, "lightcurve_extractor_{}".format(randint(0, int(1e8))) + ) + os.makedirs(temp_work_dir) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + out = hsp.extractor( + filename=evt_file_chan_sel, + fitsbinlc=os.path.join(out_dir, lc_out), + binlc=time_bin_size, + noprompt=True, + clobber=True, + ) + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return out ``` ### Constants From 424e82c4bf7f74ebded3bdc139c7bbaffc9eb127 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 17:11:06 -0500 Subject: [PATCH 019/221] Added the code cells to call the new light curve generating wrapper function in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 29 ++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 422641f3..dc99c816 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -439,7 +439,7 @@ def gen_xrism_xtend_lightcurve( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.extractor( filename=evt_file_chan_sel, - fitsbinlc=os.path.join(out_dir, lc_out), + fitsbinlc=os.path.join("..", lc_out), binlc=time_bin_size, noprompt=True, clobber=True, @@ -1083,6 +1083,33 @@ ex_path_temp = os.path.join( ### New XRISM-Xtend light curves +```{code-cell} python +lc_time_bin = Quantity(100, "s") +``` + +```{code-cell} python +# Defining the various energy bounds we want to make light curves for +xtd_lc_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") +``` + +```{code-cell} python +arg_combs = [ + [ + oi, + dc, + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + os.path.join(OUT_PATH, oi), + *cur_bnds, + lc_time_bin, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs + for cur_bnds in xtd_lc_en_bounds +] + +with mp.Pool(NUM_CORES) as p: + lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) +``` ## 4. Generating new XRISM-Xtend spectra and supporting files From a54cce0bdb1413a47c1c811cfbddd3dded801ddb Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 17:32:09 -0500 Subject: [PATCH 020/221] Allow a 'lc_bin_thresh' argument to be passed to gen_xrism_xtend_lightcurve, to control which light curve bins are thrown out because of fractional exposure. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index dc99c816..21a53760 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -364,7 +364,8 @@ def gen_xrism_xtend_lightcurve( out_dir: str, lo_en: Quantity, hi_en: Quantity, - time_bin_size: Quantity = Quantity(100, "s"), + time_bin_size: Quantity = Quantity(200, "s"), + lc_bin_thresh: float = 0.0, ): """ @@ -424,7 +425,8 @@ def gen_xrism_xtend_lightcurve( # Set up the output file name for the image we're about to generate. lc_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"en{lo_en_val}_{hi_en_val}keV-tb{time_bin_size}s-lightcurve.fits" + f"en{lo_en_val}_{hi_en_val}keV-expthresh{lc_bin_thresh}-tb{time_bin_size}s-" + f"lightcurve.fits" ) # Create a temporary working directory @@ -441,6 +443,7 @@ def gen_xrism_xtend_lightcurve( filename=evt_file_chan_sel, fitsbinlc=os.path.join("..", lc_out), binlc=time_bin_size, + lcthresh=lc_bin_thresh, noprompt=True, clobber=True, ) @@ -1000,6 +1003,8 @@ bin_factor = 4 We use... +***NEED TO APPLY GTIS TO IMAGE GENERATION AS WELL*** + ```{code-cell} python arg_combs = [ [ @@ -1058,9 +1063,9 @@ arg_combs = [ ehk_path_temp.format(oi=oi), badpix_path_temp.format(oi=oi, xdc=dc, sc=0), "NONE", + bin_factor, expmap_rad_delta, expmap_phi_bins, - bin_factor, ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -1084,12 +1089,12 @@ ex_path_temp = os.path.join( ### New XRISM-Xtend light curves ```{code-cell} python -lc_time_bin = Quantity(100, "s") +lc_time_bin = Quantity(200, "s") ``` ```{code-cell} python # Defining the various energy bounds we want to make light curves for -xtd_lc_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") +xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") ``` ```{code-cell} python From 6bd7885c3e527992a0ddecf19a50766ab46f06fd Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 09:35:17 -0500 Subject: [PATCH 021/221] Shifted the skeleton of the light curve generation subsection to the second section of product generation (with spectra, as it will be source specific). Added a template variable for light curve outputs. All in getting-started-xrism-xtend.md, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 30 +++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 21a53760..f3625abe 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -816,7 +816,7 @@ badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") ``` -## 3. Generating new XRISM-Xtend images, exposure maps, and light curves +## 3. Generating new XRISM-Xtend images and exposure maps ### Converting energy bounds to channel bounds @@ -1086,8 +1086,17 @@ ex_path_temp = os.path.join( ) ``` +## 4. Generating new XRISM-Xtend spectra and light curves + ### New XRISM-Xtend light curves + +***WILL NEED TO ADD REGION FILE SUPPORT TO THE LIGHT CURVE GENERATION WRAPPER FUNCTION*** + +***PERHAPS I FINALLY IMPLEMENT MY IDEAS ABOUT HASHING THE REGION FILES TO GENERATE UNIQUE OUTPUT FILE NAMES*** + +***ALSO NEED TO SOURCE THE REGION FILES THAT DESCRIBE WHERE THE CALIBRATION SOURCES ARE*** + ```{code-cell} python lc_time_bin = Quantity(200, "s") ``` @@ -1097,6 +1106,8 @@ lc_time_bin = Quantity(200, "s") xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") ``` +***NEEEEEEEED GTI*** + ```{code-cell} python arg_combs = [ [ @@ -1116,9 +1127,24 @@ with mp.Pool(NUM_CORES) as p: lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) ``` -## 4. Generating new XRISM-Xtend spectra and supporting files +Create a template variable for output light curve file names: + +```{code-cell} python +lc_path_temp = ( + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}" + "-tb{tb}s-lightcurve.fits" +) +``` + +### Preparing to generate new XRISM-Xtend spectra + +```{code-cell} python + +``` +### +## 5. Fitting spectral models to XRISM-Xtend spectra ## About this notebook From 1f7256e3f9e4e1f851091f5211998af31bd59f46 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 10:21:08 -0500 Subject: [PATCH 022/221] Added some commentary on setting up source extraction regions to the getting-started-xrism-xtend.md notebook. Also indicated that N132D can be treated as a point source here, and extended source analysis will be handled in another notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 39 ++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f3625abe..a3ade618 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -860,7 +860,7 @@ We will be creating new RMFs as part of the generation of XRISM-Xtend spectra in next section. For our current purpose, however, it is acceptable to use the RMFs that were included in the XRISM-Xtend archive we downloaded earlier. -The archived RMFs are generated for the entire Xtend FoV, rather than for the CCDs +The archived RMFs are generated for the entire Xtend field-of-view (FoV), rather than for the CCDs our particular target fall on, but practically speaking, that doesn't make a significant difference. @@ -1088,6 +1088,43 @@ ex_path_temp = os.path.join( ## 4. Generating new XRISM-Xtend spectra and light curves +In this section we will demonstrate how to generate source-specific data products from +XRISM-Xtend observations; light curves and spectra (along with supporting files like +RMFs and Ancillary Response Files, or ARFs). + +### Setting up data product source and background extraction regions + +Rather than extracting spectra and light curves for the entire XRISM-Xtend FoV, +*which is how the quick-look spectra and light curves contained in the archive are made*, we +want to control exactly where we are taking events from. + +That way we can focus on the particular source(s) of interest present in the +XRISM-Xtend observations we are using. + +The size, shape, placement, and number of source extraction regions you need to use for +your work will depend heavily on your science case and the type of astrophysical +source you're analyzing. + +You will find that point sources are considerably easier to deal with, as you can +generally learn all you need from a single spectrum encompassing the entire source +emission region. + +Indeed, trying to extract spectra from different spatial regions of a point source (even +if the emission *appears* extended in XRISM-Xtend images) is **not valid**, as the +apparently extended emission is caused by the PSF of the telescope optics. + +The 'blurring' of the observed emission events by the PSF is one of the reasons that +extended sources are much harder to analyze than point sources. For example, you +might want to extract spectra from a series of annular bins centered on +your extended source to see how a particular spectral property changes in different +parts of the object. + +Unfortunately, due to the PSF, each annulus will be contaminated by (and be +*contaminating* in turn) events from other annuli, scattered there by the telescope PSF - this +effect is sometimes referred to as **cross-talk** or **spatial-spectral mixing (SSM)**. Accounting +for this effect is complicated and time-consuming, so our demonstration will focus on a point source, and +extended sources will be discussed in another notebook. + ### New XRISM-Xtend light curves From 913409dec374f876d62b4009f56b8771f1a4a3ef Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 10:48:43 -0500 Subject: [PATCH 023/221] Added code cell that should write the RA-DEC version of the source region file to getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 33 +++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a3ade618..e6792aa6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1092,8 +1092,6 @@ In this section we will demonstrate how to generate source-specific data product XRISM-Xtend observations; light curves and spectra (along with supporting files like RMFs and Ancillary Response Files, or ARFs). -### Setting up data product source and background extraction regions - Rather than extracting spectra and light curves for the entire XRISM-Xtend FoV, *which is how the quick-look spectra and light curves contained in the archive are made*, we want to control exactly where we are taking events from. @@ -1125,6 +1123,37 @@ effect is sometimes referred to as **cross-talk** or **spatial-spectral mixing ( for this effect is complicated and time-consuming, so our demonstration will focus on a point source, and extended sources will be discussed in another notebook. +### Setting up data product source and background extraction regions + +There are different ways to define.... + +SOURCE: +radius of 2 arcmin + +BACKGROUND: +81.1932474 +-69.5073738 +radius of 4 arcmin + +```{code-cell} python +src_reg_rad = Quantity(2, "arcmin") + +radec_src_reg_path = os.path.join(OUT_PATH, "radec_src.reg") + +with open(radec_src_reg_path, "w") as src_rego: + src_radec_str = """ + # Region file format: DS9 version 4.1 + global color=green + icrs + circle({ra},{dec},{rad}d) + """.format( + ra=src_coord.ra.value, + dec=src_coord.dec.value, + rad=src_reg_rad.to("arcmin").value, + ) + src_rego.write(src_radec_str) +``` + ### New XRISM-Xtend light curves From feb66b702c23bdb602152d307e4f85494a5053b5 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 11:07:26 -0500 Subject: [PATCH 024/221] The getting-started-xrism-xtend.md notebooks now successfully generates source and background RA-DEC region files. They will be converted to sky coords for each observation, and possibly overlaid on every image from each observation. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 60 +++++++++++++------ 1 file changed, 42 insertions(+), 18 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index e6792aa6..f07e4169 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1127,33 +1127,57 @@ extended sources will be discussed in another notebook. There are different ways to define.... -SOURCE: -radius of 2 arcmin - -BACKGROUND: -81.1932474 --69.5073738 -radius of 4 arcmin - ```{code-cell} python +# The radius of the source extraction region src_reg_rad = Quantity(2, "arcmin") +# Where to write the new region file radec_src_reg_path = os.path.join(OUT_PATH, "radec_src.reg") +# Defining each line of the region file as an element of a list, then joining +# them all into a string with newlines +src_radec_lines = [ + "# Region file format: DS9 version 4.1", + "global color=green", + "icrs", + "circle({ra},{dec},{rad}d)", +] +src_radec_str = "\n".join(src_radec_lines).format( + ra=src_coord.ra.value, dec=src_coord.dec.value, rad=src_reg_rad.to("deg").value +) + with open(radec_src_reg_path, "w") as src_rego: - src_radec_str = """ - # Region file format: DS9 version 4.1 - global color=green - icrs - circle({ra},{dec},{rad}d) - """.format( - ra=src_coord.ra.value, - dec=src_coord.dec.value, - rad=src_reg_rad.to("arcmin").value, - ) src_rego.write(src_radec_str) ``` +We do the same to define a region from which to extract a background spectrum: + +```{code-cell} python +# The central coordinate of the background region +back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") + +# The radius of the background region +back_reg_rad = Quantity(3, "arcmin") + +# Where to write the new region file +radec_back_reg_path = os.path.join(OUT_PATH, "radec_back.reg") + +# Defining each line of the region file as an element of a list, then joining +# them all into a string with newlines +back_radec_lines = [ + "# Region file format: DS9 version 4.1", + "global color=green", + "icrs", + "circle({ra},{dec},{rad}d)", +] +back_radec_str = "\n".join(back_radec_lines).format( + ra=back_coord.ra.value, dec=back_coord.dec.value, rad=back_reg_rad.to("deg").value +) + +with open(radec_back_reg_path, "w") as back_rego: + back_rego.write(back_radec_str) +``` + ### New XRISM-Xtend light curves From 7fc83490b2a89c9f2d3da22fc4ea332ee7b008fc Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 11:22:47 -0500 Subject: [PATCH 025/221] Added source and background region support to the 'gen_xrism_xtend_lightcurve' wrapper function for XRISM-Xtend light curve generation in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 27 +++++++++++++++---- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f07e4169..b1dfef17 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -362,8 +362,10 @@ def gen_xrism_xtend_lightcurve( cur_xtend_data_class: str, event_file: str, out_dir: str, - lo_en: Quantity, - hi_en: Quantity, + src_reg_file: str, + back_reg_file: str, + lo_en: Quantity = Quantity(0.6, "keV"), + hi_en: Quantity = Quantity(13, "keV"), time_bin_size: Quantity = Quantity(200, "s"), lc_bin_thresh: float = 0.0, ): @@ -422,12 +424,15 @@ def gen_xrism_xtend_lightcurve( # PI channel limits to subset the events evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" - # Set up the output file name for the image we're about to generate. + # Set up the output file name for the light curve we're about to generate. lc_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" f"en{lo_en_val}_{hi_en_val}keV-expthresh{lc_bin_thresh}-tb{time_bin_size}s-" f"lightcurve.fits" ) + # The same file name, but with 'lightcurve' changed to 'back-lightcurve', for the + # background light curve. + lc_back_out = lc_out.replace("lightcurve", "back-lightcurve") # Create a temporary working directory temp_work_dir = os.path.join( @@ -439,11 +444,23 @@ def gen_xrism_xtend_lightcurve( # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): - out = hsp.extractor( + src_out = hsp.extractor( filename=evt_file_chan_sel, fitsbinlc=os.path.join("..", lc_out), binlc=time_bin_size, lcthresh=lc_bin_thresh, + regionfile=src_reg_file, + noprompt=True, + clobber=True, + ) + + # Now for the background light curve + back_out = hsp.extractor( + filename=evt_file_chan_sel, + fitsbinlc=os.path.join("..", lc_back_out), + binlc=time_bin_size, + lcthresh=lc_bin_thresh, + regionfile=back_reg_file, noprompt=True, clobber=True, ) @@ -451,7 +468,7 @@ def gen_xrism_xtend_lightcurve( # Make sure to remove the temporary directory rmtree(temp_work_dir) - return out + return [src_out, back_out] ``` ### Constants From e82c74dc9d56b342f755f684ae321b984a17a142 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 12:50:44 -0500 Subject: [PATCH 026/221] Added xcolf and ycolf arguments to the extractor call to generate images from XRISM-Xtend data in the getting-started-xrism-xtend.md notebook. If those arguments are missing it seems that no sky-pixel WCS is written to the resulting images, which is obviously a problem. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b1dfef17..e208159a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -72,6 +72,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc +from xga.products import Image # from typing import Tuple, Union # from warnings import warn @@ -242,6 +243,8 @@ def gen_xrism_xtend_image( noprompt=True, clobber=True, binf=im_bin, + xcolf="X", + ycolf="Y", ) # Make sure to remove the temporary directory @@ -975,7 +978,7 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ```{code-cell} python # Defining the energy bounds we want images within -xtd_im_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") +xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV") ``` Converting those energy bounds to channel bounds is straightforward, we simply divide @@ -1013,7 +1016,7 @@ factor would be required to minimize cross-talk between image pixels. As such, t should not be the primary motivation for your choice of image binning factor. ```{code-cell} python -bin_factor = 4 +bin_factors = [1, 4] ``` #### Running image generation @@ -1030,11 +1033,12 @@ arg_combs = [ evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), *cur_bnds, - bin_factor, + cur_bf, ] for oi, dcs in rel_dataclasses.items() for dc in dcs for cur_bnds in xtd_im_en_bounds + for cur_bf in bin_factors ] with mp.Pool(NUM_CORES) as p: @@ -1080,12 +1084,13 @@ arg_combs = [ ehk_path_temp.format(oi=oi), badpix_path_temp.format(oi=oi, xdc=dc, sc=0), "NONE", - bin_factor, + cur_bf, expmap_rad_delta, expmap_phi_bins, ] for oi, dcs in rel_dataclasses.items() for dc in dcs + for cur_bf in bin_factors ] with mp.Pool(NUM_CORES) as p: @@ -1195,6 +1200,12 @@ with open(radec_back_reg_path, "w") as back_rego: back_rego.write(back_radec_str) ``` +Examining... + +```{code-cell} python + +``` + ### New XRISM-Xtend light curves @@ -1246,7 +1257,14 @@ lc_path_temp = ( ### Preparing to generate new XRISM-Xtend spectra ```{code-cell} python +chos_im_en = xtd_im_en_bounds[0].to("keV") +for oi, cur_dcs in rel_dataclasses.items(): + for dc in cur_dcs: + cur_im_path = im_path_temp.format( + oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value + ) + cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) ``` ### From a76f22e1f42de7a6a62cf14a3236a0832f67f3ab Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 14:26:06 -0500 Subject: [PATCH 027/221] Regions should now be included and properly handled in the getting-started-xrism-xtend.md notebook. Implemented for the light curve generation step at the moment. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 103 ++++++++++-------- 1 file changed, 58 insertions(+), 45 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index e208159a..097b5a36 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -72,6 +72,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc +from regions import CircleSkyRegion from xga.products import Image # from typing import Tuple, Union @@ -453,6 +454,8 @@ def gen_xrism_xtend_lightcurve( binlc=time_bin_size, lcthresh=lc_bin_thresh, regionfile=src_reg_file, + xcolf="X", + ycolf="Y", noprompt=True, clobber=True, ) @@ -464,6 +467,8 @@ def gen_xrism_xtend_lightcurve( binlc=time_bin_size, lcthresh=lc_bin_thresh, regionfile=back_reg_file, + xcolf="X", + ycolf="Y", noprompt=True, clobber=True, ) @@ -483,7 +488,7 @@ jupyter: source_hidden: true --- # The name of the source we're examining in this demonstration -SRC_NAME = "LMC N132D" +SRC_NAME = "LMCN132D" # SRC_NAME = "NGC4151" # SRC_NAME = "AX J1910.7+0917" @@ -1149,69 +1154,84 @@ extended sources will be discussed in another notebook. There are different ways to define.... +#### General RA-DEC region files + ```{code-cell} python +# Where to write the new region file +radec_src_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_src.reg") + # The radius of the source extraction region src_reg_rad = Quantity(2, "arcmin") -# Where to write the new region file -radec_src_reg_path = os.path.join(OUT_PATH, "radec_src.reg") - -# Defining each line of the region file as an element of a list, then joining -# them all into a string with newlines -src_radec_lines = [ - "# Region file format: DS9 version 4.1", - "global color=green", - "icrs", - "circle({ra},{dec},{rad}d)", -] -src_radec_str = "\n".join(src_radec_lines).format( - ra=src_coord.ra.value, dec=src_coord.dec.value, rad=src_reg_rad.to("deg").value -) +# Setting up a 'regions' module circular sky region instance +src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) -with open(radec_src_reg_path, "w") as src_rego: - src_rego.write(src_radec_str) +# Write the source region to a region file +src_reg.write(radec_src_reg_path, format="ds9") ``` We do the same to define a region from which to extract a background spectrum: ```{code-cell} python +# Where to write the new region file +radec_back_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_back.reg") + # The central coordinate of the background region back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") # The radius of the background region back_reg_rad = Quantity(3, "arcmin") -# Where to write the new region file -radec_back_reg_path = os.path.join(OUT_PATH, "radec_back.reg") - -# Defining each line of the region file as an element of a list, then joining -# them all into a string with newlines -back_radec_lines = [ - "# Region file format: DS9 version 4.1", - "global color=green", - "icrs", - "circle({ra},{dec},{rad}d)", -] -back_radec_str = "\n".join(back_radec_lines).format( - ra=back_coord.ra.value, dec=back_coord.dec.value, rad=back_reg_rad.to("deg").value -) +# Setting up a 'regions' module circular sky region instance for the background region +back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) -with open(radec_back_reg_path, "w") as back_rego: - back_rego.write(back_radec_str) +# Once again writing the region to a region file as well +back_reg.write(radec_back_reg_path, format="ds9") ``` +#### Visualizing the source and background extraction regions on XRISM-Xtend images + Examining... ```{code-cell} python +chos_im_en = xtd_im_en_bounds[0].to("keV") +oi_skypix_wcs = {} +for oi, cur_dcs in rel_dataclasses.items(): + for dc in cur_dcs: + cur_im_path = im_path_temp.format( + oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value + ) + cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) + cur_im.regions = [src_reg, back_reg] + cur_im.view(src_coord_quant, zoom_in=True, view_regions=True) + + oi_skypix_wcs.setdefault(oi, cur_im.radec_wcs) ``` -### New XRISM-Xtend light curves +#### Observation specific sky-pixel coordinate region files + +```{code-cell} python +obs_src_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_src.reg") +obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_back.reg") +for oi in rel_obsids: + src_reg.to_pixel(oi_skypix_wcs[oi]).write( + obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9" + ) + back_reg.to_pixel(oi_skypix_wcs[oi]).write( + obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9" + ) +``` -***WILL NEED TO ADD REGION FILE SUPPORT TO THE LIGHT CURVE GENERATION WRAPPER FUNCTION*** +```{tip} +Events from different data classes of **the same observation** share a common sky-pixel +coordinate system, so sky-pixel region files for one are also valid for the other. +However, different data classes represent different pairs of Xtend CCDs, so there is +no shared sky coverage. +``` -***PERHAPS I FINALLY IMPLEMENT MY IDEAS ABOUT HASHING THE REGION FILES TO GENERATE UNIQUE OUTPUT FILE NAMES*** +### New XRISM-Xtend light curves ***ALSO NEED TO SOURCE THE REGION FILES THAT DESCRIBE WHERE THE CALIBRATION SOURCES ARE*** @@ -1233,6 +1253,8 @@ arg_combs = [ dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), + obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), + obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), *cur_bnds, lc_time_bin, ] @@ -1256,16 +1278,7 @@ lc_path_temp = ( ### Preparing to generate new XRISM-Xtend spectra -```{code-cell} python -chos_im_en = xtd_im_en_bounds[0].to("keV") -for oi, cur_dcs in rel_dataclasses.items(): - for dc in cur_dcs: - cur_im_path = im_path_temp.format( - oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value - ) - cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) -``` ### From aee45ae4cc0ea2a8fd885b70dd766cb0e05d217a Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 15:04:25 -0500 Subject: [PATCH 028/221] Added first pass at a function to generate XRISM Xtend spectra to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 108 +++++++++++++++++- 1 file changed, 105 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 097b5a36..73edfcd5 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -49,7 +49,7 @@ XRISM is... ### Runtime -As of 22nd November 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 26nd November 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -477,6 +477,106 @@ def gen_xrism_xtend_lightcurve( rmtree(temp_work_dir) return [src_out, back_out] + + +def gen_xrism_xtend_spectra( + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + src_reg_file: str, + back_reg_file: str, + lo_en: Quantity = Quantity(0.6, "keV"), + hi_en: Quantity = Quantity(13, "keV"), +): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend spectrum. + :param str cur_xtend_data_class: + :param str event_file: + :param str out_dir: The directory where output files should be written. + :param Quantity lo_en: Lower bound of the energy band within which we will + generate the spectrum. + :param Quantity hi_en: Upper bound of the energy band within which we will + generate the spectrum. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + + # Validity check on the passed data class + if cur_xtend_data_class[0] != "3": + raise ValueError( + f"The first digit of the Xtend data class ({cur_xtend_data_class}) " + "must be 3 for in-flight data." + ) + + # Make sure the lower and upper energy limits make sense + if lo_en > hi_en: + raise ValueError( + "The lower energy limit must be less than or equal to the upper " + "energy limit." + ) + else: + lo_en_val = lo_en.to("keV").value + hi_en_val = hi_en.to("keV").value + + # Convert the energy limits to channel limits, rounding down and up to the nearest + # integer channel for the lower and upper bounds respectively. + lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + + # Create modified input event list file path, where we use the just-calculated + # PI channel limits to subset the events + evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" + + # Set up the output file name for the light curve we're about to generate. + sp_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"en{lo_en_val}_{hi_en_val}keV-" + f"spectrum.fits" + ) + # The same file name, but with 'spectrum' changed to 'back-spectrum', for the + # background light curve. + sp_back_out = sp_out.replace("spectrum", "back-spectrum") + + # Create a temporary working directory + temp_work_dir = os.path.join( + out_dir, "spec_extractor_{}".format(randint(0, int(1e8))) + ) + os.makedirs(temp_work_dir) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + src_out = hsp.extractor( + filename=evt_file_chan_sel, + phafile=os.path.join("..", sp_out), + regionfile=src_reg_file, + xcolf="X", + ycolf="Y", + ecol="PI", + noprompt=True, + clobber=True, + ) + + # Now for the background light curve + back_out = hsp.extractor( + filename=evt_file_chan_sel, + phafile=os.path.join("..", sp_back_out), + regionfile=back_reg_file, + xcolf="X", + ycolf="Y", + ecol="PI", + noprompt=True, + clobber=True, + ) + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return [src_out, back_out] ``` ### Constants @@ -1276,7 +1376,9 @@ lc_path_temp = ( ) ``` -### Preparing to generate new XRISM-Xtend spectra +### Generate new XRISM-Xtend spectra + + @@ -1290,7 +1392,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-11-25 +Updated On: 2025-11-26 +++ From b43c00ccfdfbddf89f99cd620874adc2e99fad81 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 15:24:50 -0500 Subject: [PATCH 029/221] Started to add spec generation and grouping sections and code to the getting-started-xrism-xtend.md notebook. For issue #128. --- .../xrism/getting-started-xrism-xtend.md | 100 +++++++++++++++++- 1 file changed, 97 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 73edfcd5..91389fc8 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -479,7 +479,7 @@ def gen_xrism_xtend_lightcurve( return [src_out, back_out] -def gen_xrism_xtend_spectra( +def gen_xrism_xtend_spectrum( cur_obs_id: str, cur_xtend_data_class: str, event_file: str, @@ -1367,20 +1367,114 @@ with mp.Pool(NUM_CORES) as p: lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) ``` -Create a template variable for output light curve file names: +Create template variables for source and background light curves: ```{code-cell} python lc_path_temp = ( "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}" "-tb{tb}s-lightcurve.fits" ) + +back_lc_path_temp = ( + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV" + "-expthresh{lct}-tb{tb}s-back-lightcurve.fits" +) +``` + +### New XRISM-Xtend spectra and supporting files + +```{code-cell} python + +``` + +#### Generating the spectral files + +```{code-cell} python +spec_lo_en = Quantity(0.6, "keV") +spec_hi_en = Quantity(13, "keV") +``` + +```{code-cell} python +arg_combs = [ + [ + oi, + dc, + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + os.path.join(OUT_PATH, oi), + obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), + obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + spec_lo_en, + spec_hi_en, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs +] + +with mp.Pool(NUM_CORES) as p: + sp_result = p.starmap(gen_xrism_xtend_spectrum, arg_combs) +``` + +Create template variables for source and background spectrum files: + +```{code-cell} python +sp_path_temp = "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-spectrum.fits" + +back_sp_path_temp = ( + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-back-spectrum.fits" +) ``` -### Generate new XRISM-Xtend spectra +#### Grouping our new spectra +We will group the spectra we just generated. Grouping essentially combines +spectral channels until some minimum quality threshold is reached; in this case a +minimum of one count per grouped channel. We use the HEASoft `ftgrouppha` tool to do +this, once again through HEASoftPy. +First, we set up the grouping criteria and a template variable for the name of the +output grouped spectral files: +*** REMIND MYSELF WHETHER GROUPING FROM SOURCE SPEC IS AUTOMATICALLY APPLIED TO BACK SPEC IN XSPEC? *** +```{code-cell} python +spec_group_type = "min" +spec_group_scale = 1 + +grp_sp_path_temp = sp_path_temp.replace("-spectrum", "-{gt}grp{gs}-spectrum") +``` + +Now we run the grouping tool - though this time we do not parallelize the task, as +the grouping process is very fast, and we wish to demonstrate how you use a HEASoftPy +function directly. Though remember to look at the Global Setup section of this notebook +to see how we call HEASoftPy tools in the wrapper functions used to parallelize those +tasks. + +If you are dealing with significantly more observations than we use for this +demonstration, we do recommend that you parallelize this grouping step as we have +the other processing steps in this notebook. + +```{code-cell} python +for oi, dcs in rel_dataclasses.items(): + for cur_dc in dcs: + cur_spec = sp_path_temp.format( + oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + ) + cur_grp_spec = grp_sp_path_temp.format( + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + gt=spec_group_type, + gs=spec_group_scale, + ) + + hsp.ftgrouppha( + infile=cur_spec, + outfile=cur_grp_spec, + grouptype=spec_group_type, + groupscale=spec_group_scale, + ) +``` ### From bef26f097682fa9b6a5cd561aba7a91d8737b7e4 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 16:26:44 -0500 Subject: [PATCH 030/221] Added a subsection to the part of getting-started-xrism-xtend.md that generates new spectra which calculates correct values of BACKSCAL for the source and background spectra. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 83 ++++++++++++++++--- 1 file changed, 73 insertions(+), 10 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 91389fc8..14e9d3e6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1309,6 +1309,14 @@ for oi, cur_dcs in rel_dataclasses.items(): oi_skypix_wcs.setdefault(oi, cur_im.radec_wcs) ``` +#### Excluding the XRISM-Xtend calibration sources + +```{code-cell} python +detpix_xtend_calib_reg_path = os.path.join( + os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" +) +``` + #### Observation specific sky-pixel coordinate region files ```{code-cell} python @@ -1370,14 +1378,18 @@ with mp.Pool(NUM_CORES) as p: Create template variables for source and background light curves: ```{code-cell} python -lc_path_temp = ( - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}" - "-tb{tb}s-lightcurve.fits" +lc_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" + "-lightcurve.fits", ) -back_lc_path_temp = ( - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV" - "-expthresh{lct}-tb{tb}s-back-lightcurve.fits" +back_lc_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" + "-back-lightcurve.fits", ) ``` @@ -1417,13 +1429,60 @@ with mp.Pool(NUM_CORES) as p: Create template variables for source and background spectrum files: ```{code-cell} python -sp_path_temp = "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-spectrum.fits" +sp_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-spectrum.fits", +) -back_sp_path_temp = ( - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-back-spectrum.fits" +back_sp_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-back-spectrum.fits", ) ``` +#### Calculating the 'BACKSCAL' value for new XRISM-Xtend spectra + +***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** + +```{code-cell} python +for oi, dcs in rel_dataclasses.items(): + for cur_dc in dcs: + # Set up the path to input source and background spectra + cur_spec = sp_path_temp.format( + oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + ) + cur_bspec = back_sp_path_temp.format( + oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + ) + + # Also need to pass an exposure map, so set up a path to that + cur_ex = ex_path_temp.format( + oi=oi, + xdc=cur_dc, + rd=expmap_rad_delta.to("arcmin").value, + npb=expmap_phi_bins, + ibf=1, + ) + + # Calculate the BACKSCAL keyword, first for the source spectrum + hsp.ahbackscal( + infile=cur_spec, + regfile=obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), + expfile=cur_ex, + logfile="NONE", + ) + + # Then for the background spectrum + hsp.ahbackscal( + infile=cur_bspec, + regfile=obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + expfile=cur_ex, + logfile="NONE", + ) +``` + #### Grouping our new spectra We will group the spectra we just generated. Grouping essentially combines @@ -1456,6 +1515,7 @@ the other processing steps in this notebook. ```{code-cell} python for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: + # Set up relevant paths to the input and output spectrum cur_spec = sp_path_temp.format( oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value ) @@ -1476,7 +1536,10 @@ for oi, dcs in rel_dataclasses.items(): ) ``` -### +#### Generating XRISM-Xtend RMFs + +#### Generating XRISM-Xtend ARFs + ## 5. Fitting spectral models to XRISM-Xtend spectra From a74bc51cc9b67357dfc55a8f57c28399a576855b Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 16:55:23 -0500 Subject: [PATCH 031/221] Added first pass at the wrapper function to generate RMFs for XRISM Xtend in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 58 ++++++++++++++++++- 1 file changed, 55 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 14e9d3e6..f20cb04e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -577,6 +577,37 @@ def gen_xrism_xtend_spectrum( rmtree(temp_work_dir) return [src_out, back_out] + + +def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend RMF. + :param str out_dir: The directory where output files should be written. + """ + + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xtdrmf_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) + + # Set up the RMF file name by cannibalising the name of the spectrum file + rmf_out = os.path.basename(spec_file).split("-en")[0] + ".rmf" + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + out = hsp.xtdrmf( + infile=spec_file, + outfile=os.path.join("..", rmf_out), + noprompt=True, + clobber=True, + ) + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return out ``` ### Constants @@ -1311,6 +1342,10 @@ for oi, cur_dcs in rel_dataclasses.items(): #### Excluding the XRISM-Xtend calibration sources +***HAVE TO LOAD THE CALIBRATION SOURCE REGIONS, CONVERT TO SKY PIX SYSTEM, AND EXCLUDE THEM FROM THE EXTRACTION REGION DEFINITIONS*** + +***THIS FILE IS SET UP FOR EXCLUSION (I.E. WITH - IN FRONT OF REGIONS) SO THE REGIONS MODULE WON'T READ THEM IN, AT LEAST BY DEFAULT*** + ```{code-cell} python detpix_xtend_calib_reg_path = os.path.join( os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" @@ -1341,8 +1376,6 @@ no shared sky coverage. ### New XRISM-Xtend light curves -***ALSO NEED TO SOURCE THE REGION FILES THAT DESCRIBE WHERE THE CALIBRATION SOURCES ARE*** - ```{code-cell} python lc_time_bin = Quantity(200, "s") ``` @@ -1442,7 +1475,7 @@ back_sp_path_temp = os.path.join( ) ``` -#### Calculating the 'BACKSCAL' value for new XRISM-Xtend spectra +#### Calculating 'BACKSCAL' for new XRISM-Xtend spectra ***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** @@ -1538,6 +1571,25 @@ for oi, dcs in rel_dataclasses.items(): #### Generating XRISM-Xtend RMFs +***THIS IS ALSO GOING TO FALL OVER IN PART BECAUSE CAN'T EXTRACT SPECTRUM FROM REGIONS NOT ON THE 31100010 DATACLASS OBSERVATION OF 000128000*** + +```{code-cell} python +arg_combs = [ + [ + oi, + sp_path_temp.format( + oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + ), + os.path.join(OUT_PATH, oi), + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs +] + +with mp.Pool(NUM_CORES) as p: + rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) +``` + #### Generating XRISM-Xtend ARFs From d071486752b0708d330e85ee11b6e2e041c06acd Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 1 Dec 2025 12:24:16 -0500 Subject: [PATCH 032/221] Getting ARF generation closer to being functional in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 170 +++++++++++++++++- 1 file changed, 161 insertions(+), 9 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f20cb04e..507e523e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -484,12 +484,15 @@ def gen_xrism_xtend_spectrum( cur_xtend_data_class: str, event_file: str, out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, src_reg_file: str, back_reg_file: str, lo_en: Quantity = Quantity(0.6, "keV"), hi_en: Quantity = Quantity(13, "keV"), ): """ + IMPLICITLY ASSUMES THE REGION IS A CIRCLE :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend spectrum. :param str cur_xtend_data_class: @@ -530,15 +533,22 @@ def gen_xrism_xtend_spectrum( # PI channel limits to subset the events evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" + # Get RA, Dec, and radius values in the right format + ra_val = rel_src_coord.ra.to("deg").value.round(6) + dec_val = rel_src_coord.dec.to("deg").value.round(6) + rad_val = rel_src_radius.to("deg").value.round(4) + # Set up the output file name for the light curve we're about to generate. sp_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"en{lo_en_val}_{hi_en_val}keV-" + f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-en{lo_en_val}_{hi_en_val}keV-" f"spectrum.fits" ) # The same file name, but with 'spectrum' changed to 'back-spectrum', for the # background light curve. - sp_back_out = sp_out.replace("spectrum", "back-spectrum") + sp_back_out = sp_out.replace(f"-radius{rad_val}deg", "").replace( + "spectrum", "back-spectrum" + ) # Create a temporary working directory temp_work_dir = os.path.join( @@ -591,7 +601,7 @@ def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): os.makedirs(temp_work_dir) # Set up the RMF file name by cannibalising the name of the spectrum file - rmf_out = os.path.basename(spec_file).split("-en")[0] + ".rmf" + rmf_out = os.path.basename(spec_file).split("-ra")[0] + ".rmf" # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so @@ -608,6 +618,65 @@ def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): rmtree(temp_work_dir) return out + + +def gen_xrism_xtend_arf( + cur_obs_id: str, + out_dir: str, + expmap_file: str, + spec_file: str, + rmf_file: str, + num_photons: int, +): + """ + IMPLICITLY ASSUMES THAT WE'RE GENERATING AN ARF FOR A 'POINT SOURCE' + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend ARF. + :param str out_dir: The directory where output files should be written. + """ + + # Spectrum files generated in this demonstration notebook contain RA-Dec + # information in their file name, so we will read it out from there + radec_sec = os.path.basename(spec_file).split("-radius")[0].split("-ra")[1] + cen_strs = radec_sec.split("-dec") + ra_val, dec_val = [float(crd) for crd in cen_strs] + + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xaarfgen_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) + + # We can use the spectrum file name to set up the output ARF file name + arf_out = os.path.basename(spec_file).replace("-spectrum.fits", ".arf") + + # Set up a name for the ray-traced simulated event file required for + # XRISM ARF generation + ray_traced_evt_out = ( + f"xrism-xtend-obsid{cur_obs_id}-numphoton{num_photons}-" + f"enALL-raytracedevents.fits" + ) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + out = hsp.xaarfgen( + xrtevtfile=os.path.join("..", ray_traced_evt_out), + outfile=os.path.join("..", arf_out), + sourcetype="POINT", + numphotons=num_photons, + source_ra=ra_val, + source_dec=dec_val, + telescop="XRISM", + instrume="XTEND", + emapfile=expmap_file, + noprompt=True, + clobber=True, + ) + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return out ``` ### Constants @@ -1465,13 +1534,15 @@ Create template variables for source and background spectrum files: sp_path_temp = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-spectrum.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" + "en{lo}_{hi}keV-spectrum.fits", ) back_sp_path_temp = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-back-spectrum.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" + "en{lo}_{hi}keV-back-spectrum.fits", ) ``` @@ -1484,10 +1555,21 @@ for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up the path to input source and background spectra cur_spec = sp_path_temp.format( - oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), ) cur_bspec = back_sp_path_temp.format( - oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), ) # Also need to pass an exposure map, so set up a path to that @@ -1550,7 +1632,13 @@ for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up relevant paths to the input and output spectrum cur_spec = sp_path_temp.format( - oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), ) cur_grp_spec = grp_sp_path_temp.format( oi=oi, @@ -1578,7 +1666,13 @@ arg_combs = [ [ oi, sp_path_temp.format( - oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), ), os.path.join(OUT_PATH, oi), ] @@ -1590,8 +1684,66 @@ with mp.Pool(NUM_CORES) as p: rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) ``` +```{code-cell} python +rmf_path_temp = os.path.join( + OUT_PATH, "{oi}", "xrism-xtend-obsid{oi}-dataclass{xdc}.rmf" +) +``` + +#### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation + +```{code-cell} python + +``` + #### Generating XRISM-Xtend ARFs +```{danger} +The HEASoft task we use to generate ARFs is called **`xaarfgen`**. There is +another, very similarly named, HEASoft tool related to the construction of XRISM +ARFs, **`xaxmaarfgen`**. Be sure which one you are using! +``` + +```{code-cell} python +arf_rt_num_photons = 20000 +``` + +```{code-cell} python +arg_combs = [ + [ + oi, + os.path.join(OUT_PATH, oi), + ex_path_temp.format( + oi=oi, + xdc=dc, + rd=expmap_rad_delta.to("arcmin").value, + npb=expmap_phi_bins, + ibf=1, + ), + sp_path_temp.format( + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), + ), + rmf_path_temp.format(oi=oi, xdc=dc), + arf_rt_num_photons, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs +] + +with mp.Pool(NUM_CORES) as p: + arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) +``` + +```{warning} +Due to the high-fidelity ray-tracing method used to calculate XRISM ARFs, the runtime +of this step can be on the order of hours. We have to do ***FINISH*** +``` ## 5. Fitting spectral models to XRISM-Xtend spectra From e2c0ab306282c867b240bf14e4be39c190aad276 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 1 Dec 2025 17:25:02 -0500 Subject: [PATCH 033/221] Missed passing the RMF file to the xaarfgen call in the wrapper function for XRISM-Xtend ARF generation in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 33 ++++++++++++++++--- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 507e523e..66c4804b 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-11-25' +date: '2025-12-01' file_format: mystnb jupytext: text_representation: @@ -49,7 +49,7 @@ XRISM is... ### Runtime -As of 26nd November 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 1st December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -240,7 +240,7 @@ def gen_xrism_xtend_image( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.extractor( filename=evt_file_chan_sel, - imgfile=os.path.join(out_dir, im_out), + imgfile=os.path.join("..", im_out), noprompt=True, clobber=True, binf=im_bin, @@ -337,7 +337,7 @@ def gen_xrism_xtend_expmap( pixgtifile=pix_gti_file, delta=radial_delta, numphi=num_phi_bin, - outfile=os.path.join(out_dir, ex_out), + outfile=os.path.join("..", ex_out), badimgfile=bad_pix_file, outmaptype=out_map_type, noprompt=True, @@ -669,6 +669,7 @@ def gen_xrism_xtend_arf( telescop="XRISM", instrume="XTEND", emapfile=expmap_file, + rmffile=rmf_file, noprompt=True, clobber=True, ) @@ -1515,6 +1516,8 @@ arg_combs = [ dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), + src_coord, + src_reg_rad, obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), spec_lo_en, @@ -1647,6 +1650,9 @@ for oi, dcs in rel_dataclasses.items(): hi=spec_hi_en.value, gt=spec_group_type, gs=spec_group_scale, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), ) hsp.ftgrouppha( @@ -1708,6 +1714,19 @@ ARFs, **`xaxmaarfgen`**. Be sure which one you are using! arf_rt_num_photons = 20000 ``` +```{warning} +***MIGHT BE WRONG, BUT v6.35.2 OF HEASOFT MIGHT POINT TO THE WRONG CALDB FILE PATH MIRROR +SCATTER INFORMATION USED BY RAYTRACE (xa_xtd_scatter_20190101v001.fits RATHER THAN xa_xtd_scatter_20190101v001.fits.gz)?*** + +***HEASOFT RELEASE NOTES SEEM TO BEAR THIS IDEA UP***: + +xrtraytrace: Additional updates to fix and enable remote CalDB + usage with xrtraytrace and xaarfgen ("timeout" interval extended + for reading large CalDB files). + + +``` + ```{code-cell} python arg_combs = [ [ @@ -1740,6 +1759,10 @@ with mp.Pool(NUM_CORES) as p: arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) ``` +```{code-cell} python +arf_path_temp = sp_path_temp.replace("-spectrum", ".arf") +``` + ```{warning} Due to the high-fidelity ray-tracing method used to calculate XRISM ARFs, the runtime of this step can be on the order of hours. We have to do ***FINISH*** @@ -1753,7 +1776,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-11-26 +Updated On: 2025-12-01 +++ From f730542ca6a1ced3dd8de993df2ba601656b8343 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 1 Dec 2025 17:38:16 -0500 Subject: [PATCH 034/221] Had to pass RA-Dec region file to the arf generation tool for XRISM-Xtend in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 66c4804b..cff4df34 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -626,6 +626,7 @@ def gen_xrism_xtend_arf( expmap_file: str, spec_file: str, rmf_file: str, + src_radec_reg_file: str, num_photons: int, ): """ @@ -670,6 +671,8 @@ def gen_xrism_xtend_arf( instrume="XTEND", emapfile=expmap_file, rmffile=rmf_file, + regionfile=src_radec_reg_file, + regmode="RADEC", noprompt=True, clobber=True, ) From a2bf80e80af6ab1d71f59612b08ead036b8c2633 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 1 Dec 2025 17:42:42 -0500 Subject: [PATCH 035/221] Had to add a temporary manually-defined link to a caldb file in the wrapper function for generation XRISM products. This is only until a problem with the caldb is sorted. For issue #128. ARFS SHOULD NOW GENERATE PROPERLY --- .../xrism/getting-started-xrism-xtend.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index cff4df34..5fe832d6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -675,8 +675,12 @@ def gen_xrism_xtend_arf( regmode="RADEC", noprompt=True, clobber=True, + scatterfile="https://heasarc.gsfc.nasa.gov/FTP/caldb/data/xrism/xtend/" + "bcf/mirror/xa_xtd_scatter_20190101v001.fits.gz[1]", ) + # TODO REMOVE THE DIRECT SETTING OF SCATTERFILE ONCE CALDB IS FIXED + # Make sure to remove the temporary directory rmtree(temp_work_dir) From 8c3b0d48fb92103ccf625846935688ab85c4bd7c Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 2 Dec 2025 13:05:44 -0500 Subject: [PATCH 036/221] Removed the demonstration of generating spectra within a certain energy range from getting-started-xrism-xtend.md. If you restrict the range the RMF and ARF have to be generated to match and it is too indepth for too little gain in a getting started demo. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 88 +++++++------------ 1 file changed, 34 insertions(+), 54 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 5fe832d6..d81bdacb 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-01' +date: '2025-12-02' file_format: mystnb jupytext: text_representation: @@ -64,8 +64,7 @@ from shutil import rmtree import heasoftpy as hsp import matplotlib.pyplot as plt import numpy as np - -# import xspec as xs +import xspec as xs from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.table import Table @@ -488,8 +487,6 @@ def gen_xrism_xtend_spectrum( rel_src_radius: Quantity, src_reg_file: str, back_reg_file: str, - lo_en: Quantity = Quantity(0.6, "keV"), - hi_en: Quantity = Quantity(13, "keV"), ): """ IMPLICITLY ASSUMES THE REGION IS A CIRCLE @@ -498,10 +495,6 @@ def gen_xrism_xtend_spectrum( :param str cur_xtend_data_class: :param str event_file: :param str out_dir: The directory where output files should be written. - :param Quantity lo_en: Lower bound of the energy band within which we will - generate the spectrum. - :param Quantity hi_en: Upper bound of the energy band within which we will - generate the spectrum. :return: A tuple containing the processed ObsID, the log output of the pipeline, and a boolean flag indicating success (True) or failure (False). :rtype: Tuple[str, hsp.core.HSPResult, bool] @@ -514,25 +507,6 @@ def gen_xrism_xtend_spectrum( "must be 3 for in-flight data." ) - # Make sure the lower and upper energy limits make sense - if lo_en > hi_en: - raise ValueError( - "The lower energy limit must be less than or equal to the upper " - "energy limit." - ) - else: - lo_en_val = lo_en.to("keV").value - hi_en_val = hi_en.to("keV").value - - # Convert the energy limits to channel limits, rounding down and up to the nearest - # integer channel for the lower and upper bounds respectively. - lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) - hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) - - # Create modified input event list file path, where we use the just-calculated - # PI channel limits to subset the events - evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" - # Get RA, Dec, and radius values in the right format ra_val = rel_src_coord.ra.to("deg").value.round(6) dec_val = rel_src_coord.dec.to("deg").value.round(6) @@ -541,7 +515,7 @@ def gen_xrism_xtend_spectrum( # Set up the output file name for the light curve we're about to generate. sp_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-en{lo_en_val}_{hi_en_val}keV-" + f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-enALL-" f"spectrum.fits" ) # The same file name, but with 'spectrum' changed to 'back-spectrum', for the @@ -561,7 +535,7 @@ def gen_xrism_xtend_spectrum( # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): src_out = hsp.extractor( - filename=evt_file_chan_sel, + filename=event_file, phafile=os.path.join("..", sp_out), regionfile=src_reg_file, xcolf="X", @@ -573,7 +547,7 @@ def gen_xrism_xtend_spectrum( # Now for the background light curve back_out = hsp.extractor( - filename=evt_file_chan_sel, + filename=event_file, phafile=os.path.join("..", sp_back_out), regionfile=back_reg_file, xcolf="X", @@ -1511,11 +1485,6 @@ back_lc_path_temp = os.path.join( #### Generating the spectral files -```{code-cell} python -spec_lo_en = Quantity(0.6, "keV") -spec_hi_en = Quantity(13, "keV") -``` - ```{code-cell} python arg_combs = [ [ @@ -1527,8 +1496,6 @@ arg_combs = [ src_reg_rad, obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), - spec_lo_en, - spec_hi_en, ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -1545,14 +1512,13 @@ sp_path_temp = os.path.join( OUT_PATH, "{oi}", "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" - "en{lo}_{hi}keV-spectrum.fits", + "enALL-spectrum.fits", ) back_sp_path_temp = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" - "en{lo}_{hi}keV-back-spectrum.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "enALL-back-spectrum.fits", ) ``` @@ -1567,8 +1533,6 @@ for oi, dcs in rel_dataclasses.items(): cur_spec = sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), @@ -1576,8 +1540,6 @@ for oi, dcs in rel_dataclasses.items(): cur_bspec = back_sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), ) @@ -1644,8 +1606,6 @@ for oi, dcs in rel_dataclasses.items(): cur_spec = sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), @@ -1653,8 +1613,6 @@ for oi, dcs in rel_dataclasses.items(): cur_grp_spec = grp_sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, gt=spec_group_type, gs=spec_group_scale, ra=src_coord.ra.value.round(6), @@ -1681,8 +1639,6 @@ arg_combs = [ sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), @@ -1749,8 +1705,6 @@ arg_combs = [ sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), @@ -1767,7 +1721,7 @@ with mp.Pool(NUM_CORES) as p: ``` ```{code-cell} python -arf_path_temp = sp_path_temp.replace("-spectrum", ".arf") +arf_path_temp = sp_path_temp.replace("-spectrum.fits", ".arf") ``` ```{warning} @@ -1777,6 +1731,32 @@ of this step can be on the order of hours. We have to do ***FINISH*** ## 5. Fitting spectral models to XRISM-Xtend spectra +### Configuring PyXspec + +Now we configure some behaviors of XSPEC/pyXspec: +- The ```chatter``` parameter is set to zero to reduce printed output during fitting (note that some XSPEC messages are still shown). +- We inform XSPEC of the number of cores we have available, as some XSPEC methods can be paralleled. +- We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). + +```{code-cell} python +xs.Xset.chatter = 0 + +# XSPEC parallelisation settings +xs.Xset.parallel.leven = NUM_CORES +xs.Xset.parallel.error = NUM_CORES +xs.Xset.parallel.steppar = NUM_CORES + +# Other xspec settings +xs.Plot.area = True +xs.Plot.xAxis = "keV" +xs.Plot.background = True +xs.Fit.statMethod = "cstat" +xs.Fit.query = "no" +xs.Fit.nIterations = 500 +``` + +### Reading XRISM-Xtend spectra into pyXspec + ## About this notebook Author: David J Turner, HEASARC Staff Scientist. From d67ce11641337100a7d0e7452733c26a035051d9 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 2 Dec 2025 13:19:33 -0500 Subject: [PATCH 037/221] Have to include '__row' in the columns we manually specify should be returned by Heasarc.query_region in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index d81bdacb..69768efc 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -766,7 +766,8 @@ src_coord ```{code-cell} python col_str = ( - "obsid,name,ra,dec,time,exposure,status,public_date,xtd_dataclas1,xtd_dataclas2" + "__row,obsid,name,ra,dec,time,exposure,status,public_date," + "xtd_dataclas1,xtd_dataclas2" ) all_xrism_obs = Heasarc.query_region(src_coord, catalog_name, columns=col_str) all_xrism_obs From 83a911cf931b2d904c24468b848ad16740f5af42 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 2 Dec 2025 14:38:40 -0500 Subject: [PATCH 038/221] Hadn't passed 'radec_src_reg_path' to the gen_xrism_xtend_arf function when called in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 69768efc..7ceffd31 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1252,6 +1252,9 @@ expmap_rad_delta = Quantity(20, "arcmin") expmap_phi_bins = 1 ``` +```{code-cell} python +expmap_bin_factors = [4] +``` gen_xrism_xtend_expmap(chosen_demo_obsid, "32000010", os.path.join(OUT_PATH, chosen_demo_obsid), @@ -1278,7 +1281,7 @@ arg_combs = [ ] for oi, dcs in rel_dataclasses.items() for dc in dcs - for cur_bf in bin_factors + for cur_bf in expmap_bin_factors ] with mp.Pool(NUM_CORES) as p: @@ -1711,6 +1714,7 @@ arg_combs = [ rad=src_reg_rad.to("deg").value.round(4), ), rmf_path_temp.format(oi=oi, xdc=dc), + radec_src_reg_path, arf_rt_num_photons, ] for oi, dcs in rel_dataclasses.items() From 7ae0e4b97f8f9f8335d1c6e33bbf01814bfc9bfa Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 3 Dec 2025 13:56:10 -0500 Subject: [PATCH 039/221] Updated the 'gen_xrism_xtend_image' wrapper function in getting-started-xrism-xtend.md, so that the user no longer has to pass the ObsID and dataclass information (they are extracted from event lists). For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 189 +++++++++++++----- 1 file changed, 142 insertions(+), 47 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 7ceffd31..d22350cb 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-02' +date: '2025-12-03' file_format: mystnb jupytext: text_representation: @@ -30,26 +30,50 @@ title: Getting started with XRISM-Xtend By the end of this tutorial, you will be able to: -- Find... -- +- Identify and download XRISM observations of an interesting source. +- Prepare the XRISM-Xtend data for analysis. +- Generate XRISM-Xtend data products: + - Images + - Exposure maps + - Light curves + - Spectra and supporting files +- Perform a simple spectral analysis of a XRISM-Xtend spectrum ## Introduction -XRISM is... +The 'X-Ray Imaging and Spectroscopy Mission' (**XRISM**) is an X-ray telescope designed for high-energy-resolution +spectroscopic observations of astrophysical sources, as well as wide-field X-ray imaging. +XRISM, launched in 2023, is the result of a JAXA-NASA partnership (with involvement from ESA), and serves as nearly like-for-like replacement +of the **Hitomi** telescope, which was lost shortly after its launch in 2016. + +There are two main XRISM instruments, **Xtend** and **Resolve**. In this tutorial, we will focus on **Xtend**, which is +a wide-field CCD spectro-imaging instrument similar in concept to instruments included on many other X-ray +telescopes (XMM's EPIC detectors, Chandra's ACIS, Swift's XRT, etc.) The other instrument, **Resolve**, has its own +dedicated demonstration notebook. + +Our goal with this 'getting started' notebook is to give you the skills required to prepare XRISM-Xtend +observations for scientific use and to generate data products tailored to your science goals. It can also serve as a +template notebook to build your own analyses on top of. + +Other tutorials in this series will explore how to perform more complicated generation and analysis +of XRISM-Xtend data, but here we will focus on making single aperture light curves and spectra for an +object that can be semi-reasonably treated as a 'point' source. The supernova-remnant LMC N132D. ### Inputs -- +- The name of the source of interest - in this case *LMC N132D* ### Outputs -- +- Processed, cleaned, and calibrated XRISM-Xtend event lists. +- XRISM-Xtend images, exposure maps, light curves, spectra, and supporting files. +- Simple region files that define where light curves and spectra are extracted from. ### Runtime -As of 1st December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 3rd December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -73,12 +97,6 @@ from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc from regions import CircleSkyRegion from xga.products import Image - -# from typing import Tuple, Union -# from warnings import warn - -# from matplotlib.ticker import FuncFormatter -# from tqdm import tqdm ``` ## Global Setup @@ -130,14 +148,14 @@ def process_xrism_xtend( :rtype: Tuple[str, hsp.core.HSPResult, bool] """ - # Makes sure the specified output directory exists. - temp_outdir = os.path.join(out_dir, "temp") - os.makedirs(temp_outdir, exist_ok=True) + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xtdpipeline_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). - with contextlib.chdir(out_dir), hsp.utils.local_pfiles_context(): + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): # The processing/preparation stage of any X-ray telescope's data is the most # likely to go wrong, and we use a Python try-except as an automated way to @@ -149,7 +167,7 @@ def process_xrism_xtend( steminputs=file_stem, stemoutputs=file_stem, indir=evt_dir, - outdir=temp_outdir, + outdir=".", attitude=attitude, orbit=orbit, obsgti=obs_gti, @@ -164,18 +182,18 @@ def process_xrism_xtend( task_success = False out = str(err) - # Moves files from the temporary output directory into the - # final output directory - if os.path.exists(temp_outdir) and len(os.listdir(temp_outdir)) != 0: - for f in os.listdir(temp_outdir): - os.rename(os.path.join(temp_outdir, f), os.path.join(out_dir, f)) + # Moves files from the temporary output directory into the + # final output directory + if os.path.exists(temp_work_dir) and len(os.listdir(temp_work_dir)) != 0: + for f in os.listdir(temp_work_dir): + os.rename(os.path.join(temp_work_dir, f), os.path.join(out_dir, f)) + # Make sure to remove the temporary directory + rmtree(temp_work_dir) return cur_obs_id, out, task_success def gen_xrism_xtend_image( - cur_obs_id: str, - cur_xtend_data_class: str, event_file: str, out_dir: str, lo_en: Quantity, @@ -183,24 +201,32 @@ def gen_xrism_xtend_image( im_bin: int = 1, ): """ + This function wraps the HEASoft 'extractor' tool and is used to spatially bin + XRISM-Xtend event lists into images. The HEASoftPy interface to 'extractor' is used. - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend image. - :param str cur_xtend_data_class: - :param str event_file: + Both the energy band and the image binning factor, which controls how + many 'pixels' in the native SKY X-Y coordinate of the event list are binned into + a single image pixel, can be specified. + + The ObsID and dataclass are extracted from the header of the passed event list file. + + :param str event_file: Path to the event list (usually cleaned, but not + necessarily) we wish to generate an image from. ObsID and dataclass information + will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. :param Quantity lo_en: Lower bound of the energy band within which we will generate the image. :param Quantity hi_en: Upper bound of the energy band within which we will generate the image. - :return: A tuple containing the processed ObsID, the log output of the - pipeline, and a boolean flag indicating success (True) or failure (False). - :rtype: Tuple[str, hsp.core.HSPResult, bool] + :param int im_bin: Number of XRISM-Xtend SKY X-Y pixels to bin into a single image + pixel. """ - if cur_xtend_data_class[0] != "3": - raise ValueError( - f"The first digit of the Xtend data class ({cur_xtend_data_class}) " - "must be 3 for in-flight data." - ) + + # We can extract the ObsID and data class directly from the header of the event + # list - it is safer than having them be passed to this function separately. + with fits.open(event_file) as read_evto: + cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] + cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] # Make sure the lower and upper energy limits make sense if lo_en > hi_en: @@ -239,7 +265,7 @@ def gen_xrism_xtend_image( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.extractor( filename=evt_file_chan_sel, - imgfile=os.path.join("..", im_out), + imgfile=im_out, noprompt=True, clobber=True, binf=im_bin, @@ -247,6 +273,10 @@ def gen_xrism_xtend_image( ycolf="Y", ) + # Move the output image file to the proper output directory from + # the temporary working directory + os.rename(os.path.join(temp_work_dir, im_out), os.path.join(out_dir, im_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1216,8 +1246,6 @@ We use... ```{code-cell} python arg_combs = [ [ - oi, - dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), *cur_bnds, @@ -1255,13 +1283,6 @@ expmap_phi_bins = 1 ```{code-cell} python expmap_bin_factors = [4] ``` -gen_xrism_xtend_expmap(chosen_demo_obsid, - "32000010", - os.path.join(OUT_PATH, chosen_demo_obsid), - evt_path_temp.format(oi=chosen_demo_obsid, xdc="32000010", sc=0), - ehk_path_temp.format(oi=chosen_demo_obsid), - badpix_path_temp.format(oi=chosen_demo_obsid, xdc="32000010", sc=0), - im_bin=4, out_map_type='EFFICIENCY') ***WHAT ABOUT THIS GTI?? - xa000128000xtd_mode.gti*** @@ -1736,6 +1757,12 @@ of this step can be on the order of hours. We have to do ***FINISH*** ## 5. Fitting spectral models to XRISM-Xtend spectra +Finally, to show off the XRISM-Xtend products we just generated, we will perform +a very simple model fit to one of our spectra. + +Our demonstration of spectral model fitting to an XRISM-Xtend spectrum will be +performed using the [PyXspec](https://heasarc.gsfc.nasa.gov/docs/software/xspec/python/html/index.html) package. + ### Configuring PyXspec Now we configure some behaviors of XSPEC/pyXspec: @@ -1762,13 +1789,81 @@ xs.Fit.nIterations = 500 ### Reading XRISM-Xtend spectra into pyXspec +```{code-cell} python +chosen_demo_spec_obsid = "000128000" +chosen_demo_spec_dataclass = "31100010" +``` + +```{code-cell} python +# In case this cell is re-run, clear all previously loaded spectra +xs.AllData.clear() + +# Set up the paths to grouped source spectrum, ungrouped background +# spectrum, RMF, and ARF files +cur_spec = grp_sp_path_temp.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + gt=spec_group_type, + gs=spec_group_scale, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), +) + +cur_bspec = back_sp_path_temp.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), +) + +cur_rmf = rmf_path_temp.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, +) + +cur_arf = arf_path_temp.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), +) + +# Load the chosen spectrum (and all its supporting files) into pyXspec +xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) +``` + +### Restricting the spectral channels used for fitting + +```{code-cell} python +xs_spec.ignore("**-0.5 12.0-**") + +# Ignore any channels that have been marked as 'bad' +# This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra +# have been declared +xs.AllData.ignore("bad") +``` + +### Setting up a spectral model + +```{code-cell} python +xs.Model("tbabs*(powerlaw+apec+bbody)") +``` + +### Fitting our pyXspec model to the XRISM-Xtend spectrum + +```{code-cell} python +xs.Fit.perform() +``` + ## About this notebook Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-01 +Updated On: 2025-12-03 +++ From 14725b0ff62d56312ebebeb97a49b32b4735e6ab Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 3 Dec 2025 14:23:50 -0500 Subject: [PATCH 040/221] Altered the 'gen_xrism_xtend_expmap' function (in getting-started-xrism-xtend.md) so that it takes an event list argument in lieu of manually specifying the ObsID and dataclass of the exposure map to generate. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 161 ++++++++++-------- 1 file changed, 91 insertions(+), 70 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index d22350cb..0caaeccd 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -84,6 +84,7 @@ import multiprocessing as mp import os from random import randint from shutil import rmtree +from typing import Union import heasoftpy as hsp import matplotlib.pyplot as plt @@ -105,21 +106,22 @@ from xga.products import Image ```{code-cell} python --- -tags: [hide-input] +tags: [hide - input] jupyter: - source_hidden: true +source_hidden: true --- + def process_xrism_xtend( - cur_obs_id: str, - out_dir: str, - evt_dir: str, - attitude: str, - orbit: str, - obs_gti: str, - mkf_filter: str, - file_stem: str, - extended_housekeeping: str, - xtend_housekeeping: str, + cur_obs_id: str, + out_dir: str, + evt_dir: str, + attitude: str, + orbit: str, + obs_gti: str, + mkf_filter: str, + file_stem: str, + extended_housekeeping: str, + xtend_housekeeping: str, ): """ A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process @@ -194,11 +196,11 @@ def process_xrism_xtend( def gen_xrism_xtend_image( - event_file: str, - out_dir: str, - lo_en: Quantity, - hi_en: Quantity, - im_bin: int = 1, + event_file: str, + out_dir: str, + lo_en: Quantity, + hi_en: Quantity, + im_bin: int = 1, ): """ This function wraps the HEASoft 'extractor' tool and is used to spatially bin @@ -284,34 +286,46 @@ def gen_xrism_xtend_image( def gen_xrism_xtend_expmap( - cur_obs_id: str, - cur_xtend_data_class: str, - out_dir: str, - gti_file: str, - extend_hk_file: str, - bad_pix_file: str, - pix_gti_file: str = "NONE", - im_bin: int = 1, - radial_delta: float = Quantity(20.0, "arcmin"), - num_phi_bin: int = 1, + event_file: str, + out_dir: str, + gti_file: str, + extend_hk_file: str, + bad_pix_file: str, + pix_gti_file: str = "NONE", + im_bin: int = 1, + radial_delta: Union[float, Quantity] = Quantity(20.0, "arcmin"), + num_phi_bin: int = 1, ): """ + Function that wraps the HEASoftPy interface to the XRISM-Xtend 'xaexpmap' + task, which is used to generate exposure maps for XRISM-Xtend observations. - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend exposure map. - :param str cur_xtend_data_class: - :param str event_file: + :param str event_file: Event list of the observation + dataclass you wish to + generate an exposure map for. No event data are used in the creation of the + event list, but some information in the file headers is useful. :param str out_dir: The directory where output files should be written. - :return: A tuple containing the processed ObsID, the log output of the - pipeline, and a boolean flag indicating success (True) or failure (False). - :rtype: Tuple[str, hsp.core.HSPResult, bool] + :param str gti_file: File defining the good-time-intervals of the observation + and observation dataclass for which we are generating an exposure map (often + the event list itself is passed). + :param str extend_hk_file: + :param str bad_pix_file: + :param str pix_gti_file: Optional file defining the good-time-intervals of + individual XRISM-Xtend pixels. If not provided, the default value of 'NONE' is + passed to 'xaexpmap'. + :param im_bin: Number of XRISM-Xtend SKY X-Y pixels to bin into a single exposure + map pixel. Defaults to 1, and any other value will also result in an + 'im_bin=1' being generated. + :param float/Quantity radial_delta: Radial increment for the annular grid for + which the attitude histogram will be calculated. + :param int num_phi_bin: Number of azimuth (phi) bins in the first annular region + over which attitude histogram bins will be calculated """ - # Validity check on the passed data class - if cur_xtend_data_class[0] != "3": - raise ValueError( - f"The first digit of the Xtend data class ({cur_xtend_data_class}) " - "must be 3 for in-flight data." - ) + # We can extract the ObsID and data class directly from the header of the event + # list - it is safer than having them be passed to this function separately. + with fits.open(event_file) as read_evto: + cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] + cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] # Make sure the radial_delta value is in arcminutes/is convertible to arcmins # Also will assume that radial_delta is in arcmin if it is not a Quantity object @@ -366,7 +380,7 @@ def gen_xrism_xtend_expmap( pixgtifile=pix_gti_file, delta=radial_delta, numphi=num_phi_bin, - outfile=os.path.join("..", ex_out), + outfile=ex_out, badimgfile=bad_pix_file, outmaptype=out_map_type, noprompt=True, @@ -376,14 +390,22 @@ def gen_xrism_xtend_expmap( # If the user wants a spatially binned exposure map, we run the fimgbin task if im_bin != 1: rebin_out = hsp.fimgbin( - infile=os.path.join("..", ex_out), - outfile=os.path.join("..", binned_ex_out), + infile=ex_out, + outfile=binned_ex_out, xbinsize=im_bin, noprompt=True, clobber=True, ) out = [out, rebin_out] + # Move the im_bin=1 exposure map (guaranteed to have been generated) up to the + # final output directory + os.rename(os.path.join(temp_work_dir, ex_out), os.path.join(out_dir, ex_out)) + # Then do the same for the spatially binned exposure map, if it was requested + if im_bin != 1: + os.rename(os.path.join(temp_work_dir, binned_ex_out), + os.path.join(out_dir, binned_ex_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -391,16 +413,16 @@ def gen_xrism_xtend_expmap( def gen_xrism_xtend_lightcurve( - cur_obs_id: str, - cur_xtend_data_class: str, - event_file: str, - out_dir: str, - src_reg_file: str, - back_reg_file: str, - lo_en: Quantity = Quantity(0.6, "keV"), - hi_en: Quantity = Quantity(13, "keV"), - time_bin_size: Quantity = Quantity(200, "s"), - lc_bin_thresh: float = 0.0, + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + src_reg_file: str, + back_reg_file: str, + lo_en: Quantity = Quantity(0.6, "keV"), + hi_en: Quantity = Quantity(13, "keV"), + time_bin_size: Quantity = Quantity(200, "s"), + lc_bin_thresh: float = 0.0, ): """ @@ -509,14 +531,14 @@ def gen_xrism_xtend_lightcurve( def gen_xrism_xtend_spectrum( - cur_obs_id: str, - cur_xtend_data_class: str, - event_file: str, - out_dir: str, - rel_src_coord: SkyCoord, - rel_src_radius: Quantity, - src_reg_file: str, - back_reg_file: str, + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, + src_reg_file: str, + back_reg_file: str, ): """ IMPLICITLY ASSUMES THE REGION IS A CIRCLE @@ -625,13 +647,13 @@ def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): def gen_xrism_xtend_arf( - cur_obs_id: str, - out_dir: str, - expmap_file: str, - spec_file: str, - rmf_file: str, - src_radec_reg_file: str, - num_photons: int, + cur_obs_id: str, + out_dir: str, + expmap_file: str, + spec_file: str, + rmf_file: str, + src_radec_reg_file: str, + num_photons: int, ): """ IMPLICITLY ASSUMES THAT WE'RE GENERATING AN ARF FOR A 'POINT SOURCE' @@ -680,7 +702,7 @@ def gen_xrism_xtend_arf( noprompt=True, clobber=True, scatterfile="https://heasarc.gsfc.nasa.gov/FTP/caldb/data/xrism/xtend/" - "bcf/mirror/xa_xtd_scatter_20190101v001.fits.gz[1]", + "bcf/mirror/xa_xtd_scatter_20190101v001.fits.gz[1]", ) # TODO REMOVE THE DIRECT SETTING OF SCATTERFILE ONCE CALDB IS FIXED @@ -1289,8 +1311,7 @@ expmap_bin_factors = [4] ```{code-cell} python arg_combs = [ [ - oi, - dc, + evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), evt_path_temp.format(oi=oi, xdc=dc, sc=0), ehk_path_temp.format(oi=oi), From 46cba22943dd1bdb15d56dfe9eef7f10fdb9893c Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 3 Dec 2025 14:44:52 -0500 Subject: [PATCH 041/221] Altered the time_bin_size gen_xrism_xtend_lightcurve function to not require ObsID and dataclass arguments. Also added a docstring. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 40 ++++++++++--------- 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0caaeccd..b99329d5 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -413,8 +413,6 @@ def gen_xrism_xtend_expmap( def gen_xrism_xtend_lightcurve( - cur_obs_id: str, - cur_xtend_data_class: str, event_file: str, out_dir: str, src_reg_file: str, @@ -425,26 +423,30 @@ def gen_xrism_xtend_lightcurve( lc_bin_thresh: float = 0.0, ): """ + Function that wraps the HEASoftPy interface to the HEASoft extractor tool, set + up to generate light curves from XRISM-Xtend observations. The function will + generate a light curve for the source region and a background light curve for + the background region. - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend light curve. - :param str cur_xtend_data_class: - :param str event_file: + :param str event_file: Path to the event list (usually cleaned, but not + necessarily) we wish to generate a XRISM-Xtend light curve from. ObsID and + dataclass information will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. :param Quantity lo_en: Lower bound of the energy band within which we will generate the light curve. :param Quantity hi_en: Upper bound of the energy band within which we will generate the light curve. - :return: A tuple containing the processed ObsID, the log output of the - pipeline, and a boolean flag indicating success (True) or failure (False). - :rtype: Tuple[str, hsp.core.HSPResult, bool] + :param Quantity time_bin_size: The size of the time bins used to generate the + light curve. + :param float lc_bin_thresh: When constructing a light curve, any bins whose + exposure is less than lc_bin_thresh*time_bin_size are ignored. """ - # Validity check on the passed data class - if cur_xtend_data_class[0] != "3": - raise ValueError( - f"The first digit of the Xtend data class ({cur_xtend_data_class}) " - "must be 3 for in-flight data." - ) + # We can extract the ObsID and data class directly from the header of the event + # list - it is safer than having them be passed to this function separately. + with fits.open(event_file) as read_evto: + cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] + cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] # Check the units of the passed time bin size - also if the passed value is # a float or integer, we'll assume it is in seconds @@ -501,7 +503,7 @@ def gen_xrism_xtend_lightcurve( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): src_out = hsp.extractor( filename=evt_file_chan_sel, - fitsbinlc=os.path.join("..", lc_out), + fitsbinlc=lc_out, binlc=time_bin_size, lcthresh=lc_bin_thresh, regionfile=src_reg_file, @@ -514,7 +516,7 @@ def gen_xrism_xtend_lightcurve( # Now for the background light curve back_out = hsp.extractor( filename=evt_file_chan_sel, - fitsbinlc=os.path.join("..", lc_back_out), + fitsbinlc=lc_back_out, binlc=time_bin_size, lcthresh=lc_bin_thresh, regionfile=back_reg_file, @@ -524,6 +526,10 @@ def gen_xrism_xtend_lightcurve( clobber=True, ) + # Move the light curves up from the temporary directory + os.rename(os.path.join(temp_work_dir, lc_out), os.path.join(out_dir, lc_out)) + os.rename(os.path.join(temp_work_dir, lc_back_out), os.path.join(out_dir, lc_back_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1487,8 +1493,6 @@ xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") ```{code-cell} python arg_combs = [ [ - oi, - dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), From 875dc772171d5aa960286a71672f34ca25fa83d5 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 3 Dec 2025 17:00:21 -0500 Subject: [PATCH 042/221] Moved the xspec import down the notebook --- .../xrism/getting-started-xrism-xtend.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b99329d5..1c1ff42e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -89,7 +89,6 @@ from typing import Union import heasoftpy as hsp import matplotlib.pyplot as plt import numpy as np -import xspec as xs from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.table import Table @@ -1796,6 +1795,8 @@ Now we configure some behaviors of XSPEC/pyXspec: - We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). ```{code-cell} python +import xspec as xs + xs.Xset.chatter = 0 # XSPEC parallelisation settings From d9acfc51610fbe06059732be793553dd1fe0e90a Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 4 Dec 2025 12:10:06 -0500 Subject: [PATCH 043/221] Fixed an issue with an incorrect dataclass being passed to RMF generation (perhaps will alter the function to take an event list like some of the other wrapper functions - could be safer). In getting-started-xrism-xtend.md, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 1c1ff42e..dd7cf344 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1687,7 +1687,7 @@ arg_combs = [ oi, sp_path_temp.format( oi=oi, - xdc=cur_dc, + xdc=dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), From da24bb49c4f4825ae5e887d63483f5d05696aae0 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 10:30:13 -0500 Subject: [PATCH 044/221] Explicitly defined the name of the table in the event list that contains GTI information when generating images, light curves, and spectra using extractor. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 102 ++++++++++++++---- 1 file changed, 81 insertions(+), 21 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index dd7cf344..5d3e8896 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-03' +date: '2025-12-05' file_format: mystnb jupytext: text_representation: @@ -58,8 +58,9 @@ template notebook to build your own analyses on top of. Other tutorials in this series will explore how to perform more complicated generation and analysis of XRISM-Xtend data, but here we will focus on making single aperture light curves and spectra for an -object that can be semi-reasonably treated as a 'point' source. The supernova-remnant LMC N132D. +object that can be semi-reasonably treated as a 'point' source; the supernova-remnant LMC N132D. +We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstration. ### Inputs @@ -73,7 +74,7 @@ object that can be semi-reasonably treated as a 'point' source. The supernova-re ### Runtime -As of 3rd December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 5th December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -272,6 +273,7 @@ def gen_xrism_xtend_image( binf=im_bin, xcolf="X", ycolf="Y", + gti="GTI", ) # Move the output image file to the proper output directory from @@ -508,6 +510,7 @@ def gen_xrism_xtend_lightcurve( regionfile=src_reg_file, xcolf="X", ycolf="Y", + gti="GTI", noprompt=True, clobber=True, ) @@ -521,6 +524,7 @@ def gen_xrism_xtend_lightcurve( regionfile=back_reg_file, xcolf="X", ycolf="Y", + gti="GTI", noprompt=True, clobber=True, ) @@ -598,6 +602,7 @@ def gen_xrism_xtend_spectrum( xcolf="X", ycolf="Y", ecol="PI", + gti="GTI", noprompt=True, clobber=True, ) @@ -610,6 +615,7 @@ def gen_xrism_xtend_spectrum( xcolf="X", ycolf="Y", ecol="PI", + gti="GTI", noprompt=True, clobber=True, ) @@ -706,12 +712,8 @@ def gen_xrism_xtend_arf( regmode="RADEC", noprompt=True, clobber=True, - scatterfile="https://heasarc.gsfc.nasa.gov/FTP/caldb/data/xrism/xtend/" - "bcf/mirror/xa_xtd_scatter_20190101v001.fits.gz[1]", ) - # TODO REMOVE THE DIRECT SETTING OF SCATTERFILE ONCE CALDB IS FIXED - # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1075,6 +1077,8 @@ evt_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") ``` +### Good-time-intervals + ### Identifying problem pixels ```{code-cell} python @@ -1083,9 +1087,18 @@ badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") ## 3. Generating new XRISM-Xtend images and exposure maps +The XRISM-Xtend data have now been prepared for scientific use, with the most important +output being the cleaned event list(s); remember that one observation can produce +**two** cleaned event lists if Xtend was operating in a windowed or burst mode. + +We will now demonstrate how to generate new XRISM-Xtend data products tailored to your +scientific needs. Images and exposure maps can be generated for the entire +field-of-view (FoV), rather than having to focus on a particular source, so we will +start with them. + ### Converting energy bounds to channel bounds -The data products we generate in this section can all benefit from selecting events +The data products we generate in this section (and the next) can all benefit from selecting events from within a specific energy range. This might be because your source of interest only emits in a narrow energy range, and you don't care about the rest, or because different mechanisms emit at different energies, and you wish to separate them. @@ -1125,8 +1138,8 @@ We will be creating new RMFs as part of the generation of XRISM-Xtend spectra in next section. For our current purpose, however, it is acceptable to use the RMFs that were included in the XRISM-Xtend archive we downloaded earlier. -The archived RMFs are generated for the entire Xtend field-of-view (FoV), rather than for the CCDs -our particular target fall on, but practically speaking, that doesn't make a significant +The archived RMFs are generated for the entire Xtend FoV, rather than for the CCDs +our particular target falls on, but practically speaking, that doesn't make a significant difference. Using observation 000128000 as an example, we determine the path to the relevant @@ -1171,7 +1184,7 @@ e_bounds[90:110] ``` We can use this file to visualize the basic linear mapping between energy and -channel - it will not be the most interesting figure you've ever seen: +channel - *it will be the most boring figure you've ever seen*: ```{code-cell} python --- @@ -1179,13 +1192,18 @@ tags: [hide-input] jupyter: source_hidden: true --- + +# Set up the figure plt.figure(figsize=(5.5, 5.5)) +# Configuring the axis ticks plt.minorticks_on() plt.tick_params(which="both", direction="in", top=True, right=True) +# Calculate the mid-point of each energy bin mid_ens = (e_bounds["E_MIN"] + e_bounds["E_MAX"]) / 2 +# Plot the relationship between channel and the energy bin mid-points plt.plot(e_bounds["CHANNEL"], mid_ens, color="navy", alpha=0.9, label="XRISM-Xtend") plt.xlim(0) @@ -1204,10 +1222,13 @@ Finally, we can validate our assumed relationship between energy and channel by calculating the mean change in minimum energy between adjacent channels: ```{code-cell} python -# -rmf_ev_per_chan = Quantity(np.diff(e_bounds["E_MIN"].data).mean(), "keV/chan").to( - "eV/chan" -) +# Calculates the energy change from one to channel to the next, then finds the +# mean value of those energy changes +mean_en_diffs = np.diff(e_bounds["E_MIN"].data).mean() + +# Set up the result in an astropy quantity and convert to eV-per-channel for +# easier comparison to the assumed relationship +rmf_ev_per_chan = Quantity(mean_en_diffs, "keV/chan").to("eV/chan") rmf_ev_per_chan ``` @@ -1219,8 +1240,26 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ### New XRISM-Xtend images +We've established that we understand XRISM-Xtend's relationship between energy and +channel. Now we can use that relationship to choose the energy bounds we generate +data products within and convert them to the channel values required by XRISM HEASoft +tasks. + +We recommend that you generate images first, as examining them is a good way to spot +any problems or unusual features of the prepared and cleaned observations. + #### Image energy bounds +We are going to generate images within the following energy bounds: +- 0.6-2.0 keV +- 2.0-10.0 keV +- ***0.4-2.0 keV*** [not recommended] +- ***0.4-10.0 keV*** [not recommended] + +The bands that have a lower bound of ***0.4 keV*** are ***not recommended***, as there +are issues with XRISM-Xtend data below *0.6 keV*. We are generating them to +demonstrate those issues. + ```{code-cell} python # Defining the energy bounds we want images within xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV") @@ -1266,7 +1305,28 @@ bin_factors = [1, 4] #### Running image generation -We use... +There is no HEASoft tool specifically for generating XRISM-Xtend images, but there is a +generalized HEASoft image (and other data products) generation task that we can use. + +If you have previously generated images, light curves, or spectra from HEASARC-hosted +X-ray data on the command line, you may well have come across `XSELECT`; a HEASoft +tool for interactively generating data products from event lists. + +When creating data products, `XSELECT` calls the HEASoft `extractor` task, which we +will now use to demonstrate the creation of XRISM-Xtend images. + +As with all uses of HEASoft tasks in this notebook, our call to `extractor` will be +through the HEASoftPy Python interface - specifically the `hsp.extractor` function. + +We have implemented a wrapper to this function in the 'Global Setup: Functions' section +of this notebook, primarily so that we can easily multiprocess the generation of images +in different energy bands, binning factors, observations, and dataclasses. + +Image generation is not a particularly computationally intensive task, but if you are +addressing a large number of observations (or making many images per observation), it +is a good idea to run them in parallel! + + ***NEED TO APPLY GTIS TO IMAGE GENERATION AS WELL*** @@ -1400,7 +1460,7 @@ src_reg_rad = Quantity(2, "arcmin") src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) # Write the source region to a region file -src_reg.write(radec_src_reg_path, format="ds9") +src_reg.write(radec_src_reg_path, format="ds9", overwrite=True) ``` We do the same to define a region from which to extract a background spectrum: @@ -1419,7 +1479,7 @@ back_reg_rad = Quantity(3, "arcmin") back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) # Once again writing the region to a region file as well -back_reg.write(radec_back_reg_path, format="ds9") +back_reg.write(radec_back_reg_path, format="ds9", overwrite=True) ``` #### Visualizing the source and background extraction regions on XRISM-Xtend images @@ -1462,10 +1522,10 @@ obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_back.re for oi in rel_obsids: src_reg.to_pixel(oi_skypix_wcs[oi]).write( - obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9" + obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9", overwrite=True ) back_reg.to_pixel(oi_skypix_wcs[oi]).write( - obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9" + obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9", overwrite=True ) ``` @@ -1753,7 +1813,7 @@ arg_combs = [ ), sp_path_temp.format( oi=oi, - xdc=cur_dc, + xdc=dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), From bef4242d4610bba6c9dbf6071077d2aa7ccebbc5 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 10:42:39 -0500 Subject: [PATCH 045/221] Moved the definition of most output path template variables to the 'Constants' section of the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 143 +++++++++++------- 1 file changed, 85 insertions(+), 58 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 5d3e8896..079734e1 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -738,6 +738,74 @@ TASK_CHATTER = 3 # The approximate linear relationship between Xtend PI and event energy XTD_EV_PER_CHAN = (1 / Quantity(166.7, "chan/keV")).to("eV/chan") + + +# ------------- Set up output file path templates -------------- +# --------- IMAGES --------- +IM_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-imbinfactor{ibf}-en{lo}_{hi}keV-image.fits", +) +# -------------------------- + + +# -------- EXPMAPS --------- +EX_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-attraddelta{rd}arcmin-" + "attphibin{npb}-imbinfactor{ibf}-enALL-expmap.fits", +) +# -------------------------- + + +# ------ LIGHTCURVES ------- +LC_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" + "-lightcurve.fits", +) + +BACK_LC_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" + "-back-lightcurve.fits", +) +# -------------------------- + + +# -------- SPECTRA --------- +SP_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" + "enALL-spectrum.fits", +) + +BACK_SP_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "enALL-back-spectrum.fits", +) +# -------------------------- + +# ----- GROUPEDSPECTRA ----- +GRP_SP_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum", "-{gt}grp{gs}-spectrum") +# -------------------------- + +# ---------- RMF ----------- +RMF_PATH_TEMP = os.path.join( + OUT_PATH, "{oi}", "xrism-xtend-obsid{oi}-dataclass{xdc}.rmf" +) +# -------------------------- + +# ---------- ARF ----------- +ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") +# -------------------------- +# -------------------------------------------------------------- ``` ### Configuration @@ -1327,9 +1395,6 @@ addressing a large number of observations (or making many images per observation is a good idea to run them in parallel! - -***NEED TO APPLY GTIS TO IMAGE GENERATION AS WELL*** - ```{code-cell} python arg_combs = [ [ @@ -1351,11 +1416,7 @@ with mp.Pool(NUM_CORES) as p: Once again we set up a template variable for output image file names: ```{code-cell} python -im_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-imbinfactor{ibf}-en{lo}_{hi}keV-image.fits", -) + ``` ### New XRISM-Xtend exposure maps @@ -1398,12 +1459,7 @@ with mp.Pool(NUM_CORES) as p: Set up a template variable for output exposure map file names: ```{code-cell} python -ex_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-attraddelta{rd}arcmin-" - "attphibin{npb}-imbinfactor{ibf}-enALL-expmap.fits", -) + ``` ## 4. Generating new XRISM-Xtend spectra and light curves @@ -1492,7 +1548,7 @@ chos_im_en = xtd_im_en_bounds[0].to("keV") oi_skypix_wcs = {} for oi, cur_dcs in rel_dataclasses.items(): for dc in cur_dcs: - cur_im_path = im_path_temp.format( + cur_im_path = IM_PATH_TEMP.format( oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value ) cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) @@ -1547,8 +1603,6 @@ lc_time_bin = Quantity(200, "s") xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") ``` -***NEEEEEEEED GTI*** - ```{code-cell} python arg_combs = [ [ @@ -1571,19 +1625,7 @@ with mp.Pool(NUM_CORES) as p: Create template variables for source and background light curves: ```{code-cell} python -lc_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" - "-lightcurve.fits", -) -back_lc_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" - "-back-lightcurve.fits", -) ``` ### New XRISM-Xtend spectra and supporting files @@ -1617,18 +1659,7 @@ with mp.Pool(NUM_CORES) as p: Create template variables for source and background spectrum files: ```{code-cell} python -sp_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" - "enALL-spectrum.fits", -) -back_sp_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "enALL-back-spectrum.fits", -) ``` #### Calculating 'BACKSCAL' for new XRISM-Xtend spectra @@ -1639,14 +1670,14 @@ back_sp_path_temp = os.path.join( for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up the path to input source and background spectra - cur_spec = sp_path_temp.format( + cur_spec = SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), ) - cur_bspec = back_sp_path_temp.format( + cur_bspec = BACK_SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, ra=src_coord.ra.value.round(6), @@ -1654,7 +1685,7 @@ for oi, dcs in rel_dataclasses.items(): ) # Also need to pass an exposure map, so set up a path to that - cur_ex = ex_path_temp.format( + cur_ex = EX_PATH_TEMP.format( oi=oi, xdc=cur_dc, rd=expmap_rad_delta.to("arcmin").value, @@ -1695,7 +1726,6 @@ output grouped spectral files: spec_group_type = "min" spec_group_scale = 1 -grp_sp_path_temp = sp_path_temp.replace("-spectrum", "-{gt}grp{gs}-spectrum") ``` Now we run the grouping tool - though this time we do not parallelize the task, as @@ -1712,14 +1742,14 @@ the other processing steps in this notebook. for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up relevant paths to the input and output spectrum - cur_spec = sp_path_temp.format( + cur_spec = SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), ) - cur_grp_spec = grp_sp_path_temp.format( + cur_grp_spec = grp_SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, gt=spec_group_type, @@ -1745,7 +1775,7 @@ for oi, dcs in rel_dataclasses.items(): arg_combs = [ [ oi, - sp_path_temp.format( + SP_PATH_TEMP.format( oi=oi, xdc=dc, ra=src_coord.ra.value.round(6), @@ -1763,9 +1793,7 @@ with mp.Pool(NUM_CORES) as p: ``` ```{code-cell} python -rmf_path_temp = os.path.join( - OUT_PATH, "{oi}", "xrism-xtend-obsid{oi}-dataclass{xdc}.rmf" -) + ``` #### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation @@ -1804,21 +1832,21 @@ arg_combs = [ [ oi, os.path.join(OUT_PATH, oi), - ex_path_temp.format( + EX_PATH_TEMP.format( oi=oi, xdc=dc, rd=expmap_rad_delta.to("arcmin").value, npb=expmap_phi_bins, ibf=1, ), - sp_path_temp.format( + SP_PATH_TEMP.format( oi=oi, xdc=dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), ), - rmf_path_temp.format(oi=oi, xdc=dc), + RMF_PATH_TEMP.format(oi=oi, xdc=dc), radec_src_reg_path, arf_rt_num_photons, ] @@ -1831,7 +1859,6 @@ with mp.Pool(NUM_CORES) as p: ``` ```{code-cell} python -arf_path_temp = sp_path_temp.replace("-spectrum.fits", ".arf") ``` ```{warning} @@ -1886,7 +1913,7 @@ xs.AllData.clear() # Set up the paths to grouped source spectrum, ungrouped background # spectrum, RMF, and ARF files -cur_spec = grp_sp_path_temp.format( +cur_spec = grp_SP_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, gt=spec_group_type, @@ -1896,19 +1923,19 @@ cur_spec = grp_sp_path_temp.format( rad=src_reg_rad.to("deg").value.round(4), ) -cur_bspec = back_sp_path_temp.format( +cur_bspec = BACK_SP_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), ) -cur_rmf = rmf_path_temp.format( +cur_rmf = RMF_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, ) -cur_arf = arf_path_temp.format( +cur_arf = ARF_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, ra=src_coord.ra.value.round(6), From 24802d6872a78b8d063a94cc78246a2cfd0f7a1d Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 10:47:45 -0500 Subject: [PATCH 046/221] Replaced the im_out definition in gen_xrism_xtend_image with something that uses the template path variable set up in the constants section of getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 079734e1..f462fd8c 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -250,10 +250,10 @@ def gen_xrism_xtend_image( evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" # Set up the output file name for the image we're about to generate. - im_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"imbinfactor{im_bin}-en{lo_en_val}_{hi_en_val}keV-image.fits" - ) + im_out = os.path.basename(IM_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + ibf=im_bin, lo=lo_en_val, + hi=hi_en_val) # Create a temporary working directory temp_work_dir = os.path.join( From 569316c324d4da647bed772696808a4fc358cf0f Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 10:51:17 -0500 Subject: [PATCH 047/221] Replaced the ex_out and binned_ex_out definitions in gen_xrism_xtend_expmap with something that uses the template path variable set up in the constants section of getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f462fd8c..a5d64ed3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -350,21 +350,25 @@ def gen_xrism_xtend_expmap( out_map_type = "EXPOSURE" ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) + # Set up the output file name for the exposure map we're about to generate. - ex_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"attraddelta{radial_delta}arcmin-attphibin{num_phi_bin}-" - f"imbinfactor1-enALL-{ex_type}.fits" - ) + ex_out = os.path.basename(EX_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + rd=radial_delta, + npb=num_phi_bin, + ibf=1) # If the user wants to bin up the exposure map, we'll need to set up another # output file name with the bin factor set to the input value (this variable # is not used if the user does not want to bin the map) - binned_ex_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"attraddelta{radial_delta}arcmin-attphibin{num_phi_bin}-" - f"imbinfactor{im_bin}-enALL-{ex_type}.fits" - ) + binned_ex_out = os.path.basename(EX_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + rd=radial_delta, + npb=num_phi_bin, + ibf=im_vbin) # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) From c7c749c5668c9b2c43bc91d2de127861f8a99ef5 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 11:25:16 -0500 Subject: [PATCH 048/221] Altered the light curve and spectrum generation wrapper functions in the getting-started-xrism-xtend.md notebook, so that they use the constant template variables to name their output files. Also made light curve generation wrapper take ra,dec,radius information for the light curve file names, and moved the spectrum generation wrapper to extract ObsID and data class from the passed event file. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 112 ++++++++++++------ 1 file changed, 74 insertions(+), 38 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a5d64ed3..8d77d4a3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -420,6 +420,8 @@ def gen_xrism_xtend_expmap( def gen_xrism_xtend_lightcurve( event_file: str, out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, src_reg_file: str, back_reg_file: str, lo_en: Quantity = Quantity(0.6, "keV"), @@ -437,6 +439,14 @@ def gen_xrism_xtend_lightcurve( necessarily) we wish to generate a XRISM-Xtend light curve from. ObsID and dataclass information will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. + :param SkyCoord rel_src_coord: The source coordinate (RA, Dec) of the + source region for which we wish to generate a light curve. + :param Quantity rel_src_radius: The radius of the source region for which we wish + to generate a light curve. + :param str src_reg_file: Path to the region file defining the source region for + which we wish to generate a light curve. + :param str back_reg_file: Path to the region file defining the background region + for which we wish to generate a light curve. :param Quantity lo_en: Lower bound of the energy band within which we will generate the light curve. :param Quantity hi_en: Upper bound of the energy band within which we will @@ -453,6 +463,11 @@ def gen_xrism_xtend_lightcurve( cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] + # Get RA, Dec, and radius values in the right format + ra_val = rel_src_coord.ra.to("deg").value.round(6) + dec_val = rel_src_coord.dec.to("deg").value.round(6) + rad_val = rel_src_radius.to("deg").value.round(4) + # Check the units of the passed time bin size - also if the passed value is # a float or integer, we'll assume it is in seconds if not isinstance(time_bin_size, Quantity): @@ -487,14 +502,26 @@ def gen_xrism_xtend_lightcurve( evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" # Set up the output file name for the light curve we're about to generate. - lc_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"en{lo_en_val}_{hi_en_val}keV-expthresh{lc_bin_thresh}-tb{time_bin_size}s-" - f"lightcurve.fits" - ) - # The same file name, but with 'lightcurve' changed to 'back-lightcurve', for the - # background light curve. - lc_back_out = lc_out.replace("lightcurve", "back-lightcurve") + lc_out = os.path.basename(LC_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val, + rad=rad_val, + lo=lo_en_val, + hi=hi_en_val, + lct=lc_bin_thresh, + tb=time_bin_size) + + # The same file name, but with 'lightcurve' changed to 'back-lightcurve', and the + # radius information information removed, for the background light curve. + lc_back_out = os.path.basename(BACK_LC_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val, + lo=lo_en_val, + hi=hi_en_val, + lct=lc_bin_thresh, + tb=time_bin_size) # Create a temporary working directory temp_work_dir = os.path.join( @@ -544,8 +571,6 @@ def gen_xrism_xtend_lightcurve( def gen_xrism_xtend_spectrum( - cur_obs_id: str, - cur_xtend_data_class: str, event_file: str, out_dir: str, rel_src_coord: SkyCoord, @@ -554,40 +579,47 @@ def gen_xrism_xtend_spectrum( back_reg_file: str, ): """ - IMPLICITLY ASSUMES THE REGION IS A CIRCLE + Function that wraps the HEASoftPy interface to the HEASoft extractor tool, set + up to generate spectra from XRISM-Xtend observations. The function will + generate a spectrum for the source region and a background spectrum for + the background region. - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend spectrum. - :param str cur_xtend_data_class: - :param str event_file: + :param str event_file: Path to the event list (usually cleaned, but not + necessarily) we wish to generate a XRISM-Xtend spectrum from. ObsID and + dataclass information will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. - :return: A tuple containing the processed ObsID, the log output of the - pipeline, and a boolean flag indicating success (True) or failure (False). - :rtype: Tuple[str, hsp.core.HSPResult, bool] + :param SkyCoord rel_src_coord: The source coordinate (RA, Dec) of the + source region for which we wish to generate a light curve. + :param Quantity rel_src_radius: The radius of the source region for which we wish + to generate a light curve. + :param str src_reg_file: Path to the region file defining the source region for + which we wish to generate a light curve. + :param str back_reg_file: Path to the region file defining the background region + for which we wish to generate a light curve. """ - # Validity check on the passed data class - if cur_xtend_data_class[0] != "3": - raise ValueError( - f"The first digit of the Xtend data class ({cur_xtend_data_class}) " - "must be 3 for in-flight data." - ) + # We can extract the ObsID and data class directly from the header of the event + # list - it is safer than having them be passed to this function separately. + with fits.open(event_file) as read_evto: + cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] + cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] # Get RA, Dec, and radius values in the right format ra_val = rel_src_coord.ra.to("deg").value.round(6) dec_val = rel_src_coord.dec.to("deg").value.round(6) rad_val = rel_src_radius.to("deg").value.round(4) - # Set up the output file name for the light curve we're about to generate. + # Set up the output file names for the source and background spectra we're + # about to generate. sp_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-enALL-" f"spectrum.fits" ) - # The same file name, but with 'spectrum' changed to 'back-spectrum', for the - # background light curve. - sp_back_out = sp_out.replace(f"-radius{rad_val}deg", "").replace( - "spectrum", "back-spectrum" - ) + back_sp_out = os.path.basename(BACK_SP_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val) # Create a temporary working directory temp_work_dir = os.path.join( @@ -601,7 +633,7 @@ def gen_xrism_xtend_spectrum( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): src_out = hsp.extractor( filename=event_file, - phafile=os.path.join("..", sp_out), + phafile=sp_out, regionfile=src_reg_file, xcolf="X", ycolf="Y", @@ -614,7 +646,7 @@ def gen_xrism_xtend_spectrum( # Now for the background light curve back_out = hsp.extractor( filename=event_file, - phafile=os.path.join("..", sp_back_out), + phafile=sp_back_out, regionfile=back_reg_file, xcolf="X", ycolf="Y", @@ -624,6 +656,10 @@ def gen_xrism_xtend_spectrum( clobber=True, ) + # Move the spectra up from the temporary directory + os.rename(os.path.join(temp_work_dir, sp_out), os.path.join(out_dir, sp_out)) + os.rename(os.path.join(temp_work_dir, sp_back_out), os.path.join(out_dir, sp_back_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -768,15 +804,15 @@ EX_PATH_TEMP = os.path.join( LC_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" - "-lightcurve.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" \ + "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-lightcurve.fits", ) BACK_LC_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" - "-back-lightcurve.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" \ + "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-back-lightcurve.fits", ) # -------------------------- @@ -792,7 +828,7 @@ SP_PATH_TEMP = os.path.join( BACK_SP_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "enALL-back-spectrum.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-enALL-back-spectrum.fits", ) # -------------------------- @@ -1612,6 +1648,8 @@ arg_combs = [ [ evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), + src_coord, + src_reg_rad, obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), *cur_bnds, @@ -1643,8 +1681,6 @@ Create template variables for source and background light curves: ```{code-cell} python arg_combs = [ [ - oi, - dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), src_coord, From 4565f939098502078b62ac1c3f7c7cf86d7962b4 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 11:31:07 -0500 Subject: [PATCH 049/221] Updated the gen_xrism_xtend_rmf wrapper function so that it doesn't require an ObsID input, the ObsID is extracted from the passed spectrum. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 8d77d4a3..b8f4bd05 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -666,13 +666,20 @@ def gen_xrism_xtend_spectrum( return [src_out, back_out] -def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): +def gen_xrism_xtend_rmf(spec_file: str, out_dir: str): """ + A wrapper around the XRISM-Xtend-specific RMF generation tool implemented as + part of HEASoft (and called here through HEASoftPy). - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend RMF. + :param str spec_file: The path to the spectrum file for which to generate an RMF. :param str out_dir: The directory where output files should be written. """ + # We can extract the ObsID directly from the header of the spectrum file - it is + # safer than having the user pass it separately + with fits.open(spec_file) as read_speco: + cur_obs_id = read_speco[0].header["OBS_ID"] + # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xtdrmf_{}".format(randint(0, int(1e8)))) os.makedirs(temp_work_dir) @@ -686,11 +693,14 @@ def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.xtdrmf( infile=spec_file, - outfile=os.path.join("..", rmf_out), + outfile=rmf_out, noprompt=True, clobber=True, ) + # Move the RMF up from the temporary directory + os.rename(os.path.join(temp_work_dir, rmf_out), os.path.join(out_dir, rmf_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1814,7 +1824,6 @@ for oi, dcs in rel_dataclasses.items(): ```{code-cell} python arg_combs = [ [ - oi, SP_PATH_TEMP.format( oi=oi, xdc=dc, From 89c8fa21e6037b35fe9e7da990f5567cac097c66 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 11:47:44 -0500 Subject: [PATCH 050/221] Polished the gen_xrism_xtend_arf wrapper function a little, making sure that re-runs would use previously generated matching ray-traced event files, and adding a docstring. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 49 ++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b8f4bd05..989c25dc 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -708,7 +708,6 @@ def gen_xrism_xtend_rmf(spec_file: str, out_dir: str): def gen_xrism_xtend_arf( - cur_obs_id: str, out_dir: str, expmap_file: str, spec_file: str, @@ -717,12 +716,36 @@ def gen_xrism_xtend_arf( num_photons: int, ): """ - IMPLICITLY ASSUMES THAT WE'RE GENERATING AN ARF FOR A 'POINT SOURCE' + A wrapper function for the HEASoft `xaarfgen` task, which we use to generate + ARFs for XRISM-Xtend spectra. + + IMPORTANT: The way we have set up the call to `xaarfgen` implicitly assumes that + the spectrum was generated for a POINT SOURCE. Using this setup to generate + an ARF for an extended source WOULD NOT BE VALID. + + This function can take a long time to run, primarily because of the ray-tracing + step (and the acquisition of a large CalDB file necessary for this step, if + using remote CalDB). The ray-tracing time will scale with the value + of 'num_photons', with the XRISM team estimating ~1 minute per 100,000 photons + (though note this does not include time to download the previously mentioned + CalDB file). - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend ARF. :param str out_dir: The directory where output files should be written. + :param str expmap_file: The path to the exposure map file necessary to generate + the ARF. + :param str spec_file: The path to the spectrum file for which to generate an ARF. + :param str rmf_file: The path to the RMF file necessary to generate an ARF. + :param str src_radec_reg_file: The path to the region file defining the source + region for which to generate an ARF. + :param int num_photons: The number of photons to simulate in the ray-tracing + portion of XRISM-Xtend ARF generation. """ + # We can extract the ObsID directly from the header of the spectrum file - it is + # safer than having the user pass it separately + with fits.open(spec_file) as read_speco: + cur_obs_id = read_speco[0].header["OBS_ID"] + # Spectrum files generated in this demonstration notebook contain RA-Dec # information in their file name, so we will read it out from there radec_sec = os.path.basename(spec_file).split("-radius")[0].split("-ra")[1] @@ -743,13 +766,22 @@ def gen_xrism_xtend_arf( f"enALL-raytracedevents.fits" ) + # If a ray-traced event file with the same already exists, we're just going + # to point to it with the absolute path (saves on re-running expensive + # ray tracing). + if os.path.exists(os.path.join(os.path.abspath(out_dir), ray_traced_evt_out)): + ray_traced_exists = True + ray_traced_evt_out = os.path.abspath(os.path.join(out_dir, ray_traced_evt_out)) + else: + ray_traced_exists = False + # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.xaarfgen( - xrtevtfile=os.path.join("..", ray_traced_evt_out), - outfile=os.path.join("..", arf_out), + xrtevtfile=ray_traced_evt_out, + outfile=arf_out, sourcetype="POINT", numphotons=num_photons, source_ra=ra_val, @@ -764,6 +796,12 @@ def gen_xrism_xtend_arf( clobber=True, ) + # Move the ARF and ray traced event files up from the temporary directory + os.rename(os.path.join(temp_work_dir, arf_out), os.path.join(out_dir, arf_out)) + # If the ray traced file already existed, we don't need to move anything + if not ray_traced_exists: + os.rename(os.path.join(temp_work_dir, ray_traced_evt_out), os.path.join(out_dir, ray_traced_evt_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1879,7 +1917,6 @@ xrtraytrace: Additional updates to fix and enable remote CalDB ```{code-cell} python arg_combs = [ [ - oi, os.path.join(OUT_PATH, oi), EX_PATH_TEMP.format( oi=oi, From cfdaab6cbc39a5683559b0d5cb9bf2d0b752fc1e Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 12:31:22 -0500 Subject: [PATCH 051/221] Added much commentary to the exposure map generation section of the getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 68 ++++++++++--------- 1 file changed, 37 insertions(+), 31 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 989c25dc..1a72ba80 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1249,7 +1249,7 @@ output being the cleaned event list(s); remember that one observation can produc We will now demonstrate how to generate new XRISM-Xtend data products tailored to your scientific needs. Images and exposure maps can be generated for the entire -field-of-view (FoV), rather than having to focus on a particular source, so we will +field-of-view (FoV; or at least the entire FoV of a particular observation mode, e.g., full window, 1/8th window, etc.), rather than having to focus on a particular source, so we will start with them. ### Converting energy bounds to channel bounds @@ -1482,7 +1482,6 @@ Image generation is not a particularly computationally intensive task, but if yo addressing a large number of observations (or making many images per observation), it is a good idea to run them in parallel! - ```{code-cell} python arg_combs = [ [ @@ -1501,26 +1500,51 @@ with mp.Pool(NUM_CORES) as p: im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` -Once again we set up a template variable for output image file names: +### New XRISM-Xtend exposure maps -```{code-cell} python +We also generate exposure maps for the entire FoV of a particular observation mode, rather +than for a particular source. The exposure map serves both to tell us the exposure time +at any given pixel of our image (assuming the image and exposure map are spatially +binned the same way), and also as a useful way to tell exactly which parts of the sky +are covered by the observation. -``` +The latter capability is of particular importance for the generation/analysis of +spectra, and the creation of Ancillary Response Files (ARFs), which describe the +effective sensitivity of Xtend as a function of energy. -### New XRISM-Xtend exposure maps +Unlike for image creation, XRISM does have a dedicated HEASoft task for the +generation of exposure maps; `xaexpmap`. We have once again set up a wrapper function +in the 'Global Setup: Functions' section of this notebook to make it easier to run +this task in parallel. + +There are two `xaexpmap` configuration options which control how the +attitude (essentially pointing) of XRISM over the course of the observation is +binned spatially. These bins ('off-axis wedges' as the +[`xaexpmap` documentation](https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html) +describes them) are where the initial 'time intervals' of observation coverage are calculated: +- **Radial Delta** - Passed to `xaexpmap` as `delta`. Radial increment (in arcmin) for the annular grid for which the attitude histogram will be calculated. The annuli are centered on the optical axis (off-axis angle = 0), and the central circle has a radius equal to `delta`. +- **Number of azimuthal bins** - Passed to `xaexpmap` as `numphi`. Number of azimuth (phi) bins in the first annular region over which attitude histogram bins will be calculated (i.e., this annular region lies between `delta` and 2*`delta` arcmin from the center of the annuli). The zeroth annular region is a full circle of radius `delta` and the nth annular region has an outer radius of (n+1)*`delta`, and `numphi`*n azimuthal bins. -Exposure maps... +The documentation for `xaexpmap` notes that you can force the attitude histogram to have a single bin, by choosing a radial delta that is much larger than any expected attitude variation during an observation. +We choose to create exposure maps from only one attitude histogram bin, by passing a large radial delta and requiring a single azimuthal bin: ```{code-cell} python expmap_rad_delta = Quantity(20, "arcmin") expmap_phi_bins = 1 ``` +Our wrapper function for `xaexpmap` also contains an optional call to the HEASoft FTOOLS `fimgbin` task, which +is used to re-bin the exposure map to a coarser spatial resolution (in this case to match the second +binning factor we generated images for). + +The `xaexpmap` task creates exposure maps with 1 image pixel per Sky X-Y coordinate system pixel, so we +only need to specify binning factors **that do not equal 1** here (adding 1 to this +list would be redundant and would waste compute time): ```{code-cell} python expmap_bin_factors = [4] ``` -***WHAT ABOUT THIS GTI?? - xa000128000xtd_mode.gti*** +Finally, we run the exposure map generation: ```{code-cell} python arg_combs = [ @@ -1544,17 +1568,11 @@ with mp.Pool(NUM_CORES) as p: ex_result = p.starmap(gen_xrism_xtend_expmap, arg_combs) ``` -Set up a template variable for output exposure map file names: - -```{code-cell} python - -``` - ## 4. Generating new XRISM-Xtend spectra and light curves In this section we will demonstrate how to generate source-specific data products from XRISM-Xtend observations; light curves and spectra (along with supporting files like -RMFs and Ancillary Response Files, or ARFs). +RMFs and ARFs). Rather than extracting spectra and light curves for the entire XRISM-Xtend FoV, *which is how the quick-look spectra and light curves contained in the archive are made*, we @@ -1712,17 +1730,10 @@ with mp.Pool(NUM_CORES) as p: lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) ``` -Create template variables for source and background light curves: - -```{code-cell} python - -``` ### New XRISM-Xtend spectra and supporting files -```{code-cell} python -``` #### Generating the spectral files @@ -1879,9 +1890,6 @@ with mp.Pool(NUM_CORES) as p: rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) ``` -```{code-cell} python - -``` #### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation @@ -1911,7 +1919,6 @@ xrtraytrace: Additional updates to fix and enable remote CalDB usage with xrtraytrace and xaarfgen ("timeout" interval extended for reading large CalDB files). - ``` ```{code-cell} python @@ -1944,18 +1951,15 @@ with mp.Pool(NUM_CORES) as p: arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) ``` -```{code-cell} python -``` - ```{warning} Due to the high-fidelity ray-tracing method used to calculate XRISM ARFs, the runtime -of this step can be on the order of hours. We have to do ***FINISH*** +of this step can be on the order of hours. ``` ## 5. Fitting spectral models to XRISM-Xtend spectra Finally, to show off the XRISM-Xtend products we just generated, we will perform -a very simple model fit to one of our spectra. +a simple model fit to one of our spectra. Our demonstration of spectral model fitting to an XRISM-Xtend spectrum will be performed using the [PyXspec](https://heasarc.gsfc.nasa.gov/docs/software/xspec/python/html/index.html) package. @@ -2076,6 +2080,8 @@ HEASoftPy HEASARC Page: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/hea HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html +HEASoft XRISM `xaexpmap` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html + ### Acknowledgements From ec8fef69d070112fe0c1a99688a6eb01bed56c5b Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 15:19:31 -0500 Subject: [PATCH 052/221] Spent an inordinate amount of time adding a subsection on the calibration source region files, how to convert them to different coordinate systems, and then plot them on images. In the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 119 +++++++++++++++--- 1 file changed, 103 insertions(+), 16 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 1a72ba80..b306da10 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -96,7 +96,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc -from regions import CircleSkyRegion +from regions import CircleSkyRegion, Regions from xga.products import Image ``` @@ -1500,6 +1500,10 @@ with mp.Pool(NUM_CORES) as p: im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` +#### Problematic pixels + + + ### New XRISM-Xtend exposure maps We also generate exposure maps for the entire FoV of a particular observation mode, rather @@ -1605,12 +1609,31 @@ effect is sometimes referred to as **cross-talk** or **spatial-spectral mixing ( for this effect is complicated and time-consuming, so our demonstration will focus on a point source, and extended sources will be discussed in another notebook. -### Setting up data product source and background extraction regions +### Setting up source and background extraction regions + +To define exactly where we want to extract events from to build our data products, we +will construct 'region files' in the commonly-used 'DS9' format. + +In this demonstration we will not provide guidance on how to choose particular +source/background regions for your science case, or give detailed information +about the DS9 region format and its capabilities. -There are different ways to define.... +Instead, we will show you how to construct basic region files using the +astropy-affiliated `regions` module. + +Most high-energy missions use three common coordinate systems: +- **Detector (DET) X-Y** - This coordinate system is aligned with the detector; a coordinate in this system will always represent the same physical position on the detector. +- **Sky X-Y** - A transformed version of the DET X-Y coordinate system, aligned with the roll angle of the telescope. **Within a single observation**, a Sky X-Y coordinate will always represent the same physical position on the sky. +- **RA-DEC** - The familiar right ascension and declination coordinate system. + +You need to be careful about which coordinate system you use with which tools, as some +tasks will not accept regions in all coordinate systems. #### General RA-DEC region files +We define a `CircleSkyRegion` instance (a class of the `regions` module) centered on +our target source, with a radius of 2 arcminutes. + ```{code-cell} python # Where to write the new region file radec_src_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_src.reg") @@ -1625,7 +1648,8 @@ src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) src_reg.write(radec_src_reg_path, format="ds9", overwrite=True) ``` -We do the same to define a region from which to extract a background spectrum: +We do the same to define a region from which to extract a background spectrum, though +this region is of a different size and is not centered on the source: ```{code-cell} python # Where to write the new region file @@ -1644,6 +1668,80 @@ back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) back_reg.write(radec_back_reg_path, format="ds9", overwrite=True) ``` +#### Excluding the XRISM-Xtend calibration sources + +The XRISM-Xtend instrument has two calibration sources in its FoV, which present as +bright circles on opposite edges of the detector. While highly useful for the +calibration of Xtend's energy scale, we do not want to accidentally include +calibration events in spectra or light curves we extract. + +As such, we have to define regions to exclude these sources from our data products. + +This will be quite simple, as HEASoft includes a pre-made region file that defines the +location and extent of the emission from the calibration sources. + +A small difficulty arises from the fact that this pre-made region file is defined in +detector coordinates, rather than the RA-Dec coordinates we've been using so far. That +will be pretty easy to deal with, however, as HEASoft includes a tool to transform region +files between different coordinate systems. + +```{code-cell} python +# The path to the included calibration source region file +detpix_xtend_calib_reg_path = os.path.join( + os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" +) + +# Setting up the output file for the RA-Dec calibration source regions +radec_xtend_calib_reg_path = os.path.join(OUT_PATH, "{oi}", "radec_{oi}_calsrc.reg") +``` + +We only need to generate one RA-Dec calibration source region file for each +observation, as the observations with data from multiple data modes share a common +coordinate system. + +The HEASoft tool `coordpnt` is used to perform the transformation from detector to +RA-Dec coordinates - it requires pointing coordinate and telescope roll angle +information which we extract from previously generated image files. + +```{code-cell} python +for oi, dcs in rel_dataclasses.items(): + cur_im_path = IM_PATH_TEMP.format(oi=oi, xdc=dcs[0], ibf=1, lo=chos_im_en[0].value, + hi=chos_im_en[1].value) + with fits.open(cur_im_path) as imago: + cur_pnt_ra = imago[0].header['RA_PNT'] + cur_pnt_dec = imago[0].header['DEC_PNT'] + cur_roll_ang = imago[0].header['PA_NOM'] + + # Call the HEASoft task that converts from detector coordinates to RA-Dec + hsp.coordpnt(input=detpix_xtend_calib_reg_path, + outfile=radec_xtend_calib_reg_path.format(oi=oi), + telescop='XRISM', instrume='XTEND', ra=cur_pnt_ra, dec=cur_pnt_dec, + roll=cur_roll_ang, startsys='DET', stopsys='RADEC', clobber=True) +``` + +Finally, we have to pull the RA-Dec calibration regions from the transformed region +file. The `regions` module provides functions to read-in region files in +various formats and coordinate systems but as the calibration region file was defined +to subtract the regions (each region in the file has a negative sign applied to it), +we need to manually read in the file. + +Note that we also prepend a global color to the region string, so that when we plot +the calibration regions, they will appear as a different color than the source and +background regions: + +```{code-cell} python +cal_regs = {} +for oi in rel_obsids: + with open(radec_xtend_calib_reg_path.format(oi=oi), 'r') as calbo: + cur_cal_reg_str = "global color=white\n" + calbo.read().replace('-ellipse', 'ellipse') + # The '.regions' just retrieves a list of region objects, we don't need to + # keep the calibration regions in the regions module 'Regions' class they + # are read into + cal_regs[oi] = Regions.parse(cur_cal_reg_str, format='ds9').regions +cal_regs +``` + + #### Visualizing the source and background extraction regions on XRISM-Xtend images Examining... @@ -1658,23 +1756,12 @@ for oi, cur_dcs in rel_dataclasses.items(): oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value ) cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) - cur_im.regions = [src_reg, back_reg] + cur_im.regions = [src_reg, back_reg] + cal_regs[oi] cur_im.view(src_coord_quant, zoom_in=True, view_regions=True) oi_skypix_wcs.setdefault(oi, cur_im.radec_wcs) ``` -#### Excluding the XRISM-Xtend calibration sources - -***HAVE TO LOAD THE CALIBRATION SOURCE REGIONS, CONVERT TO SKY PIX SYSTEM, AND EXCLUDE THEM FROM THE EXTRACTION REGION DEFINITIONS*** - -***THIS FILE IS SET UP FOR EXCLUSION (I.E. WITH - IN FRONT OF REGIONS) SO THE REGIONS MODULE WON'T READ THEM IN, AT LEAST BY DEFAULT*** - -```{code-cell} python -detpix_xtend_calib_reg_path = os.path.join( - os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" -) -``` #### Observation specific sky-pixel coordinate region files From eea00353cc366e2f87e25b3d107fc02f9f0450f0 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 16:30:09 -0500 Subject: [PATCH 053/221] Altered where in the getting-started-xrism-xtend.md notebook the RA-Dec system region files are written. Also made sure both those region files and the Sky X-Y coordinate ones exclude calibration source regions. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 129 ++++++++++-------- 1 file changed, 70 insertions(+), 59 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b306da10..e36cefdd 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1623,51 +1623,12 @@ astropy-affiliated `regions` module. Most high-energy missions use three common coordinate systems: - **Detector (DET) X-Y** - This coordinate system is aligned with the detector; a coordinate in this system will always represent the same physical position on the detector. -- **Sky X-Y** - A transformed version of the DET X-Y coordinate system, aligned with the roll angle of the telescope. **Within a single observation**, a Sky X-Y coordinate will always represent the same physical position on the sky. +- **Sky X-Y** - A transformed version of the DETX-DETY coordinate system, aligned with the roll angle of the telescope. **Within a single observation**, a Sky X-Y coordinate will always represent the same physical position on the sky. - **RA-DEC** - The familiar right ascension and declination coordinate system. You need to be careful about which coordinate system you use with which tools, as some tasks will not accept regions in all coordinate systems. -#### General RA-DEC region files - -We define a `CircleSkyRegion` instance (a class of the `regions` module) centered on -our target source, with a radius of 2 arcminutes. - -```{code-cell} python -# Where to write the new region file -radec_src_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_src.reg") - -# The radius of the source extraction region -src_reg_rad = Quantity(2, "arcmin") - -# Setting up a 'regions' module circular sky region instance -src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) - -# Write the source region to a region file -src_reg.write(radec_src_reg_path, format="ds9", overwrite=True) -``` - -We do the same to define a region from which to extract a background spectrum, though -this region is of a different size and is not centered on the source: - -```{code-cell} python -# Where to write the new region file -radec_back_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_back.reg") - -# The central coordinate of the background region -back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") - -# The radius of the background region -back_reg_rad = Quantity(3, "arcmin") - -# Setting up a 'regions' module circular sky region instance for the background region -back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) - -# Once again writing the region to a region file as well -back_reg.write(radec_back_reg_path, format="ds9", overwrite=True) -``` - #### Excluding the XRISM-Xtend calibration sources The XRISM-Xtend instrument has two calibration sources in its FoV, which present as @@ -1721,26 +1682,53 @@ for oi, dcs in rel_dataclasses.items(): Finally, we have to pull the RA-Dec calibration regions from the transformed region file. The `regions` module provides functions to read-in region files in -various formats and coordinate systems but as the calibration region file was defined -to subtract the regions (each region in the file has a negative sign applied to it), -we need to manually read in the file. +various formats and coordinate systems; it also understands that the regions in +these files are to be excluded (indicated by a '-' prefix). -Note that we also prepend a global color to the region string, so that when we plot -the calibration regions, they will appear as a different color than the source and -background regions: +Note that we also set the calibration region's color to be white so that when we plot +them, they will appear as a different color than the source and background regions: ```{code-cell} python cal_regs = {} for oi in rel_obsids: - with open(radec_xtend_calib_reg_path.format(oi=oi), 'r') as calbo: - cur_cal_reg_str = "global color=white\n" + calbo.read().replace('-ellipse', 'ellipse') - # The '.regions' just retrieves a list of region objects, we don't need to - # keep the calibration regions in the regions module 'Regions' class they - # are read into - cal_regs[oi] = Regions.parse(cur_cal_reg_str, format='ds9').regions + cur_cal_regs = Regions(radec_xtend_calib_reg_path.format(oi=oi), format='ds9') + for cur_reg in cur_cal_regs: + cur_reg.visual['color'] = 'white' + + # The '.regions' just retrieves a list of region objects, we don't need to keep + # the calibration regions in the regions module 'Regions' class they are read into + cal_regs[oi] = cur_cal_regs.regions + cal_regs ``` +#### Source and background RA-DEC region files + +We define a `CircleSkyRegion` instance (a class of the `regions` module) centered on +our target source, with a radius of 2 arcminutes. + +```{code-cell} python +# The radius of the source extraction region +src_reg_rad = Quantity(2, "arcmin") + +# Setting up a 'regions' module circular sky region instance +src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) +``` + +We do the same to define a region from which to extract a background spectrum, though +this region is of a different size and is not centered on the source: + +```{code-cell} python +# The central coordinate of the background region +back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") + +# The radius of the background region +back_reg_rad = Quantity(3, "arcmin") + +# Setting up a 'regions' module circular sky region instance for the background region +back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) +``` + #### Visualizing the source and background extraction regions on XRISM-Xtend images @@ -1762,19 +1750,42 @@ for oi, cur_dcs in rel_dataclasses.items(): oi_skypix_wcs.setdefault(oi, cur_im.radec_wcs) ``` +#### Writing observation-specific RA-Dec and sky-pixel coordinate region files + -#### Observation specific sky-pixel coordinate region files ```{code-cell} python -obs_src_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_src.reg") -obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_back.reg") +# Where to write the new RA-Dec source region file - the double {{}} around 'oi' just +# means that the f-string will fill in the SRC_NAME and leave '{oi}' to be +# formatted later +radec_src_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_src.reg") +# Where to write the new RA-Dec background region file +radec_back_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_back.reg") + +# The file path templates for the source and background Sky X-Y system region files +obs_src_reg_path_temp = os.path.join(OUT_PATH, '{oi}', f"skypix_{{oi}}_{SRC_NAME}_src.reg") +obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_back.reg") for oi in rel_obsids: - src_reg.to_pixel(oi_skypix_wcs[oi]).write( - obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9", overwrite=True + # We set up a combination of the source region and the calibration source + # regions - the calibration regions have been set up to be excluded, which will + # be reflected in the final region files we write + src_comb_regs = Regions([src_reg] + cal_regs[oi]) + # The same but for the background region + back_comb_regs = Regions([back_reg] + cal_regs[oi]) + + # Write the RA-Dec source region file + src_comb_regs.write(radec_src_reg_path.format(oi=oi), format="ds9", overwrite=True) + # And the RA-Dec background region file + back_reg.write(radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True) + + # Now we repeat the exercise, but use the Sky<->RA-Dec WCS information we pulled + # from the images to convert the regions to the Sky X-Y system + src_comb_regs.to_pixel(oi_skypix_wcs[oi]).write( + obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True ) - back_reg.to_pixel(oi_skypix_wcs[oi]).write( - obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9", overwrite=True + back_comb_regs.to_pixel(oi_skypix_wcs[oi]).write( + obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True ) ``` From 07f6b8565e2190247872a49d0e2e63aacffc4690 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 16:53:04 -0500 Subject: [PATCH 054/221] Added commentary to the part of the getting-started-xrism-xtend.md notebook that writes out region files, and the part that plots regions on top of previously generated XRISM-Xtend images. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 29 +++++++++++++++++-- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index e36cefdd..f17c0fc8 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1729,10 +1729,17 @@ back_reg_rad = Quantity(3, "arcmin") back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) ``` - #### Visualizing the source and background extraction regions on XRISM-Xtend images -Examining... +We should inspect the regions in-situ to make sure they look sensible - first, our +previously generated images are loaded in as `Image` class (from the `XGA` Python +module) instances. + +The regions we created are then assigned to each image's `regions`, and +the `.view()` method is called with the `view_regions=True` argument to display them. + +Additionally, we extract the RA-Dec <-> Sky X-Y WCS from one image per observation so +that we can use it later on to transform our RA-Dec regions into Sky X-Y regions. ```{code-cell} python chos_im_en = xtd_im_en_bounds[0].to("keV") @@ -1752,7 +1759,23 @@ for oi, cur_dcs in rel_dataclasses.items(): #### Writing observation-specific RA-Dec and sky-pixel coordinate region files +Now we've set up the regions and visualized them, we'll write them to disk as region +files that can be passed to the HEASoft tasks used to generate spectra and light curves. + +We set up instances of the `Regions` class of the astropy-affiliated `regions` module +for the source and background regions plus calibration regions. The `write()` method +is then used to save a DS9-formatted region file to disk. + +The calibration source regions are set up to be excluded, and the output files will +reflect that. + +We write two versions each of the source and region files, one version in the RA-Dec +coordinate system, and the other in the Sky X-Y system (different tasks have different +requirements for the coordinate system they accept). +Our RA-Dec regions are converted to Sky X-Y using a feature of the `regions` +module, using the WCS information we pulled from the images when we visualized them +with the source, background, and calibration source regions overplotted ```{code-cell} python # Where to write the new RA-Dec source region file - the double {{}} around 'oi' just @@ -1763,7 +1786,7 @@ radec_src_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_sr radec_back_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_back.reg") # The file path templates for the source and background Sky X-Y system region files -obs_src_reg_path_temp = os.path.join(OUT_PATH, '{oi}', f"skypix_{{oi}}_{SRC_NAME}_src.reg") +obs_src_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_src.reg") obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_back.reg") for oi in rel_obsids: From b8b0c146cdf44e30f65692a20c5d316915c26e31 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 17:25:18 -0500 Subject: [PATCH 055/221] The meta data of the functions cell in getting-started-xrism-xtend.md had its formatting screwed up somehow, which was stopping the black formatter from doing its job (I hope). This should fix that. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 369 +++++++++--------- 1 file changed, 194 insertions(+), 175 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f17c0fc8..26184e99 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -106,22 +106,21 @@ from xga.products import Image ```{code-cell} python --- -tags: [hide - input] +tags: [hide-input] jupyter: -source_hidden: true + source_hidden: true --- - def process_xrism_xtend( - cur_obs_id: str, - out_dir: str, - evt_dir: str, - attitude: str, - orbit: str, - obs_gti: str, - mkf_filter: str, - file_stem: str, - extended_housekeeping: str, - xtend_housekeeping: str, + cur_obs_id: str, + out_dir: str, + evt_dir: str, + attitude: str, + orbit: str, + obs_gti: str, + mkf_filter: str, + file_stem: str, + extended_housekeeping: str, + xtend_housekeeping: str, ): """ A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process @@ -196,11 +195,11 @@ def process_xrism_xtend( def gen_xrism_xtend_image( - event_file: str, - out_dir: str, - lo_en: Quantity, - hi_en: Quantity, - im_bin: int = 1, + event_file: str, + out_dir: str, + lo_en: Quantity, + hi_en: Quantity, + im_bin: int = 1, ): """ This function wraps the HEASoft 'extractor' tool and is used to spatially bin @@ -250,10 +249,9 @@ def gen_xrism_xtend_image( evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" # Set up the output file name for the image we're about to generate. - im_out = os.path.basename(IM_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - ibf=im_bin, lo=lo_en_val, - hi=hi_en_val) + im_out = os.path.basename(IM_PATH_TEMP).format( + oi=cur_obs_id, xdc=cur_xtend_data_class, ibf=im_bin, lo=lo_en_val, hi=hi_en_val + ) # Create a temporary working directory temp_work_dir = os.path.join( @@ -287,15 +285,15 @@ def gen_xrism_xtend_image( def gen_xrism_xtend_expmap( - event_file: str, - out_dir: str, - gti_file: str, - extend_hk_file: str, - bad_pix_file: str, - pix_gti_file: str = "NONE", - im_bin: int = 1, - radial_delta: Union[float, Quantity] = Quantity(20.0, "arcmin"), - num_phi_bin: int = 1, + event_file: str, + out_dir: str, + gti_file: str, + extend_hk_file: str, + bad_pix_file: str, + pix_gti_file: str = "NONE", + im_bin: int = 1, + radial_delta: Union[float, Quantity] = Quantity(20.0, "arcmin"), + num_phi_bin: int = 1, ): """ Function that wraps the HEASoftPy interface to the XRISM-Xtend 'xaexpmap' @@ -346,29 +344,29 @@ def gen_xrism_xtend_expmap( # Two variants of exposure map can be generated by the function we're about to # call; the default is a map of the integrated exposure time for each pixel, and # the second (not recommended by the documentation) is a flat-fielding map - # TODO REINSTATE WHEN WE HAVE A BETTER UNDERSTANDING OF POTENTIAL USER USES out_map_type = "EXPOSURE" - ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" + # TODO REINSTATE WHEN WE HAVE A BETTER UNDERSTANDING OF POTENTIAL USER USES + # ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) os.makedirs(temp_work_dir) # Set up the output file name for the exposure map we're about to generate. - ex_out = os.path.basename(EX_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - rd=radial_delta, - npb=num_phi_bin, - ibf=1) + ex_out = os.path.basename(EX_PATH_TEMP).format( + oi=cur_obs_id, xdc=cur_xtend_data_class, rd=radial_delta, npb=num_phi_bin, ibf=1 + ) # If the user wants to bin up the exposure map, we'll need to set up another # output file name with the bin factor set to the input value (this variable # is not used if the user does not want to bin the map) - binned_ex_out = os.path.basename(EX_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - rd=radial_delta, - npb=num_phi_bin, - ibf=im_vbin) + binned_ex_out = os.path.basename(EX_PATH_TEMP).format( + oi=cur_obs_id, + xdc=cur_xtend_data_class, + rd=radial_delta, + npb=num_phi_bin, + ibf=im_bin, + ) # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) @@ -408,8 +406,10 @@ def gen_xrism_xtend_expmap( os.rename(os.path.join(temp_work_dir, ex_out), os.path.join(out_dir, ex_out)) # Then do the same for the spatially binned exposure map, if it was requested if im_bin != 1: - os.rename(os.path.join(temp_work_dir, binned_ex_out), - os.path.join(out_dir, binned_ex_out)) + os.rename( + os.path.join(temp_work_dir, binned_ex_out), + os.path.join(out_dir, binned_ex_out), + ) # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -418,16 +418,16 @@ def gen_xrism_xtend_expmap( def gen_xrism_xtend_lightcurve( - event_file: str, - out_dir: str, - rel_src_coord: SkyCoord, - rel_src_radius: Quantity, - src_reg_file: str, - back_reg_file: str, - lo_en: Quantity = Quantity(0.6, "keV"), - hi_en: Quantity = Quantity(13, "keV"), - time_bin_size: Quantity = Quantity(200, "s"), - lc_bin_thresh: float = 0.0, + event_file: str, + out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, + src_reg_file: str, + back_reg_file: str, + lo_en: Quantity = Quantity(0.6, "keV"), + hi_en: Quantity = Quantity(13, "keV"), + time_bin_size: Quantity = Quantity(200, "s"), + lc_bin_thresh: float = 0.0, ): """ Function that wraps the HEASoftPy interface to the HEASoft extractor tool, set @@ -502,26 +502,30 @@ def gen_xrism_xtend_lightcurve( evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" # Set up the output file name for the light curve we're about to generate. - lc_out = os.path.basename(LC_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - ra=ra_val, - dec=dec_val, - rad=rad_val, - lo=lo_en_val, - hi=hi_en_val, - lct=lc_bin_thresh, - tb=time_bin_size) + lc_out = os.path.basename(LC_PATH_TEMP).format( + oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val, + rad=rad_val, + lo=lo_en_val, + hi=hi_en_val, + lct=lc_bin_thresh, + tb=time_bin_size, + ) # The same file name, but with 'lightcurve' changed to 'back-lightcurve', and the # radius information information removed, for the background light curve. - lc_back_out = os.path.basename(BACK_LC_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - ra=ra_val, - dec=dec_val, - lo=lo_en_val, - hi=hi_en_val, - lct=lc_bin_thresh, - tb=time_bin_size) + lc_back_out = os.path.basename(BACK_LC_PATH_TEMP).format( + oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val, + lo=lo_en_val, + hi=hi_en_val, + lct=lc_bin_thresh, + tb=time_bin_size, + ) # Create a temporary working directory temp_work_dir = os.path.join( @@ -562,7 +566,9 @@ def gen_xrism_xtend_lightcurve( # Move the light curves up from the temporary directory os.rename(os.path.join(temp_work_dir, lc_out), os.path.join(out_dir, lc_out)) - os.rename(os.path.join(temp_work_dir, lc_back_out), os.path.join(out_dir, lc_back_out)) + os.rename( + os.path.join(temp_work_dir, lc_back_out), os.path.join(out_dir, lc_back_out) + ) # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -571,12 +577,12 @@ def gen_xrism_xtend_lightcurve( def gen_xrism_xtend_spectrum( - event_file: str, - out_dir: str, - rel_src_coord: SkyCoord, - rel_src_radius: Quantity, - src_reg_file: str, - back_reg_file: str, + event_file: str, + out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, + src_reg_file: str, + back_reg_file: str, ): """ Function that wraps the HEASoftPy interface to the HEASoft extractor tool, set @@ -611,15 +617,12 @@ def gen_xrism_xtend_spectrum( # Set up the output file names for the source and background spectra we're # about to generate. - sp_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-enALL-" - f"spectrum.fits" + sp_out = os.path.basename(SP_PATH_TEMP).format( + oi=cur_obs_id, xdc=cur_xtend_data_class, ra=ra_val, dec=dec_val, rad=rad_val + ) + sp_back_out = os.path.basename(BACK_SP_PATH_TEMP).format( + oi=cur_obs_id, xdc=cur_xtend_data_class, ra=ra_val, dec=dec_val ) - back_sp_out = os.path.basename(BACK_SP_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - ra=ra_val, - dec=dec_val) # Create a temporary working directory temp_work_dir = os.path.join( @@ -658,7 +661,9 @@ def gen_xrism_xtend_spectrum( # Move the spectra up from the temporary directory os.rename(os.path.join(temp_work_dir, sp_out), os.path.join(out_dir, sp_out)) - os.rename(os.path.join(temp_work_dir, sp_back_out), os.path.join(out_dir, sp_back_out)) + os.rename( + os.path.join(temp_work_dir, sp_back_out), os.path.join(out_dir, sp_back_out) + ) # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -675,16 +680,12 @@ def gen_xrism_xtend_rmf(spec_file: str, out_dir: str): :param str out_dir: The directory where output files should be written. """ - # We can extract the ObsID directly from the header of the spectrum file - it is - # safer than having the user pass it separately - with fits.open(spec_file) as read_speco: - cur_obs_id = read_speco[0].header["OBS_ID"] - # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xtdrmf_{}".format(randint(0, int(1e8)))) os.makedirs(temp_work_dir) - # Set up the RMF file name by cannibalising the name of the spectrum file + # Set up the RMF file name by cannibalising the name of the spectrum file - this + # means we don't have to worry about identifying the ObsID rmf_out = os.path.basename(spec_file).split("-ra")[0] + ".rmf" # Using dual contexts, one that moves us into the output directory for the @@ -708,12 +709,12 @@ def gen_xrism_xtend_rmf(spec_file: str, out_dir: str): def gen_xrism_xtend_arf( - out_dir: str, - expmap_file: str, - spec_file: str, - rmf_file: str, - src_radec_reg_file: str, - num_photons: int, + out_dir: str, + expmap_file: str, + spec_file: str, + rmf_file: str, + src_radec_reg_file: str, + num_photons: int, ): """ A wrapper function for the HEASoft `xaarfgen` task, which we use to generate @@ -800,7 +801,10 @@ def gen_xrism_xtend_arf( os.rename(os.path.join(temp_work_dir, arf_out), os.path.join(out_dir, arf_out)) # If the ray traced file already existed, we don't need to move anything if not ray_traced_exists: - os.rename(os.path.join(temp_work_dir, ray_traced_evt_out), os.path.join(out_dir, ray_traced_evt_out)) + os.rename( + os.path.join(temp_work_dir, ray_traced_evt_out), + os.path.join(out_dir, ray_traced_evt_out), + ) # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -826,6 +830,57 @@ TASK_CHATTER = 3 # The approximate linear relationship between Xtend PI and event energy XTD_EV_PER_CHAN = (1 / Quantity(166.7, "chan/keV")).to("eV/chan") +``` + +### Configuration + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +# ------------- Configure global package settings -------------- +# Raise Python exceptions if a heasoftpy task fails +# TODO Remove once this becomes a default in heasoftpy +hsp.Config.allow_failure = False + +# Set up the method for spawning processes. +mp.set_start_method("fork", force=True) +# -------------------------------------------------------------- + +# ------------- Setting how many cores we can use -------------- +NUM_CORES = None +total_cores = os.cpu_count() + +if NUM_CORES is None: + NUM_CORES = total_cores +elif not isinstance(NUM_CORES, int): + raise TypeError( + "If manually overriding 'NUM_CORES', you must set it to an integer value." + ) +elif isinstance(NUM_CORES, int) and NUM_CORES > total_cores: + raise ValueError( + f"If manually overriding 'NUM_CORES', the value must be less than or " + f"equal to the total available cores ({total_cores})." + ) +# -------------------------------------------------------------- + +# -------------- Set paths and create directories -------------- +if os.path.exists("../../../_data"): + ROOT_DATA_DIR = "../../../_data/XRISM/" +else: + ROOT_DATA_DIR = "XRISM/" + +ROOT_DATA_DIR = os.path.abspath(ROOT_DATA_DIR) + +# Make sure the download directory exists. +os.makedirs(ROOT_DATA_DIR, exist_ok=True) + +# Setup path and directory into which we save output files from this example. +OUT_PATH = os.path.abspath("XRISM_output") +os.makedirs(OUT_PATH, exist_ok=True) +# -------------------------------------------------------------- # ------------- Set up output file path templates -------------- @@ -852,14 +907,14 @@ EX_PATH_TEMP = os.path.join( LC_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" \ + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-lightcurve.fits", ) BACK_LC_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" \ + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-back-lightcurve.fits", ) # -------------------------- @@ -896,57 +951,6 @@ ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") # -------------------------------------------------------------- ``` -### Configuration - -```{code-cell} python ---- -tags: [hide-input] -jupyter: - source_hidden: true ---- -# ------------- Configure global package settings -------------- -# Raise Python exceptions if a heasoftpy task fails -# TODO Remove once this becomes a default in heasoftpy -hsp.Config.allow_failure = False - -# Set up the method for spawning processes. -mp.set_start_method("fork", force=True) -# -------------------------------------------------------------- - -# ------------- Setting how many cores we can use -------------- -NUM_CORES = None -total_cores = os.cpu_count() - -if NUM_CORES is None: - NUM_CORES = total_cores -elif not isinstance(NUM_CORES, int): - raise TypeError( - "If manually overriding 'NUM_CORES', you must set it to an integer value." - ) -elif isinstance(NUM_CORES, int) and NUM_CORES > total_cores: - raise ValueError( - f"If manually overriding 'NUM_CORES', the value must be less than or " - f"equal to the total available cores ({total_cores})." - ) -# -------------------------------------------------------------- - -# -------------- Set paths and create directories -------------- -if os.path.exists("../../../_data"): - ROOT_DATA_DIR = "../../../_data/XRISM/" -else: - ROOT_DATA_DIR = "XRISM/" - -ROOT_DATA_DIR = os.path.abspath(ROOT_DATA_DIR) - -# Make sure the download directory exists. -os.makedirs(ROOT_DATA_DIR, exist_ok=True) - -# Setup path and directory into which we save output files from this example. -OUT_PATH = os.path.abspath("XRISM_output") -os.makedirs(OUT_PATH, exist_ok=True) -# -------------------------------------------------------------- -``` - *** ## 1. Finding and downloading XRISM observations of **NAMEHERE** @@ -1348,7 +1352,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - # Set up the figure plt.figure(figsize=(5.5, 5.5)) @@ -1532,6 +1535,7 @@ describes them) are where the initial 'time intervals' of observation coverage a The documentation for `xaexpmap` notes that you can force the attitude histogram to have a single bin, by choosing a radial delta that is much larger than any expected attitude variation during an observation. We choose to create exposure maps from only one attitude histogram bin, by passing a large radial delta and requiring a single azimuthal bin: + ```{code-cell} python expmap_rad_delta = Quantity(20, "arcmin") expmap_phi_bins = 1 @@ -1544,6 +1548,7 @@ binning factor we generated images for). The `xaexpmap` task creates exposure maps with 1 image pixel per Sky X-Y coordinate system pixel, so we only need to specify binning factors **that do not equal 1** here (adding 1 to this list would be redundant and would waste compute time): + ```{code-cell} python expmap_bin_factors = [4] ``` @@ -1665,19 +1670,30 @@ RA-Dec coordinates - it requires pointing coordinate and telescope roll angle information which we extract from previously generated image files. ```{code-cell} python +chos_im_en = xtd_im_en_bounds[2] + for oi, dcs in rel_dataclasses.items(): - cur_im_path = IM_PATH_TEMP.format(oi=oi, xdc=dcs[0], ibf=1, lo=chos_im_en[0].value, - hi=chos_im_en[1].value) + cur_im_path = IM_PATH_TEMP.format( + oi=oi, xdc=dcs[0], ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value + ) with fits.open(cur_im_path) as imago: - cur_pnt_ra = imago[0].header['RA_PNT'] - cur_pnt_dec = imago[0].header['DEC_PNT'] - cur_roll_ang = imago[0].header['PA_NOM'] + cur_pnt_ra = imago[0].header["RA_PNT"] + cur_pnt_dec = imago[0].header["DEC_PNT"] + cur_roll_ang = imago[0].header["PA_NOM"] # Call the HEASoft task that converts from detector coordinates to RA-Dec - hsp.coordpnt(input=detpix_xtend_calib_reg_path, - outfile=radec_xtend_calib_reg_path.format(oi=oi), - telescop='XRISM', instrume='XTEND', ra=cur_pnt_ra, dec=cur_pnt_dec, - roll=cur_roll_ang, startsys='DET', stopsys='RADEC', clobber=True) + hsp.coordpnt( + input=detpix_xtend_calib_reg_path, + outfile=radec_xtend_calib_reg_path.format(oi=oi), + telescop="XRISM", + instrume="XTEND", + ra=cur_pnt_ra, + dec=cur_pnt_dec, + roll=cur_roll_ang, + startsys="DET", + stopsys="RADEC", + clobber=True, + ) ``` Finally, we have to pull the RA-Dec calibration regions from the transformed region @@ -1691,9 +1707,9 @@ them, they will appear as a different color than the source and background regio ```{code-cell} python cal_regs = {} for oi in rel_obsids: - cur_cal_regs = Regions(radec_xtend_calib_reg_path.format(oi=oi), format='ds9') + cur_cal_regs = Regions(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") for cur_reg in cur_cal_regs: - cur_reg.visual['color'] = 'white' + cur_reg.visual["color"] = "white" # The '.regions' just retrieves a list of region objects, we don't need to keep # the calibration regions in the regions module 'Regions' class they are read into @@ -1783,11 +1799,17 @@ with the source, background, and calibration source regions overplotted # formatted later radec_src_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_src.reg") # Where to write the new RA-Dec background region file -radec_back_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_back.reg") +radec_back_reg_path = os.path.join( + OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_back.reg" +) # The file path templates for the source and background Sky X-Y system region files -obs_src_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_src.reg") -obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_back.reg") +obs_src_reg_path_temp = os.path.join( + OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_src.reg" +) +obs_back_reg_path_temp = os.path.join( + OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_back.reg" +) for oi in rel_obsids: # We set up a combination of the source region and the calibration source @@ -1851,7 +1873,6 @@ with mp.Pool(NUM_CORES) as p: lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) ``` - ### New XRISM-Xtend spectra and supporting files @@ -1945,7 +1966,6 @@ output grouped spectral files: ```{code-cell} python spec_group_type = "min" spec_group_scale = 1 - ``` Now we run the grouping tool - though this time we do not parallelize the task, as @@ -1969,7 +1989,7 @@ for oi, dcs in rel_dataclasses.items(): dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), ) - cur_grp_spec = grp_SP_PATH_TEMP.format( + cur_grp_spec = GRP_SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, gt=spec_group_type, @@ -2011,7 +2031,6 @@ with mp.Pool(NUM_CORES) as p: rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) ``` - #### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation ```{code-cell} python @@ -2093,7 +2112,7 @@ Now we configure some behaviors of XSPEC/pyXspec: - We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). ```{code-cell} python -import xspec as xs +import xspec as xs # noqa: E402 xs.Xset.chatter = 0 @@ -2124,7 +2143,7 @@ xs.AllData.clear() # Set up the paths to grouped source spectrum, ungrouped background # spectrum, RMF, and ARF files -cur_spec = grp_SP_PATH_TEMP.format( +cur_spec = GRP_SP_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, gt=spec_group_type, From 65b18dae31a9dc41a1028838cf57fa910d8fad66 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 17:35:20 -0500 Subject: [PATCH 056/221] Commented out the xspec and arfgen portions, and did a horrible bodge wrt selecting the right dataclasses. Want to do a test PR and see if this runs on CircleCI. For issue #128 ALSO COMMENTED OUT XMM CCF ACQUISITION IN CIRCLECI YML, WILL REALLY NEED TO REMEMBER TO PUT THAT BACK --- .circleci/config.yml | 56 ++++---- .../xrism/getting-started-xrism-xtend.md | 134 +++++++++--------- 2 files changed, 97 insertions(+), 93 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index ca08ff75..1b773c97 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -45,34 +45,34 @@ jobs: # Some missions (e.g. XMM, Chandra, eROSITA) require that their calibration files are available locally. We # don't want to download them every single time, so we're going to cache them for future runs. - - run: - name: Acquiring/validating XMM-CCFs - no_output_timeout: 30m - command: | - # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same - # version number as currently defined in the environment variable, we have to download the data - if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then - - # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the - # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions - # available, so we don't want to delete the directory. - mkdir -p /home/jovyan/xmm-ccf/ccf-files - - # We do delete the version file, if it exists, as if we get to this point then we are either downloading - # the data for the first time, or the version number has changed. - [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver - - # Make a new version file - this goes a level up from the directory where the files will actually - # be stored (ccf-files), because the rsync process will delete the version file - echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver - - # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The - # neatest way is to make a new conda environment that just contains rsync - micromamba create -n rsync-env -y -c conda-forge rsync - - # Actually rsync the calibration files - micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files - fi +# - run: +# name: Acquiring/validating XMM-CCFs +# no_output_timeout: 30m +# command: | +# # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same +# # version number as currently defined in the environment variable, we have to download the data +# if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then +# +# # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the +# # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions +# # available, so we don't want to delete the directory. +# mkdir -p /home/jovyan/xmm-ccf/ccf-files +# +# # We do delete the version file, if it exists, as if we get to this point then we are either downloading +# # the data for the first time, or the version number has changed. +# [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver +# +# # Make a new version file - this goes a level up from the directory where the files will actually +# # be stored (ccf-files), because the rsync process will delete the version file +# echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver +# +# # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The +# # neatest way is to make a new conda environment that just contains rsync +# micromamba create -n rsync-env -y -c conda-forge rsync +# +# # Actually rsync the calibration files +# micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files +# fi # We also download the Chandra CalDB - run: diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 26184e99..ac2c2fc1 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1897,15 +1897,16 @@ with mp.Pool(NUM_CORES) as p: sp_result = p.starmap(gen_xrism_xtend_spectrum, arg_combs) ``` -Create template variables for source and background spectrum files: +#### Calculating 'BACKSCAL' for new XRISM-Xtend spectra -```{code-cell} python +***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** -``` -#### Calculating 'BACKSCAL' for new XRISM-Xtend spectra +***TERRIBLE BODGE MUST FIX*** -***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** +```{code-cell} python +rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} +``` ```{code-cell} python for oi, dcs in rel_dataclasses.items(): @@ -2087,8 +2088,8 @@ arg_combs = [ for dc in dcs ] -with mp.Pool(NUM_CORES) as p: - arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) +# with mp.Pool(NUM_CORES) as p: +# arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) ``` ```{warning} @@ -2112,22 +2113,25 @@ Now we configure some behaviors of XSPEC/pyXspec: - We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). ```{code-cell} python -import xspec as xs # noqa: E402 +# The strange comment on the end of this line is for the benefit of our +# automated code-checking processes. You shouldn't import modules anywhere but +# the top of your file, but this is unfortunately necessary at the moment +# import xspec as xs # noqa: E402 -xs.Xset.chatter = 0 +# xs.Xset.chatter = 0 # XSPEC parallelisation settings -xs.Xset.parallel.leven = NUM_CORES -xs.Xset.parallel.error = NUM_CORES -xs.Xset.parallel.steppar = NUM_CORES - -# Other xspec settings -xs.Plot.area = True -xs.Plot.xAxis = "keV" -xs.Plot.background = True -xs.Fit.statMethod = "cstat" -xs.Fit.query = "no" -xs.Fit.nIterations = 500 +# xs.Xset.parallel.leven = NUM_CORES +# xs.Xset.parallel.error = NUM_CORES +# xs.Xset.parallel.steppar = NUM_CORES +# +# # Other xspec settings +# xs.Plot.area = True +# xs.Plot.xAxis = "keV" +# xs.Plot.background = True +# xs.Fit.statMethod = "cstat" +# xs.Fit.query = "no" +# xs.Fit.nIterations = 500 ``` ### Reading XRISM-Xtend spectra into pyXspec @@ -2139,65 +2143,65 @@ chosen_demo_spec_dataclass = "31100010" ```{code-cell} python # In case this cell is re-run, clear all previously loaded spectra -xs.AllData.clear() - -# Set up the paths to grouped source spectrum, ungrouped background -# spectrum, RMF, and ARF files -cur_spec = GRP_SP_PATH_TEMP.format( - oi=chosen_demo_spec_obsid, - xdc=chosen_demo_spec_dataclass, - gt=spec_group_type, - gs=spec_group_scale, - ra=src_coord.ra.value.round(6), - dec=src_coord.dec.value.round(6), - rad=src_reg_rad.to("deg").value.round(4), -) - -cur_bspec = BACK_SP_PATH_TEMP.format( - oi=chosen_demo_spec_obsid, - xdc=chosen_demo_spec_dataclass, - ra=src_coord.ra.value.round(6), - dec=src_coord.dec.value.round(6), -) - -cur_rmf = RMF_PATH_TEMP.format( - oi=chosen_demo_spec_obsid, - xdc=chosen_demo_spec_dataclass, -) - -cur_arf = ARF_PATH_TEMP.format( - oi=chosen_demo_spec_obsid, - xdc=chosen_demo_spec_dataclass, - ra=src_coord.ra.value.round(6), - dec=src_coord.dec.value.round(6), - rad=src_reg_rad.to("deg").value.round(4), -) - -# Load the chosen spectrum (and all its supporting files) into pyXspec -xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) +# xs.AllData.clear() +# +# # Set up the paths to grouped source spectrum, ungrouped background +# # spectrum, RMF, and ARF files +# cur_spec = GRP_SP_PATH_TEMP.format( +# oi=chosen_demo_spec_obsid, +# xdc=chosen_demo_spec_dataclass, +# gt=spec_group_type, +# gs=spec_group_scale, +# ra=src_coord.ra.value.round(6), +# dec=src_coord.dec.value.round(6), +# rad=src_reg_rad.to("deg").value.round(4), +# ) +# +# cur_bspec = BACK_SP_PATH_TEMP.format( +# oi=chosen_demo_spec_obsid, +# xdc=chosen_demo_spec_dataclass, +# ra=src_coord.ra.value.round(6), +# dec=src_coord.dec.value.round(6), +# ) +# +# cur_rmf = RMF_PATH_TEMP.format( +# oi=chosen_demo_spec_obsid, +# xdc=chosen_demo_spec_dataclass, +# ) +# +# cur_arf = ARF_PATH_TEMP.format( +# oi=chosen_demo_spec_obsid, +# xdc=chosen_demo_spec_dataclass, +# ra=src_coord.ra.value.round(6), +# dec=src_coord.dec.value.round(6), +# rad=src_reg_rad.to("deg").value.round(4), +# ) +# +# # Load the chosen spectrum (and all its supporting files) into pyXspec +# xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) ``` ### Restricting the spectral channels used for fitting ```{code-cell} python -xs_spec.ignore("**-0.5 12.0-**") - -# Ignore any channels that have been marked as 'bad' -# This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra -# have been declared -xs.AllData.ignore("bad") +# xs_spec.ignore("**-0.5 12.0-**") +# +# # Ignore any channels that have been marked as 'bad' +# # This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra +# # have been declared +# xs.AllData.ignore("bad") ``` ### Setting up a spectral model ```{code-cell} python -xs.Model("tbabs*(powerlaw+apec+bbody)") +# xs.Model("tbabs*(powerlaw+apec+bbody)") ``` ### Fitting our pyXspec model to the XRISM-Xtend spectrum ```{code-cell} python -xs.Fit.perform() +# xs.Fit.perform() ``` ## About this notebook @@ -2206,7 +2210,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-03 +Updated On: 2025-12-05 +++ From c3c5035510233580d8ed35d576a998bd17a2a2cf Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 17:39:24 -0500 Subject: [PATCH 057/221] Specifically selected two out of three observations in the getting-started-xrism-xtend.md notebook. Will need to change where/how that happens. For issue #128 and the test PR --- .../xrism/getting-started-xrism-xtend.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index ac2c2fc1..c722a220 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -998,6 +998,12 @@ For an active mission (i.e., actively collecting data and adding to the archive) public_times = Time(all_xrism_obs["public_date"], format="mjd") avail_xrism_obs = all_xrism_obs[public_times <= Time.now()] +# TODO MAKE MORE PERMANENT ELSEWHERE +avail_xrism_obs = avail_xrism_obs[ + (avail_xrism_obs["obsid"] == "000128000") + | (avail_xrism_obs["obsid"] == "000126000") +] + # Define a couple of useful variables that make accessing information in the # table a little easier later on in the notebook # Create an array of the relevant ObsIDs From 352b69684f26331a67c6cfa0dbb3a7738b5571da Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 18:49:56 -0500 Subject: [PATCH 058/221] Increased the time out of notebook execution in the sphinx configuration file. --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index 5cbed18f..a59e3b38 100644 --- a/conf.py +++ b/conf.py @@ -32,7 +32,7 @@ '**/*_template*', '**/README.md', '*_template*'] # MyST-NB configuration -nb_execution_timeout = 1200 +nb_execution_timeout = 2400 nb_merge_streams = True nb_execution_mode = "cache" # nb_execution_mode = "force" From bd902205da18c691ae00acb7a03b8ee308dcd2ab Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 09:39:14 -0500 Subject: [PATCH 059/221] Added something that should print xtdpipeline outputs for any observations that failed processing in the getting-started-xrism-xtend.md notebook. Also made sure that the rel_dataclasses variable is modified to remove failed ObsIDs, if necessary. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index c722a220..fc982938 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -953,7 +953,7 @@ ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") *** -## 1. Finding and downloading XRISM observations of **NAMEHERE** +## 1. Finding and downloading XRISM observations of LMC N132D ### Determining the name of the XRISM observation summary table @@ -963,7 +963,7 @@ catalog_name = Heasarc.list_catalogs(master=True, keywords="xrism")[0]["name"] catalog_name ``` -### What are the coordinates of **NAMEHERE**? +### What are the coordinates of LMC N132D? To search for relevant observations, we have to know the coordinates of our source. The astropy module allows us to look up a source name in CDS' Sesame name @@ -1215,10 +1215,22 @@ with mp.Pool(NUM_CORES) as p: xtd_pipe_problem_ois = [all_out[0] for all_out in pipe_result if not all_out[2]] rel_obsids = [oi for oi in rel_obsids if oi not in xtd_pipe_problem_ois] +rel_dataclasses = {oi: rel_dataclasses[oi] for oi in rel_obsids} xtd_pipe_problem_ois ``` +We also include a code snippet that will print the output of the `xtdpipeline` run for any +observations that appear to have failed: + +```{code-cell} python +if len(xtd_pipe_problem_ois) != 0: + for all_out in pipe_result: + if all_out[0] in xtd_pipe_problem_ois: + print(all_out[2]) + print("\n\n") +``` + ```{warning} Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` From 797d3233b72598e0968bceb578f12321a2e45a92 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 10:42:18 -0500 Subject: [PATCH 060/221] Fixed some small bugs with xtdpipeline error printing, and the reading in of a regions file from disk into the 'regions' module. In the getting-started-xrism-xtend.md notebook. --- .../xrism/getting-started-xrism-xtend.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index fc982938..733421fc 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1227,7 +1227,7 @@ observations that appear to have failed: if len(xtd_pipe_problem_ois) != 0: for all_out in pipe_result: if all_out[0] in xtd_pipe_problem_ois: - print(all_out[2]) + print(all_out[1]) print("\n\n") ``` @@ -1725,7 +1725,7 @@ them, they will appear as a different color than the source and background regio ```{code-cell} python cal_regs = {} for oi in rel_obsids: - cur_cal_regs = Regions(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") + cur_cal_regs = Regions.read(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") for cur_reg in cur_cal_regs: cur_reg.visual["color"] = "white" From dd5dd0d3f3458d8962c38a3f5d6dc02b7b8574e3 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 10:43:51 -0500 Subject: [PATCH 061/221] Made some changes to the getting-started-xrism-xtend.md notebook where it writes 'final' source and background region files in both RA-Dec and Sky-XY coordinate systems. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 733421fc..5d5bdfea 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1840,16 +1840,15 @@ for oi in rel_obsids: # Write the RA-Dec source region file src_comb_regs.write(radec_src_reg_path.format(oi=oi), format="ds9", overwrite=True) # And the RA-Dec background region file - back_reg.write(radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True) + back_comb_regs.write(radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True) # Now we repeat the exercise, but use the Sky<->RA-Dec WCS information we pulled # from the images to convert the regions to the Sky X-Y system - src_comb_regs.to_pixel(oi_skypix_wcs[oi]).write( - obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True - ) - back_comb_regs.to_pixel(oi_skypix_wcs[oi]).write( - obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True - ) + src_comb_skyXY_regs = Regions([r.to_pixel(oi_skypix_wcs[oi]) for r in src_comb_regs]) + back_comb_skyXY_regs = Regions([r.to_pixel(oi_skypix_wcs[oi]) for r in back_comb_regs]) + + src_comb_skyXY_regs.write(obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True) + back_comb_skyXY_regs.write(obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True) ``` ```{tip} From edf600f13dd56bfde9ab1ba17d5ca234d274b523 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 8 Dec 2025 15:44:25 +0000 Subject: [PATCH 062/221] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- .../xrism/getting-started-xrism-xtend.md | 22 ++++++++++++++----- 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 5d5bdfea..8ebd4fc5 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1840,15 +1840,25 @@ for oi in rel_obsids: # Write the RA-Dec source region file src_comb_regs.write(radec_src_reg_path.format(oi=oi), format="ds9", overwrite=True) # And the RA-Dec background region file - back_comb_regs.write(radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True) + back_comb_regs.write( + radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True + ) # Now we repeat the exercise, but use the Sky<->RA-Dec WCS information we pulled # from the images to convert the regions to the Sky X-Y system - src_comb_skyXY_regs = Regions([r.to_pixel(oi_skypix_wcs[oi]) for r in src_comb_regs]) - back_comb_skyXY_regs = Regions([r.to_pixel(oi_skypix_wcs[oi]) for r in back_comb_regs]) - - src_comb_skyXY_regs.write(obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True) - back_comb_skyXY_regs.write(obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True) + src_comb_skyXY_regs = Regions( + [r.to_pixel(oi_skypix_wcs[oi]) for r in src_comb_regs] + ) + back_comb_skyXY_regs = Regions( + [r.to_pixel(oi_skypix_wcs[oi]) for r in back_comb_regs] + ) + + src_comb_skyXY_regs.write( + obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True + ) + back_comb_skyXY_regs.write( + obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True + ) ``` ```{tip} From 86ae11c5661b45d46634fd00b1f99b2b41753a05 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 11:56:42 -0500 Subject: [PATCH 063/221] Made sure all the coordinates that go into region files are in the ICRS file - that stops issues with a HEASArc tool trying to read coordinates in different frames and falling over. In getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 8ebd4fc5..86ab3d5f 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -974,7 +974,7 @@ You could also set up a SkyCoord object directly, if you already know the coordi ``` ```{code-cell} python -src_coord = SkyCoord.from_name(SRC_NAME) +src_coord = SkyCoord.from_name(SRC_NAME).transform_to("icrs") # This will be useful later on in the notebook, for functions that take # coordinates as an astropy Quantity. src_coord_quant = Quantity([src_coord.ra, src_coord.dec]) @@ -1728,6 +1728,9 @@ for oi in rel_obsids: cur_cal_regs = Regions.read(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") for cur_reg in cur_cal_regs: cur_reg.visual["color"] = "white" + # Make sure the frame is consistent with the source/back regions later, as + # otherwise HEASoft tools will get confused + cur_reg.center = cur_reg.center.transform_to("icrs") # The '.regions' just retrieves a list of region objects, we don't need to keep # the calibration regions in the regions module 'Regions' class they are read into @@ -1754,7 +1757,7 @@ this region is of a different size and is not centered on the source: ```{code-cell} python # The central coordinate of the background region -back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") +back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg", frame="icrs") # The radius of the background region back_reg_rad = Quantity(3, "arcmin") @@ -2108,7 +2111,7 @@ arg_combs = [ rad=src_reg_rad.to("deg").value.round(4), ), RMF_PATH_TEMP.format(oi=oi, xdc=dc), - radec_src_reg_path, + radec_src_reg_path.format(oi=oi), arf_rt_num_photons, ] for oi, dcs in rel_dataclasses.items() From d8736f88a292697dd7aea7a3c88015cc5762a7a8 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 12:18:33 -0500 Subject: [PATCH 064/221] Increased notebook execution timeout in the CircleCI config. --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 80b79e25..d13bca94 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -148,7 +148,7 @@ jobs: # Now we're going to start building the documentation - run: name: Build HTML rendering of notebooks - no_output_timeout: 30m + no_output_timeout: 60m command: | micromamba run -n build_docs sphinx-build -b html . _build/html -nWT --keep-going sed -E -i.bak '/caption-text/{N; s/.+caption-text.+\n
    /
      /; P;D;}' _build/html/index.html From 94071f53db15ff65a3aa52aafe54aa22f4b5745c Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 13:36:41 -0500 Subject: [PATCH 065/221] Removed some extraneous formatting arguments in getting-started-xrism-xtend.md --- .../xrism/getting-started-xrism-xtend.md | 21 ++++++++----------- 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 86ab3d5f..ba2d1703 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-05' +date: '2025-12-08' file_format: mystnb jupytext: text_representation: @@ -74,7 +74,7 @@ We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstr ### Runtime -As of 5th December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 8th December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -1889,8 +1889,8 @@ arg_combs = [ os.path.join(OUT_PATH, oi), src_coord, src_reg_rad, - obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), - obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + obs_src_reg_path_temp.format(oi=oi), + obs_back_reg_path_temp.format(oi=oi), *cur_bnds, lc_time_bin, ] @@ -1916,8 +1916,8 @@ arg_combs = [ os.path.join(OUT_PATH, oi), src_coord, src_reg_rad, - obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), - obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + obs_src_reg_path_temp.format(oi=oi), + obs_back_reg_path_temp.format(oi=oi), ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -1929,9 +1929,6 @@ with mp.Pool(NUM_CORES) as p: #### Calculating 'BACKSCAL' for new XRISM-Xtend spectra -***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** - - ***TERRIBLE BODGE MUST FIX*** ```{code-cell} python @@ -1968,7 +1965,7 @@ for oi, dcs in rel_dataclasses.items(): # Calculate the BACKSCAL keyword, first for the source spectrum hsp.ahbackscal( infile=cur_spec, - regfile=obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), + regfile=obs_src_reg_path_temp.format(oi=oi), expfile=cur_ex, logfile="NONE", ) @@ -1976,7 +1973,7 @@ for oi, dcs in rel_dataclasses.items(): # Then for the background spectrum hsp.ahbackscal( infile=cur_bspec, - regfile=obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + regfile=obs_back_reg_path_temp.format(oi=oi), expfile=cur_ex, logfile="NONE", ) @@ -2240,7 +2237,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-05 +Updated On: 2025-12-08 +++ From ad059a68f405c0682f4e1e01ac6e00dd3e4b8994 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 16:56:52 -0500 Subject: [PATCH 066/221] Finished explaining the light curve generation section, also moved it and produced net light curve files using the BACKSCAL values calculated for the source and background spectra. All in the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 212 +++++++++++++++--- 1 file changed, 179 insertions(+), 33 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index ba2d1703..a606af32 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -917,6 +917,13 @@ BACK_LC_PATH_TEMP = os.path.join( "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-back-lightcurve.fits", ) + +NET_LC_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" + "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-net-lightcurve.fits", +) # -------------------------- @@ -1871,42 +1878,9 @@ However, different data classes represent different pairs of Xtend CCDs, so ther no shared sky coverage. ``` -### New XRISM-Xtend light curves - -```{code-cell} python -lc_time_bin = Quantity(200, "s") -``` - -```{code-cell} python -# Defining the various energy bounds we want to make light curves for -xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") -``` - -```{code-cell} python -arg_combs = [ - [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), - os.path.join(OUT_PATH, oi), - src_coord, - src_reg_rad, - obs_src_reg_path_temp.format(oi=oi), - obs_back_reg_path_temp.format(oi=oi), - *cur_bnds, - lc_time_bin, - ] - for oi, dcs in rel_dataclasses.items() - for dc in dcs - for cur_bnds in xtd_lc_en_bounds -] - -with mp.Pool(NUM_CORES) as p: - lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) -``` ### New XRISM-Xtend spectra and supporting files - - #### Generating the spectral files ```{code-cell} python @@ -1929,6 +1903,10 @@ with mp.Pool(NUM_CORES) as p: #### Calculating 'BACKSCAL' for new XRISM-Xtend spectra +Our calculation of 'BACKSCAL' doesn't only benefit our spectra analyses, as when we +demonstrate the creation of light curves later in this notebook, we can also use +the values to weight our subtraction of the background. + ***TERRIBLE BODGE MUST FIX*** ```{code-cell} python @@ -1936,6 +1914,9 @@ rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} ``` ```{code-cell} python +spec_backscals = {oi: {dc: 0 for dc in rel_dataclasses[oi]} for oi in rel_obsids} +bspec_backscals = {oi: {dc: 0 for dc in rel_dataclasses[oi]} for oi in rel_obsids} + for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up the path to input source and background spectra @@ -1977,6 +1958,15 @@ for oi, dcs in rel_dataclasses.items(): expfile=cur_ex, logfile="NONE", ) + + # For good measure, and because we're going to need them later for + # net light curve calculation, we read the backscal values into Python + # First, the source spectrum + with fits.open(cur_spec) as src_specco: + spec_backscals[oi][cur_dc] = src_specco["SPECTRUM"].header["BACKSCAL"] + # Now the background + with fits.open(cur_bspec) as back_specco: + bspec_backscals[oi][cur_dc] = back_specco["SPECTRUM"].header["BACKSCAL"] ``` #### Grouping our new spectra @@ -2124,6 +2114,162 @@ Due to the high-fidelity ray-tracing method used to calculate XRISM ARFs, the ru of this step can be on the order of hours. ``` +### New XRISM-Xtend light curves + +Now we can quickly demonstrate how to generate XRISM-Xtend light curves - it is +rather simpler than the creation of new spectra. + +There is no XRISM-Xtend-specific task for the generation of light curves, so we +once again turn to HEASoft's `extractor` tool (we used it to create XRISM-Xtend +images in Section 3). + +By providing a slightly different set of inputs to `extractor`, we can tell it to +bin the cleaned event lists in time, rather than in space, and thus produce a +light curve. + +We'll make sure to generate light curves within the source and background regions +that we defined in the previous part of this section, which we can then use to +produce net light curves for our source. + +The primary input we need to provide is the time step, or time bin size, which +controls the temporal resolution of the output light curve. This uniform sampling is +the simplest method of dividing an event list into a light curve, but other methods +exist (requiring each time bin to reach a minimum signal-to-noise for instance). + +Your choice of uniform time bin size will depend on your particular science case, and +practical considerations based on the length of the overall observation and the +observed count-rate of the source. + +```{code-cell} python +lc_time_bin = Quantity(200, "s") +``` + +It is also very common to want to specify the events included in each by setting +lower and upper energy bounds - this may allow you to focus on light emitted by +a particular process you're interested in, or to exclude energy bands that are +not relevant to your science case. + +We define three energy bands that cover much of the useful energy range of the +XRISM-Xtend instrument: +- 0.6-2.0 keV +- 2.0-6.0 keV +- 6.0-10.0 keV + +Though again, your choices will depend on what you're trying to learn. + +```{code-cell} python +# Defining the various energy bounds we want to make light curves for +xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") +``` + +#### Generating source and background light curves + +Using another wrapper function around the HEASoftPy interface to `extractor`, we can +now generate the light curves within the source and background regions, for each of +the specified energy bands. + +As with previous steps, our motivation for writing a wrapper function (defined in the +Global Setup section) is to make it easy for us to run generation of different +light curves simultaneously: + +```{code-cell} python +arg_combs = [ + [ + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + os.path.join(OUT_PATH, oi), + src_coord, + src_reg_rad, + obs_src_reg_path_temp.format(oi=oi), + obs_back_reg_path_temp.format(oi=oi), + *cur_bnds, + lc_time_bin, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs + for cur_bnds in xtd_lc_en_bounds +] + +with mp.Pool(NUM_CORES) as p: + lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) +``` + +#### Calculating net light curves + +Unlike with the spectra we generated earlier in this section, we will produce 'net' +light curves, with the background light curve scaled and subtracted from the source. + +The applied scaling is to effectively normalize the area within which the background +light curve was extracted to the source light curve. + +We have already performed the measurement of the extraction regions for source and +background products - when we used HEASoft to calculate the 'BACKSCAL' keyword after +our spectra were generated. + +At the time we made sure to read those 'BACKSCAL' values into Python, specifically +for this purpose. + +With the scaling known, we can use the HEASoft `lcmath` tool (through the HEASoftPy +interface) to subtract the background from the source light curve. This operation is +computationally cheap for the number of light curves we are working with, but you +should consider parallelizing this step if you are working with significantly more: + +```{code-cell} python +for oi, dcs in rel_dataclasses.items(): + for cur_dc in dcs: + for cur_bnds in xtd_lc_en_bounds: + # Constructing the file paths to the source and background light curves + cur_lc = LC_PATH_TEMP.format( + oi=oi, + xdc=dc, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), + lo=cur_bnds[0].value, + hi=cur_bnds[1].value, + tct=0.0, + tb=lc_time_bin.to("s").value, + ) + + cur_blc = BACK_LC_PATH_TEMP.format( + oi=oi, + xdc=dc, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + lo=cur_bnds[0].value, + hi=cur_bnds[1].value, + tct=0.0, + tb=lc_time_bin.to("s").value, + ) + + # Now we construct the output file path for the final net light curve + cur_nlc = NET_LC_PATH_TEMP.format( + oi=oi, + xdc=dc, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), + lo=cur_bnds[0].value, + hi=cur_bnds[1].value, + tct=0.0, + tb=lc_time_bin.to("s").value, + ) + + # The 'lcmath' tool is sensitive to long paths, so we fetch the relative + # paths to pass in + cur_lc = os.path.relpath(cur_lc) + cur_blc = os.path.relpath(cur_blc) + cur_nlc = os.path.relpath(cur_nlc) + + # Calculate the scaling that should be applied to the background + # light curve before subtraction + cur_back_multi = spec_backscals[oi][cur_dc] / bspec_backscals[oi][cur_dc] + + # Run the tool to produce a net light curve + hsp.lcmath( + infile=cur_lc, bgfile=cur_blc, outfile="", multi=1, multb=cur_back_multi + ) +``` + ## 5. Fitting spectral models to XRISM-Xtend spectra Finally, to show off the XRISM-Xtend products we just generated, we will perform From 95a2312dbafe9041da5b2cf72c7d6ed883f0823a Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 10:52:07 -0500 Subject: [PATCH 067/221] Added a HEASoft version check to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 35 +++++++++++++++++-- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a606af32..a9c50ebd 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -96,6 +96,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc +from packaging.version import Version from regions import CircleSkyRegion, Regions from xga.products import Image ``` @@ -1085,16 +1086,40 @@ on a regular basis, but please report any issues or suggestions to the HEASARC H Both the HEASoft and HEASoftPy package versions can be retrieved from the HEASoftPy module. +The HEASoftPy version: + +```{code-cell} python +hsp.__version__ +``` + The HEASoft version: ```{code-cell} python -hsp.fversion() +fver_out = hsp.fversion() +fver_out ``` -The HEASoftPy version: +It is likely that this tutorial will not run all the way through if you are using +a version of HEASoft older than **v6.36**, so we will check for that and raise an +error if it is the case. First, extract the version string from the `fversion` output, and +set up a `Version` object: ```{code-cell} python -hsp.__version__ +fver_out.output[0].split("_")[-1] +HEA_VER = Version(fver_out.output[0].split("_")[-1]) +HEA_VER +``` + +Now we can use inequality operators on another Version object to check if our version +of HEASoft meets the requirements: + +```{code-cell} python +if HEA_VER < Version("V6.36"): + raise ValueError( + "We strongly recommend using HEASoft v6.36 or later for this " + "tutorial - you may run rest of the notebook yourself, but " + "ARF generation will likely fail." + ) ``` ### Setting up file paths to pass to the XRISM-Xtend pipeline @@ -1903,6 +1928,10 @@ with mp.Pool(NUM_CORES) as p: #### Calculating 'BACKSCAL' for new XRISM-Xtend spectra +Spectral data products generated for high-energy missions typically contain a +measurement of their extraction region area. This is in order to scale source and +background spectra properly when + Our calculation of 'BACKSCAL' doesn't only benefit our spectra analyses, as when we demonstrate the creation of light curves later in this notebook, we can also use the values to weight our subtraction of the background. From adc60db4fb92b9b44902f7ee4df42235b94330e3 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 13:12:19 -0500 Subject: [PATCH 068/221] Added a very short, but top level, section about what do with observations that have multiple dataclasses - in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 123 ++++++++++++++++-- 1 file changed, 109 insertions(+), 14 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a9c50ebd..bd247fa7 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1460,6 +1460,7 @@ any problems or unusual features of the prepared and cleaned observations. #### Image energy bounds We are going to generate images within the following energy bounds: +- 0.6-10.0 keV - 0.6-2.0 keV - 2.0-10.0 keV - ***0.4-2.0 keV*** [not recommended] @@ -1471,7 +1472,9 @@ demonstrate those issues. ```{code-cell} python # Defining the energy bounds we want images within -xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV") +xtd_im_en_bounds = Quantity( + [[0.6, 10.0], [0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV" +) ``` Converting those energy bounds to channel bounds is straightforward, we simply divide @@ -1553,10 +1556,6 @@ with mp.Pool(NUM_CORES) as p: im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` -#### Problematic pixels - - - ### New XRISM-Xtend exposure maps We also generate exposure maps for the entire FoV of a particular observation mode, rather @@ -1627,7 +1626,89 @@ with mp.Pool(NUM_CORES) as p: ex_result = p.starmap(gen_xrism_xtend_expmap, arg_combs) ``` -## 4. Generating new XRISM-Xtend spectra and light curves +## 4. Handling observations with multiple dataclasses + +As we've already mentioned, the XRISM-Xtend detector can operate in two different +data modes simultaneously, with the read-out area of some of the CCDs restricted in +order to increase the read-out speed and improve timing resolution. + +If the proposer of the observation has requested an increased-timing-resolution +mode, then odds are the target of their observation will be placed on that +fast-timing-mode portion of the detector. + +As such, if that is the target you are also interested in (whether you are the +original proposer or are performing some archival analysis), then the other half +of the XRISM-Xtend detector may not be that interesting to you. + +In such cases we will have to decide which dataclass we're going to use from +this point onwards. + +The first part of the next section of this demonstration will show you how to set up +the regions from which source and background data products will be extracted. + +Event lists from the same observation with different dataclasses are taken from +different halves of the detector and are not co-aligned. That means the source region +we're about to set up for our target will not overlap with the coverage of the other +dataclass. + +That would result in empty light curve and spectrum products, and errors +from 'BACKSCAL' calculation, and RMF and ARF generation. + + +### Visualize separate XRISM-Xtend 000128000 dataclass images + +```{code-cell} python +fast_im_path = IM_PATH_TEMP.format( + oi="000128000", xdc="31100010", lo="0.6", hi="10.0", ibf=1 +) +half_im_path = IM_PATH_TEMP.format( + oi="000128000", xdc="32000010", lo="0.6", hi="10.0", ibf=1 +) + +fast_im = Image( + fast_im_path, + "000128000", + "Xtend", + "", + "", + "", + Quantity(0.6, "keV"), + Quantity(10.0, "keV"), +) +half_im = Image( + half_im_path, + "000128000", + "Xtend", + "", + "", + "", + Quantity(0.6, "keV"), + Quantity(10.0, "keV"), +) +``` + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +fig, ax_arr = plt.subplots(ncols=2, figsize=(6, 6), width_ratios=[3, 1]) +plt.subplots_adjust(wspace=0) + +half_im.get_view(ax_arr[0], zoom_in=True, custom_title="32000010") +fast_im.get_view(ax_arr[1], src_coord_quant, zoom_in=True, custom_title="31100010") + +plt.show() +``` + +### Choosing our dataclasses + +```{code-cell} python +rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} +``` + +## 5. Generating new XRISM-Xtend spectra and light curves In this section we will demonstrate how to generate source-specific data products from XRISM-Xtend observations; light curves and spectra (along with supporting files like @@ -1903,11 +1984,31 @@ However, different data classes represent different pairs of Xtend CCDs, so ther no shared sky coverage. ``` - ### New XRISM-Xtend spectra and supporting files +With the source and background regions defined, we have everything we need to generate +XRISM-Xtend spectra for our target source. + +Though there is no XRISM-Xtend-specific HEASoft task for spectrum generation, we can +use the same tool that we used to create our images in Section 3, and that we will use +later in this section to make light curves - `extractor`. + +Spectral files are just as simple a data product as images and light curves. All we're +doing to make a spectrum is binning the events in our extraction regions in a 1D +channel space (which will become energy), rather than X-Y, or time. The complexities +of spectral generation come when creating the supporting files (RMFs and ARFs) +required to actually fit models to the spectra. + #### Generating the spectral files +We set up a spectrum-generation-specific region wrapper function for the HEASoftPy +interface to the `extractor` task (see the Global Setup: Functions section near the +top of the notebook). + +This once again allows us to parallelize the generation of spectra for different +observations, though it is worth noting we aren't producing multiple variants of the +spectrum like we did with the different-energy-band images. + ```{code-cell} python arg_combs = [ [ @@ -1936,12 +2037,6 @@ Our calculation of 'BACKSCAL' doesn't only benefit our spectra analyses, as when demonstrate the creation of light curves later in this notebook, we can also use the values to weight our subtraction of the background. -***TERRIBLE BODGE MUST FIX*** - -```{code-cell} python -rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} -``` - ```{code-cell} python spec_backscals = {oi: {dc: 0 for dc in rel_dataclasses[oi]} for oi in rel_obsids} bspec_backscals = {oi: {dc: 0 for dc in rel_dataclasses[oi]} for oi in rel_obsids} @@ -2299,7 +2394,7 @@ for oi, dcs in rel_dataclasses.items(): ) ``` -## 5. Fitting spectral models to XRISM-Xtend spectra +## 6. Fitting spectral models to XRISM-Xtend spectra Finally, to show off the XRISM-Xtend products we just generated, we will perform a simple model fit to one of our spectra. From d28880e88414f041861b8c07660a565e9b867d10 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 13:23:19 -0500 Subject: [PATCH 069/221] Finished the dataclass choice section for now. --- .../xrism/getting-started-xrism-xtend.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index bd247fa7..f57c2763 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1657,14 +1657,16 @@ from 'BACKSCAL' calculation, and RMF and ARF generation. ### Visualize separate XRISM-Xtend 000128000 dataclass images +To make our point, and to give an example of the inspection you may want to perform +before choosing the right dataclass for your target, we will visualize +the 0.6-10.0 keV images we generated for 000128000 in the last section: + ```{code-cell} python +# Set up the path to the image, and XGA Image class instance, for the '31100010' +# dataclass, which is the small-window, fast-readout, mode fast_im_path = IM_PATH_TEMP.format( oi="000128000", xdc="31100010", lo="0.6", hi="10.0", ibf=1 ) -half_im_path = IM_PATH_TEMP.format( - oi="000128000", xdc="32000010", lo="0.6", hi="10.0", ibf=1 -) - fast_im = Image( fast_im_path, "000128000", @@ -1675,6 +1677,12 @@ fast_im = Image( Quantity(0.6, "keV"), Quantity(10.0, "keV"), ) + +# Set up the path to the image, and XGA Image class instance, for the '32000010' +# dataclass, which is rest of the Xtend detector running as normal +half_im_path = IM_PATH_TEMP.format( + oi="000128000", xdc="32000010", lo="0.6", hi="10.0", ibf=1 +) half_im = Image( half_im_path, "000128000", @@ -1687,6 +1695,8 @@ half_im = Image( ) ``` +We can quite clearly see that the source of interest, indicated by the +cross-hair, is on the small-window, fast-readout, '31100010' dataclass image: ```{code-cell} python --- tags: [hide-input] @@ -1704,10 +1714,18 @@ plt.show() ### Choosing our dataclasses +Based on our inspection, we choose the right dataclass for the '000128000' observation: ```{code-cell} python rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} ``` +```{hint} +Manual inspection is far from the only way of making this choice - it would be easy to +automate this process by, for instance, retrieving the exposure value corresponding to +the source position from the exposure map; a non-zero exposure for a particular dataclass +would indicate that it contains your source. +``` + ## 5. Generating new XRISM-Xtend spectra and light curves In this section we will demonstrate how to generate source-specific data products from From 173c851e15055f6ea525ee661f4fae2bf910420a Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 15:01:46 -0500 Subject: [PATCH 070/221] Added commentary --- .../xrism/getting-started-xrism-xtend.md | 36 +++++++++---------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f57c2763..2df16064 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2169,7 +2169,22 @@ for oi, dcs in rel_dataclasses.items(): #### Generating XRISM-Xtend RMFs -***THIS IS ALSO GOING TO FALL OVER IN PART BECAUSE CAN'T EXTRACT SPECTRUM FROM REGIONS NOT ON THE 31100010 DATACLASS OBSERVATION OF 000128000*** +In order for the spectral model fitting software of our choice (XSPEC, for +instance) to be able to map the channels of a spectrum to an energy, we need to +generate RMFs. + +We already discussed RMFs, and even used them to perform our own conversion between +XRISM-Xtend spectral channels and energy, in Section 3 - there we used the RMF that +was included in the original data download. + +Now we wish to generate new RMFs, to ensure they are entirely up-to-date! + +We make use of the XRISM-Xtend specific HEASoft task `xtdrmf` - the only input it +requires is the path to the spectral file for which we wish to generate an RMF. + +Our `gen_xrism_xtend_rmf` function (defined in the Global Setup: Functions section near +the top of the notebook) wraps the HEASoftPy interface to the `xtdrmf` task. We now use +it to generate RMFs in parallel for all of our new spectra: ```{code-cell} python arg_combs = [ @@ -2191,12 +2206,6 @@ with mp.Pool(NUM_CORES) as p: rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) ``` -#### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation - -```{code-cell} python - -``` - #### Generating XRISM-Xtend ARFs ```{danger} @@ -2205,21 +2214,12 @@ another, very similarly named, HEASoft tool related to the construction of XRISM ARFs, **`xaxmaarfgen`**. Be sure which one you are using! ``` +ARFs are the final type of supporting file required to make our spectra usable + ```{code-cell} python arf_rt_num_photons = 20000 ``` -```{warning} -***MIGHT BE WRONG, BUT v6.35.2 OF HEASOFT MIGHT POINT TO THE WRONG CALDB FILE PATH MIRROR -SCATTER INFORMATION USED BY RAYTRACE (xa_xtd_scatter_20190101v001.fits RATHER THAN xa_xtd_scatter_20190101v001.fits.gz)?*** - -***HEASOFT RELEASE NOTES SEEM TO BEAR THIS IDEA UP***: - -xrtraytrace: Additional updates to fix and enable remote CalDB - usage with xrtraytrace and xaarfgen ("timeout" interval extended - for reading large CalDB files). - -``` ```{code-cell} python arg_combs = [ From d88c511b3677b5a11235c3521782bcebdd971399 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 15:50:11 -0500 Subject: [PATCH 071/221] Have now written commentary on the ARF generation section of the getting-started-xrism-xtend.md notebook. THIS MIGHT BE BASICALLY DONE AAAARGH. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 69 ++++++++++++++++++- 1 file changed, 66 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 2df16064..7944f27a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2214,12 +2214,75 @@ another, very similarly named, HEASoft tool related to the construction of XRISM ARFs, **`xaxmaarfgen`**. Be sure which one you are using! ``` -ARFs are the final type of supporting file required to make our spectra usable +ARFs are the final type of supporting file required to make our spectra usable and +describe the effective area (i.e., the sensitivity) of XRISM-Xtend as a function of +energy. + +The effective area has to be understood (and well calibrated) as we need it to help +map a spectral model, which hopefully describes what the object of interest +is _actually_ emitting (and how), to the _observed_ spectrum; that observed spectrum +has been altered across its energy range by how good XRISM-Xtend is at detecting +photons at different points in that range. + +The sensitivity of an X-ray detector is a combination of the effective area of the +actual X-ray optics (on XRISM this is the called X-ray Mirror Assembly, or XMA), and +the detector's quantum efficiency. They are both independently a function of +energy. + +ARFs are standard products for most high-energy missions, but the methods implemented +to calculate them for XRISM's instruments are quite unusual. + +The HEASoft task we need to call (`xaarfgen`) calls further HEASoft tools that perform +ray-tracing simulations of XRISM XMAs, for the location of your source on the +detector, and use those to define the X-ray optic's collecting area for a wide range +of energies. + +```{note} +If you have to generate multiple ARFs for the same source, in the same observation, you +should be aware that the raytraced event lists can be re-used (though only in this +particular scenario). +``` + +Raytracing can be a slow process, as individual events and their path through the +XMA are being simulated, but it does help to produce very accurate ARFs. There are ways +that it can be sped up, though at the cost of that accuracy - the most direct way is +to limit the number of events that are simulated. + +An argument specifying the number of events can be passed to `xaarfgen`, and for +our demonstration we are going to use a very small sample - this is primarily so the +notebook can run in a reasonable amount of time. + +**You should likely use a larger value for real analysis!** + +```{admonition} +The minimum number of raytracing photons that successfully reach the focal plane, per +raytracing energy grid point, that is acceptable to make an ARF. The number of focal +plane photons that contribute to the ARF must exceed 'minphoton' for every +energy, otherwise the program aborts. Note that even if minphoton is exceeded at +all energies, this does not guarantee that the resulting ARF is robust and +sufficiently accurate. In general, about 5000 or more photons per energy (over the +extraction region) give good results, but the actual minimum number varies +case-by-case, and fewer may be sufficient in some cases. The default value of +minphoton is deliberately very small, in order that the ARF is made and available +for diagnostic evaluation. In general, it is not recommended to set 'minphoton' to a +high value in the first place, because it is not possible to reliably estimate +what 'numphoton' should be in advance of running raytracing within xaarfgen, in order +for that value of 'minphoton' to be satisfied for all energies, which could result +in repeated failures after very long run times. It could also run into memory +problems and/or a raytracing file size that is unmanageable. +``` + +The xaarfgen documentation provides the following guidance on choosing the number of +events to simulate: ```{code-cell} python arf_rt_num_photons = 20000 ``` +So now we move onto actually running the ARF generation - using the +`gen_xrism_xtend_arf` function defined in the Global Setup: Functions section (near the top of +the notebook), which wraps the HEASoftPy interface to the `xaarfgen` task. We now use it +to generate ARFs in parallel for all of our new spectra: ```{code-cell} python arg_combs = [ @@ -2247,8 +2310,8 @@ arg_combs = [ for dc in dcs ] -# with mp.Pool(NUM_CORES) as p: -# arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) +with mp.Pool(NUM_CORES) as p: + arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) ``` ```{warning} From 5edf936e568723af9e2217d5e07f19c32cfc2a96 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 20:22:52 -0500 Subject: [PATCH 072/221] May have finally finished the writing of the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 207 ++++++++++++------ 1 file changed, 143 insertions(+), 64 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 7944f27a..83731265 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-08' +date: '2025-12-10' file_format: mystnb jupytext: text_representation: @@ -1697,6 +1697,7 @@ half_im = Image( We can quite clearly see that the source of interest, indicated by the cross-hair, is on the small-window, fast-readout, '31100010' dataclass image: + ```{code-cell} python --- tags: [hide-input] @@ -1715,6 +1716,7 @@ plt.show() ### Choosing our dataclasses Based on our inspection, we choose the right dataclass for the '000128000' observation: + ```{code-cell} python rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} ``` @@ -2254,6 +2256,8 @@ notebook can run in a reasonable amount of time. **You should likely use a larger value for real analysis!** +The xaarfgen documentation provides the following guidance on choosing the number of +events to simulate: ```{admonition} The minimum number of raytracing photons that successfully reach the focal plane, per raytracing energy grid point, that is acceptable to make an ARF. The number of focal @@ -2272,9 +2276,6 @@ in repeated failures after very long run times. It could also run into memory problems and/or a raytracing file size that is unmanageable. ``` -The xaarfgen documentation provides the following guidance on choosing the number of -events to simulate: - ```{code-cell} python arf_rt_num_photons = 20000 ``` @@ -2475,7 +2476,7 @@ for oi, dcs in rel_dataclasses.items(): ) ``` -## 6. Fitting spectral models to XRISM-Xtend spectra +## 6. Fitting a spectral model to an XRISM-Xtend spectrum Finally, to show off the XRISM-Xtend products we just generated, we will perform a simple model fit to one of our spectra. @@ -2494,92 +2495,168 @@ Now we configure some behaviors of XSPEC/pyXspec: # The strange comment on the end of this line is for the benefit of our # automated code-checking processes. You shouldn't import modules anywhere but # the top of your file, but this is unfortunately necessary at the moment -# import xspec as xs # noqa: E402 +import xspec as xs # noqa: E402 -# xs.Xset.chatter = 0 +xs.Xset.chatter = 0 # XSPEC parallelisation settings -# xs.Xset.parallel.leven = NUM_CORES -# xs.Xset.parallel.error = NUM_CORES -# xs.Xset.parallel.steppar = NUM_CORES -# -# # Other xspec settings -# xs.Plot.area = True -# xs.Plot.xAxis = "keV" -# xs.Plot.background = True -# xs.Fit.statMethod = "cstat" -# xs.Fit.query = "no" -# xs.Fit.nIterations = 500 +xs.Xset.parallel.leven = NUM_CORES +xs.Xset.parallel.error = NUM_CORES +xs.Xset.parallel.steppar = NUM_CORES + +# Other xspec settings +xs.Plot.area = True +xs.Plot.xAxis = "keV" +xs.Plot.background = True +xs.Fit.statMethod = "cstat" +xs.Fit.query = "no" +xs.Fit.nIterations = 500 +``` + +```{danger} +There is a known issue with the version of PyXspec shipped in HEASoft v6.36 (and +possibly later versions) that will cause the parallelised generation of data products +to hang forever. We avoid this here by importing PyXspec **after** all data product +generation is complete. ``` -### Reading XRISM-Xtend spectra into pyXspec +### Reading a XRISM-Xtend spectrum into pyXspec + +Here we define the ObsID and dataclass of the spectrum we want to fit: ```{code-cell} python chosen_demo_spec_obsid = "000128000" chosen_demo_spec_dataclass = "31100010" ``` +The spectrum, and all of its supporting files, are then read into pyXspec: + ```{code-cell} python # In case this cell is re-run, clear all previously loaded spectra -# xs.AllData.clear() -# -# # Set up the paths to grouped source spectrum, ungrouped background -# # spectrum, RMF, and ARF files -# cur_spec = GRP_SP_PATH_TEMP.format( -# oi=chosen_demo_spec_obsid, -# xdc=chosen_demo_spec_dataclass, -# gt=spec_group_type, -# gs=spec_group_scale, -# ra=src_coord.ra.value.round(6), -# dec=src_coord.dec.value.round(6), -# rad=src_reg_rad.to("deg").value.round(4), -# ) -# -# cur_bspec = BACK_SP_PATH_TEMP.format( -# oi=chosen_demo_spec_obsid, -# xdc=chosen_demo_spec_dataclass, -# ra=src_coord.ra.value.round(6), -# dec=src_coord.dec.value.round(6), -# ) -# -# cur_rmf = RMF_PATH_TEMP.format( -# oi=chosen_demo_spec_obsid, -# xdc=chosen_demo_spec_dataclass, -# ) -# -# cur_arf = ARF_PATH_TEMP.format( -# oi=chosen_demo_spec_obsid, -# xdc=chosen_demo_spec_dataclass, -# ra=src_coord.ra.value.round(6), -# dec=src_coord.dec.value.round(6), -# rad=src_reg_rad.to("deg").value.round(4), -# ) -# -# # Load the chosen spectrum (and all its supporting files) into pyXspec -# xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) +xs.AllData.clear() + +# Set up the paths to grouped source spectrum, ungrouped background +# spectrum, RMF, and ARF files +cur_spec = GRP_SP_PATH_TEMP.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + gt=spec_group_type, + gs=spec_group_scale, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), +) + +cur_bspec = BACK_SP_PATH_TEMP.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), +) + +cur_rmf = RMF_PATH_TEMP.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, +) + +cur_arf = ARF_PATH_TEMP.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), +) + +# Load the chosen spectrum (and all its supporting files) into pyXspec +xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) ``` ### Restricting the spectral channels used for fitting +When we analyze a spectrum by fitting a model, we often want to apply lower and +upper energy limits in order to fit the model on only a subset of the data points. + +Restricting the spectrum data points by energy allows us to cut out parts of the +spectrum that, for instance, have very low signal-to-noise, aren't relevant to our +science case, or fall outside the optimal energy range of the instrument. + +Remember that XRISM-Xtend data are not currently trustworthy around or below 0.4 keV, so +we definitely want to exclude that part of the energy range. If we didn't we would be +in danger of biasing our model fitting results, leading to unreliable or unphysical +conclusions about our source of interest. + +Here, we only make use of channels within a 0.5-10.0 keV energy range, and we also +ignore any channels that have been marked as 'bad' by any previous processing steps: + ```{code-cell} python -# xs_spec.ignore("**-0.5 12.0-**") -# -# # Ignore any channels that have been marked as 'bad' -# # This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra -# # have been declared -# xs.AllData.ignore("bad") +xs_spec.ignore("**-0.5 10.0-**") + +# Ignore any channels that have been marked as 'bad' +# This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra +# have been declared +xs.AllData.ignore("bad") ``` ### Setting up a spectral model +Now we choose the spectral model we want to fit to our spectrum. + +A full list of XSPEC model components can be found in the [XSPEC documentation](https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html). + +Our choice of model is empirically driven, chosen by someone who is not a specialist in supernova remnants, and should definitely not be considered as scientifically useful! + ```{code-cell} python -# xs.Model("tbabs*(powerlaw+apec+bbody)") +xs.Model("tbabs*(powerlaw+apec+bbody)") ``` ### Fitting our pyXspec model to the XRISM-Xtend spectrum +Performing the fit is simple: + +```{code-cell} python +xs.Fit.perform() +``` + +### Visualizing the fitted spectrum + +We want to use matplotlib to visualize the spectrum data, and the model we +just fitted to it. PyXspec allows us to extract the data it would have plotted were +we using XSPEC directly: + ```{code-cell} python -# xs.Fit.perform() +# This populates plot information attributes for the current +# spectrum and model. We can extract that information and +# plot it using matplotlib +xs.Plot() + +# These read the plotting information into handy variables +spec_en = xs.Plot.x() +spec_en_err = xs.Plot.xErr() +spec_cr = xs.Plot.y() +spec_cr_err = xs.Plot.yErr() +``` + +Now we can quite easily produce a plot of the spectrum and model: + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +plt.figure(figsize=(6, 4)) +plt.minorticks_on() +plt.tick_params(which="both", direction="in", top=True, right=True) + +plt.errorbar(spec_en, spec_cr, xerr=spec_en_err, yerr=spec_cr_err, fmt="kx", capsize=2) + +plt.xscale("log") + +plt.xlabel("Energy [keV]", fontsize=15) +plt.ylabel(r"Spectrum [ct cm$^{-2}$ s$^{-1}$ keV$^{-1}$]", fontsize=15) + +plt.tight_layout() +plt.show() ``` ## About this notebook @@ -2588,7 +2665,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-08 +Updated On: 2025-12-10 +++ @@ -2604,6 +2681,8 @@ HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/softwa HEASoft XRISM `xaexpmap` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html +XSPEC Model Components: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html + ### Acknowledgements From cb8de0f529030f97942024e9ab278efc2c6574f0 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 20:36:09 -0500 Subject: [PATCH 073/221] Added a test visualization of a light curve in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 49 ++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 83731265..e8e31d11 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -98,7 +98,7 @@ from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc from packaging.version import Version from regions import CircleSkyRegion, Regions -from xga.products import Image +from xga.products import Image, LightCurve ``` ## Global Setup @@ -2476,6 +2476,47 @@ for oi, dcs in rel_dataclasses.items(): ) ``` +#### Loading and displaying a single light curve + +We take a quick look at one of the light curves we just generated to make sure it +looks sensible. First, we specify the ObsID and dataclass of the light curve we will +use as a demonstration, as well as the energy band: + +```{code-cell} python +chosen_demo_lc_obsid = "000128000" +chosen_demo_lc_dataclass = "31100010" +chosen_demo_lc_bnds = Quantity([0.6, 2.0], "keV") +``` + +Now we set up the path and load the light curve into an XGA LightCurve object, as +it has a convenient method to generate visualizations. You could very easily load +the light curve data in directly, using astropy.io.fits, and then plot it yourself: + +```{code-cell} python +demo_lc_path = NET_LC_PATH_TEMP.format( + oi=chosen_demo_lc_obsid, + xdc=chosen_demo_lc_dataclass, + lo=chosen_demo_lc_bnds[0].value, + hi=chosen_demo_lc_bnds[1].value, + lct=0.0, + tb=lc_time_bin.value, +) +demo_lc = LightCurve( + demo_lc_path, + chosen_demo_lc_obsid, + "Xtend", + "", + "", + "", + src_coord_quant, + Quantity(0, "arcmin"), + Quantity(2.0, "arcmin"), + *chosen_demo_lc_bnds, + lc_time_bin, +) +demo_lc.view() +``` + ## 6. Fitting a spectral model to an XRISM-Xtend spectrum Finally, to show off the XRISM-Xtend products we just generated, we will perform @@ -2497,13 +2538,9 @@ Now we configure some behaviors of XSPEC/pyXspec: # the top of your file, but this is unfortunately necessary at the moment import xspec as xs # noqa: E402 +# Limits the amount of output from XSPEC that pyXspec will display xs.Xset.chatter = 0 -# XSPEC parallelisation settings -xs.Xset.parallel.leven = NUM_CORES -xs.Xset.parallel.error = NUM_CORES -xs.Xset.parallel.steppar = NUM_CORES - # Other xspec settings xs.Plot.area = True xs.Plot.xAxis = "keV" From c8785a685234182bcada6b3535f5784edd143a1b Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 20:37:46 -0500 Subject: [PATCH 074/221] Fixed a typo --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index e8e31d11..6e9f7e58 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2618,7 +2618,7 @@ spectrum that, for instance, have very low signal-to-noise, aren't relevant to o science case, or fall outside the optimal energy range of the instrument. Remember that XRISM-Xtend data are not currently trustworthy around or below 0.4 keV, so -we definitely want to exclude that part of the energy range. If we didn't we would be +we definitely want to exclude that part of the energy range. If we didn't, then we would be in danger of biasing our model fitting results, leading to unreliable or unphysical conclusions about our source of interest. From f5678f2976a7713ab5c63fc71f1da7e01ebeb019 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 20:45:46 -0500 Subject: [PATCH 075/221] Restricted the getting-started-xrism-xtend.md notebook to only use 000128000 + added some commentary. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 6e9f7e58..d0752721 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1006,11 +1006,21 @@ For an active mission (i.e., actively collecting data and adding to the archive) public_times = Time(all_xrism_obs["public_date"], format="mjd") avail_xrism_obs = all_xrism_obs[public_times <= Time.now()] -# TODO MAKE MORE PERMANENT ELSEWHERE -avail_xrism_obs = avail_xrism_obs[ - (avail_xrism_obs["obsid"] == "000128000") - | (avail_xrism_obs["obsid"] == "000126000") -] +avail_xrism_obs +``` + +We can see that there are three public XRISM observations of LMC N132D +(as of December 2025) that we could make use of for this demonstration. + +To make sure that this notebook can run in a reasonable amount of time, we +are only going to choose one of them; observation 000128000. + +Please note that we have written this notebook in such a way that you could remove the +first line of the next cell (which selects only one ObsID), and run the notebook +for all public observations. + +```{code-cell} python +avail_xrism_obs = avail_xrism_obs[avail_xrism_obs["obsid"] == "000128000"] # Define a couple of useful variables that make accessing information in the # table a little easier later on in the notebook @@ -1025,8 +1035,6 @@ rel_dataclasses = { ] for oi_ind, oi in enumerate(rel_obsids) } - -avail_xrism_obs ``` ### Downloading the selected XRISM observations @@ -1718,7 +1726,7 @@ plt.show() Based on our inspection, we choose the right dataclass for the '000128000' observation: ```{code-cell} python -rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} +rel_dataclasses = {"000128000": ["31100010"]} ``` ```{hint} From b8cec457aa668da5009031396da0ce592911bca7 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 09:39:33 -0500 Subject: [PATCH 076/221] Made sure that XSPEC model fit is plotted on top of spectrum, improved the x-axis labelling on the spectrum. Some small bug fixes. All in getting-started-xrism-xtend.md, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 41 +++++++++++++++++-- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index d0752721..85c7d8cb 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -96,6 +96,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc +from matplotlib.ticker import FuncFormatter from packaging.version import Version from regions import CircleSkyRegion, Regions from xga.products import Image, LightCurve @@ -2440,7 +2441,7 @@ for oi, dcs in rel_dataclasses.items(): rad=src_reg_rad.to("deg").value.round(4), lo=cur_bnds[0].value, hi=cur_bnds[1].value, - tct=0.0, + lct=0.0, tb=lc_time_bin.to("s").value, ) @@ -2451,7 +2452,7 @@ for oi, dcs in rel_dataclasses.items(): dec=src_coord.dec.value.round(6), lo=cur_bnds[0].value, hi=cur_bnds[1].value, - tct=0.0, + lct=0.0, tb=lc_time_bin.to("s").value, ) @@ -2464,7 +2465,7 @@ for oi, dcs in rel_dataclasses.items(): rad=src_reg_rad.to("deg").value.round(4), lo=cur_bnds[0].value, hi=cur_bnds[1].value, - tct=0.0, + lct=0.0, tb=lc_time_bin.to("s").value, ) @@ -2501,6 +2502,7 @@ it has a convenient method to generate visualizations. You could very easily loa the light curve data in directly, using astropy.io.fits, and then plot it yourself: ```{code-cell} python +# Define the path to the demo net light curve demo_lc_path = NET_LC_PATH_TEMP.format( oi=chosen_demo_lc_obsid, xdc=chosen_demo_lc_dataclass, @@ -2509,6 +2511,8 @@ demo_lc_path = NET_LC_PATH_TEMP.format( lct=0.0, tb=lc_time_bin.value, ) + +# Set up a XGA LightCurve instance demo_lc = LightCurve( demo_lc_path, chosen_demo_lc_obsid, @@ -2522,6 +2526,8 @@ demo_lc = LightCurve( *chosen_demo_lc_bnds, lc_time_bin, ) + +# Show a visualization of the LightCurve demo_lc.view() ``` @@ -2654,6 +2660,15 @@ Our choice of model is empirically driven, chosen by someone who is not a specia xs.Model("tbabs*(powerlaw+apec+bbody)") ``` +If we temporarily increase PyXspec's chatter level, we can see the default values +of each model's parameters: + +```{code-cell} python +xs.Xset.chatter = 10 +xs.AllModels.show() +xs.Xset.chatter = 0 +``` + ### Fitting our pyXspec model to the XRISM-Xtend spectrum Performing the fit is simple: @@ -2662,6 +2677,14 @@ Performing the fit is simple: xs.Fit.perform() ``` +We once again temporarily increase the chatter level, and display the fitted parameters: + +```{code-cell} python +xs.Xset.chatter = 10 +xs.AllModels.show() +xs.Xset.chatter = 0 +``` + ### Visualizing the fitted spectrum We want to use matplotlib to visualize the spectrum data, and the model we @@ -2674,11 +2697,14 @@ we using XSPEC directly: # plot it using matplotlib xs.Plot() -# These read the plotting information into handy variables +# These read out the plotting information for the SPECTRUM spec_en = xs.Plot.x() spec_en_err = xs.Plot.xErr() spec_cr = xs.Plot.y() spec_cr_err = xs.Plot.yErr() + +# And the equivalent for the MODEL +spec_mod_cr = xs.Plot.model() ``` Now we can quite easily produce a plot of the spectrum and model: @@ -2694,9 +2720,16 @@ plt.minorticks_on() plt.tick_params(which="both", direction="in", top=True, right=True) plt.errorbar(spec_en, spec_cr, xerr=spec_en_err, yerr=spec_cr_err, fmt="kx", capsize=2) +plt.plot(spec_en, spec_mod_cr, color="tab:cyan", alpha=0.7, lw=1.1) +# Make sure the energy axis is log scaled plt.xscale("log") +# Alter the formatters for the energy axis so that (for instance) 10 keV is +# displayed as '10 keV' rather than '10^1 keV' +plt.gca().xaxis.set_major_formatter(FuncFormatter(lambda inp, _: "{:g}".format(inp))) +plt.gca().xaxis.set_minor_formatter(FuncFormatter(lambda inp, _: "{:g}".format(inp))) + plt.xlabel("Energy [keV]", fontsize=15) plt.ylabel(r"Spectrum [ct cm$^{-2}$ s$^{-1}$ keV$^{-1}$]", fontsize=15) From 20e5cd6fec0f91fd0860ee90614f37479604721b Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 10:31:36 -0500 Subject: [PATCH 077/221] Added more accurate commentary on the ARF generation, also allow the passing of the minphotons argument to xaarfgen now. In getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 61 ++++++++++++------- 1 file changed, 39 insertions(+), 22 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 85c7d8cb..778bff3e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-10' +date: '2025-12-11' file_format: mystnb jupytext: text_representation: @@ -717,6 +717,7 @@ def gen_xrism_xtend_arf( rmf_file: str, src_radec_reg_file: str, num_photons: int, + min_photons: int, ): """ A wrapper function for the HEASoft `xaarfgen` task, which we use to generate @@ -740,8 +741,12 @@ def gen_xrism_xtend_arf( :param str rmf_file: The path to the RMF file necessary to generate an ARF. :param str src_radec_reg_file: The path to the region file defining the source region for which to generate an ARF. - :param int num_photons: The number of photons to simulate in the ray-tracing - portion of XRISM-Xtend ARF generation. + :param int num_photons: The number of photons, per energy grid point, per + attitude histogram, to simulate in the ray-tracing portion of + XRISM-Xtend ARF generation. + :param int min_photons: The minimum number of photons, per energy grid point, per + attitude histogram, that is required to continue to calculating an ARF at + the end of the ray-tracing portion. """ # We can extract the ObsID directly from the header of the spectrum file - it is @@ -787,6 +792,7 @@ def gen_xrism_xtend_arf( outfile=arf_out, sourcetype="POINT", numphotons=num_photons, + minphotons=min_photons, source_ra=ra_val, source_dec=dec_val, telescop="XRISM", @@ -2259,34 +2265,44 @@ XMA are being simulated, but it does help to produce very accurate ARFs. There a that it can be sped up, though at the cost of that accuracy - the most direct way is to limit the number of events that are simulated. -An argument specifying the number of events can be passed to `xaarfgen`, and for +Rather than setting an overall number of events to simulate, the `xaarfgen` task provides +an argument ('numphoton') to set the number og photons allocated to each attitude +histogram bin (in the exposure map file), per grid point in the internal energy grid. + +An argument specifying the number of events ('numphoton') can be passed to `xaarfgen`, and for our demonstration we are going to use a very small sample - this is primarily so the notebook can run in a reasonable amount of time. -**You should likely use a larger value for real analysis!** -The xaarfgen documentation provides the following guidance on choosing the number of +A second argument, `minphoton`, specifies the minimum acceptable number of raytracing photons that +successfully reach the focal plane for each raytracing energy grid point. If that minimum number is +not reached for each energy grid point during the raytracing process, ARF production will fail. + +The `xaarfgen` documentation provides the following guidance on choosing the number of events to simulate: ```{admonition} -The minimum number of raytracing photons that successfully reach the focal plane, per -raytracing energy grid point, that is acceptable to make an ARF. The number of focal -plane photons that contribute to the ARF must exceed 'minphoton' for every -energy, otherwise the program aborts. Note that even if minphoton is exceeded at -all energies, this does not guarantee that the resulting ARF is robust and -sufficiently accurate. In general, about 5000 or more photons per energy (over the -extraction region) give good results, but the actual minimum number varies -case-by-case, and fewer may be sufficient in some cases. The default value of -minphoton is deliberately very small, in order that the ARF is made and available -for diagnostic evaluation. In general, it is not recommended to set 'minphoton' to a -high value in the first place, because it is not possible to reliably estimate -what 'numphoton' should be in advance of running raytracing within xaarfgen, in order -for that value of 'minphoton' to be satisfied for all energies, which could result -in repeated failures after very long run times. It could also run into memory -problems and/or a raytracing file size that is unmanageable. +Note that even if minphoton is exceeded at all energies, this does not guarantee +that the resulting ARF is robust and sufficiently accurate. + +In general, about 5000 or more photons per energy (over the extraction region) give +good results, but the actual minimum number varies case-by-case, and fewer may be +sufficient in some cases. + +The default value of minphoton is deliberately very small, in order that the +ARF is made and available for diagnostic evaluation. In general, it is not +recommended to set 'minphoton' to a high value in the first place, because it is +not possible to reliably estimate what 'minphoton' should be in advance of +running raytracing within xaarfgen, in order for that value of 'photon' to be +satisfied for all energies, which could result in repeated failures after very long +run times. It could also run into memory problems and/or a raytracing file size that +is unmanageable. ``` +We choose the default values for both the 'minphoton' and 'numphoton' arguments: + ```{code-cell} python arf_rt_num_photons = 20000 +arf_rt_min_photons = 100 ``` So now we move onto actually running the ARF generation - using the @@ -2315,6 +2331,7 @@ arg_combs = [ RMF_PATH_TEMP.format(oi=oi, xdc=dc), radec_src_reg_path.format(oi=oi), arf_rt_num_photons, + arf_rt_min_photons, ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -2743,7 +2760,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-10 +Updated On: 2025-12-11 +++ From c0423dd5fc5f78cfb28e3b3ac2980809a16d52d4 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 10:38:10 -0500 Subject: [PATCH 078/221] Absolute idiot that I am, I wasn't passing the net light curve path to the outfile parameter of lcmath, which might explain why that tool failed!. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 778bff3e..2d75564d 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2128,6 +2128,14 @@ for oi, dcs in rel_dataclasses.items(): bspec_backscals[oi][cur_dc] = back_specco["SPECTRUM"].header["BACKSCAL"] ``` +Showing the BACKSCAL values: + +```{code-cell} python +print(spec_backscals) +print("\n") +print(bspec_backscals) +``` + #### Grouping our new spectra We will group the spectra we just generated. Grouping essentially combines @@ -2498,7 +2506,11 @@ for oi, dcs in rel_dataclasses.items(): # Run the tool to produce a net light curve hsp.lcmath( - infile=cur_lc, bgfile=cur_blc, outfile="", multi=1, multb=cur_back_multi + infile=cur_lc, + bgfile=cur_blc, + outfile=cur_nlc, + multi=1, + multb=cur_back_multi, ) ``` From 8c856ee8232387d680deab63d6bd261fe72243d8 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 12:58:09 -0500 Subject: [PATCH 079/221] Added information on datamodes/classes to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 96 +++++++++---------- 1 file changed, 48 insertions(+), 48 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 2d75564d..82a343f3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -892,6 +892,14 @@ os.makedirs(OUT_PATH, exist_ok=True) # ------------- Set up output file path templates -------------- +# ------ XTDPIPELINE ------- +# Cleaned event list path template - obviously going to be useful later +EVT_PATH_TEMP = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") + +# The path to the bad pixel map, useful for excluding dodgy pixels from data products +BADPIX_PATH_TEMP = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") +# -------------------------- + # --------- IMAGES --------- IM_PATH_TEMP = os.path.join( OUT_PATH, @@ -1202,6 +1210,31 @@ raw_evt_dir_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "xtend", "event_uf") through to generating the 'quick-look' data products (images, spectra, and light curves) included in HEASARC's XRISM archive 'products' directories. +This `xtdpipeline` task will prepare **all** data taken during a particular +observation - that means that if you are using an XRISM-Xtend observation that +was running in a data mode other than full-window, you still only have to +run `xtdpipeline` once. + +Full-window should be considered XRISM-Xtend's default data mode, but you are likely +to come across data taken in other modes. Those modes are described in the XRISM +ABC guide ([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/proposals/POG/Xtend_SXI.html#SECTION00920000000000000000)), +and we summarise them here: + +- **Full-window** - The entire Xtend detector is in the same data mode, producing a 640x640 (in raw CCD coordinates) image, at a 4 second time resolution. +- **1/8th window [NO BURST]** - Half of the detector (2 CCDs) operates 'normally' and the other has only 1/8th of the pixel rows operating. Produces a 640x80 image, at 0.5 second time resolution. +- **1/8th window [BURST]** - Similar to the 1/8th window mode but collects exposures during only a small fraction of the effective detector exposures (the 0.5 second time resolution). Avoids pile-up for very bright sources, and allows the determination of photon arrival times with ~0.06 second accuracy. + +Data taken in each data mode is assigned a different 'dataclass' so that the multiple +event lists produced when using a 1/8th window mode can be distinguished from the event +list of the half the detector that is operating 'normally'. The dataclasses are discussed in the XRISM ABC guide +([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html)) and are summarised below: +- **30000010** - All four CCDs are in full window mode. +- **31100010** - CCD1 & CCD2 in 1/8 window mode. +- **31200010** - CCD1 & CCD2 in full window + 0.1 sec burst mode. +- **31300010** - CCD1 & CCD2 in 1/8 window + 0.1 sec burst mode +- **32000010** - CCD3 & CCD4 in full window mode + + The pipeline has three stages and provides the option to start and stop the processing at any of those stages; this can be useful if you wish to re-run a stage with slightly different configuration without repeating the entire pipeline run. @@ -1211,13 +1244,6 @@ A different set of tasks is encapsulated by each stage, and they have the follow - **Stage 2** - Screening and filtering of the prepared Xtend event lists. - **Stage 3** - Generation of quick-look data products. - -***MUCH MORE SPECIFIC INFORMATION SHOULD GO HERE*** - -***INCLUDE XTEND DATA CLASS SUMMARY, AND HOW THERE MAY BE MULTIPLE EVENT LISTS PER OBSID*** - - - ```{note} We will stop the execution of `xtdpipeline` at **Stage 2**, as the latter part of this demonstration will show you how to make more customised data products than are output @@ -1282,34 +1308,6 @@ if len(xtd_pipe_problem_ois) != 0: Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` -Finally, we set up some template variables for the various useful files output by the -XRISM-Xtend processing pipeline. These include the cleaned event lists we just created, - -cleaned event lists we just created: - -***CONSIDER MORE COMMENTARY ON DATACLASS ETC. HERE*** - - -***Need to consider the 'p0' bit - apparently the '0' is a counter for splitting large datasets, but I don't know when it is used*** - -***sc == 'split counter'*** - -```{code-cell} python -# Cleaned event list path template - obviously going to be useful later -evt_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") - -# The path to the bad pixel map, useful for excluding dodgy pixels from data products -badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") -``` - -### Good-time-intervals - -### Identifying problem pixels - -```{code-cell} python - -``` - ## 3. Generating new XRISM-Xtend images and exposure maps The XRISM-Xtend data have now been prepared for scientific use, with the most important @@ -1556,7 +1554,7 @@ is a good idea to run them in parallel! ```{code-cell} python arg_combs = [ [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), *cur_bnds, cur_bf, @@ -1622,11 +1620,11 @@ Finally, we run the exposure map generation: ```{code-cell} python arg_combs = [ [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), ehk_path_temp.format(oi=oi), - badpix_path_temp.format(oi=oi, xdc=dc, sc=0), + BADPIX_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), "NONE", cur_bf, expmap_rad_delta, @@ -2047,7 +2045,7 @@ spectrum like we did with the different-energy-band images. ```{code-cell} python arg_combs = [ [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), src_coord, src_reg_rad, @@ -2415,7 +2413,7 @@ light curves simultaneously: ```{code-cell} python arg_combs = [ [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), src_coord, src_reg_rad, @@ -2778,17 +2776,17 @@ Updated On: 2025-12-11 ### Additional Resources -XRISM Data Reduction (ABC) Guide - https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide +**XRISM Data Reduction (ABC) Guide**: https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide -HEASoftPy GitHub Repository: https://github.com/HEASARC/heasoftpy +**HEASoftPy GitHub Repository**: https://github.com/HEASARC/heasoftpy -HEASoftPy HEASARC Page: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/heasoftpy.html +**HEASoftPy HEASARC Page**: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/heasoftpy.html -HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html +**HEASoft XRISM `xtdpipeline` help file**: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html -HEASoft XRISM `xaexpmap` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html +**HEASoft XRISM `xaexpmap` help file**: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html -XSPEC Model Components: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html +**XSPEC Model Components**: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html ### Acknowledgements @@ -2797,4 +2795,6 @@ XSPEC Model Components: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual [XRISM GOF & SDC (2024) - _XRISM ABC GUIDE XTEND ENERGY-CHANNEL MAPPING_ [ACCESSED 25-NOV-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/Xtend_Data_Analysis.html#SECTION001043000000000000000) -[](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html) +[XRISM GOF & SDC (2024) - _XRISM ABC GUIDE FILE NAMING CONVENTIONS_ [ACCESSED 11-DEC-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html) + +[XRISM GOF & SDC (2024) - _XRISM ABC GUIDE XTEND DATA MODES_ [ACCESSED 11-DEC-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/proposals/POG/Xtend_SXI.html#SECTION00920000000000000000) From 41e6c8574a9dff1f5c5d5a425ea1ea68ab235483 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 13:10:22 -0500 Subject: [PATCH 080/221] May possibly have actually finished the writing of getting-started-xrism-xtend.md now. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 51 ++++++++++++++++++- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 82a343f3..bc5edb77 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -978,9 +978,28 @@ ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") ## 1. Finding and downloading XRISM observations of LMC N132D +Our first task is to determine which XRISM observations are relevant to the source +that we are interested in. + +We are going in with the knowledge that LMC N132D has been observed by XRISM, but of +course there is no guarantee that _your_ source of interest has been, so this is +an important exploratory step. ### Determining the name of the XRISM observation summary table +HEASARC maintains tables that contain information about every observation taken by +each of the missions in its archive. We will use XRISM's table to find observations +that should be relevant to our source. + +The name of the XRISM observation summary table is 'xrismmastr', but as you may not +know that a priori, we demonstrate how to identify the correct table for a given +mission. + +Using the AstroQuery Python module (specifically this Heasarc object), we list all +catalogs that are a) related to XRISM, and b) are flagged as 'master' (meaning the +summary table of observations). This should only return one catalog for any +mission you pass to 'keywords': + ```{code-cell} python catalog_name = Heasarc.list_catalogs(master=True, keywords="xrism")[0]["name"] catalog_name @@ -990,7 +1009,7 @@ catalog_name To search for relevant observations, we have to know the coordinates of our source. The astropy module allows us to look up a source name in CDS' Sesame name - resolver and retrieve its coordinates. +resolver and retrieve its coordinates. ```{hint} You could also set up a SkyCoord object directly, if you already know the coordinates. @@ -1006,6 +1025,15 @@ src_coord ### Searching for relevant observations +Now that we know which catalog to search, and the coordinates of our source, we use +AstroQuery to retrieve those lines of the summary table that are within some radius +of the source coordinate. In this case we're using the default search radius for +the XRISM summary table, but you can also pass a `radius` argument to set your own. + +In this case, we also define a custom set of columns to retrieve, as the default set +does not contain the 'xtd_dataclas*' columns that we might need later. You may also +pass a wildcard `columns='*'` to retrieve all available columns. + ```{code-cell} python col_str = ( "__row,obsid,name,ra,dec,time,exposure,status,public_date," @@ -1015,7 +1043,16 @@ all_xrism_obs = Heasarc.query_region(src_coord, catalog_name, columns=col_str) all_xrism_obs ``` -For an active mission (i.e., actively collecting data and adding to the archive)... +For an active mission (i.e., actively collecting data and adding to the archive), we +will, at some point, probably come across observations that have been taken, but are +currently only available to their proposers (still in the proprietary period). + +Such proprietary observations will still appear in the XRISM summary table, and the +files could even be downloaded, but unless we took those data we won't have the +key necessary to decrypt the files. + +As such, we are going to use the 'public_date' column to filter out any observations +that are not yet publicly available: ```{code-cell} python public_times = Time(all_xrism_obs["public_date"], format="mjd") @@ -1054,11 +1091,19 @@ rel_dataclasses = { ### Downloading the selected XRISM observations +The AstroQuery `Heasarc` module makes it easy to download the data we need. Our +cut-down table of observations can be passed to the `locate_data()` method, which +will return the access links for the data on several different platforms: + ```{code-cell} python data_links = Heasarc.locate_data(avail_xrism_obs) data_links ``` +That data links table can now be passed straight to the `download_data()` method, which +will do what it says on the tin and download the files. We can also specify which +platform to pull the observations from, and in this case we select the HEASARC AWS S3 bucket: + ```{code-cell} python Heasarc.download_data(data_links, "aws", ROOT_DATA_DIR) ``` @@ -1071,6 +1116,8 @@ on SciServer, you may pass 'sciserver' to use the pre-mounted HEASARC dataset. ### What do the downloaded data directories contain? +Now we can take a quick look at the contents of the directory we just downloaded: + ```{code-cell} python glob.glob(os.path.join(ROOT_DATA_DIR, rel_obsids[0], "") + "*") ``` From 9980d17d8cd5fc1100fccea7d594e03460ecc3a9 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 15:27:01 -0500 Subject: [PATCH 081/221] Tiny aethsetic change to getting-started-xrism-xtend.md --- .../xrism/getting-started-xrism-xtend.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index bc5edb77..f4fd8529 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -74,7 +74,7 @@ We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstr ### Runtime -As of 8th December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 8th December 2025, this notebook takes ~50m to run to completion on Fornax using the 'Default Astrophysics' image and the medium server with 16GB RAM/ 4 cores. ## Imports @@ -1188,7 +1188,7 @@ if HEA_VER < Version("V6.36"): raise ValueError( "We strongly recommend using HEASoft v6.36 or later for this " "tutorial - you may run rest of the notebook yourself, but " - "ARF generation will likely fail." + "ARF generation will either fail or produce an incorrect result." ) ``` @@ -2177,7 +2177,6 @@ Showing the BACKSCAL values: ```{code-cell} python print(spec_backscals) -print("\n") print(bspec_backscals) ``` @@ -2789,7 +2788,7 @@ tags: [hide-input] jupyter: source_hidden: true --- -plt.figure(figsize=(6, 4)) +plt.figure(figsize=(7, 4)) plt.minorticks_on() plt.tick_params(which="both", direction="in", top=True, right=True) From 1c316cc3611d6ff44f80d360a6e39b622f97b5bf Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 16:19:10 -0500 Subject: [PATCH 082/221] Fixed a bug in the filling out of the demo_lc_path variable in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f4fd8529..267bcc60 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2190,8 +2190,6 @@ this, once again through HEASoftPy. First, we set up the grouping criteria and a template variable for the name of the output grouped spectral files: -*** REMIND MYSELF WHETHER GROUPING FROM SOURCE SPEC IS AUTOMATICALLY APPLIED TO BACK SPEC IN XSPEC? *** - ```{code-cell} python spec_group_type = "min" spec_group_scale = 1 @@ -2579,6 +2577,9 @@ the light curve data in directly, using astropy.io.fits, and then plot it yourse demo_lc_path = NET_LC_PATH_TEMP.format( oi=chosen_demo_lc_obsid, xdc=chosen_demo_lc_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), lo=chosen_demo_lc_bnds[0].value, hi=chosen_demo_lc_bnds[1].value, lct=0.0, From 298c42d1ffc31fe52266eede99547fdcfa35e20a Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 16:34:10 -0500 Subject: [PATCH 083/221] Pointed the getting-started-xrism-xtend.md notebook to the S3 bucket for CALDB access. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 267bcc60..12787b7a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -857,6 +857,10 @@ hsp.Config.allow_failure = False mp.set_start_method("fork", force=True) # -------------------------------------------------------------- +# ----------- Set HEASoft to use the S3-bucket CALDB ----------- +os.environ["CALDB"] = "https://nasa-heasarc.s3.amazonaws.com/caldb" +# -------------------------------------------------------------- + # ------------- Setting how many cores we can use -------------- NUM_CORES = None total_cores = os.cpu_count() @@ -1340,6 +1344,10 @@ rel_dataclasses = {oi: rel_dataclasses[oi] for oi in rel_obsids} xtd_pipe_problem_ois ``` +```{warning} +Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. +``` + We also include a code snippet that will print the output of the `xtdpipeline` run for any observations that appear to have failed: @@ -1351,8 +1359,12 @@ if len(xtd_pipe_problem_ois) != 0: print("\n\n") ``` -```{warning} -Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. +```{note} +This notebook is configured to acquire XRISM CALDB files from the HEASARC +Amazon Web Services S3 bucket - this can greatly improve the speed of some +steps later in the notebook when running on the Fornax science console. + +CALDB location configuration can be found in the 'Global Setup: Configuration' section. ``` ## 3. Generating new XRISM-Xtend images and exposure maps From 641bdc397cdbadfd562bf8caf48973e97a2d6c2a Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 16:53:15 -0500 Subject: [PATCH 084/221] Temporarily disabling the error on too-low of a HEASoft version in the getting-started-xrism-xtend.md notebook. To maybe have it build on CI/CD? For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 12787b7a..0cf80259 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1188,12 +1188,12 @@ Now we can use inequality operators on another Version object to check if our ve of HEASoft meets the requirements: ```{code-cell} python -if HEA_VER < Version("V6.36"): - raise ValueError( - "We strongly recommend using HEASoft v6.36 or later for this " - "tutorial - you may run rest of the notebook yourself, but " - "ARF generation will either fail or produce an incorrect result." - ) +# if HEA_VER < Version("V6.36"): +# raise ValueError( +# "We strongly recommend using HEASoft v6.36 or later for this " +# "tutorial - you may run rest of the notebook yourself, but " +# "ARF generation will either fail or produce an incorrect result." +# ) ``` ### Setting up file paths to pass to the XRISM-Xtend pipeline From 524899ab81996a22557252f21fbfff83203c7d9f Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 17:17:13 -0500 Subject: [PATCH 085/221] Removed a blank line --- .../xrism/getting-started-xrism-xtend.md | 1 - 1 file changed, 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0cf80259..02847a8c 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1285,7 +1285,6 @@ list of the half the detector that is operating 'normally'. The dataclasses are - **31300010** - CCD1 & CCD2 in 1/8 window + 0.1 sec burst mode - **32000010** - CCD3 & CCD4 in full window mode - The pipeline has three stages and provides the option to start and stop the processing at any of those stages; this can be useful if you wish to re-run a stage with slightly different configuration without repeating the entire pipeline run. From cc1ac3c5dbb03fdf37fd38f171c45753845dd74f Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 21:05:59 -0500 Subject: [PATCH 086/221] Made some of the event list and region file paths passed to HEASoft tasks into relative paths, to save on string length and the problems it can cause in HEASoft. In getting-started-xrism-xtend.md, for PR #150 --- .../xrism/getting-started-xrism-xtend.md | 30 +++++++++++-------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 02847a8c..0a8da9b3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -496,13 +496,11 @@ def gen_xrism_xtend_lightcurve( # Convert the energy limits to channel limits, rounding down and up to the nearest # integer channel for the lower and upper bounds respectively. + # We will use these to make a channel selection in the event list passed + # to the tool lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) - # Create modified input event list file path, where we use the just-calculated - # PI channel limits to subset the events - evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" - # Set up the output file name for the light curve we're about to generate. lc_out = os.path.basename(LC_PATH_TEMP).format( oi=cur_obs_id, @@ -539,12 +537,18 @@ def gen_xrism_xtend_lightcurve( # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + # Create modified input event list file path, where we use the just-calculated + # PI channel limits to subset the events + # This is within the chdir context, as we want to pass a relative path + # as very long string arguments can be cut off when passed to HEASoft tools + evt_file_chan_sel = os.path.relpath(event_file) + f"[PI={lo_ch}:{hi_ch}]" + src_out = hsp.extractor( filename=evt_file_chan_sel, fitsbinlc=lc_out, binlc=time_bin_size, lcthresh=lc_bin_thresh, - regionfile=src_reg_file, + regionfile=os.path.relpath(src_reg_file), xcolf="X", ycolf="Y", gti="GTI", @@ -558,7 +562,7 @@ def gen_xrism_xtend_lightcurve( fitsbinlc=lc_back_out, binlc=time_bin_size, lcthresh=lc_bin_thresh, - regionfile=back_reg_file, + regionfile=os.path.relpath(back_reg_file), xcolf="X", ycolf="Y", gti="GTI", @@ -637,9 +641,9 @@ def gen_xrism_xtend_spectrum( # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): src_out = hsp.extractor( - filename=event_file, + filename=os.path.relpath(event_file), phafile=sp_out, - regionfile=src_reg_file, + regionfile=os.path.relpath(src_reg_file), xcolf="X", ycolf="Y", ecol="PI", @@ -650,9 +654,9 @@ def gen_xrism_xtend_spectrum( # Now for the background light curve back_out = hsp.extractor( - filename=event_file, + filename=os.path.relpath(event_file), phafile=sp_back_out, - regionfile=back_reg_file, + regionfile=os.path.relpath(back_reg_file), xcolf="X", ycolf="Y", ecol="PI", @@ -797,9 +801,9 @@ def gen_xrism_xtend_arf( source_dec=dec_val, telescop="XRISM", instrume="XTEND", - emapfile=expmap_file, - rmffile=rmf_file, - regionfile=src_radec_reg_file, + emapfile=os.path.relpath(expmap_file), + rmffile=os.path.relpath(rmf_file), + regionfile=os.path.relpath(src_radec_reg_file), regmode="RADEC", noprompt=True, clobber=True, From 04ebecfe94d6ea17d321d1bebf2e14b3777a0a3a Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 12 Dec 2025 08:35:39 -0500 Subject: [PATCH 087/221] Tiny wording change, mostly to get CircleCI going again (hopefully the HEASARC website is a little happier today). For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0a8da9b3..b3563ed0 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1035,8 +1035,8 @@ src_coord Now that we know which catalog to search, and the coordinates of our source, we use AstroQuery to retrieve those lines of the summary table that are within some radius -of the source coordinate. In this case we're using the default search radius for -the XRISM summary table, but you can also pass a `radius` argument to set your own. +of the source coordinate. We're using the default search radius for +the XRISM summary table, but you can pass a `radius` argument to set your own. In this case, we also define a custom set of columns to retrieve, as the default set does not contain the 'xtd_dataclas*' columns that we might need later. You may also From 9818017e98007e8fe360a134e3c1cac141e0dc24 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 12 Dec 2025 12:47:50 -0500 Subject: [PATCH 088/221] Replaced a misuse of admonition with seealso, in the getting-started-xrism-xtend.md notebook --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b3563ed0..3b6c5019 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2345,7 +2345,7 @@ not reached for each energy grid point during the raytracing process, ARF produc The `xaarfgen` documentation provides the following guidance on choosing the number of events to simulate: -```{admonition} +```{seealso} Note that even if minphoton is exceeded at all energies, this does not guarantee that the resulting ARF is robust and sufficiently accurate. From 5f6f024cd460a1880d6725413b16c396ac02aa45 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 15 Dec 2025 10:36:49 -0500 Subject: [PATCH 089/221] Some very small typos fixed --- .../xrism/getting-started-xrism-xtend.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 3b6c5019..c8adfbe2 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -834,8 +834,6 @@ jupyter: --- # The name of the source we're examining in this demonstration SRC_NAME = "LMCN132D" -# SRC_NAME = "NGC4151" -# SRC_NAME = "AX J1910.7+0917" # Controls the verbosity of all HEASoftPy tasks TASK_CHATTER = 3 @@ -1419,8 +1417,8 @@ Alternatively, we can figure out this relationship between PI and energy by look a XRISM-Xtend Redistribution Matrix File (RMF), which exists to describe this mapping. -We will be creating new RMFs as part of the generation of XRISM-Xtend spectra in the -next section. For our current purpose, however, it is acceptable to use the RMFs that +We will be creating new RMFs as part of the generation of XRISM-Xtend spectra later +in this notebook. For our current purpose, however, it is acceptable to use the RMFs that were included in the XRISM-Xtend archive we downloaded earlier. The archived RMFs are generated for the entire Xtend FoV, rather than for the CCDs From c0e1a493e9fefe6199eefa59854ee319849156ed Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:04:40 -0500 Subject: [PATCH 090/221] Restored XMM CCF download step in config.yml --- .circleci/config.yml | 56 ++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index d13bca94..5316832a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -44,34 +44,34 @@ jobs: # Some missions (e.g. XMM, Chandra, eROSITA) require that their calibration files are available locally. We # don't want to download them every single time, so we're going to cache them for future runs. -# - run: -# name: Acquiring/validating XMM-CCFs -# no_output_timeout: 30m -# command: | -# # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same -# # version number as currently defined in the environment variable, we have to download the data -# if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then -# -# # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the -# # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions -# # available, so we don't want to delete the directory. -# mkdir -p /home/jovyan/xmm-ccf/ccf-files -# -# # We do delete the version file, if it exists, as if we get to this point then we are either downloading -# # the data for the first time, or the version number has changed. -# [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver -# -# # Make a new version file - this goes a level up from the directory where the files will actually -# # be stored (ccf-files), because the rsync process will delete the version file -# echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver -# -# # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The -# # neatest way is to make a new conda environment that just contains rsync -# micromamba create -n rsync-env -y -c conda-forge rsync -# -# # Actually rsync the calibration files -# micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files -# fi + - run: + name: Acquiring/validating XMM-CCFs + no_output_timeout: 30m + command: | + # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same + # version number as currently defined in the environment variable, we have to download the data + if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then + + # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the + # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions + # available, so we don't want to delete the directory. + mkdir -p /home/jovyan/xmm-ccf/ccf-files + + # We do delete the version file, if it exists, as if we get to this point then we are either downloading + # the data for the first time, or the version number has changed. + [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver + + # Make a new version file - this goes a level up from the directory where the files will actually + # be stored (ccf-files), because the rsync process will delete the version file + echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver + + # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The + # neatest way is to make a new conda environment that just contains rsync + micromamba create -n rsync-env -y -c conda-forge rsync + + # Actually rsync the calibration files + micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files + fi # We also download the Chandra CalDB - run: From 8d45f7abcfa353354f50ddacf956f356a1f6716c Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:07:51 -0500 Subject: [PATCH 091/221] Returning nb_execution_timeout to 1200 in conf.py - might alter it back at some point though --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index 5a37b94d..05e8732a 100644 --- a/conf.py +++ b/conf.py @@ -32,7 +32,7 @@ '**/*_template*', '**/README.md', '*_template*'] # MyST-NB configuration -nb_execution_timeout = 2400 +nb_execution_timeout = 1200 nb_merge_streams = True nb_execution_mode = "cache" # nb_execution_mode = "force" From c20003c2762a113a153492827acec96d5b5493f6 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 19 Nov 2025 16:08:15 -0500 Subject: [PATCH 092/221] Added the getting started with XRISM Xtend notebook, currently just the template, as well as the necessary index entries. For issue #128 --- index.md | 1 + .../xrism/getting-started-xrism-xtend.md | 268 ++++++++++++++++++ .../xrism/xrism_index.md | 13 + 3 files changed, 282 insertions(+) create mode 100644 tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md create mode 100644 tutorials/mission_specific_analyses/xrism/xrism_index.md diff --git a/index.md b/index.md index 3d055cb2..6655d891 100644 --- a/index.md +++ b/index.md @@ -36,6 +36,7 @@ caption: Mission specific tutorials tutorials/mission_specific_analyses/nustar/nustar_index tutorials/mission_specific_analyses/rxte/rxte_index tutorials/mission_specific_analyses/swift/swift_index +tutorials/mission_specific_analyses/xrism/xrism_index ``` ## Demonstrations of useful high-energy tools diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md new file mode 100644 index 00000000..37a5aec7 --- /dev/null +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -0,0 +1,268 @@ +--- +authors: +- name: David Turner + affiliations: ['University of Maryland, Baltimore County', 'HEASARC, NASA Goddard'] + email: djturner@umbc.edu + orcid: 0000-0001-9658-1396 + website: https://davidt3.github.io/ +- name: Kenji Hamaguchi + affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] + website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 + orcid: 0000-0001-7515-2779 +date: '2025-11-19' +file_format: mystnb +jupytext: + text_representation: + extension: .md + format_name: myst + format_version: 1.3 + jupytext_version: 1.17.3 +kernelspec: + display_name: heasoft + language: python + name: heasoft +title: Getting started with XRISM-Xtend +--- + +# Getting started with XRISM-Xtend + +## Learning Goals + +By the end of this tutorial, you will be able to: + +- Find... +- + +## Introduction + +Alter this file according to your use case but retain the basic structure and try to use the same syntax for things like section headings, numbering schemes, and bullet points. +Specifically the headings in this Intro section should not be edited to maintain consistency between notebooks. + +All contributed notebooks should be in [MyST markdown](https://mystmd.org) format. +See the [Fornax documentation](https://docs.fornax.sciencecloud.nasa.gov/markdown-and-code-dev) for more info about this. + +The Introduction should provide context and motivation. +Why should someone use this notebook? +Give background on the science or technical problem. +Point out the parts that are particularly challenging and what solutions we chose for what reasons. + +### Inputs + +- List the data, catalogs, or files needed, and where they come from. + If there are data that get downloaded to Fornax as part of this notebook, place those in a `data` directory. + Please do not change the name of this directory for consistency with other notebooks. + Do not add the contents of `data` to the repo, just the empty directory. + +### Outputs + +- List the products the notebook generates (plots, tables, derived data, etc.) +- If there are intermediate products produced by your notebook, generate an `output` directory for those data. + Please do not change the name of this directory for consistency with other notebooks. + Do not add the contents of `output` to the repo, just the empty directory. + +### Runtime + +Please report actual numbers and machine details for your notebook if it is expected to run longer or requires specific machines, for example, on Fornax. +Also, if querying archives, please include a statement like: +"This runtime is heavily dependent on archive servers which means runtime will vary for users". + +Here is a template runtime statement: +As of {Date}, this notebook takes ~{N}s to run to completion on Fornax using the ‘Default Astrophysics' image and the ‘{name: size}’ server with NGB RAM/ NCPU. + +## Imports + +This should be a list of the modules that are required to run this code. +Importantly, even those that are already installed in Fornax should be listed here so users wanting to run this locally on their own machines have the information they need to do this. + +Make sure that you have built a "requirements_notebook_name.txt" file with the modules to be imported. +The name of the notebook should be present in the name of the requirements file, as in our example "requirements_notebook_template.txt" + +```{code-cell} python +# This cell should not be edited below this line except for the name of +# the requirements_notebook_name.txt + +# Uncomment the next line to install dependencies if needed. +# %pip install -r requirements_notebook_name.txt +``` + +```{code-cell} python +import numpy as np +``` + +## Global Setup + +### Functions + +Please avoid writing functions where possible, but if they are necessary, then place them in the following +code cell - it will be minimized unless the user decides to expand it. **Please replace this text with concise +explanations of your functions or remove it if there are no functions.** + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + +# This cell will be automatically collapsed when the notebook is rendered, which helps +# to hide large and distracting functions while keeping the notebook self-contained +# and leaving them easily accessible to the user +``` + +### Constants + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + +``` + +### Configuration + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + +``` + +*** + +## 1. Data Access + +The name of this, and all future sections can change. +In general, it probably is a good idea to start with something like "Data Access". +Please note, and stick to, the existing numbering scheme. + +```{code-cell} python +# Create some example data. +data = np.random.randint(0, 100, size=100) +``` + +## 2. Data Exploration + +Describe what the data look like. +Add summary statistics, initial plots, sanity checks. + +For cuts or other data filtering and cleaning steps, explain the scientific reasoning behind them. +This helps people understand both the notebook and the data so that they're more equipped to use the data appropriately in other contexts. + ++++ + +:::{tip} +Please include a narrative for *all* your code cells to help the reader figure out what you are doing and why you chose that path. + +Using [MyST admonitions](https://mystmd.org/guide/admonitions) such as this `tip` are encouraged +::: + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + +hist, bin_edges = np.histogram(data, bins=10) +hist +``` + +:::{important} +The HEASARC-tutorials style guide requires that cells using matplotlib (or similar) to produce figures +should be isolated (i.e., only contain plotting code), and must include the following metadata to hide the +code from view (see the source of this cell for the unrendered text: + +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +::: + +For any Figures, please add a few sentences about what the users should be noticing. + ++++ + +## 3. Analysis + +The working part of the notebook. +Lay out the step-by-step analysis workflow. +Each subsection should describe what is being done and why. +These can be sections or subsections. + ++++ + +### 3.1 Design Principles + +- Make no assumptions: define terms, common acronyms, link to things you reference. +- Keep in mind who your audience is. +- Design for portability - will this notebook work on both Fornax and someone's individual laptop. +- Cells capture logical units of work. +- Use markdown before or after cells to describe what is happening in the notebook. + ++++ + +### 3.2 Style Principles + +- Follow suggestions of The Turing Way community [markdown style](https://book.the-turing-way.org/community-handbook/style) +- Write each sentence in a new line (line breaks) to make changes easier to read in PRs. +- Avoid latin abbreviation to avoid failing CI. + +#### 3.2.1 Best Practice Guidelines +It would be nice if all contributed codes did the following, but these guidelines will not be checked in a code review + +- Section titles should not end with ":". +- List items should start at the beginning of the line, no spaces first. Exception is nested lists. +- One empty line between section header and text. +- One empty line before a list and after. +- No more than one empty line between any two non-empty lines. + +```{code-cell} python + +``` + +## 4. PR Review + +Notebooks go through a two step process: first step is getting into the repo, and the second step gets it into the [published tutorials](https://nasa-fornax.github.io/fornax-demo-notebooks/). +Final notebooks are expected to go through both a science and tech review checklist. +Checklists are [here](https://github.com/nasa-fornax/fornax-demo-notebooks/tree/main/template/notebook_review_checklists.md). +Please consider these checklist requirements as you are writing your code. + +The first PR of a notebook does not need to have everything from the checklists completed, but should have all the pieces there, and the authors should be aware of the requirements. + +To complete the second step of this process and be both rendered and included in users Fornax home directories, both a science and technical reviewer will be looking at [this checklist](https://github.com/nasa-fornax/fornax-demo-notebooks/tree/main/template/notebook_review_checklists.md) to see if the new tutorial notebook meets all of the requirements, or has a reasonable excuse not to. + +Any PRs can be opened as drafts, which is in fact preferred, if authors are still working on them. + ++++ + +## About this notebook + +- **Authors:** Specific author and/or team names, plus "and the Fornax team". +- **Contact:** For help with this notebook, please open a topic in the [Fornax Community Forum](https://discourse.fornax.sciencecloud.nasa.gov/) "Support" category. +- Please edit and keep the above 2 bullet points, and remove this last line. + ++++ + +### Additional Resources + + +### Acknowledgements + +Did anyone help you? +Probably these teams did, so include them: MAST, HEASARC, & IRSA Fornax teams. + +Did you use AI for any part of this tutorial, if so please include a statement such as: +"AI: This notebook was created with assistance from OpenAI’s ChatGPT 5 model.", which is a good time to mention that this template notebook was created with assistance from OpenAI’s ChatGPT 5 model. + +### References + +This work made use of: + +- STScI style guide: https://github.com/spacetelescope/style-guides/blob/master/guides/jupyter-notebooks.md +- Fornax tech and science review guidelines: https://github.com/nasa-fornax/fornax-demo-notebooks/blob/main/template/notebook_review_checklists.md +- The Turing Way Style Guide: https://book.the-turing-way.org/community-handbook/style diff --git a/tutorials/mission_specific_analyses/xrism/xrism_index.md b/tutorials/mission_specific_analyses/xrism/xrism_index.md new file mode 100644 index 00000000..65b32608 --- /dev/null +++ b/tutorials/mission_specific_analyses/xrism/xrism_index.md @@ -0,0 +1,13 @@ +# XRISM + +Here we include notebooks that demonstrate how to use data taken by the XRISM mission. + + +```{toctree} +--- +maxdepth: 2 +caption: XRISM tutorials +--- +getting-started-xrism-xtend + +``` From 3af4be7b78d7e96a2347a4c5e225fc15e0132821 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 19 Nov 2025 18:11:02 -0500 Subject: [PATCH 093/221] Started adding things to the global configuration section of the getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 229 ++++++------------ 1 file changed, 68 insertions(+), 161 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 37a5aec7..04a3d274 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -15,7 +15,7 @@ jupytext: text_representation: extension: .md format_name: myst - format_version: 1.3 + format_version: 0.13 jupytext_version: 1.17.3 kernelspec: display_name: heasoft @@ -35,68 +35,46 @@ By the end of this tutorial, you will be able to: ## Introduction -Alter this file according to your use case but retain the basic structure and try to use the same syntax for things like section headings, numbering schemes, and bullet points. -Specifically the headings in this Intro section should not be edited to maintain consistency between notebooks. - -All contributed notebooks should be in [MyST markdown](https://mystmd.org) format. -See the [Fornax documentation](https://docs.fornax.sciencecloud.nasa.gov/markdown-and-code-dev) for more info about this. - -The Introduction should provide context and motivation. -Why should someone use this notebook? -Give background on the science or technical problem. -Point out the parts that are particularly challenging and what solutions we chose for what reasons. ### Inputs -- List the data, catalogs, or files needed, and where they come from. - If there are data that get downloaded to Fornax as part of this notebook, place those in a `data` directory. - Please do not change the name of this directory for consistency with other notebooks. - Do not add the contents of `data` to the repo, just the empty directory. +- ### Outputs -- List the products the notebook generates (plots, tables, derived data, etc.) -- If there are intermediate products produced by your notebook, generate an `output` directory for those data. - Please do not change the name of this directory for consistency with other notebooks. - Do not add the contents of `output` to the repo, just the empty directory. +- ### Runtime -Please report actual numbers and machine details for your notebook if it is expected to run longer or requires specific machines, for example, on Fornax. -Also, if querying archives, please include a statement like: -"This runtime is heavily dependent on archive servers which means runtime will vary for users". - -Here is a template runtime statement: -As of {Date}, this notebook takes ~{N}s to run to completion on Fornax using the ‘Default Astrophysics' image and the ‘{name: size}’ server with NGB RAM/ NCPU. +As of 22nd November 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports -This should be a list of the modules that are required to run this code. -Importantly, even those that are already installed in Fornax should be listed here so users wanting to run this locally on their own machines have the information they need to do this. +```{code-cell} python +# import contextlib +import multiprocessing as mp +import os -Make sure that you have built a "requirements_notebook_name.txt" file with the modules to be imported. -The name of the notebook should be present in the name of the requirements file, as in our example "requirements_notebook_template.txt" +import heasoftpy as hsp -```{code-cell} python -# This cell should not be edited below this line except for the name of -# the requirements_notebook_name.txt +# from typing import Tuple, Union +# from warnings import warn -# Uncomment the next line to install dependencies if needed. -# %pip install -r requirements_notebook_name.txt -``` -```{code-cell} python -import numpy as np +# import matplotlib.pyplot as plt +# import numpy as np +# import xspec as xs +# from astropy.coordinates import SkyCoord +# from astropy.units import Quantity +# from astroquery.heasarc import Heasarc +# from matplotlib.ticker import FuncFormatter +# from tqdm import tqdm ``` ## Global Setup ### Functions -Please avoid writing functions where possible, but if they are necessary, then place them in the following -code cell - it will be minimized unless the user decides to expand it. **Please replace this text with concise -explanations of your functions or remove it if there are no functions.** - ```{code-cell} python --- tags: [hide-input] @@ -104,9 +82,6 @@ jupyter: source_hidden: true --- -# This cell will be automatically collapsed when the notebook is rendered, which helps -# to hide large and distracting functions while keeping the notebook self-contained -# and leaving them easily accessible to the user ``` ### Constants @@ -117,7 +92,11 @@ tags: [hide-input] jupyter: source_hidden: true --- +# The name of the source we're examining in this demonstration +SRC_NAME = "AX J1910.7+0917" +# Controls the verbosity of all HEASoftPy tasks +TASK_CHATTER = 3 ``` ### Configuration @@ -128,123 +107,62 @@ tags: [hide-input] jupyter: source_hidden: true --- - +# ------------- Configure global package settings -------------- +# Raise Python exceptions if a heasoftpy task fails +# TODO Remove once this becomes a default in heasoftpy +hsp.Config.allow_failure = False + +# Set up the method for spawning processes. +mp.set_start_method("fork", force=True) +# -------------------------------------------------------------- + +# ------------- Setting how many cores we can use -------------- +NUM_CORES = None +total_cores = os.cpu_count() + +if NUM_CORES is None: + NUM_CORES = total_cores +elif not isinstance(NUM_CORES, int): + raise TypeError( + "If manually overriding 'NUM_CORES', you must set it to an integer value." + ) +elif isinstance(NUM_CORES, int) and NUM_CORES > total_cores: + raise ValueError( + f"If manually overriding 'NUM_CORES', the value must be less than or " + f"equal to the total available cores ({total_cores})." + ) +# -------------------------------------------------------------- + +# -------------- Set paths and create directories -------------- +if os.path.exists("../../../_data"): + ROOT_DATA_DIR = "../../../_data/XRISM/" +else: + ROOT_DATA_DIR = "XRISM/" + +ROOT_DATA_DIR = os.path.abspath(ROOT_DATA_DIR) + +# Make sure the download directory exists. +os.makedirs(ROOT_DATA_DIR, exist_ok=True) + +# Setup path and directory into which we save output files from this example. +OUT_PATH = os.path.abspath("XRISM_output") +os.makedirs(OUT_PATH, exist_ok=True) +# -------------------------------------------------------------- ``` *** ## 1. Data Access -The name of this, and all future sections can change. -In general, it probably is a good idea to start with something like "Data Access". -Please note, and stick to, the existing numbering scheme. - -```{code-cell} python -# Create some example data. -data = np.random.randint(0, 100, size=100) -``` - -## 2. Data Exploration - -Describe what the data look like. -Add summary statistics, initial plots, sanity checks. - -For cuts or other data filtering and cleaning steps, explain the scientific reasoning behind them. -This helps people understand both the notebook and the data so that they're more equipped to use the data appropriately in other contexts. - +++ -:::{tip} -Please include a narrative for *all* your code cells to help the reader figure out what you are doing and why you chose that path. - -Using [MyST admonitions](https://mystmd.org/guide/admonitions) such as this `tip` are encouraged -::: - -```{code-cell} python ---- -tags: [hide-input] -jupyter: - source_hidden: true ---- - -hist, bin_edges = np.histogram(data, bins=10) -hist -``` - -:::{important} -The HEASARC-tutorials style guide requires that cells using matplotlib (or similar) to produce figures -should be isolated (i.e., only contain plotting code), and must include the following metadata to hide the -code from view (see the source of this cell for the unrendered text: - ---- -tags: [hide-input] -jupyter: - source_hidden: true ---- -::: - -For any Figures, please add a few sentences about what the users should be noticing. - -+++ - -## 3. Analysis - -The working part of the notebook. -Lay out the step-by-step analysis workflow. -Each subsection should describe what is being done and why. -These can be sections or subsections. - -+++ - -### 3.1 Design Principles - -- Make no assumptions: define terms, common acronyms, link to things you reference. -- Keep in mind who your audience is. -- Design for portability - will this notebook work on both Fornax and someone's individual laptop. -- Cells capture logical units of work. -- Use markdown before or after cells to describe what is happening in the notebook. - -+++ - -### 3.2 Style Principles - -- Follow suggestions of The Turing Way community [markdown style](https://book.the-turing-way.org/community-handbook/style) -- Write each sentence in a new line (line breaks) to make changes easier to read in PRs. -- Avoid latin abbreviation to avoid failing CI. - -#### 3.2.1 Best Practice Guidelines -It would be nice if all contributed codes did the following, but these guidelines will not be checked in a code review - -- Section titles should not end with ":". -- List items should start at the beginning of the line, no spaces first. Exception is nested lists. -- One empty line between section header and text. -- One empty line before a list and after. -- No more than one empty line between any two non-empty lines. - -```{code-cell} python - -``` - -## 4. PR Review - -Notebooks go through a two step process: first step is getting into the repo, and the second step gets it into the [published tutorials](https://nasa-fornax.github.io/fornax-demo-notebooks/). -Final notebooks are expected to go through both a science and tech review checklist. -Checklists are [here](https://github.com/nasa-fornax/fornax-demo-notebooks/tree/main/template/notebook_review_checklists.md). -Please consider these checklist requirements as you are writing your code. - -The first PR of a notebook does not need to have everything from the checklists completed, but should have all the pieces there, and the authors should be aware of the requirements. - -To complete the second step of this process and be both rendered and included in users Fornax home directories, both a science and technical reviewer will be looking at [this checklist](https://github.com/nasa-fornax/fornax-demo-notebooks/tree/main/template/notebook_review_checklists.md) to see if the new tutorial notebook meets all of the requirements, or has a reasonable excuse not to. - -Any PRs can be opened as drafts, which is in fact preferred, if authors are still working on them. +## About this notebook -+++ +Author: David J Turner, HEASARC Staff Scientist. -## About this notebook +Author: Kenji Hamaguchi, XRISM GOF Scientist. -- **Authors:** Specific author and/or team names, plus "and the Fornax team". -- **Contact:** For help with this notebook, please open a topic in the [Fornax Community Forum](https://discourse.fornax.sciencecloud.nasa.gov/) "Support" category. -- Please edit and keep the above 2 bullet points, and remove this last line. +Updated On: 2025-11-19 +++ @@ -253,16 +171,5 @@ Any PRs can be opened as drafts, which is in fact preferred, if authors are stil ### Acknowledgements -Did anyone help you? -Probably these teams did, so include them: MAST, HEASARC, & IRSA Fornax teams. - -Did you use AI for any part of this tutorial, if so please include a statement such as: -"AI: This notebook was created with assistance from OpenAI’s ChatGPT 5 model.", which is a good time to mention that this template notebook was created with assistance from OpenAI’s ChatGPT 5 model. ### References - -This work made use of: - -- STScI style guide: https://github.com/spacetelescope/style-guides/blob/master/guides/jupyter-notebooks.md -- Fornax tech and science review guidelines: https://github.com/nasa-fornax/fornax-demo-notebooks/blob/main/template/notebook_review_checklists.md -- The Turing Way Style Guide: https://book.the-turing-way.org/community-handbook/style From 6258c3d06dfe61f49e1e7e9b02288e90dc252fb9 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 20 Nov 2025 13:43:01 -0500 Subject: [PATCH 094/221] Added name of possible target to xrism xtend getting started notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 04a3d274..6b5dc592 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -93,7 +93,8 @@ jupyter: source_hidden: true --- # The name of the source we're examining in this demonstration -SRC_NAME = "AX J1910.7+0917" +SRC_NAME = "NGC4151" +# SRC_NAME = "AX J1910.7+0917" # Controls the verbosity of all HEASoftPy tasks TASK_CHATTER = 3 From 9a557306aa4ea1db9b5791091947b6b1e0174bc4 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 10:36:29 -0500 Subject: [PATCH 095/221] Started writing the text of the XRISM-Xtend tutorial, including a brief introduction to the xtdpipeline, and a note about the relatively immature nature of XRISM's software and best practises. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 159 ++++++++++++++++-- 1 file changed, 149 insertions(+), 10 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 6b5dc592..959be482 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-11-19' +date: '2025-11-24' file_format: mystnb jupytext: text_representation: @@ -35,6 +35,9 @@ By the end of this tutorial, you will be able to: ## Introduction +XRISM is... + + ### Inputs @@ -52,21 +55,24 @@ As of 22nd November 2025, this notebook takes ~{N}m to run to completion on Forn ```{code-cell} python # import contextlib +import glob import multiprocessing as mp import os import heasoftpy as hsp +# import matplotlib.pyplot as plt +# import numpy as np +# import xspec as xs +from astropy.coordinates import SkyCoord +from astropy.time import Time +from astropy.units import Quantity +from astroquery.heasarc import Heasarc + # from typing import Tuple, Union # from warnings import warn -# import matplotlib.pyplot as plt -# import numpy as np -# import xspec as xs -# from astropy.coordinates import SkyCoord -# from astropy.units import Quantity -# from astroquery.heasarc import Heasarc # from matplotlib.ticker import FuncFormatter # from tqdm import tqdm ``` @@ -93,7 +99,8 @@ jupyter: source_hidden: true --- # The name of the source we're examining in this demonstration -SRC_NAME = "NGC4151" +SRC_NAME = "LMC N132D" +# SRC_NAME = "NGC4151" # SRC_NAME = "AX J1910.7+0917" # Controls the verbosity of all HEASoftPy tasks @@ -153,7 +160,134 @@ os.makedirs(OUT_PATH, exist_ok=True) *** -## 1. Data Access +## 1. Finding and downloading XRISM observations of **NAMEHERE** + + +### Determining the name of the XRISM observation summary table + +```{code-cell} python +catalog_name = Heasarc.list_catalogs(master=True, keywords="xrism")[0]["name"] +catalog_name +``` + +### What are the coordinates of **NAMEHERE**? + +To search for relevant observations, we have to know the coordinates of our +source. The astropy module allows us to look up a source name in CDS' Sesame name + resolver and retrieve its coordinates. + +```{hint} +You could also set up a SkyCoord object directly, if you already know the coordinates. +``` + +```{code-cell} python +src_coord = SkyCoord.from_name(SRC_NAME) +# This will be useful later on in the notebook, for functions that take +# coordinates as an astropy Quantity. +src_coord_quant = Quantity([src_coord.ra, src_coord.dec]) +src_coord +``` + +### Searching for relevant observations + +```{code-cell} python +all_xrism_obs = Heasarc.query_region(src_coord, catalog_name) +all_xrism_obs +``` + +For an active mission (i.e. actively collecting data and adding to the archive)... + +```{code-cell} python +public_times = Time(all_xrism_obs["public_date"], format="mjd") +avail_xrism_obs = all_xrism_obs[public_times <= Time.now()] +rel_obsids = avail_xrism_obs["obsid"].value.data + +avail_xrism_obs +``` + +### Downloading the selected XRISM observations + +```{code-cell} python +data_links = Heasarc.locate_data(avail_xrism_obs) +data_links +``` + +```{code-cell} python +Heasarc.download_data(data_links, "aws", ROOT_DATA_DIR) +``` + +```{note} +We choose to download the data from the HEASARC AWS S3 bucket, but you could +pass 'heasarc' to acquire data from the FTP server. Additionally, if you are working +on SciServer, you may pass 'sciserver' to use the pre-mounted HEASARC dataset. +``` + +### What do the downloaded data directories contain? + +```{code-cell} python +glob.glob(os.path.join(ROOT_DATA_DIR, rel_obsids[0], "") + "*") +``` + +```{code-cell} python +glob.glob(os.path.join(ROOT_DATA_DIR, rel_obsids[0], "xtend", "") + "**/*") +``` + +## 2. Processing XRISM-Xtend data + +There are multiple steps involved in processing XRISM-Xtend data into a +science-ready state. As with many NASA-affiliated high-energy missions, HEASoft +includes a beginning-to-end pipeline(s) to streamline this process for XRISM data - the +XRISM-Xtend and Resolve instruments both have their own pipelines. + +In this tutorial we are focused only on preparing and using data from XRISM's Xtend +instrument and will not discuss how to handle XRISM-Resolve data; we note however that +there is a third XRISM pipeline task in HEASoft called `xapipeline`, which can be used +to run either or both the Xtend and Resolve pipelines. It contains some convenient +functionality that can identify and automatically pass the attitude, housekeeping, etc. files. + +We will show you how to run the Xtend-specific pipeline, `xtdpipeline`, but the +use of `xapipeline` is nearly functionally identical. + +The Python interface to HEASoft, HEASoftPy, is used throughout this tutorial, and we +will implement parallel observation processing wherever possible. + +### HEASoft and HEASoftPy versions + +```{warning} +XRISM is a relatively new mission, and as such the analysis software and recommended +best practises are still immature and evolving. We are checking and updating this tutorial +on a regular basis, but please report any issues or suggestions to the HEASARC Help Desk. +``` + +Both the HEASoft and HEASoftPy package versions can be retrieved from the +HEASoftPy module. + +The HEASoft version: +```{code-cell} python +hsp.fversion() +``` + +The HEASoftPy version: +```{code-cell} python +hsp.__version__ +``` + +### Running the XRISM-Xtend pipeline + +`xtdpipeline` will take us from a brand-new set of raw XRISM-Xtend data files, all the way +through to generating the 'quick-look' data products (images, spectra, and light curves) +included in HEASARC's XRISM archive 'products' directories. + +The pipeline has three stages and provides the option to start and stop the processing +at any of those stages; this can be useful if you wish to re-run a stage with slightly +different configuration without repeating the entire pipeline run. + +A different set of tasks is encapsulated by each stage, and they have the following general goals: +- **Stage 1** - Calibration and preparation of raw Xtend data. +- **Stage 2** - Screening and filtering of the prepared Xtend event lists. +- **Stage 3** - Generation of quick-look data products. + + +++ @@ -163,12 +297,17 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-11-19 +Updated On: 2025-11-24 +++ ### Additional Resources +HEASoftPy GitHub Repository: https://github.com/HEASARC/heasoftpy + +HEASoftPy HEASARC Page: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/heasoftpy.html + +HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html ### Acknowledgements From d7d3e02c1ebb92b267328ccc75152d176e9be9db Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 11:26:21 -0500 Subject: [PATCH 096/221] Added the beginnings of the wrapper for XTEND processing, as well as the call to it, and the setting up of template file paths. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 102 +++++++++++++++++- 1 file changed, 98 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 959be482..0756961e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -54,7 +54,7 @@ As of 22nd November 2025, this notebook takes ~{N}m to run to completion on Forn ## Imports ```{code-cell} python -# import contextlib +import contextlib import glob import multiprocessing as mp import os @@ -87,7 +87,46 @@ tags: [hide-input] jupyter: source_hidden: true --- - +def process_xrism_xtend(cur_obs_id: str, out_dir: str): + """ + A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process + XRISM-Xtend observation data. This wrapper function is primarily to enable the + use of multiprocessing. + + This function is set to run xtdpipeline until the end of stage 2, excluding the + final stage that generates the 'quick-look' data products. + + :param str cur_obs_id: The ObsID of the XRISM observation to be processed. + :param str out_dir: The directory where output files should be written. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + + # Makes sure the specified output directory exists. + os.makedirs(out_dir, exist_ok=True) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(out_dir), hsp.utils.local_pfiles_context(): + + # The processing/preparation stage of any X-ray telescope's data is the most + # likely to go wrong, and we use a Python try-except as an automated way to + # collect ObsIDs that had an issue during processing. + try: + out = hsp.xtdpipeline( + indir=os.path.join(ROOT_DATA_DIR, cur_obs_id), + outdir=out_dir, + instrument="XTEND", + ) + task_success = True + + except hsp.HSPTaskException as err: + task_success = False + out = str(err) + + return cur_obs_id, out, task_success ``` ### Constants @@ -195,7 +234,7 @@ all_xrism_obs = Heasarc.query_region(src_coord, catalog_name) all_xrism_obs ``` -For an active mission (i.e. actively collecting data and adding to the archive)... +For an active mission (i.e., actively collecting data and adding to the archive)... ```{code-cell} python public_times = Time(all_xrism_obs["public_date"], format="mjd") @@ -263,15 +302,37 @@ Both the HEASoft and HEASoftPy package versions can be retrieved from the HEASoftPy module. The HEASoft version: + ```{code-cell} python hsp.fversion() ``` The HEASoftPy version: + ```{code-cell} python hsp.__version__ ``` +### Setting up file paths to pass to the XRISM-Xtend pipeline + +```{code-cell} python +att_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.att.gz") + +orbit_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.orb.gz") + +obs_gti_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}_gen.gti.gz") + +xtd_hk_path_temp = os.path.join( + ROOT_DATA_DIR, "{oi}", "xtend", "hk", "xa{oi}xtd_a0.hk.gz" +) + +mkf_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.mkf.gz") + +ehk_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.ehk.gz") + +file_stem_temp = "xa{oi}" +``` + ### Running the XRISM-Xtend pipeline `xtdpipeline` will take us from a brand-new set of raw XRISM-Xtend data files, all the way @@ -288,8 +349,41 @@ A different set of tasks is encapsulated by each stage, and they have the follow - **Stage 3** - Generation of quick-look data products. +***MUCH MORE SPECIFIC INFORMATION SHOULD GO HERE*** -+++ + +```{note} +We will stop the execution of `xtdpipeline` at **Stage 2**, as the latter part of this +demonstration will show you how to make more customised data products than are output +by default. +``` + +Though we are using the HEASoftPy `xtdpipeline` function, called +as `hsp.xtdpipeline(indir=...)`, it is called within a wrapper function we have +written in the 'Global Setup: Functions' section of this notebook. The `process_xrism_xtend` +wrapper function exists primarily to let us run the processing of different XRISM-Xtend +observations in parallel. + +We can use Python's multiprocessing module to call the wrapper function for each +of our XRISM observations, passing the relevant arguments. + +The multiprocessing pool will then execute the processing of observations +simultaneously, if there are more cores available than there are observations. + +If there are fewer cores than observations, the pool will handle the allocation of +resources to each observation's processing run, and they will be processed in parallel +until all are complete. + +```{code-cell} python +with mp.Pool(NUM_CORES) as p: + arg_combs = [[oi, os.path.join(OUT_PATH, oi), src_coord] for oi in rel_obsids] + pipe_result = p.starmap(process_xrism_xtend, arg_combs) + +xtd_pipe_problem_ois = [all_out[0] for all_out in pipe_result if not all_out[2]] +rel_obsids = [oi for oi in rel_obsids if oi not in xtd_pipe_problem_ois] + +xtd_pipe_problem_ois +``` ## About this notebook From 5d50782364dc98eea5d908d9e7250c9abe509a13 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 12:48:37 -0500 Subject: [PATCH 097/221] Added explanation of the supporting files required by xtdpipeline to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 42 +++++++++++++++++-- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0756961e..b8cd16a5 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -315,6 +315,26 @@ hsp.__version__ ### Setting up file paths to pass to the XRISM-Xtend pipeline +In order to properly prepare and calibrate XRISM-Xtend data, `xtdpipeline` must +make use of a number of housekeeping files that describe the observatory's status. + +Here we set up template file path variables to the required files so that we can +more easily pass observation-specific file paths to the XRISM-Xtend processing +function in the next section. + +The only expected difference in file name between the equivalent files of different +observations is the included ObsID string, represented by the `{oi}` placeholder. This +placeholder will be replaced by the relevant ObsID for each observation being processed. + +In summary, the supporting files required by `xtdpipeline` are: +- **Attitude file** - Describes the pointing of XRISM in many short time steps throughout the observation. +- **Orbit file** - Orbital telemetry of the XRISM spacecraft during the observation. +- **Observation good-time-intervals (GTI) file** - Contains base GTIs for the observation; used to exclude times when the spacecraft was slewing, or its attitude was inconsistent with that required to observe the target. +- **Filter file (MKF)** - The base filters used to exclude times when the instruments or spacecraft were not operating normally. +- **Extended housekeeping (EHK) file** - Contains extra information about the observation derived from attitude and orbit files, used to screen events. Much of the data relates to attitude, the South Atlantic Anomaly (SAA), and cut-off rigidity (COR). +- **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarises the electrical and thermal state of Xtend in small time steps throughout the observation. + + ```{code-cell} python att_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.att.gz") @@ -322,14 +342,23 @@ orbit_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.orb.gz") obs_gti_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}_gen.gti.gz") +# The overall XRISM observation filter file +mkf_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.mkf.gz") + +# The XRISM extended housekeeping file +ehk_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.ehk.gz") + +# The Xtend housekeeping file xtd_hk_path_temp = os.path.join( ROOT_DATA_DIR, "{oi}", "xtend", "hk", "xa{oi}xtd_a0.hk.gz" ) +``` -mkf_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.mkf.gz") - -ehk_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.ehk.gz") +`xtdpipeline` also needs the 'stem' of the input file names to be defined, so that it +can identify the relevant event list files. The way we call the pipeline, the input +stem will also be used to format output file names. +```{code-cell} python file_stem_temp = "xa{oi}" ``` @@ -385,6 +414,13 @@ rel_obsids = [oi for oi in rel_obsids if oi not in xtd_pipe_problem_ois] xtd_pipe_problem_ois ``` +```{warning} +Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. +``` + +## 3. + + ## About this notebook Author: David J Turner, HEASARC Staff Scientist. From a3ee587d0937710ca14e734cdfe2ba8cbe82f11e Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 13:01:41 -0500 Subject: [PATCH 098/221] Filled out the process_xrism_xtend docstring and its call to hsp.xtendpipeline in the getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 39 +++++++++++++++++-- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b8cd16a5..4df8f5d0 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -87,7 +87,18 @@ tags: [hide-input] jupyter: source_hidden: true --- -def process_xrism_xtend(cur_obs_id: str, out_dir: str): +def process_xrism_xtend( + cur_obs_id: str, + evt_dir: str, + out_dir: str, + attitude: str, + orbit: str, + obs_gti: str, + mkf_filter: str, + file_stem: str, + extended_housekeeping: str, + xtend_housekeeping: str, +): """ A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process XRISM-Xtend observation data. This wrapper function is primarily to enable the @@ -97,7 +108,19 @@ def process_xrism_xtend(cur_obs_id: str, out_dir: str): final stage that generates the 'quick-look' data products. :param str cur_obs_id: The ObsID of the XRISM observation to be processed. + :param str evt_dir: The directory containing the raw, unfiltered, event list + files for the observation. :param str out_dir: The directory where output files should be written. + :param str attitude: XRISM attitude file for the observation. + :param str orbit: XRISM orbit file for the observation. + :param str obs_gti: XRISM base good-time-invterval file for the observation. + :param str mkf_filter: XRISM overall filter file for the observation. + :param str file_stem: The stem of the input event list files (also used for + output file names). + :param str extended_housekeeping: Extended housekeeping file for the + XRISM observation. + :param str xtend_housekeeping: Instrument-specific Xtend housekeeping file + for the observation. :return: A tuple containing the processed ObsID, the log output of the pipeline, and a boolean flag indicating success (True) or failure (False). :rtype: Tuple[str, hsp.core.HSPResult, bool] @@ -116,9 +139,18 @@ def process_xrism_xtend(cur_obs_id: str, out_dir: str): # collect ObsIDs that had an issue during processing. try: out = hsp.xtdpipeline( - indir=os.path.join(ROOT_DATA_DIR, cur_obs_id), + entry_stage=1, + exit_stage=2, + steminputs=file_stem, + stemoutputs=file_stem, + indir=evt_dir, outdir=out_dir, - instrument="XTEND", + attitude=attitude, + orbit=orbit, + obsgti=obs_gti, + makefilter=mkf_filter, + extended_housekeeping=extended_housekeeping, + housekeeping=xtend_housekeeping, ) task_success = True @@ -334,7 +366,6 @@ In summary, the supporting files required by `xtdpipeline` are: - **Extended housekeeping (EHK) file** - Contains extra information about the observation derived from attitude and orbit files, used to screen events. Much of the data relates to attitude, the South Atlantic Anomaly (SAA), and cut-off rigidity (COR). - **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarises the electrical and thermal state of Xtend in small time steps throughout the observation. - ```{code-cell} python att_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.att.gz") From 0a8deaa63af4ad1698a8845a4121d6b45a3c085d Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 13:27:42 -0500 Subject: [PATCH 099/221] Built the list of arguments that process_xrism_xtend needs in the getting-started-xrism-xtend.md. Also put in subsection heading for the 'generating new products' part. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 44 +++++++++++++++++-- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 4df8f5d0..198693e6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -89,8 +89,8 @@ jupyter: --- def process_xrism_xtend( cur_obs_id: str, - evt_dir: str, out_dir: str, + evt_dir: str, attitude: str, orbit: str, obs_gti: str, @@ -108,9 +108,9 @@ def process_xrism_xtend( final stage that generates the 'quick-look' data products. :param str cur_obs_id: The ObsID of the XRISM observation to be processed. + :param str out_dir: The directory where output files should be written. :param str evt_dir: The directory containing the raw, unfiltered, event list files for the observation. - :param str out_dir: The directory where output files should be written. :param str attitude: XRISM attitude file for the observation. :param str orbit: XRISM orbit file for the observation. :param str obs_gti: XRISM base good-time-invterval file for the observation. @@ -367,10 +367,13 @@ In summary, the supporting files required by `xtdpipeline` are: - **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarises the electrical and thermal state of Xtend in small time steps throughout the observation. ```{code-cell} python +# File containing XRISM pointing information att_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.att.gz") +# File containing XRISM orbital telemetry orbit_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}.orb.gz") +# The base XRISM observation GTI file obs_gti_path_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "auxil", "xa{oi}_gen.gti.gz") # The overall XRISM observation filter file @@ -393,6 +396,14 @@ stem will also be used to format output file names. file_stem_temp = "xa{oi}" ``` +Finally, we set up a template variable for the directory containing the raw +Xtend event information for each observation. It contains several files, and +`xtdpipeline` will identify the ones it needs to use: + +```{code-cell} python +raw_evt_dir_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "xtend", "event_uf") +``` + ### Running the XRISM-Xtend pipeline `xtdpipeline` will take us from a brand-new set of raw XRISM-Xtend data files, all the way @@ -436,7 +447,22 @@ until all are complete. ```{code-cell} python with mp.Pool(NUM_CORES) as p: - arg_combs = [[oi, os.path.join(OUT_PATH, oi), src_coord] for oi in rel_obsids] + arg_combs = [ + [ + oi, + os.path.join(OUT_PATH, oi), + raw_evt_dir_temp.format(oi=oi), + att_path_temp.format(oi=oi), + orbit_path_temp.format(oi=oi), + obs_gti_path_temp.format(oi=oi), + mkf_path_temp.format(oi=oi), + file_stem_temp.format(oi=oi), + ehk_path_temp.format(oi=oi), + xtd_hk_path_temp.format(oi=oi), + ] + for oi in rel_obsids + ] + pipe_result = p.starmap(process_xrism_xtend, arg_combs) xtd_pipe_problem_ois = [all_out[0] for all_out in pipe_result if not all_out[2]] @@ -449,8 +475,18 @@ xtd_pipe_problem_ois Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` -## 3. +## 3. Generating new XRISM-Xtend images, exposure maps, spectra, and light curves + +### New XRISM-Xtend images + +### New XRISM-Xtend exposure maps + +### New XRISM-Xtend spectra and supporting files + +### New XRISM-Xtend light curves + +## 4. Fitting ## About this notebook From 6de752baeef51bbae154c4ce947fe127a1fd0288 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 24 Nov 2025 14:31:12 -0500 Subject: [PATCH 100/221] Made some changes to the process_xrism_xtend function, as xtdpipeline gets angry if the outdir already exists, and I like to make sure my demos move INTO the out directory to work. As such there is a lower layer temporary outdir created which is where xtdpipeline can write files. All in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 198693e6..839737f9 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -127,7 +127,8 @@ def process_xrism_xtend( """ # Makes sure the specified output directory exists. - os.makedirs(out_dir, exist_ok=True) + temp_outdir = os.path.join(out_dir, "temp") + os.makedirs(temp_outdir, exist_ok=True) # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so @@ -144,13 +145,14 @@ def process_xrism_xtend( steminputs=file_stem, stemoutputs=file_stem, indir=evt_dir, - outdir=out_dir, + outdir=temp_outdir, attitude=attitude, orbit=orbit, obsgti=obs_gti, makefilter=mkf_filter, extended_housekeeping=extended_housekeeping, housekeeping=xtend_housekeeping, + clobber=True, ) task_success = True @@ -158,6 +160,12 @@ def process_xrism_xtend( task_success = False out = str(err) + # Moves files from the temporary output directory into the + # final output directory + if os.path.exists(temp_outdir) and len(os.listdir(temp_outdir)) != 0: + for f in os.listdir(temp_outdir): + os.rename(os.path.join(temp_outdir, f), os.path.join(out_dir, f)) + return cur_obs_id, out, task_success ``` From b33a9980403ab7a48a9b4d5be14bcf6be3d62234 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 10:53:39 -0500 Subject: [PATCH 101/221] Added discussion of the mapping between XRISM-Xtend channel and energy, as well as a demonstration of how to validate our assumed relation through looking at a pre-generated RMF. All in getting-started-xrism-xtend.md, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 131 +++++++++++++++++- 1 file changed, 125 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 839737f9..b363268a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -60,11 +60,13 @@ import multiprocessing as mp import os import heasoftpy as hsp +import matplotlib.pyplot as plt +import numpy as np -# import matplotlib.pyplot as plt -# import numpy as np # import xspec as xs from astropy.coordinates import SkyCoord +from astropy.io import fits +from astropy.table import Table from astropy.time import Time from astropy.units import Quantity from astroquery.heasarc import Heasarc @@ -184,6 +186,9 @@ SRC_NAME = "LMC N132D" # Controls the verbosity of all HEASoftPy tasks TASK_CHATTER = 3 + +# The approximate linear relationship between Xtend PI and event energy +XTD_EV_PER_CHAN = (1 / Quantity(166.7, "chan/keV")).to("eV/chan") ``` ### Configuration @@ -483,18 +488,132 @@ xtd_pipe_problem_ois Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` -## 3. Generating new XRISM-Xtend images, exposure maps, spectra, and light curves +### Identifying problem pixels + +```{code-cell} python + +``` + +## 3. Generating new XRISM-Xtend images, exposure maps, and light curves + +### Converting energy bounds to channel bounds + +```{code-cell} python +XTD_EV_PER_CHAN +``` + +Alternatively, we can figure out this relationship between PI and energy by looking at +a XRISM-Xtend Redistribution Matrix File (RMF), which exists to describe this +mapping. + +We will be creating new RMFs as part of the generation of XRISM-Xtend spectra in the +next section. For our current purpose, however, it is acceptable to use the RMFs that +were included in the XRISM-Xtend archive we downloaded earlier. + +The archived RMFs are generated for the entire Xtend FoV, rather than for the CCDs +our particular target fall on, but practically speaking, that doesn't make a significant +difference. + +Using observation 000128000 as an example, we determine the path to the relevant +pre-generated RMF. We only expect a single file, and include a validity check to +ensure that this does not change in future versions of the archive: + +```{code-cell} python +chosen_demo_obsid = "000128000" + +pregen_rmf_wildcard = os.path.join( + ROOT_DATA_DIR, "{oi}", "xtend", "products", "xa{oi}xtd_p*.rmf*" +) +poss_rmfs = glob.glob(pregen_rmf_wildcard.format(oi=chosen_demo_obsid)) +print(poss_rmfs) + +# Check how many RMF files we found - there should only be one +if len(poss_rmfs) != 1: + raise ValueError(f"Expected exactly one RMF file, but found {len(poss_rmfs)}.") +else: + pregen_rmf_path = poss_rmfs[0] +``` + +XRISM-Xtend RMFs are written in the FITS file format, and so can be read into +Python using the `astropy.io.fits` module: + +```{code-cell} python +# Loading the fits file using astropy +with fits.open(pregen_rmf_path) as rmfo: + # Iterate through the tables in the RMF, printing their names + for tab in rmfo: + print(tab.name) + + # Associate the EBOUNDS table with a variable, so it can be used outside + # the fits.open context + e_bounds = rmfo["EBOUNDS"].data + +# Convert the read-out energy bound information to an astropy Table, mainly +# because it will look nicer whe we show it below +e_bounds = Table(e_bounds) +# Display a subset of the table +e_bounds[90:110] +``` + +We can use this file to visualize the basic linear mapping between energy and +channel - it will not be the most interesting figure you've ever seen: + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +plt.figure(figsize=(5.5, 5.5)) + +plt.minorticks_on() +plt.tick_params(which="both", direction="in", top=True, right=True) + +mid_ens = (e_bounds["E_MIN"] + e_bounds["E_MAX"]) / 2 + +plt.plot(e_bounds["CHANNEL"], mid_ens, color="navy", alpha=0.9, label="XRISM-Xtend") + +plt.xlim(0) +plt.ylim(0) + +plt.xlabel("Channel [PI]", fontsize=15) +plt.ylabel("Central Energy [keV]", fontsize=15) + +plt.legend(fontsize=14) + +plt.tight_layout() +plt.show() +``` + +Finally, we can validate our assumed relationship between energy and channel by +calculating the mean change in minimum energy between adjacent channels: + +```{code-cell} python +# +rmf_ev_per_chan = Quantity(np.diff(e_bounds["E_MIN"].data).mean(), "keV/chan").to( + "eV/chan" +) +rmf_ev_per_chan +``` + +Clearly, our assumed relationship is valid: + +```{code-cell} python +rmf_ev_per_chan / XTD_EV_PER_CHAN +``` ### New XRISM-Xtend images -### New XRISM-Xtend exposure maps -### New XRISM-Xtend spectra and supporting files + +### New XRISM-Xtend exposure maps ### New XRISM-Xtend light curves -## 4. Fitting +## 4. Generating new XRISM-Xtend spectra and supporting files + + ## About this notebook From f27399f34f12b6df2899e4ec0c366bb2f976aa78 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 11:21:58 -0500 Subject: [PATCH 102/221] Added more explanation of why a mapping between channel and energy is required for generating energy bound data products for XRISM-Xtend to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 33 ++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b363268a..b70d3b28 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -74,7 +74,6 @@ from astroquery.heasarc import Heasarc # from typing import Tuple, Union # from warnings import warn - # from matplotlib.ticker import FuncFormatter # from tqdm import tqdm ``` @@ -498,6 +497,34 @@ Processing XRISM-Xtend data can take a long time, up to several hours for a sing ### Converting energy bounds to channel bounds +The data products we generate in this section can all benefit from selecting events +from within a specific energy range. This might be because your source of interest only +emits in a narrow energy range, and you don't care about the rest, or because different +mechanisms emit at different energies, and you wish to separate them. + +Such filtering needs to be performed at the event list level so that the resulting +subset of events can be binned in spatial and temporal dimensions to produce +images and light curves. + +The event lists of most high-energy missions (including XRISM) do not directly store +event energies - instead they contain the pulse-height-amplitude (PHA), and/or the +pulse-invariant (PI) channel (calculated from PHA and instrument gain tables) information. + +This is because the calibration of detector-channel to energy, the understanding of the +behaviors of the instrument and its electronics, and the performance of the detectors +can all change dramatically over time. + +All that said, the tools we will use to generate our energy-bounded images and light +curves do not take _energy_ bounds as an input, but rather _channel_ bounds. + +Thus, we have the responsibility of determining equivalent channel bounds for our +hopefully-physics-driven energy-bound choices. For images and light curves, we can +safely assume a perfect linear relationship between energy and channel. + +The XRISM ABC guide provides the following mapping +([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/Xtend_Data_Analysis.html#SECTION001043000000000000000)) +for Xtend: + ```{code-cell} python XTD_EV_PER_CHAN ``` @@ -627,6 +654,8 @@ Updated On: 2025-11-24 ### Additional Resources +XRISM Data Reduction (ABC) Guide - https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide + HEASoftPy GitHub Repository: https://github.com/HEASARC/heasoftpy HEASoftPy HEASARC Page: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/heasoftpy.html @@ -637,3 +666,5 @@ HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/softwa ### References + +[XRISM GOF & SDC (2024) - _XRISM ABC GUIDE XTEND ENERGY-CHANNEL MAPPING_ [ACCESSED 25-NOV-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/Xtend_Data_Analysis.html#SECTION001043000000000000000) From 99c654273ec8e7cbcb9a3b4034a442b63bd2ce30 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 11:55:07 -0500 Subject: [PATCH 103/221] Added the beginnings of a wrapper for extractor to generate XRISM-Xtend images (gen_xrism_xtend_image) to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b70d3b28..70dcd517 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -168,6 +168,74 @@ def process_xrism_xtend( os.rename(os.path.join(temp_outdir, f), os.path.join(out_dir, f)) return cur_obs_id, out, task_success + + +def gen_xrism_xtend_image( + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + lo_en: Quantity, + hi_en: Quantity, +): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend image. + :param str cur_xtend_data_class: + :param str event_file: + :param str out_dir: The directory where output files should be written. + :param Quantity lo_en: Lower bound of the energy band within which we will + generate the image. + :param Quantity hi_en: Upper bound of the energy band within which we will + generate the image. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + if cur_xtend_data_class[0] != "3": + raise ValueError( + f"The first digit of the Xtend data class ({cur_xtend_data_class}) " + "must be 3 for in-flight data." + ) + + # Make sure the lower and upper energy limits make sense + if lo_en > hi_en: + raise ValueError( + "The lower energy limit must be less than or equal to the upper " + "energy limit." + ) + else: + lo_en_val = lo_en.to("keV").value + hi_en_val = hi_en.to("keV").value + + # Convert the energy limits to channel limits, rounding down and up to the nearest + # integer channel for the lower and upper bounds respectively. + lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + + # Create modified input event list file path, where we use the just-calculated + # PI channel limits to subset the events + evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" + + # Set up the output file name for the image we're about to generate. + im_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"en{lo_en_val}_{hi_en_val}keV-image.fits" + ) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(out_dir), hsp.utils.local_pfiles_context(): + + # The processing/preparation stage of any X-ray telescope's data is the most + # likely to go wrong, and we use a Python try-except as an automated way to + # collect ObsIDs that had an issue during processing. + out = hsp.extractor( + filename=evt_file_chan_sel, imgfile=im_out, noprompt=True, clobber=True + ) + + return out ``` ### Constants @@ -631,7 +699,22 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ### New XRISM-Xtend images +```{code-cell} python +# Defining the energy bounds we want images within +xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.3, 10.0]], "keV") + +# Convert energy bounds to channel bounds +xtd_im_ch_bounds = (xtd_im_en_bounds / XTD_EV_PER_CHAN).to("chan") +xtd_im_ch_bounds[:, 0] = np.floor(xtd_im_ch_bounds[:, 0]) +xtd_im_ch_bounds[:, 1] = np.ceil(xtd_im_ch_bounds[:, 1]) +xtd_im_ch_bounds = xtd_im_ch_bounds.astype(int) +xtd_im_ch_bounds +``` +```{code-cell} python +# hsp.extractor(filename=evt_file_chan_sel, imgfile='testo_img.fits', noprompt=True, +# clobber=True) +``` ### New XRISM-Xtend exposure maps From f364089c584935c7cf88a1921667efa7cfdeaa87 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 13:16:04 -0500 Subject: [PATCH 104/221] Added bits and bobs to the getting-started-xrism-xtend.md notebook to make the image generation work. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 88 +++++++++++++++++-- 1 file changed, 80 insertions(+), 8 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 70dcd517..cc7a9120 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -58,6 +58,8 @@ import contextlib import glob import multiprocessing as mp import os +from random import randint +from shutil import rmtree import heasoftpy as hsp import matplotlib.pyplot as plt @@ -223,18 +225,29 @@ def gen_xrism_xtend_image( f"en{lo_en_val}_{hi_en_val}keV-image.fits" ) + # Create a temporary working directory + temp_work_dir = os.path.join( + out_dir, "im_extractor_{}".format(randint(0, int(1e8))) + ) + os.makedirs(temp_work_dir) + # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). - with contextlib.chdir(out_dir), hsp.utils.local_pfiles_context(): - + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): # The processing/preparation stage of any X-ray telescope's data is the most # likely to go wrong, and we use a Python try-except as an automated way to # collect ObsIDs that had an issue during processing. out = hsp.extractor( - filename=evt_file_chan_sel, imgfile=im_out, noprompt=True, clobber=True + filename=evt_file_chan_sel, + imgfile=os.path.join(out_dir, im_out), + noprompt=True, + clobber=True, ) + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + return out ``` @@ -342,7 +355,10 @@ src_coord ### Searching for relevant observations ```{code-cell} python -all_xrism_obs = Heasarc.query_region(src_coord, catalog_name) +col_str = ( + "obsid,name,ra,dec,time,exposure,status,public_date,xtd_dataclas1,xtd_dataclas2" +) +all_xrism_obs = Heasarc.query_region(src_coord, catalog_name, columns=col_str) all_xrism_obs ``` @@ -351,7 +367,20 @@ For an active mission (i.e., actively collecting data and adding to the archive) ```{code-cell} python public_times = Time(all_xrism_obs["public_date"], format="mjd") avail_xrism_obs = all_xrism_obs[public_times <= Time.now()] + +# Define a couple of useful variables that make accessing information in the +# table a little easier later on in the notebook +# Create an array of the relevant ObsIDs rel_obsids = avail_xrism_obs["obsid"].value.data +# Create a dictionary connecting ObsIDs to their associated Xtend data classes +rel_dataclasses = { + oi: [ + dc + for dc in avail_xrism_obs[oi_ind][["xtd_dataclas1", "xtd_dataclas2"]].values() + if dc != "" + ] + for oi_ind, oi in enumerate(rel_obsids) +} avail_xrism_obs ``` @@ -444,7 +473,7 @@ In summary, the supporting files required by `xtdpipeline` are: - **Observation good-time-intervals (GTI) file** - Contains base GTIs for the observation; used to exclude times when the spacecraft was slewing, or its attitude was inconsistent with that required to observe the target. - **Filter file (MKF)** - The base filters used to exclude times when the instruments or spacecraft were not operating normally. - **Extended housekeeping (EHK) file** - Contains extra information about the observation derived from attitude and orbit files, used to screen events. Much of the data relates to attitude, the South Atlantic Anomaly (SAA), and cut-off rigidity (COR). -- **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarises the electrical and thermal state of Xtend in small time steps throughout the observation. +- **Xtend housekeeping (HK) file** - An instrument-specific housekeeping file that summarizes the electrical and thermal state of Xtend in small time steps throughout the observation. ```{code-cell} python # File containing XRISM pointing information @@ -502,6 +531,9 @@ A different set of tasks is encapsulated by each stage, and they have the follow ***MUCH MORE SPECIFIC INFORMATION SHOULD GO HERE*** +***INCLUDE XTEND DATA CLASS SUMMARY, AND HOW THERE MAY BE MULTIPLE EVENT LISTS PER OBSID*** + + ```{note} We will stop the execution of `xtdpipeline` at **Stage 2**, as the latter part of this @@ -555,6 +587,19 @@ xtd_pipe_problem_ois Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` +Finally, we set up a template variable for the cleaned event lists we just created: + +***CONSIDER MORE COMMENTARY ON DATACLASS ETC. HERE*** + + +***Need to consider the 'p0' bit - apparently the '0' is a counter for splitting large datasets, but I don't know when it is used*** + +***sc == 'split counter'*** + +```{code-cell} python +evt_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") +``` + ### Identifying problem pixels ```{code-cell} python @@ -701,8 +746,16 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ```{code-cell} python # Defining the energy bounds we want images within -xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.3, 10.0]], "keV") +xtd_im_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") +``` + +Converting those energy bounds to channel bounds is straightforward, we simply divide +the energy values by our assumed mapping between energy and channel. + +The resulting lower and upper bound channel values are rounded down and up to the +nearest integer channel respectively. +```{code-cell} python # Convert energy bounds to channel bounds xtd_im_ch_bounds = (xtd_im_en_bounds / XTD_EV_PER_CHAN).to("chan") xtd_im_ch_bounds[:, 0] = np.floor(xtd_im_ch_bounds[:, 0]) @@ -711,9 +764,28 @@ xtd_im_ch_bounds = xtd_im_ch_bounds.astype(int) xtd_im_ch_bounds ``` +```{note} +Though we demonstrate how to convert energy to channel bounds above, the wrapper +function for image generation will repeat this exercise, as it will write +energy bounds into output file names. +``` + ```{code-cell} python -# hsp.extractor(filename=evt_file_chan_sel, imgfile='testo_img.fits', noprompt=True, -# clobber=True) +arg_combs = [ + [ + oi, + dc, + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + os.path.join(OUT_PATH, oi), + *cur_bnds, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs + for cur_bnds in xtd_im_en_bounds +] + +with mp.Pool(NUM_CORES) as p: + im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` ### New XRISM-Xtend exposure maps From 50903ffe7437295f4dc9894a2423eae469734ca4 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 14:20:50 -0500 Subject: [PATCH 105/221] Added an image binning factor subsection to the getting-started-xrism-xtend.md notebook, and included some commentary. May add a bit about how to use an event list to find the size of the event pixels. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 36 +++++++++++++++++-- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index cc7a9120..0145cc85 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-11-24' +date: '2025-11-25' file_format: mystnb jupytext: text_representation: @@ -179,6 +179,7 @@ def gen_xrism_xtend_image( out_dir: str, lo_en: Quantity, hi_en: Quantity, + im_bin: int = 1, ): """ @@ -222,7 +223,7 @@ def gen_xrism_xtend_image( # Set up the output file name for the image we're about to generate. im_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"en{lo_en_val}_{hi_en_val}keV-image.fits" + f"imbinfactor{im_bin}-en{lo_en_val}_{hi_en_val}keV-image.fits" ) # Create a temporary working directory @@ -243,6 +244,7 @@ def gen_xrism_xtend_image( imgfile=os.path.join(out_dir, im_out), noprompt=True, clobber=True, + binf=im_bin, ) # Make sure to remove the temporary directory @@ -744,6 +746,8 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ### New XRISM-Xtend images +#### Image energy bounds + ```{code-cell} python # Defining the energy bounds we want images within xtd_im_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") @@ -770,6 +774,27 @@ function for image generation will repeat this exercise, as it will write energy bounds into output file names. ``` +#### Image binning factor + +When generating images, you might wish to bin the event X-Y sky coordinate system so +that one pixel of the output image represents a grouping of 'event pixels'. + +This binning could be motivated by increasing the signal-to-noise of each pixel or +reducing the size of the output image file, or your own scientific purpose. + +It is worth noting that the Xtend **event pixel** size dramatically subsamples the +point-spread-function (PSF) size induced by the X-ray optics, so an extreme binning +factor would be required to minimize cross-talk between image pixels. As such, this +should not be the primary motivation for your choice of image binning factor. + +```{code-cell} python +bin_factor = 4 +``` + +#### Running image generation + +We use... + ```{code-cell} python arg_combs = [ [ @@ -778,6 +803,7 @@ arg_combs = [ evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), *cur_bnds, + bin_factor, ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -790,6 +816,10 @@ with mp.Pool(NUM_CORES) as p: ### New XRISM-Xtend exposure maps +```{code-cell} python + +``` + ### New XRISM-Xtend light curves @@ -803,7 +833,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-11-24 +Updated On: 2025-11-25 +++ From 3c109f2d02070377fe72d1a6b9180046af29f5cc Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 15:42:35 -0500 Subject: [PATCH 106/221] Added first go at a wrapper function for XRISM Xtend exposure map generation to the getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 118 +++++++++++++++++- 1 file changed, 115 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0145cc85..0171b5da 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -236,9 +236,6 @@ def gen_xrism_xtend_image( # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): - # The processing/preparation stage of any X-ray telescope's data is the most - # likely to go wrong, and we use a Python try-except as an automated way to - # collect ObsIDs that had an issue during processing. out = hsp.extractor( filename=evt_file_chan_sel, imgfile=os.path.join(out_dir, im_out), @@ -251,6 +248,111 @@ def gen_xrism_xtend_image( rmtree(temp_work_dir) return out + + +def gen_xrism_xtend_expmap( + cur_obs_id: str, + cur_xtend_data_class: str, + out_dir: str, + gti_file: str, + extend_hk_file: str, + bad_pix_file: str, + pix_gti_file: str = "NONE", + im_bin: int = 1, + out_map_type: str = "EXPOSURE", + radial_delta: float = Quantity(20.0, "arcmin"), + num_phi_bin: int = 1, +): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend exposure map. + :param str cur_xtend_data_class: + :param str event_file: + :param str out_dir: The directory where output files should be written. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + + # Validity check on the passed data class + if cur_xtend_data_class[0] != "3": + raise ValueError( + f"The first digit of the Xtend data class ({cur_xtend_data_class}) " + "must be 3 for in-flight data." + ) + + # Make sure the radial_delta value is in arcminutes/is convertible to arcmins + # Also will assume that radial_delta is in arcmin if it is not a Quantity object + if not isinstance(radial_delta, Quantity): + radial_delta = Quantity(radial_delta, "arcmin") + elif radial_delta.unit.is_equivalent("arcmin"): + radial_delta = radial_delta.to("arcmin") + else: + raise ValueError( + f"The 'radial_delta' argument must be in arcmin or convertible to " + f"arcmin, not {radial_delta.unit}." + ) + + # Now we're certain of 'radial_delta's unit, we read out the value + radial_delta = radial_delta.value.astype(float) + + # Two variants of exposure map can be generated by the function we're about to + # call; the default is a map of the integrated exposure time for each pixel, and + # the second (not recommended by the documentation) is a flat-fielding map + ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" + + # Set up the output file name for the exposure map we're about to generate. + ex_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"attraddelta{radial_delta}arcmin-attphibin{num_phi_bin}-" + f"imbinfactor1-enALL-{ex_type}.fits" + ) + + # If the user wants to bin up the exposure map, we'll need to set up another + # output file name with the bin factor set to the input value (this variable + # is not used if the user does not want to bin the map) + binned_ex_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"attraddelta{radial_delta}arcmin-attphibin{num_phi_bin}-" + f"imbinfactor{im_bin}-enALL-{ex_type}.fits" + ) + + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + out = hsp.xaexpmap( + instrume="XTEND", + ehkfile=extend_hk_file, + gtifile=gti_file, + pixgtifile=pix_gti_file, + delta=radial_delta, + numphi=num_phi_bin, + outfile=os.path.join(out_dir, ex_out), + badimgfile=bad_pix_file, + noprompt=True, + clobber=True, + ) + + # If the user wants a spatially binned exposure map, we run the fimgbin task + if im_bin != 1: + rebin_out = hsp.fimgbin( + infile=os.path.join(out_dir, ex_out), + outfile=os.path.join(out_dir, binned_ex_out), + xbinsize=im_bin, + noprompt=True, + clobber=True, + ) + out = [out, rebin_out] + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return out ``` ### Constants @@ -816,6 +918,16 @@ with mp.Pool(NUM_CORES) as p: ### New XRISM-Xtend exposure maps +xaexpmap \ + instrume=XTEND \ + outfile="n132d_r2m_src.expo" \ + ehkfile="xa000126000.ehk" \ + gtifile="xa000126000xtd_p030000010_cl.evt" \ + badimgfile="xa000126000xtd_p030000010.bimg" \ + pixgtifile=NONE \ + outmaptype=EXPOSURE delta=20.0 numphi=1 stopsys=SKY \ + clobber=yes mode=hl + ```{code-cell} python ``` From 7cddd2e989dc1fbeffcaadf1101da3125f522227 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 16:13:41 -0500 Subject: [PATCH 107/221] Think I've worked out the kinks in the 'gen_xrism_xtend_expmap' function in the getting-started-xrism-xtend.md notebook. Seems to work fine, including spatial re-binning and creating different types of map. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0171b5da..47fea099 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -334,6 +334,7 @@ def gen_xrism_xtend_expmap( numphi=num_phi_bin, outfile=os.path.join(out_dir, ex_out), badimgfile=bad_pix_file, + outmaptype=out_map_type, noprompt=True, clobber=True, ) @@ -341,8 +342,8 @@ def gen_xrism_xtend_expmap( # If the user wants a spatially binned exposure map, we run the fimgbin task if im_bin != 1: rebin_out = hsp.fimgbin( - infile=os.path.join(out_dir, ex_out), - outfile=os.path.join(out_dir, binned_ex_out), + infile=os.path.join("..", ex_out), + outfile=os.path.join("..", binned_ex_out), xbinsize=im_bin, noprompt=True, clobber=True, @@ -928,6 +929,10 @@ xaexpmap \ outmaptype=EXPOSURE delta=20.0 numphi=1 stopsys=SKY \ clobber=yes mode=hl +```{code-cell} python +badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") +``` + ```{code-cell} python ``` From 565f9fd70c84e57520f499913bb98cea4e7e8562 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 16:53:15 -0500 Subject: [PATCH 108/221] Added the code that calls the exposure map wrapper function to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 76 ++++++++++++++++--- 1 file changed, 64 insertions(+), 12 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 47fea099..4de1228d 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -259,7 +259,6 @@ def gen_xrism_xtend_expmap( bad_pix_file: str, pix_gti_file: str = "NONE", im_bin: int = 1, - out_map_type: str = "EXPOSURE", radial_delta: float = Quantity(20.0, "arcmin"), num_phi_bin: int = 1, ): @@ -299,6 +298,8 @@ def gen_xrism_xtend_expmap( # Two variants of exposure map can be generated by the function we're about to # call; the default is a map of the integrated exposure time for each pixel, and # the second (not recommended by the documentation) is a flat-fielding map + # TODO REINSTATE WHEN WE HAVE A BETTER UNDERSTANDING OF POTENTIAL USER USES + out_map_type = "EXPOSURE" ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" # Set up the output file name for the exposure map we're about to generate. @@ -692,7 +693,10 @@ xtd_pipe_problem_ois Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` -Finally, we set up a template variable for the cleaned event lists we just created: +Finally, we set up some template variables for the various useful files output by the +XRISM-Xtend processing pipeline. These include the cleaned event lists we just created, + +cleaned event lists we just created: ***CONSIDER MORE COMMENTARY ON DATACLASS ETC. HERE*** @@ -702,7 +706,11 @@ Finally, we set up a template variable for the cleaned event lists we just creat ***sc == 'split counter'*** ```{code-cell} python +# Cleaned event list path template - obviously going to be useful later evt_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") + +# The path to the bad pixel map, useful for excluding dodgy pixels from data products +badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") ``` ### Identifying problem pixels @@ -917,24 +925,66 @@ with mp.Pool(NUM_CORES) as p: im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` +Once again we set up a template variable for output image file names: + +```{code-cell} python +im_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-imbinfactor{ibf}-en{lo}_{hi}keV-image.fits", +) +``` + ### New XRISM-Xtend exposure maps -xaexpmap \ - instrume=XTEND \ - outfile="n132d_r2m_src.expo" \ - ehkfile="xa000126000.ehk" \ - gtifile="xa000126000xtd_p030000010_cl.evt" \ - badimgfile="xa000126000xtd_p030000010.bimg" \ - pixgtifile=NONE \ - outmaptype=EXPOSURE delta=20.0 numphi=1 stopsys=SKY \ - clobber=yes mode=hl +Exposure maps... ```{code-cell} python -badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") +expmap_rad_delta = Quantity(20, "arcmin") +expmap_phi_bins = 1 ``` +gen_xrism_xtend_expmap(chosen_demo_obsid, + "32000010", + os.path.join(OUT_PATH, chosen_demo_obsid), + evt_path_temp.format(oi=chosen_demo_obsid, xdc="32000010", sc=0), + ehk_path_temp.format(oi=chosen_demo_obsid), + badpix_path_temp.format(oi=chosen_demo_obsid, xdc="32000010", sc=0), + im_bin=4, out_map_type='EFFICIENCY') + +***WHAT ABOUT THIS GTI?? - xa000128000xtd_mode.gti*** + ```{code-cell} python +arg_combs = [ + [ + oi, + dc, + os.path.join(OUT_PATH, oi), + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + ehk_path_temp.format(oi=oi), + badpix_path_temp.format(oi=oi, xdc=dc, sc=0), + "NONE", + expmap_rad_delta, + expmap_phi_bins, + bin_factor, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs +] +with mp.Pool(NUM_CORES) as p: + ex_result = p.starmap(gen_xrism_xtend_expmap, arg_combs) +``` + +Set up a template variable for output exposure map file names: + +```{code-cell} python +ex_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-attraddelta{rd}arcmin-" + "attphibin{npb}-imbinfactor{ibf}-enALL-expmap.fits", +) ``` ### New XRISM-Xtend light curves @@ -970,3 +1020,5 @@ HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/softwa ### References [XRISM GOF & SDC (2024) - _XRISM ABC GUIDE XTEND ENERGY-CHANNEL MAPPING_ [ACCESSED 25-NOV-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/Xtend_Data_Analysis.html#SECTION001043000000000000000) + +[](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html) From b9861f123a68d3a54020dcec11eab1b981405f13 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 17:05:31 -0500 Subject: [PATCH 109/221] Added first pass at a wrapper function for XRISM light curve generation to the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 96 ++++++++++++++++++- 1 file changed, 95 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 4de1228d..422641f3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -70,7 +70,7 @@ from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.table import Table from astropy.time import Time -from astropy.units import Quantity +from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc # from typing import Tuple, Union @@ -355,6 +355,100 @@ def gen_xrism_xtend_expmap( rmtree(temp_work_dir) return out + + +def gen_xrism_xtend_lightcurve( + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + lo_en: Quantity, + hi_en: Quantity, + time_bin_size: Quantity = Quantity(100, "s"), +): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend light curve. + :param str cur_xtend_data_class: + :param str event_file: + :param str out_dir: The directory where output files should be written. + :param Quantity lo_en: Lower bound of the energy band within which we will + generate the light curve. + :param Quantity hi_en: Upper bound of the energy band within which we will + generate the light curve. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + + # Validity check on the passed data class + if cur_xtend_data_class[0] != "3": + raise ValueError( + f"The first digit of the Xtend data class ({cur_xtend_data_class}) " + "must be 3 for in-flight data." + ) + + # Check the units of the passed time bin size - also if the passed value is + # a float or integer, we'll assume it is in seconds + if not isinstance(time_bin_size, Quantity): + time_bin_size = Quantity(time_bin_size, "s") + elif not time_bin_size.unit.is_equivalent("s"): + raise UnitConversionError( + f"The 'time_bin_size' argument ({time_bin_size}) " + "must be an astropy Quantity that is convertible " + "to seconds." + ) + + # Convert the time bin size to seconds and convert it to a simple integer/float + time_bin_size = time_bin_size.to("s").value + + # Make sure the lower and upper energy limits make sense + if lo_en > hi_en: + raise ValueError( + "The lower energy limit must be less than or equal to the upper " + "energy limit." + ) + else: + lo_en_val = lo_en.to("keV").value + hi_en_val = hi_en.to("keV").value + + # Convert the energy limits to channel limits, rounding down and up to the nearest + # integer channel for the lower and upper bounds respectively. + lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + + # Create modified input event list file path, where we use the just-calculated + # PI channel limits to subset the events + evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" + + # Set up the output file name for the image we're about to generate. + lc_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"en{lo_en_val}_{hi_en_val}keV-tb{time_bin_size}s-lightcurve.fits" + ) + + # Create a temporary working directory + temp_work_dir = os.path.join( + out_dir, "lightcurve_extractor_{}".format(randint(0, int(1e8))) + ) + os.makedirs(temp_work_dir) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + out = hsp.extractor( + filename=evt_file_chan_sel, + fitsbinlc=os.path.join(out_dir, lc_out), + binlc=time_bin_size, + noprompt=True, + clobber=True, + ) + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return out ``` ### Constants From 0692c5d10653476b4fdc46148f63c3d2faf63a31 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 17:11:06 -0500 Subject: [PATCH 110/221] Added the code cells to call the new light curve generating wrapper function in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 29 ++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 422641f3..dc99c816 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -439,7 +439,7 @@ def gen_xrism_xtend_lightcurve( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.extractor( filename=evt_file_chan_sel, - fitsbinlc=os.path.join(out_dir, lc_out), + fitsbinlc=os.path.join("..", lc_out), binlc=time_bin_size, noprompt=True, clobber=True, @@ -1083,6 +1083,33 @@ ex_path_temp = os.path.join( ### New XRISM-Xtend light curves +```{code-cell} python +lc_time_bin = Quantity(100, "s") +``` + +```{code-cell} python +# Defining the various energy bounds we want to make light curves for +xtd_lc_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") +``` + +```{code-cell} python +arg_combs = [ + [ + oi, + dc, + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + os.path.join(OUT_PATH, oi), + *cur_bnds, + lc_time_bin, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs + for cur_bnds in xtd_lc_en_bounds +] + +with mp.Pool(NUM_CORES) as p: + lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) +``` ## 4. Generating new XRISM-Xtend spectra and supporting files From 4e98412ccf9476705fd90385c44d9845b0faf1b0 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 25 Nov 2025 17:32:09 -0500 Subject: [PATCH 111/221] Allow a 'lc_bin_thresh' argument to be passed to gen_xrism_xtend_lightcurve, to control which light curve bins are thrown out because of fractional exposure. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index dc99c816..21a53760 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -364,7 +364,8 @@ def gen_xrism_xtend_lightcurve( out_dir: str, lo_en: Quantity, hi_en: Quantity, - time_bin_size: Quantity = Quantity(100, "s"), + time_bin_size: Quantity = Quantity(200, "s"), + lc_bin_thresh: float = 0.0, ): """ @@ -424,7 +425,8 @@ def gen_xrism_xtend_lightcurve( # Set up the output file name for the image we're about to generate. lc_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"en{lo_en_val}_{hi_en_val}keV-tb{time_bin_size}s-lightcurve.fits" + f"en{lo_en_val}_{hi_en_val}keV-expthresh{lc_bin_thresh}-tb{time_bin_size}s-" + f"lightcurve.fits" ) # Create a temporary working directory @@ -441,6 +443,7 @@ def gen_xrism_xtend_lightcurve( filename=evt_file_chan_sel, fitsbinlc=os.path.join("..", lc_out), binlc=time_bin_size, + lcthresh=lc_bin_thresh, noprompt=True, clobber=True, ) @@ -1000,6 +1003,8 @@ bin_factor = 4 We use... +***NEED TO APPLY GTIS TO IMAGE GENERATION AS WELL*** + ```{code-cell} python arg_combs = [ [ @@ -1058,9 +1063,9 @@ arg_combs = [ ehk_path_temp.format(oi=oi), badpix_path_temp.format(oi=oi, xdc=dc, sc=0), "NONE", + bin_factor, expmap_rad_delta, expmap_phi_bins, - bin_factor, ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -1084,12 +1089,12 @@ ex_path_temp = os.path.join( ### New XRISM-Xtend light curves ```{code-cell} python -lc_time_bin = Quantity(100, "s") +lc_time_bin = Quantity(200, "s") ``` ```{code-cell} python # Defining the various energy bounds we want to make light curves for -xtd_lc_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") +xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") ``` ```{code-cell} python From f2643c1601046f6b55d7d956e7f472aa1e5e8f3c Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 09:35:17 -0500 Subject: [PATCH 112/221] Shifted the skeleton of the light curve generation subsection to the second section of product generation (with spectra, as it will be source specific). Added a template variable for light curve outputs. All in getting-started-xrism-xtend.md, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 30 +++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 21a53760..f3625abe 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -816,7 +816,7 @@ badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") ``` -## 3. Generating new XRISM-Xtend images, exposure maps, and light curves +## 3. Generating new XRISM-Xtend images and exposure maps ### Converting energy bounds to channel bounds @@ -1086,8 +1086,17 @@ ex_path_temp = os.path.join( ) ``` +## 4. Generating new XRISM-Xtend spectra and light curves + ### New XRISM-Xtend light curves + +***WILL NEED TO ADD REGION FILE SUPPORT TO THE LIGHT CURVE GENERATION WRAPPER FUNCTION*** + +***PERHAPS I FINALLY IMPLEMENT MY IDEAS ABOUT HASHING THE REGION FILES TO GENERATE UNIQUE OUTPUT FILE NAMES*** + +***ALSO NEED TO SOURCE THE REGION FILES THAT DESCRIBE WHERE THE CALIBRATION SOURCES ARE*** + ```{code-cell} python lc_time_bin = Quantity(200, "s") ``` @@ -1097,6 +1106,8 @@ lc_time_bin = Quantity(200, "s") xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") ``` +***NEEEEEEEED GTI*** + ```{code-cell} python arg_combs = [ [ @@ -1116,9 +1127,24 @@ with mp.Pool(NUM_CORES) as p: lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) ``` -## 4. Generating new XRISM-Xtend spectra and supporting files +Create a template variable for output light curve file names: + +```{code-cell} python +lc_path_temp = ( + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}" + "-tb{tb}s-lightcurve.fits" +) +``` + +### Preparing to generate new XRISM-Xtend spectra + +```{code-cell} python + +``` +### +## 5. Fitting spectral models to XRISM-Xtend spectra ## About this notebook From 790e190e0be72ea37969422f9d81bf182a62e27d Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 10:21:08 -0500 Subject: [PATCH 113/221] Added some commentary on setting up source extraction regions to the getting-started-xrism-xtend.md notebook. Also indicated that N132D can be treated as a point source here, and extended source analysis will be handled in another notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 39 ++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f3625abe..a3ade618 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -860,7 +860,7 @@ We will be creating new RMFs as part of the generation of XRISM-Xtend spectra in next section. For our current purpose, however, it is acceptable to use the RMFs that were included in the XRISM-Xtend archive we downloaded earlier. -The archived RMFs are generated for the entire Xtend FoV, rather than for the CCDs +The archived RMFs are generated for the entire Xtend field-of-view (FoV), rather than for the CCDs our particular target fall on, but practically speaking, that doesn't make a significant difference. @@ -1088,6 +1088,43 @@ ex_path_temp = os.path.join( ## 4. Generating new XRISM-Xtend spectra and light curves +In this section we will demonstrate how to generate source-specific data products from +XRISM-Xtend observations; light curves and spectra (along with supporting files like +RMFs and Ancillary Response Files, or ARFs). + +### Setting up data product source and background extraction regions + +Rather than extracting spectra and light curves for the entire XRISM-Xtend FoV, +*which is how the quick-look spectra and light curves contained in the archive are made*, we +want to control exactly where we are taking events from. + +That way we can focus on the particular source(s) of interest present in the +XRISM-Xtend observations we are using. + +The size, shape, placement, and number of source extraction regions you need to use for +your work will depend heavily on your science case and the type of astrophysical +source you're analyzing. + +You will find that point sources are considerably easier to deal with, as you can +generally learn all you need from a single spectrum encompassing the entire source +emission region. + +Indeed, trying to extract spectra from different spatial regions of a point source (even +if the emission *appears* extended in XRISM-Xtend images) is **not valid**, as the +apparently extended emission is caused by the PSF of the telescope optics. + +The 'blurring' of the observed emission events by the PSF is one of the reasons that +extended sources are much harder to analyze than point sources. For example, you +might want to extract spectra from a series of annular bins centered on +your extended source to see how a particular spectral property changes in different +parts of the object. + +Unfortunately, due to the PSF, each annulus will be contaminated by (and be +*contaminating* in turn) events from other annuli, scattered there by the telescope PSF - this +effect is sometimes referred to as **cross-talk** or **spatial-spectral mixing (SSM)**. Accounting +for this effect is complicated and time-consuming, so our demonstration will focus on a point source, and +extended sources will be discussed in another notebook. + ### New XRISM-Xtend light curves From 36433f28f4ad015f69baa1e12fd0a1c1df508116 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 10:48:43 -0500 Subject: [PATCH 114/221] Added code cell that should write the RA-DEC version of the source region file to getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 33 +++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a3ade618..e6792aa6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1092,8 +1092,6 @@ In this section we will demonstrate how to generate source-specific data product XRISM-Xtend observations; light curves and spectra (along with supporting files like RMFs and Ancillary Response Files, or ARFs). -### Setting up data product source and background extraction regions - Rather than extracting spectra and light curves for the entire XRISM-Xtend FoV, *which is how the quick-look spectra and light curves contained in the archive are made*, we want to control exactly where we are taking events from. @@ -1125,6 +1123,37 @@ effect is sometimes referred to as **cross-talk** or **spatial-spectral mixing ( for this effect is complicated and time-consuming, so our demonstration will focus on a point source, and extended sources will be discussed in another notebook. +### Setting up data product source and background extraction regions + +There are different ways to define.... + +SOURCE: +radius of 2 arcmin + +BACKGROUND: +81.1932474 +-69.5073738 +radius of 4 arcmin + +```{code-cell} python +src_reg_rad = Quantity(2, "arcmin") + +radec_src_reg_path = os.path.join(OUT_PATH, "radec_src.reg") + +with open(radec_src_reg_path, "w") as src_rego: + src_radec_str = """ + # Region file format: DS9 version 4.1 + global color=green + icrs + circle({ra},{dec},{rad}d) + """.format( + ra=src_coord.ra.value, + dec=src_coord.dec.value, + rad=src_reg_rad.to("arcmin").value, + ) + src_rego.write(src_radec_str) +``` + ### New XRISM-Xtend light curves From e712d7a80b31db1841bb870d77a652a563ce6760 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 11:07:26 -0500 Subject: [PATCH 115/221] The getting-started-xrism-xtend.md notebooks now successfully generates source and background RA-DEC region files. They will be converted to sky coords for each observation, and possibly overlaid on every image from each observation. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 60 +++++++++++++------ 1 file changed, 42 insertions(+), 18 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index e6792aa6..f07e4169 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1127,33 +1127,57 @@ extended sources will be discussed in another notebook. There are different ways to define.... -SOURCE: -radius of 2 arcmin - -BACKGROUND: -81.1932474 --69.5073738 -radius of 4 arcmin - ```{code-cell} python +# The radius of the source extraction region src_reg_rad = Quantity(2, "arcmin") +# Where to write the new region file radec_src_reg_path = os.path.join(OUT_PATH, "radec_src.reg") +# Defining each line of the region file as an element of a list, then joining +# them all into a string with newlines +src_radec_lines = [ + "# Region file format: DS9 version 4.1", + "global color=green", + "icrs", + "circle({ra},{dec},{rad}d)", +] +src_radec_str = "\n".join(src_radec_lines).format( + ra=src_coord.ra.value, dec=src_coord.dec.value, rad=src_reg_rad.to("deg").value +) + with open(radec_src_reg_path, "w") as src_rego: - src_radec_str = """ - # Region file format: DS9 version 4.1 - global color=green - icrs - circle({ra},{dec},{rad}d) - """.format( - ra=src_coord.ra.value, - dec=src_coord.dec.value, - rad=src_reg_rad.to("arcmin").value, - ) src_rego.write(src_radec_str) ``` +We do the same to define a region from which to extract a background spectrum: + +```{code-cell} python +# The central coordinate of the background region +back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") + +# The radius of the background region +back_reg_rad = Quantity(3, "arcmin") + +# Where to write the new region file +radec_back_reg_path = os.path.join(OUT_PATH, "radec_back.reg") + +# Defining each line of the region file as an element of a list, then joining +# them all into a string with newlines +back_radec_lines = [ + "# Region file format: DS9 version 4.1", + "global color=green", + "icrs", + "circle({ra},{dec},{rad}d)", +] +back_radec_str = "\n".join(back_radec_lines).format( + ra=back_coord.ra.value, dec=back_coord.dec.value, rad=back_reg_rad.to("deg").value +) + +with open(radec_back_reg_path, "w") as back_rego: + back_rego.write(back_radec_str) +``` + ### New XRISM-Xtend light curves From c9127e4ab2b5f24a18e040efb7e20fc43d0363e6 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 11:22:47 -0500 Subject: [PATCH 116/221] Added source and background region support to the 'gen_xrism_xtend_lightcurve' wrapper function for XRISM-Xtend light curve generation in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 27 +++++++++++++++---- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f07e4169..b1dfef17 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -362,8 +362,10 @@ def gen_xrism_xtend_lightcurve( cur_xtend_data_class: str, event_file: str, out_dir: str, - lo_en: Quantity, - hi_en: Quantity, + src_reg_file: str, + back_reg_file: str, + lo_en: Quantity = Quantity(0.6, "keV"), + hi_en: Quantity = Quantity(13, "keV"), time_bin_size: Quantity = Quantity(200, "s"), lc_bin_thresh: float = 0.0, ): @@ -422,12 +424,15 @@ def gen_xrism_xtend_lightcurve( # PI channel limits to subset the events evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" - # Set up the output file name for the image we're about to generate. + # Set up the output file name for the light curve we're about to generate. lc_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" f"en{lo_en_val}_{hi_en_val}keV-expthresh{lc_bin_thresh}-tb{time_bin_size}s-" f"lightcurve.fits" ) + # The same file name, but with 'lightcurve' changed to 'back-lightcurve', for the + # background light curve. + lc_back_out = lc_out.replace("lightcurve", "back-lightcurve") # Create a temporary working directory temp_work_dir = os.path.join( @@ -439,11 +444,23 @@ def gen_xrism_xtend_lightcurve( # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): - out = hsp.extractor( + src_out = hsp.extractor( filename=evt_file_chan_sel, fitsbinlc=os.path.join("..", lc_out), binlc=time_bin_size, lcthresh=lc_bin_thresh, + regionfile=src_reg_file, + noprompt=True, + clobber=True, + ) + + # Now for the background light curve + back_out = hsp.extractor( + filename=evt_file_chan_sel, + fitsbinlc=os.path.join("..", lc_back_out), + binlc=time_bin_size, + lcthresh=lc_bin_thresh, + regionfile=back_reg_file, noprompt=True, clobber=True, ) @@ -451,7 +468,7 @@ def gen_xrism_xtend_lightcurve( # Make sure to remove the temporary directory rmtree(temp_work_dir) - return out + return [src_out, back_out] ``` ### Constants From 7ad4e43fe8da5d2fe46304a108458b9705547ef1 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 12:50:44 -0500 Subject: [PATCH 117/221] Added xcolf and ycolf arguments to the extractor call to generate images from XRISM-Xtend data in the getting-started-xrism-xtend.md notebook. If those arguments are missing it seems that no sky-pixel WCS is written to the resulting images, which is obviously a problem. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b1dfef17..e208159a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -72,6 +72,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc +from xga.products import Image # from typing import Tuple, Union # from warnings import warn @@ -242,6 +243,8 @@ def gen_xrism_xtend_image( noprompt=True, clobber=True, binf=im_bin, + xcolf="X", + ycolf="Y", ) # Make sure to remove the temporary directory @@ -975,7 +978,7 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ```{code-cell} python # Defining the energy bounds we want images within -xtd_im_en_bounds = Quantity([[0.4, 2.0], [0.6, 2.0], [2.0, 10.0], [0.4, 10.0]], "keV") +xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV") ``` Converting those energy bounds to channel bounds is straightforward, we simply divide @@ -1013,7 +1016,7 @@ factor would be required to minimize cross-talk between image pixels. As such, t should not be the primary motivation for your choice of image binning factor. ```{code-cell} python -bin_factor = 4 +bin_factors = [1, 4] ``` #### Running image generation @@ -1030,11 +1033,12 @@ arg_combs = [ evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), *cur_bnds, - bin_factor, + cur_bf, ] for oi, dcs in rel_dataclasses.items() for dc in dcs for cur_bnds in xtd_im_en_bounds + for cur_bf in bin_factors ] with mp.Pool(NUM_CORES) as p: @@ -1080,12 +1084,13 @@ arg_combs = [ ehk_path_temp.format(oi=oi), badpix_path_temp.format(oi=oi, xdc=dc, sc=0), "NONE", - bin_factor, + cur_bf, expmap_rad_delta, expmap_phi_bins, ] for oi, dcs in rel_dataclasses.items() for dc in dcs + for cur_bf in bin_factors ] with mp.Pool(NUM_CORES) as p: @@ -1195,6 +1200,12 @@ with open(radec_back_reg_path, "w") as back_rego: back_rego.write(back_radec_str) ``` +Examining... + +```{code-cell} python + +``` + ### New XRISM-Xtend light curves @@ -1246,7 +1257,14 @@ lc_path_temp = ( ### Preparing to generate new XRISM-Xtend spectra ```{code-cell} python +chos_im_en = xtd_im_en_bounds[0].to("keV") +for oi, cur_dcs in rel_dataclasses.items(): + for dc in cur_dcs: + cur_im_path = im_path_temp.format( + oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value + ) + cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) ``` ### From 3ba4fb4e48e078a071bdfae22c42f3c7d2440a84 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 14:26:06 -0500 Subject: [PATCH 118/221] Regions should now be included and properly handled in the getting-started-xrism-xtend.md notebook. Implemented for the light curve generation step at the moment. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 103 ++++++++++-------- 1 file changed, 58 insertions(+), 45 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index e208159a..097b5a36 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -72,6 +72,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc +from regions import CircleSkyRegion from xga.products import Image # from typing import Tuple, Union @@ -453,6 +454,8 @@ def gen_xrism_xtend_lightcurve( binlc=time_bin_size, lcthresh=lc_bin_thresh, regionfile=src_reg_file, + xcolf="X", + ycolf="Y", noprompt=True, clobber=True, ) @@ -464,6 +467,8 @@ def gen_xrism_xtend_lightcurve( binlc=time_bin_size, lcthresh=lc_bin_thresh, regionfile=back_reg_file, + xcolf="X", + ycolf="Y", noprompt=True, clobber=True, ) @@ -483,7 +488,7 @@ jupyter: source_hidden: true --- # The name of the source we're examining in this demonstration -SRC_NAME = "LMC N132D" +SRC_NAME = "LMCN132D" # SRC_NAME = "NGC4151" # SRC_NAME = "AX J1910.7+0917" @@ -1149,69 +1154,84 @@ extended sources will be discussed in another notebook. There are different ways to define.... +#### General RA-DEC region files + ```{code-cell} python +# Where to write the new region file +radec_src_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_src.reg") + # The radius of the source extraction region src_reg_rad = Quantity(2, "arcmin") -# Where to write the new region file -radec_src_reg_path = os.path.join(OUT_PATH, "radec_src.reg") - -# Defining each line of the region file as an element of a list, then joining -# them all into a string with newlines -src_radec_lines = [ - "# Region file format: DS9 version 4.1", - "global color=green", - "icrs", - "circle({ra},{dec},{rad}d)", -] -src_radec_str = "\n".join(src_radec_lines).format( - ra=src_coord.ra.value, dec=src_coord.dec.value, rad=src_reg_rad.to("deg").value -) +# Setting up a 'regions' module circular sky region instance +src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) -with open(radec_src_reg_path, "w") as src_rego: - src_rego.write(src_radec_str) +# Write the source region to a region file +src_reg.write(radec_src_reg_path, format="ds9") ``` We do the same to define a region from which to extract a background spectrum: ```{code-cell} python +# Where to write the new region file +radec_back_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_back.reg") + # The central coordinate of the background region back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") # The radius of the background region back_reg_rad = Quantity(3, "arcmin") -# Where to write the new region file -radec_back_reg_path = os.path.join(OUT_PATH, "radec_back.reg") - -# Defining each line of the region file as an element of a list, then joining -# them all into a string with newlines -back_radec_lines = [ - "# Region file format: DS9 version 4.1", - "global color=green", - "icrs", - "circle({ra},{dec},{rad}d)", -] -back_radec_str = "\n".join(back_radec_lines).format( - ra=back_coord.ra.value, dec=back_coord.dec.value, rad=back_reg_rad.to("deg").value -) +# Setting up a 'regions' module circular sky region instance for the background region +back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) -with open(radec_back_reg_path, "w") as back_rego: - back_rego.write(back_radec_str) +# Once again writing the region to a region file as well +back_reg.write(radec_back_reg_path, format="ds9") ``` +#### Visualizing the source and background extraction regions on XRISM-Xtend images + Examining... ```{code-cell} python +chos_im_en = xtd_im_en_bounds[0].to("keV") +oi_skypix_wcs = {} +for oi, cur_dcs in rel_dataclasses.items(): + for dc in cur_dcs: + cur_im_path = im_path_temp.format( + oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value + ) + cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) + cur_im.regions = [src_reg, back_reg] + cur_im.view(src_coord_quant, zoom_in=True, view_regions=True) + + oi_skypix_wcs.setdefault(oi, cur_im.radec_wcs) ``` -### New XRISM-Xtend light curves +#### Observation specific sky-pixel coordinate region files + +```{code-cell} python +obs_src_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_src.reg") +obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_back.reg") +for oi in rel_obsids: + src_reg.to_pixel(oi_skypix_wcs[oi]).write( + obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9" + ) + back_reg.to_pixel(oi_skypix_wcs[oi]).write( + obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9" + ) +``` -***WILL NEED TO ADD REGION FILE SUPPORT TO THE LIGHT CURVE GENERATION WRAPPER FUNCTION*** +```{tip} +Events from different data classes of **the same observation** share a common sky-pixel +coordinate system, so sky-pixel region files for one are also valid for the other. +However, different data classes represent different pairs of Xtend CCDs, so there is +no shared sky coverage. +``` -***PERHAPS I FINALLY IMPLEMENT MY IDEAS ABOUT HASHING THE REGION FILES TO GENERATE UNIQUE OUTPUT FILE NAMES*** +### New XRISM-Xtend light curves ***ALSO NEED TO SOURCE THE REGION FILES THAT DESCRIBE WHERE THE CALIBRATION SOURCES ARE*** @@ -1233,6 +1253,8 @@ arg_combs = [ dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), + obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), + obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), *cur_bnds, lc_time_bin, ] @@ -1256,16 +1278,7 @@ lc_path_temp = ( ### Preparing to generate new XRISM-Xtend spectra -```{code-cell} python -chos_im_en = xtd_im_en_bounds[0].to("keV") -for oi, cur_dcs in rel_dataclasses.items(): - for dc in cur_dcs: - cur_im_path = im_path_temp.format( - oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value - ) - cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) -``` ### From c6a7fc2f47dcd216ee27d6318862469c96b6268e Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 15:04:25 -0500 Subject: [PATCH 119/221] Added first pass at a function to generate XRISM Xtend spectra to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 108 +++++++++++++++++- 1 file changed, 105 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 097b5a36..73edfcd5 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -49,7 +49,7 @@ XRISM is... ### Runtime -As of 22nd November 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 26nd November 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -477,6 +477,106 @@ def gen_xrism_xtend_lightcurve( rmtree(temp_work_dir) return [src_out, back_out] + + +def gen_xrism_xtend_spectra( + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + src_reg_file: str, + back_reg_file: str, + lo_en: Quantity = Quantity(0.6, "keV"), + hi_en: Quantity = Quantity(13, "keV"), +): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend spectrum. + :param str cur_xtend_data_class: + :param str event_file: + :param str out_dir: The directory where output files should be written. + :param Quantity lo_en: Lower bound of the energy band within which we will + generate the spectrum. + :param Quantity hi_en: Upper bound of the energy band within which we will + generate the spectrum. + :return: A tuple containing the processed ObsID, the log output of the + pipeline, and a boolean flag indicating success (True) or failure (False). + :rtype: Tuple[str, hsp.core.HSPResult, bool] + """ + + # Validity check on the passed data class + if cur_xtend_data_class[0] != "3": + raise ValueError( + f"The first digit of the Xtend data class ({cur_xtend_data_class}) " + "must be 3 for in-flight data." + ) + + # Make sure the lower and upper energy limits make sense + if lo_en > hi_en: + raise ValueError( + "The lower energy limit must be less than or equal to the upper " + "energy limit." + ) + else: + lo_en_val = lo_en.to("keV").value + hi_en_val = hi_en.to("keV").value + + # Convert the energy limits to channel limits, rounding down and up to the nearest + # integer channel for the lower and upper bounds respectively. + lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) + + # Create modified input event list file path, where we use the just-calculated + # PI channel limits to subset the events + evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" + + # Set up the output file name for the light curve we're about to generate. + sp_out = ( + f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" + f"en{lo_en_val}_{hi_en_val}keV-" + f"spectrum.fits" + ) + # The same file name, but with 'spectrum' changed to 'back-spectrum', for the + # background light curve. + sp_back_out = sp_out.replace("spectrum", "back-spectrum") + + # Create a temporary working directory + temp_work_dir = os.path.join( + out_dir, "spec_extractor_{}".format(randint(0, int(1e8))) + ) + os.makedirs(temp_work_dir) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + src_out = hsp.extractor( + filename=evt_file_chan_sel, + phafile=os.path.join("..", sp_out), + regionfile=src_reg_file, + xcolf="X", + ycolf="Y", + ecol="PI", + noprompt=True, + clobber=True, + ) + + # Now for the background light curve + back_out = hsp.extractor( + filename=evt_file_chan_sel, + phafile=os.path.join("..", sp_back_out), + regionfile=back_reg_file, + xcolf="X", + ycolf="Y", + ecol="PI", + noprompt=True, + clobber=True, + ) + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return [src_out, back_out] ``` ### Constants @@ -1276,7 +1376,9 @@ lc_path_temp = ( ) ``` -### Preparing to generate new XRISM-Xtend spectra +### Generate new XRISM-Xtend spectra + + @@ -1290,7 +1392,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-11-25 +Updated On: 2025-11-26 +++ From 87b82dc4cde81c1e7606a4a2a43ce092dcb3fbf0 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 15:24:50 -0500 Subject: [PATCH 120/221] Started to add spec generation and grouping sections and code to the getting-started-xrism-xtend.md notebook. For issue #128. --- .../xrism/getting-started-xrism-xtend.md | 100 +++++++++++++++++- 1 file changed, 97 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 73edfcd5..91389fc8 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -479,7 +479,7 @@ def gen_xrism_xtend_lightcurve( return [src_out, back_out] -def gen_xrism_xtend_spectra( +def gen_xrism_xtend_spectrum( cur_obs_id: str, cur_xtend_data_class: str, event_file: str, @@ -1367,20 +1367,114 @@ with mp.Pool(NUM_CORES) as p: lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) ``` -Create a template variable for output light curve file names: +Create template variables for source and background light curves: ```{code-cell} python lc_path_temp = ( "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}" "-tb{tb}s-lightcurve.fits" ) + +back_lc_path_temp = ( + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV" + "-expthresh{lct}-tb{tb}s-back-lightcurve.fits" +) +``` + +### New XRISM-Xtend spectra and supporting files + +```{code-cell} python + +``` + +#### Generating the spectral files + +```{code-cell} python +spec_lo_en = Quantity(0.6, "keV") +spec_hi_en = Quantity(13, "keV") +``` + +```{code-cell} python +arg_combs = [ + [ + oi, + dc, + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + os.path.join(OUT_PATH, oi), + obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), + obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + spec_lo_en, + spec_hi_en, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs +] + +with mp.Pool(NUM_CORES) as p: + sp_result = p.starmap(gen_xrism_xtend_spectrum, arg_combs) +``` + +Create template variables for source and background spectrum files: + +```{code-cell} python +sp_path_temp = "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-spectrum.fits" + +back_sp_path_temp = ( + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-back-spectrum.fits" +) ``` -### Generate new XRISM-Xtend spectra +#### Grouping our new spectra +We will group the spectra we just generated. Grouping essentially combines +spectral channels until some minimum quality threshold is reached; in this case a +minimum of one count per grouped channel. We use the HEASoft `ftgrouppha` tool to do +this, once again through HEASoftPy. +First, we set up the grouping criteria and a template variable for the name of the +output grouped spectral files: +*** REMIND MYSELF WHETHER GROUPING FROM SOURCE SPEC IS AUTOMATICALLY APPLIED TO BACK SPEC IN XSPEC? *** +```{code-cell} python +spec_group_type = "min" +spec_group_scale = 1 + +grp_sp_path_temp = sp_path_temp.replace("-spectrum", "-{gt}grp{gs}-spectrum") +``` + +Now we run the grouping tool - though this time we do not parallelize the task, as +the grouping process is very fast, and we wish to demonstrate how you use a HEASoftPy +function directly. Though remember to look at the Global Setup section of this notebook +to see how we call HEASoftPy tools in the wrapper functions used to parallelize those +tasks. + +If you are dealing with significantly more observations than we use for this +demonstration, we do recommend that you parallelize this grouping step as we have +the other processing steps in this notebook. + +```{code-cell} python +for oi, dcs in rel_dataclasses.items(): + for cur_dc in dcs: + cur_spec = sp_path_temp.format( + oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + ) + cur_grp_spec = grp_sp_path_temp.format( + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + gt=spec_group_type, + gs=spec_group_scale, + ) + + hsp.ftgrouppha( + infile=cur_spec, + outfile=cur_grp_spec, + grouptype=spec_group_type, + groupscale=spec_group_scale, + ) +``` ### From 5f06d3a0244430d5a85c809f484e1574f281f76b Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 16:26:44 -0500 Subject: [PATCH 121/221] Added a subsection to the part of getting-started-xrism-xtend.md that generates new spectra which calculates correct values of BACKSCAL for the source and background spectra. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 83 ++++++++++++++++--- 1 file changed, 73 insertions(+), 10 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 91389fc8..14e9d3e6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1309,6 +1309,14 @@ for oi, cur_dcs in rel_dataclasses.items(): oi_skypix_wcs.setdefault(oi, cur_im.radec_wcs) ``` +#### Excluding the XRISM-Xtend calibration sources + +```{code-cell} python +detpix_xtend_calib_reg_path = os.path.join( + os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" +) +``` + #### Observation specific sky-pixel coordinate region files ```{code-cell} python @@ -1370,14 +1378,18 @@ with mp.Pool(NUM_CORES) as p: Create template variables for source and background light curves: ```{code-cell} python -lc_path_temp = ( - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}" - "-tb{tb}s-lightcurve.fits" +lc_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" + "-lightcurve.fits", ) -back_lc_path_temp = ( - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV" - "-expthresh{lct}-tb{tb}s-back-lightcurve.fits" +back_lc_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" + "-back-lightcurve.fits", ) ``` @@ -1417,13 +1429,60 @@ with mp.Pool(NUM_CORES) as p: Create template variables for source and background spectrum files: ```{code-cell} python -sp_path_temp = "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-spectrum.fits" +sp_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-spectrum.fits", +) -back_sp_path_temp = ( - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-back-spectrum.fits" +back_sp_path_temp = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-back-spectrum.fits", ) ``` +#### Calculating the 'BACKSCAL' value for new XRISM-Xtend spectra + +***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** + +```{code-cell} python +for oi, dcs in rel_dataclasses.items(): + for cur_dc in dcs: + # Set up the path to input source and background spectra + cur_spec = sp_path_temp.format( + oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + ) + cur_bspec = back_sp_path_temp.format( + oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + ) + + # Also need to pass an exposure map, so set up a path to that + cur_ex = ex_path_temp.format( + oi=oi, + xdc=cur_dc, + rd=expmap_rad_delta.to("arcmin").value, + npb=expmap_phi_bins, + ibf=1, + ) + + # Calculate the BACKSCAL keyword, first for the source spectrum + hsp.ahbackscal( + infile=cur_spec, + regfile=obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), + expfile=cur_ex, + logfile="NONE", + ) + + # Then for the background spectrum + hsp.ahbackscal( + infile=cur_bspec, + regfile=obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + expfile=cur_ex, + logfile="NONE", + ) +``` + #### Grouping our new spectra We will group the spectra we just generated. Grouping essentially combines @@ -1456,6 +1515,7 @@ the other processing steps in this notebook. ```{code-cell} python for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: + # Set up relevant paths to the input and output spectrum cur_spec = sp_path_temp.format( oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value ) @@ -1476,7 +1536,10 @@ for oi, dcs in rel_dataclasses.items(): ) ``` -### +#### Generating XRISM-Xtend RMFs + +#### Generating XRISM-Xtend ARFs + ## 5. Fitting spectral models to XRISM-Xtend spectra From dad4dbd8b08826cb9768578e8741e092b11f8931 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 26 Nov 2025 16:55:23 -0500 Subject: [PATCH 122/221] Added first pass at the wrapper function to generate RMFs for XRISM Xtend in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 58 ++++++++++++++++++- 1 file changed, 55 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 14e9d3e6..f20cb04e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -577,6 +577,37 @@ def gen_xrism_xtend_spectrum( rmtree(temp_work_dir) return [src_out, back_out] + + +def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): + """ + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend RMF. + :param str out_dir: The directory where output files should be written. + """ + + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xtdrmf_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) + + # Set up the RMF file name by cannibalising the name of the spectrum file + rmf_out = os.path.basename(spec_file).split("-en")[0] + ".rmf" + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + out = hsp.xtdrmf( + infile=spec_file, + outfile=os.path.join("..", rmf_out), + noprompt=True, + clobber=True, + ) + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return out ``` ### Constants @@ -1311,6 +1342,10 @@ for oi, cur_dcs in rel_dataclasses.items(): #### Excluding the XRISM-Xtend calibration sources +***HAVE TO LOAD THE CALIBRATION SOURCE REGIONS, CONVERT TO SKY PIX SYSTEM, AND EXCLUDE THEM FROM THE EXTRACTION REGION DEFINITIONS*** + +***THIS FILE IS SET UP FOR EXCLUSION (I.E. WITH - IN FRONT OF REGIONS) SO THE REGIONS MODULE WON'T READ THEM IN, AT LEAST BY DEFAULT*** + ```{code-cell} python detpix_xtend_calib_reg_path = os.path.join( os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" @@ -1341,8 +1376,6 @@ no shared sky coverage. ### New XRISM-Xtend light curves -***ALSO NEED TO SOURCE THE REGION FILES THAT DESCRIBE WHERE THE CALIBRATION SOURCES ARE*** - ```{code-cell} python lc_time_bin = Quantity(200, "s") ``` @@ -1442,7 +1475,7 @@ back_sp_path_temp = os.path.join( ) ``` -#### Calculating the 'BACKSCAL' value for new XRISM-Xtend spectra +#### Calculating 'BACKSCAL' for new XRISM-Xtend spectra ***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** @@ -1538,6 +1571,25 @@ for oi, dcs in rel_dataclasses.items(): #### Generating XRISM-Xtend RMFs +***THIS IS ALSO GOING TO FALL OVER IN PART BECAUSE CAN'T EXTRACT SPECTRUM FROM REGIONS NOT ON THE 31100010 DATACLASS OBSERVATION OF 000128000*** + +```{code-cell} python +arg_combs = [ + [ + oi, + sp_path_temp.format( + oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + ), + os.path.join(OUT_PATH, oi), + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs +] + +with mp.Pool(NUM_CORES) as p: + rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) +``` + #### Generating XRISM-Xtend ARFs From e512cf71e96b82cf4aae7ccfaf30e2399c2b8666 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 1 Dec 2025 12:24:16 -0500 Subject: [PATCH 123/221] Getting ARF generation closer to being functional in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 170 +++++++++++++++++- 1 file changed, 161 insertions(+), 9 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f20cb04e..507e523e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -484,12 +484,15 @@ def gen_xrism_xtend_spectrum( cur_xtend_data_class: str, event_file: str, out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, src_reg_file: str, back_reg_file: str, lo_en: Quantity = Quantity(0.6, "keV"), hi_en: Quantity = Quantity(13, "keV"), ): """ + IMPLICITLY ASSUMES THE REGION IS A CIRCLE :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend spectrum. :param str cur_xtend_data_class: @@ -530,15 +533,22 @@ def gen_xrism_xtend_spectrum( # PI channel limits to subset the events evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" + # Get RA, Dec, and radius values in the right format + ra_val = rel_src_coord.ra.to("deg").value.round(6) + dec_val = rel_src_coord.dec.to("deg").value.round(6) + rad_val = rel_src_radius.to("deg").value.round(4) + # Set up the output file name for the light curve we're about to generate. sp_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"en{lo_en_val}_{hi_en_val}keV-" + f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-en{lo_en_val}_{hi_en_val}keV-" f"spectrum.fits" ) # The same file name, but with 'spectrum' changed to 'back-spectrum', for the # background light curve. - sp_back_out = sp_out.replace("spectrum", "back-spectrum") + sp_back_out = sp_out.replace(f"-radius{rad_val}deg", "").replace( + "spectrum", "back-spectrum" + ) # Create a temporary working directory temp_work_dir = os.path.join( @@ -591,7 +601,7 @@ def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): os.makedirs(temp_work_dir) # Set up the RMF file name by cannibalising the name of the spectrum file - rmf_out = os.path.basename(spec_file).split("-en")[0] + ".rmf" + rmf_out = os.path.basename(spec_file).split("-ra")[0] + ".rmf" # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so @@ -608,6 +618,65 @@ def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): rmtree(temp_work_dir) return out + + +def gen_xrism_xtend_arf( + cur_obs_id: str, + out_dir: str, + expmap_file: str, + spec_file: str, + rmf_file: str, + num_photons: int, +): + """ + IMPLICITLY ASSUMES THAT WE'RE GENERATING AN ARF FOR A 'POINT SOURCE' + + :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend ARF. + :param str out_dir: The directory where output files should be written. + """ + + # Spectrum files generated in this demonstration notebook contain RA-Dec + # information in their file name, so we will read it out from there + radec_sec = os.path.basename(spec_file).split("-radius")[0].split("-ra")[1] + cen_strs = radec_sec.split("-dec") + ra_val, dec_val = [float(crd) for crd in cen_strs] + + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xaarfgen_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) + + # We can use the spectrum file name to set up the output ARF file name + arf_out = os.path.basename(spec_file).replace("-spectrum.fits", ".arf") + + # Set up a name for the ray-traced simulated event file required for + # XRISM ARF generation + ray_traced_evt_out = ( + f"xrism-xtend-obsid{cur_obs_id}-numphoton{num_photons}-" + f"enALL-raytracedevents.fits" + ) + + # Using dual contexts, one that moves us into the output directory for the + # duration, and another that creates a new set of HEASoft parameter files (so + # there are no clashes with other processes). + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + out = hsp.xaarfgen( + xrtevtfile=os.path.join("..", ray_traced_evt_out), + outfile=os.path.join("..", arf_out), + sourcetype="POINT", + numphotons=num_photons, + source_ra=ra_val, + source_dec=dec_val, + telescop="XRISM", + instrume="XTEND", + emapfile=expmap_file, + noprompt=True, + clobber=True, + ) + + # Make sure to remove the temporary directory + rmtree(temp_work_dir) + + return out ``` ### Constants @@ -1465,13 +1534,15 @@ Create template variables for source and background spectrum files: sp_path_temp = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-spectrum.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" + "en{lo}_{hi}keV-spectrum.fits", ) back_sp_path_temp = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-back-spectrum.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" + "en{lo}_{hi}keV-back-spectrum.fits", ) ``` @@ -1484,10 +1555,21 @@ for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up the path to input source and background spectra cur_spec = sp_path_temp.format( - oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), ) cur_bspec = back_sp_path_temp.format( - oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), ) # Also need to pass an exposure map, so set up a path to that @@ -1550,7 +1632,13 @@ for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up relevant paths to the input and output spectrum cur_spec = sp_path_temp.format( - oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), ) cur_grp_spec = grp_sp_path_temp.format( oi=oi, @@ -1578,7 +1666,13 @@ arg_combs = [ [ oi, sp_path_temp.format( - oi=oi, xdc=cur_dc, lo=spec_lo_en.value, hi=spec_hi_en.value + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), ), os.path.join(OUT_PATH, oi), ] @@ -1590,8 +1684,66 @@ with mp.Pool(NUM_CORES) as p: rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) ``` +```{code-cell} python +rmf_path_temp = os.path.join( + OUT_PATH, "{oi}", "xrism-xtend-obsid{oi}-dataclass{xdc}.rmf" +) +``` + +#### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation + +```{code-cell} python + +``` + #### Generating XRISM-Xtend ARFs +```{danger} +The HEASoft task we use to generate ARFs is called **`xaarfgen`**. There is +another, very similarly named, HEASoft tool related to the construction of XRISM +ARFs, **`xaxmaarfgen`**. Be sure which one you are using! +``` + +```{code-cell} python +arf_rt_num_photons = 20000 +``` + +```{code-cell} python +arg_combs = [ + [ + oi, + os.path.join(OUT_PATH, oi), + ex_path_temp.format( + oi=oi, + xdc=dc, + rd=expmap_rad_delta.to("arcmin").value, + npb=expmap_phi_bins, + ibf=1, + ), + sp_path_temp.format( + oi=oi, + xdc=cur_dc, + lo=spec_lo_en.value, + hi=spec_hi_en.value, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), + ), + rmf_path_temp.format(oi=oi, xdc=dc), + arf_rt_num_photons, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs +] + +with mp.Pool(NUM_CORES) as p: + arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) +``` + +```{warning} +Due to the high-fidelity ray-tracing method used to calculate XRISM ARFs, the runtime +of this step can be on the order of hours. We have to do ***FINISH*** +``` ## 5. Fitting spectral models to XRISM-Xtend spectra From a34a2a6a4b682c0dd1bdc72acaeaf5183cad6fbb Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 1 Dec 2025 17:25:02 -0500 Subject: [PATCH 124/221] Missed passing the RMF file to the xaarfgen call in the wrapper function for XRISM-Xtend ARF generation in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 33 ++++++++++++++++--- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 507e523e..66c4804b 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-11-25' +date: '2025-12-01' file_format: mystnb jupytext: text_representation: @@ -49,7 +49,7 @@ XRISM is... ### Runtime -As of 26nd November 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 1st December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -240,7 +240,7 @@ def gen_xrism_xtend_image( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.extractor( filename=evt_file_chan_sel, - imgfile=os.path.join(out_dir, im_out), + imgfile=os.path.join("..", im_out), noprompt=True, clobber=True, binf=im_bin, @@ -337,7 +337,7 @@ def gen_xrism_xtend_expmap( pixgtifile=pix_gti_file, delta=radial_delta, numphi=num_phi_bin, - outfile=os.path.join(out_dir, ex_out), + outfile=os.path.join("..", ex_out), badimgfile=bad_pix_file, outmaptype=out_map_type, noprompt=True, @@ -669,6 +669,7 @@ def gen_xrism_xtend_arf( telescop="XRISM", instrume="XTEND", emapfile=expmap_file, + rmffile=rmf_file, noprompt=True, clobber=True, ) @@ -1515,6 +1516,8 @@ arg_combs = [ dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), + src_coord, + src_reg_rad, obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), spec_lo_en, @@ -1647,6 +1650,9 @@ for oi, dcs in rel_dataclasses.items(): hi=spec_hi_en.value, gt=spec_group_type, gs=spec_group_scale, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), ) hsp.ftgrouppha( @@ -1708,6 +1714,19 @@ ARFs, **`xaxmaarfgen`**. Be sure which one you are using! arf_rt_num_photons = 20000 ``` +```{warning} +***MIGHT BE WRONG, BUT v6.35.2 OF HEASOFT MIGHT POINT TO THE WRONG CALDB FILE PATH MIRROR +SCATTER INFORMATION USED BY RAYTRACE (xa_xtd_scatter_20190101v001.fits RATHER THAN xa_xtd_scatter_20190101v001.fits.gz)?*** + +***HEASOFT RELEASE NOTES SEEM TO BEAR THIS IDEA UP***: + +xrtraytrace: Additional updates to fix and enable remote CalDB + usage with xrtraytrace and xaarfgen ("timeout" interval extended + for reading large CalDB files). + + +``` + ```{code-cell} python arg_combs = [ [ @@ -1740,6 +1759,10 @@ with mp.Pool(NUM_CORES) as p: arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) ``` +```{code-cell} python +arf_path_temp = sp_path_temp.replace("-spectrum", ".arf") +``` + ```{warning} Due to the high-fidelity ray-tracing method used to calculate XRISM ARFs, the runtime of this step can be on the order of hours. We have to do ***FINISH*** @@ -1753,7 +1776,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-11-26 +Updated On: 2025-12-01 +++ From 491e74aa81fa9fb603ddb2c48fd6310e493c80f4 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 1 Dec 2025 17:38:16 -0500 Subject: [PATCH 125/221] Had to pass RA-Dec region file to the arf generation tool for XRISM-Xtend in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 66c4804b..cff4df34 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -626,6 +626,7 @@ def gen_xrism_xtend_arf( expmap_file: str, spec_file: str, rmf_file: str, + src_radec_reg_file: str, num_photons: int, ): """ @@ -670,6 +671,8 @@ def gen_xrism_xtend_arf( instrume="XTEND", emapfile=expmap_file, rmffile=rmf_file, + regionfile=src_radec_reg_file, + regmode="RADEC", noprompt=True, clobber=True, ) From 408876475cef99ce968914fd366e9037c24ca8f6 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 1 Dec 2025 17:42:42 -0500 Subject: [PATCH 126/221] Had to add a temporary manually-defined link to a caldb file in the wrapper function for generation XRISM products. This is only until a problem with the caldb is sorted. For issue #128. ARFS SHOULD NOW GENERATE PROPERLY --- .../xrism/getting-started-xrism-xtend.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index cff4df34..5fe832d6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -675,8 +675,12 @@ def gen_xrism_xtend_arf( regmode="RADEC", noprompt=True, clobber=True, + scatterfile="https://heasarc.gsfc.nasa.gov/FTP/caldb/data/xrism/xtend/" + "bcf/mirror/xa_xtd_scatter_20190101v001.fits.gz[1]", ) + # TODO REMOVE THE DIRECT SETTING OF SCATTERFILE ONCE CALDB IS FIXED + # Make sure to remove the temporary directory rmtree(temp_work_dir) From 9de57e8f8fd9a94b375556d18d30b79ec0017e70 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 2 Dec 2025 13:05:44 -0500 Subject: [PATCH 127/221] Removed the demonstration of generating spectra within a certain energy range from getting-started-xrism-xtend.md. If you restrict the range the RMF and ARF have to be generated to match and it is too indepth for too little gain in a getting started demo. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 88 +++++++------------ 1 file changed, 34 insertions(+), 54 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 5fe832d6..d81bdacb 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-01' +date: '2025-12-02' file_format: mystnb jupytext: text_representation: @@ -64,8 +64,7 @@ from shutil import rmtree import heasoftpy as hsp import matplotlib.pyplot as plt import numpy as np - -# import xspec as xs +import xspec as xs from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.table import Table @@ -488,8 +487,6 @@ def gen_xrism_xtend_spectrum( rel_src_radius: Quantity, src_reg_file: str, back_reg_file: str, - lo_en: Quantity = Quantity(0.6, "keV"), - hi_en: Quantity = Quantity(13, "keV"), ): """ IMPLICITLY ASSUMES THE REGION IS A CIRCLE @@ -498,10 +495,6 @@ def gen_xrism_xtend_spectrum( :param str cur_xtend_data_class: :param str event_file: :param str out_dir: The directory where output files should be written. - :param Quantity lo_en: Lower bound of the energy band within which we will - generate the spectrum. - :param Quantity hi_en: Upper bound of the energy band within which we will - generate the spectrum. :return: A tuple containing the processed ObsID, the log output of the pipeline, and a boolean flag indicating success (True) or failure (False). :rtype: Tuple[str, hsp.core.HSPResult, bool] @@ -514,25 +507,6 @@ def gen_xrism_xtend_spectrum( "must be 3 for in-flight data." ) - # Make sure the lower and upper energy limits make sense - if lo_en > hi_en: - raise ValueError( - "The lower energy limit must be less than or equal to the upper " - "energy limit." - ) - else: - lo_en_val = lo_en.to("keV").value - hi_en_val = hi_en.to("keV").value - - # Convert the energy limits to channel limits, rounding down and up to the nearest - # integer channel for the lower and upper bounds respectively. - lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) - hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) - - # Create modified input event list file path, where we use the just-calculated - # PI channel limits to subset the events - evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" - # Get RA, Dec, and radius values in the right format ra_val = rel_src_coord.ra.to("deg").value.round(6) dec_val = rel_src_coord.dec.to("deg").value.round(6) @@ -541,7 +515,7 @@ def gen_xrism_xtend_spectrum( # Set up the output file name for the light curve we're about to generate. sp_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-en{lo_en_val}_{hi_en_val}keV-" + f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-enALL-" f"spectrum.fits" ) # The same file name, but with 'spectrum' changed to 'back-spectrum', for the @@ -561,7 +535,7 @@ def gen_xrism_xtend_spectrum( # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): src_out = hsp.extractor( - filename=evt_file_chan_sel, + filename=event_file, phafile=os.path.join("..", sp_out), regionfile=src_reg_file, xcolf="X", @@ -573,7 +547,7 @@ def gen_xrism_xtend_spectrum( # Now for the background light curve back_out = hsp.extractor( - filename=evt_file_chan_sel, + filename=event_file, phafile=os.path.join("..", sp_back_out), regionfile=back_reg_file, xcolf="X", @@ -1511,11 +1485,6 @@ back_lc_path_temp = os.path.join( #### Generating the spectral files -```{code-cell} python -spec_lo_en = Quantity(0.6, "keV") -spec_hi_en = Quantity(13, "keV") -``` - ```{code-cell} python arg_combs = [ [ @@ -1527,8 +1496,6 @@ arg_combs = [ src_reg_rad, obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), - spec_lo_en, - spec_hi_en, ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -1545,14 +1512,13 @@ sp_path_temp = os.path.join( OUT_PATH, "{oi}", "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" - "en{lo}_{hi}keV-spectrum.fits", + "enALL-spectrum.fits", ) back_sp_path_temp = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" - "en{lo}_{hi}keV-back-spectrum.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "enALL-back-spectrum.fits", ) ``` @@ -1567,8 +1533,6 @@ for oi, dcs in rel_dataclasses.items(): cur_spec = sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), @@ -1576,8 +1540,6 @@ for oi, dcs in rel_dataclasses.items(): cur_bspec = back_sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), ) @@ -1644,8 +1606,6 @@ for oi, dcs in rel_dataclasses.items(): cur_spec = sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), @@ -1653,8 +1613,6 @@ for oi, dcs in rel_dataclasses.items(): cur_grp_spec = grp_sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, gt=spec_group_type, gs=spec_group_scale, ra=src_coord.ra.value.round(6), @@ -1681,8 +1639,6 @@ arg_combs = [ sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), @@ -1749,8 +1705,6 @@ arg_combs = [ sp_path_temp.format( oi=oi, xdc=cur_dc, - lo=spec_lo_en.value, - hi=spec_hi_en.value, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), @@ -1767,7 +1721,7 @@ with mp.Pool(NUM_CORES) as p: ``` ```{code-cell} python -arf_path_temp = sp_path_temp.replace("-spectrum", ".arf") +arf_path_temp = sp_path_temp.replace("-spectrum.fits", ".arf") ``` ```{warning} @@ -1777,6 +1731,32 @@ of this step can be on the order of hours. We have to do ***FINISH*** ## 5. Fitting spectral models to XRISM-Xtend spectra +### Configuring PyXspec + +Now we configure some behaviors of XSPEC/pyXspec: +- The ```chatter``` parameter is set to zero to reduce printed output during fitting (note that some XSPEC messages are still shown). +- We inform XSPEC of the number of cores we have available, as some XSPEC methods can be paralleled. +- We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). + +```{code-cell} python +xs.Xset.chatter = 0 + +# XSPEC parallelisation settings +xs.Xset.parallel.leven = NUM_CORES +xs.Xset.parallel.error = NUM_CORES +xs.Xset.parallel.steppar = NUM_CORES + +# Other xspec settings +xs.Plot.area = True +xs.Plot.xAxis = "keV" +xs.Plot.background = True +xs.Fit.statMethod = "cstat" +xs.Fit.query = "no" +xs.Fit.nIterations = 500 +``` + +### Reading XRISM-Xtend spectra into pyXspec + ## About this notebook Author: David J Turner, HEASARC Staff Scientist. From 7a4ff42190d3abddf38155c8570e9f8c69151545 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 2 Dec 2025 13:19:33 -0500 Subject: [PATCH 128/221] Have to include '__row' in the columns we manually specify should be returned by Heasarc.query_region in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index d81bdacb..69768efc 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -766,7 +766,8 @@ src_coord ```{code-cell} python col_str = ( - "obsid,name,ra,dec,time,exposure,status,public_date,xtd_dataclas1,xtd_dataclas2" + "__row,obsid,name,ra,dec,time,exposure,status,public_date," + "xtd_dataclas1,xtd_dataclas2" ) all_xrism_obs = Heasarc.query_region(src_coord, catalog_name, columns=col_str) all_xrism_obs From 4cea26e661a7f6054347e2d175c9bbd8a82bac19 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 2 Dec 2025 14:38:40 -0500 Subject: [PATCH 129/221] Hadn't passed 'radec_src_reg_path' to the gen_xrism_xtend_arf function when called in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 69768efc..7ceffd31 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1252,6 +1252,9 @@ expmap_rad_delta = Quantity(20, "arcmin") expmap_phi_bins = 1 ``` +```{code-cell} python +expmap_bin_factors = [4] +``` gen_xrism_xtend_expmap(chosen_demo_obsid, "32000010", os.path.join(OUT_PATH, chosen_demo_obsid), @@ -1278,7 +1281,7 @@ arg_combs = [ ] for oi, dcs in rel_dataclasses.items() for dc in dcs - for cur_bf in bin_factors + for cur_bf in expmap_bin_factors ] with mp.Pool(NUM_CORES) as p: @@ -1711,6 +1714,7 @@ arg_combs = [ rad=src_reg_rad.to("deg").value.round(4), ), rmf_path_temp.format(oi=oi, xdc=dc), + radec_src_reg_path, arf_rt_num_photons, ] for oi, dcs in rel_dataclasses.items() From 2409a16cbe6a2c4e10b59dcfae74ea72958ffc1a Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 3 Dec 2025 13:56:10 -0500 Subject: [PATCH 130/221] Updated the 'gen_xrism_xtend_image' wrapper function in getting-started-xrism-xtend.md, so that the user no longer has to pass the ObsID and dataclass information (they are extracted from event lists). For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 189 +++++++++++++----- 1 file changed, 142 insertions(+), 47 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 7ceffd31..d22350cb 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-02' +date: '2025-12-03' file_format: mystnb jupytext: text_representation: @@ -30,26 +30,50 @@ title: Getting started with XRISM-Xtend By the end of this tutorial, you will be able to: -- Find... -- +- Identify and download XRISM observations of an interesting source. +- Prepare the XRISM-Xtend data for analysis. +- Generate XRISM-Xtend data products: + - Images + - Exposure maps + - Light curves + - Spectra and supporting files +- Perform a simple spectral analysis of a XRISM-Xtend spectrum ## Introduction -XRISM is... +The 'X-Ray Imaging and Spectroscopy Mission' (**XRISM**) is an X-ray telescope designed for high-energy-resolution +spectroscopic observations of astrophysical sources, as well as wide-field X-ray imaging. +XRISM, launched in 2023, is the result of a JAXA-NASA partnership (with involvement from ESA), and serves as nearly like-for-like replacement +of the **Hitomi** telescope, which was lost shortly after its launch in 2016. + +There are two main XRISM instruments, **Xtend** and **Resolve**. In this tutorial, we will focus on **Xtend**, which is +a wide-field CCD spectro-imaging instrument similar in concept to instruments included on many other X-ray +telescopes (XMM's EPIC detectors, Chandra's ACIS, Swift's XRT, etc.) The other instrument, **Resolve**, has its own +dedicated demonstration notebook. + +Our goal with this 'getting started' notebook is to give you the skills required to prepare XRISM-Xtend +observations for scientific use and to generate data products tailored to your science goals. It can also serve as a +template notebook to build your own analyses on top of. + +Other tutorials in this series will explore how to perform more complicated generation and analysis +of XRISM-Xtend data, but here we will focus on making single aperture light curves and spectra for an +object that can be semi-reasonably treated as a 'point' source. The supernova-remnant LMC N132D. ### Inputs -- +- The name of the source of interest - in this case *LMC N132D* ### Outputs -- +- Processed, cleaned, and calibrated XRISM-Xtend event lists. +- XRISM-Xtend images, exposure maps, light curves, spectra, and supporting files. +- Simple region files that define where light curves and spectra are extracted from. ### Runtime -As of 1st December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 3rd December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -73,12 +97,6 @@ from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc from regions import CircleSkyRegion from xga.products import Image - -# from typing import Tuple, Union -# from warnings import warn - -# from matplotlib.ticker import FuncFormatter -# from tqdm import tqdm ``` ## Global Setup @@ -130,14 +148,14 @@ def process_xrism_xtend( :rtype: Tuple[str, hsp.core.HSPResult, bool] """ - # Makes sure the specified output directory exists. - temp_outdir = os.path.join(out_dir, "temp") - os.makedirs(temp_outdir, exist_ok=True) + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xtdpipeline_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). - with contextlib.chdir(out_dir), hsp.utils.local_pfiles_context(): + with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): # The processing/preparation stage of any X-ray telescope's data is the most # likely to go wrong, and we use a Python try-except as an automated way to @@ -149,7 +167,7 @@ def process_xrism_xtend( steminputs=file_stem, stemoutputs=file_stem, indir=evt_dir, - outdir=temp_outdir, + outdir=".", attitude=attitude, orbit=orbit, obsgti=obs_gti, @@ -164,18 +182,18 @@ def process_xrism_xtend( task_success = False out = str(err) - # Moves files from the temporary output directory into the - # final output directory - if os.path.exists(temp_outdir) and len(os.listdir(temp_outdir)) != 0: - for f in os.listdir(temp_outdir): - os.rename(os.path.join(temp_outdir, f), os.path.join(out_dir, f)) + # Moves files from the temporary output directory into the + # final output directory + if os.path.exists(temp_work_dir) and len(os.listdir(temp_work_dir)) != 0: + for f in os.listdir(temp_work_dir): + os.rename(os.path.join(temp_work_dir, f), os.path.join(out_dir, f)) + # Make sure to remove the temporary directory + rmtree(temp_work_dir) return cur_obs_id, out, task_success def gen_xrism_xtend_image( - cur_obs_id: str, - cur_xtend_data_class: str, event_file: str, out_dir: str, lo_en: Quantity, @@ -183,24 +201,32 @@ def gen_xrism_xtend_image( im_bin: int = 1, ): """ + This function wraps the HEASoft 'extractor' tool and is used to spatially bin + XRISM-Xtend event lists into images. The HEASoftPy interface to 'extractor' is used. - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend image. - :param str cur_xtend_data_class: - :param str event_file: + Both the energy band and the image binning factor, which controls how + many 'pixels' in the native SKY X-Y coordinate of the event list are binned into + a single image pixel, can be specified. + + The ObsID and dataclass are extracted from the header of the passed event list file. + + :param str event_file: Path to the event list (usually cleaned, but not + necessarily) we wish to generate an image from. ObsID and dataclass information + will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. :param Quantity lo_en: Lower bound of the energy band within which we will generate the image. :param Quantity hi_en: Upper bound of the energy band within which we will generate the image. - :return: A tuple containing the processed ObsID, the log output of the - pipeline, and a boolean flag indicating success (True) or failure (False). - :rtype: Tuple[str, hsp.core.HSPResult, bool] + :param int im_bin: Number of XRISM-Xtend SKY X-Y pixels to bin into a single image + pixel. """ - if cur_xtend_data_class[0] != "3": - raise ValueError( - f"The first digit of the Xtend data class ({cur_xtend_data_class}) " - "must be 3 for in-flight data." - ) + + # We can extract the ObsID and data class directly from the header of the event + # list - it is safer than having them be passed to this function separately. + with fits.open(event_file) as read_evto: + cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] + cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] # Make sure the lower and upper energy limits make sense if lo_en > hi_en: @@ -239,7 +265,7 @@ def gen_xrism_xtend_image( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.extractor( filename=evt_file_chan_sel, - imgfile=os.path.join("..", im_out), + imgfile=im_out, noprompt=True, clobber=True, binf=im_bin, @@ -247,6 +273,10 @@ def gen_xrism_xtend_image( ycolf="Y", ) + # Move the output image file to the proper output directory from + # the temporary working directory + os.rename(os.path.join(temp_work_dir, im_out), os.path.join(out_dir, im_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1216,8 +1246,6 @@ We use... ```{code-cell} python arg_combs = [ [ - oi, - dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), *cur_bnds, @@ -1255,13 +1283,6 @@ expmap_phi_bins = 1 ```{code-cell} python expmap_bin_factors = [4] ``` -gen_xrism_xtend_expmap(chosen_demo_obsid, - "32000010", - os.path.join(OUT_PATH, chosen_demo_obsid), - evt_path_temp.format(oi=chosen_demo_obsid, xdc="32000010", sc=0), - ehk_path_temp.format(oi=chosen_demo_obsid), - badpix_path_temp.format(oi=chosen_demo_obsid, xdc="32000010", sc=0), - im_bin=4, out_map_type='EFFICIENCY') ***WHAT ABOUT THIS GTI?? - xa000128000xtd_mode.gti*** @@ -1736,6 +1757,12 @@ of this step can be on the order of hours. We have to do ***FINISH*** ## 5. Fitting spectral models to XRISM-Xtend spectra +Finally, to show off the XRISM-Xtend products we just generated, we will perform +a very simple model fit to one of our spectra. + +Our demonstration of spectral model fitting to an XRISM-Xtend spectrum will be +performed using the [PyXspec](https://heasarc.gsfc.nasa.gov/docs/software/xspec/python/html/index.html) package. + ### Configuring PyXspec Now we configure some behaviors of XSPEC/pyXspec: @@ -1762,13 +1789,81 @@ xs.Fit.nIterations = 500 ### Reading XRISM-Xtend spectra into pyXspec +```{code-cell} python +chosen_demo_spec_obsid = "000128000" +chosen_demo_spec_dataclass = "31100010" +``` + +```{code-cell} python +# In case this cell is re-run, clear all previously loaded spectra +xs.AllData.clear() + +# Set up the paths to grouped source spectrum, ungrouped background +# spectrum, RMF, and ARF files +cur_spec = grp_sp_path_temp.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + gt=spec_group_type, + gs=spec_group_scale, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), +) + +cur_bspec = back_sp_path_temp.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), +) + +cur_rmf = rmf_path_temp.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, +) + +cur_arf = arf_path_temp.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), +) + +# Load the chosen spectrum (and all its supporting files) into pyXspec +xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) +``` + +### Restricting the spectral channels used for fitting + +```{code-cell} python +xs_spec.ignore("**-0.5 12.0-**") + +# Ignore any channels that have been marked as 'bad' +# This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra +# have been declared +xs.AllData.ignore("bad") +``` + +### Setting up a spectral model + +```{code-cell} python +xs.Model("tbabs*(powerlaw+apec+bbody)") +``` + +### Fitting our pyXspec model to the XRISM-Xtend spectrum + +```{code-cell} python +xs.Fit.perform() +``` + ## About this notebook Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-01 +Updated On: 2025-12-03 +++ From 8157dfc4a8f424cf95fb3e688e5504a23c6f9a7b Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 3 Dec 2025 14:23:50 -0500 Subject: [PATCH 131/221] Altered the 'gen_xrism_xtend_expmap' function (in getting-started-xrism-xtend.md) so that it takes an event list argument in lieu of manually specifying the ObsID and dataclass of the exposure map to generate. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 161 ++++++++++-------- 1 file changed, 91 insertions(+), 70 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index d22350cb..0caaeccd 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -84,6 +84,7 @@ import multiprocessing as mp import os from random import randint from shutil import rmtree +from typing import Union import heasoftpy as hsp import matplotlib.pyplot as plt @@ -105,21 +106,22 @@ from xga.products import Image ```{code-cell} python --- -tags: [hide-input] +tags: [hide - input] jupyter: - source_hidden: true +source_hidden: true --- + def process_xrism_xtend( - cur_obs_id: str, - out_dir: str, - evt_dir: str, - attitude: str, - orbit: str, - obs_gti: str, - mkf_filter: str, - file_stem: str, - extended_housekeeping: str, - xtend_housekeeping: str, + cur_obs_id: str, + out_dir: str, + evt_dir: str, + attitude: str, + orbit: str, + obs_gti: str, + mkf_filter: str, + file_stem: str, + extended_housekeeping: str, + xtend_housekeeping: str, ): """ A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process @@ -194,11 +196,11 @@ def process_xrism_xtend( def gen_xrism_xtend_image( - event_file: str, - out_dir: str, - lo_en: Quantity, - hi_en: Quantity, - im_bin: int = 1, + event_file: str, + out_dir: str, + lo_en: Quantity, + hi_en: Quantity, + im_bin: int = 1, ): """ This function wraps the HEASoft 'extractor' tool and is used to spatially bin @@ -284,34 +286,46 @@ def gen_xrism_xtend_image( def gen_xrism_xtend_expmap( - cur_obs_id: str, - cur_xtend_data_class: str, - out_dir: str, - gti_file: str, - extend_hk_file: str, - bad_pix_file: str, - pix_gti_file: str = "NONE", - im_bin: int = 1, - radial_delta: float = Quantity(20.0, "arcmin"), - num_phi_bin: int = 1, + event_file: str, + out_dir: str, + gti_file: str, + extend_hk_file: str, + bad_pix_file: str, + pix_gti_file: str = "NONE", + im_bin: int = 1, + radial_delta: Union[float, Quantity] = Quantity(20.0, "arcmin"), + num_phi_bin: int = 1, ): """ + Function that wraps the HEASoftPy interface to the XRISM-Xtend 'xaexpmap' + task, which is used to generate exposure maps for XRISM-Xtend observations. - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend exposure map. - :param str cur_xtend_data_class: - :param str event_file: + :param str event_file: Event list of the observation + dataclass you wish to + generate an exposure map for. No event data are used in the creation of the + event list, but some information in the file headers is useful. :param str out_dir: The directory where output files should be written. - :return: A tuple containing the processed ObsID, the log output of the - pipeline, and a boolean flag indicating success (True) or failure (False). - :rtype: Tuple[str, hsp.core.HSPResult, bool] + :param str gti_file: File defining the good-time-intervals of the observation + and observation dataclass for which we are generating an exposure map (often + the event list itself is passed). + :param str extend_hk_file: + :param str bad_pix_file: + :param str pix_gti_file: Optional file defining the good-time-intervals of + individual XRISM-Xtend pixels. If not provided, the default value of 'NONE' is + passed to 'xaexpmap'. + :param im_bin: Number of XRISM-Xtend SKY X-Y pixels to bin into a single exposure + map pixel. Defaults to 1, and any other value will also result in an + 'im_bin=1' being generated. + :param float/Quantity radial_delta: Radial increment for the annular grid for + which the attitude histogram will be calculated. + :param int num_phi_bin: Number of azimuth (phi) bins in the first annular region + over which attitude histogram bins will be calculated """ - # Validity check on the passed data class - if cur_xtend_data_class[0] != "3": - raise ValueError( - f"The first digit of the Xtend data class ({cur_xtend_data_class}) " - "must be 3 for in-flight data." - ) + # We can extract the ObsID and data class directly from the header of the event + # list - it is safer than having them be passed to this function separately. + with fits.open(event_file) as read_evto: + cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] + cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] # Make sure the radial_delta value is in arcminutes/is convertible to arcmins # Also will assume that radial_delta is in arcmin if it is not a Quantity object @@ -366,7 +380,7 @@ def gen_xrism_xtend_expmap( pixgtifile=pix_gti_file, delta=radial_delta, numphi=num_phi_bin, - outfile=os.path.join("..", ex_out), + outfile=ex_out, badimgfile=bad_pix_file, outmaptype=out_map_type, noprompt=True, @@ -376,14 +390,22 @@ def gen_xrism_xtend_expmap( # If the user wants a spatially binned exposure map, we run the fimgbin task if im_bin != 1: rebin_out = hsp.fimgbin( - infile=os.path.join("..", ex_out), - outfile=os.path.join("..", binned_ex_out), + infile=ex_out, + outfile=binned_ex_out, xbinsize=im_bin, noprompt=True, clobber=True, ) out = [out, rebin_out] + # Move the im_bin=1 exposure map (guaranteed to have been generated) up to the + # final output directory + os.rename(os.path.join(temp_work_dir, ex_out), os.path.join(out_dir, ex_out)) + # Then do the same for the spatially binned exposure map, if it was requested + if im_bin != 1: + os.rename(os.path.join(temp_work_dir, binned_ex_out), + os.path.join(out_dir, binned_ex_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -391,16 +413,16 @@ def gen_xrism_xtend_expmap( def gen_xrism_xtend_lightcurve( - cur_obs_id: str, - cur_xtend_data_class: str, - event_file: str, - out_dir: str, - src_reg_file: str, - back_reg_file: str, - lo_en: Quantity = Quantity(0.6, "keV"), - hi_en: Quantity = Quantity(13, "keV"), - time_bin_size: Quantity = Quantity(200, "s"), - lc_bin_thresh: float = 0.0, + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + src_reg_file: str, + back_reg_file: str, + lo_en: Quantity = Quantity(0.6, "keV"), + hi_en: Quantity = Quantity(13, "keV"), + time_bin_size: Quantity = Quantity(200, "s"), + lc_bin_thresh: float = 0.0, ): """ @@ -509,14 +531,14 @@ def gen_xrism_xtend_lightcurve( def gen_xrism_xtend_spectrum( - cur_obs_id: str, - cur_xtend_data_class: str, - event_file: str, - out_dir: str, - rel_src_coord: SkyCoord, - rel_src_radius: Quantity, - src_reg_file: str, - back_reg_file: str, + cur_obs_id: str, + cur_xtend_data_class: str, + event_file: str, + out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, + src_reg_file: str, + back_reg_file: str, ): """ IMPLICITLY ASSUMES THE REGION IS A CIRCLE @@ -625,13 +647,13 @@ def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): def gen_xrism_xtend_arf( - cur_obs_id: str, - out_dir: str, - expmap_file: str, - spec_file: str, - rmf_file: str, - src_radec_reg_file: str, - num_photons: int, + cur_obs_id: str, + out_dir: str, + expmap_file: str, + spec_file: str, + rmf_file: str, + src_radec_reg_file: str, + num_photons: int, ): """ IMPLICITLY ASSUMES THAT WE'RE GENERATING AN ARF FOR A 'POINT SOURCE' @@ -680,7 +702,7 @@ def gen_xrism_xtend_arf( noprompt=True, clobber=True, scatterfile="https://heasarc.gsfc.nasa.gov/FTP/caldb/data/xrism/xtend/" - "bcf/mirror/xa_xtd_scatter_20190101v001.fits.gz[1]", + "bcf/mirror/xa_xtd_scatter_20190101v001.fits.gz[1]", ) # TODO REMOVE THE DIRECT SETTING OF SCATTERFILE ONCE CALDB IS FIXED @@ -1289,8 +1311,7 @@ expmap_bin_factors = [4] ```{code-cell} python arg_combs = [ [ - oi, - dc, + evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), evt_path_temp.format(oi=oi, xdc=dc, sc=0), ehk_path_temp.format(oi=oi), From 3f01780a0fded3c4a70306ab7bdc05de5a6b80d3 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 3 Dec 2025 14:44:52 -0500 Subject: [PATCH 132/221] Altered the time_bin_size gen_xrism_xtend_lightcurve function to not require ObsID and dataclass arguments. Also added a docstring. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 40 ++++++++++--------- 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0caaeccd..b99329d5 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -413,8 +413,6 @@ def gen_xrism_xtend_expmap( def gen_xrism_xtend_lightcurve( - cur_obs_id: str, - cur_xtend_data_class: str, event_file: str, out_dir: str, src_reg_file: str, @@ -425,26 +423,30 @@ def gen_xrism_xtend_lightcurve( lc_bin_thresh: float = 0.0, ): """ + Function that wraps the HEASoftPy interface to the HEASoft extractor tool, set + up to generate light curves from XRISM-Xtend observations. The function will + generate a light curve for the source region and a background light curve for + the background region. - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend light curve. - :param str cur_xtend_data_class: - :param str event_file: + :param str event_file: Path to the event list (usually cleaned, but not + necessarily) we wish to generate a XRISM-Xtend light curve from. ObsID and + dataclass information will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. :param Quantity lo_en: Lower bound of the energy band within which we will generate the light curve. :param Quantity hi_en: Upper bound of the energy band within which we will generate the light curve. - :return: A tuple containing the processed ObsID, the log output of the - pipeline, and a boolean flag indicating success (True) or failure (False). - :rtype: Tuple[str, hsp.core.HSPResult, bool] + :param Quantity time_bin_size: The size of the time bins used to generate the + light curve. + :param float lc_bin_thresh: When constructing a light curve, any bins whose + exposure is less than lc_bin_thresh*time_bin_size are ignored. """ - # Validity check on the passed data class - if cur_xtend_data_class[0] != "3": - raise ValueError( - f"The first digit of the Xtend data class ({cur_xtend_data_class}) " - "must be 3 for in-flight data." - ) + # We can extract the ObsID and data class directly from the header of the event + # list - it is safer than having them be passed to this function separately. + with fits.open(event_file) as read_evto: + cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] + cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] # Check the units of the passed time bin size - also if the passed value is # a float or integer, we'll assume it is in seconds @@ -501,7 +503,7 @@ def gen_xrism_xtend_lightcurve( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): src_out = hsp.extractor( filename=evt_file_chan_sel, - fitsbinlc=os.path.join("..", lc_out), + fitsbinlc=lc_out, binlc=time_bin_size, lcthresh=lc_bin_thresh, regionfile=src_reg_file, @@ -514,7 +516,7 @@ def gen_xrism_xtend_lightcurve( # Now for the background light curve back_out = hsp.extractor( filename=evt_file_chan_sel, - fitsbinlc=os.path.join("..", lc_back_out), + fitsbinlc=lc_back_out, binlc=time_bin_size, lcthresh=lc_bin_thresh, regionfile=back_reg_file, @@ -524,6 +526,10 @@ def gen_xrism_xtend_lightcurve( clobber=True, ) + # Move the light curves up from the temporary directory + os.rename(os.path.join(temp_work_dir, lc_out), os.path.join(out_dir, lc_out)) + os.rename(os.path.join(temp_work_dir, lc_back_out), os.path.join(out_dir, lc_back_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1487,8 +1493,6 @@ xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") ```{code-cell} python arg_combs = [ [ - oi, - dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), From 41c956b3781263c9c1aabe187b9bbd39974ec3bc Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 3 Dec 2025 17:00:21 -0500 Subject: [PATCH 133/221] Moved the xspec import down the notebook --- .../xrism/getting-started-xrism-xtend.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b99329d5..1c1ff42e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -89,7 +89,6 @@ from typing import Union import heasoftpy as hsp import matplotlib.pyplot as plt import numpy as np -import xspec as xs from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.table import Table @@ -1796,6 +1795,8 @@ Now we configure some behaviors of XSPEC/pyXspec: - We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). ```{code-cell} python +import xspec as xs + xs.Xset.chatter = 0 # XSPEC parallelisation settings From 76e1341447a3b7dc1950d1a00f011a8c49dd76eb Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 4 Dec 2025 12:10:06 -0500 Subject: [PATCH 134/221] Fixed an issue with an incorrect dataclass being passed to RMF generation (perhaps will alter the function to take an event list like some of the other wrapper functions - could be safer). In getting-started-xrism-xtend.md, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 1c1ff42e..dd7cf344 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1687,7 +1687,7 @@ arg_combs = [ oi, sp_path_temp.format( oi=oi, - xdc=cur_dc, + xdc=dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), From 562a5d272aac4492cabdc79e3049cca48dcdd69a Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 10:30:13 -0500 Subject: [PATCH 135/221] Explicitly defined the name of the table in the event list that contains GTI information when generating images, light curves, and spectra using extractor. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 102 ++++++++++++++---- 1 file changed, 81 insertions(+), 21 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index dd7cf344..5d3e8896 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-03' +date: '2025-12-05' file_format: mystnb jupytext: text_representation: @@ -58,8 +58,9 @@ template notebook to build your own analyses on top of. Other tutorials in this series will explore how to perform more complicated generation and analysis of XRISM-Xtend data, but here we will focus on making single aperture light curves and spectra for an -object that can be semi-reasonably treated as a 'point' source. The supernova-remnant LMC N132D. +object that can be semi-reasonably treated as a 'point' source; the supernova-remnant LMC N132D. +We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstration. ### Inputs @@ -73,7 +74,7 @@ object that can be semi-reasonably treated as a 'point' source. The supernova-re ### Runtime -As of 3rd December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 5th December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -272,6 +273,7 @@ def gen_xrism_xtend_image( binf=im_bin, xcolf="X", ycolf="Y", + gti="GTI", ) # Move the output image file to the proper output directory from @@ -508,6 +510,7 @@ def gen_xrism_xtend_lightcurve( regionfile=src_reg_file, xcolf="X", ycolf="Y", + gti="GTI", noprompt=True, clobber=True, ) @@ -521,6 +524,7 @@ def gen_xrism_xtend_lightcurve( regionfile=back_reg_file, xcolf="X", ycolf="Y", + gti="GTI", noprompt=True, clobber=True, ) @@ -598,6 +602,7 @@ def gen_xrism_xtend_spectrum( xcolf="X", ycolf="Y", ecol="PI", + gti="GTI", noprompt=True, clobber=True, ) @@ -610,6 +615,7 @@ def gen_xrism_xtend_spectrum( xcolf="X", ycolf="Y", ecol="PI", + gti="GTI", noprompt=True, clobber=True, ) @@ -706,12 +712,8 @@ def gen_xrism_xtend_arf( regmode="RADEC", noprompt=True, clobber=True, - scatterfile="https://heasarc.gsfc.nasa.gov/FTP/caldb/data/xrism/xtend/" - "bcf/mirror/xa_xtd_scatter_20190101v001.fits.gz[1]", ) - # TODO REMOVE THE DIRECT SETTING OF SCATTERFILE ONCE CALDB IS FIXED - # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1075,6 +1077,8 @@ evt_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") ``` +### Good-time-intervals + ### Identifying problem pixels ```{code-cell} python @@ -1083,9 +1087,18 @@ badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") ## 3. Generating new XRISM-Xtend images and exposure maps +The XRISM-Xtend data have now been prepared for scientific use, with the most important +output being the cleaned event list(s); remember that one observation can produce +**two** cleaned event lists if Xtend was operating in a windowed or burst mode. + +We will now demonstrate how to generate new XRISM-Xtend data products tailored to your +scientific needs. Images and exposure maps can be generated for the entire +field-of-view (FoV), rather than having to focus on a particular source, so we will +start with them. + ### Converting energy bounds to channel bounds -The data products we generate in this section can all benefit from selecting events +The data products we generate in this section (and the next) can all benefit from selecting events from within a specific energy range. This might be because your source of interest only emits in a narrow energy range, and you don't care about the rest, or because different mechanisms emit at different energies, and you wish to separate them. @@ -1125,8 +1138,8 @@ We will be creating new RMFs as part of the generation of XRISM-Xtend spectra in next section. For our current purpose, however, it is acceptable to use the RMFs that were included in the XRISM-Xtend archive we downloaded earlier. -The archived RMFs are generated for the entire Xtend field-of-view (FoV), rather than for the CCDs -our particular target fall on, but practically speaking, that doesn't make a significant +The archived RMFs are generated for the entire Xtend FoV, rather than for the CCDs +our particular target falls on, but practically speaking, that doesn't make a significant difference. Using observation 000128000 as an example, we determine the path to the relevant @@ -1171,7 +1184,7 @@ e_bounds[90:110] ``` We can use this file to visualize the basic linear mapping between energy and -channel - it will not be the most interesting figure you've ever seen: +channel - *it will be the most boring figure you've ever seen*: ```{code-cell} python --- @@ -1179,13 +1192,18 @@ tags: [hide-input] jupyter: source_hidden: true --- + +# Set up the figure plt.figure(figsize=(5.5, 5.5)) +# Configuring the axis ticks plt.minorticks_on() plt.tick_params(which="both", direction="in", top=True, right=True) +# Calculate the mid-point of each energy bin mid_ens = (e_bounds["E_MIN"] + e_bounds["E_MAX"]) / 2 +# Plot the relationship between channel and the energy bin mid-points plt.plot(e_bounds["CHANNEL"], mid_ens, color="navy", alpha=0.9, label="XRISM-Xtend") plt.xlim(0) @@ -1204,10 +1222,13 @@ Finally, we can validate our assumed relationship between energy and channel by calculating the mean change in minimum energy between adjacent channels: ```{code-cell} python -# -rmf_ev_per_chan = Quantity(np.diff(e_bounds["E_MIN"].data).mean(), "keV/chan").to( - "eV/chan" -) +# Calculates the energy change from one to channel to the next, then finds the +# mean value of those energy changes +mean_en_diffs = np.diff(e_bounds["E_MIN"].data).mean() + +# Set up the result in an astropy quantity and convert to eV-per-channel for +# easier comparison to the assumed relationship +rmf_ev_per_chan = Quantity(mean_en_diffs, "keV/chan").to("eV/chan") rmf_ev_per_chan ``` @@ -1219,8 +1240,26 @@ rmf_ev_per_chan / XTD_EV_PER_CHAN ### New XRISM-Xtend images +We've established that we understand XRISM-Xtend's relationship between energy and +channel. Now we can use that relationship to choose the energy bounds we generate +data products within and convert them to the channel values required by XRISM HEASoft +tasks. + +We recommend that you generate images first, as examining them is a good way to spot +any problems or unusual features of the prepared and cleaned observations. + #### Image energy bounds +We are going to generate images within the following energy bounds: +- 0.6-2.0 keV +- 2.0-10.0 keV +- ***0.4-2.0 keV*** [not recommended] +- ***0.4-10.0 keV*** [not recommended] + +The bands that have a lower bound of ***0.4 keV*** are ***not recommended***, as there +are issues with XRISM-Xtend data below *0.6 keV*. We are generating them to +demonstrate those issues. + ```{code-cell} python # Defining the energy bounds we want images within xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV") @@ -1266,7 +1305,28 @@ bin_factors = [1, 4] #### Running image generation -We use... +There is no HEASoft tool specifically for generating XRISM-Xtend images, but there is a +generalized HEASoft image (and other data products) generation task that we can use. + +If you have previously generated images, light curves, or spectra from HEASARC-hosted +X-ray data on the command line, you may well have come across `XSELECT`; a HEASoft +tool for interactively generating data products from event lists. + +When creating data products, `XSELECT` calls the HEASoft `extractor` task, which we +will now use to demonstrate the creation of XRISM-Xtend images. + +As with all uses of HEASoft tasks in this notebook, our call to `extractor` will be +through the HEASoftPy Python interface - specifically the `hsp.extractor` function. + +We have implemented a wrapper to this function in the 'Global Setup: Functions' section +of this notebook, primarily so that we can easily multiprocess the generation of images +in different energy bands, binning factors, observations, and dataclasses. + +Image generation is not a particularly computationally intensive task, but if you are +addressing a large number of observations (or making many images per observation), it +is a good idea to run them in parallel! + + ***NEED TO APPLY GTIS TO IMAGE GENERATION AS WELL*** @@ -1400,7 +1460,7 @@ src_reg_rad = Quantity(2, "arcmin") src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) # Write the source region to a region file -src_reg.write(radec_src_reg_path, format="ds9") +src_reg.write(radec_src_reg_path, format="ds9", overwrite=True) ``` We do the same to define a region from which to extract a background spectrum: @@ -1419,7 +1479,7 @@ back_reg_rad = Quantity(3, "arcmin") back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) # Once again writing the region to a region file as well -back_reg.write(radec_back_reg_path, format="ds9") +back_reg.write(radec_back_reg_path, format="ds9", overwrite=True) ``` #### Visualizing the source and background extraction regions on XRISM-Xtend images @@ -1462,10 +1522,10 @@ obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_back.re for oi in rel_obsids: src_reg.to_pixel(oi_skypix_wcs[oi]).write( - obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9" + obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9", overwrite=True ) back_reg.to_pixel(oi_skypix_wcs[oi]).write( - obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9" + obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9", overwrite=True ) ``` @@ -1753,7 +1813,7 @@ arg_combs = [ ), sp_path_temp.format( oi=oi, - xdc=cur_dc, + xdc=dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), From e3880b18139ced24bb4332da57e79e33378b89de Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 10:42:39 -0500 Subject: [PATCH 136/221] Moved the definition of most output path template variables to the 'Constants' section of the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 143 +++++++++++------- 1 file changed, 85 insertions(+), 58 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 5d3e8896..079734e1 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -738,6 +738,74 @@ TASK_CHATTER = 3 # The approximate linear relationship between Xtend PI and event energy XTD_EV_PER_CHAN = (1 / Quantity(166.7, "chan/keV")).to("eV/chan") + + +# ------------- Set up output file path templates -------------- +# --------- IMAGES --------- +IM_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-imbinfactor{ibf}-en{lo}_{hi}keV-image.fits", +) +# -------------------------- + + +# -------- EXPMAPS --------- +EX_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-attraddelta{rd}arcmin-" + "attphibin{npb}-imbinfactor{ibf}-enALL-expmap.fits", +) +# -------------------------- + + +# ------ LIGHTCURVES ------- +LC_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" + "-lightcurve.fits", +) + +BACK_LC_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" + "-back-lightcurve.fits", +) +# -------------------------- + + +# -------- SPECTRA --------- +SP_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" + "enALL-spectrum.fits", +) + +BACK_SP_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "enALL-back-spectrum.fits", +) +# -------------------------- + +# ----- GROUPEDSPECTRA ----- +GRP_SP_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum", "-{gt}grp{gs}-spectrum") +# -------------------------- + +# ---------- RMF ----------- +RMF_PATH_TEMP = os.path.join( + OUT_PATH, "{oi}", "xrism-xtend-obsid{oi}-dataclass{xdc}.rmf" +) +# -------------------------- + +# ---------- ARF ----------- +ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") +# -------------------------- +# -------------------------------------------------------------- ``` ### Configuration @@ -1327,9 +1395,6 @@ addressing a large number of observations (or making many images per observation is a good idea to run them in parallel! - -***NEED TO APPLY GTIS TO IMAGE GENERATION AS WELL*** - ```{code-cell} python arg_combs = [ [ @@ -1351,11 +1416,7 @@ with mp.Pool(NUM_CORES) as p: Once again we set up a template variable for output image file names: ```{code-cell} python -im_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-imbinfactor{ibf}-en{lo}_{hi}keV-image.fits", -) + ``` ### New XRISM-Xtend exposure maps @@ -1398,12 +1459,7 @@ with mp.Pool(NUM_CORES) as p: Set up a template variable for output exposure map file names: ```{code-cell} python -ex_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-attraddelta{rd}arcmin-" - "attphibin{npb}-imbinfactor{ibf}-enALL-expmap.fits", -) + ``` ## 4. Generating new XRISM-Xtend spectra and light curves @@ -1492,7 +1548,7 @@ chos_im_en = xtd_im_en_bounds[0].to("keV") oi_skypix_wcs = {} for oi, cur_dcs in rel_dataclasses.items(): for dc in cur_dcs: - cur_im_path = im_path_temp.format( + cur_im_path = IM_PATH_TEMP.format( oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value ) cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) @@ -1547,8 +1603,6 @@ lc_time_bin = Quantity(200, "s") xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") ``` -***NEEEEEEEED GTI*** - ```{code-cell} python arg_combs = [ [ @@ -1571,19 +1625,7 @@ with mp.Pool(NUM_CORES) as p: Create template variables for source and background light curves: ```{code-cell} python -lc_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" - "-lightcurve.fits", -) -back_lc_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" - "-back-lightcurve.fits", -) ``` ### New XRISM-Xtend spectra and supporting files @@ -1617,18 +1659,7 @@ with mp.Pool(NUM_CORES) as p: Create template variables for source and background spectrum files: ```{code-cell} python -sp_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" - "enALL-spectrum.fits", -) -back_sp_path_temp = os.path.join( - OUT_PATH, - "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "enALL-back-spectrum.fits", -) ``` #### Calculating 'BACKSCAL' for new XRISM-Xtend spectra @@ -1639,14 +1670,14 @@ back_sp_path_temp = os.path.join( for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up the path to input source and background spectra - cur_spec = sp_path_temp.format( + cur_spec = SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), ) - cur_bspec = back_sp_path_temp.format( + cur_bspec = BACK_SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, ra=src_coord.ra.value.round(6), @@ -1654,7 +1685,7 @@ for oi, dcs in rel_dataclasses.items(): ) # Also need to pass an exposure map, so set up a path to that - cur_ex = ex_path_temp.format( + cur_ex = EX_PATH_TEMP.format( oi=oi, xdc=cur_dc, rd=expmap_rad_delta.to("arcmin").value, @@ -1695,7 +1726,6 @@ output grouped spectral files: spec_group_type = "min" spec_group_scale = 1 -grp_sp_path_temp = sp_path_temp.replace("-spectrum", "-{gt}grp{gs}-spectrum") ``` Now we run the grouping tool - though this time we do not parallelize the task, as @@ -1712,14 +1742,14 @@ the other processing steps in this notebook. for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up relevant paths to the input and output spectrum - cur_spec = sp_path_temp.format( + cur_spec = SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), ) - cur_grp_spec = grp_sp_path_temp.format( + cur_grp_spec = grp_SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, gt=spec_group_type, @@ -1745,7 +1775,7 @@ for oi, dcs in rel_dataclasses.items(): arg_combs = [ [ oi, - sp_path_temp.format( + SP_PATH_TEMP.format( oi=oi, xdc=dc, ra=src_coord.ra.value.round(6), @@ -1763,9 +1793,7 @@ with mp.Pool(NUM_CORES) as p: ``` ```{code-cell} python -rmf_path_temp = os.path.join( - OUT_PATH, "{oi}", "xrism-xtend-obsid{oi}-dataclass{xdc}.rmf" -) + ``` #### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation @@ -1804,21 +1832,21 @@ arg_combs = [ [ oi, os.path.join(OUT_PATH, oi), - ex_path_temp.format( + EX_PATH_TEMP.format( oi=oi, xdc=dc, rd=expmap_rad_delta.to("arcmin").value, npb=expmap_phi_bins, ibf=1, ), - sp_path_temp.format( + SP_PATH_TEMP.format( oi=oi, xdc=dc, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), ), - rmf_path_temp.format(oi=oi, xdc=dc), + RMF_PATH_TEMP.format(oi=oi, xdc=dc), radec_src_reg_path, arf_rt_num_photons, ] @@ -1831,7 +1859,6 @@ with mp.Pool(NUM_CORES) as p: ``` ```{code-cell} python -arf_path_temp = sp_path_temp.replace("-spectrum.fits", ".arf") ``` ```{warning} @@ -1886,7 +1913,7 @@ xs.AllData.clear() # Set up the paths to grouped source spectrum, ungrouped background # spectrum, RMF, and ARF files -cur_spec = grp_sp_path_temp.format( +cur_spec = grp_SP_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, gt=spec_group_type, @@ -1896,19 +1923,19 @@ cur_spec = grp_sp_path_temp.format( rad=src_reg_rad.to("deg").value.round(4), ) -cur_bspec = back_sp_path_temp.format( +cur_bspec = BACK_SP_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, ra=src_coord.ra.value.round(6), dec=src_coord.dec.value.round(6), ) -cur_rmf = rmf_path_temp.format( +cur_rmf = RMF_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, ) -cur_arf = arf_path_temp.format( +cur_arf = ARF_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, ra=src_coord.ra.value.round(6), From 5a6ac94e0bbec4dd173ba27bdf91712bd74e9ccc Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 10:47:45 -0500 Subject: [PATCH 137/221] Replaced the im_out definition in gen_xrism_xtend_image with something that uses the template path variable set up in the constants section of getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 079734e1..f462fd8c 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -250,10 +250,10 @@ def gen_xrism_xtend_image( evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" # Set up the output file name for the image we're about to generate. - im_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"imbinfactor{im_bin}-en{lo_en_val}_{hi_en_val}keV-image.fits" - ) + im_out = os.path.basename(IM_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + ibf=im_bin, lo=lo_en_val, + hi=hi_en_val) # Create a temporary working directory temp_work_dir = os.path.join( From e6ade8474ade9fbb8b2bb2259e403a343bad121e Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 10:51:17 -0500 Subject: [PATCH 138/221] Replaced the ex_out and binned_ex_out definitions in gen_xrism_xtend_expmap with something that uses the template path variable set up in the constants section of getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f462fd8c..a5d64ed3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -350,21 +350,25 @@ def gen_xrism_xtend_expmap( out_map_type = "EXPOSURE" ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" + # Create a temporary working directory + temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) + os.makedirs(temp_work_dir) + # Set up the output file name for the exposure map we're about to generate. - ex_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"attraddelta{radial_delta}arcmin-attphibin{num_phi_bin}-" - f"imbinfactor1-enALL-{ex_type}.fits" - ) + ex_out = os.path.basename(EX_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + rd=radial_delta, + npb=num_phi_bin, + ibf=1) # If the user wants to bin up the exposure map, we'll need to set up another # output file name with the bin factor set to the input value (this variable # is not used if the user does not want to bin the map) - binned_ex_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"attraddelta{radial_delta}arcmin-attphibin{num_phi_bin}-" - f"imbinfactor{im_bin}-enALL-{ex_type}.fits" - ) + binned_ex_out = os.path.basename(EX_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + rd=radial_delta, + npb=num_phi_bin, + ibf=im_vbin) # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) From 6593598281324e467e8976396cdddb423f0342e4 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 11:25:16 -0500 Subject: [PATCH 139/221] Altered the light curve and spectrum generation wrapper functions in the getting-started-xrism-xtend.md notebook, so that they use the constant template variables to name their output files. Also made light curve generation wrapper take ra,dec,radius information for the light curve file names, and moved the spectrum generation wrapper to extract ObsID and data class from the passed event file. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 112 ++++++++++++------ 1 file changed, 74 insertions(+), 38 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a5d64ed3..8d77d4a3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -420,6 +420,8 @@ def gen_xrism_xtend_expmap( def gen_xrism_xtend_lightcurve( event_file: str, out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, src_reg_file: str, back_reg_file: str, lo_en: Quantity = Quantity(0.6, "keV"), @@ -437,6 +439,14 @@ def gen_xrism_xtend_lightcurve( necessarily) we wish to generate a XRISM-Xtend light curve from. ObsID and dataclass information will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. + :param SkyCoord rel_src_coord: The source coordinate (RA, Dec) of the + source region for which we wish to generate a light curve. + :param Quantity rel_src_radius: The radius of the source region for which we wish + to generate a light curve. + :param str src_reg_file: Path to the region file defining the source region for + which we wish to generate a light curve. + :param str back_reg_file: Path to the region file defining the background region + for which we wish to generate a light curve. :param Quantity lo_en: Lower bound of the energy band within which we will generate the light curve. :param Quantity hi_en: Upper bound of the energy band within which we will @@ -453,6 +463,11 @@ def gen_xrism_xtend_lightcurve( cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] + # Get RA, Dec, and radius values in the right format + ra_val = rel_src_coord.ra.to("deg").value.round(6) + dec_val = rel_src_coord.dec.to("deg").value.round(6) + rad_val = rel_src_radius.to("deg").value.round(4) + # Check the units of the passed time bin size - also if the passed value is # a float or integer, we'll assume it is in seconds if not isinstance(time_bin_size, Quantity): @@ -487,14 +502,26 @@ def gen_xrism_xtend_lightcurve( evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" # Set up the output file name for the light curve we're about to generate. - lc_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"en{lo_en_val}_{hi_en_val}keV-expthresh{lc_bin_thresh}-tb{time_bin_size}s-" - f"lightcurve.fits" - ) - # The same file name, but with 'lightcurve' changed to 'back-lightcurve', for the - # background light curve. - lc_back_out = lc_out.replace("lightcurve", "back-lightcurve") + lc_out = os.path.basename(LC_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val, + rad=rad_val, + lo=lo_en_val, + hi=hi_en_val, + lct=lc_bin_thresh, + tb=time_bin_size) + + # The same file name, but with 'lightcurve' changed to 'back-lightcurve', and the + # radius information information removed, for the background light curve. + lc_back_out = os.path.basename(BACK_LC_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val, + lo=lo_en_val, + hi=hi_en_val, + lct=lc_bin_thresh, + tb=time_bin_size) # Create a temporary working directory temp_work_dir = os.path.join( @@ -544,8 +571,6 @@ def gen_xrism_xtend_lightcurve( def gen_xrism_xtend_spectrum( - cur_obs_id: str, - cur_xtend_data_class: str, event_file: str, out_dir: str, rel_src_coord: SkyCoord, @@ -554,40 +579,47 @@ def gen_xrism_xtend_spectrum( back_reg_file: str, ): """ - IMPLICITLY ASSUMES THE REGION IS A CIRCLE + Function that wraps the HEASoftPy interface to the HEASoft extractor tool, set + up to generate spectra from XRISM-Xtend observations. The function will + generate a spectrum for the source region and a background spectrum for + the background region. - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend spectrum. - :param str cur_xtend_data_class: - :param str event_file: + :param str event_file: Path to the event list (usually cleaned, but not + necessarily) we wish to generate a XRISM-Xtend spectrum from. ObsID and + dataclass information will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. - :return: A tuple containing the processed ObsID, the log output of the - pipeline, and a boolean flag indicating success (True) or failure (False). - :rtype: Tuple[str, hsp.core.HSPResult, bool] + :param SkyCoord rel_src_coord: The source coordinate (RA, Dec) of the + source region for which we wish to generate a light curve. + :param Quantity rel_src_radius: The radius of the source region for which we wish + to generate a light curve. + :param str src_reg_file: Path to the region file defining the source region for + which we wish to generate a light curve. + :param str back_reg_file: Path to the region file defining the background region + for which we wish to generate a light curve. """ - # Validity check on the passed data class - if cur_xtend_data_class[0] != "3": - raise ValueError( - f"The first digit of the Xtend data class ({cur_xtend_data_class}) " - "must be 3 for in-flight data." - ) + # We can extract the ObsID and data class directly from the header of the event + # list - it is safer than having them be passed to this function separately. + with fits.open(event_file) as read_evto: + cur_obs_id = read_evto["EVENTS"].header["OBS_ID"] + cur_xtend_data_class = read_evto["EVENTS"].header["DATACLAS"] # Get RA, Dec, and radius values in the right format ra_val = rel_src_coord.ra.to("deg").value.round(6) dec_val = rel_src_coord.dec.to("deg").value.round(6) rad_val = rel_src_radius.to("deg").value.round(4) - # Set up the output file name for the light curve we're about to generate. + # Set up the output file names for the source and background spectra we're + # about to generate. sp_out = ( f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-enALL-" f"spectrum.fits" ) - # The same file name, but with 'spectrum' changed to 'back-spectrum', for the - # background light curve. - sp_back_out = sp_out.replace(f"-radius{rad_val}deg", "").replace( - "spectrum", "back-spectrum" - ) + back_sp_out = os.path.basename(BACK_SP_PATH_TEMP).format(oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val) # Create a temporary working directory temp_work_dir = os.path.join( @@ -601,7 +633,7 @@ def gen_xrism_xtend_spectrum( with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): src_out = hsp.extractor( filename=event_file, - phafile=os.path.join("..", sp_out), + phafile=sp_out, regionfile=src_reg_file, xcolf="X", ycolf="Y", @@ -614,7 +646,7 @@ def gen_xrism_xtend_spectrum( # Now for the background light curve back_out = hsp.extractor( filename=event_file, - phafile=os.path.join("..", sp_back_out), + phafile=sp_back_out, regionfile=back_reg_file, xcolf="X", ycolf="Y", @@ -624,6 +656,10 @@ def gen_xrism_xtend_spectrum( clobber=True, ) + # Move the spectra up from the temporary directory + os.rename(os.path.join(temp_work_dir, sp_out), os.path.join(out_dir, sp_out)) + os.rename(os.path.join(temp_work_dir, sp_back_out), os.path.join(out_dir, sp_back_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -768,15 +804,15 @@ EX_PATH_TEMP = os.path.join( LC_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" - "-lightcurve.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" \ + "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-lightcurve.fits", ) BACK_LC_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-en{lo}_{hi}keV-expthresh{lct}-tb{tb}s" - "-back-lightcurve.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" \ + "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-back-lightcurve.fits", ) # -------------------------- @@ -792,7 +828,7 @@ SP_PATH_TEMP = os.path.join( BACK_SP_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "enALL-back-spectrum.fits", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-enALL-back-spectrum.fits", ) # -------------------------- @@ -1612,6 +1648,8 @@ arg_combs = [ [ evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), + src_coord, + src_reg_rad, obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), *cur_bnds, @@ -1643,8 +1681,6 @@ Create template variables for source and background light curves: ```{code-cell} python arg_combs = [ [ - oi, - dc, evt_path_temp.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), src_coord, From 3af12015c5bcd2e6e7adcdcf600951ecc353e77f Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 11:31:07 -0500 Subject: [PATCH 140/221] Updated the gen_xrism_xtend_rmf wrapper function so that it doesn't require an ObsID input, the ObsID is extracted from the passed spectrum. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 8d77d4a3..b8f4bd05 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -666,13 +666,20 @@ def gen_xrism_xtend_spectrum( return [src_out, back_out] -def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): +def gen_xrism_xtend_rmf(spec_file: str, out_dir: str): """ + A wrapper around the XRISM-Xtend-specific RMF generation tool implemented as + part of HEASoft (and called here through HEASoftPy). - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend RMF. + :param str spec_file: The path to the spectrum file for which to generate an RMF. :param str out_dir: The directory where output files should be written. """ + # We can extract the ObsID directly from the header of the spectrum file - it is + # safer than having the user pass it separately + with fits.open(spec_file) as read_speco: + cur_obs_id = read_speco[0].header["OBS_ID"] + # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xtdrmf_{}".format(randint(0, int(1e8)))) os.makedirs(temp_work_dir) @@ -686,11 +693,14 @@ def gen_xrism_xtend_rmf(cur_obs_id: str, spec_file: str, out_dir: str): with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.xtdrmf( infile=spec_file, - outfile=os.path.join("..", rmf_out), + outfile=rmf_out, noprompt=True, clobber=True, ) + # Move the RMF up from the temporary directory + os.rename(os.path.join(temp_work_dir, rmf_out), os.path.join(out_dir, rmf_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1814,7 +1824,6 @@ for oi, dcs in rel_dataclasses.items(): ```{code-cell} python arg_combs = [ [ - oi, SP_PATH_TEMP.format( oi=oi, xdc=dc, From 1b3a412700d4d3404d99b3afead942163d36c3bb Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 11:47:44 -0500 Subject: [PATCH 141/221] Polished the gen_xrism_xtend_arf wrapper function a little, making sure that re-runs would use previously generated matching ray-traced event files, and adding a docstring. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 49 ++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b8f4bd05..989c25dc 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -708,7 +708,6 @@ def gen_xrism_xtend_rmf(spec_file: str, out_dir: str): def gen_xrism_xtend_arf( - cur_obs_id: str, out_dir: str, expmap_file: str, spec_file: str, @@ -717,12 +716,36 @@ def gen_xrism_xtend_arf( num_photons: int, ): """ - IMPLICITLY ASSUMES THAT WE'RE GENERATING AN ARF FOR A 'POINT SOURCE' + A wrapper function for the HEASoft `xaarfgen` task, which we use to generate + ARFs for XRISM-Xtend spectra. + + IMPORTANT: The way we have set up the call to `xaarfgen` implicitly assumes that + the spectrum was generated for a POINT SOURCE. Using this setup to generate + an ARF for an extended source WOULD NOT BE VALID. + + This function can take a long time to run, primarily because of the ray-tracing + step (and the acquisition of a large CalDB file necessary for this step, if + using remote CalDB). The ray-tracing time will scale with the value + of 'num_photons', with the XRISM team estimating ~1 minute per 100,000 photons + (though note this does not include time to download the previously mentioned + CalDB file). - :param str cur_obs_id: The XRISM ObsID for which to generate an Xtend ARF. :param str out_dir: The directory where output files should be written. + :param str expmap_file: The path to the exposure map file necessary to generate + the ARF. + :param str spec_file: The path to the spectrum file for which to generate an ARF. + :param str rmf_file: The path to the RMF file necessary to generate an ARF. + :param str src_radec_reg_file: The path to the region file defining the source + region for which to generate an ARF. + :param int num_photons: The number of photons to simulate in the ray-tracing + portion of XRISM-Xtend ARF generation. """ + # We can extract the ObsID directly from the header of the spectrum file - it is + # safer than having the user pass it separately + with fits.open(spec_file) as read_speco: + cur_obs_id = read_speco[0].header["OBS_ID"] + # Spectrum files generated in this demonstration notebook contain RA-Dec # information in their file name, so we will read it out from there radec_sec = os.path.basename(spec_file).split("-radius")[0].split("-ra")[1] @@ -743,13 +766,22 @@ def gen_xrism_xtend_arf( f"enALL-raytracedevents.fits" ) + # If a ray-traced event file with the same already exists, we're just going + # to point to it with the absolute path (saves on re-running expensive + # ray tracing). + if os.path.exists(os.path.join(os.path.abspath(out_dir), ray_traced_evt_out)): + ray_traced_exists = True + ray_traced_evt_out = os.path.abspath(os.path.join(out_dir, ray_traced_evt_out)) + else: + ray_traced_exists = False + # Using dual contexts, one that moves us into the output directory for the # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): out = hsp.xaarfgen( - xrtevtfile=os.path.join("..", ray_traced_evt_out), - outfile=os.path.join("..", arf_out), + xrtevtfile=ray_traced_evt_out, + outfile=arf_out, sourcetype="POINT", numphotons=num_photons, source_ra=ra_val, @@ -764,6 +796,12 @@ def gen_xrism_xtend_arf( clobber=True, ) + # Move the ARF and ray traced event files up from the temporary directory + os.rename(os.path.join(temp_work_dir, arf_out), os.path.join(out_dir, arf_out)) + # If the ray traced file already existed, we don't need to move anything + if not ray_traced_exists: + os.rename(os.path.join(temp_work_dir, ray_traced_evt_out), os.path.join(out_dir, ray_traced_evt_out)) + # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -1879,7 +1917,6 @@ xrtraytrace: Additional updates to fix and enable remote CalDB ```{code-cell} python arg_combs = [ [ - oi, os.path.join(OUT_PATH, oi), EX_PATH_TEMP.format( oi=oi, From c9091eed3d0e0c4b410997e85f8b51dc1f1304a6 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 12:31:22 -0500 Subject: [PATCH 142/221] Added much commentary to the exposure map generation section of the getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 68 ++++++++++--------- 1 file changed, 37 insertions(+), 31 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 989c25dc..1a72ba80 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1249,7 +1249,7 @@ output being the cleaned event list(s); remember that one observation can produc We will now demonstrate how to generate new XRISM-Xtend data products tailored to your scientific needs. Images and exposure maps can be generated for the entire -field-of-view (FoV), rather than having to focus on a particular source, so we will +field-of-view (FoV; or at least the entire FoV of a particular observation mode, e.g., full window, 1/8th window, etc.), rather than having to focus on a particular source, so we will start with them. ### Converting energy bounds to channel bounds @@ -1482,7 +1482,6 @@ Image generation is not a particularly computationally intensive task, but if yo addressing a large number of observations (or making many images per observation), it is a good idea to run them in parallel! - ```{code-cell} python arg_combs = [ [ @@ -1501,26 +1500,51 @@ with mp.Pool(NUM_CORES) as p: im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` -Once again we set up a template variable for output image file names: +### New XRISM-Xtend exposure maps -```{code-cell} python +We also generate exposure maps for the entire FoV of a particular observation mode, rather +than for a particular source. The exposure map serves both to tell us the exposure time +at any given pixel of our image (assuming the image and exposure map are spatially +binned the same way), and also as a useful way to tell exactly which parts of the sky +are covered by the observation. -``` +The latter capability is of particular importance for the generation/analysis of +spectra, and the creation of Ancillary Response Files (ARFs), which describe the +effective sensitivity of Xtend as a function of energy. -### New XRISM-Xtend exposure maps +Unlike for image creation, XRISM does have a dedicated HEASoft task for the +generation of exposure maps; `xaexpmap`. We have once again set up a wrapper function +in the 'Global Setup: Functions' section of this notebook to make it easier to run +this task in parallel. + +There are two `xaexpmap` configuration options which control how the +attitude (essentially pointing) of XRISM over the course of the observation is +binned spatially. These bins ('off-axis wedges' as the +[`xaexpmap` documentation](https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html) +describes them) are where the initial 'time intervals' of observation coverage are calculated: +- **Radial Delta** - Passed to `xaexpmap` as `delta`. Radial increment (in arcmin) for the annular grid for which the attitude histogram will be calculated. The annuli are centered on the optical axis (off-axis angle = 0), and the central circle has a radius equal to `delta`. +- **Number of azimuthal bins** - Passed to `xaexpmap` as `numphi`. Number of azimuth (phi) bins in the first annular region over which attitude histogram bins will be calculated (i.e., this annular region lies between `delta` and 2*`delta` arcmin from the center of the annuli). The zeroth annular region is a full circle of radius `delta` and the nth annular region has an outer radius of (n+1)*`delta`, and `numphi`*n azimuthal bins. -Exposure maps... +The documentation for `xaexpmap` notes that you can force the attitude histogram to have a single bin, by choosing a radial delta that is much larger than any expected attitude variation during an observation. +We choose to create exposure maps from only one attitude histogram bin, by passing a large radial delta and requiring a single azimuthal bin: ```{code-cell} python expmap_rad_delta = Quantity(20, "arcmin") expmap_phi_bins = 1 ``` +Our wrapper function for `xaexpmap` also contains an optional call to the HEASoft FTOOLS `fimgbin` task, which +is used to re-bin the exposure map to a coarser spatial resolution (in this case to match the second +binning factor we generated images for). + +The `xaexpmap` task creates exposure maps with 1 image pixel per Sky X-Y coordinate system pixel, so we +only need to specify binning factors **that do not equal 1** here (adding 1 to this +list would be redundant and would waste compute time): ```{code-cell} python expmap_bin_factors = [4] ``` -***WHAT ABOUT THIS GTI?? - xa000128000xtd_mode.gti*** +Finally, we run the exposure map generation: ```{code-cell} python arg_combs = [ @@ -1544,17 +1568,11 @@ with mp.Pool(NUM_CORES) as p: ex_result = p.starmap(gen_xrism_xtend_expmap, arg_combs) ``` -Set up a template variable for output exposure map file names: - -```{code-cell} python - -``` - ## 4. Generating new XRISM-Xtend spectra and light curves In this section we will demonstrate how to generate source-specific data products from XRISM-Xtend observations; light curves and spectra (along with supporting files like -RMFs and Ancillary Response Files, or ARFs). +RMFs and ARFs). Rather than extracting spectra and light curves for the entire XRISM-Xtend FoV, *which is how the quick-look spectra and light curves contained in the archive are made*, we @@ -1712,17 +1730,10 @@ with mp.Pool(NUM_CORES) as p: lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) ``` -Create template variables for source and background light curves: - -```{code-cell} python - -``` ### New XRISM-Xtend spectra and supporting files -```{code-cell} python -``` #### Generating the spectral files @@ -1879,9 +1890,6 @@ with mp.Pool(NUM_CORES) as p: rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) ``` -```{code-cell} python - -``` #### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation @@ -1911,7 +1919,6 @@ xrtraytrace: Additional updates to fix and enable remote CalDB usage with xrtraytrace and xaarfgen ("timeout" interval extended for reading large CalDB files). - ``` ```{code-cell} python @@ -1944,18 +1951,15 @@ with mp.Pool(NUM_CORES) as p: arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) ``` -```{code-cell} python -``` - ```{warning} Due to the high-fidelity ray-tracing method used to calculate XRISM ARFs, the runtime -of this step can be on the order of hours. We have to do ***FINISH*** +of this step can be on the order of hours. ``` ## 5. Fitting spectral models to XRISM-Xtend spectra Finally, to show off the XRISM-Xtend products we just generated, we will perform -a very simple model fit to one of our spectra. +a simple model fit to one of our spectra. Our demonstration of spectral model fitting to an XRISM-Xtend spectrum will be performed using the [PyXspec](https://heasarc.gsfc.nasa.gov/docs/software/xspec/python/html/index.html) package. @@ -2076,6 +2080,8 @@ HEASoftPy HEASARC Page: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/hea HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html +HEASoft XRISM `xaexpmap` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html + ### Acknowledgements From 26d02930bfb4a856dcdea71ca86e4043bc755723 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 15:19:31 -0500 Subject: [PATCH 143/221] Spent an inordinate amount of time adding a subsection on the calibration source region files, how to convert them to different coordinate systems, and then plot them on images. In the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 119 +++++++++++++++--- 1 file changed, 103 insertions(+), 16 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 1a72ba80..b306da10 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -96,7 +96,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc -from regions import CircleSkyRegion +from regions import CircleSkyRegion, Regions from xga.products import Image ``` @@ -1500,6 +1500,10 @@ with mp.Pool(NUM_CORES) as p: im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` +#### Problematic pixels + + + ### New XRISM-Xtend exposure maps We also generate exposure maps for the entire FoV of a particular observation mode, rather @@ -1605,12 +1609,31 @@ effect is sometimes referred to as **cross-talk** or **spatial-spectral mixing ( for this effect is complicated and time-consuming, so our demonstration will focus on a point source, and extended sources will be discussed in another notebook. -### Setting up data product source and background extraction regions +### Setting up source and background extraction regions + +To define exactly where we want to extract events from to build our data products, we +will construct 'region files' in the commonly-used 'DS9' format. + +In this demonstration we will not provide guidance on how to choose particular +source/background regions for your science case, or give detailed information +about the DS9 region format and its capabilities. -There are different ways to define.... +Instead, we will show you how to construct basic region files using the +astropy-affiliated `regions` module. + +Most high-energy missions use three common coordinate systems: +- **Detector (DET) X-Y** - This coordinate system is aligned with the detector; a coordinate in this system will always represent the same physical position on the detector. +- **Sky X-Y** - A transformed version of the DET X-Y coordinate system, aligned with the roll angle of the telescope. **Within a single observation**, a Sky X-Y coordinate will always represent the same physical position on the sky. +- **RA-DEC** - The familiar right ascension and declination coordinate system. + +You need to be careful about which coordinate system you use with which tools, as some +tasks will not accept regions in all coordinate systems. #### General RA-DEC region files +We define a `CircleSkyRegion` instance (a class of the `regions` module) centered on +our target source, with a radius of 2 arcminutes. + ```{code-cell} python # Where to write the new region file radec_src_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_src.reg") @@ -1625,7 +1648,8 @@ src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) src_reg.write(radec_src_reg_path, format="ds9", overwrite=True) ``` -We do the same to define a region from which to extract a background spectrum: +We do the same to define a region from which to extract a background spectrum, though +this region is of a different size and is not centered on the source: ```{code-cell} python # Where to write the new region file @@ -1644,6 +1668,80 @@ back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) back_reg.write(radec_back_reg_path, format="ds9", overwrite=True) ``` +#### Excluding the XRISM-Xtend calibration sources + +The XRISM-Xtend instrument has two calibration sources in its FoV, which present as +bright circles on opposite edges of the detector. While highly useful for the +calibration of Xtend's energy scale, we do not want to accidentally include +calibration events in spectra or light curves we extract. + +As such, we have to define regions to exclude these sources from our data products. + +This will be quite simple, as HEASoft includes a pre-made region file that defines the +location and extent of the emission from the calibration sources. + +A small difficulty arises from the fact that this pre-made region file is defined in +detector coordinates, rather than the RA-Dec coordinates we've been using so far. That +will be pretty easy to deal with, however, as HEASoft includes a tool to transform region +files between different coordinate systems. + +```{code-cell} python +# The path to the included calibration source region file +detpix_xtend_calib_reg_path = os.path.join( + os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" +) + +# Setting up the output file for the RA-Dec calibration source regions +radec_xtend_calib_reg_path = os.path.join(OUT_PATH, "{oi}", "radec_{oi}_calsrc.reg") +``` + +We only need to generate one RA-Dec calibration source region file for each +observation, as the observations with data from multiple data modes share a common +coordinate system. + +The HEASoft tool `coordpnt` is used to perform the transformation from detector to +RA-Dec coordinates - it requires pointing coordinate and telescope roll angle +information which we extract from previously generated image files. + +```{code-cell} python +for oi, dcs in rel_dataclasses.items(): + cur_im_path = IM_PATH_TEMP.format(oi=oi, xdc=dcs[0], ibf=1, lo=chos_im_en[0].value, + hi=chos_im_en[1].value) + with fits.open(cur_im_path) as imago: + cur_pnt_ra = imago[0].header['RA_PNT'] + cur_pnt_dec = imago[0].header['DEC_PNT'] + cur_roll_ang = imago[0].header['PA_NOM'] + + # Call the HEASoft task that converts from detector coordinates to RA-Dec + hsp.coordpnt(input=detpix_xtend_calib_reg_path, + outfile=radec_xtend_calib_reg_path.format(oi=oi), + telescop='XRISM', instrume='XTEND', ra=cur_pnt_ra, dec=cur_pnt_dec, + roll=cur_roll_ang, startsys='DET', stopsys='RADEC', clobber=True) +``` + +Finally, we have to pull the RA-Dec calibration regions from the transformed region +file. The `regions` module provides functions to read-in region files in +various formats and coordinate systems but as the calibration region file was defined +to subtract the regions (each region in the file has a negative sign applied to it), +we need to manually read in the file. + +Note that we also prepend a global color to the region string, so that when we plot +the calibration regions, they will appear as a different color than the source and +background regions: + +```{code-cell} python +cal_regs = {} +for oi in rel_obsids: + with open(radec_xtend_calib_reg_path.format(oi=oi), 'r') as calbo: + cur_cal_reg_str = "global color=white\n" + calbo.read().replace('-ellipse', 'ellipse') + # The '.regions' just retrieves a list of region objects, we don't need to + # keep the calibration regions in the regions module 'Regions' class they + # are read into + cal_regs[oi] = Regions.parse(cur_cal_reg_str, format='ds9').regions +cal_regs +``` + + #### Visualizing the source and background extraction regions on XRISM-Xtend images Examining... @@ -1658,23 +1756,12 @@ for oi, cur_dcs in rel_dataclasses.items(): oi=oi, xdc=dc, ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value ) cur_im = Image(cur_im_path, oi, "Xtend", "", "", "", *chos_im_en) - cur_im.regions = [src_reg, back_reg] + cur_im.regions = [src_reg, back_reg] + cal_regs[oi] cur_im.view(src_coord_quant, zoom_in=True, view_regions=True) oi_skypix_wcs.setdefault(oi, cur_im.radec_wcs) ``` -#### Excluding the XRISM-Xtend calibration sources - -***HAVE TO LOAD THE CALIBRATION SOURCE REGIONS, CONVERT TO SKY PIX SYSTEM, AND EXCLUDE THEM FROM THE EXTRACTION REGION DEFINITIONS*** - -***THIS FILE IS SET UP FOR EXCLUSION (I.E. WITH - IN FRONT OF REGIONS) SO THE REGIONS MODULE WON'T READ THEM IN, AT LEAST BY DEFAULT*** - -```{code-cell} python -detpix_xtend_calib_reg_path = os.path.join( - os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" -) -``` #### Observation specific sky-pixel coordinate region files From 7dc58ad138bb76290fb60d7cd17d240a3b2bc6e2 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 16:30:09 -0500 Subject: [PATCH 144/221] Altered where in the getting-started-xrism-xtend.md notebook the RA-Dec system region files are written. Also made sure both those region files and the Sky X-Y coordinate ones exclude calibration source regions. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 129 ++++++++++-------- 1 file changed, 70 insertions(+), 59 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b306da10..e36cefdd 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1623,51 +1623,12 @@ astropy-affiliated `regions` module. Most high-energy missions use three common coordinate systems: - **Detector (DET) X-Y** - This coordinate system is aligned with the detector; a coordinate in this system will always represent the same physical position on the detector. -- **Sky X-Y** - A transformed version of the DET X-Y coordinate system, aligned with the roll angle of the telescope. **Within a single observation**, a Sky X-Y coordinate will always represent the same physical position on the sky. +- **Sky X-Y** - A transformed version of the DETX-DETY coordinate system, aligned with the roll angle of the telescope. **Within a single observation**, a Sky X-Y coordinate will always represent the same physical position on the sky. - **RA-DEC** - The familiar right ascension and declination coordinate system. You need to be careful about which coordinate system you use with which tools, as some tasks will not accept regions in all coordinate systems. -#### General RA-DEC region files - -We define a `CircleSkyRegion` instance (a class of the `regions` module) centered on -our target source, with a radius of 2 arcminutes. - -```{code-cell} python -# Where to write the new region file -radec_src_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_src.reg") - -# The radius of the source extraction region -src_reg_rad = Quantity(2, "arcmin") - -# Setting up a 'regions' module circular sky region instance -src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) - -# Write the source region to a region file -src_reg.write(radec_src_reg_path, format="ds9", overwrite=True) -``` - -We do the same to define a region from which to extract a background spectrum, though -this region is of a different size and is not centered on the source: - -```{code-cell} python -# Where to write the new region file -radec_back_reg_path = os.path.join(OUT_PATH, f"radec_{SRC_NAME}_back.reg") - -# The central coordinate of the background region -back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") - -# The radius of the background region -back_reg_rad = Quantity(3, "arcmin") - -# Setting up a 'regions' module circular sky region instance for the background region -back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) - -# Once again writing the region to a region file as well -back_reg.write(radec_back_reg_path, format="ds9", overwrite=True) -``` - #### Excluding the XRISM-Xtend calibration sources The XRISM-Xtend instrument has two calibration sources in its FoV, which present as @@ -1721,26 +1682,53 @@ for oi, dcs in rel_dataclasses.items(): Finally, we have to pull the RA-Dec calibration regions from the transformed region file. The `regions` module provides functions to read-in region files in -various formats and coordinate systems but as the calibration region file was defined -to subtract the regions (each region in the file has a negative sign applied to it), -we need to manually read in the file. +various formats and coordinate systems; it also understands that the regions in +these files are to be excluded (indicated by a '-' prefix). -Note that we also prepend a global color to the region string, so that when we plot -the calibration regions, they will appear as a different color than the source and -background regions: +Note that we also set the calibration region's color to be white so that when we plot +them, they will appear as a different color than the source and background regions: ```{code-cell} python cal_regs = {} for oi in rel_obsids: - with open(radec_xtend_calib_reg_path.format(oi=oi), 'r') as calbo: - cur_cal_reg_str = "global color=white\n" + calbo.read().replace('-ellipse', 'ellipse') - # The '.regions' just retrieves a list of region objects, we don't need to - # keep the calibration regions in the regions module 'Regions' class they - # are read into - cal_regs[oi] = Regions.parse(cur_cal_reg_str, format='ds9').regions + cur_cal_regs = Regions(radec_xtend_calib_reg_path.format(oi=oi), format='ds9') + for cur_reg in cur_cal_regs: + cur_reg.visual['color'] = 'white' + + # The '.regions' just retrieves a list of region objects, we don't need to keep + # the calibration regions in the regions module 'Regions' class they are read into + cal_regs[oi] = cur_cal_regs.regions + cal_regs ``` +#### Source and background RA-DEC region files + +We define a `CircleSkyRegion` instance (a class of the `regions` module) centered on +our target source, with a radius of 2 arcminutes. + +```{code-cell} python +# The radius of the source extraction region +src_reg_rad = Quantity(2, "arcmin") + +# Setting up a 'regions' module circular sky region instance +src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) +``` + +We do the same to define a region from which to extract a background spectrum, though +this region is of a different size and is not centered on the source: + +```{code-cell} python +# The central coordinate of the background region +back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") + +# The radius of the background region +back_reg_rad = Quantity(3, "arcmin") + +# Setting up a 'regions' module circular sky region instance for the background region +back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) +``` + #### Visualizing the source and background extraction regions on XRISM-Xtend images @@ -1762,19 +1750,42 @@ for oi, cur_dcs in rel_dataclasses.items(): oi_skypix_wcs.setdefault(oi, cur_im.radec_wcs) ``` +#### Writing observation-specific RA-Dec and sky-pixel coordinate region files + -#### Observation specific sky-pixel coordinate region files ```{code-cell} python -obs_src_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_src.reg") -obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", "skypix_{oi}_{n}_back.reg") +# Where to write the new RA-Dec source region file - the double {{}} around 'oi' just +# means that the f-string will fill in the SRC_NAME and leave '{oi}' to be +# formatted later +radec_src_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_src.reg") +# Where to write the new RA-Dec background region file +radec_back_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_back.reg") + +# The file path templates for the source and background Sky X-Y system region files +obs_src_reg_path_temp = os.path.join(OUT_PATH, '{oi}', f"skypix_{{oi}}_{SRC_NAME}_src.reg") +obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_back.reg") for oi in rel_obsids: - src_reg.to_pixel(oi_skypix_wcs[oi]).write( - obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9", overwrite=True + # We set up a combination of the source region and the calibration source + # regions - the calibration regions have been set up to be excluded, which will + # be reflected in the final region files we write + src_comb_regs = Regions([src_reg] + cal_regs[oi]) + # The same but for the background region + back_comb_regs = Regions([back_reg] + cal_regs[oi]) + + # Write the RA-Dec source region file + src_comb_regs.write(radec_src_reg_path.format(oi=oi), format="ds9", overwrite=True) + # And the RA-Dec background region file + back_reg.write(radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True) + + # Now we repeat the exercise, but use the Sky<->RA-Dec WCS information we pulled + # from the images to convert the regions to the Sky X-Y system + src_comb_regs.to_pixel(oi_skypix_wcs[oi]).write( + obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True ) - back_reg.to_pixel(oi_skypix_wcs[oi]).write( - obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), format="ds9", overwrite=True + back_comb_regs.to_pixel(oi_skypix_wcs[oi]).write( + obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True ) ``` From 09b6f8cbef7f520fec4597949754a76bcdccde4b Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 16:53:04 -0500 Subject: [PATCH 145/221] Added commentary to the part of the getting-started-xrism-xtend.md notebook that writes out region files, and the part that plots regions on top of previously generated XRISM-Xtend images. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 29 +++++++++++++++++-- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index e36cefdd..f17c0fc8 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1729,10 +1729,17 @@ back_reg_rad = Quantity(3, "arcmin") back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) ``` - #### Visualizing the source and background extraction regions on XRISM-Xtend images -Examining... +We should inspect the regions in-situ to make sure they look sensible - first, our +previously generated images are loaded in as `Image` class (from the `XGA` Python +module) instances. + +The regions we created are then assigned to each image's `regions`, and +the `.view()` method is called with the `view_regions=True` argument to display them. + +Additionally, we extract the RA-Dec <-> Sky X-Y WCS from one image per observation so +that we can use it later on to transform our RA-Dec regions into Sky X-Y regions. ```{code-cell} python chos_im_en = xtd_im_en_bounds[0].to("keV") @@ -1752,7 +1759,23 @@ for oi, cur_dcs in rel_dataclasses.items(): #### Writing observation-specific RA-Dec and sky-pixel coordinate region files +Now we've set up the regions and visualized them, we'll write them to disk as region +files that can be passed to the HEASoft tasks used to generate spectra and light curves. + +We set up instances of the `Regions` class of the astropy-affiliated `regions` module +for the source and background regions plus calibration regions. The `write()` method +is then used to save a DS9-formatted region file to disk. + +The calibration source regions are set up to be excluded, and the output files will +reflect that. + +We write two versions each of the source and region files, one version in the RA-Dec +coordinate system, and the other in the Sky X-Y system (different tasks have different +requirements for the coordinate system they accept). +Our RA-Dec regions are converted to Sky X-Y using a feature of the `regions` +module, using the WCS information we pulled from the images when we visualized them +with the source, background, and calibration source regions overplotted ```{code-cell} python # Where to write the new RA-Dec source region file - the double {{}} around 'oi' just @@ -1763,7 +1786,7 @@ radec_src_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_sr radec_back_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_back.reg") # The file path templates for the source and background Sky X-Y system region files -obs_src_reg_path_temp = os.path.join(OUT_PATH, '{oi}', f"skypix_{{oi}}_{SRC_NAME}_src.reg") +obs_src_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_src.reg") obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_back.reg") for oi in rel_obsids: From 06893fb773ebc96080c8ac286bc65b123b4fb02f Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 17:25:18 -0500 Subject: [PATCH 146/221] The meta data of the functions cell in getting-started-xrism-xtend.md had its formatting screwed up somehow, which was stopping the black formatter from doing its job (I hope). This should fix that. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 369 +++++++++--------- 1 file changed, 194 insertions(+), 175 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f17c0fc8..26184e99 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -106,22 +106,21 @@ from xga.products import Image ```{code-cell} python --- -tags: [hide - input] +tags: [hide-input] jupyter: -source_hidden: true + source_hidden: true --- - def process_xrism_xtend( - cur_obs_id: str, - out_dir: str, - evt_dir: str, - attitude: str, - orbit: str, - obs_gti: str, - mkf_filter: str, - file_stem: str, - extended_housekeeping: str, - xtend_housekeeping: str, + cur_obs_id: str, + out_dir: str, + evt_dir: str, + attitude: str, + orbit: str, + obs_gti: str, + mkf_filter: str, + file_stem: str, + extended_housekeeping: str, + xtend_housekeeping: str, ): """ A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process @@ -196,11 +195,11 @@ def process_xrism_xtend( def gen_xrism_xtend_image( - event_file: str, - out_dir: str, - lo_en: Quantity, - hi_en: Quantity, - im_bin: int = 1, + event_file: str, + out_dir: str, + lo_en: Quantity, + hi_en: Quantity, + im_bin: int = 1, ): """ This function wraps the HEASoft 'extractor' tool and is used to spatially bin @@ -250,10 +249,9 @@ def gen_xrism_xtend_image( evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" # Set up the output file name for the image we're about to generate. - im_out = os.path.basename(IM_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - ibf=im_bin, lo=lo_en_val, - hi=hi_en_val) + im_out = os.path.basename(IM_PATH_TEMP).format( + oi=cur_obs_id, xdc=cur_xtend_data_class, ibf=im_bin, lo=lo_en_val, hi=hi_en_val + ) # Create a temporary working directory temp_work_dir = os.path.join( @@ -287,15 +285,15 @@ def gen_xrism_xtend_image( def gen_xrism_xtend_expmap( - event_file: str, - out_dir: str, - gti_file: str, - extend_hk_file: str, - bad_pix_file: str, - pix_gti_file: str = "NONE", - im_bin: int = 1, - radial_delta: Union[float, Quantity] = Quantity(20.0, "arcmin"), - num_phi_bin: int = 1, + event_file: str, + out_dir: str, + gti_file: str, + extend_hk_file: str, + bad_pix_file: str, + pix_gti_file: str = "NONE", + im_bin: int = 1, + radial_delta: Union[float, Quantity] = Quantity(20.0, "arcmin"), + num_phi_bin: int = 1, ): """ Function that wraps the HEASoftPy interface to the XRISM-Xtend 'xaexpmap' @@ -346,29 +344,29 @@ def gen_xrism_xtend_expmap( # Two variants of exposure map can be generated by the function we're about to # call; the default is a map of the integrated exposure time for each pixel, and # the second (not recommended by the documentation) is a flat-fielding map - # TODO REINSTATE WHEN WE HAVE A BETTER UNDERSTANDING OF POTENTIAL USER USES out_map_type = "EXPOSURE" - ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" + # TODO REINSTATE WHEN WE HAVE A BETTER UNDERSTANDING OF POTENTIAL USER USES + # ex_type = "expmap" if out_map_type == "EXPOSURE" else "flatfieldmap" # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) os.makedirs(temp_work_dir) # Set up the output file name for the exposure map we're about to generate. - ex_out = os.path.basename(EX_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - rd=radial_delta, - npb=num_phi_bin, - ibf=1) + ex_out = os.path.basename(EX_PATH_TEMP).format( + oi=cur_obs_id, xdc=cur_xtend_data_class, rd=radial_delta, npb=num_phi_bin, ibf=1 + ) # If the user wants to bin up the exposure map, we'll need to set up another # output file name with the bin factor set to the input value (this variable # is not used if the user does not want to bin the map) - binned_ex_out = os.path.basename(EX_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - rd=radial_delta, - npb=num_phi_bin, - ibf=im_vbin) + binned_ex_out = os.path.basename(EX_PATH_TEMP).format( + oi=cur_obs_id, + xdc=cur_xtend_data_class, + rd=radial_delta, + npb=num_phi_bin, + ibf=im_bin, + ) # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xaexpmap_{}".format(randint(0, int(1e8)))) @@ -408,8 +406,10 @@ def gen_xrism_xtend_expmap( os.rename(os.path.join(temp_work_dir, ex_out), os.path.join(out_dir, ex_out)) # Then do the same for the spatially binned exposure map, if it was requested if im_bin != 1: - os.rename(os.path.join(temp_work_dir, binned_ex_out), - os.path.join(out_dir, binned_ex_out)) + os.rename( + os.path.join(temp_work_dir, binned_ex_out), + os.path.join(out_dir, binned_ex_out), + ) # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -418,16 +418,16 @@ def gen_xrism_xtend_expmap( def gen_xrism_xtend_lightcurve( - event_file: str, - out_dir: str, - rel_src_coord: SkyCoord, - rel_src_radius: Quantity, - src_reg_file: str, - back_reg_file: str, - lo_en: Quantity = Quantity(0.6, "keV"), - hi_en: Quantity = Quantity(13, "keV"), - time_bin_size: Quantity = Quantity(200, "s"), - lc_bin_thresh: float = 0.0, + event_file: str, + out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, + src_reg_file: str, + back_reg_file: str, + lo_en: Quantity = Quantity(0.6, "keV"), + hi_en: Quantity = Quantity(13, "keV"), + time_bin_size: Quantity = Quantity(200, "s"), + lc_bin_thresh: float = 0.0, ): """ Function that wraps the HEASoftPy interface to the HEASoft extractor tool, set @@ -502,26 +502,30 @@ def gen_xrism_xtend_lightcurve( evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" # Set up the output file name for the light curve we're about to generate. - lc_out = os.path.basename(LC_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - ra=ra_val, - dec=dec_val, - rad=rad_val, - lo=lo_en_val, - hi=hi_en_val, - lct=lc_bin_thresh, - tb=time_bin_size) + lc_out = os.path.basename(LC_PATH_TEMP).format( + oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val, + rad=rad_val, + lo=lo_en_val, + hi=hi_en_val, + lct=lc_bin_thresh, + tb=time_bin_size, + ) # The same file name, but with 'lightcurve' changed to 'back-lightcurve', and the # radius information information removed, for the background light curve. - lc_back_out = os.path.basename(BACK_LC_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - ra=ra_val, - dec=dec_val, - lo=lo_en_val, - hi=hi_en_val, - lct=lc_bin_thresh, - tb=time_bin_size) + lc_back_out = os.path.basename(BACK_LC_PATH_TEMP).format( + oi=cur_obs_id, + xdc=cur_xtend_data_class, + ra=ra_val, + dec=dec_val, + lo=lo_en_val, + hi=hi_en_val, + lct=lc_bin_thresh, + tb=time_bin_size, + ) # Create a temporary working directory temp_work_dir = os.path.join( @@ -562,7 +566,9 @@ def gen_xrism_xtend_lightcurve( # Move the light curves up from the temporary directory os.rename(os.path.join(temp_work_dir, lc_out), os.path.join(out_dir, lc_out)) - os.rename(os.path.join(temp_work_dir, lc_back_out), os.path.join(out_dir, lc_back_out)) + os.rename( + os.path.join(temp_work_dir, lc_back_out), os.path.join(out_dir, lc_back_out) + ) # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -571,12 +577,12 @@ def gen_xrism_xtend_lightcurve( def gen_xrism_xtend_spectrum( - event_file: str, - out_dir: str, - rel_src_coord: SkyCoord, - rel_src_radius: Quantity, - src_reg_file: str, - back_reg_file: str, + event_file: str, + out_dir: str, + rel_src_coord: SkyCoord, + rel_src_radius: Quantity, + src_reg_file: str, + back_reg_file: str, ): """ Function that wraps the HEASoftPy interface to the HEASoft extractor tool, set @@ -611,15 +617,12 @@ def gen_xrism_xtend_spectrum( # Set up the output file names for the source and background spectra we're # about to generate. - sp_out = ( - f"xrism-xtend-obsid{cur_obs_id}-dataclass{cur_xtend_data_class}-" - f"ra{ra_val}-dec{dec_val}-radius{rad_val}deg-enALL-" - f"spectrum.fits" + sp_out = os.path.basename(SP_PATH_TEMP).format( + oi=cur_obs_id, xdc=cur_xtend_data_class, ra=ra_val, dec=dec_val, rad=rad_val + ) + sp_back_out = os.path.basename(BACK_SP_PATH_TEMP).format( + oi=cur_obs_id, xdc=cur_xtend_data_class, ra=ra_val, dec=dec_val ) - back_sp_out = os.path.basename(BACK_SP_PATH_TEMP).format(oi=cur_obs_id, - xdc=cur_xtend_data_class, - ra=ra_val, - dec=dec_val) # Create a temporary working directory temp_work_dir = os.path.join( @@ -658,7 +661,9 @@ def gen_xrism_xtend_spectrum( # Move the spectra up from the temporary directory os.rename(os.path.join(temp_work_dir, sp_out), os.path.join(out_dir, sp_out)) - os.rename(os.path.join(temp_work_dir, sp_back_out), os.path.join(out_dir, sp_back_out)) + os.rename( + os.path.join(temp_work_dir, sp_back_out), os.path.join(out_dir, sp_back_out) + ) # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -675,16 +680,12 @@ def gen_xrism_xtend_rmf(spec_file: str, out_dir: str): :param str out_dir: The directory where output files should be written. """ - # We can extract the ObsID directly from the header of the spectrum file - it is - # safer than having the user pass it separately - with fits.open(spec_file) as read_speco: - cur_obs_id = read_speco[0].header["OBS_ID"] - # Create a temporary working directory temp_work_dir = os.path.join(out_dir, "xtdrmf_{}".format(randint(0, int(1e8)))) os.makedirs(temp_work_dir) - # Set up the RMF file name by cannibalising the name of the spectrum file + # Set up the RMF file name by cannibalising the name of the spectrum file - this + # means we don't have to worry about identifying the ObsID rmf_out = os.path.basename(spec_file).split("-ra")[0] + ".rmf" # Using dual contexts, one that moves us into the output directory for the @@ -708,12 +709,12 @@ def gen_xrism_xtend_rmf(spec_file: str, out_dir: str): def gen_xrism_xtend_arf( - out_dir: str, - expmap_file: str, - spec_file: str, - rmf_file: str, - src_radec_reg_file: str, - num_photons: int, + out_dir: str, + expmap_file: str, + spec_file: str, + rmf_file: str, + src_radec_reg_file: str, + num_photons: int, ): """ A wrapper function for the HEASoft `xaarfgen` task, which we use to generate @@ -800,7 +801,10 @@ def gen_xrism_xtend_arf( os.rename(os.path.join(temp_work_dir, arf_out), os.path.join(out_dir, arf_out)) # If the ray traced file already existed, we don't need to move anything if not ray_traced_exists: - os.rename(os.path.join(temp_work_dir, ray_traced_evt_out), os.path.join(out_dir, ray_traced_evt_out)) + os.rename( + os.path.join(temp_work_dir, ray_traced_evt_out), + os.path.join(out_dir, ray_traced_evt_out), + ) # Make sure to remove the temporary directory rmtree(temp_work_dir) @@ -826,6 +830,57 @@ TASK_CHATTER = 3 # The approximate linear relationship between Xtend PI and event energy XTD_EV_PER_CHAN = (1 / Quantity(166.7, "chan/keV")).to("eV/chan") +``` + +### Configuration + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +# ------------- Configure global package settings -------------- +# Raise Python exceptions if a heasoftpy task fails +# TODO Remove once this becomes a default in heasoftpy +hsp.Config.allow_failure = False + +# Set up the method for spawning processes. +mp.set_start_method("fork", force=True) +# -------------------------------------------------------------- + +# ------------- Setting how many cores we can use -------------- +NUM_CORES = None +total_cores = os.cpu_count() + +if NUM_CORES is None: + NUM_CORES = total_cores +elif not isinstance(NUM_CORES, int): + raise TypeError( + "If manually overriding 'NUM_CORES', you must set it to an integer value." + ) +elif isinstance(NUM_CORES, int) and NUM_CORES > total_cores: + raise ValueError( + f"If manually overriding 'NUM_CORES', the value must be less than or " + f"equal to the total available cores ({total_cores})." + ) +# -------------------------------------------------------------- + +# -------------- Set paths and create directories -------------- +if os.path.exists("../../../_data"): + ROOT_DATA_DIR = "../../../_data/XRISM/" +else: + ROOT_DATA_DIR = "XRISM/" + +ROOT_DATA_DIR = os.path.abspath(ROOT_DATA_DIR) + +# Make sure the download directory exists. +os.makedirs(ROOT_DATA_DIR, exist_ok=True) + +# Setup path and directory into which we save output files from this example. +OUT_PATH = os.path.abspath("XRISM_output") +os.makedirs(OUT_PATH, exist_ok=True) +# -------------------------------------------------------------- # ------------- Set up output file path templates -------------- @@ -852,14 +907,14 @@ EX_PATH_TEMP = os.path.join( LC_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" \ + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-lightcurve.fits", ) BACK_LC_PATH_TEMP = os.path.join( OUT_PATH, "{oi}", - "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" \ + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-back-lightcurve.fits", ) # -------------------------- @@ -896,57 +951,6 @@ ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") # -------------------------------------------------------------- ``` -### Configuration - -```{code-cell} python ---- -tags: [hide-input] -jupyter: - source_hidden: true ---- -# ------------- Configure global package settings -------------- -# Raise Python exceptions if a heasoftpy task fails -# TODO Remove once this becomes a default in heasoftpy -hsp.Config.allow_failure = False - -# Set up the method for spawning processes. -mp.set_start_method("fork", force=True) -# -------------------------------------------------------------- - -# ------------- Setting how many cores we can use -------------- -NUM_CORES = None -total_cores = os.cpu_count() - -if NUM_CORES is None: - NUM_CORES = total_cores -elif not isinstance(NUM_CORES, int): - raise TypeError( - "If manually overriding 'NUM_CORES', you must set it to an integer value." - ) -elif isinstance(NUM_CORES, int) and NUM_CORES > total_cores: - raise ValueError( - f"If manually overriding 'NUM_CORES', the value must be less than or " - f"equal to the total available cores ({total_cores})." - ) -# -------------------------------------------------------------- - -# -------------- Set paths and create directories -------------- -if os.path.exists("../../../_data"): - ROOT_DATA_DIR = "../../../_data/XRISM/" -else: - ROOT_DATA_DIR = "XRISM/" - -ROOT_DATA_DIR = os.path.abspath(ROOT_DATA_DIR) - -# Make sure the download directory exists. -os.makedirs(ROOT_DATA_DIR, exist_ok=True) - -# Setup path and directory into which we save output files from this example. -OUT_PATH = os.path.abspath("XRISM_output") -os.makedirs(OUT_PATH, exist_ok=True) -# -------------------------------------------------------------- -``` - *** ## 1. Finding and downloading XRISM observations of **NAMEHERE** @@ -1348,7 +1352,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - # Set up the figure plt.figure(figsize=(5.5, 5.5)) @@ -1532,6 +1535,7 @@ describes them) are where the initial 'time intervals' of observation coverage a The documentation for `xaexpmap` notes that you can force the attitude histogram to have a single bin, by choosing a radial delta that is much larger than any expected attitude variation during an observation. We choose to create exposure maps from only one attitude histogram bin, by passing a large radial delta and requiring a single azimuthal bin: + ```{code-cell} python expmap_rad_delta = Quantity(20, "arcmin") expmap_phi_bins = 1 @@ -1544,6 +1548,7 @@ binning factor we generated images for). The `xaexpmap` task creates exposure maps with 1 image pixel per Sky X-Y coordinate system pixel, so we only need to specify binning factors **that do not equal 1** here (adding 1 to this list would be redundant and would waste compute time): + ```{code-cell} python expmap_bin_factors = [4] ``` @@ -1665,19 +1670,30 @@ RA-Dec coordinates - it requires pointing coordinate and telescope roll angle information which we extract from previously generated image files. ```{code-cell} python +chos_im_en = xtd_im_en_bounds[2] + for oi, dcs in rel_dataclasses.items(): - cur_im_path = IM_PATH_TEMP.format(oi=oi, xdc=dcs[0], ibf=1, lo=chos_im_en[0].value, - hi=chos_im_en[1].value) + cur_im_path = IM_PATH_TEMP.format( + oi=oi, xdc=dcs[0], ibf=1, lo=chos_im_en[0].value, hi=chos_im_en[1].value + ) with fits.open(cur_im_path) as imago: - cur_pnt_ra = imago[0].header['RA_PNT'] - cur_pnt_dec = imago[0].header['DEC_PNT'] - cur_roll_ang = imago[0].header['PA_NOM'] + cur_pnt_ra = imago[0].header["RA_PNT"] + cur_pnt_dec = imago[0].header["DEC_PNT"] + cur_roll_ang = imago[0].header["PA_NOM"] # Call the HEASoft task that converts from detector coordinates to RA-Dec - hsp.coordpnt(input=detpix_xtend_calib_reg_path, - outfile=radec_xtend_calib_reg_path.format(oi=oi), - telescop='XRISM', instrume='XTEND', ra=cur_pnt_ra, dec=cur_pnt_dec, - roll=cur_roll_ang, startsys='DET', stopsys='RADEC', clobber=True) + hsp.coordpnt( + input=detpix_xtend_calib_reg_path, + outfile=radec_xtend_calib_reg_path.format(oi=oi), + telescop="XRISM", + instrume="XTEND", + ra=cur_pnt_ra, + dec=cur_pnt_dec, + roll=cur_roll_ang, + startsys="DET", + stopsys="RADEC", + clobber=True, + ) ``` Finally, we have to pull the RA-Dec calibration regions from the transformed region @@ -1691,9 +1707,9 @@ them, they will appear as a different color than the source and background regio ```{code-cell} python cal_regs = {} for oi in rel_obsids: - cur_cal_regs = Regions(radec_xtend_calib_reg_path.format(oi=oi), format='ds9') + cur_cal_regs = Regions(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") for cur_reg in cur_cal_regs: - cur_reg.visual['color'] = 'white' + cur_reg.visual["color"] = "white" # The '.regions' just retrieves a list of region objects, we don't need to keep # the calibration regions in the regions module 'Regions' class they are read into @@ -1783,11 +1799,17 @@ with the source, background, and calibration source regions overplotted # formatted later radec_src_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_src.reg") # Where to write the new RA-Dec background region file -radec_back_reg_path = os.path.join(OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_back.reg") +radec_back_reg_path = os.path.join( + OUT_PATH, "{oi}", f"radec_{{oi}}_{SRC_NAME}_back.reg" +) # The file path templates for the source and background Sky X-Y system region files -obs_src_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_src.reg") -obs_back_reg_path_temp = os.path.join(OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_back.reg") +obs_src_reg_path_temp = os.path.join( + OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_src.reg" +) +obs_back_reg_path_temp = os.path.join( + OUT_PATH, "{oi}", f"skypix_{{oi}}_{SRC_NAME}_back.reg" +) for oi in rel_obsids: # We set up a combination of the source region and the calibration source @@ -1851,7 +1873,6 @@ with mp.Pool(NUM_CORES) as p: lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) ``` - ### New XRISM-Xtend spectra and supporting files @@ -1945,7 +1966,6 @@ output grouped spectral files: ```{code-cell} python spec_group_type = "min" spec_group_scale = 1 - ``` Now we run the grouping tool - though this time we do not parallelize the task, as @@ -1969,7 +1989,7 @@ for oi, dcs in rel_dataclasses.items(): dec=src_coord.dec.value.round(6), rad=src_reg_rad.to("deg").value.round(4), ) - cur_grp_spec = grp_SP_PATH_TEMP.format( + cur_grp_spec = GRP_SP_PATH_TEMP.format( oi=oi, xdc=cur_dc, gt=spec_group_type, @@ -2011,7 +2031,6 @@ with mp.Pool(NUM_CORES) as p: rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) ``` - #### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation ```{code-cell} python @@ -2093,7 +2112,7 @@ Now we configure some behaviors of XSPEC/pyXspec: - We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). ```{code-cell} python -import xspec as xs +import xspec as xs # noqa: E402 xs.Xset.chatter = 0 @@ -2124,7 +2143,7 @@ xs.AllData.clear() # Set up the paths to grouped source spectrum, ungrouped background # spectrum, RMF, and ARF files -cur_spec = grp_SP_PATH_TEMP.format( +cur_spec = GRP_SP_PATH_TEMP.format( oi=chosen_demo_spec_obsid, xdc=chosen_demo_spec_dataclass, gt=spec_group_type, From ad8e734d4888c9e89831ea8cbae3a8fd5d8ceecd Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 17:35:20 -0500 Subject: [PATCH 147/221] Commented out the xspec and arfgen portions, and did a horrible bodge wrt selecting the right dataclasses. Want to do a test PR and see if this runs on CircleCI. For issue #128 ALSO COMMENTED OUT XMM CCF ACQUISITION IN CIRCLECI YML, WILL REALLY NEED TO REMEMBER TO PUT THAT BACK --- .circleci/config.yml | 56 ++++---- .../xrism/getting-started-xrism-xtend.md | 134 +++++++++--------- 2 files changed, 97 insertions(+), 93 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 66c91789..a5e4a083 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -60,34 +60,34 @@ jobs: # Some missions (e.g. XMM, Chandra, eROSITA) require that their calibration files are available locally. We # don't want to download them every single time, so we're going to cache them for future runs. - - run: - name: Acquiring/validating XMM-CCFs - no_output_timeout: 30m - command: | - # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same - # version number as currently defined in the environment variable, we have to download the data - if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then - - # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the - # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions - # available, so we don't want to delete the directory. - mkdir -p /home/jovyan/xmm-ccf/ccf-files - - # We do delete the version file, if it exists, as if we get to this point then we are either downloading - # the data for the first time, or the version number has changed. - [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver - - # Make a new version file - this goes a level up from the directory where the files will actually - # be stored (ccf-files), because the rsync process will delete the version file - echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver - - # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The - # neatest way is to make a new conda environment that just contains rsync - micromamba create -n rsync-env -y -c conda-forge rsync - - # Actually rsync the calibration files - micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files - fi +# - run: +# name: Acquiring/validating XMM-CCFs +# no_output_timeout: 30m +# command: | +# # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same +# # version number as currently defined in the environment variable, we have to download the data +# if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then +# +# # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the +# # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions +# # available, so we don't want to delete the directory. +# mkdir -p /home/jovyan/xmm-ccf/ccf-files +# +# # We do delete the version file, if it exists, as if we get to this point then we are either downloading +# # the data for the first time, or the version number has changed. +# [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver +# +# # Make a new version file - this goes a level up from the directory where the files will actually +# # be stored (ccf-files), because the rsync process will delete the version file +# echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver +# +# # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The +# # neatest way is to make a new conda environment that just contains rsync +# micromamba create -n rsync-env -y -c conda-forge rsync +# +# # Actually rsync the calibration files +# micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files +# fi # We also download the Chandra CalDB - run: diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 26184e99..ac2c2fc1 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1897,15 +1897,16 @@ with mp.Pool(NUM_CORES) as p: sp_result = p.starmap(gen_xrism_xtend_spectrum, arg_combs) ``` -Create template variables for source and background spectrum files: +#### Calculating 'BACKSCAL' for new XRISM-Xtend spectra -```{code-cell} python +***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** -``` -#### Calculating 'BACKSCAL' for new XRISM-Xtend spectra +***TERRIBLE BODGE MUST FIX*** -***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** +```{code-cell} python +rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} +``` ```{code-cell} python for oi, dcs in rel_dataclasses.items(): @@ -2087,8 +2088,8 @@ arg_combs = [ for dc in dcs ] -with mp.Pool(NUM_CORES) as p: - arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) +# with mp.Pool(NUM_CORES) as p: +# arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) ``` ```{warning} @@ -2112,22 +2113,25 @@ Now we configure some behaviors of XSPEC/pyXspec: - We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). ```{code-cell} python -import xspec as xs # noqa: E402 +# The strange comment on the end of this line is for the benefit of our +# automated code-checking processes. You shouldn't import modules anywhere but +# the top of your file, but this is unfortunately necessary at the moment +# import xspec as xs # noqa: E402 -xs.Xset.chatter = 0 +# xs.Xset.chatter = 0 # XSPEC parallelisation settings -xs.Xset.parallel.leven = NUM_CORES -xs.Xset.parallel.error = NUM_CORES -xs.Xset.parallel.steppar = NUM_CORES - -# Other xspec settings -xs.Plot.area = True -xs.Plot.xAxis = "keV" -xs.Plot.background = True -xs.Fit.statMethod = "cstat" -xs.Fit.query = "no" -xs.Fit.nIterations = 500 +# xs.Xset.parallel.leven = NUM_CORES +# xs.Xset.parallel.error = NUM_CORES +# xs.Xset.parallel.steppar = NUM_CORES +# +# # Other xspec settings +# xs.Plot.area = True +# xs.Plot.xAxis = "keV" +# xs.Plot.background = True +# xs.Fit.statMethod = "cstat" +# xs.Fit.query = "no" +# xs.Fit.nIterations = 500 ``` ### Reading XRISM-Xtend spectra into pyXspec @@ -2139,65 +2143,65 @@ chosen_demo_spec_dataclass = "31100010" ```{code-cell} python # In case this cell is re-run, clear all previously loaded spectra -xs.AllData.clear() - -# Set up the paths to grouped source spectrum, ungrouped background -# spectrum, RMF, and ARF files -cur_spec = GRP_SP_PATH_TEMP.format( - oi=chosen_demo_spec_obsid, - xdc=chosen_demo_spec_dataclass, - gt=spec_group_type, - gs=spec_group_scale, - ra=src_coord.ra.value.round(6), - dec=src_coord.dec.value.round(6), - rad=src_reg_rad.to("deg").value.round(4), -) - -cur_bspec = BACK_SP_PATH_TEMP.format( - oi=chosen_demo_spec_obsid, - xdc=chosen_demo_spec_dataclass, - ra=src_coord.ra.value.round(6), - dec=src_coord.dec.value.round(6), -) - -cur_rmf = RMF_PATH_TEMP.format( - oi=chosen_demo_spec_obsid, - xdc=chosen_demo_spec_dataclass, -) - -cur_arf = ARF_PATH_TEMP.format( - oi=chosen_demo_spec_obsid, - xdc=chosen_demo_spec_dataclass, - ra=src_coord.ra.value.round(6), - dec=src_coord.dec.value.round(6), - rad=src_reg_rad.to("deg").value.round(4), -) - -# Load the chosen spectrum (and all its supporting files) into pyXspec -xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) +# xs.AllData.clear() +# +# # Set up the paths to grouped source spectrum, ungrouped background +# # spectrum, RMF, and ARF files +# cur_spec = GRP_SP_PATH_TEMP.format( +# oi=chosen_demo_spec_obsid, +# xdc=chosen_demo_spec_dataclass, +# gt=spec_group_type, +# gs=spec_group_scale, +# ra=src_coord.ra.value.round(6), +# dec=src_coord.dec.value.round(6), +# rad=src_reg_rad.to("deg").value.round(4), +# ) +# +# cur_bspec = BACK_SP_PATH_TEMP.format( +# oi=chosen_demo_spec_obsid, +# xdc=chosen_demo_spec_dataclass, +# ra=src_coord.ra.value.round(6), +# dec=src_coord.dec.value.round(6), +# ) +# +# cur_rmf = RMF_PATH_TEMP.format( +# oi=chosen_demo_spec_obsid, +# xdc=chosen_demo_spec_dataclass, +# ) +# +# cur_arf = ARF_PATH_TEMP.format( +# oi=chosen_demo_spec_obsid, +# xdc=chosen_demo_spec_dataclass, +# ra=src_coord.ra.value.round(6), +# dec=src_coord.dec.value.round(6), +# rad=src_reg_rad.to("deg").value.round(4), +# ) +# +# # Load the chosen spectrum (and all its supporting files) into pyXspec +# xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) ``` ### Restricting the spectral channels used for fitting ```{code-cell} python -xs_spec.ignore("**-0.5 12.0-**") - -# Ignore any channels that have been marked as 'bad' -# This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra -# have been declared -xs.AllData.ignore("bad") +# xs_spec.ignore("**-0.5 12.0-**") +# +# # Ignore any channels that have been marked as 'bad' +# # This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra +# # have been declared +# xs.AllData.ignore("bad") ``` ### Setting up a spectral model ```{code-cell} python -xs.Model("tbabs*(powerlaw+apec+bbody)") +# xs.Model("tbabs*(powerlaw+apec+bbody)") ``` ### Fitting our pyXspec model to the XRISM-Xtend spectrum ```{code-cell} python -xs.Fit.perform() +# xs.Fit.perform() ``` ## About this notebook @@ -2206,7 +2210,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-03 +Updated On: 2025-12-05 +++ From 10a9be4c723c781c95e58cd02c4610e98eeabaa3 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 17:39:24 -0500 Subject: [PATCH 148/221] Specifically selected two out of three observations in the getting-started-xrism-xtend.md notebook. Will need to change where/how that happens. For issue #128 and the test PR --- .../xrism/getting-started-xrism-xtend.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index ac2c2fc1..c722a220 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -998,6 +998,12 @@ For an active mission (i.e., actively collecting data and adding to the archive) public_times = Time(all_xrism_obs["public_date"], format="mjd") avail_xrism_obs = all_xrism_obs[public_times <= Time.now()] +# TODO MAKE MORE PERMANENT ELSEWHERE +avail_xrism_obs = avail_xrism_obs[ + (avail_xrism_obs["obsid"] == "000128000") + | (avail_xrism_obs["obsid"] == "000126000") +] + # Define a couple of useful variables that make accessing information in the # table a little easier later on in the notebook # Create an array of the relevant ObsIDs From c044db26a9a2cf80d5ae396ffff143ed52091f4a Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 5 Dec 2025 18:49:56 -0500 Subject: [PATCH 149/221] Increased the time out of notebook execution in the sphinx configuration file. --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index cdffde54..b82ea7ab 100644 --- a/conf.py +++ b/conf.py @@ -24,7 +24,7 @@ # ---------------------------- MyST configuration ---------------------------- # MyST-NB configuration -nb_execution_timeout = 1200 +nb_execution_timeout = 2400 nb_merge_streams = True nb_execution_mode = "cache" nb_scroll_outputs = True From 17c475ede84e09e80788d88d10325f39c3d0fd4b Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 09:39:14 -0500 Subject: [PATCH 150/221] Added something that should print xtdpipeline outputs for any observations that failed processing in the getting-started-xrism-xtend.md notebook. Also made sure that the rel_dataclasses variable is modified to remove failed ObsIDs, if necessary. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index c722a220..fc982938 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -953,7 +953,7 @@ ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") *** -## 1. Finding and downloading XRISM observations of **NAMEHERE** +## 1. Finding and downloading XRISM observations of LMC N132D ### Determining the name of the XRISM observation summary table @@ -963,7 +963,7 @@ catalog_name = Heasarc.list_catalogs(master=True, keywords="xrism")[0]["name"] catalog_name ``` -### What are the coordinates of **NAMEHERE**? +### What are the coordinates of LMC N132D? To search for relevant observations, we have to know the coordinates of our source. The astropy module allows us to look up a source name in CDS' Sesame name @@ -1215,10 +1215,22 @@ with mp.Pool(NUM_CORES) as p: xtd_pipe_problem_ois = [all_out[0] for all_out in pipe_result if not all_out[2]] rel_obsids = [oi for oi in rel_obsids if oi not in xtd_pipe_problem_ois] +rel_dataclasses = {oi: rel_dataclasses[oi] for oi in rel_obsids} xtd_pipe_problem_ois ``` +We also include a code snippet that will print the output of the `xtdpipeline` run for any +observations that appear to have failed: + +```{code-cell} python +if len(xtd_pipe_problem_ois) != 0: + for all_out in pipe_result: + if all_out[0] in xtd_pipe_problem_ois: + print(all_out[2]) + print("\n\n") +``` + ```{warning} Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` From f8e2b33bcd5563e1720cb647b1e50d6f574cb541 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 10:42:18 -0500 Subject: [PATCH 151/221] Fixed some small bugs with xtdpipeline error printing, and the reading in of a regions file from disk into the 'regions' module. In the getting-started-xrism-xtend.md notebook. --- .../xrism/getting-started-xrism-xtend.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index fc982938..733421fc 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1227,7 +1227,7 @@ observations that appear to have failed: if len(xtd_pipe_problem_ois) != 0: for all_out in pipe_result: if all_out[0] in xtd_pipe_problem_ois: - print(all_out[2]) + print(all_out[1]) print("\n\n") ``` @@ -1725,7 +1725,7 @@ them, they will appear as a different color than the source and background regio ```{code-cell} python cal_regs = {} for oi in rel_obsids: - cur_cal_regs = Regions(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") + cur_cal_regs = Regions.read(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") for cur_reg in cur_cal_regs: cur_reg.visual["color"] = "white" From 4095749c9d93fdd6d207e7491ceb96107990982b Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 10:43:51 -0500 Subject: [PATCH 152/221] Made some changes to the getting-started-xrism-xtend.md notebook where it writes 'final' source and background region files in both RA-Dec and Sky-XY coordinate systems. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 733421fc..5d5bdfea 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1840,16 +1840,15 @@ for oi in rel_obsids: # Write the RA-Dec source region file src_comb_regs.write(radec_src_reg_path.format(oi=oi), format="ds9", overwrite=True) # And the RA-Dec background region file - back_reg.write(radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True) + back_comb_regs.write(radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True) # Now we repeat the exercise, but use the Sky<->RA-Dec WCS information we pulled # from the images to convert the regions to the Sky X-Y system - src_comb_regs.to_pixel(oi_skypix_wcs[oi]).write( - obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True - ) - back_comb_regs.to_pixel(oi_skypix_wcs[oi]).write( - obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True - ) + src_comb_skyXY_regs = Regions([r.to_pixel(oi_skypix_wcs[oi]) for r in src_comb_regs]) + back_comb_skyXY_regs = Regions([r.to_pixel(oi_skypix_wcs[oi]) for r in back_comb_regs]) + + src_comb_skyXY_regs.write(obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True) + back_comb_skyXY_regs.write(obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True) ``` ```{tip} From 734a86b4053e1045bb59f3b8af58f7ad7a844810 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 8 Dec 2025 15:44:25 +0000 Subject: [PATCH 153/221] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- .../xrism/getting-started-xrism-xtend.md | 22 ++++++++++++++----- 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 5d5bdfea..8ebd4fc5 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1840,15 +1840,25 @@ for oi in rel_obsids: # Write the RA-Dec source region file src_comb_regs.write(radec_src_reg_path.format(oi=oi), format="ds9", overwrite=True) # And the RA-Dec background region file - back_comb_regs.write(radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True) + back_comb_regs.write( + radec_back_reg_path.format(oi=oi), format="ds9", overwrite=True + ) # Now we repeat the exercise, but use the Sky<->RA-Dec WCS information we pulled # from the images to convert the regions to the Sky X-Y system - src_comb_skyXY_regs = Regions([r.to_pixel(oi_skypix_wcs[oi]) for r in src_comb_regs]) - back_comb_skyXY_regs = Regions([r.to_pixel(oi_skypix_wcs[oi]) for r in back_comb_regs]) - - src_comb_skyXY_regs.write(obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True) - back_comb_skyXY_regs.write(obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True) + src_comb_skyXY_regs = Regions( + [r.to_pixel(oi_skypix_wcs[oi]) for r in src_comb_regs] + ) + back_comb_skyXY_regs = Regions( + [r.to_pixel(oi_skypix_wcs[oi]) for r in back_comb_regs] + ) + + src_comb_skyXY_regs.write( + obs_src_reg_path_temp.format(oi=oi), format="ds9", overwrite=True + ) + back_comb_skyXY_regs.write( + obs_back_reg_path_temp.format(oi=oi), format="ds9", overwrite=True + ) ``` ```{tip} From ac704eb920811656d44fdd6b16ee9bd3a4ed582f Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 11:56:42 -0500 Subject: [PATCH 154/221] Made sure all the coordinates that go into region files are in the ICRS file - that stops issues with a HEASArc tool trying to read coordinates in different frames and falling over. In getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 8ebd4fc5..86ab3d5f 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -974,7 +974,7 @@ You could also set up a SkyCoord object directly, if you already know the coordi ``` ```{code-cell} python -src_coord = SkyCoord.from_name(SRC_NAME) +src_coord = SkyCoord.from_name(SRC_NAME).transform_to("icrs") # This will be useful later on in the notebook, for functions that take # coordinates as an astropy Quantity. src_coord_quant = Quantity([src_coord.ra, src_coord.dec]) @@ -1728,6 +1728,9 @@ for oi in rel_obsids: cur_cal_regs = Regions.read(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") for cur_reg in cur_cal_regs: cur_reg.visual["color"] = "white" + # Make sure the frame is consistent with the source/back regions later, as + # otherwise HEASoft tools will get confused + cur_reg.center = cur_reg.center.transform_to("icrs") # The '.regions' just retrieves a list of region objects, we don't need to keep # the calibration regions in the regions module 'Regions' class they are read into @@ -1754,7 +1757,7 @@ this region is of a different size and is not centered on the source: ```{code-cell} python # The central coordinate of the background region -back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg") +back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg", frame="icrs") # The radius of the background region back_reg_rad = Quantity(3, "arcmin") @@ -2108,7 +2111,7 @@ arg_combs = [ rad=src_reg_rad.to("deg").value.round(4), ), RMF_PATH_TEMP.format(oi=oi, xdc=dc), - radec_src_reg_path, + radec_src_reg_path.format(oi=oi), arf_rt_num_photons, ] for oi, dcs in rel_dataclasses.items() From b55eeea15c032dddac39b39047e29e765b8d50ea Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 12:18:33 -0500 Subject: [PATCH 155/221] Increased notebook execution timeout in the CircleCI config. --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index a5e4a083..7bc63652 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -164,7 +164,7 @@ jobs: # Now we're going to start building the documentation - run: name: Build HTML rendering of notebooks - no_output_timeout: 30m + no_output_timeout: 60m command: | # The first command sets the HEASARC_NOTEBOOKS_TO_BUILD environment variable, which tells the build # process which notebooks to build. It allows granular control over which notebooks are built From d33065ed59b8685025b024523d1253de3eae742c Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 13:36:41 -0500 Subject: [PATCH 156/221] Removed some extraneous formatting arguments in getting-started-xrism-xtend.md --- .../xrism/getting-started-xrism-xtend.md | 21 ++++++++----------- 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 86ab3d5f..ba2d1703 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-05' +date: '2025-12-08' file_format: mystnb jupytext: text_representation: @@ -74,7 +74,7 @@ We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstr ### Runtime -As of 5th December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 8th December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. ## Imports @@ -1889,8 +1889,8 @@ arg_combs = [ os.path.join(OUT_PATH, oi), src_coord, src_reg_rad, - obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), - obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + obs_src_reg_path_temp.format(oi=oi), + obs_back_reg_path_temp.format(oi=oi), *cur_bnds, lc_time_bin, ] @@ -1916,8 +1916,8 @@ arg_combs = [ os.path.join(OUT_PATH, oi), src_coord, src_reg_rad, - obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), - obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + obs_src_reg_path_temp.format(oi=oi), + obs_back_reg_path_temp.format(oi=oi), ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -1929,9 +1929,6 @@ with mp.Pool(NUM_CORES) as p: #### Calculating 'BACKSCAL' for new XRISM-Xtend spectra -***AT THIS POINT THINGS WILL FALL OVER BECAUSE THE REGIONS I DEFINED ARE NOT ON THE 32000010 DATACLASS OBSERVATION OF 000128000*** - - ***TERRIBLE BODGE MUST FIX*** ```{code-cell} python @@ -1968,7 +1965,7 @@ for oi, dcs in rel_dataclasses.items(): # Calculate the BACKSCAL keyword, first for the source spectrum hsp.ahbackscal( infile=cur_spec, - regfile=obs_src_reg_path_temp.format(oi=oi, n=SRC_NAME), + regfile=obs_src_reg_path_temp.format(oi=oi), expfile=cur_ex, logfile="NONE", ) @@ -1976,7 +1973,7 @@ for oi, dcs in rel_dataclasses.items(): # Then for the background spectrum hsp.ahbackscal( infile=cur_bspec, - regfile=obs_back_reg_path_temp.format(oi=oi, n=SRC_NAME), + regfile=obs_back_reg_path_temp.format(oi=oi), expfile=cur_ex, logfile="NONE", ) @@ -2240,7 +2237,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-05 +Updated On: 2025-12-08 +++ From 9b099f0bf7c73c72aea5bc6cb4275163c31d91f9 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 8 Dec 2025 16:56:52 -0500 Subject: [PATCH 157/221] Finished explaining the light curve generation section, also moved it and produced net light curve files using the BACKSCAL values calculated for the source and background spectra. All in the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 212 +++++++++++++++--- 1 file changed, 179 insertions(+), 33 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index ba2d1703..a606af32 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -917,6 +917,13 @@ BACK_LC_PATH_TEMP = os.path.join( "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-" "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-back-lightcurve.fits", ) + +NET_LC_PATH_TEMP = os.path.join( + OUT_PATH, + "{oi}", + "xrism-xtend-obsid{oi}-dataclass{xdc}-ra{ra}-dec{dec}-radius{rad}deg-" + "en{lo}_{hi}keV-expthresh{lct}-tb{tb}s-net-lightcurve.fits", +) # -------------------------- @@ -1871,42 +1878,9 @@ However, different data classes represent different pairs of Xtend CCDs, so ther no shared sky coverage. ``` -### New XRISM-Xtend light curves - -```{code-cell} python -lc_time_bin = Quantity(200, "s") -``` - -```{code-cell} python -# Defining the various energy bounds we want to make light curves for -xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") -``` - -```{code-cell} python -arg_combs = [ - [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), - os.path.join(OUT_PATH, oi), - src_coord, - src_reg_rad, - obs_src_reg_path_temp.format(oi=oi), - obs_back_reg_path_temp.format(oi=oi), - *cur_bnds, - lc_time_bin, - ] - for oi, dcs in rel_dataclasses.items() - for dc in dcs - for cur_bnds in xtd_lc_en_bounds -] - -with mp.Pool(NUM_CORES) as p: - lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) -``` ### New XRISM-Xtend spectra and supporting files - - #### Generating the spectral files ```{code-cell} python @@ -1929,6 +1903,10 @@ with mp.Pool(NUM_CORES) as p: #### Calculating 'BACKSCAL' for new XRISM-Xtend spectra +Our calculation of 'BACKSCAL' doesn't only benefit our spectra analyses, as when we +demonstrate the creation of light curves later in this notebook, we can also use +the values to weight our subtraction of the background. + ***TERRIBLE BODGE MUST FIX*** ```{code-cell} python @@ -1936,6 +1914,9 @@ rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} ``` ```{code-cell} python +spec_backscals = {oi: {dc: 0 for dc in rel_dataclasses[oi]} for oi in rel_obsids} +bspec_backscals = {oi: {dc: 0 for dc in rel_dataclasses[oi]} for oi in rel_obsids} + for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: # Set up the path to input source and background spectra @@ -1977,6 +1958,15 @@ for oi, dcs in rel_dataclasses.items(): expfile=cur_ex, logfile="NONE", ) + + # For good measure, and because we're going to need them later for + # net light curve calculation, we read the backscal values into Python + # First, the source spectrum + with fits.open(cur_spec) as src_specco: + spec_backscals[oi][cur_dc] = src_specco["SPECTRUM"].header["BACKSCAL"] + # Now the background + with fits.open(cur_bspec) as back_specco: + bspec_backscals[oi][cur_dc] = back_specco["SPECTRUM"].header["BACKSCAL"] ``` #### Grouping our new spectra @@ -2124,6 +2114,162 @@ Due to the high-fidelity ray-tracing method used to calculate XRISM ARFs, the ru of this step can be on the order of hours. ``` +### New XRISM-Xtend light curves + +Now we can quickly demonstrate how to generate XRISM-Xtend light curves - it is +rather simpler than the creation of new spectra. + +There is no XRISM-Xtend-specific task for the generation of light curves, so we +once again turn to HEASoft's `extractor` tool (we used it to create XRISM-Xtend +images in Section 3). + +By providing a slightly different set of inputs to `extractor`, we can tell it to +bin the cleaned event lists in time, rather than in space, and thus produce a +light curve. + +We'll make sure to generate light curves within the source and background regions +that we defined in the previous part of this section, which we can then use to +produce net light curves for our source. + +The primary input we need to provide is the time step, or time bin size, which +controls the temporal resolution of the output light curve. This uniform sampling is +the simplest method of dividing an event list into a light curve, but other methods +exist (requiring each time bin to reach a minimum signal-to-noise for instance). + +Your choice of uniform time bin size will depend on your particular science case, and +practical considerations based on the length of the overall observation and the +observed count-rate of the source. + +```{code-cell} python +lc_time_bin = Quantity(200, "s") +``` + +It is also very common to want to specify the events included in each by setting +lower and upper energy bounds - this may allow you to focus on light emitted by +a particular process you're interested in, or to exclude energy bands that are +not relevant to your science case. + +We define three energy bands that cover much of the useful energy range of the +XRISM-Xtend instrument: +- 0.6-2.0 keV +- 2.0-6.0 keV +- 6.0-10.0 keV + +Though again, your choices will depend on what you're trying to learn. + +```{code-cell} python +# Defining the various energy bounds we want to make light curves for +xtd_lc_en_bounds = Quantity([[0.6, 2.0], [2.0, 6.0], [6.0, 10.0]], "keV") +``` + +#### Generating source and background light curves + +Using another wrapper function around the HEASoftPy interface to `extractor`, we can +now generate the light curves within the source and background regions, for each of +the specified energy bands. + +As with previous steps, our motivation for writing a wrapper function (defined in the +Global Setup section) is to make it easy for us to run generation of different +light curves simultaneously: + +```{code-cell} python +arg_combs = [ + [ + evt_path_temp.format(oi=oi, xdc=dc, sc=0), + os.path.join(OUT_PATH, oi), + src_coord, + src_reg_rad, + obs_src_reg_path_temp.format(oi=oi), + obs_back_reg_path_temp.format(oi=oi), + *cur_bnds, + lc_time_bin, + ] + for oi, dcs in rel_dataclasses.items() + for dc in dcs + for cur_bnds in xtd_lc_en_bounds +] + +with mp.Pool(NUM_CORES) as p: + lc_result = p.starmap(gen_xrism_xtend_lightcurve, arg_combs) +``` + +#### Calculating net light curves + +Unlike with the spectra we generated earlier in this section, we will produce 'net' +light curves, with the background light curve scaled and subtracted from the source. + +The applied scaling is to effectively normalize the area within which the background +light curve was extracted to the source light curve. + +We have already performed the measurement of the extraction regions for source and +background products - when we used HEASoft to calculate the 'BACKSCAL' keyword after +our spectra were generated. + +At the time we made sure to read those 'BACKSCAL' values into Python, specifically +for this purpose. + +With the scaling known, we can use the HEASoft `lcmath` tool (through the HEASoftPy +interface) to subtract the background from the source light curve. This operation is +computationally cheap for the number of light curves we are working with, but you +should consider parallelizing this step if you are working with significantly more: + +```{code-cell} python +for oi, dcs in rel_dataclasses.items(): + for cur_dc in dcs: + for cur_bnds in xtd_lc_en_bounds: + # Constructing the file paths to the source and background light curves + cur_lc = LC_PATH_TEMP.format( + oi=oi, + xdc=dc, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), + lo=cur_bnds[0].value, + hi=cur_bnds[1].value, + tct=0.0, + tb=lc_time_bin.to("s").value, + ) + + cur_blc = BACK_LC_PATH_TEMP.format( + oi=oi, + xdc=dc, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + lo=cur_bnds[0].value, + hi=cur_bnds[1].value, + tct=0.0, + tb=lc_time_bin.to("s").value, + ) + + # Now we construct the output file path for the final net light curve + cur_nlc = NET_LC_PATH_TEMP.format( + oi=oi, + xdc=dc, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), + lo=cur_bnds[0].value, + hi=cur_bnds[1].value, + tct=0.0, + tb=lc_time_bin.to("s").value, + ) + + # The 'lcmath' tool is sensitive to long paths, so we fetch the relative + # paths to pass in + cur_lc = os.path.relpath(cur_lc) + cur_blc = os.path.relpath(cur_blc) + cur_nlc = os.path.relpath(cur_nlc) + + # Calculate the scaling that should be applied to the background + # light curve before subtraction + cur_back_multi = spec_backscals[oi][cur_dc] / bspec_backscals[oi][cur_dc] + + # Run the tool to produce a net light curve + hsp.lcmath( + infile=cur_lc, bgfile=cur_blc, outfile="", multi=1, multb=cur_back_multi + ) +``` + ## 5. Fitting spectral models to XRISM-Xtend spectra Finally, to show off the XRISM-Xtend products we just generated, we will perform From b1f0b49d71b0db3e6dec71604eb674f382a6f1a5 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 10:52:07 -0500 Subject: [PATCH 158/221] Added a HEASoft version check to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 35 +++++++++++++++++-- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a606af32..a9c50ebd 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -96,6 +96,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc +from packaging.version import Version from regions import CircleSkyRegion, Regions from xga.products import Image ``` @@ -1085,16 +1086,40 @@ on a regular basis, but please report any issues or suggestions to the HEASARC H Both the HEASoft and HEASoftPy package versions can be retrieved from the HEASoftPy module. +The HEASoftPy version: + +```{code-cell} python +hsp.__version__ +``` + The HEASoft version: ```{code-cell} python -hsp.fversion() +fver_out = hsp.fversion() +fver_out ``` -The HEASoftPy version: +It is likely that this tutorial will not run all the way through if you are using +a version of HEASoft older than **v6.36**, so we will check for that and raise an +error if it is the case. First, extract the version string from the `fversion` output, and +set up a `Version` object: ```{code-cell} python -hsp.__version__ +fver_out.output[0].split("_")[-1] +HEA_VER = Version(fver_out.output[0].split("_")[-1]) +HEA_VER +``` + +Now we can use inequality operators on another Version object to check if our version +of HEASoft meets the requirements: + +```{code-cell} python +if HEA_VER < Version("V6.36"): + raise ValueError( + "We strongly recommend using HEASoft v6.36 or later for this " + "tutorial - you may run rest of the notebook yourself, but " + "ARF generation will likely fail." + ) ``` ### Setting up file paths to pass to the XRISM-Xtend pipeline @@ -1903,6 +1928,10 @@ with mp.Pool(NUM_CORES) as p: #### Calculating 'BACKSCAL' for new XRISM-Xtend spectra +Spectral data products generated for high-energy missions typically contain a +measurement of their extraction region area. This is in order to scale source and +background spectra properly when + Our calculation of 'BACKSCAL' doesn't only benefit our spectra analyses, as when we demonstrate the creation of light curves later in this notebook, we can also use the values to weight our subtraction of the background. From 5b4a8d5a406064cc17dd8e5bf90c49d048ef758a Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 13:12:19 -0500 Subject: [PATCH 159/221] Added a very short, but top level, section about what do with observations that have multiple dataclasses - in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 123 ++++++++++++++++-- 1 file changed, 109 insertions(+), 14 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a9c50ebd..bd247fa7 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1460,6 +1460,7 @@ any problems or unusual features of the prepared and cleaned observations. #### Image energy bounds We are going to generate images within the following energy bounds: +- 0.6-10.0 keV - 0.6-2.0 keV - 2.0-10.0 keV - ***0.4-2.0 keV*** [not recommended] @@ -1471,7 +1472,9 @@ demonstrate those issues. ```{code-cell} python # Defining the energy bounds we want images within -xtd_im_en_bounds = Quantity([[0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV") +xtd_im_en_bounds = Quantity( + [[0.6, 10.0], [0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV" +) ``` Converting those energy bounds to channel bounds is straightforward, we simply divide @@ -1553,10 +1556,6 @@ with mp.Pool(NUM_CORES) as p: im_result = p.starmap(gen_xrism_xtend_image, arg_combs) ``` -#### Problematic pixels - - - ### New XRISM-Xtend exposure maps We also generate exposure maps for the entire FoV of a particular observation mode, rather @@ -1627,7 +1626,89 @@ with mp.Pool(NUM_CORES) as p: ex_result = p.starmap(gen_xrism_xtend_expmap, arg_combs) ``` -## 4. Generating new XRISM-Xtend spectra and light curves +## 4. Handling observations with multiple dataclasses + +As we've already mentioned, the XRISM-Xtend detector can operate in two different +data modes simultaneously, with the read-out area of some of the CCDs restricted in +order to increase the read-out speed and improve timing resolution. + +If the proposer of the observation has requested an increased-timing-resolution +mode, then odds are the target of their observation will be placed on that +fast-timing-mode portion of the detector. + +As such, if that is the target you are also interested in (whether you are the +original proposer or are performing some archival analysis), then the other half +of the XRISM-Xtend detector may not be that interesting to you. + +In such cases we will have to decide which dataclass we're going to use from +this point onwards. + +The first part of the next section of this demonstration will show you how to set up +the regions from which source and background data products will be extracted. + +Event lists from the same observation with different dataclasses are taken from +different halves of the detector and are not co-aligned. That means the source region +we're about to set up for our target will not overlap with the coverage of the other +dataclass. + +That would result in empty light curve and spectrum products, and errors +from 'BACKSCAL' calculation, and RMF and ARF generation. + + +### Visualize separate XRISM-Xtend 000128000 dataclass images + +```{code-cell} python +fast_im_path = IM_PATH_TEMP.format( + oi="000128000", xdc="31100010", lo="0.6", hi="10.0", ibf=1 +) +half_im_path = IM_PATH_TEMP.format( + oi="000128000", xdc="32000010", lo="0.6", hi="10.0", ibf=1 +) + +fast_im = Image( + fast_im_path, + "000128000", + "Xtend", + "", + "", + "", + Quantity(0.6, "keV"), + Quantity(10.0, "keV"), +) +half_im = Image( + half_im_path, + "000128000", + "Xtend", + "", + "", + "", + Quantity(0.6, "keV"), + Quantity(10.0, "keV"), +) +``` + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +fig, ax_arr = plt.subplots(ncols=2, figsize=(6, 6), width_ratios=[3, 1]) +plt.subplots_adjust(wspace=0) + +half_im.get_view(ax_arr[0], zoom_in=True, custom_title="32000010") +fast_im.get_view(ax_arr[1], src_coord_quant, zoom_in=True, custom_title="31100010") + +plt.show() +``` + +### Choosing our dataclasses + +```{code-cell} python +rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} +``` + +## 5. Generating new XRISM-Xtend spectra and light curves In this section we will demonstrate how to generate source-specific data products from XRISM-Xtend observations; light curves and spectra (along with supporting files like @@ -1903,11 +1984,31 @@ However, different data classes represent different pairs of Xtend CCDs, so ther no shared sky coverage. ``` - ### New XRISM-Xtend spectra and supporting files +With the source and background regions defined, we have everything we need to generate +XRISM-Xtend spectra for our target source. + +Though there is no XRISM-Xtend-specific HEASoft task for spectrum generation, we can +use the same tool that we used to create our images in Section 3, and that we will use +later in this section to make light curves - `extractor`. + +Spectral files are just as simple a data product as images and light curves. All we're +doing to make a spectrum is binning the events in our extraction regions in a 1D +channel space (which will become energy), rather than X-Y, or time. The complexities +of spectral generation come when creating the supporting files (RMFs and ARFs) +required to actually fit models to the spectra. + #### Generating the spectral files +We set up a spectrum-generation-specific region wrapper function for the HEASoftPy +interface to the `extractor` task (see the Global Setup: Functions section near the +top of the notebook). + +This once again allows us to parallelize the generation of spectra for different +observations, though it is worth noting we aren't producing multiple variants of the +spectrum like we did with the different-energy-band images. + ```{code-cell} python arg_combs = [ [ @@ -1936,12 +2037,6 @@ Our calculation of 'BACKSCAL' doesn't only benefit our spectra analyses, as when demonstrate the creation of light curves later in this notebook, we can also use the values to weight our subtraction of the background. -***TERRIBLE BODGE MUST FIX*** - -```{code-cell} python -rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} -``` - ```{code-cell} python spec_backscals = {oi: {dc: 0 for dc in rel_dataclasses[oi]} for oi in rel_obsids} bspec_backscals = {oi: {dc: 0 for dc in rel_dataclasses[oi]} for oi in rel_obsids} @@ -2299,7 +2394,7 @@ for oi, dcs in rel_dataclasses.items(): ) ``` -## 5. Fitting spectral models to XRISM-Xtend spectra +## 6. Fitting spectral models to XRISM-Xtend spectra Finally, to show off the XRISM-Xtend products we just generated, we will perform a simple model fit to one of our spectra. From b756c1825742bbbe98d6052ac42144f6c9524069 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 13:23:19 -0500 Subject: [PATCH 160/221] Finished the dataclass choice section for now. --- .../xrism/getting-started-xrism-xtend.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index bd247fa7..f57c2763 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1657,14 +1657,16 @@ from 'BACKSCAL' calculation, and RMF and ARF generation. ### Visualize separate XRISM-Xtend 000128000 dataclass images +To make our point, and to give an example of the inspection you may want to perform +before choosing the right dataclass for your target, we will visualize +the 0.6-10.0 keV images we generated for 000128000 in the last section: + ```{code-cell} python +# Set up the path to the image, and XGA Image class instance, for the '31100010' +# dataclass, which is the small-window, fast-readout, mode fast_im_path = IM_PATH_TEMP.format( oi="000128000", xdc="31100010", lo="0.6", hi="10.0", ibf=1 ) -half_im_path = IM_PATH_TEMP.format( - oi="000128000", xdc="32000010", lo="0.6", hi="10.0", ibf=1 -) - fast_im = Image( fast_im_path, "000128000", @@ -1675,6 +1677,12 @@ fast_im = Image( Quantity(0.6, "keV"), Quantity(10.0, "keV"), ) + +# Set up the path to the image, and XGA Image class instance, for the '32000010' +# dataclass, which is rest of the Xtend detector running as normal +half_im_path = IM_PATH_TEMP.format( + oi="000128000", xdc="32000010", lo="0.6", hi="10.0", ibf=1 +) half_im = Image( half_im_path, "000128000", @@ -1687,6 +1695,8 @@ half_im = Image( ) ``` +We can quite clearly see that the source of interest, indicated by the +cross-hair, is on the small-window, fast-readout, '31100010' dataclass image: ```{code-cell} python --- tags: [hide-input] @@ -1704,10 +1714,18 @@ plt.show() ### Choosing our dataclasses +Based on our inspection, we choose the right dataclass for the '000128000' observation: ```{code-cell} python rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} ``` +```{hint} +Manual inspection is far from the only way of making this choice - it would be easy to +automate this process by, for instance, retrieving the exposure value corresponding to +the source position from the exposure map; a non-zero exposure for a particular dataclass +would indicate that it contains your source. +``` + ## 5. Generating new XRISM-Xtend spectra and light curves In this section we will demonstrate how to generate source-specific data products from From de9baec02d7efba7fdb84890fa4cd4b819d24277 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 15:01:46 -0500 Subject: [PATCH 161/221] Added commentary --- .../xrism/getting-started-xrism-xtend.md | 36 +++++++++---------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f57c2763..2df16064 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2169,7 +2169,22 @@ for oi, dcs in rel_dataclasses.items(): #### Generating XRISM-Xtend RMFs -***THIS IS ALSO GOING TO FALL OVER IN PART BECAUSE CAN'T EXTRACT SPECTRUM FROM REGIONS NOT ON THE 31100010 DATACLASS OBSERVATION OF 000128000*** +In order for the spectral model fitting software of our choice (XSPEC, for +instance) to be able to map the channels of a spectrum to an energy, we need to +generate RMFs. + +We already discussed RMFs, and even used them to perform our own conversion between +XRISM-Xtend spectral channels and energy, in Section 3 - there we used the RMF that +was included in the original data download. + +Now we wish to generate new RMFs, to ensure they are entirely up-to-date! + +We make use of the XRISM-Xtend specific HEASoft task `xtdrmf` - the only input it +requires is the path to the spectral file for which we wish to generate an RMF. + +Our `gen_xrism_xtend_rmf` function (defined in the Global Setup: Functions section near +the top of the notebook) wraps the HEASoftPy interface to the `xtdrmf` task. We now use +it to generate RMFs in parallel for all of our new spectra: ```{code-cell} python arg_combs = [ @@ -2191,12 +2206,6 @@ with mp.Pool(NUM_CORES) as p: rmf_result = p.starmap(gen_xrism_xtend_rmf, arg_combs) ``` -#### Ray-tracing simulated events in preparation for XRISM-Xtend ARF generation - -```{code-cell} python - -``` - #### Generating XRISM-Xtend ARFs ```{danger} @@ -2205,21 +2214,12 @@ another, very similarly named, HEASoft tool related to the construction of XRISM ARFs, **`xaxmaarfgen`**. Be sure which one you are using! ``` +ARFs are the final type of supporting file required to make our spectra usable + ```{code-cell} python arf_rt_num_photons = 20000 ``` -```{warning} -***MIGHT BE WRONG, BUT v6.35.2 OF HEASOFT MIGHT POINT TO THE WRONG CALDB FILE PATH MIRROR -SCATTER INFORMATION USED BY RAYTRACE (xa_xtd_scatter_20190101v001.fits RATHER THAN xa_xtd_scatter_20190101v001.fits.gz)?*** - -***HEASOFT RELEASE NOTES SEEM TO BEAR THIS IDEA UP***: - -xrtraytrace: Additional updates to fix and enable remote CalDB - usage with xrtraytrace and xaarfgen ("timeout" interval extended - for reading large CalDB files). - -``` ```{code-cell} python arg_combs = [ From 09a62ebd5327db390f090d203d5a037f25db9ae5 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 15:50:11 -0500 Subject: [PATCH 162/221] Have now written commentary on the ARF generation section of the getting-started-xrism-xtend.md notebook. THIS MIGHT BE BASICALLY DONE AAAARGH. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 69 ++++++++++++++++++- 1 file changed, 66 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 2df16064..7944f27a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2214,12 +2214,75 @@ another, very similarly named, HEASoft tool related to the construction of XRISM ARFs, **`xaxmaarfgen`**. Be sure which one you are using! ``` -ARFs are the final type of supporting file required to make our spectra usable +ARFs are the final type of supporting file required to make our spectra usable and +describe the effective area (i.e., the sensitivity) of XRISM-Xtend as a function of +energy. + +The effective area has to be understood (and well calibrated) as we need it to help +map a spectral model, which hopefully describes what the object of interest +is _actually_ emitting (and how), to the _observed_ spectrum; that observed spectrum +has been altered across its energy range by how good XRISM-Xtend is at detecting +photons at different points in that range. + +The sensitivity of an X-ray detector is a combination of the effective area of the +actual X-ray optics (on XRISM this is the called X-ray Mirror Assembly, or XMA), and +the detector's quantum efficiency. They are both independently a function of +energy. + +ARFs are standard products for most high-energy missions, but the methods implemented +to calculate them for XRISM's instruments are quite unusual. + +The HEASoft task we need to call (`xaarfgen`) calls further HEASoft tools that perform +ray-tracing simulations of XRISM XMAs, for the location of your source on the +detector, and use those to define the X-ray optic's collecting area for a wide range +of energies. + +```{note} +If you have to generate multiple ARFs for the same source, in the same observation, you +should be aware that the raytraced event lists can be re-used (though only in this +particular scenario). +``` + +Raytracing can be a slow process, as individual events and their path through the +XMA are being simulated, but it does help to produce very accurate ARFs. There are ways +that it can be sped up, though at the cost of that accuracy - the most direct way is +to limit the number of events that are simulated. + +An argument specifying the number of events can be passed to `xaarfgen`, and for +our demonstration we are going to use a very small sample - this is primarily so the +notebook can run in a reasonable amount of time. + +**You should likely use a larger value for real analysis!** + +```{admonition} +The minimum number of raytracing photons that successfully reach the focal plane, per +raytracing energy grid point, that is acceptable to make an ARF. The number of focal +plane photons that contribute to the ARF must exceed 'minphoton' for every +energy, otherwise the program aborts. Note that even if minphoton is exceeded at +all energies, this does not guarantee that the resulting ARF is robust and +sufficiently accurate. In general, about 5000 or more photons per energy (over the +extraction region) give good results, but the actual minimum number varies +case-by-case, and fewer may be sufficient in some cases. The default value of +minphoton is deliberately very small, in order that the ARF is made and available +for diagnostic evaluation. In general, it is not recommended to set 'minphoton' to a +high value in the first place, because it is not possible to reliably estimate +what 'numphoton' should be in advance of running raytracing within xaarfgen, in order +for that value of 'minphoton' to be satisfied for all energies, which could result +in repeated failures after very long run times. It could also run into memory +problems and/or a raytracing file size that is unmanageable. +``` + +The xaarfgen documentation provides the following guidance on choosing the number of +events to simulate: ```{code-cell} python arf_rt_num_photons = 20000 ``` +So now we move onto actually running the ARF generation - using the +`gen_xrism_xtend_arf` function defined in the Global Setup: Functions section (near the top of +the notebook), which wraps the HEASoftPy interface to the `xaarfgen` task. We now use it +to generate ARFs in parallel for all of our new spectra: ```{code-cell} python arg_combs = [ @@ -2247,8 +2310,8 @@ arg_combs = [ for dc in dcs ] -# with mp.Pool(NUM_CORES) as p: -# arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) +with mp.Pool(NUM_CORES) as p: + arf_result = p.starmap(gen_xrism_xtend_arf, arg_combs) ``` ```{warning} From a1213d57a9ecfeb68da3b05e6c93142b8377170b Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 20:22:52 -0500 Subject: [PATCH 163/221] May have finally finished the writing of the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 207 ++++++++++++------ 1 file changed, 143 insertions(+), 64 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 7944f27a..83731265 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-08' +date: '2025-12-10' file_format: mystnb jupytext: text_representation: @@ -1697,6 +1697,7 @@ half_im = Image( We can quite clearly see that the source of interest, indicated by the cross-hair, is on the small-window, fast-readout, '31100010' dataclass image: + ```{code-cell} python --- tags: [hide-input] @@ -1715,6 +1716,7 @@ plt.show() ### Choosing our dataclasses Based on our inspection, we choose the right dataclass for the '000128000' observation: + ```{code-cell} python rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} ``` @@ -2254,6 +2256,8 @@ notebook can run in a reasonable amount of time. **You should likely use a larger value for real analysis!** +The xaarfgen documentation provides the following guidance on choosing the number of +events to simulate: ```{admonition} The minimum number of raytracing photons that successfully reach the focal plane, per raytracing energy grid point, that is acceptable to make an ARF. The number of focal @@ -2272,9 +2276,6 @@ in repeated failures after very long run times. It could also run into memory problems and/or a raytracing file size that is unmanageable. ``` -The xaarfgen documentation provides the following guidance on choosing the number of -events to simulate: - ```{code-cell} python arf_rt_num_photons = 20000 ``` @@ -2475,7 +2476,7 @@ for oi, dcs in rel_dataclasses.items(): ) ``` -## 6. Fitting spectral models to XRISM-Xtend spectra +## 6. Fitting a spectral model to an XRISM-Xtend spectrum Finally, to show off the XRISM-Xtend products we just generated, we will perform a simple model fit to one of our spectra. @@ -2494,92 +2495,168 @@ Now we configure some behaviors of XSPEC/pyXspec: # The strange comment on the end of this line is for the benefit of our # automated code-checking processes. You shouldn't import modules anywhere but # the top of your file, but this is unfortunately necessary at the moment -# import xspec as xs # noqa: E402 +import xspec as xs # noqa: E402 -# xs.Xset.chatter = 0 +xs.Xset.chatter = 0 # XSPEC parallelisation settings -# xs.Xset.parallel.leven = NUM_CORES -# xs.Xset.parallel.error = NUM_CORES -# xs.Xset.parallel.steppar = NUM_CORES -# -# # Other xspec settings -# xs.Plot.area = True -# xs.Plot.xAxis = "keV" -# xs.Plot.background = True -# xs.Fit.statMethod = "cstat" -# xs.Fit.query = "no" -# xs.Fit.nIterations = 500 +xs.Xset.parallel.leven = NUM_CORES +xs.Xset.parallel.error = NUM_CORES +xs.Xset.parallel.steppar = NUM_CORES + +# Other xspec settings +xs.Plot.area = True +xs.Plot.xAxis = "keV" +xs.Plot.background = True +xs.Fit.statMethod = "cstat" +xs.Fit.query = "no" +xs.Fit.nIterations = 500 +``` + +```{danger} +There is a known issue with the version of PyXspec shipped in HEASoft v6.36 (and +possibly later versions) that will cause the parallelised generation of data products +to hang forever. We avoid this here by importing PyXspec **after** all data product +generation is complete. ``` -### Reading XRISM-Xtend spectra into pyXspec +### Reading a XRISM-Xtend spectrum into pyXspec + +Here we define the ObsID and dataclass of the spectrum we want to fit: ```{code-cell} python chosen_demo_spec_obsid = "000128000" chosen_demo_spec_dataclass = "31100010" ``` +The spectrum, and all of its supporting files, are then read into pyXspec: + ```{code-cell} python # In case this cell is re-run, clear all previously loaded spectra -# xs.AllData.clear() -# -# # Set up the paths to grouped source spectrum, ungrouped background -# # spectrum, RMF, and ARF files -# cur_spec = GRP_SP_PATH_TEMP.format( -# oi=chosen_demo_spec_obsid, -# xdc=chosen_demo_spec_dataclass, -# gt=spec_group_type, -# gs=spec_group_scale, -# ra=src_coord.ra.value.round(6), -# dec=src_coord.dec.value.round(6), -# rad=src_reg_rad.to("deg").value.round(4), -# ) -# -# cur_bspec = BACK_SP_PATH_TEMP.format( -# oi=chosen_demo_spec_obsid, -# xdc=chosen_demo_spec_dataclass, -# ra=src_coord.ra.value.round(6), -# dec=src_coord.dec.value.round(6), -# ) -# -# cur_rmf = RMF_PATH_TEMP.format( -# oi=chosen_demo_spec_obsid, -# xdc=chosen_demo_spec_dataclass, -# ) -# -# cur_arf = ARF_PATH_TEMP.format( -# oi=chosen_demo_spec_obsid, -# xdc=chosen_demo_spec_dataclass, -# ra=src_coord.ra.value.round(6), -# dec=src_coord.dec.value.round(6), -# rad=src_reg_rad.to("deg").value.round(4), -# ) -# -# # Load the chosen spectrum (and all its supporting files) into pyXspec -# xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) +xs.AllData.clear() + +# Set up the paths to grouped source spectrum, ungrouped background +# spectrum, RMF, and ARF files +cur_spec = GRP_SP_PATH_TEMP.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + gt=spec_group_type, + gs=spec_group_scale, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), +) + +cur_bspec = BACK_SP_PATH_TEMP.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), +) + +cur_rmf = RMF_PATH_TEMP.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, +) + +cur_arf = ARF_PATH_TEMP.format( + oi=chosen_demo_spec_obsid, + xdc=chosen_demo_spec_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), +) + +# Load the chosen spectrum (and all its supporting files) into pyXspec +xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) ``` ### Restricting the spectral channels used for fitting +When we analyze a spectrum by fitting a model, we often want to apply lower and +upper energy limits in order to fit the model on only a subset of the data points. + +Restricting the spectrum data points by energy allows us to cut out parts of the +spectrum that, for instance, have very low signal-to-noise, aren't relevant to our +science case, or fall outside the optimal energy range of the instrument. + +Remember that XRISM-Xtend data are not currently trustworthy around or below 0.4 keV, so +we definitely want to exclude that part of the energy range. If we didn't we would be +in danger of biasing our model fitting results, leading to unreliable or unphysical +conclusions about our source of interest. + +Here, we only make use of channels within a 0.5-10.0 keV energy range, and we also +ignore any channels that have been marked as 'bad' by any previous processing steps: + ```{code-cell} python -# xs_spec.ignore("**-0.5 12.0-**") -# -# # Ignore any channels that have been marked as 'bad' -# # This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra -# # have been declared -# xs.AllData.ignore("bad") +xs_spec.ignore("**-0.5 10.0-**") + +# Ignore any channels that have been marked as 'bad' +# This CANNOT be done on a spectrum-by-spectrum basis, only after all spectra +# have been declared +xs.AllData.ignore("bad") ``` ### Setting up a spectral model +Now we choose the spectral model we want to fit to our spectrum. + +A full list of XSPEC model components can be found in the [XSPEC documentation](https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html). + +Our choice of model is empirically driven, chosen by someone who is not a specialist in supernova remnants, and should definitely not be considered as scientifically useful! + ```{code-cell} python -# xs.Model("tbabs*(powerlaw+apec+bbody)") +xs.Model("tbabs*(powerlaw+apec+bbody)") ``` ### Fitting our pyXspec model to the XRISM-Xtend spectrum +Performing the fit is simple: + +```{code-cell} python +xs.Fit.perform() +``` + +### Visualizing the fitted spectrum + +We want to use matplotlib to visualize the spectrum data, and the model we +just fitted to it. PyXspec allows us to extract the data it would have plotted were +we using XSPEC directly: + ```{code-cell} python -# xs.Fit.perform() +# This populates plot information attributes for the current +# spectrum and model. We can extract that information and +# plot it using matplotlib +xs.Plot() + +# These read the plotting information into handy variables +spec_en = xs.Plot.x() +spec_en_err = xs.Plot.xErr() +spec_cr = xs.Plot.y() +spec_cr_err = xs.Plot.yErr() +``` + +Now we can quite easily produce a plot of the spectrum and model: + +```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- +plt.figure(figsize=(6, 4)) +plt.minorticks_on() +plt.tick_params(which="both", direction="in", top=True, right=True) + +plt.errorbar(spec_en, spec_cr, xerr=spec_en_err, yerr=spec_cr_err, fmt="kx", capsize=2) + +plt.xscale("log") + +plt.xlabel("Energy [keV]", fontsize=15) +plt.ylabel(r"Spectrum [ct cm$^{-2}$ s$^{-1}$ keV$^{-1}$]", fontsize=15) + +plt.tight_layout() +plt.show() ``` ## About this notebook @@ -2588,7 +2665,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-08 +Updated On: 2025-12-10 +++ @@ -2604,6 +2681,8 @@ HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/softwa HEASoft XRISM `xaexpmap` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html +XSPEC Model Components: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html + ### Acknowledgements From a7922df816a52456d6b5a4cc7d0e0647854cb44a Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 20:36:09 -0500 Subject: [PATCH 164/221] Added a test visualization of a light curve in the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 49 ++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 83731265..e8e31d11 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -98,7 +98,7 @@ from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc from packaging.version import Version from regions import CircleSkyRegion, Regions -from xga.products import Image +from xga.products import Image, LightCurve ``` ## Global Setup @@ -2476,6 +2476,47 @@ for oi, dcs in rel_dataclasses.items(): ) ``` +#### Loading and displaying a single light curve + +We take a quick look at one of the light curves we just generated to make sure it +looks sensible. First, we specify the ObsID and dataclass of the light curve we will +use as a demonstration, as well as the energy band: + +```{code-cell} python +chosen_demo_lc_obsid = "000128000" +chosen_demo_lc_dataclass = "31100010" +chosen_demo_lc_bnds = Quantity([0.6, 2.0], "keV") +``` + +Now we set up the path and load the light curve into an XGA LightCurve object, as +it has a convenient method to generate visualizations. You could very easily load +the light curve data in directly, using astropy.io.fits, and then plot it yourself: + +```{code-cell} python +demo_lc_path = NET_LC_PATH_TEMP.format( + oi=chosen_demo_lc_obsid, + xdc=chosen_demo_lc_dataclass, + lo=chosen_demo_lc_bnds[0].value, + hi=chosen_demo_lc_bnds[1].value, + lct=0.0, + tb=lc_time_bin.value, +) +demo_lc = LightCurve( + demo_lc_path, + chosen_demo_lc_obsid, + "Xtend", + "", + "", + "", + src_coord_quant, + Quantity(0, "arcmin"), + Quantity(2.0, "arcmin"), + *chosen_demo_lc_bnds, + lc_time_bin, +) +demo_lc.view() +``` + ## 6. Fitting a spectral model to an XRISM-Xtend spectrum Finally, to show off the XRISM-Xtend products we just generated, we will perform @@ -2497,13 +2538,9 @@ Now we configure some behaviors of XSPEC/pyXspec: # the top of your file, but this is unfortunately necessary at the moment import xspec as xs # noqa: E402 +# Limits the amount of output from XSPEC that pyXspec will display xs.Xset.chatter = 0 -# XSPEC parallelisation settings -xs.Xset.parallel.leven = NUM_CORES -xs.Xset.parallel.error = NUM_CORES -xs.Xset.parallel.steppar = NUM_CORES - # Other xspec settings xs.Plot.area = True xs.Plot.xAxis = "keV" From b85a6ca1dfee9e11b70152ce3e87a5a209e74a4b Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 20:37:46 -0500 Subject: [PATCH 165/221] Fixed a typo --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index e8e31d11..6e9f7e58 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2618,7 +2618,7 @@ spectrum that, for instance, have very low signal-to-noise, aren't relevant to o science case, or fall outside the optimal energy range of the instrument. Remember that XRISM-Xtend data are not currently trustworthy around or below 0.4 keV, so -we definitely want to exclude that part of the energy range. If we didn't we would be +we definitely want to exclude that part of the energy range. If we didn't, then we would be in danger of biasing our model fitting results, leading to unreliable or unphysical conclusions about our source of interest. From 722b4e6002f4084a835bea7843dea407282775aa Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 10 Dec 2025 20:45:46 -0500 Subject: [PATCH 166/221] Restricted the getting-started-xrism-xtend.md notebook to only use 000128000 + added some commentary. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 6e9f7e58..d0752721 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1006,11 +1006,21 @@ For an active mission (i.e., actively collecting data and adding to the archive) public_times = Time(all_xrism_obs["public_date"], format="mjd") avail_xrism_obs = all_xrism_obs[public_times <= Time.now()] -# TODO MAKE MORE PERMANENT ELSEWHERE -avail_xrism_obs = avail_xrism_obs[ - (avail_xrism_obs["obsid"] == "000128000") - | (avail_xrism_obs["obsid"] == "000126000") -] +avail_xrism_obs +``` + +We can see that there are three public XRISM observations of LMC N132D +(as of December 2025) that we could make use of for this demonstration. + +To make sure that this notebook can run in a reasonable amount of time, we +are only going to choose one of them; observation 000128000. + +Please note that we have written this notebook in such a way that you could remove the +first line of the next cell (which selects only one ObsID), and run the notebook +for all public observations. + +```{code-cell} python +avail_xrism_obs = avail_xrism_obs[avail_xrism_obs["obsid"] == "000128000"] # Define a couple of useful variables that make accessing information in the # table a little easier later on in the notebook @@ -1025,8 +1035,6 @@ rel_dataclasses = { ] for oi_ind, oi in enumerate(rel_obsids) } - -avail_xrism_obs ``` ### Downloading the selected XRISM observations @@ -1718,7 +1726,7 @@ plt.show() Based on our inspection, we choose the right dataclass for the '000128000' observation: ```{code-cell} python -rel_dataclasses = {"000128000": ["31100010"], "000126000": ["30000010"]} +rel_dataclasses = {"000128000": ["31100010"]} ``` ```{hint} From 2b5292eddb98a96a5b70069beeccb38755149923 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 09:39:33 -0500 Subject: [PATCH 167/221] Made sure that XSPEC model fit is plotted on top of spectrum, improved the x-axis labelling on the spectrum. Some small bug fixes. All in getting-started-xrism-xtend.md, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 41 +++++++++++++++++-- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index d0752721..85c7d8cb 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -96,6 +96,7 @@ from astropy.table import Table from astropy.time import Time from astropy.units import Quantity, UnitConversionError from astroquery.heasarc import Heasarc +from matplotlib.ticker import FuncFormatter from packaging.version import Version from regions import CircleSkyRegion, Regions from xga.products import Image, LightCurve @@ -2440,7 +2441,7 @@ for oi, dcs in rel_dataclasses.items(): rad=src_reg_rad.to("deg").value.round(4), lo=cur_bnds[0].value, hi=cur_bnds[1].value, - tct=0.0, + lct=0.0, tb=lc_time_bin.to("s").value, ) @@ -2451,7 +2452,7 @@ for oi, dcs in rel_dataclasses.items(): dec=src_coord.dec.value.round(6), lo=cur_bnds[0].value, hi=cur_bnds[1].value, - tct=0.0, + lct=0.0, tb=lc_time_bin.to("s").value, ) @@ -2464,7 +2465,7 @@ for oi, dcs in rel_dataclasses.items(): rad=src_reg_rad.to("deg").value.round(4), lo=cur_bnds[0].value, hi=cur_bnds[1].value, - tct=0.0, + lct=0.0, tb=lc_time_bin.to("s").value, ) @@ -2501,6 +2502,7 @@ it has a convenient method to generate visualizations. You could very easily loa the light curve data in directly, using astropy.io.fits, and then plot it yourself: ```{code-cell} python +# Define the path to the demo net light curve demo_lc_path = NET_LC_PATH_TEMP.format( oi=chosen_demo_lc_obsid, xdc=chosen_demo_lc_dataclass, @@ -2509,6 +2511,8 @@ demo_lc_path = NET_LC_PATH_TEMP.format( lct=0.0, tb=lc_time_bin.value, ) + +# Set up a XGA LightCurve instance demo_lc = LightCurve( demo_lc_path, chosen_demo_lc_obsid, @@ -2522,6 +2526,8 @@ demo_lc = LightCurve( *chosen_demo_lc_bnds, lc_time_bin, ) + +# Show a visualization of the LightCurve demo_lc.view() ``` @@ -2654,6 +2660,15 @@ Our choice of model is empirically driven, chosen by someone who is not a specia xs.Model("tbabs*(powerlaw+apec+bbody)") ``` +If we temporarily increase PyXspec's chatter level, we can see the default values +of each model's parameters: + +```{code-cell} python +xs.Xset.chatter = 10 +xs.AllModels.show() +xs.Xset.chatter = 0 +``` + ### Fitting our pyXspec model to the XRISM-Xtend spectrum Performing the fit is simple: @@ -2662,6 +2677,14 @@ Performing the fit is simple: xs.Fit.perform() ``` +We once again temporarily increase the chatter level, and display the fitted parameters: + +```{code-cell} python +xs.Xset.chatter = 10 +xs.AllModels.show() +xs.Xset.chatter = 0 +``` + ### Visualizing the fitted spectrum We want to use matplotlib to visualize the spectrum data, and the model we @@ -2674,11 +2697,14 @@ we using XSPEC directly: # plot it using matplotlib xs.Plot() -# These read the plotting information into handy variables +# These read out the plotting information for the SPECTRUM spec_en = xs.Plot.x() spec_en_err = xs.Plot.xErr() spec_cr = xs.Plot.y() spec_cr_err = xs.Plot.yErr() + +# And the equivalent for the MODEL +spec_mod_cr = xs.Plot.model() ``` Now we can quite easily produce a plot of the spectrum and model: @@ -2694,9 +2720,16 @@ plt.minorticks_on() plt.tick_params(which="both", direction="in", top=True, right=True) plt.errorbar(spec_en, spec_cr, xerr=spec_en_err, yerr=spec_cr_err, fmt="kx", capsize=2) +plt.plot(spec_en, spec_mod_cr, color="tab:cyan", alpha=0.7, lw=1.1) +# Make sure the energy axis is log scaled plt.xscale("log") +# Alter the formatters for the energy axis so that (for instance) 10 keV is +# displayed as '10 keV' rather than '10^1 keV' +plt.gca().xaxis.set_major_formatter(FuncFormatter(lambda inp, _: "{:g}".format(inp))) +plt.gca().xaxis.set_minor_formatter(FuncFormatter(lambda inp, _: "{:g}".format(inp))) + plt.xlabel("Energy [keV]", fontsize=15) plt.ylabel(r"Spectrum [ct cm$^{-2}$ s$^{-1}$ keV$^{-1}$]", fontsize=15) From 4c40668e56f285275cae7cfd8a9992dec15ce1a6 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 10:31:36 -0500 Subject: [PATCH 168/221] Added more accurate commentary on the ARF generation, also allow the passing of the minphotons argument to xaarfgen now. In getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 61 ++++++++++++------- 1 file changed, 39 insertions(+), 22 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 85c7d8cb..778bff3e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-10' +date: '2025-12-11' file_format: mystnb jupytext: text_representation: @@ -717,6 +717,7 @@ def gen_xrism_xtend_arf( rmf_file: str, src_radec_reg_file: str, num_photons: int, + min_photons: int, ): """ A wrapper function for the HEASoft `xaarfgen` task, which we use to generate @@ -740,8 +741,12 @@ def gen_xrism_xtend_arf( :param str rmf_file: The path to the RMF file necessary to generate an ARF. :param str src_radec_reg_file: The path to the region file defining the source region for which to generate an ARF. - :param int num_photons: The number of photons to simulate in the ray-tracing - portion of XRISM-Xtend ARF generation. + :param int num_photons: The number of photons, per energy grid point, per + attitude histogram, to simulate in the ray-tracing portion of + XRISM-Xtend ARF generation. + :param int min_photons: The minimum number of photons, per energy grid point, per + attitude histogram, that is required to continue to calculating an ARF at + the end of the ray-tracing portion. """ # We can extract the ObsID directly from the header of the spectrum file - it is @@ -787,6 +792,7 @@ def gen_xrism_xtend_arf( outfile=arf_out, sourcetype="POINT", numphotons=num_photons, + minphotons=min_photons, source_ra=ra_val, source_dec=dec_val, telescop="XRISM", @@ -2259,34 +2265,44 @@ XMA are being simulated, but it does help to produce very accurate ARFs. There a that it can be sped up, though at the cost of that accuracy - the most direct way is to limit the number of events that are simulated. -An argument specifying the number of events can be passed to `xaarfgen`, and for +Rather than setting an overall number of events to simulate, the `xaarfgen` task provides +an argument ('numphoton') to set the number og photons allocated to each attitude +histogram bin (in the exposure map file), per grid point in the internal energy grid. + +An argument specifying the number of events ('numphoton') can be passed to `xaarfgen`, and for our demonstration we are going to use a very small sample - this is primarily so the notebook can run in a reasonable amount of time. -**You should likely use a larger value for real analysis!** -The xaarfgen documentation provides the following guidance on choosing the number of +A second argument, `minphoton`, specifies the minimum acceptable number of raytracing photons that +successfully reach the focal plane for each raytracing energy grid point. If that minimum number is +not reached for each energy grid point during the raytracing process, ARF production will fail. + +The `xaarfgen` documentation provides the following guidance on choosing the number of events to simulate: ```{admonition} -The minimum number of raytracing photons that successfully reach the focal plane, per -raytracing energy grid point, that is acceptable to make an ARF. The number of focal -plane photons that contribute to the ARF must exceed 'minphoton' for every -energy, otherwise the program aborts. Note that even if minphoton is exceeded at -all energies, this does not guarantee that the resulting ARF is robust and -sufficiently accurate. In general, about 5000 or more photons per energy (over the -extraction region) give good results, but the actual minimum number varies -case-by-case, and fewer may be sufficient in some cases. The default value of -minphoton is deliberately very small, in order that the ARF is made and available -for diagnostic evaluation. In general, it is not recommended to set 'minphoton' to a -high value in the first place, because it is not possible to reliably estimate -what 'numphoton' should be in advance of running raytracing within xaarfgen, in order -for that value of 'minphoton' to be satisfied for all energies, which could result -in repeated failures after very long run times. It could also run into memory -problems and/or a raytracing file size that is unmanageable. +Note that even if minphoton is exceeded at all energies, this does not guarantee +that the resulting ARF is robust and sufficiently accurate. + +In general, about 5000 or more photons per energy (over the extraction region) give +good results, but the actual minimum number varies case-by-case, and fewer may be +sufficient in some cases. + +The default value of minphoton is deliberately very small, in order that the +ARF is made and available for diagnostic evaluation. In general, it is not +recommended to set 'minphoton' to a high value in the first place, because it is +not possible to reliably estimate what 'minphoton' should be in advance of +running raytracing within xaarfgen, in order for that value of 'photon' to be +satisfied for all energies, which could result in repeated failures after very long +run times. It could also run into memory problems and/or a raytracing file size that +is unmanageable. ``` +We choose the default values for both the 'minphoton' and 'numphoton' arguments: + ```{code-cell} python arf_rt_num_photons = 20000 +arf_rt_min_photons = 100 ``` So now we move onto actually running the ARF generation - using the @@ -2315,6 +2331,7 @@ arg_combs = [ RMF_PATH_TEMP.format(oi=oi, xdc=dc), radec_src_reg_path.format(oi=oi), arf_rt_num_photons, + arf_rt_min_photons, ] for oi, dcs in rel_dataclasses.items() for dc in dcs @@ -2743,7 +2760,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-10 +Updated On: 2025-12-11 +++ From 18f0d6533b5a42863ca7d5e4fd2e3fc569561276 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 10:38:10 -0500 Subject: [PATCH 169/221] Absolute idiot that I am, I wasn't passing the net light curve path to the outfile parameter of lcmath, which might explain why that tool failed!. In the getting-started-xrism-xtend.md notebook, for issue #128 --- .../xrism/getting-started-xrism-xtend.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 778bff3e..2d75564d 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2128,6 +2128,14 @@ for oi, dcs in rel_dataclasses.items(): bspec_backscals[oi][cur_dc] = back_specco["SPECTRUM"].header["BACKSCAL"] ``` +Showing the BACKSCAL values: + +```{code-cell} python +print(spec_backscals) +print("\n") +print(bspec_backscals) +``` + #### Grouping our new spectra We will group the spectra we just generated. Grouping essentially combines @@ -2498,7 +2506,11 @@ for oi, dcs in rel_dataclasses.items(): # Run the tool to produce a net light curve hsp.lcmath( - infile=cur_lc, bgfile=cur_blc, outfile="", multi=1, multb=cur_back_multi + infile=cur_lc, + bgfile=cur_blc, + outfile=cur_nlc, + multi=1, + multb=cur_back_multi, ) ``` From cab782b6d6ee83efa06399ff5fbde45adab0daca Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 12:58:09 -0500 Subject: [PATCH 170/221] Added information on datamodes/classes to the getting-started-xrism-xtend.md notebook. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 96 +++++++++---------- 1 file changed, 48 insertions(+), 48 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 2d75564d..82a343f3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -892,6 +892,14 @@ os.makedirs(OUT_PATH, exist_ok=True) # ------------- Set up output file path templates -------------- +# ------ XTDPIPELINE ------- +# Cleaned event list path template - obviously going to be useful later +EVT_PATH_TEMP = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") + +# The path to the bad pixel map, useful for excluding dodgy pixels from data products +BADPIX_PATH_TEMP = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") +# -------------------------- + # --------- IMAGES --------- IM_PATH_TEMP = os.path.join( OUT_PATH, @@ -1202,6 +1210,31 @@ raw_evt_dir_temp = os.path.join(ROOT_DATA_DIR, "{oi}", "xtend", "event_uf") through to generating the 'quick-look' data products (images, spectra, and light curves) included in HEASARC's XRISM archive 'products' directories. +This `xtdpipeline` task will prepare **all** data taken during a particular +observation - that means that if you are using an XRISM-Xtend observation that +was running in a data mode other than full-window, you still only have to +run `xtdpipeline` once. + +Full-window should be considered XRISM-Xtend's default data mode, but you are likely +to come across data taken in other modes. Those modes are described in the XRISM +ABC guide ([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/proposals/POG/Xtend_SXI.html#SECTION00920000000000000000)), +and we summarise them here: + +- **Full-window** - The entire Xtend detector is in the same data mode, producing a 640x640 (in raw CCD coordinates) image, at a 4 second time resolution. +- **1/8th window [NO BURST]** - Half of the detector (2 CCDs) operates 'normally' and the other has only 1/8th of the pixel rows operating. Produces a 640x80 image, at 0.5 second time resolution. +- **1/8th window [BURST]** - Similar to the 1/8th window mode but collects exposures during only a small fraction of the effective detector exposures (the 0.5 second time resolution). Avoids pile-up for very bright sources, and allows the determination of photon arrival times with ~0.06 second accuracy. + +Data taken in each data mode is assigned a different 'dataclass' so that the multiple +event lists produced when using a 1/8th window mode can be distinguished from the event +list of the half the detector that is operating 'normally'. The dataclasses are discussed in the XRISM ABC guide +([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html)) and are summarised below: +- **30000010** - All four CCDs are in full window mode. +- **31100010** - CCD1 & CCD2 in 1/8 window mode. +- **31200010** - CCD1 & CCD2 in full window + 0.1 sec burst mode. +- **31300010** - CCD1 & CCD2 in 1/8 window + 0.1 sec burst mode +- **32000010** - CCD3 & CCD4 in full window mode + + The pipeline has three stages and provides the option to start and stop the processing at any of those stages; this can be useful if you wish to re-run a stage with slightly different configuration without repeating the entire pipeline run. @@ -1211,13 +1244,6 @@ A different set of tasks is encapsulated by each stage, and they have the follow - **Stage 2** - Screening and filtering of the prepared Xtend event lists. - **Stage 3** - Generation of quick-look data products. - -***MUCH MORE SPECIFIC INFORMATION SHOULD GO HERE*** - -***INCLUDE XTEND DATA CLASS SUMMARY, AND HOW THERE MAY BE MULTIPLE EVENT LISTS PER OBSID*** - - - ```{note} We will stop the execution of `xtdpipeline` at **Stage 2**, as the latter part of this demonstration will show you how to make more customised data products than are output @@ -1282,34 +1308,6 @@ if len(xtd_pipe_problem_ois) != 0: Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. ``` -Finally, we set up some template variables for the various useful files output by the -XRISM-Xtend processing pipeline. These include the cleaned event lists we just created, - -cleaned event lists we just created: - -***CONSIDER MORE COMMENTARY ON DATACLASS ETC. HERE*** - - -***Need to consider the 'p0' bit - apparently the '0' is a counter for splitting large datasets, but I don't know when it is used*** - -***sc == 'split counter'*** - -```{code-cell} python -# Cleaned event list path template - obviously going to be useful later -evt_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}_cl.evt") - -# The path to the bad pixel map, useful for excluding dodgy pixels from data products -badpix_path_temp = os.path.join(OUT_PATH, "{oi}", "xa{oi}xtd_p{sc}{xdc}.bimg") -``` - -### Good-time-intervals - -### Identifying problem pixels - -```{code-cell} python - -``` - ## 3. Generating new XRISM-Xtend images and exposure maps The XRISM-Xtend data have now been prepared for scientific use, with the most important @@ -1556,7 +1554,7 @@ is a good idea to run them in parallel! ```{code-cell} python arg_combs = [ [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), *cur_bnds, cur_bf, @@ -1622,11 +1620,11 @@ Finally, we run the exposure map generation: ```{code-cell} python arg_combs = [ [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), ehk_path_temp.format(oi=oi), - badpix_path_temp.format(oi=oi, xdc=dc, sc=0), + BADPIX_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), "NONE", cur_bf, expmap_rad_delta, @@ -2047,7 +2045,7 @@ spectrum like we did with the different-energy-band images. ```{code-cell} python arg_combs = [ [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), src_coord, src_reg_rad, @@ -2415,7 +2413,7 @@ light curves simultaneously: ```{code-cell} python arg_combs = [ [ - evt_path_temp.format(oi=oi, xdc=dc, sc=0), + EVT_PATH_TEMP.format(oi=oi, xdc=dc, sc=0), os.path.join(OUT_PATH, oi), src_coord, src_reg_rad, @@ -2778,17 +2776,17 @@ Updated On: 2025-12-11 ### Additional Resources -XRISM Data Reduction (ABC) Guide - https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide +**XRISM Data Reduction (ABC) Guide**: https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide -HEASoftPy GitHub Repository: https://github.com/HEASARC/heasoftpy +**HEASoftPy GitHub Repository**: https://github.com/HEASARC/heasoftpy -HEASoftPy HEASARC Page: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/heasoftpy.html +**HEASoftPy HEASARC Page**: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/heasoftpy.html -HEASoft XRISM `xtdpipeline` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html +**HEASoft XRISM `xtdpipeline` help file**: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html -HEASoft XRISM `xaexpmap` help file: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html +**HEASoft XRISM `xaexpmap` help file**: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html -XSPEC Model Components: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html +**XSPEC Model Components**: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html ### Acknowledgements @@ -2797,4 +2795,6 @@ XSPEC Model Components: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual [XRISM GOF & SDC (2024) - _XRISM ABC GUIDE XTEND ENERGY-CHANNEL MAPPING_ [ACCESSED 25-NOV-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/Xtend_Data_Analysis.html#SECTION001043000000000000000) -[](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html) +[XRISM GOF & SDC (2024) - _XRISM ABC GUIDE FILE NAMING CONVENTIONS_ [ACCESSED 11-DEC-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html) + +[XRISM GOF & SDC (2024) - _XRISM ABC GUIDE XTEND DATA MODES_ [ACCESSED 11-DEC-2025]](https://heasarc.gsfc.nasa.gov/docs/xrism/proposals/POG/Xtend_SXI.html#SECTION00920000000000000000) From 6131852b68b6f6466cf1ebc12693555db631e59c Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 13:10:22 -0500 Subject: [PATCH 171/221] May possibly have actually finished the writing of getting-started-xrism-xtend.md now. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 51 ++++++++++++++++++- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 82a343f3..bc5edb77 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -978,9 +978,28 @@ ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") ## 1. Finding and downloading XRISM observations of LMC N132D +Our first task is to determine which XRISM observations are relevant to the source +that we are interested in. + +We are going in with the knowledge that LMC N132D has been observed by XRISM, but of +course there is no guarantee that _your_ source of interest has been, so this is +an important exploratory step. ### Determining the name of the XRISM observation summary table +HEASARC maintains tables that contain information about every observation taken by +each of the missions in its archive. We will use XRISM's table to find observations +that should be relevant to our source. + +The name of the XRISM observation summary table is 'xrismmastr', but as you may not +know that a priori, we demonstrate how to identify the correct table for a given +mission. + +Using the AstroQuery Python module (specifically this Heasarc object), we list all +catalogs that are a) related to XRISM, and b) are flagged as 'master' (meaning the +summary table of observations). This should only return one catalog for any +mission you pass to 'keywords': + ```{code-cell} python catalog_name = Heasarc.list_catalogs(master=True, keywords="xrism")[0]["name"] catalog_name @@ -990,7 +1009,7 @@ catalog_name To search for relevant observations, we have to know the coordinates of our source. The astropy module allows us to look up a source name in CDS' Sesame name - resolver and retrieve its coordinates. +resolver and retrieve its coordinates. ```{hint} You could also set up a SkyCoord object directly, if you already know the coordinates. @@ -1006,6 +1025,15 @@ src_coord ### Searching for relevant observations +Now that we know which catalog to search, and the coordinates of our source, we use +AstroQuery to retrieve those lines of the summary table that are within some radius +of the source coordinate. In this case we're using the default search radius for +the XRISM summary table, but you can also pass a `radius` argument to set your own. + +In this case, we also define a custom set of columns to retrieve, as the default set +does not contain the 'xtd_dataclas*' columns that we might need later. You may also +pass a wildcard `columns='*'` to retrieve all available columns. + ```{code-cell} python col_str = ( "__row,obsid,name,ra,dec,time,exposure,status,public_date," @@ -1015,7 +1043,16 @@ all_xrism_obs = Heasarc.query_region(src_coord, catalog_name, columns=col_str) all_xrism_obs ``` -For an active mission (i.e., actively collecting data and adding to the archive)... +For an active mission (i.e., actively collecting data and adding to the archive), we +will, at some point, probably come across observations that have been taken, but are +currently only available to their proposers (still in the proprietary period). + +Such proprietary observations will still appear in the XRISM summary table, and the +files could even be downloaded, but unless we took those data we won't have the +key necessary to decrypt the files. + +As such, we are going to use the 'public_date' column to filter out any observations +that are not yet publicly available: ```{code-cell} python public_times = Time(all_xrism_obs["public_date"], format="mjd") @@ -1054,11 +1091,19 @@ rel_dataclasses = { ### Downloading the selected XRISM observations +The AstroQuery `Heasarc` module makes it easy to download the data we need. Our +cut-down table of observations can be passed to the `locate_data()` method, which +will return the access links for the data on several different platforms: + ```{code-cell} python data_links = Heasarc.locate_data(avail_xrism_obs) data_links ``` +That data links table can now be passed straight to the `download_data()` method, which +will do what it says on the tin and download the files. We can also specify which +platform to pull the observations from, and in this case we select the HEASARC AWS S3 bucket: + ```{code-cell} python Heasarc.download_data(data_links, "aws", ROOT_DATA_DIR) ``` @@ -1071,6 +1116,8 @@ on SciServer, you may pass 'sciserver' to use the pre-mounted HEASARC dataset. ### What do the downloaded data directories contain? +Now we can take a quick look at the contents of the directory we just downloaded: + ```{code-cell} python glob.glob(os.path.join(ROOT_DATA_DIR, rel_obsids[0], "") + "*") ``` From 01b83658710d60521b5a1947ba5dd6334c417ecb Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 15:27:01 -0500 Subject: [PATCH 172/221] Tiny aethsetic change to getting-started-xrism-xtend.md --- .../xrism/getting-started-xrism-xtend.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index bc5edb77..f4fd8529 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -74,7 +74,7 @@ We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstr ### Runtime -As of 8th December 2025, this notebook takes ~{N}m to run to completion on Fornax using the 'Default Astrophysics' image and the small server with 8GB RAM/ 2 cores. +As of 8th December 2025, this notebook takes ~50m to run to completion on Fornax using the 'Default Astrophysics' image and the medium server with 16GB RAM/ 4 cores. ## Imports @@ -1188,7 +1188,7 @@ if HEA_VER < Version("V6.36"): raise ValueError( "We strongly recommend using HEASoft v6.36 or later for this " "tutorial - you may run rest of the notebook yourself, but " - "ARF generation will likely fail." + "ARF generation will either fail or produce an incorrect result." ) ``` @@ -2177,7 +2177,6 @@ Showing the BACKSCAL values: ```{code-cell} python print(spec_backscals) -print("\n") print(bspec_backscals) ``` @@ -2789,7 +2788,7 @@ tags: [hide-input] jupyter: source_hidden: true --- -plt.figure(figsize=(6, 4)) +plt.figure(figsize=(7, 4)) plt.minorticks_on() plt.tick_params(which="both", direction="in", top=True, right=True) From 1c9972af0b541f576a0ebc0775ee861aa76f7d8d Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 16:19:10 -0500 Subject: [PATCH 173/221] Fixed a bug in the filling out of the demo_lc_path variable in getting-started-xrism-xtend.md. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f4fd8529..267bcc60 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2190,8 +2190,6 @@ this, once again through HEASoftPy. First, we set up the grouping criteria and a template variable for the name of the output grouped spectral files: -*** REMIND MYSELF WHETHER GROUPING FROM SOURCE SPEC IS AUTOMATICALLY APPLIED TO BACK SPEC IN XSPEC? *** - ```{code-cell} python spec_group_type = "min" spec_group_scale = 1 @@ -2579,6 +2577,9 @@ the light curve data in directly, using astropy.io.fits, and then plot it yourse demo_lc_path = NET_LC_PATH_TEMP.format( oi=chosen_demo_lc_obsid, xdc=chosen_demo_lc_dataclass, + ra=src_coord.ra.value.round(6), + dec=src_coord.dec.value.round(6), + rad=src_reg_rad.to("deg").value.round(4), lo=chosen_demo_lc_bnds[0].value, hi=chosen_demo_lc_bnds[1].value, lct=0.0, From 4e4428732c5e21cce0a840d2ffc996bd7571864f Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 16:34:10 -0500 Subject: [PATCH 174/221] Pointed the getting-started-xrism-xtend.md notebook to the S3 bucket for CALDB access. For issue #128 --- .../xrism/getting-started-xrism-xtend.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 267bcc60..12787b7a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -857,6 +857,10 @@ hsp.Config.allow_failure = False mp.set_start_method("fork", force=True) # -------------------------------------------------------------- +# ----------- Set HEASoft to use the S3-bucket CALDB ----------- +os.environ["CALDB"] = "https://nasa-heasarc.s3.amazonaws.com/caldb" +# -------------------------------------------------------------- + # ------------- Setting how many cores we can use -------------- NUM_CORES = None total_cores = os.cpu_count() @@ -1340,6 +1344,10 @@ rel_dataclasses = {oi: rel_dataclasses[oi] for oi in rel_obsids} xtd_pipe_problem_ois ``` +```{warning} +Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. +``` + We also include a code snippet that will print the output of the `xtdpipeline` run for any observations that appear to have failed: @@ -1351,8 +1359,12 @@ if len(xtd_pipe_problem_ois) != 0: print("\n\n") ``` -```{warning} -Processing XRISM-Xtend data can take a long time, up to several hours for a single observation. +```{note} +This notebook is configured to acquire XRISM CALDB files from the HEASARC +Amazon Web Services S3 bucket - this can greatly improve the speed of some +steps later in the notebook when running on the Fornax science console. + +CALDB location configuration can be found in the 'Global Setup: Configuration' section. ``` ## 3. Generating new XRISM-Xtend images and exposure maps From 1efc9305c4a4b47e1378b2b0d7c0f4b3a0445457 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 16:53:15 -0500 Subject: [PATCH 175/221] Temporarily disabling the error on too-low of a HEASoft version in the getting-started-xrism-xtend.md notebook. To maybe have it build on CI/CD? For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 12787b7a..0cf80259 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1188,12 +1188,12 @@ Now we can use inequality operators on another Version object to check if our ve of HEASoft meets the requirements: ```{code-cell} python -if HEA_VER < Version("V6.36"): - raise ValueError( - "We strongly recommend using HEASoft v6.36 or later for this " - "tutorial - you may run rest of the notebook yourself, but " - "ARF generation will either fail or produce an incorrect result." - ) +# if HEA_VER < Version("V6.36"): +# raise ValueError( +# "We strongly recommend using HEASoft v6.36 or later for this " +# "tutorial - you may run rest of the notebook yourself, but " +# "ARF generation will either fail or produce an incorrect result." +# ) ``` ### Setting up file paths to pass to the XRISM-Xtend pipeline From 10082dc0563b097946524813b05d4e197660f433 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 17:17:13 -0500 Subject: [PATCH 176/221] Removed a blank line --- .../xrism/getting-started-xrism-xtend.md | 1 - 1 file changed, 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0cf80259..02847a8c 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1285,7 +1285,6 @@ list of the half the detector that is operating 'normally'. The dataclasses are - **31300010** - CCD1 & CCD2 in 1/8 window + 0.1 sec burst mode - **32000010** - CCD3 & CCD4 in full window mode - The pipeline has three stages and provides the option to start and stop the processing at any of those stages; this can be useful if you wish to re-run a stage with slightly different configuration without repeating the entire pipeline run. From 64b69fbd73bd5fba1dbb8eff777bbb79317bc030 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 11 Dec 2025 21:05:59 -0500 Subject: [PATCH 177/221] Made some of the event list and region file paths passed to HEASoft tasks into relative paths, to save on string length and the problems it can cause in HEASoft. In getting-started-xrism-xtend.md, for PR #150 --- .../xrism/getting-started-xrism-xtend.md | 30 +++++++++++-------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 02847a8c..0a8da9b3 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -496,13 +496,11 @@ def gen_xrism_xtend_lightcurve( # Convert the energy limits to channel limits, rounding down and up to the nearest # integer channel for the lower and upper bounds respectively. + # We will use these to make a channel selection in the event list passed + # to the tool lo_ch = np.floor((lo_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) hi_ch = np.ceil((hi_en / XTD_EV_PER_CHAN).to("chan")).value.astype(int) - # Create modified input event list file path, where we use the just-calculated - # PI channel limits to subset the events - evt_file_chan_sel = f"{event_file}[PI={lo_ch}:{hi_ch}]" - # Set up the output file name for the light curve we're about to generate. lc_out = os.path.basename(LC_PATH_TEMP).format( oi=cur_obs_id, @@ -539,12 +537,18 @@ def gen_xrism_xtend_lightcurve( # duration, and another that creates a new set of HEASoft parameter files (so # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): + # Create modified input event list file path, where we use the just-calculated + # PI channel limits to subset the events + # This is within the chdir context, as we want to pass a relative path + # as very long string arguments can be cut off when passed to HEASoft tools + evt_file_chan_sel = os.path.relpath(event_file) + f"[PI={lo_ch}:{hi_ch}]" + src_out = hsp.extractor( filename=evt_file_chan_sel, fitsbinlc=lc_out, binlc=time_bin_size, lcthresh=lc_bin_thresh, - regionfile=src_reg_file, + regionfile=os.path.relpath(src_reg_file), xcolf="X", ycolf="Y", gti="GTI", @@ -558,7 +562,7 @@ def gen_xrism_xtend_lightcurve( fitsbinlc=lc_back_out, binlc=time_bin_size, lcthresh=lc_bin_thresh, - regionfile=back_reg_file, + regionfile=os.path.relpath(back_reg_file), xcolf="X", ycolf="Y", gti="GTI", @@ -637,9 +641,9 @@ def gen_xrism_xtend_spectrum( # there are no clashes with other processes). with contextlib.chdir(temp_work_dir), hsp.utils.local_pfiles_context(): src_out = hsp.extractor( - filename=event_file, + filename=os.path.relpath(event_file), phafile=sp_out, - regionfile=src_reg_file, + regionfile=os.path.relpath(src_reg_file), xcolf="X", ycolf="Y", ecol="PI", @@ -650,9 +654,9 @@ def gen_xrism_xtend_spectrum( # Now for the background light curve back_out = hsp.extractor( - filename=event_file, + filename=os.path.relpath(event_file), phafile=sp_back_out, - regionfile=back_reg_file, + regionfile=os.path.relpath(back_reg_file), xcolf="X", ycolf="Y", ecol="PI", @@ -797,9 +801,9 @@ def gen_xrism_xtend_arf( source_dec=dec_val, telescop="XRISM", instrume="XTEND", - emapfile=expmap_file, - rmffile=rmf_file, - regionfile=src_radec_reg_file, + emapfile=os.path.relpath(expmap_file), + rmffile=os.path.relpath(rmf_file), + regionfile=os.path.relpath(src_radec_reg_file), regmode="RADEC", noprompt=True, clobber=True, From 740fb4d67525f3b832b9705e229b0f34ce4ae7bf Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 12 Dec 2025 08:35:39 -0500 Subject: [PATCH 178/221] Tiny wording change, mostly to get CircleCI going again (hopefully the HEASARC website is a little happier today). For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0a8da9b3..b3563ed0 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1035,8 +1035,8 @@ src_coord Now that we know which catalog to search, and the coordinates of our source, we use AstroQuery to retrieve those lines of the summary table that are within some radius -of the source coordinate. In this case we're using the default search radius for -the XRISM summary table, but you can also pass a `radius` argument to set your own. +of the source coordinate. We're using the default search radius for +the XRISM summary table, but you can pass a `radius` argument to set your own. In this case, we also define a custom set of columns to retrieve, as the default set does not contain the 'xtd_dataclas*' columns that we might need later. You may also From 48595effe2726e285a09b2bbaa5660e7cb089339 Mon Sep 17 00:00:00 2001 From: David Turner Date: Fri, 12 Dec 2025 12:47:50 -0500 Subject: [PATCH 179/221] Replaced a misuse of admonition with seealso, in the getting-started-xrism-xtend.md notebook --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b3563ed0..3b6c5019 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2345,7 +2345,7 @@ not reached for each energy grid point during the raytracing process, ARF produc The `xaarfgen` documentation provides the following guidance on choosing the number of events to simulate: -```{admonition} +```{seealso} Note that even if minphoton is exceeded at all energies, this does not guarantee that the resulting ARF is robust and sufficiently accurate. From 8395a49dd15f18c3f384439329e27bb1057800a6 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 15 Dec 2025 10:36:49 -0500 Subject: [PATCH 180/221] Some very small typos fixed --- .../xrism/getting-started-xrism-xtend.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 3b6c5019..c8adfbe2 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -834,8 +834,6 @@ jupyter: --- # The name of the source we're examining in this demonstration SRC_NAME = "LMCN132D" -# SRC_NAME = "NGC4151" -# SRC_NAME = "AX J1910.7+0917" # Controls the verbosity of all HEASoftPy tasks TASK_CHATTER = 3 @@ -1419,8 +1417,8 @@ Alternatively, we can figure out this relationship between PI and energy by look a XRISM-Xtend Redistribution Matrix File (RMF), which exists to describe this mapping. -We will be creating new RMFs as part of the generation of XRISM-Xtend spectra in the -next section. For our current purpose, however, it is acceptable to use the RMFs that +We will be creating new RMFs as part of the generation of XRISM-Xtend spectra later +in this notebook. For our current purpose, however, it is acceptable to use the RMFs that were included in the XRISM-Xtend archive we downloaded earlier. The archived RMFs are generated for the entire Xtend FoV, rather than for the CCDs From 384bed9646b90adba4fcfefabb90264d61090cd3 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:04:40 -0500 Subject: [PATCH 181/221] Restored XMM CCF download step in config.yml --- .circleci/config.yml | 56 ++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 7bc63652..5788136f 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -60,34 +60,34 @@ jobs: # Some missions (e.g. XMM, Chandra, eROSITA) require that their calibration files are available locally. We # don't want to download them every single time, so we're going to cache them for future runs. -# - run: -# name: Acquiring/validating XMM-CCFs -# no_output_timeout: 30m -# command: | -# # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same -# # version number as currently defined in the environment variable, we have to download the data -# if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then -# -# # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the -# # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions -# # available, so we don't want to delete the directory. -# mkdir -p /home/jovyan/xmm-ccf/ccf-files -# -# # We do delete the version file, if it exists, as if we get to this point then we are either downloading -# # the data for the first time, or the version number has changed. -# [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver -# -# # Make a new version file - this goes a level up from the directory where the files will actually -# # be stored (ccf-files), because the rsync process will delete the version file -# echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver -# -# # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The -# # neatest way is to make a new conda environment that just contains rsync -# micromamba create -n rsync-env -y -c conda-forge rsync -# -# # Actually rsync the calibration files -# micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files -# fi + - run: + name: Acquiring/validating XMM-CCFs + no_output_timeout: 30m + command: | + # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same + # version number as currently defined in the environment variable, we have to download the data + if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then + + # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the + # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions + # available, so we don't want to delete the directory. + mkdir -p /home/jovyan/xmm-ccf/ccf-files + + # We do delete the version file, if it exists, as if we get to this point then we are either downloading + # the data for the first time, or the version number has changed. + [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver + + # Make a new version file - this goes a level up from the directory where the files will actually + # be stored (ccf-files), because the rsync process will delete the version file + echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver + + # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The + # neatest way is to make a new conda environment that just contains rsync + micromamba create -n rsync-env -y -c conda-forge rsync + + # Actually rsync the calibration files + micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files + fi # We also download the Chandra CalDB - run: From 82ead7acec6ac127b2738e952ab035fd2b1330de Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:07:51 -0500 Subject: [PATCH 182/221] Returning nb_execution_timeout to 1200 in conf.py - might alter it back at some point though --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index b82ea7ab..cdffde54 100644 --- a/conf.py +++ b/conf.py @@ -24,7 +24,7 @@ # ---------------------------- MyST configuration ---------------------------- # MyST-NB configuration -nb_execution_timeout = 2400 +nb_execution_timeout = 1200 nb_merge_streams = True nb_execution_mode = "cache" nb_scroll_outputs = True From ddde78364f39b5262f97cc89dfd61b9e6923a791 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:17:15 -0500 Subject: [PATCH 183/221] Commented out the XMM CCF acquisition step again in config.yml, for speed testing purposes. --- .circleci/config.yml | 56 ++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 5788136f..7bc63652 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -60,34 +60,34 @@ jobs: # Some missions (e.g. XMM, Chandra, eROSITA) require that their calibration files are available locally. We # don't want to download them every single time, so we're going to cache them for future runs. - - run: - name: Acquiring/validating XMM-CCFs - no_output_timeout: 30m - command: | - # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same - # version number as currently defined in the environment variable, we have to download the data - if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then - - # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the - # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions - # available, so we don't want to delete the directory. - mkdir -p /home/jovyan/xmm-ccf/ccf-files - - # We do delete the version file, if it exists, as if we get to this point then we are either downloading - # the data for the first time, or the version number has changed. - [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver - - # Make a new version file - this goes a level up from the directory where the files will actually - # be stored (ccf-files), because the rsync process will delete the version file - echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver - - # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The - # neatest way is to make a new conda environment that just contains rsync - micromamba create -n rsync-env -y -c conda-forge rsync - - # Actually rsync the calibration files - micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files - fi +# - run: +# name: Acquiring/validating XMM-CCFs +# no_output_timeout: 30m +# command: | +# # If we can't see the xmm_ccf directory, or the version file we generate doesn't contain the same +# # version number as currently defined in the environment variable, we have to download the data +# if [ ! -d /home/jovyan/xmm-ccf ] || [[ ! "$(<"/home/jovyan/xmm-ccf/xmm-ccf.ver")" == "$CIRCLECI_XMM_CCF_VER" ]]; then +# +# # Makes the xmm-ccf directory, if it doesn't already exist. This is a different approach to the +# # what we do for the Chandra CalDB below, as rsync will update files if there are newer versions +# # available, so we don't want to delete the directory. +# mkdir -p /home/jovyan/xmm-ccf/ccf-files +# +# # We do delete the version file, if it exists, as if we get to this point then we are either downloading +# # the data for the first time, or the version number has changed. +# [ -f /home/jovyan/xmm-ccf/xmm-ccf.ver ] && rm /home/jovyan/xmm-ccf/xmm-ccf.ver +# +# # Make a new version file - this goes a level up from the directory where the files will actually +# # be stored (ccf-files), because the rsync process will delete the version file +# echo "${CIRCLECI_XMM_CCF_VER}" > /home/jovyan/xmm-ccf/xmm-ccf.ver +# +# # We don't include rsync in the Fornax-Hea image, so unfortunately we'll have to install it now. The +# # neatest way is to make a new conda environment that just contains rsync +# micromamba create -n rsync-env -y -c conda-forge rsync +# +# # Actually rsync the calibration files +# micromamba run -n rsync-env rsync -v -a --delete --delete-after --force --include='*.CCF' --exclude='*/' sasdev-xmm.esac.esa.int::XMM_VALID_CCF /home/jovyan/xmm-ccf/ccf-files +# fi # We also download the Chandra CalDB - run: From 5c5fe36b6d6028ca74b801d60fa8791a7299e06d Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:19:59 -0500 Subject: [PATCH 184/221] Tiny grammar change suggested by the IDE (uurgh), just meant to trigger a CircleCI run. In getting-started-xrism-xtend.md. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index c8adfbe2..59ac4e73 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1426,7 +1426,7 @@ our particular target falls on, but practically speaking, that doesn't make a si difference. Using observation 000128000 as an example, we determine the path to the relevant -pre-generated RMF. We only expect a single file, and include a validity check to +pre-generated RMF. We only expect a single file and include a validity check to ensure that this does not change in future versions of the archive: ```{code-cell} python From b713f9898f36375c4ef751c335ec6117d504b774 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:29:59 -0500 Subject: [PATCH 185/221] Small typos fixed in the getting-started-xrism-xtend.md notebook. --- .../xrism/getting-started-xrism-xtend.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 59ac4e73..9e9c86e8 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-11' +date: '2025-12-16' file_format: mystnb jupytext: text_representation: @@ -1454,6 +1454,7 @@ with fits.open(pregen_rmf_path) as rmfo: # Iterate through the tables in the RMF, printing their names for tab in rmfo: print(tab.name) + print("") # Associate the EBOUNDS table with a variable, so it can be used outside # the fits.open context @@ -1876,7 +1877,7 @@ will be pretty easy to deal with, however, as HEASoft includes a tool to transfo files between different coordinate systems. ```{code-cell} python -# The path to the included calibration source region file +# The path to the HEASoft-supplied XRISM-Xtend calibration source region file detpix_xtend_calib_reg_path = os.path.join( os.environ["HEADAS"], "refdata", "calsrc_XTD_det.reg" ) @@ -2830,7 +2831,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-11 +Updated On: 2025-12-16 +++ From 184fa0c8d8bdad0545e000590c4a911ad4aa5d52 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:38:55 -0500 Subject: [PATCH 186/221] Terrible replication of a change --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 7bc63652..342429ff 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -105,7 +105,7 @@ jobs: mkdir -p /home/jovyan/chandra-caldb # Download the Chandra CalDB - wget https://cxc.cfa.harvard.edu/cdaftp/arcftp/caldb/caldb_${CHANDRA_CALDB_VER}_main.tar.gz + wget https://cxc.cfa.harvard.edu/cdaftp/arcftp/ChandraCalDB/tars/caldb_${CHANDRA_CALDB_VER}_main.tar.gz # Unpack the archive into a specific directory tar xzf caldb_${CHANDRA_CALDB_VER}_main.tar.gz -C /home/jovyan/chandra-caldb From 2a153e728dd7781cc74346b3937868c9fc1662fa Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:40:05 -0500 Subject: [PATCH 187/221] Grammar fix --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 9e9c86e8..d46ac4a7 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -44,7 +44,7 @@ By the end of this tutorial, you will be able to: The 'X-Ray Imaging and Spectroscopy Mission' (**XRISM**) is an X-ray telescope designed for high-energy-resolution spectroscopic observations of astrophysical sources, as well as wide-field X-ray imaging. -XRISM, launched in 2023, is the result of a JAXA-NASA partnership (with involvement from ESA), and serves as nearly like-for-like replacement +XRISM, launched in 2023, is the result of a JAXA-NASA partnership (with involvement from ESA), and serves as a nearly like-for-like replacement of the **Hitomi** telescope, which was lost shortly after its launch in 2016. There are two main XRISM instruments, **Xtend** and **Resolve**. In this tutorial, we will focus on **Xtend**, which is From 2748a21309fff4ef53f3cfabb02858ae1bef8dce Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 10:56:53 -0500 Subject: [PATCH 188/221] Small grammar changes to getting-started-xrism-xtend.md. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index d46ac4a7..b1aeb780 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -988,7 +988,7 @@ Our first task is to determine which XRISM observations are relevant to the sour that we are interested in. We are going in with the knowledge that LMC N132D has been observed by XRISM, but of -course there is no guarantee that _your_ source of interest has been, so this is +course, there is no guarantee that _your_ source of interest has been, so this is an important exploratory step. ### Determining the name of the XRISM observation summary table @@ -1074,7 +1074,7 @@ To make sure that this notebook can run in a reasonable amount of time, we are only going to choose one of them; observation 000128000. Please note that we have written this notebook in such a way that you could remove the -first line of the next cell (which selects only one ObsID), and run the notebook +first line of the next cell (which selects only one ObsID) and run the notebook for all public observations. ```{code-cell} python @@ -2258,7 +2258,7 @@ We already discussed RMFs, and even used them to perform our own conversion betw XRISM-Xtend spectral channels and energy, in Section 3 - there we used the RMF that was included in the original data download. -Now we wish to generate new RMFs, to ensure they are entirely up-to-date! +Now we wish to generate new RMFs, so we can ensure they are entirely up to date! We make use of the XRISM-Xtend specific HEASoft task `xtdrmf` - the only input it requires is the path to the spectral file for which we wish to generate an RMF. From 885f2daeb7cdb3dd3a490af99ad8ce5ab45daa80 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 11:03:56 -0500 Subject: [PATCH 189/221] Small grammar fix --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b1aeb780..f7f610ae 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1842,7 +1842,7 @@ extended sources will be discussed in another notebook. ### Setting up source and background extraction regions To define exactly where we want to extract events from to build our data products, we -will construct 'region files' in the commonly-used 'DS9' format. +will construct 'region files' in the commonly used 'DS9' format. In this demonstration we will not provide guidance on how to choose particular source/background regions for your science case, or give detailed information From 0d712df8ba8551b33ce8b262916d585dc8ab73f7 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 11:49:51 -0500 Subject: [PATCH 190/221] Small grammar correction - meant to trigger the CircleCI process --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f7f610ae..f6be82cf 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1054,7 +1054,7 @@ will, at some point, probably come across observations that have been taken, but currently only available to their proposers (still in the proprietary period). Such proprietary observations will still appear in the XRISM summary table, and the -files could even be downloaded, but unless we took those data we won't have the +files could even be downloaded, but unless we took those data, we won't have the key necessary to decrypt the files. As such, we are going to use the 'public_date' column to filter out any observations From 831ccde2271efa125ff72c6030443cd4833ce1f7 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 13:04:49 -0500 Subject: [PATCH 191/221] Fixed lots of style issues in getting-started-xrism-xtend.md. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f6be82cf..3e5261aa 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -74,7 +74,7 @@ We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstr ### Runtime -As of 8th December 2025, this notebook takes ~50m to run to completion on Fornax using the 'Default Astrophysics' image and the medium server with 16GB RAM/ 4 cores. +As of 8th December 2025, this notebook takes ~50 m to run to completion on Fornax using the 'Default Astrophysics' image and the medium server with 16GB RAM/ 4 cores. ## Imports @@ -1271,7 +1271,7 @@ run `xtdpipeline` once. Full-window should be considered XRISM-Xtend's default data mode, but you are likely to come across data taken in other modes. Those modes are described in the XRISM ABC guide ([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/proposals/POG/Xtend_SXI.html#SECTION00920000000000000000)), -and we summarise them here: +and we summarize them here: - **Full-window** - The entire Xtend detector is in the same data mode, producing a 640x640 (in raw CCD coordinates) image, at a 4 second time resolution. - **1/8th window [NO BURST]** - Half of the detector (2 CCDs) operates 'normally' and the other has only 1/8th of the pixel rows operating. Produces a 640x80 image, at 0.5 second time resolution. @@ -1280,7 +1280,7 @@ and we summarise them here: Data taken in each data mode is assigned a different 'dataclass' so that the multiple event lists produced when using a 1/8th window mode can be distinguished from the event list of the half the detector that is operating 'normally'. The dataclasses are discussed in the XRISM ABC guide -([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html)) and are summarised below: +([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html)) and are summarized below: - **30000010** - All four CCDs are in full window mode. - **31100010** - CCD1 & CCD2 in 1/8 window mode. - **31200010** - CCD1 & CCD2 in full window + 0.1 sec burst mode. @@ -1733,7 +1733,7 @@ from 'BACKSCAL' calculation, and RMF and ARF generation. To make our point, and to give an example of the inspection you may want to perform before choosing the right dataclass for your target, we will visualize -the 0.6-10.0 keV images we generated for 000128000 in the last section: +the 0.6–10.0 keV images we generated for 000128000 in the last section: ```{code-cell} python # Set up the path to the image, and XGA Image class instance, for the '31100010' @@ -1982,7 +1982,7 @@ module) instances. The regions we created are then assigned to each image's `regions`, and the `.view()` method is called with the `view_regions=True` argument to display them. -Additionally, we extract the RA-Dec <-> Sky X-Y WCS from one image per observation so +Additionally, we extract the RA-Dec ↔ Sky X-Y WCS from one image per observation so that we can use it later on to transform our RA-Dec regions into Sky X-Y regions. ```{code-cell} python @@ -2713,7 +2713,7 @@ xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cu ### Restricting the spectral channels used for fitting When we analyze a spectrum by fitting a model, we often want to apply lower and -upper energy limits in order to fit the model on only a subset of the data points. +upper energy limits to fit the model using only a subset of the data points. Restricting the spectrum data points by energy allows us to cut out parts of the spectrum that, for instance, have very low signal-to-noise, aren't relevant to our From ed6bb9c9e418bcbd03198148c8e36b55c3e80bf0 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 16 Dec 2025 13:39:49 -0500 Subject: [PATCH 192/221] Upping the nb_execution_timeout again, 1200 isn't enough for the XRISM-Xtend notebook. For PR #150 --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index cdffde54..b82ea7ab 100644 --- a/conf.py +++ b/conf.py @@ -24,7 +24,7 @@ # ---------------------------- MyST configuration ---------------------------- # MyST-NB configuration -nb_execution_timeout = 1200 +nb_execution_timeout = 2400 nb_merge_streams = True nb_execution_mode = "cache" nb_scroll_outputs = True From 2c54af504308bf143b9ea5fa26afec0cb2783636 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 17 Dec 2025 09:36:49 -0500 Subject: [PATCH 193/221] Switched the pr-build-docs CircleCI job to use the fornax-hea:update-addHEASoft6.36 image, so we can test the XRISM Xtend notebook using that version and hopefully have the ARF generation work properly. For PR #150 --- .circleci/config.yml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 342429ff..7194025a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -19,10 +19,9 @@ jobs: # We're using a docker container as a run environment - particularly the NASA Fornax # high-energy image that we maintain as part of the Fornax initiative docker: - # The Fornax-Hea images can definitely be used here, just we're having some issues with what supporting files - # software are included at the moment, so it is easier to use a different image and just install HEASoft - # at the moment - - image: ghcr.io/nasa-fornax/fornax-images/fornax-hea:20251030_1555 + # We make use of the NASA Fornax high-energy astrophysics image, which has + # pre-existing environments for most of the missions we support. + - image: ghcr.io/nasa-fornax/fornax-images/fornax-hea:update-addHEASoft6.36 # This sets the amount of compute that we'll use to run the on-PR builds/tests - this particular setup corresponds # to 2 CPUs and 4GB of RAM as of 16th September 2025 resource_class: medium From 35d8e53517ad54b6f5c5e79a75a8b007b6625d84 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 17 Dec 2025 09:56:43 -0500 Subject: [PATCH 194/221] Rejigged a sentence slightly. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 3e5261aa..fb8ad1dd 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1633,9 +1633,9 @@ with mp.Pool(NUM_CORES) as p: ### New XRISM-Xtend exposure maps We also generate exposure maps for the entire FoV of a particular observation mode, rather -than for a particular source. The exposure map serves both to tell us the exposure time -at any given pixel of our image (assuming the image and exposure map are spatially -binned the same way), and also as a useful way to tell exactly which parts of the sky +than for a particular source. The exposure map gives us the exposure time (unsurprisingly) +at any given pixel of our image (assuming the image and exposure map are +binned the same way). Exposure maps are also a useful way to tell exactly which parts of the sky are covered by the observation. The latter capability is of particular importance for the generation/analysis of From 3d446affef41aa213dd59d1499addbe76e3c7ace Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 17 Dec 2025 10:31:13 -0500 Subject: [PATCH 195/221] More typos fixed, I REALLY JUST WANT THIS TO RUN ON CIRCLECI COME ON SSL CERTIFICATES DON'T FAIL ME. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index fb8ad1dd..8cf42fe4 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1002,7 +1002,7 @@ know that a priori, we demonstrate how to identify the correct table for a given mission. Using the AstroQuery Python module (specifically this Heasarc object), we list all -catalogs that are a) related to XRISM, and b) are flagged as 'master' (meaning the +catalogs that are **(a)** related to XRISM, and **(b)** are flagged as 'master' (meaning the summary table of observations). This should only return one catalog for any mission you pass to 'keywords': @@ -2251,10 +2251,10 @@ for oi, dcs in rel_dataclasses.items(): #### Generating XRISM-Xtend RMFs In order for the spectral model fitting software of our choice (XSPEC, for -instance) to be able to map the channels of a spectrum to an energy, we need to +instance) to be able to map spectrum channels to energies, we need to generate RMFs. -We already discussed RMFs, and even used them to perform our own conversion between +We have already discussed RMFs and even used them to perform our own conversion between XRISM-Xtend spectral channels and energy, in Section 3 - there we used the RMF that was included in the original data download. @@ -2430,10 +2430,10 @@ produce net light curves for our source. The primary input we need to provide is the time step, or time bin size, which controls the temporal resolution of the output light curve. This uniform sampling is the simplest method of dividing an event list into a light curve, but other methods -exist (requiring each time bin to reach a minimum signal-to-noise for instance). +exist (requiring each time bin to reach a minimum signal-to-noise, for instance). -Your choice of uniform time bin size will depend on your particular science case, and -practical considerations based on the length of the overall observation and the +Your choice of uniform time bin size will depend on your particular science case, as +well as practical considerations based on the length of the overall observation and the observed count-rate of the source. ```{code-cell} python @@ -2765,7 +2765,7 @@ Performing the fit is simple: xs.Fit.perform() ``` -We once again temporarily increase the chatter level, and display the fitted parameters: +We once again temporarily increase the chatter level and display the fitted parameters: ```{code-cell} python xs.Xset.chatter = 10 From c719704fff68ca736b11c0fafa5ae69904cb14d9 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 17 Dec 2025 17:50:45 -0500 Subject: [PATCH 196/221] Restored the getting-started-xrism-xtend.md check for HEASoft version. --- .../xrism/getting-started-xrism-xtend.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 8cf42fe4..efeb7e2f 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1186,16 +1186,15 @@ HEA_VER = Version(fver_out.output[0].split("_")[-1]) HEA_VER ``` -Now we can use inequality operators on another Version object to check if our version -of HEASoft meets the requirements: +We can now check that `HEA_VER` is greater than the minimum required version: ```{code-cell} python -# if HEA_VER < Version("V6.36"): -# raise ValueError( -# "We strongly recommend using HEASoft v6.36 or later for this " -# "tutorial - you may run rest of the notebook yourself, but " -# "ARF generation will either fail or produce an incorrect result." -# ) +if HEA_VER < Version("v6.36"): + raise ValueError( + "We strongly recommend using HEASoft v6.36 or later for this " + "tutorial - you may run rest of the notebook yourself, but " + "ARF generation will either fail or produce an incorrect result." + ) ``` ### Setting up file paths to pass to the XRISM-Xtend pipeline From d177b4c6d49cbe62fd1302b3c953ddd7b5c03210 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 22 Jan 2026 18:09:30 -0500 Subject: [PATCH 197/221] Made some small typo fixes in the getting-started-xrism-xtend.md notebook. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 23 +++++++++---------- 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index efeb7e2f..f0aa6b64 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1272,9 +1272,9 @@ to come across data taken in other modes. Those modes are described in the XRISM ABC guide ([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/proposals/POG/Xtend_SXI.html#SECTION00920000000000000000)), and we summarize them here: -- **Full-window** - The entire Xtend detector is in the same data mode, producing a 640x640 (in raw CCD coordinates) image, at a 4 second time resolution. -- **1/8th window [NO BURST]** - Half of the detector (2 CCDs) operates 'normally' and the other has only 1/8th of the pixel rows operating. Produces a 640x80 image, at 0.5 second time resolution. -- **1/8th window [BURST]** - Similar to the 1/8th window mode but collects exposures during only a small fraction of the effective detector exposures (the 0.5 second time resolution). Avoids pile-up for very bright sources, and allows the determination of photon arrival times with ~0.06 second accuracy. +- **Full-window** - The entire Xtend detector is in the same data mode, producing a 640x640 (in raw CCD coordinates) image, at a 4-second time resolution. +- **1/8th window [NO BURST]** - Half of the detector (2 CCDs) operates 'normally' and the other has only 1/8th of the pixel rows operating. Produces a 640x80 image, at 0.5-second time resolution. +- **1/8th window [BURST]** - Similar to the 1/8th window mode but collects exposures during only a small fraction of the effective detector exposures (the 0.5-second time resolution). Avoids pile-up for very bright sources and allows the determination of photon arrival times with ~0.06 second accuracy. Data taken in each data mode is assigned a different 'dataclass' so that the multiple event lists produced when using a 1/8th window mode can be distinguished from the event @@ -1536,8 +1536,8 @@ We are going to generate images within the following energy bounds: - 0.6-10.0 keV - 0.6-2.0 keV - 2.0-10.0 keV -- ***0.4-2.0 keV*** [not recommended] -- ***0.4-10.0 keV*** [not recommended] +- ***0.4–2.0 keV*** [not recommended] +- ***0.4–10.0 keV*** [not recommended] The bands that have a lower bound of ***0.4 keV*** are ***not recommended***, as there are issues with XRISM-Xtend data below *0.6 keV*. We are generating them to @@ -1844,7 +1844,7 @@ To define exactly where we want to extract events from to build our data product will construct 'region files' in the commonly used 'DS9' format. In this demonstration we will not provide guidance on how to choose particular -source/background regions for your science case, or give detailed information +source/background regions for your science case or give detailed information about the DS9 region format and its capabilities. Instead, we will show you how to construct basic region files using the @@ -2209,7 +2209,7 @@ spec_group_scale = 1 ``` Now we run the grouping tool - though this time we do not parallelize the task, as -the grouping process is very fast, and we wish to demonstrate how you use a HEASoftPy +the grouping process is fast, and we wish to demonstrate how you use a HEASoftPy function directly. Though remember to look at the Global Setup section of this notebook to see how we call HEASoftPy tools in the wrapper functions used to parallelize those tasks. @@ -2304,10 +2304,9 @@ is _actually_ emitting (and how), to the _observed_ spectrum; that observed spec has been altered across its energy range by how good XRISM-Xtend is at detecting photons at different points in that range. -The sensitivity of an X-ray detector is a combination of the effective area of the -actual X-ray optics (on XRISM this is the called X-ray Mirror Assembly, or XMA), and -the detector's quantum efficiency. They are both independently a function of -energy. +The sensitivity of an X-ray detector is a combination of the X-ray optic's (on XRISM +this is the called X-ray Mirror Assembly, or XMA) effective area and the detector's +quantum efficiency. They are both independently a function of energy. ARFs are standard products for most high-energy missions, but the methods implemented to calculate them for XRISM's instruments are quite unusual. @@ -2723,7 +2722,7 @@ we definitely want to exclude that part of the energy range. If we didn't, then in danger of biasing our model fitting results, leading to unreliable or unphysical conclusions about our source of interest. -Here, we only make use of channels within a 0.5-10.0 keV energy range, and we also +Here, we only make use of channels within a 0.5–10.0 keV energy range, and we also ignore any channels that have been marked as 'bad' by any previous processing steps: ```{code-cell} python From f18abc39e9d9bddc2b627e2794e69f6967029c2d Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Feb 2026 10:03:42 -0500 Subject: [PATCH 198/221] Fixed typos in parameter descriptions in a function docstring. --- .../xrism/getting-started-xrism-xtend.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f0aa6b64..71eb66a6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -605,9 +605,9 @@ def gen_xrism_xtend_spectrum( :param Quantity rel_src_radius: The radius of the source region for which we wish to generate a light curve. :param str src_reg_file: Path to the region file defining the source region for - which we wish to generate a light curve. + which we wish to generate a spectrum. :param str back_reg_file: Path to the region file defining the background region - for which we wish to generate a light curve. + for which we wish to generate a spectrum. """ # We can extract the ObsID and data class directly from the header of the event From 06ec860c3d6e84d8b0c05f5099d660184dd90f6e Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Feb 2026 19:26:57 -0500 Subject: [PATCH 199/221] Added a way to control 'calc_modegti' for xtdpipeline, and set to False to see if it'll run in time on CircleCI. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 71eb66a6..3ba1350c 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -123,6 +123,7 @@ def process_xrism_xtend( file_stem: str, extended_housekeeping: str, xtend_housekeeping: str, + calc_modegti: bool = True, ): """ A wrapper for the HEASoftPy xtdpipeline task, which is used to prepare and process @@ -146,6 +147,8 @@ def process_xrism_xtend( XRISM observation. :param str xtend_housekeeping: Instrument-specific Xtend housekeeping file for the observation. + :param bool calc_modegti: Whether to execute `xtdmodegti` to create the + segment and mode GTI for Xtend using the exposure file. Default is True. :return: A tuple containing the processed ObsID, the log output of the pipeline, and a boolean flag indicating success (True) or failure (False). :rtype: Tuple[str, hsp.core.HSPResult, bool] @@ -177,6 +180,7 @@ def process_xrism_xtend( makefilter=mkf_filter, extended_housekeeping=extended_housekeeping, housekeeping=xtend_housekeeping, + calc_modegti=calc_modegti, clobber=True, ) task_success = True @@ -1331,6 +1335,7 @@ with mp.Pool(NUM_CORES) as p: file_stem_temp.format(oi=oi), ehk_path_temp.format(oi=oi), xtd_hk_path_temp.format(oi=oi), + False, ] for oi in rel_obsids ] From 21e6954524e794853ec946a57f497fd57059bad1 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Feb 2026 20:30:25 -0500 Subject: [PATCH 200/221] Changed the CircleCI resource class to 'medium+' to try and get the XRISM notebook running (not sure if this is the issue, but worth a try). For PR #150 and issue #128 --- .circleci/config.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 253ccf62..d5f7c7c9 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -50,8 +50,8 @@ jobs: # at the moment - image: ghcr.io/nasa-fornax/fornax-images/fornax-hea:update-addHEASoft6.36 # This sets the amount of compute that we'll use to run the on-PR builds/tests - this particular setup corresponds - # to 2 CPUs and 4GB of RAM as of 16th September 2025 - resource_class: medium + # to 3 CPUs and 6GB of RAM as of 2nd February 2026 + resource_class: medium+ # Setting up the environment variables that define the support data versions to load. Use a # YAML anchor to keep the values consistent across jobs <<: *version-env From e455c8402c3c3eef53db11b9294024b6c70ba518 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Feb 2026 21:21:34 -0500 Subject: [PATCH 201/221] Switching CircleCi resource to large In an effort to get the XRISM notebook build to finish. --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index d5f7c7c9..a223b587 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -51,7 +51,7 @@ jobs: - image: ghcr.io/nasa-fornax/fornax-images/fornax-hea:update-addHEASoft6.36 # This sets the amount of compute that we'll use to run the on-PR builds/tests - this particular setup corresponds # to 3 CPUs and 6GB of RAM as of 2nd February 2026 - resource_class: medium+ + resource_class: large # Setting up the environment variables that define the support data versions to load. Use a # YAML anchor to keep the values consistent across jobs <<: *version-env From c45ea27daa9547d5904fd4e3340dc4afa97eb01d Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 3 Feb 2026 10:04:49 -0500 Subject: [PATCH 202/221] Added more of an explanation of the calc_modegti option, and why I've set it to False in this demonstration. Also a warning that they should probably turn it back on. For issue #128 and PR #150 --- .../xrism/getting-started-xrism-xtend.md | 40 +++++++++++++++++-- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 3ba1350c..4cd92690 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2025-12-16' +date: '2026-02-03' file_format: mystnb jupytext: text_representation: @@ -74,7 +74,7 @@ We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstr ### Runtime -As of 8th December 2025, this notebook takes ~50 m to run to completion on Fornax using the 'Default Astrophysics' image and the medium server with 16GB RAM/ 4 cores. +As of 2nd February 2026, this notebook takes ~50 m to run to completion on Fornax using the 'Default Astrophysics' image and the medium server with 16GB RAM/ 4 cores. ## Imports @@ -1305,12 +1305,42 @@ demonstration will show you how to make more customised data products than are o by default. ``` +There are a great many arguments that can be passed to the `xtdpipeline` task to +modify its behaviors and exercise finer control over its outputs - please see the +[`xtdpipeline` documentation](https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdpipeline.html) +for a full overview. + +One optional argument that we change from its default value is `calc_modegti`, which +is normally set to `True` (or 'yes', if you're running `xtdpipeline` from the command line). This +controls whether the `xtdpipeline` task will calculate new GTIs for each _separate_ dataclass +of a particular observation (if there are multiple dataclasses are present). + +Generating individual GTIs for different dataclasses (using the `xtdmodegti` HEASoft task; [see the documentation](https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdmodegti.html)) +allows for a more precise exclusion of detector `dead time`, which may well be different for different dataclasses. + +We have disabled this behavior due to the significant increase in processing time required for this step. + +```{code-cell} python +calc_dataclass_specific_gti = False +``` + +```{warning} +Depending on your exact science case, you may wish to set +`calc_dataclass_specific_gti=True` to re-enable the `xtdmodegti` step of `xtdpipeline`. +Doing so will likely increase the run time, but may be necessary if you care about +very precise timings and count-rates. +``` + Though we are using the HEASoftPy `xtdpipeline` function, called as `hsp.xtdpipeline(indir=...)`, it is called within a wrapper function we have written in the 'Global Setup: Functions' section of this notebook. The `process_xrism_xtend` wrapper function exists primarily to let us run the processing of different XRISM-Xtend observations in parallel. +We do not allow for every argument supported by `xtdpipeline` to be passed to the wrapper +function, but you could copy and modify `process_xrism_xtend` to suit your needs, or +create an entirely new wrapper function. + We can use Python's multiprocessing module to call the wrapper function for each of our XRISM observations, passing the relevant arguments. @@ -1335,7 +1365,7 @@ with mp.Pool(NUM_CORES) as p: file_stem_temp.format(oi=oi), ehk_path_temp.format(oi=oi), xtd_hk_path_temp.format(oi=oi), - False, + calc_dataclass_specific_gti, ] for oi in rel_obsids ] @@ -2834,7 +2864,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2025-12-16 +Updated On: 2026-02-03 +++ @@ -2850,6 +2880,8 @@ Updated On: 2025-12-16 **HEASoft XRISM `xaexpmap` help file**: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html +**HEASoft XRISM `xtdmodegti` help file**: https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xtdmodegti.html + **XSPEC Model Components**: https://heasarc.gsfc.nasa.gov/docs/software/xspec/manual/node128.html ### Acknowledgements From 6f3aea6a9dd7007d5afdcf4dd1b02de201bc3f5c Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 3 Feb 2026 11:09:50 -0500 Subject: [PATCH 203/221] Hopefully made the calibration and background regions more visually distinct from the source region in the getting-started-xrism-xtend.md image plot. The output figure should be more accessible to people with different forms of color blindness now. For issue #128 and PR #150 --- .../xrism/getting-started-xrism-xtend.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 4cd92690..0c699fdb 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1960,8 +1960,9 @@ file. The `regions` module provides functions to read-in region files in various formats and coordinate systems; it also understands that the regions in these files are to be excluded (indicated by a '-' prefix). -Note that we also set the calibration region's color to be white so that when we plot -them, they will appear as a different color than the source and background regions: +Note that we also set each calibration region's color to white and line-style to +'dotted'. This is so that when we plot them on the same image as the source and +background regions (a little later in this notebook) they will be visually distinct. ```{code-cell} python cal_regs = {} @@ -1969,6 +1970,7 @@ for oi in rel_obsids: cur_cal_regs = Regions.read(radec_xtend_calib_reg_path.format(oi=oi), format="ds9") for cur_reg in cur_cal_regs: cur_reg.visual["color"] = "white" + cur_reg.visual["linestyle"] = "dotted" # Make sure the frame is consistent with the source/back regions later, as # otherwise HEASoft tools will get confused cur_reg.center = cur_reg.center.transform_to("icrs") @@ -1994,7 +1996,9 @@ src_reg = CircleSkyRegion(src_coord, src_reg_rad, visual={"color": "green"}) ``` We do the same to define a region from which to extract a background spectrum, though -this region is of a different size and is not centered on the source: +this region is of a different size and is not centered on the source. We also set a +line-style for the background region, to distinguish it from the source and +calibration regions: ```{code-cell} python # The central coordinate of the background region @@ -2004,7 +2008,9 @@ back_coord = SkyCoord(81.1932474, -69.5073738, unit="deg", frame="icrs") back_reg_rad = Quantity(3, "arcmin") # Setting up a 'regions' module circular sky region instance for the background region -back_reg = CircleSkyRegion(back_coord, back_reg_rad, visual={"color": "red"}) +back_reg = CircleSkyRegion( + back_coord, back_reg_rad, visual={"color": "red", "linestyle": "dashed"} +) ``` #### Visualizing the source and background extraction regions on XRISM-Xtend images From 50d34a4d9ecc8bff95bd1ee6791c78b08dc0fc24 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 3 Feb 2026 11:36:24 -0500 Subject: [PATCH 204/221] Added links/mention of the XRISM helpdesk to the getting-started-xrism-xtend.md notebook. Also collapsed the HEASoft version comparison cell. --- .../xrism/getting-started-xrism-xtend.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 0c699fdb..fd99966d 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1140,7 +1140,7 @@ glob.glob(os.path.join(ROOT_DATA_DIR, rel_obsids[0], "xtend", "") + "**/*") There are multiple steps involved in processing XRISM-Xtend data into a science-ready state. As with many NASA-affiliated high-energy missions, HEASoft -includes a beginning-to-end pipeline(s) to streamline this process for XRISM data - the +includes a beginning-to-end pipeline to streamline this process for XRISM data - the XRISM-Xtend and Resolve instruments both have their own pipelines. In this tutorial we are focused only on preparing and using data from XRISM's Xtend @@ -1150,17 +1150,18 @@ to run either or both the Xtend and Resolve pipelines. It contains some convenie functionality that can identify and automatically pass the attitude, housekeeping, etc. files. We will show you how to run the Xtend-specific pipeline, `xtdpipeline`, but the -use of `xapipeline` is nearly functionally identical. +use of `xapipeline` is nearly identical. The Python interface to HEASoft, HEASoftPy, is used throughout this tutorial, and we -will implement parallel observation processing wherever possible. +will implement parallel observation processing wherever possible (even though we have +only selected a single observation). ### HEASoft and HEASoftPy versions ```{warning} XRISM is a relatively new mission, and as such the analysis software and recommended best practises are still immature and evolving. We are checking and updating this tutorial -on a regular basis, but please report any issues or suggestions to the HEASARC Help Desk. +on a regular basis, but please report any issues, or make suggestions, to the [XRISM Help Desk](https://heasarc.gsfc.nasa.gov/cgi-bin/Feedback?selected=xrism). ``` Both the HEASoft and HEASoftPy package versions can be retrieved from the @@ -1193,6 +1194,12 @@ HEA_VER We can now check that `HEA_VER` is greater than the minimum required version: ```{code-cell} python +--- +tags: [hide-input] +jupyter: + source_hidden: true +--- + if HEA_VER < Version("v6.36"): raise ValueError( "We strongly recommend using HEASoft v6.36 or later for this " @@ -2876,6 +2883,8 @@ Updated On: 2026-02-03 ### Additional Resources +**XRISM Help Desk**: https://heasarc.gsfc.nasa.gov/cgi-bin/Feedback?selected=xrism + **XRISM Data Reduction (ABC) Guide**: https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide **HEASoftPy GitHub Repository**: https://github.com/HEASARC/heasoftpy From 09573583b9a01a7e2b6230386fb18d2d87f0b96a Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 3 Feb 2026 12:17:11 -0500 Subject: [PATCH 205/221] Altered the look of the spec data + model figure produced at the end of getting-started-xrism-xtend.md notebook. For issue #128 and PR #150 --- .../xrism/getting-started-xrism-xtend.md | 68 +++++++++++-------- 1 file changed, 41 insertions(+), 27 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index fd99966d..f2ed1e60 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1199,7 +1199,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - if HEA_VER < Version("v6.36"): raise ValueError( "We strongly recommend using HEASoft v6.36 or later for this " @@ -1339,8 +1338,8 @@ very precise timings and count-rates. ``` Though we are using the HEASoftPy `xtdpipeline` function, called -as `hsp.xtdpipeline(indir=...)`, it is called within a wrapper function we have -written in the 'Global Setup: Functions' section of this notebook. The `process_xrism_xtend` +as `hsp.xtdpipeline(indir=...)`, we wrap it in a function defined in +the 'Global Setup: Functions' section of this notebook. The `process_xrism_xtend` wrapper function exists primarily to let us run the processing of different XRISM-Xtend observations in parallel. @@ -1404,7 +1403,7 @@ if len(xtd_pipe_problem_ois) != 0: ```{note} This notebook is configured to acquire XRISM CALDB files from the HEASARC Amazon Web Services S3 bucket - this can greatly improve the speed of some -steps later in the notebook when running on the Fornax science console. +steps later in the notebook when running on the Fornax Science Console. CALDB location configuration can be found in the 'Global Setup: Configuration' section. ``` @@ -1633,7 +1632,7 @@ bin_factors = [1, 4] #### Running image generation There is no HEASoft tool specifically for generating XRISM-Xtend images, but there is a -generalized HEASoft image (and other data products) generation task that we can use. +generalized HEASoft image (and other data product) generation task that we can use. If you have previously generated images, light curves, or spectra from HEASARC-hosted X-ray data on the command line, you may well have come across `XSELECT`; a HEASoft @@ -1689,8 +1688,8 @@ in the 'Global Setup: Functions' section of this notebook to make it easier to r this task in parallel. There are two `xaexpmap` configuration options which control how the -attitude (essentially pointing) of XRISM over the course of the observation is -binned spatially. These bins ('off-axis wedges' as the +attitude (essentially where the telescope is pointing) of XRISM over the course of +the observation is binned spatially. These bins ('off-axis wedges' as the [`xaexpmap` documentation](https://heasarc.gsfc.nasa.gov/docs/software/lheasoft/help/xaexpmap.html) describes them) are where the initial 'time intervals' of observation coverage are calculated: - **Radial Delta** - Passed to `xaexpmap` as `delta`. Radial increment (in arcmin) for the annular grid for which the attitude histogram will be calculated. The annuli are centered on the optical axis (off-axis angle = 0), and the central circle has a radius equal to `delta`. @@ -1913,9 +1912,9 @@ This will be quite simple, as HEASoft includes a pre-made region file that defin location and extent of the emission from the calibration sources. A small difficulty arises from the fact that this pre-made region file is defined in -detector coordinates, rather than the RA-Dec coordinates we've been using so far. That -will be pretty easy to deal with, however, as HEASoft includes a tool to transform region -files between different coordinate systems. +detector coordinates, rather than the RA-Dec coordinates we're going to use for the +source region. That will be pretty easy to deal with, however, as HEASoft includes a +tool to transform region files between different coordinate systems. ```{code-cell} python # The path to the HEASoft-supplied XRISM-Xtend calibration source region file @@ -2174,7 +2173,7 @@ Spectral data products generated for high-energy missions typically contain a measurement of their extraction region area. This is in order to scale source and background spectra properly when -Our calculation of 'BACKSCAL' doesn't only benefit our spectra analyses, as when we +Our calculation of 'BACKSCAL' doesn't only benefit our spectral analyses, as when we demonstrate the creation of light curves later in this notebook, we can also use the values to weight our subtraction of the background. @@ -2376,7 +2375,7 @@ that it can be sped up, though at the cost of that accuracy - the most direct wa to limit the number of events that are simulated. Rather than setting an overall number of events to simulate, the `xaarfgen` task provides -an argument ('numphoton') to set the number og photons allocated to each attitude +an argument ('numphoton') to set the number of photons allocated to each attitude histogram bin (in the exposure map file), per grid point in the internal energy grid. An argument specifying the number of events ('numphoton') can be passed to `xaarfgen`, and for @@ -2391,24 +2390,24 @@ not reached for each energy grid point during the raytracing process, ARF produc The `xaarfgen` documentation provides the following guidance on choosing the number of events to simulate: ```{seealso} -Note that even if minphoton is exceeded at all energies, this does not guarantee +Note that even if `minphoton` is exceeded at all energies, this does not guarantee that the resulting ARF is robust and sufficiently accurate. In general, about 5000 or more photons per energy (over the extraction region) give good results, but the actual minimum number varies case-by-case, and fewer may be sufficient in some cases. -The default value of minphoton is deliberately very small, in order that the +The default value of `minphoton` is deliberately very small, in order that the ARF is made and available for diagnostic evaluation. In general, it is not -recommended to set 'minphoton' to a high value in the first place, because it is -not possible to reliably estimate what 'minphoton' should be in advance of -running raytracing within xaarfgen, in order for that value of 'photon' to be +recommended to set `minphoton` to a high value in the first place, because it is +not possible to reliably estimate what `minphoton` should be in advance of +running raytracing within `xaarfgen`, in order for that value of 'photon' to be satisfied for all energies, which could result in repeated failures after very long run times. It could also run into memory problems and/or a raytracing file size that is unmanageable. ``` -We choose the default values for both the 'minphoton' and 'numphoton' arguments: +We choose the default values for both the `minphoton` and `numphoton` arguments: ```{code-cell} python arf_rt_num_photons = 20000 @@ -2675,7 +2674,7 @@ performed using the [PyXspec](https://heasarc.gsfc.nasa.gov/docs/software/xspec/ ### Configuring PyXspec -Now we configure some behaviors of XSPEC/pyXspec: +Now we configure some behaviors of XSPEC/PyXspec: - The ```chatter``` parameter is set to zero to reduce printed output during fitting (note that some XSPEC messages are still shown). - We inform XSPEC of the number of cores we have available, as some XSPEC methods can be paralleled. - We tell XSPEC to use the Cash statistic for fitting (the reason we grouped our spectra earlier). @@ -2686,7 +2685,7 @@ Now we configure some behaviors of XSPEC/pyXspec: # the top of your file, but this is unfortunately necessary at the moment import xspec as xs # noqa: E402 -# Limits the amount of output from XSPEC that pyXspec will display +# Limits the amount of output from XSPEC that PyXspec will display xs.Xset.chatter = 0 # Other xspec settings @@ -2700,12 +2699,12 @@ xs.Fit.nIterations = 500 ```{danger} There is a known issue with the version of PyXspec shipped in HEASoft v6.36 (and -possibly later versions) that will cause the parallelised generation of data products +possibly later versions) that can cause the parallelised generation of data products to hang forever. We avoid this here by importing PyXspec **after** all data product generation is complete. ``` -### Reading a XRISM-Xtend spectrum into pyXspec +### Reading a XRISM-Xtend spectrum into PyXspec Here we define the ObsID and dataclass of the spectrum we want to fit: @@ -2714,7 +2713,7 @@ chosen_demo_spec_obsid = "000128000" chosen_demo_spec_dataclass = "31100010" ``` -The spectrum, and all of its supporting files, are then read into pyXspec: +The spectrum, and all of its supporting files, are then read into PyXspec: ```{code-cell} python # In case this cell is re-run, clear all previously loaded spectra @@ -2752,7 +2751,7 @@ cur_arf = ARF_PATH_TEMP.format( rad=src_reg_rad.to("deg").value.round(4), ) -# Load the chosen spectrum (and all its supporting files) into pyXspec +# Load the chosen spectrum (and all its supporting files) into PyXspec xs_spec = xs.Spectrum(cur_spec, backFile=cur_bspec, respFile=cur_rmf, arfFile=cur_arf) ``` @@ -2803,7 +2802,7 @@ xs.AllModels.show() xs.Xset.chatter = 0 ``` -### Fitting our pyXspec model to the XRISM-Xtend spectrum +### Fitting our PyXspec model to the XRISM-Xtend spectrum Performing the fit is simple: @@ -2853,8 +2852,19 @@ plt.figure(figsize=(7, 4)) plt.minorticks_on() plt.tick_params(which="both", direction="in", top=True, right=True) -plt.errorbar(spec_en, spec_cr, xerr=spec_en_err, yerr=spec_cr_err, fmt="kx", capsize=2) -plt.plot(spec_en, spec_mod_cr, color="tab:cyan", alpha=0.7, lw=1.1) +plt.errorbar( + spec_en, + spec_cr, + xerr=spec_en_err, + yerr=spec_cr_err, + fmt="kx", + capsize=2, + alpha=0.7, + label="XRISM-Xtend data", +) +plt.plot( + spec_en, spec_mod_cr, color="deepskyblue", alpha=1, lw=1.2, label="Fitted model" +) # Make sure the energy axis is log scaled plt.xscale("log") @@ -2864,9 +2874,13 @@ plt.xscale("log") plt.gca().xaxis.set_major_formatter(FuncFormatter(lambda inp, _: "{:g}".format(inp))) plt.gca().xaxis.set_minor_formatter(FuncFormatter(lambda inp, _: "{:g}".format(inp))) +# Label the X and Y axes plt.xlabel("Energy [keV]", fontsize=15) plt.ylabel(r"Spectrum [ct cm$^{-2}$ s$^{-1}$ keV$^{-1}$]", fontsize=15) +# Add a legend +plt.legend(loc="best", fontsize=14) + plt.tight_layout() plt.show() ``` From 09b2d49c558ac2c15a603297c98a24ba986fc6a6 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 3 Feb 2026 12:49:17 -0500 Subject: [PATCH 206/221] Changed the look of the final spectrum plot again, and switched the XSPEC model to something a little more suitable (hopefully?). All in getting-started-xrism-xtend.md, for issue #128 and PR #150 --- .../xrism/getting-started-xrism-xtend.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index f2ed1e60..a69e718e 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2790,7 +2790,7 @@ A full list of XSPEC model components can be found in the [XSPEC documentation]( Our choice of model is empirically driven, chosen by someone who is not a specialist in supernova remnants, and should definitely not be considered as scientifically useful! ```{code-cell} python -xs.Model("tbabs*(powerlaw+apec+bbody)") +xs.Model("vapec+vrnei+powerlaw") ``` If we temporarily increase PyXspec's chatter level, we can see the default values @@ -2859,11 +2859,18 @@ plt.errorbar( yerr=spec_cr_err, fmt="kx", capsize=2, - alpha=0.7, + alpha=0.5, label="XRISM-Xtend data", + zorder=1, ) plt.plot( - spec_en, spec_mod_cr, color="deepskyblue", alpha=1, lw=1.2, label="Fitted model" + spec_en, + spec_mod_cr, + color="mediumorchid", + alpha=1, + lw=2, + label="Fitted model", + zorder=2, ) # Make sure the energy axis is log scaled From 96b8953ec79f9548c5c251045a92c61b46e8f779 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 10 Feb 2026 11:03:22 -0500 Subject: [PATCH 207/221] Small typo corrections to the gen_xrism_xtend_spectrum function docstring. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a69e718e..00daf3d8 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -605,9 +605,9 @@ def gen_xrism_xtend_spectrum( dataclass information will be extracted from the EVENTS table header. :param str out_dir: The directory where output files should be written. :param SkyCoord rel_src_coord: The source coordinate (RA, Dec) of the - source region for which we wish to generate a light curve. + source region for which we wish to generate a spectrum. :param Quantity rel_src_radius: The radius of the source region for which we wish - to generate a light curve. + to generate a spectrum. :param str src_reg_file: Path to the region file defining the source region for which we wish to generate a spectrum. :param str back_reg_file: Path to the region file defining the background region From a97a54a3ff1e29567aac04d6c7dce22ec14dc43b Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 11 Feb 2026 09:22:07 -0500 Subject: [PATCH 208/221] Small comment typo fixed --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 00daf3d8..11d1588a 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -656,7 +656,7 @@ def gen_xrism_xtend_spectrum( clobber=True, ) - # Now for the background light curve + # Now for the background spectrum back_out = hsp.extractor( filename=os.path.relpath(event_file), phafile=sp_back_out, From 68abe3583fa6b95885da62c1d3b8d53908717a14 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Mar 2026 12:17:22 -0500 Subject: [PATCH 209/221] Cleaned up some of the collapsed cells in the getting-started-xrism-xtend.md notebook a little. --- .../xrism/getting-started-xrism-xtend.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 11d1588a..49d33db1 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2026-02-03' +date: '2026-03-02' file_format: mystnb jupytext: text_representation: @@ -64,7 +64,7 @@ We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstr ### Inputs -- The name of the source of interest - in this case *LMC N132D* +- The name of the source of interest, in this case *LMC N132D* ### Outputs @@ -74,7 +74,7 @@ We make use of the HEASoftPy interface to HEASoft tasks throughout this demonstr ### Runtime -As of 2nd February 2026, this notebook takes ~50 m to run to completion on Fornax using the 'Default Astrophysics' image and the medium server with 16GB RAM/ 4 cores. +As of 2nd March 2026, this notebook takes ~50 m to run to completion on Fornax using the 'Default Astrophysics' image and the medium server with 16GB RAM/ 4 cores. ## Imports @@ -112,6 +112,7 @@ tags: [hide-input] jupyter: source_hidden: true --- + def process_xrism_xtend( cur_obs_id: str, out_dir: str, @@ -836,6 +837,7 @@ tags: [hide-input] jupyter: source_hidden: true --- + # The name of the source we're examining in this demonstration SRC_NAME = "LMCN132D" @@ -854,6 +856,7 @@ tags: [hide-input] jupyter: source_hidden: true --- + # ------------- Configure global package settings -------------- # Raise Python exceptions if a heasoftpy task fails # TODO Remove once this becomes a default in heasoftpy @@ -1199,6 +1202,7 @@ tags: [hide-input] jupyter: source_hidden: true --- + if HEA_VER < Version("v6.36"): raise ValueError( "We strongly recommend using HEASoft v6.36 or later for this " @@ -1284,11 +1288,11 @@ and we summarize them here: - **Full-window** - The entire Xtend detector is in the same data mode, producing a 640x640 (in raw CCD coordinates) image, at a 4-second time resolution. - **1/8th window [NO BURST]** - Half of the detector (2 CCDs) operates 'normally' and the other has only 1/8th of the pixel rows operating. Produces a 640x80 image, at 0.5-second time resolution. -- **1/8th window [BURST]** - Similar to the 1/8th window mode but collects exposures during only a small fraction of the effective detector exposures (the 0.5-second time resolution). Avoids pile-up for very bright sources and allows the determination of photon arrival times with ~0.06 second accuracy. +- **1/8th window [BURST]** - Similar to the 1/8th window mode but collects exposures during only a small fraction of the effective detector exposures (the 0.5-second time resolution). Avoids pile-up for very bright sources and allows the determination of photon arrival times with ~0.06-second accuracy. Data taken in each data mode is assigned a different 'dataclass' so that the multiple event lists produced when using a 1/8th window mode can be distinguished from the event -list of the half the detector that is operating 'normally'. The dataclasses are discussed in the XRISM ABC guide +list of the half of the detector that is operating 'normally'. The dataclasses are discussed in the XRISM ABC guide ([XRISM GOF & SDC 2024](https://heasarc.gsfc.nasa.gov/docs/xrism/analysis/abc_guide/XRISM_Data_Specifics.html)) and are summarized below: - **30000010** - All four CCDs are in full window mode. - **31100010** - CCD1 & CCD2 in 1/8 window mode. @@ -1516,6 +1520,7 @@ tags: [hide-input] jupyter: source_hidden: true --- + # Set up the figure plt.figure(figsize=(5.5, 5.5)) @@ -1818,6 +1823,7 @@ tags: [hide-input] jupyter: source_hidden: true --- + fig, ax_arr = plt.subplots(ncols=2, figsize=(6, 6), width_ratios=[3, 1]) plt.subplots_adjust(wspace=0) @@ -2848,10 +2854,14 @@ tags: [hide-input] jupyter: source_hidden: true --- + +# Visualizing the fitted XRISM-Xtend spectrum +# First, set up the figure size, configure the axis tick appearance plt.figure(figsize=(7, 4)) plt.minorticks_on() plt.tick_params(which="both", direction="in", top=True, right=True) +# Show the spectrum data points, with uncertainties plt.errorbar( spec_en, spec_cr, @@ -2863,6 +2873,7 @@ plt.errorbar( label="XRISM-Xtend data", zorder=1, ) +# Overplot (zorder=2 will ensure the fit line goes over the data) the model that we fit using PyXpec. plt.plot( spec_en, spec_mod_cr, @@ -2898,7 +2909,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2026-02-03 +Updated On: 2026-03-02 +++ From 6f2af276710c5cc55cbb268c26705de05c83c3ac Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Mar 2026 12:53:49 -0500 Subject: [PATCH 210/221] Fixed a comment that didn't pass pre-commit on CI/CD, and added another nonsense comment that definitely won't pass, to see if pre-commit is working locally now. --- .../xrism/getting-started-xrism-xtend.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 49d33db1..1863cf62 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2873,7 +2873,9 @@ plt.errorbar( label="XRISM-Xtend data", zorder=1, ) -# Overplot (zorder=2 will ensure the fit line goes over the data) the model that we fit using PyXpec. +# Overplot (zorder=2 will ensure the fit line goes over the data) the model +# that we fit using PyXpec. +# TESTIOOOOOOOOOOOODODODODODODODODODODODODODODODODODSODSOADSOADSOADSKODASJNODSAJODSAIODAADSOU plt.plot( spec_en, spec_mod_cr, From 0ae25e35a73a1ac3ebe8af86b3183c8763033c25 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Mar 2026 12:59:13 -0500 Subject: [PATCH 211/221] Made the nonsense overlong comment even worse, still trying to test local pre-commit on this machine --- .../xrism/getting-started-xrism-xtend.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 1863cf62..a37ad904 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2873,9 +2873,9 @@ plt.errorbar( label="XRISM-Xtend data", zorder=1, ) -# Overplot (zorder=2 will ensure the fit line goes over the data) the model +# Overplot (zorder=2 will ensure the fit line goes over the data) the model # that we fit using PyXpec. -# TESTIOOOOOOOOOOOODODODODODODODODODODODODODODODODODSODSOADSOADSOADSKODASJNODSAJODSAIODAADSOU +# TESTIOOOOOOOOOOOODODODODODODODODODODODODODODODODODSODSOADSOADSOADSKODASJNODSAJODSAIODAADSOUdsa plt.plot( spec_en, spec_mod_cr, From 4fa86f74ed72887419db3b87ccb60e5498f63a41 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Mar 2026 13:06:34 -0500 Subject: [PATCH 212/221] Removed the nonsense comment, will have to find a better way to test why pre-commit isn't quite working right --- .../xrism/getting-started-xrism-xtend.md | 1 - 1 file changed, 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index a37ad904..4c11c49f 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2875,7 +2875,6 @@ plt.errorbar( ) # Overplot (zorder=2 will ensure the fit line goes over the data) the model # that we fit using PyXpec. -# TESTIOOOOOOOOOOOODODODODODODODODODODODODODODODODODSODSOADSOADSOADSKODASJNODSAJODSAIODAADSOUdsa plt.plot( spec_en, spec_mod_cr, From ea50e33b977c8043d7fe830a5c10629cbc6cc0a0 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Mar 2026 13:11:23 -0500 Subject: [PATCH 213/221] Added a single new line to trigger a rebuild of getting-started-xrism-xtend.md now that the pre-commit issue is sort of fixed --- .../xrism/getting-started-xrism-xtend.md | 1 + 1 file changed, 1 insertion(+) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 4c11c49f..2c11bc4c 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2873,6 +2873,7 @@ plt.errorbar( label="XRISM-Xtend data", zorder=1, ) + # Overplot (zorder=2 will ensure the fit line goes over the data) the model # that we fit using PyXpec. plt.plot( From 8b7f199388a0084e0a6afba4cbeecb444066cc34 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Mar 2026 13:53:36 -0500 Subject: [PATCH 214/221] Tiny change to trigger rebuild please? --- .../xrism/getting-started-xrism-xtend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 2c11bc4c..24d8d6a9 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2602,7 +2602,7 @@ for oi, dcs in rel_dataclasses.items(): ) # The 'lcmath' tool is sensitive to long paths, so we fetch the relative - # paths to pass in + # paths to pass it instead of the absolute paths cur_lc = os.path.relpath(cur_lc) cur_blc = os.path.relpath(cur_blc) cur_nlc = os.path.relpath(cur_nlc) From 7ebbe46ed66e0886d3ba1522b502981a48aa06d1 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Mar 2026 16:16:21 -0500 Subject: [PATCH 215/221] Changed getting-started-xrism-xtend.md to only include a single 'not recommended' energy band in the image generation part, two seemed excessive. --- .../xrism/getting-started-xrism-xtend.md | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 24d8d6a9..d31d57b6 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -112,7 +112,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - def process_xrism_xtend( cur_obs_id: str, out_dir: str, @@ -837,7 +836,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - # The name of the source we're examining in this demonstration SRC_NAME = "LMCN132D" @@ -856,7 +854,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - # ------------- Configure global package settings -------------- # Raise Python exceptions if a heasoftpy task fails # TODO Remove once this becomes a default in heasoftpy @@ -1202,7 +1199,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - if HEA_VER < Version("v6.36"): raise ValueError( "We strongly recommend using HEASoft v6.36 or later for this " @@ -1520,7 +1516,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - # Set up the figure plt.figure(figsize=(5.5, 5.5)) @@ -1583,17 +1578,14 @@ We are going to generate images within the following energy bounds: - 0.6-2.0 keV - 2.0-10.0 keV - ***0.4–2.0 keV*** [not recommended] -- ***0.4–10.0 keV*** [not recommended] -The bands that have a lower bound of ***0.4 keV*** are ***not recommended***, as there +A lower bound of ***0.4 keV*** is ***not recommended***, as there are issues with XRISM-Xtend data below *0.6 keV*. We are generating them to demonstrate those issues. ```{code-cell} python # Defining the energy bounds we want images within -xtd_im_en_bounds = Quantity( - [[0.6, 10.0], [0.6, 2.0], [2.0, 10.0], [0.4, 2.0], [0.4, 10.0]], "keV" -) +xtd_im_en_bounds = Quantity([[0.6, 10.0], [0.6, 2.0], [2.0, 10.0], [0.4, 2.0]], "keV") ``` Converting those energy bounds to channel bounds is straightforward, we simply divide @@ -1823,7 +1815,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - fig, ax_arr = plt.subplots(ncols=2, figsize=(6, 6), width_ratios=[3, 1]) plt.subplots_adjust(wspace=0) @@ -2854,7 +2845,6 @@ tags: [hide-input] jupyter: source_hidden: true --- - # Visualizing the fitted XRISM-Xtend spectrum # First, set up the figure size, configure the axis tick appearance plt.figure(figsize=(7, 4)) From f16169af6af67bf75873ded78b45e09f3a1f37dd Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 2 Mar 2026 19:06:26 -0500 Subject: [PATCH 216/221] Added the new core-determining logic I came up with for the RASS notebook to the getting-started-xrism-xtend.md notebook. Sets the number of available cores differently when running on CircleCI. --- .../xrism/getting-started-xrism-xtend.md | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index d31d57b6..60e7de88 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -868,11 +868,33 @@ os.environ["CALDB"] = "https://nasa-heasarc.s3.amazonaws.com/caldb" # -------------------------------------------------------------- # ------------- Setting how many cores we can use -------------- -NUM_CORES = None +# We use a service called CircleCI to execute, test, and validate these notebooks +# as we're writing and maintaining them. Unfortunately we have to treat the +# determination of the number of cores we can use differently, as the +# 'os.cpu_count()' call will return the number of cores of the host machine, rather +# than the number that have actually been allocated to us. +if "CIRCLECI" in os.environ and bool(os.environ["CIRCLECI"]): + # Here we read the CPU quota (total CPU time allowed) and the CPU period (how + # long the scheduling window is) from a cgroup (a linux kernel feature) file. + # Dividing one by t'other provides the number of cores we've been allocated. + with open("/sys/fs/cgroup/cpu.max", "r") as cpu_maxo: + quota, period = cpu_maxo.read().strip().split() + NUM_CORES = int(quota) // int(period) + +# If you, the reader, are running this notebook yourself, this is the +# part that is relevant to you - you can override the default number of cores +# used by setting this variable to an integer value. +else: + NUM_CORES = None + +# Determines the number of CPU cores available total_cores = os.cpu_count() +# If NUM_CORES is None, then we use the number of cores returned by 'os.cpu_count()' if NUM_CORES is None: NUM_CORES = total_cores +# Otherwise, NUM_CORES has been overridden (either by the user, or because we're +# running on CircleCI, and we do a validity check. elif not isinstance(NUM_CORES, int): raise TypeError( "If manually overriding 'NUM_CORES', you must set it to an integer value." From f7a8a47be3f71de6199302309e8c958e4910ad55 Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 3 Mar 2026 12:33:09 -0500 Subject: [PATCH 217/221] Pointed the circleci config at the main fornax-hea image, and got rid of the step where we install extra deps (shouldn't be needed with this image). --- .circleci/config.yml | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 25a2e5a4..f1637b6e 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -48,7 +48,7 @@ jobs: # The Fornax-Hea images can definitely be used here, just we're having some issues with what supporting files # software are included at the moment, so it is easier to use a different image and just install HEASoft # at the moment - - image: ghcr.io/nasa-fornax/fornax-images/fornax-hea:update-addHEASoft6.36 + - image: ghcr.io/nasa-fornax/fornax-images/fornax-hea:main # This sets the amount of compute that we'll use to run the on-PR builds/tests - this particular setup corresponds # to 3 CPUs and 6GB of RAM as of 2nd February 2026 resource_class: large @@ -125,19 +125,19 @@ jobs: rm $ENV_DIR/heasoft/heasoft/spectral/modelData ln -s $(pwd)/xspec-data/modelData $ENV_DIR/heasoft/heasoft/spectral/modelData - - run: - name: Installing extra dependencies - # TODO THIS METHOD OF DEFINING DEPS IS NOT GOOD ENOUGH, EVEN FOR A TEMPORARY SOLUTION - command: | - micromamba install -y -c conda-forge -n heasoft astroquery pyvo tqdm aplpy s3fs boto3 scikit-learn umap-learn - micromamba run -n heasoft pip install xga - micromamba install -y -c conda-forge -n sas astroquery pyvo tqdm aplpy s3fs boto3 - micromamba run -n sas pip install xga + #- run: + # name: Installing extra dependencies + # # TODO THIS METHOD OF DEFINING DEPS IS NOT GOOD ENOUGH, EVEN FOR A TEMPORARY SOLUTION + # command: | + # micromamba install -y -c conda-forge -n heasoft astroquery pyvo tqdm aplpy s3fs boto3 scikit-learn umap-learn + # micromamba run -n heasoft pip install xga + # micromamba install -y -c conda-forge -n sas astroquery pyvo tqdm aplpy s3fs boto3 + # micromamba run -n sas pip install xga - run: name: Create the Sphinx build environment command: | - micromamba create -n build_docs -y -c conda-forge sphinx sphinx-book-theme sphinx-copybutton myst-nb + micromamba create -n build_docs -y -c conda-forge "sphinx<9" sphinx-book-theme sphinx-copybutton myst-nb # To ensure that the build environment can activate the HEASoft, CIAO, etc. kernels when required - run: From bebc25d25d0530e9c39cbe676c68030491425ffe Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 3 Mar 2026 12:41:01 -0500 Subject: [PATCH 218/221] Trying to make the net-light-curve code execute again in the getting-started-xrism-xtend.md notebook. --- .../xrism/getting-started-xrism-xtend.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 60e7de88..80033868 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2026-03-02' +date: '2026-03-03' file_format: mystnb jupytext: text_representation: @@ -2573,6 +2573,10 @@ interface) to subtract the background from the source light curve. This operatio computationally cheap for the number of light curves we are working with, but you should consider parallelizing this step if you are working with significantly more: +```{code-cell} python +os.listdir("XRISM_output/000128000") +``` + ```{code-cell} python for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: @@ -2923,7 +2927,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2026-03-02 +Updated On: 2026-03-03 +++ From 9ab2dcd26ede3e2bc45936264b7b1414d9857a1d Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 3 Mar 2026 15:11:38 -0500 Subject: [PATCH 219/221] Making sure to use directory and heasoftpy par context managers for the bit of getting-started-xrism-xtend.md that is failing on CircleCI. Who knows, maybe it'll even work this time? --- .../xrism/getting-started-xrism-xtend.md | 45 ++++++++++--------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 80033868..b78b0874 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -2573,10 +2573,6 @@ interface) to subtract the background from the source light curve. This operatio computationally cheap for the number of light curves we are working with, but you should consider parallelizing this step if you are working with significantly more: -```{code-cell} python -os.listdir("XRISM_output/000128000") -``` - ```{code-cell} python for oi, dcs in rel_dataclasses.items(): for cur_dc in dcs: @@ -2618,24 +2614,29 @@ for oi, dcs in rel_dataclasses.items(): tb=lc_time_bin.to("s").value, ) - # The 'lcmath' tool is sensitive to long paths, so we fetch the relative - # paths to pass it instead of the absolute paths - cur_lc = os.path.relpath(cur_lc) - cur_blc = os.path.relpath(cur_blc) - cur_nlc = os.path.relpath(cur_nlc) - - # Calculate the scaling that should be applied to the background - # light curve before subtraction - cur_back_multi = spec_backscals[oi][cur_dc] / bspec_backscals[oi][cur_dc] - - # Run the tool to produce a net light curve - hsp.lcmath( - infile=cur_lc, - bgfile=cur_blc, - outfile=cur_nlc, - multi=1, - multb=cur_back_multi, - ) + with contextlib.chdir( + os.path.join(OUT_PATH, oi) + ), hsp.utils.local_pfiles_context(): + # The 'lcmath' tool is sensitive to long paths, so we fetch the + # relative paths to pass it instead of the absolute paths + cur_lc = os.path.relpath(cur_lc) + cur_blc = os.path.relpath(cur_blc) + cur_nlc = os.path.relpath(cur_nlc) + + # Calculate the scaling that should be applied to the background + # light curve before subtraction + cur_back_multi = ( + spec_backscals[oi][cur_dc] / bspec_backscals[oi][cur_dc] + ) + + # Run the tool to produce a net light curve + hsp.lcmath( + infile=cur_lc, + bgfile=cur_blc, + outfile=cur_nlc, + multi=1, + multb=cur_back_multi, + ) ``` #### Loading and displaying a single light curve From 07139153d56b05380cfe4d5908a50b129d5d87bd Mon Sep 17 00:00:00 2001 From: David Turner Date: Sat, 13 Jun 2026 11:52:27 -0400 Subject: [PATCH 220/221] Added cal file specification to the getting-started-xrism-xtend.md notebook. For PR #150 --- .../xrism/getting-started-xrism-xtend.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index b78b0874..2929b108 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -9,7 +9,7 @@ authors: affiliations: ['University of Maryland, Baltimore County', 'XRISM GOF, NASA Goddard'] website: https://science.gsfc.nasa.gov/sci/bio/kenji.hamaguchi-1 orcid: 0000-0001-7515-2779 -date: '2026-03-03' +date: '2026-06-13' file_format: mystnb jupytext: text_representation: @@ -21,6 +21,11 @@ kernelspec: display_name: heasoft language: python name: heasoft +execution: + cal-files: + xmm-ccf: False + chandra: False + xspec-models: True title: Getting started with XRISM-Xtend --- @@ -2928,7 +2933,7 @@ Author: David J Turner, HEASARC Staff Scientist. Author: Kenji Hamaguchi, XRISM GOF Scientist. -Updated On: 2026-03-03 +Updated On: 2026-06-13 +++ From f0fa11305076fd1372afdb7a05abb86cae893355 Mon Sep 17 00:00:00 2001 From: David Turner Date: Sat, 13 Jun 2026 20:28:47 -0500 Subject: [PATCH 221/221] Some small changes to the getting-started-xrism-xtend.md notebook - should make sure the light curve visualization includes the object name. --- .../xrism/getting-started-xrism-xtend.md | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md index 2929b108..0520d5ae 100644 --- a/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md +++ b/tutorials/mission_specific_analyses/xrism/getting-started-xrism-xtend.md @@ -1013,7 +1013,7 @@ ARF_PATH_TEMP = SP_PATH_TEMP.replace("-spectrum.fits", ".arf") *** -## 1. Finding and downloading XRISM observations of LMC N132D +## 1. Acquiring XRISM observations of LMC N132D Our first task is to determine which XRISM observations are relevant to the source that we are interested in. @@ -1022,7 +1022,7 @@ We are going in with the knowledge that LMC N132D has been observed by XRISM, bu course, there is no guarantee that _your_ source of interest has been, so this is an important exploratory step. -### Determining the name of the XRISM observation summary table +### Finding the name of the XRISM observation summary table HEASARC maintains tables that contain information about every observation taken by each of the missions in its archive. We will use XRISM's table to find observations @@ -1366,7 +1366,7 @@ very precise timings and count-rates. Though we are using the HEASoftPy `xtdpipeline` function, called as `hsp.xtdpipeline(indir=...)`, we wrap it in a function defined in -the 'Global Setup: Functions' section of this notebook. The `process_xrism_xtend` +the ['Global Setup: Functions'](#functions) section of this notebook. The `process_xrism_xtend` wrapper function exists primarily to let us run the processing of different XRISM-Xtend observations in parallel. @@ -1432,7 +1432,7 @@ This notebook is configured to acquire XRISM CALDB files from the HEASARC Amazon Web Services S3 bucket - this can greatly improve the speed of some steps later in the notebook when running on the Fornax Science Console. -CALDB location configuration can be found in the 'Global Setup: Configuration' section. +CALDB location configuration can be found in the ['Global Setup: Configuration'](#configuration) section. ``` ## 3. Generating new XRISM-Xtend images and exposure maps @@ -1668,7 +1668,7 @@ will now use to demonstrate the creation of XRISM-Xtend images. As with all uses of HEASoft tasks in this notebook, our call to `extractor` will be through the HEASoftPy Python interface - specifically the `hsp.extractor` function. -We have implemented a wrapper to this function in the 'Global Setup: Functions' section +We have implemented a wrapper to this function in the ['Global Setup: Functions'](#functions) section of this notebook, primarily so that we can easily multiprocess the generation of images in different energy bands, binning factors, observations, and dataclasses. @@ -1708,7 +1708,7 @@ effective sensitivity of Xtend as a function of energy. Unlike for image creation, XRISM does have a dedicated HEASoft task for the generation of exposure maps; `xaexpmap`. We have once again set up a wrapper function -in the 'Global Setup: Functions' section of this notebook to make it easier to run +in the ['Global Setup: Functions'](#functions) section of this notebook to make it easier to run this task in parallel. There are two `xaexpmap` configuration options which control how the @@ -1793,7 +1793,7 @@ That would result in empty light curve and spectrum products, and errors from 'BACKSCAL' calculation, and RMF and ARF generation. -### Visualize separate XRISM-Xtend 000128000 dataclass images +### Examining different dataclass images To make our point, and to give an example of the inspection you may want to perform before choosing the right dataclass for your target, we will visualize @@ -2043,9 +2043,9 @@ back_reg = CircleSkyRegion( ) ``` -#### Visualizing the source and background extraction regions on XRISM-Xtend images +#### Visualizing source and background extraction regions -We should inspect the regions in-situ to make sure they look sensible - first, our +We should inspect the regions in-situ to make sure they look sensible – first, our previously generated images are loaded in as `Image` class (from the `XGA` Python module) instances. @@ -2166,7 +2166,7 @@ required to actually fit models to the spectra. #### Generating the spectral files We set up a spectrum-generation-specific region wrapper function for the HEASoftPy -interface to the `extractor` task (see the Global Setup: Functions section near the +interface to the `extractor` task (see the ['Global Setup: Functions'](#functions) section near the top of the notebook). This once again allows us to parallelize the generation of spectra for different @@ -2281,7 +2281,7 @@ spec_group_scale = 1 Now we run the grouping tool - though this time we do not parallelize the task, as the grouping process is fast, and we wish to demonstrate how you use a HEASoftPy -function directly. Though remember to look at the Global Setup section of this notebook +function directly. Though remember to look at the ['Global Setup'](#global-setup) section of this notebook to see how we call HEASoftPy tools in the wrapper functions used to parallelize those tasks. @@ -2333,7 +2333,7 @@ Now we wish to generate new RMFs, so we can ensure they are entirely up to date! We make use of the XRISM-Xtend specific HEASoft task `xtdrmf` - the only input it requires is the path to the spectral file for which we wish to generate an RMF. -Our `gen_xrism_xtend_rmf` function (defined in the Global Setup: Functions section near +Our `gen_xrism_xtend_rmf` function (defined in the ['Global Setup: Functions'](#functions) section near the top of the notebook) wraps the HEASoftPy interface to the `xtdrmf` task. We now use it to generate RMFs in parallel for all of our new spectra: @@ -2438,8 +2438,8 @@ arf_rt_num_photons = 20000 arf_rt_min_photons = 100 ``` -So now we move onto actually running the ARF generation - using the -`gen_xrism_xtend_arf` function defined in the Global Setup: Functions section (near the top of +So now we move onto actually running the ARF generation – using the +`gen_xrism_xtend_arf` function defined in the ['Global Setup: Functions'](#functions) section (near the top of the notebook), which wraps the HEASoftPy interface to the `xaarfgen` task. We now use it to generate ARFs in parallel for all of our new spectra: @@ -2534,7 +2534,7 @@ now generate the light curves within the source and background regions, for each the specified energy bands. As with previous steps, our motivation for writing a wrapper function (defined in the -Global Setup section) is to make it easy for us to run generation of different +['Global Setup'](#global-setup) section) is to make it easy for us to run generation of different light curves simultaneously: ```{code-cell} python @@ -2689,11 +2689,13 @@ demo_lc = LightCurve( lc_time_bin, ) +demo_lc.src_name = SRC_NAME + # Show a visualization of the LightCurve demo_lc.view() ``` -## 6. Fitting a spectral model to an XRISM-Xtend spectrum +## 6. Fitting a model to a XRISM-Xtend spectrum Finally, to show off the XRISM-Xtend products we just generated, we will perform a simple model fit to one of our spectra.