Using the tutorial example to analyze the nicer data, an error occurs,

[py3u]

Hello, I am trying to use the tutorial example and pyxspec, heasoftpy to analyze nicer data, and an error occurs,
detailed code：

# get the on-axis rmf
res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
             filter='-', date='-', time='-',expr='-',codename='MATRIX')
rmf = [x.split()[0] for x in res.output if 'nixtiref'  in x][0]

# get the on-axis arf
res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
             filter='-', date='-', time='-',expr='-',codename='SPECRESP')
arf = [x.split()[0] for x in res.output if 'nixtiaveonaxis'  in x][0]

---

IndexError Traceback (most recent call last)
Cell In[7], line 4
1 # get the on-axis rmf
2 res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
3 filter='-', date='-', time='-',expr='-',codename='MATRIX')
----> 4 rmf = [x.split()[0] for x in res.output if 'nixtiref' in x][0]
6 # get the on-axis arf
7 res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
8 filter='-', date='-', time='-',expr='-',codename='SPECRESP')

IndexError: list index out of range

Note:
heasoftpy 1.2.1 pypi_0 pypi

[zoghbi-a]

@mfacorcoran?

[mfacorcoran]

i'll check it out

[zoghbi-a]

The error is likely resulting from the CALDB environment variable not being defined. Can ensure you run the notebook where it is defined, or define it explicitly at the top of the notebook with something like:

import os
os.environ['CALDB'] = '/path/to/caldb'

[mfacorcoran]

or to use the HEASARC caldb you can define:

os.environ['CALDB'] = 'https://heasarc.gsfc.nasa.gov/FTP/caldb'

if you don't have the caldb installed locally

[py3u]

I set up caldb locally, for example:
caldbinfo INST NICER XTI
... using gpcaldbinfo 1.0.0
** caldbinfo 1.0.2
... using infocaldb 1.0.1
...... environ-var/logical CALDB defined
......... CALDB path = /home/ethan/workdir/caldb
... environ-var/logical CALDBCONFIG defined
... CALDBCONFIG file = /home/ethan/workdir/caldb/software/tools/caldb.config
...... environ-var/logical CALDBALIAS defined
......... CALDBALIAS file = /home/ethan/workdir/caldb/software/tools/alias_con
fig.fits
...... CALDB is configured for the XTI instrument onboard NICER
......... Cal Index File: /home/ethan/workdir/caldb/data/nicer/xti/caldb.indx
......... Instrument directory: /home/ethan/workdir/caldb/data/nicer/xti
... CALDB appears to be set-up & accessible
** caldbinfo 1.0.2 completed successfully

At the same time, I also called the python built-in library os, and edited the following code in the jupyter cell:

import os

os.environ['CALDB'] = '/home/jian/workdir/caldb'

Still have the same error，

# get the on-axis rmf
res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
             filter='-', date='-', time='-',expr='-',codename='MATRIX')
rmf = [x.split()[0] for x in res.output if 'nixtiref'  in x][0]

# get the on-axis arf
res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
             filter='-', date='-', time='-',expr='-',codename='SPECRESP')
arf = [x.split()[0] for x in res.output if 'nixtiaveonaxis'  in x][0]

---

IndexError Traceback (most recent call last)
Cell In[7], line 4
1 # get the on-axis rmf
2 res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
3 filter='-', date='-', time='-',expr='-',codename='MATRIX')
----> 4 rmf = [x.split()[0] for x in res.output if 'nixtiref' in x][0]
6 # get the on-axis arf
7 res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
8 filter='-', date='-', time='-',expr='-',codename='SPECRESP')

IndexError: list index out of range

[mfacorcoran]

If you do the following in your notebook:

import heasoftpy as hsp
import os

and set chatter=5 when running hsp.quzcif:

res = hsp.quzcif(mission='nicer', instrument='xti',detector='-',
...: filter='-', date='-', time='-',expr='-',codename='MATRIX', chatter=5)

print (res.output)
for o in res.stdout:
print(o)

what do you get?

res.output should be:

['https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xti/bcf/rmfbase/nixtirmfbase20170601v001.fits 2',
'https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xti/cpf/rmf/nixtiref20170601v003.rmf 2',
'https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xti/cpf/rmf/nixtidiagresp20170601v001.rmf 2', '']

this is what I get when I run that notebook cell

[py3u]

I can't run it, it seems that there is an error here,

Create the nicerl2 task

tstart = Time.now()
print(f'Start at: {tstart.iso[:19]}')
nicerl2 = hsp.HSPTask('nicerl2')
​
nicerl2.clobber="yes"
#nicerl2.cldir=outdir
#nicerl2.mkffile=mkf
# add the KP values to the mkf file during nicerl2 processing
nicerl2.geomag_path="https://heasarc.gsfc.nasa.gov/FTP/caldb/data/gen/pcf/geomag/"
nicerl2.geomag_columns="kp_noaa.fits(KP)"
​
resl2 = nicerl2(indir=nicerobsID, noprompt=True, cldir=outdir, mkfile=mkf)
​
tend = Time.now()
print(f'End at: {tend.iso[:19]}')
print(f'nicerl2 took: {(tend.mjd-tstart.mjd)*86400} seconds')
​
if resl2.returncode != 0:
    print('\n')
    for o in resl2.output[:]:
        print(o)

Start at: 2023-05-05 12:58:38
End at: 2023-05-05 12:58:38
nicerl2 took: 0.16225771978497505 seconds

ERROR: input directory 4142010107 does not exist at /home/ethan/workdir/heasoft-6.31.1/x86_64-pc-linux-gnu-libc2.31/bin/nicerl2 line 253.

Task nicerl2 1.27 terminating with status -1
nicerl2 1.27

  ang_dist = 0.015
  attfile = $INDIR/auxil/ni$OBSID.att
  autoscreen = yes
  br_earth = 30
  calstation = FLIGHT
  cldir = /home/ethan/tmp/nicer/nicerl2_output/4142010107_out
  cleanup = 1
  cleanup_ufa_files = 0
  clfile = $CLDIR/ni$OBSID_0mpu7_cl.evt
  clobber = 1
  cor_range = *-*
  dec = OBJ
  detlist = launch
  elv = 15
  filtcolumns = NICERV4
  geomag_columns = kp_noaa.fits(KP)
  geomag_path = https://heasarc.gsfc.nasa.gov/FTP/caldb/data/gen/pcf/geomag/
  gtifiles = NONE
  gzip_thresh = 50
  history = 1
  hkdir = $INDIR/xti/hk
  hkpat = $HKDIR/ni??????????_?mpu?.hk{,.gz}
  incremental = 0
  indir = 4142010107
  issmanfile = CALDB
  leapinit = AUTO
  min_fpm = 7
  mkfile = /home/ethan/tmp/nicer/nicerl2_output/4142010107_out/ni4142010107.mkf
  mpugtimerge = OR
  mpugtiscr = DEFAULT
  mpulist = 0-6
  nicercal_filtexpr = EVENT_FLAGS=bxxxx00
  nicerclean_args = NONE
  nicersaafilt = yes
  nimaketime_gtiexpr = NONE
  niprefilter = 1
  niprefilter2 = 1
  niprefilter2_coltypes = FILTCOLUMNS
  noise25scr = DEFAULT
  noiseextscr = DEFAULT
  orbfile = $INDIR/auxil/ni$OBSID.orb
  overonly_expr = NONE
  overonly_range = *-30
  overonlyscr = DEFAULT
  picalfile = CALDB
  pifastcalfile = CALDB
  pirange = 20:1500
  prefilter_columns = FILTCOLUMNS
  ra = OBJ
  robofile = CALDB
  roundrobbinscr = DEFAULT
  saafilt = no
  saaregfile = NONE
  st_valid = yes
  tasks = ALL
  timebiascalfile = CALDB
  trackfilt = yes
  trumpetfilt = yes
  trumpetkeep = GOOD
  ufafile = $CLDIR/ni$OBSID_0mpu7_ufa.evt
  ufdir = $INDIR/xti/event_uf
  ufpat = $UFDIR/ni??????????_?mpu?_uf.evt{,.gz}
  underonly_range = 0-500
  underonlyscr = DEFAULT
  vehiclefile = CALDB

MPU List: 0-6
Requested task operations: SCREEN,CALMERGE,MKF
Processed prefilter_columns: NICERV4,-MPU_OVER_COUNT,-MPU_UNDER_COUNT,-MPU_XRAY_COUNT,-MPU_ALL_COUNT
Processed niprefilter2_coltypes: BASE,NICERV4
Processed geomag_columns: kp_noaa.fits(KP)

[mfacorcoran]

the problem above seems to be that the input obsid directory, 4142010107 does not exist. But that seems to be separate from the hsp.quzcif error.

Please just try to run the hsp.quzcif code I sent earlier, i.e.

import heasoftpy as hsp
import os
res = hsp.quzcif(mission='nicer', instrument='xti',detector='-', filter='-', date='-', time='-',expr='-',codename='MATRIX', chatter=5)

print (res.output)
for o in res.stdout:
       print(o)

which you can do without running nicerl2

[py3u]

The result of running

import heasoftpy as hsp
import os
res = hsp.quzcif(mission='nicer', instrument='xti',detector='-', filter='-', date='-', time='-',expr='-',codename='MATRIX', chatter=5)

print (res.output)
for o in res.stdout:
       print(o)

is as follows:

[' ERROR - rcnfgl 2.1: Error reading record          5 of CALDBCONFIG file', ' ERROR - rcnfig 2.1.1: Could not find a match for NICER and XTI in caldbconfig f', '         ile: STOPPING', '']
 
E
R
R
O
R
 
-
 
r
c
n
f
g
l
 
2
.
1
:
 
E
r
r
o
r
 
r
e
a
d
i
n
g
 
r
e
c
o
r
d
 
 
 
 
 
 
 
 
 
 
5
 
o
f
 
C
A
L
D
B
C
O
N
F
I
G
 
f
i
l
e


 
E
R
R
O
R
 
-
 
r
c
n
f
i
g
 
2
.
1
.
1
:
 
C
o
u
l
d
 
n
o
t
 
f
i
n
d
 
a
 
m
a
t
c
h
 
f
o
r
 
N
I
C
E
R
 
a
n
d
 
X
T
I
 
i
n
 
c
a
l
d
b
c
o
n
f
i
g
 
f


 
 
 
 
 
 
 
 
 
i
l
e
:
 
S
T
O
P
P
I
N
G

[mfacorcoran]

Dear Jian,

This is helpful - I think the problem is that your CALDBCONFIG and CALDBALIAS environment variables are not defined. Please see https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_remote_access.html for instructions on how to download the caldbconfig and caldbalias files and how to define the environment variables. Hopefully that solves your problem

[py3u]

Very strange, I can run it in sciserver_cookbooks, but an error will appear on my own computer.

[mfacorcoran]

Hi Jian,

you should add these statements:

os.environ['CALDB'] = '/FTP/caldb'
os.environ['CALDBCONFIG'] = '/FTP/caldb/software/tools/caldb.config'
os.environ['CALDBALIAS']= '/FTP/caldb/software/tools/alias_config.fits'

to your copy of the notebooks, then caldb access should work properly.

[py3u]

When I run this code,

import os

os.environ['CALDB'] = 'https://heasarc.gsfc.nasa.gov/FTP/caldb'
os.environ['CALDBCONFIG'] = '/home/jian/workdir/caldb/software/tools/caldb.config'
os.environ['CALDBALIAS']= '/home/jian/workdir/caldb/software/tools/alias_config.fits'

[' ...... environ-var/logical CALDB defined', ' ......... CALDB path = https://heasarc.gsfc.nasa.gov/FTP/caldb', 'CALDB_PATH=(https://heasarc.gsfc.nasa.gov/FTP/caldb)', ' ... environ-var/logical CALDBCONFIG defined', ' ... CALDBCONFIG file = /home/jian/workdir/caldb/software/tools/caldb.config', 'CALDBCONFIG=(/home/jian/workdir/caldb/software/tools/caldb.config)', ' ...... environ-var/logical CALDBALIAS defined', ' ......... CALDBALIAS file = /home/jian/workdir/caldb/software/tools/alias_con', ' fig.fits', 'CALDBALIAS=(/home/jian/workdir/caldb/software/tools/alias_config.fits)', ' ...... CALDB is configured for the XTI instrument onboard NICER', ' ......... Cal Index File: https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xt', ' i/caldb.indx', 'CALDB_INDEX=(https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xti/caldb.indx)', ' ......... Instrument directory: https://heasarc.gsfc.nasa.gov/FTP/caldb/data/ni', ' cer/xti', 'CALDB_INST_DIR=(https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xti)', 'gtcalf 1.2.3 : Searching CALDB for quality values = 0', ' ... using cifsl3 1.0.1', ' ...... Searching for CAL_QUAL = 0', ' WARNING - cifsl3 1.0.1: Row 5 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 8 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 9 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 11 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 12 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 16 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 17 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 23 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 27 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 32 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 33 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 45 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 47 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 48 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 49 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 50 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 51 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 52 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 55 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 56 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 58 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 59 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 60 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 61 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 62 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 63 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 64 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 65 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 66 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 67 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 68 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 69 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 70 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 71 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 72 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 73 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 74 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 75 quality 0 matches quality = 0', ' WARNING - cifsl3 1.0.1: Row 76 quality 0 matches quality = 0', ' ...... Quality: Found 39 matches out of 76 rows', ' ...... Telescope: Found 39 matches out of 39 rows', ' ...... Instrument XTI matches instrument= XTI Row number 5', ' ...... Instrument XTI matches instrument= XTI Row number 8', ' ...... Instrument XTI matches instrument= XTI Row number 9', ' ...... Instrument XTI matches instrument= XTI Row number 11', ' ...... Instrument XTI matches instrument= XTI Row number 12', ' ...... Instrument XTI matches instrument= XTI Row number 16', ' ...... Instrument XTI matches instrument= XTI Row number 17', ' ...... Instrument XTI matches instrument= XTI Row number 23', ' ...... Instrument XTI matches instrument= XTI Row number 27', ' ...... Instrument XTI matches instrument= XTI Row number 32', ' ...... Instrument XTI matches instrument= XTI Row number 33', ' ...... Instrument XTI matches instrument= XTI Row number 45', ' ...... Instrument XTI matches instrument= XTI Row number 47', ' ...... Instrument XTI matches instrument= XTI Row number 48', ' ...... Instrument XTI matches instrument= XTI Row number 49', ' ...... Instrument XTI matches instrument= XTI Row number 50', ' ...... Instrument XTI matches instrument= XTI Row number 51', ' ...... Instrument XTI matches instrument= XTI Row number 52', ' ...... Instrument XTI matches instrument= XTI Row number 55', ' ...... Instrument XTI matches instrument= XTI Row number 56', ' ...... Instrument XTI matches instrument= XTI Row number 58', ' ...... Instrument XTI matches instrument= XTI Row number 59', ' ...... Instrument XTI matches instrument= XTI Row number 60', ' ...... Instrument XTI matches instrument= XTI Row number 61', ' ...... Instrument XTI matches instrument= XTI Row number 62', ' ...... Instrument XTI matches instrument= XTI Row number 63', ' ...... Instrument XTI matches instrument= XTI Row number 64', ' ...... Instrument XTI matches instrument= XTI Row number 65', ' ...... Instrument XTI matches instrument= XTI Row number 66', ' ...... Instrument XTI matches instrument= XTI Row number 67', ' ...... Instrument XTI matches instrument= XTI Row number 68', ' ...... Instrument XTI matches instrument= XTI Row number 69', ' ...... Instrument XTI matches instrument= XTI Row number 70', ' ...... Instrument XTI matches instrument= XTI Row number 71', ' ...... Instrument XTI matches instrument= XTI Row number 72', ' ...... Instrument XTI matches instrument= XTI Row number 73', ' ...... Instrument XTI matches instrument= XTI Row number 74', ' ...... Instrument XTI matches instrument= XTI Row number 75', ' ...... Instrument XTI matches instrument= XTI Row number 76', ' ...... Instrument: Found 39 matches out of 39 rows', ' ...... Detector: Found 39 matches out of 39 rows', ' ...... Filter: Found 39 matches out of 39 rows', ' ... CNAME MATRIX matches CNAME = MATRIX from CIF row number 59', ' ... CNAME MATRIX matches CNAME = MATRIX from CIF row number 62', ' ... CNAME MATRIX matches CNAME = MATRIX from CIF row number 75', ' ...... CAL_CNAME: Found 3 matches out of 39 rows', ' ...... Boundary: Found 3 matches out of 3 rows', ' ... Found 3 rows after final time selections', 'https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xti/bcf/rmfbase/nixtirmfbase20170601v001.fits 2', 'https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xti/cpf/rmf/nixtiref20170601v003.rmf 2', 'https://heasarc.gsfc.nasa.gov/FTP/caldb/data/nicer/xti/cpf/rmf/nixtidiagresp20170601v001.rmf 2', '']

[mfacorcoran]

I think you left out the call to hsp.quzcif in the above, but this looks like the output you'd received if you have chatter = 5, so I think this is what you expect.

[py3u]

Thank you, I would like to ask a question! I use Conda to automatically generate a base environment when creating an environment on my local computer. I want to change the default base environment to HEASoft's Python environment (similar to the settings on SciServer). What should I do? thank you!