Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Python Package Management with uv

Use uv exclusively for Python package management in this project.

## Package Management Commands

- All Python dependencies **must be installed, synchronized, and locked** using uv
- Never use pip, pip-tools, poetry, or conda directly for dependency management

Use these commands:

- Install dependencies: `uv add <package>`
- Remove dependencies: `uv remove <package>`
- Sync environment: `uv sync`
- Lock dependencies: `uv lock`

## Running Python Code

- Run a Python script with `uv run <script-name>.py`
- Run Python tools with `uv run <tool>` (e.g. `uv run pytest`, `uv run ruff`, `uv run mypy`, `uv run pre-commit`)
- Launch a Python REPL with `uv run python`


## Linting and formatting

This project uses Ruff for both linting and formatting. Do not call Black,
flake8, isort, or pylint.

- Lint: `uv run ruff check .`
- Lint and auto-fix: `uv run ruff check --fix .`
- Format: `uv run ruff format .`
- Check formatting without writing: `uv run ruff format --check .`
- Always invoke Ruff through `uv run` so it resolves to the project's
virtual environment.

Ruff configuration lives in `pyproject.toml` under `[tool.ruff]`. Do not
add a separate `ruff.toml` or `.ruff.toml`. Do not add inline `# noqa`
comments without a rule code.

## Create test, Update readme, and Git commit for each new feature
Create a unit tests. Add tests as new features are added. Organize the tests accordingly.
Update readme.md.
Create a detailed git commit after accomplishing a unique feature.

## Screenshot
Take a screenshot by running scripts/screenshot_gui.py whenever the GUI is changed.
Then use the new figure in readme.md

## gitignore
Do not track large files locally and in git history. Update .gitignore if needed.

## Session limit
When the prompt starts `session limit`, it means we hit a session limit and restarted.
The goal is marked with `goal`. Based on git status, the modified files are marked with
`file1`. Review these local files, check the current implementation, and tell me what the immediate next step should be.
202 changes: 202 additions & 0 deletions LIGHTKURVE_TGLC_BUG_REPORT.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
# OverflowError reading TGLC light curves with `quality_bitmask='hardest'` under NumPy 2

## Summary

Calling `LightCurve.read(path, format="tglc", quality_bitmask="hardest")` —
or the equivalent via `search.download(quality_bitmask="hardest")` — fails
with:

```
OverflowError: Python integer 65535 out of bounds for int16
```

which `lightkurve` surfaces to the user as:

```
LightkurveError: Error in reading Data product <path> of type TGLC.
This file may be corrupt due to an interrupted download. Please remove it
from your disk and try again.
```

The diagnostic is misleading: **the file is not corrupt** — `astropy.io.fits.verify('exception')` passes on it. The failure is in the TGLC reader's quality-mask step, and it is triggered purely by the choice of `quality_bitmask`. `"none"`, `"default"`, and `"hard"` work; `"hardest"` does not.

## Reproducer (synthetic, no network)

```python
import numpy as np
from lightkurve.utils import TessQualityFlags

# TGLC HLSP stores ``tess_flags`` as signed int16 (FITS format='I').
quality = np.zeros(100, dtype=np.int16)

TessQualityFlags.create_quality_mask(
quality_array=quality,
bitmask=TessQualityFlags.HARDEST_BITMASK, # 65535
)
# -> OverflowError: Python integer 65535 out of bounds for int16
```

## Reproducer (real data)

```python
import lightkurve as lk

search = lk.search_lightcurve("TIC 360906004", author="TGLC")
lc = search[0].download(quality_bitmask="hardest")
# -> LightkurveError: Error in reading Data product ... of type TGLC.
```

(Any TGLC HLSP file reproduces this; TIC 360906004 sector 11 is a small one.)

## Full traceback

```
Traceback (most recent call last):
File ".../lightkurve/io/read.py", line 134, in read
out = self.registry.read(cls, *args, **kwargs)
File ".../astropy/io/registry/core.py", line 221, in read
data = reader(*args, **kwargs)
File ".../lightkurve/io/tglc.py", line 55, in read_tglc_lightcurve
quality_mask = TessQualityFlags.create_quality_mask(
File ".../lightkurve/utils.py", line 114, in create_quality_mask
quality_mask = (quality_array & bitmask) == 0
OverflowError: Python integer 65535 out of bounds for int16

The above exception was the direct cause of the following exception:

LightkurveError: Error in reading Data product
<cache>/hlsp_tglc_tess_ffi_gaiaid-5842130724965127040-s0011-cam3-ccd1_tess_v1_llc.fits
of type TGLC.
This file may be corrupt due to an interrupted download. Please remove it
from your disk and try again.
```

## Root cause

The TGLC HLSP stores `tess_flags` as **signed 16-bit int** (FITS column
`format='I'`). `read_tglc_lightcurve` feeds that column straight into
`TessQualityFlags.create_quality_mask`, which performs:

```python
quality_mask = (quality_array & bitmask) == 0
```

When `bitmask` is a Python `int` larger than the int16 max (32767), NumPy
2.0+ refuses the bitwise AND and raises `OverflowError`. NumPy 1.x silently
truncated this. The relevant `TessQualityFlags` constants:

| Bitmask name | Value | Fits int16? |
|-------------------|-------|-------------|
| `DEFAULT_BITMASK` | 175 | yes |
| `HARD_BITMASK` | 24319 | yes |
| `HARDEST_BITMASK` | 65535 | **no** |

So the bug is exposed for any reader/caller that combines a TGLC quality
column with `quality_bitmask="hardest"` (or any integer > 32767).

Two notes:

* The reader's user-facing error blames a "corrupt download", which is
incorrect and sends users on a futile redownload loop. The cause is an
adapter-level dtype mismatch, not a transport issue.
* Other lightkurve readers that route int32 quality columns through the
same helper are unaffected — this is specifically the int16 column the
TGLC HLSP exposes.

## Proposed fix

Cast the quality column to int32 before the bitwise AND. One-line change
in `src/lightkurve/io/tglc.py`:

```diff
--- a/src/lightkurve/io/tglc.py
+++ b/src/lightkurve/io/tglc.py
@@
- quality_mask = TessQualityFlags.create_quality_mask(
- quality_array=lc["quality"], bitmask=quality_bitmask
- )
+ # TGLC stores ``tess_flags`` as signed int16. Cast to int32 before the
+ # bitwise AND in create_quality_mask so masks > 32767 (e.g. HARDEST =
+ # 65535) don't raise ``OverflowError`` under NumPy 2.
+ quality_mask = TessQualityFlags.create_quality_mask(
+ quality_array=np.asarray(lc["quality"], dtype=np.int32),
+ bitmask=quality_bitmask,
+ )
```

`numpy` is already imported in this module (`import numpy as np` at the top).
The cast is local to the mask computation; the stored `lc["quality"]` column
keeps its original dtype.

A slightly broader fix would normalize the dtype inside
`TessQualityFlags.create_quality_mask` itself (e.g. with
`np.asarray(quality_array, dtype=np.int32)`), which would also protect any
other reader that ever passes a sub-int32 quality column. Either fix
resolves the symptom; the reader-local cast is the minimal change.

## Suggested regression test

```python
def test_tglc_hardest_bitmask_no_int16_overflow():
"""TGLC reader must not overflow int16 when HARDEST_BITMASK is used.

The TGLC HLSP stores ``tess_flags`` as int16 (FITS format='I').
Under NumPy 2, ``int16_array & 65535`` raises OverflowError; the
reader must cast to a wider dtype before the bitwise AND.
"""
import lightkurve as lk

res = lk.search_lightcurve("TIC 360906004", author="TGLC")
assert len(res) > 0, "TIC 360906004 should have at least one TGLC product"
lc = res[0].download(quality_bitmask="hardest")
assert lc is not None
assert len(lc) > 0
```

A network-free variant using only `lightkurve.utils.TessQualityFlags`:

```python
def test_create_quality_mask_accepts_int16_quality_array():
import numpy as np
from lightkurve.utils import TessQualityFlags

quality = np.zeros(8, dtype=np.int16)
mask = TessQualityFlags.create_quality_mask(
quality_array=quality,
bitmask=TessQualityFlags.HARDEST_BITMASK,
)
assert mask.all()
```

## Environment

* Python 3.10
* lightkurve 2.6.0
* numpy 2.2.6
* astropy 6.1.7

The bug is **dtype-driven** (int16 column + Python int > 32767) and the
behavior change is **NumPy 2** (which rejects out-of-bounds Python ints in
bitwise ops with smaller-dtype arrays). It will reproduce on any combination
of lightkurve >= the TGLC reader's introduction with NumPy >= 2.0.

## Impact

Any pipeline that requests `quality_bitmask="hardest"` (or any custom
integer > 32767) when reading TGLC HLSP light curves fails immediately on
download/read. The misleading "file may be corrupt" message wastes user time
on redownload attempts that cannot succeed. Downstream tools that wrap
lightkurve (e.g. quicklook pipelines that use TGLC for FFI targets) inherit
the failure.

## Verified locally

Applied the one-line patch to a venv copy of
`lightkurve/io/tglc.py` and re-ran with `quality_bitmask="hardest"`:

```
sector: 11 cadences: 1180 finite cal_psf_flux: 1180/1180
```

i.e. the read now succeeds and produces the same light curve as
`quality_bitmask="default"`, with the user's selected mask applied.
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@ Although `quicklook` is optimized to find transiting exoplanets, it can also det

## Features

- **Multi-pipeline support** -- SPOC, TESS-SPOC, QLP, CDIPS, PATHOS, TGLC, TASOC
- **Multi-pipeline support** -- SPOC, TESS-SPOC, QLP, CDIPS, PATHOS, TGLC, TASOC, T16
- **Flux / light-curve type** -- PDCSAP or SAP for SPOC; aperture or PSF photometry for TGLC, with an automatic best-quality default
- **Automated detrending** -- biweight, cosine, median, GP, and other [wotan](https://github.com/hippke/wotan) methods
- **Stellar rotation** -- Generalized Lomb-Scargle (GLS) periodogram
- **Transit detection** -- Transit Least Squares (TLS) periodogram
- **Neighbor check** -- Gaia source overlay on archival sky images
- **Neighbor check** -- Gaia source overlay on cached archival sky images, with optional nearby SIMBAD object labels
- **Batch processing** -- `--each-sector` mode, GNU parallel support, and candidate ranking tools
- **Web GUI** -- Flask-based interface with live progress, job queue, and gallery
- **HDF5 output** -- full TLS results saved for downstream filtering
Expand Down Expand Up @@ -70,6 +70,12 @@ ql --name TOI-125.01 --each-sector -save

# Run all sectors with 4 parallel workers
ql --name TOI-125.01 --each-sector -j 4 -save

# Run every available pipeline on the latest sector
ql --name TOI-125.01 --each-pipeline -save

# TGLC PSF photometry with nearby SIMBAD objects overlaid
ql --name TOI-125.01 --pipeline tglc --fluxtype psf -show_simbad -save
```

### Python API
Expand Down Expand Up @@ -126,7 +132,7 @@ The 9-panel figure shows:
| 3 | Phase-folded light curve at rotation period |
| 4 | Flattened light curve + detected transits |
| 5 | TLS periodogram (orbital period) |
| 6 | TESS aperture + Gaia sources on archival image |
| 6 | TESS aperture + Gaia sources (and optional SIMBAD objects) on archival image |
| 7 | Phase-folded transit (odd/even comparison) |
| 8 | Secondary eclipse check at phase 0.5 |
| 9 | Summary of stellar and companion parameters |
Expand Down
16 changes: 13 additions & 3 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ ql --name TARGET [options]
| Flag | Default | Description |
|------|---------|-------------|
| `--sector SECTOR` | `-1` (latest) | TESS sector number |
| `--pipeline PIPELINE` | `spoc` | Light curve pipeline: `spoc`, `tess-spoc`, `qlp`, `cdips`, `pathos`, `tglc`, `tasoc` |
| `--fluxtype TYPE` | `pdcsap` | Flux column: `pdcsap` or `sap` |
| `--pipeline PIPELINE` | `spoc` | Light curve pipeline: `spoc`, `tess-spoc`, `qlp`, `cdips`, `pathos`, `tglc`, `tasoc`, `t16` |
| `--fluxtype TYPE` | `pdcsap` | Light-curve type. SPOC: `pdcsap` or `sap`. TGLC: `aperture`, `psf`, or `auto` (best-quality default) |
| `--exptime SECONDS` | auto | Exposure time in seconds |
| `--quality_bitmask MASK` | `default` | Quality mask: `none`, `default`, `hard`, `hardest` |

Expand Down Expand Up @@ -55,16 +55,20 @@ ql --name TARGET [options]
| `--outdir DIR` | `.` | Output directory |
| `--suffix TEXT` | none | Suffix appended to output filenames |
| `--survey NAME` | `dss1` | Archival image survey for overlay |
| `-show_simbad` | off | Overplot nearby non-stellar SIMBAD objects on the archival image |
| `-save` | off | Save figure (.png) and TLS results (.h5) |
| `-verbose` | off | Print detailed progress |
| `-overwrite` | off | Overwrite existing output files |

### Multi-sector mode
### Multi-sector / multi-pipeline mode

| Flag | Default | Description |
|------|---------|-------------|
| `--each-sector` | off | Run on every available sector individually |
| `--each-pipeline` | off | Run on every available pipeline for the latest sector (mutually exclusive with `--each-sector`) |
| `-j, --jobs N` | `1` | Number of parallel jobs for `--each-sector` |
| `--nice INC` | unchanged | Lower CPU priority by this increment (POSIX; e.g. `19` = lowest) |
| `--cores N` | auto | CPU cores used by TLS per run (default: `cpu_count // 2` single run, `cpu_count // jobs` for `--each-sector`) |

### Examples

Expand All @@ -87,6 +91,12 @@ ql --name TOI-125.01 --each-sector -save
# Run all sectors with 4 parallel workers
ql --name TOI-125.01 --each-sector -j 4 -save

# Run every available pipeline on the latest sector
ql --name TOI-125.01 --each-pipeline -save

# TGLC PSF photometry with nearby SIMBAD objects overlaid
ql --name TOI-125.01 --pipeline tglc --fluxtype psf -show_simbad -save

# Pipe output to a log file
ql --name TIC-52368076 -verbose -save | tee output.log
```
Expand Down
9 changes: 6 additions & 3 deletions docs/gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@ This starts a local server at [http://127.0.0.1:5000](http://127.0.0.1:5000).

1. Enter a target name (TOI, TIC, or common name)
2. Adjust pipeline parameters (sector, pipeline, flux type, etc.)
3. Click **Run QuickLook**
4. Watch live progress with step-by-step updates and ETA
3. Optionally toggle **Show SIMBAD** to overlay nearby non-stellar SIMBAD objects on the archival image (on by default in the GUI)
4. Click **Run QuickLook**
5. Watch live progress with step-by-step updates and ETA

### Each-sector mode

Expand Down Expand Up @@ -51,7 +52,9 @@ Completed jobs display the output figure inline. Click to view full size. Result

### Gallery

The `/gallery` page shows all previously generated output figures with search, sorting, and pagination.
The `/gallery` page shows all previously generated output figures with search, sorting, and pagination. Click a thumbnail to open it in a zoomable modal -- pan by dragging, and use the **left/right arrow keys** to step between figures without closing the viewer.

![QuickLook Gallery](img/ql-gui-gallery.png)

### Cancelling jobs

Expand Down
Binary file added docs/img/ql-gui-gallery.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/img/ql-gui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 3 additions & 1 deletion docs/pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ The [Transit Least Squares](https://ui.adsabs.harvard.edu/abs/2019A%26A...623A..

**Panel 6: Archival image + Gaia overlay**

The TESS aperture (blue polygon) overlaid on an archival sky survey image (DSS by default). Orange and red circles show nearby Gaia sources, scaled by brightness. This panel helps identify:
The TESS aperture (blue polygon) overlaid on an archival sky survey image (DSS by default). Orange and red circles show nearby Gaia sources, scaled by brightness. With `-show_simbad` (CLI) or the GUI checkbox, nearby non-stellar SIMBAD objects within the field of view are also labelled by their condensed object type (e.g. `EclBin`), which helps flag known eclipsing binaries or other variables near the target. DSS cutouts are cached under `~/.astropy/cache` so repeated runs of the same field do not re-download the image. This panel helps identify:

- Blended neighbors that could be the true source of the signal
- Background eclipsing binaries contaminating the aperture
Expand Down Expand Up @@ -116,4 +116,6 @@ When `-save` is used, QuickLook produces:

Example: `WASP-21_s56_pdcsap_sc.png` and `WASP-21_s56_pdcsap_sc_tls.h5`

For TGLC runs, the flux token encodes the photometry method: `--fluxtype aperture` writes `tglc_aper` and `--fluxtype psf` writes `tglc_psf` (e.g. `TIC123_s11_tglc_psf_lc.png`). This keeps aperture and PSF runs of the same target/sector from overwriting each other on disk. SPOC stems and default (auto) TGLC stems are unchanged.

The HDF5 file contains the TLS periodogram, best-fit parameters, stellar parameters, and metadata. Use `read_tls` to extract a summary CSV from a directory of these files.
Loading
Loading