Skip to content

Fix bug parsing raw Fitbit sleep intraday JSON data#247

Merged
jenniferfedor merged 1 commit into
developfrom
bugfix/sleep-intraday-shortdata
Mar 26, 2026
Merged

Fix bug parsing raw Fitbit sleep intraday JSON data#247
jenniferfedor merged 1 commit into
developfrom
bugfix/sleep-intraday-shortdata

Conversation

@jenniferfedor
Copy link
Copy Markdown
Collaborator

When a raw JSON Fitbit sleep record has type stages, we expect the intraday levels element in the sleep record to contain two groupings of data: data and shortData. Per Fitbit's documentation, the data grouping contains the sleep stages (rem, light, deep) and any wake periods that were > 3 minutes in duration. The shortData grouping contains wake periods that were <= 3 minutes in duration. We merge the sleep stage labels contained in these data and shortData groupings in the mutation script that handles parsing raw Fitbit sleep intraday JSON data.


However, we've encountered a handful of sleep records with type stages that inexplicably do not contain the expected shortData grouping. This does not seem to be related to whether or not the record represents the main sleep episode. When the shortData element is missing, the following error occurs when applying the mutation script:

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  KeyError: 'shortData'

Detailed traceback: 
  File "/rapids/src/data/streams/mutations/fitbit/parse_sleep_intraday_json.py", line 139, in main
    parsed_data = parseSleepData(json_raw)
  File "/rapids/src/data/streams/mutations/fitbit/parse_sleep_intraday_json.py", line 128, in parseSleepData
    records_intraday = parseOneRecordForV12(record, device_id, d_is_main_sleep, records_intraday, type_episode_id)
  File "/rapids/src/data/streams/mutations/fitbit/parse_sleep_intraday_json.py", line 97, in parseOneRecordForV12
    for data in mergeLongAndShortData(record["levels"]):
  File "/rapids/src/data/streams/mutations/fitbit/parse_sleep_intraday_json.py", line 35, in mergeLongAndShortData
    for data in data_intraday["shortData"]:


To fix this, we now first check if the shortData element is present in the levels data for the record. If so, we resample the shortData and merge it with the resampled "long" data. Otherwise, we only use the resampled "long" data. We also make some modifications to this script to improve efficiency (e.g., appending data to a list and then converting the complete list to a DataFrame rather than appending rows of data directly to a DataFrame) and readability.

@jenniferfedor jenniferfedor merged commit 8ed03c1 into develop Mar 26, 2026
2 checks passed
@jenniferfedor jenniferfedor deleted the bugfix/sleep-intraday-shortdata branch March 26, 2026 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant