Skip to content

Add keep alive file to datasets pod #66

@jgriss

Description

@jgriss

In certain cases, the datasets pod does not exist as expected. Add a keep-alive file as for the worker pod.

Example error

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_datasets/reactome_analysis_dataset_fetcher.py", line 188, in _on_new_request
    (data, summary) = dataset_fetcher.load_dataset(request.parameters, self._get_mq())
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_datasets/dataset_fetchers/expression_atlas_fetcher.py", line 116, in load_dataset
    loaded_data = self.load_r_data(download_files, reactome_mq)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_datasets/dataset_fetchers/expression_atlas_fetcher.py", line 392, in load_r_data
    reactome_mq.sleep(1)
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_utils/reactome_mq.py", line 164, in sleep
    self._connect().sleep(duration)
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 850, in sleep
    self.process_data_events(time_limit)
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 827, in process_data_events
    self._flush_output(timer.is_ready, common_terminator)
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 523, in _flush_output
    raise self._closed_result.value.error
pika.exceptions.StreamLostError: Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_datasets/__main__.py", line 38, in main
    dataset_worker.start_listening()
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_datasets/reactome_analysis_dataset_fetcher.py", line 76, in start_listening
    mq.process_analysis(self._on_new_request)
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_utils/reactome_mq.py", line 199, in process_analysis
    self._channel.start_consuming()
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 1865, in start_consuming
    self._process_data_events(time_limit=None)
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 2026, in _process_data_events
    self.connection.process_data_events(time_limit=time_limit)
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 833, in process_data_events
    self._dispatch_channel_events()
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 567, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 1492, in _dispatch_events
    consumer_info.on_message_callback(self, evt.method,
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_datasets/reactome_analysis_dataset_fetcher.py", line 210, in _on_new_request
    self._acknowledge_message(ch, method)
  File "/usr/local/lib/python3.11/dist-packages/reactome_analysis_datasets/reactome_analysis_dataset_fetcher.py", line 95, in _acknowledge_message
    channel.basic_ack(delivery_tag=method.delivery_tag)
  File "/usr/lib/python3/dist-packages/pika/adapters/blocking_connection.py", line 2111, in basic_ack
    self._impl.basic_ack(delivery_tag=delivery_tag, multiple=multiple)
  File "/usr/lib/python3/dist-packages/pika/channel.py", line 203, in basic_ack
    self._raise_if_not_open()
  File "/usr/lib/python3/dist-packages/pika/channel.py", line 1393, in _raise_if_not_open
    raise exceptions.ChannelWrongStateError('Channel is closed.')
pika.exceptions.ChannelWrongStateError: Channel is closed.
[2024-07-14 17:19:52,679] DEBUG [1] [reactome_analysis_datasets.reactome_analysis_dataset_fetcher.shutdown:63] Shutting down gracefully.
[2024-07-14 17:19:52,679] DEBUG [1] [reactome_analysis_datasets.reactome_analysis_dataset_fetcher.shutdown:65] Closing MQ connection.
[2024-07-14 17:19:52,679] INFO [1] [__main__.main:45] Exiting.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions