Skip to content

TypeError: cannot pickle '_thread.lock' object #20

Description

@vcbd

Hello,
The celery cluster main command seems to crash just after scenes detection on Debian 13 (I checked), fresh pipx reinstall of alabamaEncoder. The command output is:

Input Video: 1440x1080 @ 23.98 fps, YUV420P10LE, SDR, 733.58 Kb/s, HEVC
Using celery
Got lan ip: xx.xx.xx.xx
Number of available workers: 1
Using dynamic vmaf targeting
Input file is not HDR, disabling HDR mode
Checking files: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 550/550 [00:00<00:00, 233323.27file/s]
Starting encoding of 550 out of 550 scenes
Encoding:   0%|                                                                                                         | 0.00/38.0k [00:00<?, ?frame/s]Not starting constant updates,  false
Traceback (most recent call last):
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/serialization.py", line 41, in _reraise_errors
    yield
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/serialization.py", line 220, in dumps
    payload = encoder(data)
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/serialization.py", line 347, in pickle_dumps
    return dumper(obj, protocol=pickle_protocol)
TypeError: cannot pickle '_thread.lock' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xxx/.local/bin/alabamaEncoder", line 7, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/alabamaEncode/cli/__main__.py", line 159, in main
    asyncio.run(job.run_pipeline())  # this runs the whole encoding process
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/alabamaEncode/core/job.py", line 251, in run_pipeline
    await asyncio.create_task(
    ...<13 lines>...
    )
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/alabamaEncode/parallel_execution/execute_commands.py", line 57, in execute_commands
    run_command_on_celery.delay(command) for command in command_objects
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/celery/app/task.py", line 444, in delay
    return self.apply_async(args, kwargs)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/celery/app/task.py", line 601, in apply_async
    return app.send_task(
           ~~~~~~~~~~~~~^
        self.name, args, kwargs, task_id=task_id, producer=producer,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        **options
        ^^^^^^^^^
    )
    ^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/celery/app/base.py", line 930, in send_task
    amqp.send_task_message(P, name, message, **options)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/celery/app/amqp.py", line 523, in send_task_message
    ret = producer.publish(
        body,
    ...<8 lines>...
        **properties
    )
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/messaging.py", line 178, in publish
    body, content_type, content_encoding = self._prepare(
                                           ~~~~~~~~~~~~~^
        body, serializer, content_type, content_encoding,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        compression, headers)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/messaging.py", line 280, in _prepare
    body) = dumps(body, serializer=serializer)
            ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/serialization.py", line 219, in dumps
    with _reraise_errors(EncodeError):
         ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/serialization.py", line 45, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/exceptions.py", line 34, in reraise
    raise value.with_traceback(tb)
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/serialization.py", line 41, in _reraise_errors
    yield
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/serialization.py", line 220, in dumps
    payload = encoder(data)
  File "/home/xxx/.local/pipx/venvs/alabamaencoder/lib/python3.13/site-packages/kombu/serialization.py", line 347, in pickle_dumps
    return dumper(obj, protocol=pickle_protocol)
kombu.exceptions.EncodeError: cannot pickle '_thread.lock' object
Current Session Runtime: 1 second(s), Runtime From Previous Sessions: 185 second(s), Total Runtime: 186 second(s)
Encoding:   0%|          | 0.00/38.0k [00:00<?, ?frame/s]

What am I doing wrong?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions