fix(server): fix database locked when marking deployments ready by uinstinct · Pull Request #22967 · PrefectHQ/prefect

uinstinct · 2026-08-28T03:40:55Z

Fix the repeated database locked error when workers poll. This was happening only in sqlite and not postgres.

2 transactions mark_work_queues_ready and mark_deployments_rady were reading and writing concurrently. Hence, added with_for_update to ask for the write lock at start.

related to #19771

no ai tool used.

the error from terminal output:

Running: /Users/instinct/Desktop/working/prefect-apps/prefect/.venv/bin/python -m uvicorn --factory 
prefect.server.api.server:create_app --host 127.0.0.1 --port 4200 --log-level debug
09:10:31.002 | ERROR   | docket.dependencies - ↩ [     4ms] mark_deployments_ready(work_queue_ids=...){mark_deployments_ready:work_pool:01f5fa7c-7fdc-42d8-a273-1a993c1c298c}
Traceback (most recent call last):
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        cursor, str_statement, effective_parameters, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 180, in execute
    self._adapt_connection._handle_exception(error)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 340, in _handle_exception
    raise error
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 162, in execute
    self.await_(_cursor.execute(operation, parameters))
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
    return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/aiosqlite/cursor.py", line 40, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/aiosqlite/cursor.py", line 32, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/aiosqlite/core.py", line 160, in _execute
    return await future
           ^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/aiosqlite/core.py", line 63, in _connection_worker_thread
    result = function()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/docket/worker.py", line 1111, in _execute
    result = await execution.function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
        *execution.args, **final_kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/src/prefect/server/models/deployments.py", line 1330, in mark_deployments_ready
    await session.execute(
    ...<7 lines>...
    )
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
    result = await greenlet_spawn(
             ^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 201, in greenlet_spawn
    result = context.throw(*sys.exc_info())
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
           ~~~~~~~~~~~~~~~~~~~~~~^
        statement,
        ^^^^^^^^^^
    ...<4 lines>...
        _add_event=_add_event,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self,
        ^^^^^
    ...<4 lines>...
        conn,
        ^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1660, in orm_execute_statement
    return super().orm_execute_statement(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        session,
        ^^^^^^^^
    ...<4 lines>...
        conn,
        ^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
        statement, params or {}, execution_options=execution_options
    )
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
        self,
        distilled_parameters,
        execution_options or NO_OPTIONS,
    )
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
    return connection._execute_clauseelement(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self, distilled_params, execution_options
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
        dialect,
    ...<8 lines>...
        cache_hit=cache_hit,
    )
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ~~~~~~~~~~~~~~~~~~~~~~~~~^
        dialect, context, statement, parameters
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        e, str_statement, effective_parameters, cursor, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        cursor, str_statement, effective_parameters, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 180, in execute
    self._adapt_connection._handle_exception(error)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 340, in _handle_exception
    raise error
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 162, in execute
    self.await_(_cursor.execute(operation, parameters))
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
    return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/aiosqlite/cursor.py", line 40, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/aiosqlite/cursor.py", line 32, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/aiosqlite/core.py", line 160, in _execute
    return await future
           ^^^^^^^^^^^^
  File "/Users/instinct/Desktop/working/prefect-apps/prefect/.venv/lib/python3.13/site-packages/aiosqlite/core.py", line 63, in _connection_worker_thread
    result = function()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: WITH locked AS 
(SELECT deployment.id AS id, deployment.status AS status 
FROM deployment 
WHERE deployment.id IN (SELECT 1 FROM (SELECT 1) WHERE 1!=1) OR deployment.work_queue_id IN (:work_queue_id_1_1) ORDER BY deployment.id)
 UPDATE deployment SET last_polled=:last_polled, status=:status, updated=deployment.updated WHERE deployment.id IN (SELECT locked.id 
FROM locked) RETURNING id]
[parameters: {'last_polled': '2026-08-28 03:40:31.000221', 'status': 'READY', 'work_queue_id_1_1': '83341ed5-d7f6-44a7-bf0e-afa60f279066'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

Checklist

This pull request references any related issue by including "closes <link to issue>"
- If no issue exists and your change is not a small fix, please create an issue first.
If this is a complex change, a maintainer has confirmed the proposed approach on the linked issue.
If this pull request adds or changes functionality, it includes tests or explains why tests are not needed.
If this pull request changes user-facing behavior, it updates documentation or explains why documentation is not needed.
If this pull request removes docs files, it includes redirect settings in mint.json.
If this pull request adds functions or classes, it includes helpful docstrings.

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

codspeed-hq · 2026-08-28T03:55:21Z

Merging this PR will not alter performance

✅ 8 untouched benchmarks
⏩ 1 skipped benchmark¹

_{Comparing uinstinct:sqlite-busy-snapshot (d8ca691) with main (97d0101)}

1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

desertaxle

Thanks for the PR @uinstinct! Can you add some regression tests covering these changes?

fix(server): fix database locked when marking deployments ready

4a52212

uinstinct requested review from chrisguidry, desertaxle and zzstoatzz as code owners August 28, 2026 03:40

devin-ai-integration Bot reviewed Aug 28, 2026

View reviewed changes

desertaxle requested changes Sep 1, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

add tests

ca0b62e

uinstinct requested a review from desertaxle September 1, 2026 15:27

uinstinct force-pushed the sqlite-busy-snapshot branch from 4e96f65 to ca0b62e Compare September 1, 2026 15:28

timing safe sqlite lock regressions

d8ca691

Navigation Menu

Uh oh!

fix(server): fix database locked when marking deployments ready - #22967

fix(server): fix database locked when marking deployments ready#22967
uinstinct wants to merge 3 commits into
PrefectHQ:mainfrom
uinstinct:sqlite-busy-snapshot

uinstinct commented Aug 28, 2026 •

edited by desertaxle

Loading

Uh oh!

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

codspeed-hq Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

desertaxle left a comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

uinstinct commented Aug 28, 2026 • edited by desertaxle Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

codspeed-hq Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

desertaxle left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

uinstinct commented Aug 28, 2026 •

edited by desertaxle

Loading

devin-ai-integration Bot left a comment •

edited

Loading

codspeed-hq Bot commented Aug 28, 2026 •

edited

Loading