A stalled connection to the embedding service blocks the whole indexing batch for request_timeout

[bakiburakogun]

Describe the Bug

NetworkEmbeddings._get_embedding() passes a single value as the timeout of the embedding request:

response = niquests.post(
    f'{emconf.base_url.removesuffix("/")}/embeddings',
    json=data,
    timeout=emconf.request_timeout,
    ...
)

With one value it applies to the read, and there is no separate bound on establishing the connection. When a connection to the embedding service stalls during the TLS/HTTP2 handshake — TCP connects, the handshake never completes — the worker sits in recv for the whole request_timeout, which defaults to 1800 s.

That is expensive because of how the work is batched: files_indexing_thread dispatches the batch across workers and then waits for every one of them (Waiting for file chunk 1/8 future to complete). One stalled handshake holds up the entire batch, and no further queue items are fetched until it returns.

Evidence

Captured with py-spy on a stalled worker:

Thread 1629 (idle): "MainThread"
    sync_recv_gro (urllib3/contrib/ssa/_gro.py:205)
    __exchange_until (urllib3/backend/hface.py:1018)
    _post_conn (urllib3/backend/hface.py:712)
    connect (urllib3/connection.py:894)
    _validate_conn (urllib3/connectionpool.py:820)
    _make_request (urllib3/connectionpool.py:1281)
    urlopen (urllib3/connectionpool.py:1821)
    send (niquests/adapters.py:927)
    request (niquests/api.py:122)
    post (niquests/api.py:401)
    _get_embedding (context_chat_backend/network_em.py:99)
    embed_documents (context_chat_backend/network_em.py:170)
    add_texts (langchain_postgres/vectorstores.py:885)
    add_indocuments (context_chat_backend/vectordb/pgvector.py:186)
    _process_sources (context_chat_backend/chain/ingest/injest.py:420)
    embed_sources (context_chat_backend/chain/ingest/injest.py:446)

It is inside connect(), in the handshake exchange, not in reading a response.

The signature in the logs is a batch whose workers all finish together at the timeout rather than at their own pace:

Dispatching 8 file chunk(s) and 0 provider chunk(s)
Waiting for file chunk 1/8 future to complete
Subprocess ... finished in 302262.22 ms
Subprocess ... finished in 302492.21 ms
Subprocess ... finished in 303569.71 ms
Subprocess ... finished in 305397.98 ms

Worth noting how well the service was otherwise: a direct request to the same endpoint from the same container returned in 0.28 s for one text and 4.37 s for 100 texts of 2000 characters, and 24 concurrent fresh connections all completed in 0.6 s. The stall is intermittent and only shows up under the sustained load of the real indexing run.

Impact

Measured on our deployment, before and after adding a connection timeout, everything else unchanged:

	before	after
worker duration	~295–305 s, all finishing together	1.4–85 s
batch interval	~5 min	~1.5 min
indexing rate	5.2 documents/min	60–65 documents/min

Roughly a twelvefold difference. Before the change the run looked stopped; operators reasonably read it as a hang rather than as slow progress.

Suggested fix

Give the connection its own bound, so a stalled handshake fails quickly and the retry that _get_embedding() already implements can land on a healthy connection:

timeout=(HTTP_CONNECT_TIMEOUT, emconf.request_timeout),

PR follows.

Setup Details

Nextcloud version: 34.0.3
context_chat / context_chat_backend: 5.4.0 / 5.4.1
Deployment: manual-install deploy daemon, four backend instances, external embedding service, PostgreSQL 18 with pgvector

[bakiburakogun]

Following up with something that changes the picture: the connection timeout alone does not fix this.

After running with timeout=(15, request_timeout) I still caught workers stuck for 28 minutes, in the same place:

sync_recv_gro (urllib3/contrib/ssa/_gro.py:205)
__exchange_until (urllib3/backend/hface.py:1018)
_post_conn (urllib3/backend/hface.py:712)
connect (urllib3/connection.py:894)
...
post → _get_embedding (network_em.py)

The stall sits in _post_conn, i.e. after the socket is connected, while the HTTP/2 handshake is being exchanged. The connect timeout has already been satisfied by then, and the read timeout has not started, so nothing bounds it. 15 s or 1800 s makes no difference — the request waits indefinitely.

Forcing HTTP/1.1 avoids that code path entirely, and with it the stall:

with niquests.Session(disable_http2=True) as session:
    response = session.post(...)

Measured across four backend instances, seven minutes each, everything else equal:

	before	after
nodes doing work	1 of 4 (three stalled)	4 of 4
documents indexed	56	258

Before the change three of the four instances were sitting in that handshake for tens of minutes at a time; the aggregate rate looked acceptable only because the fourth kept going.

I have not filed this as a separate issue because it is the same failure — I would rather let you decide the shape of the fix. The options I see:

• keep the connection timeout as a general safety net and additionally disable HTTP/2 for the embedding requests,
• or make the protocol configurable, since an embedding endpoint behind a proxy that mishandles HTTP/2 is not going to be unique to us.

Happy to extend #346 either way. The timeout in that PR is still worth having on its own — it bounds the ordinary connect path — but on its own it does not solve what the issue describes, and I did not want that to stand uncorrected.

[kyteinsky]

I'm not quite sure I understand the second issue, can you write it in your own words?
It seems the external embedding service is misbehaving?

[bakiburakogun]

In my own words: the embedding service is part of it, but it is not the whole story.

niquests.post() gives us two bounds, and neither one covers the window the workers are stuck in. The connect timeout covers the TCP connect, which succeeds. The read timeout starts once the request has been written. Between those two sits the protocol handshake, and that is where py-spy finds every stalled worker.

In the version we run (context_chat_backend:5.4.1, niquests 3.20.1, urllib3-future 2.22.901) that exchange is a while True loop in urllib3/backend/hface.py:

1013    next_timer = protocol.next_timer() if _has_next_timer else None
1014    sub = SubTimeout(sock, next_timer, send_pending)
1015    try:
1016        with sub:
1018            data_in = sync_recv_gro(sock, blocksize)
...
1049    if sub.timer_fired:
1050        continue

SubTimeout lowers the socket timeout to the protocol's own timer, and its docstring says what happens when that timer wins: the callback is invoked and the SocketTimeout exception is suppressed. The loop then goes around again. Nothing between the start of __exchange_until (line 934) and line 1060 records elapsed time, so there is no overall deadline on the loop, only on each individual recv.

That is consistent with what I measured: moving the connect timeout between 15 s and 1800 s made no difference to the stalls, because neither value bounds that window.

So the way I would put it is that the remote end triggers it, and the client turns it into a 28-minute stall rather than an error. Something on the path to our embedding service does not complete the HTTP/2 handshake reliably under sustained load, and there is nothing on our side to give up. Forcing HTTP/1.1 helps because it avoids that code path, not because it fixes the peer.

I checked the current release too (niquests 3.21.1, urllib3-future 2.24.905) and the loop is unchanged there, so this is not something an upgrade would resolve.

On whether the service is misbehaving: in isolation it looks healthy. A direct request returns in 0.28 s for one text and 4.37 s for 100 texts of 2000 characters, and 24 concurrent fresh connections all complete in 0.6 s. It only stalls under the real indexing load, and I have not been able to work out what is different about that traffic.

The two options I mentioned still stand and I am happy to take whichever you prefer:

• keep the connect timeout from fix(embeddings): bound the connection separately from the read #346 as a general safety net and additionally disable HTTP/2 for the embedding requests, or
• make the protocol configurable, since an endpoint behind a proxy that mishandles HTTP/2 is unlikely to be unique to us.

If you would rather see this tracked in urllib3-future instead, I can open it there, since the missing bound is in their code rather than here.