Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -428,7 +428,7 @@ iex> {:ok, channel} = GRPC.Stub.connect("localhost:50051",

When the connection drops, the adapter will attempt to reconnect up to `retry` times using **exponential backoff with jitter**. The delay starts at ~1 second and grows up to a maximum of 120 seconds. If all attempts are exhausted, the parent process receives a `{:elixir_grpc, :connection_down, pid}` message.

By default, `:retry` is `0` (no reconnection attempts).
By default, `:retry` is `0` (no reconnection attempts). It accepts any non-negative integer or `:infinity`.

> **Note:** Any in-flight requests at the time of the drop will fail immediately. Reconnection only re-establishes the transport connection — it does not replay requests.

Expand Down
7 changes: 7 additions & 0 deletions grpc/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,20 @@

## Unreleased

### Enhancements

* The Mint adapter's `:retry` option explicitly supports `:infinity`, reconnecting for as long as the connection process lives. This previously worked due Erlang term ordering (every atom sorts above every integer).
* `GRPC.Client.Adapter` gained an optional `validate_opts/1` callback. Adapters that implement it have their `:adapter_opts` validated in the caller by `GRPC.Client.Connection.connect/2`, so configuration errors raise there instead of crashing the spawned connection process.

### Behavior Changes

* The Mint adapter now enforces the requested `:timeout`/`:deadline` on unary receives. A unary call that never receives a response fails with `DEADLINE_EXCEEDED` after the documented 10s default instead of blocking indefinitely, and an explicit `:deadline` now takes precedence over `:timeout`.
* Test suites that define a Mox mock for the `GRPC.Client.Adapter` behaviour must stub the new validation callback. Mox generates optional callbacks on mocks, so `GRPC.Client.Connection` calls `validate_opts/1` on the mock and Mox raises `UnexpectedCallError` when it is not stubbed. Adapters that implement the behaviour with `@behaviour` and do not define `validate_opts/1` are unaffected.

### Bug Fixes

* The Gun adapter no longer shares a named channel's connection process across Erlang nodes. It was registered in `:global`, so a node connecting with a `:name` already used on another node adopted the remote connection process; `connect/2` returned `{:ok, channel}`, but every RPC on it then raised `ArgumentError` because `GRPC.Stub.call/5` calls `Process.alive?/1` on the connection pid and that raises for remote pids. Connection processes are now registered in the node-local `GRPC.Client.Registry`, so reuse is per node.
* Invalid Mint `:retry` values now raise `ArgumentError`. Because the old comparisons relied on term ordering, bad input failed silently and in two different directions: a negative integer behaved as no-retry, while *any* atom lead to infinite reconnection.

## v1.0.4 (2026-0-15)

Expand Down
2 changes: 1 addition & 1 deletion grpc/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ iex> {:ok, channel} = GRPC.Stub.connect("localhost:50051",

When the connection drops, the adapter will attempt to reconnect up to `retry` times using **exponential backoff with jitter**. The delay starts at ~1 second and grows up to a maximum of 120 seconds. If all attempts are exhausted, the parent process receives a `{:elixir_grpc, :connection_down, pid}` message.

By default, `:retry` is `0` (no reconnection attempts).
By default, `:retry` is `0` (no reconnection attempts). It accepts any non-negative integer or `:infinity`.

> **Note:** Any in-flight requests at the time of the drop will fail immediately. Reconnection only re-establishes the transport connection — it does not replay requests.

Expand Down
4 changes: 4 additions & 0 deletions grpc/lib/grpc/client/adapter.ex
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ defmodule GRPC.Client.Adapter do
@callback connect(channel :: struct(), opts :: keyword()) ::
{:ok, struct()} | {:error, any()}

@callback validate_opts(opts :: keyword()) :: :ok | {:error, String.t()}

@callback disconnect(channel :: struct()) :: {:ok, struct()} | {:error, any()}

@callback send_request(stream :: Stream.t(), contents :: iodata(), opts :: keyword()) ::
Expand Down Expand Up @@ -43,4 +45,6 @@ defmodule GRPC.Client.Adapter do
Cancel a stream in a streaming client.
"""
@callback cancel(stream :: Stream.t()) :: :ok | {:error, any()}

@optional_callbacks validate_opts: 1
end
58 changes: 37 additions & 21 deletions grpc/lib/grpc/client/adapters/mint.ex
Original file line number Diff line number Diff line change
Expand Up @@ -38,38 +38,26 @@ if Code.ensure_loaded?(Mint.HTTP) do
window size ensures that the number of packages exchanges is smaller, thus speeding up the requests by reducing the
amount of networks round trip, with the cost of having larger packages reaching the server per connection.
Check [Mint.HTTP2.setting() type](https://hexdocs.pm/mint/Mint.HTTP2.html#t:setting/0) for additional configs.
* `:retry`: Number of reconnection attempts when the connection drops. Defaults to `0` (no retries).
* `:retry`: Number of reconnection attempts when the connection drops, or `:infinity`
to keep reconnecting for as long as the process lives. Defaults to `0` (no retries).
Uses exponential backoff with jitter between attempts.
"""
@impl true
def connect(%{host: host, port: port} = channel, opts \\ []) do
{config_opts, opts} = Keyword.pop(opts, :config_options, [])
def connect(%Channel{} = channel, opts \\ []) do
{retry, opts} = Keyword.pop(opts, :retry, 0)
module_opts = Application.get_env(:grpc, __MODULE__, config_opts)

opts =
channel
|> connect_opts(opts)
|> merge_opts(module_opts)
|> Keyword.put(:retry, retry)

Process.flag(:trap_exit, true)

channel
|> mint_scheme()
|> ConnectionProcess.start_link(host, port, opts)
|> case do
{:ok, pid} ->
{:ok, %{channel | adapter_payload: %{conn_pid: pid}}}

error ->
{:error, "Error while opening connection: #{inspect(error)}"}
with :ok <- validate_retry(retry),
{:ok, pid} <- start_connection_process(channel, opts, retry) do
{:ok, %{channel | adapter_payload: %{conn_pid: pid}}}
end
catch
:exit, reason ->
{:error, "Error while opening connection: #{inspect(reason)}"}
end

@impl true
def validate_opts(opts), do: validate_retry(opts[:retry])

@impl true
def disconnect(%{adapter_payload: %{conn_pid: pid}} = channel)
when is_pid(pid) do
Expand Down Expand Up @@ -143,6 +131,13 @@ if Code.ensure_loaded?(Mint.HTTP) do
ConnectionProcess.cancel(conn_pid, request_ref)
end

defp validate_retry(nil), do: :ok
defp validate_retry(:infinity), do: :ok
defp validate_retry(retry) when is_integer(retry) and retry >= 0, do: :ok

defp validate_retry(retry),
do: {:error, ":retry must be a non-negative integer or :infinity, got: #{inspect(retry)}"}

defp connect_opts(%Channel{scheme: "https"} = channel, opts) do
%Credential{ssl: ssl} = Map.get(channel, :cred) || %Credential{}

Expand Down Expand Up @@ -177,6 +172,27 @@ if Code.ensure_loaded?(Mint.HTTP) do
defp mint_scheme(%Channel{scheme: "https"} = _channel), do: :https
defp mint_scheme(_channel), do: :http

defp start_connection_process(channel, opts, retry) do
{config_opts, opts} = Keyword.pop(opts, :config_options, [])
module_opts = Application.get_env(:grpc, __MODULE__, config_opts)

opts =
channel
|> connect_opts(opts)
|> merge_opts(module_opts)
|> Keyword.put(:retry, retry)

Process.flag(:trap_exit, true)

channel
|> mint_scheme()
|> ConnectionProcess.start_link(channel.host, channel.port, opts)
|> case do
{:ok, _} = ok -> ok
error -> {:error, "Error while opening connection: #{inspect(error)}"}
end
end

defp do_receive_data(%{payload: %{stream_response_pid: pid}}, request_type, opts)
when request_type in [:bidirectional_stream, :server_stream] do
produce_trailers? = opts[:return_headers] == true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ if Code.ensure_loaded?(Mint.HTTP) do

@impl true
def handle_info(:reconnect, state) do
attempt_reconnect(state)
maybe_attempt_reconnect(state)
end

def handle_info(message, state) do
Expand Down Expand Up @@ -444,23 +444,23 @@ if Code.ensure_loaded?(Mint.HTTP) do

clean_state = State.update_request_stream_queue(%{new_state | requests: %{}}, :queue.new())

if clean_state.retry > 0 do
attempt_reconnect(clean_state)
else
send(clean_state.parent, {:elixir_grpc, :connection_down, self()})
{:noreply, clean_state}
end
maybe_attempt_reconnect(clean_state)
end

defp end_stream_response(pid, error) do
StreamResponseProcess.consume(pid, :error, error)
StreamResponseProcess.done(pid)
end

defp attempt_reconnect(%{retry: max, retry_attempt: attempt} = state)
when attempt >= max do
defp maybe_attempt_reconnect(%{retry: 0} = state) do
send(state.parent, {:elixir_grpc, :connection_down, self()})
{:noreply, state}
end

defp maybe_attempt_reconnect(%{retry_attempt: attempt} = state)
when State.retries_exhausted?(state) do
Logger.warning(
"Connection retry exhausted (#{attempt}/#{max}) for #{state.scheme}://#{state.host}:#{state.port}"
"Connection retry exhausted (#{attempt}/#{state.retry}) for #{state.scheme}://#{state.host}:#{state.port}"
)

:telemetry.execute(
Expand All @@ -473,7 +473,7 @@ if Code.ensure_loaded?(Mint.HTTP) do
{:noreply, state}
end

defp attempt_reconnect(state) do
defp maybe_attempt_reconnect(state) do
next_attempt = state.retry_attempt + 1

Logger.info(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ if Code.ensure_loaded?(Mint.HTTP) do
}
end

defguard retries_exhausted?(state)
when state.retry != :infinity and state.retry_attempt >= state.retry

def update_conn(state, conn) do
%{state | conn: conn}
end
Expand Down
15 changes: 12 additions & 3 deletions grpc/lib/grpc/client/connection.ex
Original file line number Diff line number Diff line change
Expand Up @@ -1231,7 +1231,7 @@ defmodule GRPC.Client.Connection do
resolver = Keyword.get(opts, :resolver, GRPC.Client.Resolver)
adapter = Keyword.get(opts, :adapter, GRPC.Client.Adapters.Gun)

validate_adapter_opts!(opts[:adapter_opts])
validate_adapter_opts!(adapter, opts[:adapter_opts])

{norm_target, norm_opts, scheme} = normalize_target_and_opts(target, opts)
cred = resolve_credential(norm_opts[:cred], scheme)
Expand Down Expand Up @@ -1268,9 +1268,18 @@ defmodule GRPC.Client.Connection do
defp resolve_credential(nil, _scheme), do: nil
defp resolve_credential(other, _scheme), do: other

defp validate_adapter_opts!(opts) when is_list(opts), do: :ok
defp validate_adapter_opts!(adapter, opts) when is_list(opts) do
if Code.ensure_loaded?(adapter) and function_exported?(adapter, :validate_opts, 1) do
case adapter.validate_opts(opts) do
:ok -> :ok
{:error, message} -> raise ArgumentError, message
end
end

:ok
end

defp validate_adapter_opts!(_),
defp validate_adapter_opts!(_adapter, _),
do: raise(ArgumentError, ":adapter_opts must be a keyword list if present")

defp build_compressor_list(compressor, accepted) when is_list(accepted) do
Expand Down
20 changes: 20 additions & 0 deletions grpc/test/grpc/adapters/mint/connection_process_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,7 @@ defmodule GRPC.Client.Adapters.Mint.ConnectionProcessTest do

describe "handle_info - connection_closed - no requests" do
setup :valid_connection
setup :attach_reconnect_telemetry

test "send a message to parent process to inform the connection is down", %{
state: state
Expand All @@ -419,6 +420,7 @@ defmodule GRPC.Client.Adapters.Mint.ConnectionProcessTest do
assert new_state.conn.state == :closed
assert new_state.retry == 0
assert_receive {:elixir_grpc, :connection_down, _pid}, 500
refute_received {:telemetry, [:grpc, :client, :mint, :reconnect, :exhausted], _, _}
end
end

Expand Down Expand Up @@ -468,6 +470,22 @@ defmodule GRPC.Client.Adapters.Mint.ConnectionProcessTest do
end
end

describe "handle_info - connection_closed - retry: :infinity" do
setup :valid_connection_with_infinite_retry
setup :attach_reconnect_telemetry

test "reconnects when the connection drops", %{state: state} do
tcp_message = {:tcp_closed, state.conn.socket}

assert {:noreply, new_state} = ConnectionProcess.handle_info(tcp_message, state)
assert Mint.HTTP.open?(new_state.conn)
assert new_state.retry_attempt == 0

refute_received {:elixir_grpc, :connection_down, _pid}
refute_received {:telemetry, [:grpc, :client, :mint, :reconnect, :exhausted], _, _}
end
end

describe "handle_info - connection_closed - with request" do
setup :valid_connection
setup :valid_stream_request
Expand Down Expand Up @@ -833,6 +851,8 @@ defmodule GRPC.Client.Adapters.Mint.ConnectionProcessTest do

defp valid_connection_with_retry(ctx), do: valid_connection(ctx, retry: 3)

defp valid_connection_with_infinite_retry(ctx), do: valid_connection(ctx, retry: :infinity)

defp attach_reconnect_telemetry(_ctx) do
test_pid = self()
handler_id = "test-reconnect-telemetry-#{inspect(test_pid)}"
Expand Down
39 changes: 39 additions & 0 deletions grpc/test/grpc/adapters/mint_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -249,5 +249,44 @@ defmodule GRPC.Client.Adapters.MintTest do

assert state.retry == 0
end

test "accepts :infinity as the retry budget", %{port: port} do
channel = build(:channel, adapter: Mint, port: port, host: "localhost")

{:ok, connected} = Mint.connect(channel, retry: :infinity)
state = :sys.get_state(connected.adapter_payload.conn_pid)

assert state.retry == :infinity
end

test "rejects a retry budget that is neither a non-negative integer nor :infinity", %{
port: port
} do
channel = build(:channel, adapter: Mint, port: port, host: "localhost")

assert {:error, message} = Mint.connect(channel, retry: -1)
assert message =~ ":retry must be a non-negative integer or :infinity"

assert {:error, message} = Mint.connect(channel, retry: :forever)
assert message =~ ":retry must be a non-negative integer or :infinity"
end
end

describe "validate_opts/1" do
test "accepts an unset retry, a non-negative integer or :infinity" do
assert :ok == Mint.validate_opts([])
assert :ok == Mint.validate_opts(retry: nil)
assert :ok == Mint.validate_opts(retry: 0)
assert :ok == Mint.validate_opts(retry: 3)
assert :ok == Mint.validate_opts(retry: :infinity)
end

test "returns an error for anything else" do
assert {:error, message} = Mint.validate_opts(retry: -1)
assert message =~ ":retry must be a non-negative integer or :infinity"

assert {:error, message} = Mint.validate_opts(retry: :forever)
assert message =~ ":retry must be a non-negative integer or :infinity"
end
end
end
9 changes: 9 additions & 0 deletions grpc/test/grpc/client/connection_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,15 @@ defmodule GRPC.Client.ConnectionTest do
end
end

test "raises in the caller on invalid adapter-specific options" do
assert_raise ArgumentError, ~r/:retry must be a non-negative integer or :infinity/, fn ->
Connection.connect("ipv4:127.0.0.1:50051",
adapter: GRPC.Client.Adapters.Mint,
adapter_opts: [retry: -1]
)
end
end

test "interceptor init/1 runs once per connect", %{
ref: ref,
target: target,
Expand Down