GRPC.Client.Connection never prunes a dead channel — permanent for non-DNS targets

[pfitz]

Describe the bug

When an adapter connection process dies, GRPC.Client.Connection never prunes the corresponding channel from the load balancer. pick_channel/2 keeps handing out a %GRPC.Channel{} whose adapter_payload.conn_pid is a dead pid.

How long that lasts depends entirely on the resolver:

Target scheme	Recovery
dns://…	Bounded by resolve_interval (default 30s)
ipv4: / ipv6: / unix:	Never — the channel stays dead for the lifetime of the connection

The second row is the surprising one, and I think it's the actual bug: for literal-address targets there is no background re-resolution at all, so nothing ever runs the code path that would notice.

Mechanism

Line references are grpc 1.0.3, lib/grpc/client/connection.ex.

Adapter connection processes are linked to the Connection GenServer — stated in the comment at :555. When one dies, the GenServer receives {:EXIT, pid, reason} and falls through to the catch-all at :584:

def handle_info({:EXIT, pid, reason}, state) do
  Logger.warning(
    "#{inspect(__MODULE__)} received :EXIT from #{inspect(pid)} reason: #{inspect(reason)}"
  )

  {:noreply, state}
end

It logs and returns. It does not prune the channel, does not call lb_mod.update/2, and does not mark anything unhealthy.

lb_mod.update/2 has exactly one call site — reconcile_lb/3 at :947 — reached only from rebalance_after_reconcile/2 (:926), reached only from handle_resolve_result/2 (:868), i.e. only on a {:resolver_update, result} message. Liveness is consulted there, via channel_alive?/1 (:908, :954). So the library knows how to detect this; the detection just never runs on the death itself.

That leaves re-resolution as the only recovery path — and only GRPC.Client.Resolver.DNS implements the optional init/2 that starts the periodic timer (dns_resolver.ex:124). IPv4, IPv6 and Unix don't, so GRPC.Client.Resolver.init/2 returns {:ok, nil} and there is no tick to recover on.

This compounds with GRPC.Client.LoadBalancing.PickFirst, whose pick/1 returns the stored channel unconditionally with no liveness check — so with a dead channel at element 0, 100% of picks return it.

The underlying gap is that the GRPC.Client.LoadBalancing behaviour has no failure-feedback callback:

@callback init(opts :: keyword()) :: {:ok, state} | {:error, reason}
@callback pick(state :: any()) :: {:ok, struct(), new_state} | {:error, reason}
@callback update(state :: any(), new_channels :: [struct()]) :: {:ok, new_state} | {:error, reason}

pick/1 receives only the balancer's own state, so no policy — including a custom one — can learn that its previous pick failed. grpc-go and grpc-java expose subchannel connectivity state to the policy for exactly this.

To Reproduce

Self-contained script (needs only a loopback HTTP/2 server):

Mix.install([{:grpc, "1.0.3"}, {:mint, "~> 1.9"}, {:bandit, "~> 1.0"}])

defmodule EchoPlug do
  @behaviour Plug
  @impl true
  def init(opts), do: opts
  @impl true
  def call(conn, _opts), do: Plug.Conn.send_resp(conn, 200, "ok")
end

defmodule Repro do
  alias GRPC.Client.Connection

  @port 50_555
  @resolve_interval 30_000

  def run(target) do
    {:ok, _} = Bandit.start_link(plug: EchoPlug, scheme: :http, port: @port)

    {:ok, _} =
      Connection.start_link(
        name: :repro,
        target: target,
        adapter: GRPC.Client.Adapters.Mint
      )

    :ok = await_ready()
    {:ok, ch} = Connection.get_channel(:repro)

    pid = pick_pid(ch)
    IO.puts("1. picked        conn_pid=#{inspect(pid)} alive?=#{Process.alive?(pid)}")

    ref = Process.monitor(pid)
    Process.exit(pid, :kill)
    receive do: ({:DOWN, ^ref, _, _, _} -> :ok)
    Process.sleep(200)
    IO.puts("2. killed it.    alive?=#{Process.alive?(pid)}")

    pid2 = pick_pid(ch)
    IO.puts("3. picked again  conn_pid=#{inspect(pid2)} alive?=#{Process.alive?(pid2)} (same? #{pid2 == pid})")

    IO.puts("4. waiting #{div(@resolve_interval, 1000) + 5}s for a resolver tick...")
    Process.sleep(@resolve_interval + 5_000)

    pid3 = pick_pid(ch)
    IO.puts("5. after tick    conn_pid=#{inspect(pid3)} alive?=#{Process.alive?(pid3)} (still dead pid? #{pid3 == pid})")
  end

  defp pick_pid(ch) do
    {:ok, picked} = Connection.pick_channel(ch)
    picked.adapter_payload.conn_pid
  end

  defp await_ready(tries \\ 50)
  defp await_ready(0), do: {:error, :timeout}

  defp await_ready(tries) do
    with {:ok, ch} <- Connection.get_channel(:repro),
         {:ok, _} <- Connection.pick_channel(ch) do
      :ok
    else
      _ ->
        Process.sleep(100)
        await_ready(tries - 1)
    end
  end
end

Repro.run(System.argv() |> List.first() || "ipv4:127.0.0.1:50555")

Killing the adapter process stands in for a peer reset or network drop; the point is that Connection reacts identically however the process died.

Observed

elixir repro.exs "ipv4:127.0.0.1:50555" — literal address, never recovers:

07:40:09.803 [info] Running EchoPlug with Bandit 1.12.4 at 0.0.0.0:50555 (http)
1. picked        conn_pid=#PID<0.546.0> alive?=true

07:40:09.911 [warning] GRPC.Client.Connection received :EXIT from #PID<0.546.0> reason: :killed
2. killed it.    alive?=false
3. picked again  conn_pid=#PID<0.546.0> alive?=false (same? true)
4. waiting 35s for a resolver tick...
5. after tick    conn_pid=#PID<0.546.0> alive?=false (still dead pid? true)

elixir repro.exs "dns://localtest.me:50555" — same until the resolver tick, which recovers it:

07:40:50.965 [info] Running EchoPlug with Bandit 1.12.4 at 0.0.0.0:50555 (http)
1. picked        conn_pid=#PID<0.546.0> alive?=true

07:40:51.577 [warning] GRPC.Client.Connection received :EXIT from #PID<0.546.0> reason: :killed
2. killed it.    alive?=false
3. picked again  conn_pid=#PID<0.546.0> alive?=false (same? true)
4. waiting 35s for a resolver tick...
5. after tick    conn_pid=#PID<0.551.0> alive?=true (still dead pid? false)

Note the [warning] line: that is the :584 handler, and it is the only thing the library does in response to the death.

(localtest.me publicly resolves to 127.0.0.1. dns://localhost does not work here — the DNS resolver issues real DNS queries and localhost typically only exists in /etc/hosts.)

Expected behavior

A dead channel should stop being picked, without waiting for a resolver tick — and regardless of whether the target's resolver has one at all.

Possible directions (deferring to your design preference)

1. Prune on death: in the {:EXIT, …} handler, drop the matching channel from real_channels and call lb_mod.update/2. Smallest change, fixes the permanent case.
2. Have pick/1 skip channels failing channel_alive?/1, so a stale entry can't be handed out even between reconciles.
3. Longer term, add a failure-feedback callback to GRPC.Client.LoadBalancing so policies can react to failures — this is what would let a custom policy implement health-aware behaviour.

I've opened a PR implementing (1), since it is self-contained and fixes the never-recovers case. It also restarts the establish loop when the pruned channel was the last connected one, otherwise pruning would just replace "serves a dead channel" with "serves nothing" for literal-address targets. Happy to rework it if you'd prefer a different shape — (3) in particular is a design call that's yours, not mine.

Relation to #563

#563 reports a FunctionClauseError on GOAWAY that kills ConnectionProcess, and notes the same downstream symptom ("keeps serving a channel whose conn_pid is dead"). These look complementary rather than duplicate: #563 is one cause of a dead conn_pid, whereas this issue is that GRPC.Client.Connection doesn't recover from a dead conn_pid however it arose. Fixing #563 removes one source; the pruning gap would remain for ordinary transport failures.

Also possibly relevant to #464, where the periodic refresh was described as the mechanism that keeps load-balancing decisions current — that refresh only exists for dns:// targets.

Versions:

• grpc 1.0.3, mint 1.9, Elixir 1.20.2, Erlang/OTP 29
• Adapter: GRPC.Client.Adapters.Mint