Cover DNS entrypoint failover and negative fallback cases

[dkropachev]

Jira task: https://scylladb.atlassian.net/browse/DRIVER-821
Jira epic: https://scylladb.atlassian.net/browse/DRIVER-815

This issue applies DRIVER-815 to scylladb/alternator-client-python.

Problem

This Alternator client can be configured with DNS entrypoints that resolve to multiple records. Some records can be unavailable, stale, blackholed, misconfigured, or point to a node that cannot provide usable live-node data. The client must not permanently fail or get stuck on the first bad DNS record when another DNS record or the original seed endpoint can recover discovery.

This issue covers negative DNS entrypoint and live-node fallback scenarios for the Python Alternator client.

Expected behavior

When a DNS entrypoint resolves to multiple records, the client should try usable alternatives before failing the discovery cycle. If one resolved record is broken, the client should move to the next resolved record, fetch /localnodes from the first reachable valid endpoint, learn the live node set, and continue routing normally.

When a client session is already active and all currently known live nodes become unavailable, the client should fall back to the original seed endpoints or DNS entrypoints, resolve them again, use the first reachable valid DNS record, relearn live nodes from /localnodes, and keep working normally.

When routing scope is configured, DNS fallback should discover the cluster through any reachable entrypoint before applying scope rules. If the scope is valid or has a configured fallback scope, the client should keep working.

Negative scenarios to cover

1. Initial discovery: first DNS record is broken, later DNS record is reachable.
2. Initial discovery: several leading DNS records are broken, later DNS record is reachable.
3. Active session: client has learned live nodes, then all known live nodes go down or become unreachable.
4. Active-session recovery: after known nodes fail, client falls back to original seed endpoints or DNS entrypoints, resolves DNS again, learns a fresh live-node list, and continues serving requests.
5. Partial live-node failure: some known nodes fail during refresh, but at least one known node or DNS fallback endpoint is reachable.
6. All DNS records unavailable: client reports a clear failure according to existing retry/backoff behavior and does not hang indefinitely.
7. DNS record is reachable but /localnodes fails with connection reset, timeout, transport error, or non-2xx response; client tries another DNS record when available.
8. DNS record is reachable but /localnodes returns malformed, empty, or unusable node data; client tries another DNS record when available and otherwise fails clearly.
9. Wrong datacenter, rack, or scope with fallback configured: client falls back to the configured scope and keeps working if matching nodes exist.
10. After recovery, normal operations route to learned live nodes instead of repeatedly using the broken DNS record.
11. NXDOMAIN, SERVFAIL, DNS timeout, or an empty answer fails clearly for the current cycle, retains the original DNS seed, and succeeds after a later re-resolution returns usable records.
12. Active-session recovery re-resolves the original hostname and can use newly added or reordered records instead of remaining pinned to stale addresses.
13. Failure, empty data, or invalid data from one configured seed does not prevent discovery through another configured seed.
14. Address-level fallback preserves configured hostname, scheme, and port for HTTP Host, TLS SNI/certificate validation, and request-signing semantics.
15. Concurrent requests and overlapping refreshes observe either the previous valid learned-node set or the new valid set, never a partial or transiently empty set.
16. DNS lookup, per-address connection, validation, retry/backoff, cancellation, and shutdown remain bounded and do not create a busy loop.

Acceptance criteria

• Add deterministic tests for DNS record fallback during initial discovery, including several leading failures.
• Add deterministic tests where an address accepts a connection but /localnodes returns non-2xx, malformed, empty, or unusable data, then verify discovery continues with the next resolved address.
• Add active-session recovery tests where all known live nodes are down and the client falls back to original seed endpoints or DNS entrypoints.
• Add DNS re-resolution tests for initial resolution failure, later recovery, and changed answers during an active session.
• Cover wrong scope with fallback when the client supports fallback scopes.
• Verify /localnodes is fetched from a later reachable DNS record and learned live nodes replace the DNS entrypoint according to existing discovery behavior.
• Verify normal requests continue after DNS fallback recovery.
• Verify multiple seed entrypoints have bounded fallback behavior.
• Verify address-level fallback preserves logical hostname semantics for Host, TLS, and signing.
• Verify concurrent refreshes update the learned-node set atomically and respect cancellation/shutdown bounds.
• Prefer unit-level resolver/discovery coverage plus integration-level scenarios where feasible.
• Do not regress successful DNS entrypoint behavior covered by DRIVER-39.

Current repository gap

Connection-failure, seed-recovery, and scope paths exist; reachable-but-invalid /localnodes address fallback and equivalent sync/async coverage remain.

Related work

Builds on DRIVER-39, which added successful DNS entrypoint discovery coverage.

[dkropachev]

Implementation: #107

Jira task: https://scylladb.atlassian.net/browse/DRIVER-821
Jira epic: https://scylladb.atlassian.net/browse/DRIVER-815