Repository navigation
Online sockets for gRPC games: RecvMsg/SendMsg, C resolver, game errno, EventFd in a network mode - #2589
Open
eyasbzour wants to merge 51 commits into
Open
Online sockets for gRPC games: RecvMsg/SendMsg, C resolver, game errno, EventFd in a network mode#2589eyasbzour wants to merge 51 commits into
eyasbzour wants to merge 51 commits into
Conversation
Replace unconditional failure and shared task completion with a precision-checked kernel NTP query and independent SDK event lifetimes. Preserve real failure results and release the guest core across contended service locks.
Translate the Horizon IPv4 socket ABIs, retain generation-safe descriptor lifetimes and bound local discovery. Publish real guest contexts across blocking hooks using the existing per-thread gate stack. Add LDN discovery, membership and lifecycle plus local play-history registration from privately imported resources. Preserve upstream offline account and TLS boundaries. Synthetic and real loopback fixtures, ASan/UBSan, the full host suite, the production HLE registry, iPhoneOS compilation and synthesized resource-decoder tests pass. Mario Kart 3.0.5 hardware results are owner reports from the originating fork, documented separately from this rebased branch. No frontend, Nextendo integration or game/firmware data is included.
Keep the test thread heap-owned like production, release its handle after joining, and include it in the local-network test target. Stop socket fixtures at failed prerequisites rather than continuing with invalid descriptors.
Add the system permission description required by the native socket transport. Remove redirect-only resolver bookkeeping from the upstream local-only backend; no routing provider or UI is included.
…y copyout Kernel recvfrom copyout cannot invoke the guest dirty-page signal hook. Prepare the pending span before each receive attempt, including after readiness waits and partial stream reads, without retrying EFAULT or admitting unrelated invalid memory. The real dirty-tracker fixture passes 43 checks normally and under ASan/UBSan. Negative controls reproduce missing preparation and the wait boundary. The full host suites, iOS compile check and documentation checks also pass.
…e copyout The kernel getsockopt copyout cannot invoke the dirty-page fault hook. Prepare the validated result span without changing SetSockOpt inputs, invalid-memory failures, result sizes or errno translation. The expanded real dirty-tracker fixture reproduces 16 failures in 87 checks before this change and passes all 87 normally and under ASan/UBSan afterward. This synthetic reproduction does not establish a game disconnect cause.
Use owned DNSService queries and cancellation identities instead of leaving resolver work and late completions alive. Retire native references on one serial queue, fence finalization, and retain the offline and synchronous resolver paths.
Use owned DNSService queries and cancellation identities instead of leaving resolver work and late completions alive. Retire native references on one serial queue, fence finalization, and retain the offline and synchronous resolver paths.
Keep per-session read-only snapshots and scoped eligibility checks behind a default-off provider hook. Exercise startup, delivery lifetime and sanitizer failures with original offline fixtures; compose the TLS dependency without replacing upstream filesystem behavior.
Replace unconditional failure and shared task completion with a precision-checked kernel NTP query and independent SDK event lifetimes. Preserve real failure results and release the guest core across contended service locks.
Check Foundation fixture construction and bounded DER reads explicitly, initialize the filesystem fixture handle, and remove a fixed filesystem null-argument baseline entry. No new suppression or baseline expansion.
…tely Ordinary-buffer TLS, policy and resolver tests use the explicit no-tracker fixture hook. The real socket-copyout fixture keeps its registered dirty tracker and links the NSD backend needed by this stack.
I preserved upstream's readback allocation expectation and the existing explicit serial-table release. The LAN/LDN transport and tracked socket copyout corrections remain unchanged. Local network fixtures, 87 normal and sanitized copyout checks, 61 Metal geometry checks, and the documentation tests pass after the merge.
…stream-online-backend-proposal # Conflicts: # dm/host/Makefile # dm/host/hle_net_backend.c # dm/host/tests/Makefile # docs/agents/symptoms.md # docs/reference/services.md
…stream-online-backend-proposal
…stream-online-backend-proposal
…stream-online-backend-proposal # Conflicts: # dm/host/tests/Makefile
…stream-network-clock-main # Conflicts: # docs/agents/symptoms.md
I port only the two-line renderer correction from seedds/nxnative commit 5ccbedd. Existing key 152 indexed one past the 152-element g_red_once array. Capacity 153 covers that key, and the upstream compile-time assertion rejects future keys outside the array. The source macro fixture reproduces the old global-buffer-overflow under ASan, accepts key 152 with ASan/UBSan, and rejects key 153 at compile time. No Python, lint baseline, suppression, shader-chain or profile changes from the upstream maintenance commit are included.
…stream-cancellable-resolver
…stream-online-backend-proposal
I merge frozen upstream main a3d85ae after it overlaps the earlier typed-flow maintenance in this draft. I reconcile the three parser/test conflicts hunk by hunk to upstream's canonical implementations and remove the coupled tuple/count remnants. Those files and their ratchet now match main; the backend host sources remain byte-identical to the preceding 8f03fe1 head. 79 focused parser/shader-chain/source-cache tests, 17 documentation/Xcode tests and 286 output-equivalence comparisons pass. The isolated upstream pinned Ruff/Pyright environment passes 389/389 with zero new/stale findings. The inherited upstream shader chain is now 0ad559cdbb28. I have not rebuilt or qualified game outputs for that chain. The draft scope/dependency and game-qualification limits remain in place.
Super Mario Bros. Wonder's NPLN gRPC failed with 2321-4992 (UNAVAILABLE) before it opened a socket: nnsocketGetAddrInfo returned EAI_FAIL for every SDK that exports both C++ GetAddrInfo overloads, and Wonder's does. In Wonder 1.2.1's sdk, nnsocketGetAddrInfo branches to the plain-addrinfo overload, and FromBufferInternal<addrinfo> stores the address at +0x20 (AddrInfo: +0x18), so the C resolver now gets that layout whenever the SDK has it. RecvMsg and SendMsg, which gRPC's TCP endpoint reads and writes with, scatter and gather through one host buffer using Horizon's 0x30-byte msghdr. Ancillary data is reported as none on receive and refused on send. Name lookups in a network mode are logged with their result; resolver options are logged, and a set one is accepted without being applied. services.md has the entry; t_net covers the round trip and the resolver layout.
…e cases On Horizon nn::socket's errno is the C library's: detail::SetLastErrno and GetLastErrno write and read *__errno_location(). The host kept its own copy only, so code that reads errno itself saw a stale value. Super Mario Bros. Wonder's gRPC sets EAFNOSUPPORT when it leaves IPv6 (main+0xb61480), then reads errno after its non-blocking connect and took that stale value for the connect's EINPROGRESS (tcp_client_posix.cc:299), so it never sent a byte. Every socket hook that fails now also writes the game's errno through its __errno_location, and GetLastErrno reads it in a network mode. RecvMsg and SendMsg no longer fail a stream message over the 16 MiB buffer: it moves in part, as a real socket may. A datagram that long fails with EMSGSIZE. A datagram longer than the iovecs is cut to them and reported with MSG_TRUNC; the backend's new dm_net_socket_type tells the two apart. t_net covers the errno write and read, EMSGSIZE and MSG_TRUNC.
Brings the branch (seedds seedds#961 plus the gRPC socket fixes) up to current main. The conflicts follow how eyasbzour/nxnative's nextendo/wonder resolved the same ones: - hle_net.c / hle_offline.c: main's offline layer (the event descriptor table, Select as Poll over its sets, Star Fox's SDK 21 overloads) with seedds#961's transport in a network mode. The SDK 21 overloads go to the transport too. Main's ssl stubs stay out (seedds#961's hle_ssl.c serves them), and the SDK 4 GetPeerName is registered once, in hle_net.c. EventFd keeps main's offline table and still fails in a network mode. - hle_account.c: main's SDK-built AsyncContext (no constructor takeover) with seedds#961's account requests; with Nextendo off a context answers as main's does, and a move constructor copies the vtable. - hle_fs.c, the Makefiles, build_ipa.sh, gen_build.py, env_switches.tsv, main.c, the Mario Kart 8 README: both sides kept. - hle_bcat.c drops HLE_FORCE_BIG, which main removed (every takeover now runs on the host stack), and t_net_eventfd links the socket backend and stubs guest_sym, which hle_net.c now calls.
…ols/docs_symptoms.py)
…ket table gRPC (nn::npln) wakes its pollers with an event descriptor and polls it with its sockets. In a network mode EventFd failed, and the host's own event table numbered from 0 as the transport's sockets do, so a descriptor could have shared a socket's number while Poll went to the transport, which knew nothing of it. dm_net_eventfd takes a socket-table slot (type NET_EVENT) whose host descriptor is the read end of a local socket pair holding one byte while the counter is above zero. Poll, Select, Close, finalize and the generation checks then cover sockets and event descriptors alike; socket operations on one fail with ENOTSOCK. Reads honour the semaphore and never-block flags, a blocking read waits in the transport's bracketed slices and fails with EBADF if closed meanwhile, and a write that would pass the counter's maximum waits or fails with EAGAIN. Offline the host's counter table is unchanged. t_net covers mixed Poll and Select, the flags, a read blocked across a write and across a close. On this branch the hooks also write a failed call to the game's errno, as every socket hook does here; t_net checks it for the EAGAIN of an empty never-blocking read.
… clock and wait stubs hle_bcat_apple.m takes the game's BCAT title IDs from DM_BCAT_TITLES and DM_BCAT_PLAIN_TITLE (test_no_title_ids_in_the_host); the BCAT fixtures pass Splatoon 2's. t_hle_wait, t_network_clock and t_net define the clock and wait symbols main's hle_os.c and hle_thread.c now provide.
github-actions
Bot
added
host
…e first one listed iOS has no routing table to read and getifaddrs lists cellular before Wi-Fi, so in a Nextendo session nifm reported the cellular address while the phone played over Wi-Fi, and Pia offered it to other players for a direct link. The first pass now takes the adapter whose address a UDP socket connected past the local networks gets (connect sends nothing); any adapter if none matches. The adapter is logged when it changes.
eyasbzour
marked this pull request as ready for review
November 13, 2026 18:37
hle_oe.c's dm_set_opmode calls dm_hid_opmode_changed and oe::ExecuteProgram calls dm_host_shutdown_then; the test stubs them as t_notify_sched.c does.
…e_account.c expects hle_account.c keeps main's SDK-built AsyncContext, so there is no constructor hook to call: the test's constructor writes nothing, as the SDK's writes only guest memory the host never reads. Added checks: a move constructor writes the vtable and a null IPC pointer and carries the request; with Nextendo off a context answers as the offline host does.
…c's too The check wanted cdevice=Mac..., but a virtual Mac (a CI runner) reports VirtualMac2,1. It now compares with hw.model, as save_sync.c device_name reads it.
eyasbzour
added a commit
to eyasbzour/nxnative
that referenced
this pull request
Nov 13, 2026
…eded The survey docs still said no PRs go to seedds and that Mush8 code may be ported. The descriptor table, EventFd and NPLN gaps are filled by seedds#2589, the client ID is a build setting, and the Mario Kart menu sites move to a kart branch check.
eyasbzour
added a commit
to eyasbzour/nxnative
that referenced
this pull request
Nov 13, 2026
…eded The survey docs still said no PRs go to seedds and that Mush8 code may be ported. The descriptor table, EventFd and NPLN gaps are filled by seedds#2589, the client ID is a build setting, and the Mario Kart menu sites move to a kart branch check. (cherry picked from commit 500a94b)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This builds on #961 (RobiiAragon's opt-in online backend), which is still open, so until #961 merges the diff shows its roughly 10k lines too. The commits that are new here:
b8e25957Serve gRPC's name lookup and RecvMsg/SendMsg in a network modeaa03bca5Set the game's own errno on a failed socket call; RecvMsg/SendMsg edge cases8aace781Merge seedds/main (465 commits; notes below)51c46eceRegenerateddocs/agents/symptoms.mdf1afbb84EventFd in a network mode is a slot of the transport's socket table8a36c156,e3fa880aMerge seedds/main again (no conflicts)e6d91b7fThe BCAT title IDs come from the build instead of the host source (test_no_title_ids_in_the_host), and three tests define the clock and wait symbols that main'shle_os.candhle_thread.cnow provideae3a5f02iOS online: the game is told the address of the adapter the internet route uses, not the first one listedfb1d0b3e,b8bec5a9,cb5bdacdTests that broke after merging main:t_oe_handoffstubs whathle_oe.cnow calls,t_nextendo_accountlets the SDK build the AsyncContext, andt_save_synccompares the device withhw.model(a CI runner is aVirtualMac2,1)Why
Games whose online code is NPLN (Super Mario Bros. Wonder and other recent titles) talk to their servers over gRPC, which links its own OpenSSL and does TLS itself over the C socket interface. With #961 alone, Wonder's online menu fails with 2321-4992 (gRPC's UNAVAILABLE) at four separate points. Each fix below sits at the SDK level, with no game offsets in the code, so it applies to any game that makes the same calls.
nnsocketGetAddrInforeturned EAI_FAIL whenever an SDK exports both C++GetAddrInfooverloads, and Wonder's does. The C call takes the plainaddrinfolayout. In Wonder 1.2.1's SDK it branches straight to that overload (sdk+0x1481c0), so the C resolver now uses that layout whenever the SDK has it.gRPC's TCP endpoint reads and writes only through
RecvMsgandSendMsg, which were unsupported. They now scatter and gather through one host buffer with Horizon's 0x30-bytemsghdr. A message over 16 MiB moves in part on a stream socket, as a real socket may, and fails with EMSGSIZE on a datagram one. A datagram longer than the iovecs is cut and reported withMSG_TRUNC. Ancillary data is refused rather than dropped.On Horizon,
nn::socket's errno is the C library's:detail::SetLastErrnowrites*__errno_location(). The host kept only its own copy. gRPC sets EAFNOSUPPORT itself when it falls back from IPv6, then reads errno after a non-blocking connect, so it took that stale value for EINPROGRESS and dropped the address. Every socket hook that fails now also writes the game's errno, andGetLastErrnoreads it.gRPC wakes its pollers with an event descriptor and polls it together with its sockets. In a network mode
EventFdfailed, and main's offline table numbered its descriptors from 0, the same as the transport's sockets. An event descriptor is now a slot in the transport's socket table, backed by a local socket pair, so Poll and Select see sockets and event descriptors together. Offline nothing changes.On iOS,
net_routehas no routing table to read, so the adapternifmreports was the first IPv4 onegetifaddrslists. With mobile data on, that is usually the cellular one, even while the phone plays over Wi-Fi. Pia gives the addressnifmreports to the other players as its local candidate for a direct link. In a Nextendo session the adapter is now the one whose address a UDP socket connected past the local networks gets (connecton UDP sends nothing). If no adapter has that address, any adapter is used, as before. The chosen adapter's name is logged when it changes.Name lookups and resolver options are logged once, in red when they fail, so the next game that stops before its first socket shows why.
Merging main
Main's offline socket layer (the event descriptor table, Select as Poll over its sets, Star Fox's SDK 21 overloads) now sits under #961's transport, and the SDK 21 overloads go to the transport in a network mode. Main's ssl stubs stay out because #961's
hle_ssl.cserves those calls. The SDK 4GetPeerNameis registered once.hle_account.ckeeps main's SDK-builtAsyncContext: there is no constructor takeover, a context with Nextendo off answers as main's does, and a move constructor copies the vtable.hle_bcat.cdropsHLE_FORCE_BIG, since main now runs every takeover on the host stack.Tested
t_netcovers the gathered and scattered round trip, EMSGSIZE,MSG_TRUNC, the resolver layout, the game's errno on a failed call and on an empty event read, and mixed Poll and Select over sockets and event descriptors. The tests workflow on the fork: https://github.com/eyasbzour/nxnative/actions/runs/43498950204Not in this PR: the Nextendo sign-in for the Mac and iPhone, per-game patches, and the launcher. Those stay in the fork.