Skip to content

SimBrief prefetch: departure, gate pre-warm, optional alternate - #30

Open
MarcelL94 wants to merge 5 commits into
flysimdev:developfrom
MarcelL94:feature/simbrief-prefetch-pr
Open

MarcelL94 wants to merge 5 commits into
flysimdev:developfrom
MarcelL94:feature/simbrief-prefetch-pr

Conversation

@MarcelL94

@MarcelL94 MarcelL94 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What it does

An optional SimBrief prefetch. It stays off until a SimBrief username or Pilot ID is set (simbrief-user). SimBrief failures are only logged; they never make the mount fail.

Three commits, in the order you asked for:

  1. Settings, Setup step, OFP fetch/parse and the departure prefetch. When streaming starts, the 3 × 4 DSF tiles X-Plane loads around the departure are queued, each texture at the level the paging model predicts from the gate, at Warm priority at most (R2). Also in this commit: the download budget simbrief-max-mb (default 1000 MB, R4), cache trimming that removes areas X-Plane never opened first (R5), one User-Agent orthostreamxp/<version> for every server (R7), the username kept out of all logs (R8), a 10 s timeout with the fetch in the background (R10), skipping plans older than 6 h (R11), and a summary per phase when streaming stops (R12).
  2. The gate pre-warm. It covers the arrival, then the climb to TOC (capped at 200 NM / 25 min), at Background+1.. priorities, below everything else (R3). A waiting ground request gives way to load-ahead (H1). On the takeoff roll (> 40 kt), requests still waiting are dropped, both the ground ones and the departure's (H4). If the first real position is more than 50 km from the departure, the departure requests are dropped (R11). Build memory is given back once the pre-warm finishes. The Home button Reload SimBrief flight plan is the only re-fetch; there is no polling (R9).
  3. The alternate, behind its own switch simbrief-alternate, off by default.

How the work is split, so each commit makes sense on its own:

  • The "aircraft is elsewhere" drop and the takeoff drop need the position watch, so they come in commit 2. Until commit 2, waiting departure requests are not dropped at takeoff.
  • The Reload button mainly exists to re-arm the gate watch for a new plan, so it is in commit 2 too.
  • The OFP parser reads the alternate in commit 1 already (it is part of parsing the plan); only commit 3 uses it.
  • Two load-ahead changes are in commit 1 because the departure prefetch depends on them. They help every flight, not only SimBrief ones. First, a level asked for ahead on an empty texture fetches its base tile first. Second, nothing is fetched ahead coarser than X-Plane's smallest read (64 KB chunks: level 4 of a 4096 px texture). The second change removed all 926 ZL18 waits at EDDV.

R6: reusing the existing models (your call before I rework anything)

Need Now Existing model Proposal
Mip per texture PagingModel.MipFor PagingModel Already reused
Which DSF tiles X-Plane loads SceneryArea.TilesAround (3 × 4 rule) SceneryLoads only observes DSF opens Move the 3 × 4 rule and DsfTerrain next to SceneryLoads
Texture order around a point MipWarmer.PrefetchPlaced PrefetchPlanner (track-based) Check whether PrefetchPlanner.Select can take a point; otherwise keep it and say why
On ground / takeoff roll GroundWatch on the camera's motion AircraftPositionListener has no on-ground or ground-speed data Add X-Plane's ground speed and on-ground flag to AircraftPositionListener and let GroundWatch use them. This also fixes the camera-speed issue in outside views (M4)

Testing

  • dotnet build OrthoStreamXP.slnx: 0 warnings after each commit (TreatWarningsAsErrors, AnalysisMode=Recommended).
  • dotnet test --solution OrthoStreamXP.slnx on Windows with WinFsp (MountSmokeTests ran): 406 / 430 / 432 passed, 0 failed, 0 skipped after commits 1 / 2 / 3.
  • Rebased on the current develop (65deb09).
  • Not yet flown in this exact form. The numbers below come from earlier test builds of the same feature on the working branch.

Numbers so far (earlier builds; to be redone)

EDDV, empty cache, Ultra, 128 download workers, X-Plane's first load at the gate:

Run Tile loading Reads that waited Downloaded Real imagery
No prefetch 56 s 1,237 245 MB 100 %
Departure prefetch + base tiles first + coarsest read 51 s 0 212 MB 100 %

EDDV → ENGM, 121 min, gate pre-warm on (alternate included in that build): 33,714 reads, 1 wait (4 ms), 3 flat far reads, 0 failed builds. At ENGM, 95–98 % of the textures X-Plane loaded already had what it wanted, against 1–6 % on rows that were not pre-warmed. There was no control flight, so the 0 waits at ENGM are not a measured difference yet. Working set went up to 4 GB during the pre-warm in that build. The memory give-back in commit 2 addresses that but has not been flown yet.

Still to do before ready for review: 3–5 flights of different lengths and regions with the per-phase summary (R12), then the table above, a decision on the 1000 MB default and on whether the alternate is worth keeping, and your answer on R6.

🤖 Generated with Claude Code

@flysimdev

Copy link
Copy Markdown
Owner

@MarcelL94
Thanks for opening this as a draft, Marcel. This is impressive work: the structure is clear, the tests are thorough (I get 432/432 locally with no warnings), and you're upfront about what isn't measured yet. That makes it much easier to review. I'm really looking forward to this one.

Following up on my notes in the issue discussion

Several of my open questions are answered by the code:

  • Download budget: only real downloads count; tiles from the cache don't. 👍
  • Cache trim: never-opened areas are recognised through area-usage.json, and an area counts as normal again once X-Plane opens it. 👍
  • User-Agent: one shared value everywhere, and the old 0.2 is gone. 👍
  • Privacy: the username and Pilot ID stay out of logs, error texts and flight reports. 👍

Still open from that list: the cold-cache run, a control flight without the prefetch, and the length of the pause from the memory give-back (the log line already records it, so just post the number).

R6: my answers

  • Move the 3 × 4 rule and DsfTerrain next to SceneryLoads: yes. Please make it a separate refactor-only commit before the feature commits.
  • Texture order around a point: no need to check. PrefetchPlanner.Select with motion: null already gives a circle around a point, nearest first. Please use it instead of the custom placement.
  • Ground speed and on-ground flag: yes. XPlaneUdp already subscribes to a list of datarefs, so adding groundspeed and onground_any should be small. It also fixes the outside-view camera issue.

To fix before review

  1. An unexpected SimBrief answer can break Stop. SimBriefClient.Parse only catches JsonException. If a field has the wrong type (fetch as a string, status as a number, navlog as ""), it throws InvalidOperationException. That error ends up in WhenDone(), and StopAsync only catches cancellation there, so the listener and the packages are never disposed. Please treat any unexpected shape as "could not be read", and let Stop log a failure from the SimBrief work and carry on.
  2. The departure prefetch has the H1 problem. In PrefetchPlaced, departure requests overwrite whatever is waiting (_waiting[key] = request). Then AddOrOutrank ignores X-Plane's own load-ahead request while the departure request waits, and the takeoff roll drops both. It shows up when you press Reload while X-Plane is running. Please apply the same rule as for the ground requests, with a test.
  3. Without mip streaming, the departure prefetch isn't tracked. The older queue path isn't counted against the budget, isn't in the per-phase summary, and isn't cancelled on the takeoff roll. Please cover it, or document that the budget doesn't apply there.

The two load-ahead changes in commit 1
Fetching the base tile first and never fetching coarser than X-Plane's smallest read both sound like real improvements, and removing all 926 ZL18 waits is a great result. But they change streaming for every user, not just SimBrief users. Could you put them in their own commit, and measure them on their own (no SimBrief)? A normal cruise flight would also confirm that total downloads don't go up. Right now the before/after table mixes three changes in one row, so we can't tell which one did the work.

Suggestions for Sunday's flights

  • One flight starting with an empty cache.
  • Press Reload while X-Plane is already running (this exercises item 2).
  • At the gate, move the camera fast in an outside view, to check it doesn't count as a takeoff roll.
  • One short flight with mip streaming off.
  • One control flight without the prefetch, same route and same cache state.

For the release notes, later

  • The cache trim now removes never-opened areas first for all users.
  • The User-Agent changes from 0.2 to the real version.

That's a lot of points, but most are small, and the foundation is solid. Good luck with the flights on Sunday. I'm looking forward to the numbers and the next round. 🛫

@MarcelL94
MarcelL94 force-pushed the feature/simbrief-prefetch-pr branch from 1950ea7 to 058c7db Compare November 12, 2026 17:06
MarcelL94 added a commit to MarcelL94/OrthoStreamXP that referenced this pull request Nov 12, 2026
…ve-back

- R6: the 3 x 4 rule next to SceneryLoads and DsfTerrain in its own file; X-Plane's
  groundspeed and onground_any for the gate and the takeoff roll; the airports ordered
  by PrefetchPlanner.Select without motion (the climb keeps its own order).
- flysimdev#30 item 1: SimBriefClient.Parse never throws on an unexpected shape; a throwing
  fetch is "no plan"; StopAsync logs a SimBrief failure and still disposes everything.
- flysimdev#30 item 2 (H1): load-ahead outranks the departure, the departure the ground
  pre-warm, so the takeoff roll never drops load-ahead's request.
- flysimdev#30 item 3: without mip streaming only the departure is built; documented and logged.
- simbrief-max-mb is now simbrief-download-limit; the setup summary says "on", not the user.
- The memory give-back waits for 10 s without reads or builds and logs the pause.
- Tests for all of it; a log-filter in LoadMeasureTests no longer counts other tests' lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MarcelL94

MarcelL94 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review! I've worked through your notes and force-pushed the branch, rebased on the current develop (2f498d1). The description above still shows the old state; I'll update it together with the numbers after Sunday's flights.

Commit structure

Now five commits, each building with 0 warnings and passing all tests on its own (388 / 392 / 434 / 466 / 468):

  1. Refactor only: the 3 × 4 rule (SceneryLoads.TilesAround) and the TERT reader (DsfTerrain) next to SceneryLoads. Nothing calls them yet. Both are new in this PR, so the commit adds them in their final place rather than moving them.
  2. The two load-ahead changes on their own (base tile first, nothing coarser than X-Plane's smallest read), so they can be measured without SimBrief.
  3. The departure prefetch.
  4. The gate pre-warm.
  5. The alternate.

R6

  • PrefetchPlanner.Select(motion: null) now orders the departure, the arrival and the alternate; the custom placement is gone. The order value stays in metres, so departure requests rank against load-ahead's own requests of the same priority on the same scale. The climb keeps its own ordering, because its distance is to the nearest point on a line, not to a point.
  • groundspeed and onground_any are subscribed in XPlaneUdp. GroundWatch uses them for the gate and the takeoff roll, so moving or lifting the camera in an outside view no longer counts. Lifting off ends the pre-warm too, even if no roll faster than 40 kt was seen. Without the datarefs it falls back to the old logic. Side effect for every user: Home's speed and the flight-report samples now use X-Plane's ground speed.

Your three points

  1. Unexpected SimBrief answer breaks Stop. Parse checks the value kind everywhere and treats any unexpected shape as "could not be read". Your examples are covered by tests: fetch as a string, status as a number, navlog as "", a non-object fix, a number out of range. A fetch that throws becomes "no plan". StopAsync logs a failure from the SimBrief work and still disposes the listener and the packages.
  2. H1 for the departure. One ranking in AddOrOutrank for every SimBrief phase: load-ahead and the flight path, then the departure, then the ground pre-warm. A departure request no longer overwrites a waiting load-ahead request, and load-ahead's request replaces a waiting departure one, so the takeoff roll can't drop it. Tested both ways, including a departure queued again over textures X-Plane has already loaded (the Reload case).
  3. Without mip streaming. Documented and logged at start rather than covered. In that mode only the departure is built, bounded by the background queue (4 × dds-build-workers textures). It doesn't count against the budget, isn't in the per-phase summary and isn't dropped on the roll. The arrival and the climb need mip streaming anyway.

Other changes

  • Setting name: simbrief-max-mb is now simbrief-download-limit (MB), in the style of the other settings. It was never released, so there is no migration.
  • Privacy: the Setup summary said "SimBrief: ". It now says "on". The docs say to remove simbrief-user from the ini before attaching it to a bug report.
  • Memory give-back (R1): it was one blocking, compacting full GC, which pauses every thread, the file-system threads included. It now waits until no build is running and X-Plane has read nothing for 10 s, and gives up after 10 min or when new pre-warm work arrives. It logs the pause (GC.GetTotalPauseDuration) and, 60 s later, the working set plus the reads and builds since. I'll post the pause from Sunday.
  • Cache trim: a new test shows that an area that was only pre-warmed and later flown is trimmed by its age like any other.
  • Budget, one correction to my earlier claim: only real downloads count, but the tile store only keeps base tiles. A texture whose base tile is cached is skipped as "already there", and that costs nothing. A full-detail level built later downloads its tiles again, and those bytes do count.

One request: CI hasn't run on this PR yet. The workflow is waiting for approval (fork PR from a first-time contributor). Could you approve it?

MarcelL94 and others added 5 commits November 13, 2026 10:33
Refactor only, no behaviour change: nothing calls them yet. SceneryLoads observes the
DSF tiles X-Plane opens; TilesAround says which ones it opens around a place (three
rows by four columns, leaning to the nearer half of the column, checked at EDDV).
DsfTerrain reads a DSF's TERT table without reading the rest of the file. The SimBrief
prefetch builds on both in the next commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est read

Two changes to load-ahead for every flight, not only SimBrief ones:

- A level asked for ahead on a texture with nothing yet (the flight-path prefetch)
  fetches its base tile first, in the base tiles' own places. Load-ahead skips a
  texture that is whole already, so without this the low-zoom base (far scenery,
  colour matching, the report's map) stayed empty there. If the base tile fails, the
  request goes instead of being tried every pass, and only real build starts count.
- Nothing is fetched ahead coarser than X-Plane's smallest read: it reads the file's
  last 64 KB chunk, so it takes at least level 4 of a 4096 px texture. For ZL18
  textures that is two levels finer than the base tile; fetching only the base tile
  made all 926 ZL18 textures at EDDV wait while X-Plane loaded them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Optional, off until a SimBrief username or Pilot ID is set (simbrief-user).

- Setup step 6 "SimBrief" with a connection test, and a SimBrief tab in Settings
  (username, test, download limit per session).
- Fetch and parse the latest OFP (origin, destination, alternate, navlog) once per
  streaming session, 10 s timeout, in the background. Any answer of an unexpected
  shape is "could not be read", a failing fetch is "no plan": logged, never a mount
  failure, and the stop logs a failure of the SimBrief work and still disposes
  everything. The username is never logged nor shown in the setup summary;
  User-Agent orthostreamxp/<version> for every server.
- Departure prefetch: the 3 x 4 DSF tiles X-Plane loads around the departure,
  ordered by PrefetchPlanner.Select without motion, each texture to the level the
  paging model predicts from the gate, at Warm at most. A departure request never
  replaces a waiting load-ahead request, and load-ahead's replaces it (H1).
  Without mip streaming (or unreadable DSFs): the textures within
  simbrief-radius-km at full detail, bounded by the background queue, outside the
  budget (documented and logged).
- A plan older than 6 h, or an aircraft already > 50 km away, skips the prefetch.
- Download budget simbrief-download-limit (1000 MB): when spent, the waiting
  SimBrief requests are dropped; load-ahead and X-Plane's reads never count, nor do
  tiles the cache has.
- Areas X-Plane never opened go first when the cache is trimmed; once flown they
  are trimmed by age like any other.
- Per-phase summary at stop (queued, builds, tiles/MB, build time, hit rate).
- Home: a session switch and the flight plan line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Once the aircraft has stood at the departure for 30 s, the 3 x 4 tiles around the
  arrival, then the climb to TOC (capped at 200 NM / 25 min), are queued at
  Background+1.. priorities, below everything else (R3). Load-ahead outranks the
  departure, the departure outranks the ground pre-warm (H1).
- The gate and the takeoff roll come from X-Plane's own groundspeed and
  onground_any datarefs (R6): moving or lifting the camera in an outside view no
  longer counts (M4). Without them, the speed from the positions and the camera
  height as before. Home's speed and the flight-report samples use X-Plane's
  ground speed too.
- On the takeoff roll (> 40 kt, or airborne) the requests still waiting are dropped,
  the ground ones and the departure's (H4). If the first real position is more
  than 50 km from the departure, the departure requests are dropped (R11).
- The build memory is given back once the pre-warm is built and it is quiet: no
  build and no read from X-Plane for 10 s (a compacting GC pauses every thread,
  R1). The pause and the working set a minute later are logged.
- Home: "Reload SimBrief flight plan", the only re-fetch, no polling (R9). A new
  plan drops the old one's waiting requests, builds its departure (also when the
  old plan was too old to have built it), restarts the download limit, and
  watches the new gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
simbrief-alternate = on (Setup and Settings): the ground pre-warm builds the
alternate's 3 x 4 tiles after the arrival, before the climb. Off by default: it is
rarely flown and spends downloads from the same limit. An alternate that is the
arrival or the departure is not built twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MarcelL94
MarcelL94 force-pushed the feature/simbrief-prefetch-pr branch from 058c7db to 07c9355 Compare November 13, 2026 09:38
@MarcelL94
MarcelL94 marked this pull request as ready for review November 13, 2026 19:16
@MarcelL94 MarcelL94 changed the title [WIP] SimBrief prefetch: departure, gate pre-warm, optional alternate SimBrief prefetch: departure, gate pre-warm, optional alternate Nov 13, 2026
@MarcelL94

Copy link
Copy Markdown
Contributor Author

Hi,

Quick update on the SimBrief prefetch feature: the implementation is complete, and the happy path works as I intended. Unfortunately, due to unforeseen personal time constraints, I won’t be able to run all the remaining edge-case tests before the end of the year.

Rather than leaving the work sitting in my fork, I’ve opened this PR against develop so it’s available to you. The edge cases haven’t been fully tested yet, so please keep that in mind.

It’s entirely up to you how to proceed. You could merge it into develop for further testing, keep it open until the edge cases are covered, or close it if it doesn’t fit your plans. I’m happy to answer questions about the implementation in the meantime, and I’ll come back to it when I have more time.

Thanks for the project and for considering the contribution!

@flysimdev

Copy link
Copy Markdown
Owner

@MarcelL94
Hi,

Thanks for the update, and for putting this PR up so it isn't stuck in your fork. Glad to hear the main flow is working the way you planned.

No rush on the edge cases. I'm happy to leave the PR open until you have time to come back to it. Take care of things on your side first, the project isn't going anywhere.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants