From e3a2699f4e01a538e8b2ab2adac5c82a01c663f6 Mon Sep 17 00:00:00 2001 From: Toby Murray Date: Fri, 7 Aug 2026 21:56:26 -0400 Subject: [PATCH 1/3] docs: write down the SharedData convention the SDK already ships StrideLut and OutdoorStrideCalibrator both default to ../SharedData/stride.json, and Running and Treadmill are both consumers of it, but the directory appears nowhere in Docs/. A third app has no way to learn the convention exists, or the two rules that make it work: mkdir before opening (f_open does not create parents), and treat every read as optional because a new watch has never written the file. Documents what the source demonstrably does, cited line by line, plus the simulator's unclamped ".." putting the shared files beside Output/ rather than inside it. Five things I could not settle from the SDK source, and so left out of the page rather than guess at. Answers would each turn into a paragraph: 1. Where does SharedData/ sit on the device? deploy.md puts apps under Apps//, which would make it Apps/SharedData/, but the SDK never says so and app paths are sandbox-relative. 2. Does it survive uninstalling the app that created it? If not, removing and reinstalling Running costs the user their calibration. 3. Is it visible and writable over USB mass storage? Decides whether data can be seeded from a desktop, and whether a user can clear a bad calibration by hand. 4. What happens when two apps write the same file at once? Whether the filesystem offers atomic rename, and whether any locking exists. 5. Is the name reserved -- can an app create a directory that collides with it? --- Docs/index.rst | 1 + Docs/shared-data.md | 88 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+) create mode 100644 Docs/shared-data.md diff --git a/Docs/index.rst b/Docs/index.rst index c9768245..f76b2d15 100644 --- a/Docs/index.rst +++ b/Docs/index.rst @@ -69,6 +69,7 @@ Next steps: TouchGFX-Port-Architecture touchgfx-widgets FitFiles-Structure + shared-data .. toctree:: :maxdepth: 4 diff --git a/Docs/shared-data.md b/Docs/shared-data.md new file mode 100644 index 00000000..2f5581e9 --- /dev/null +++ b/Docs/shared-data.md @@ -0,0 +1,88 @@ +# Shared Data Between Apps + +`SharedData/` is a directory that every app can read and write. An app reaches it from its +own root as `../SharedData/`. + +It exists because some data belongs to the user, not to an app. Stride calibration is the +case the SDK already ships: the Running app measures it outdoors with GNSS, and the +Treadmill app needs it indoors where there is no GNSS. + +For app-private files, see the [Files tutorial](Tutorials/Files/ARCHITECTURE.md). + +## The path + +```cpp +static constexpr const char *kPath = "../SharedData/stride.json"; +``` + +Paths are relative to the app's own root. The whole path, filename included, must fit +`IFileSystem::skMaxPathLen` (256 bytes). + +## Who uses it today + +| Path | Written by | Read by | +| --- | --- | --- | +| `../SharedData/stride.json` | `OutdoorStrideCalibrator::finalise()` ([`OutdoorStrideCalibrator.hpp:59`](../Libs/Header/SDK/Calibration/OutdoorStrideCalibrator.hpp)) | `StrideLut` ([`StrideLut.hpp:67`](../Libs/Header/SDK/Calibration/StrideLut.hpp)) | +| `../SharedData/stride_trace.csv` | Running app, when tracing is on ([`Running/.../Service.cpp:789`](../Examples/Apps/Running/Software/Libs/Sources/Service.cpp)) | diagnostic only | +| `../SharedData/stride_deleted.json` | Treadmill app, backing up the LUT before deleting it ([`Treadmill/.../Service.cpp:1177`](../Examples/Apps/Treadmill/Software/Libs/Sources/Service.cpp)) | recovery only | + +## Create the directory before opening a file in it + +`IFileSystem::mkdir()` creates missing parent directories. Opening a file does not. A writer +that skips the `mkdir` works on your watch and fails on one that has never run a calibrating +app: + +```cpp +// Ensure the SharedData directory exists (FatFs f_open does not create +// missing parents). "Already exists" counts as success. +const char *slash = std::strrchr(mPath, '/'); +if (slash != nullptr) { + char dir[SDK::Interface::IFileSystem::skMaxPathLen] {}; + std::snprintf(dir, sizeof(dir), "%.*s", static_cast(slash - mPath), mPath); + if (!mFs.mkdir(dir)) { + return false; + } +} +``` + +[`OutdoorStrideCalibrator.cpp:296-306`](../Libs/Source/Calibration/OutdoorStrideCalibrator.cpp) + +## Every read is optional + +Your reader has to work on a watch where nothing has written the file yet. That is the +normal state of a new device, not an edge case. + +`StrideLut::loadFromFile()` handles it by clearing itself and returning `false` when the +file is absent or will not open +([`StrideLut.cpp:163-175`](../Libs/Source/Calibration/StrideLut.cpp)). The caller gets an +all-zero LUT and falls back to a default model. + +## Share only what is shared + +App-specific data stays in the app's own root even when it is closely related to something +shared. The Treadmill app keeps its delta-LUT out: + +```cpp +/// Delta-LUT filename in the Treadmill app's own root (NOT under SharedData). +``` + +[`CadenceStrideModelConfig.hpp:72`](../Libs/Header/SDK/Calibration/CadenceStrideModelConfig.hpp) + +Ask whether another app would be *right* to read the file, not just curious about it. A +user's stride length is theirs and follows them between apps. A treadmill's calibration +offset only means anything inside the model that produced it. + +## Expect interrupted writes + +Apps are scheduled independently, and a watch can lose power mid-write. Write so that a torn +file is detectable: keep a backup copy before replacing (the Treadmill app does this with +`stride_deleted.json`), or use a format whose reader rejects a truncated file. The failure +to design against is a reader that silently accepts half a record. + +## In the simulator + +The simulator's filesystem root is `Output/` +([`Kernel.cpp:17`](../Libs/Source/Simulator/Kernel/Kernel.cpp)). It passes `..` through to +the host filesystem instead of clamping it to that root, so `../SharedData/` lands beside +`Output/` and not inside it. Apps behave correctly. The files are just not where you would +first look for them. From 77c96111e1c1c4da775410c5625c834469cf7a09 Mon Sep 17 00:00:00 2001 From: Ross Ryles Date: Sat, 8 Aug 2026 12:39:25 +0100 Subject: [PATCH 2/3] docs: fill in the SharedData rules that only the kernel knows The page landed the SDK-visible half of the convention accurately. This adds the half that is not visible from this repo, plus two corrections to claims the SDK source does not support. The load-bearing omission is that "../SharedData/" is a whitelist, not parent traversal. The kernel resolves "../SharedData" and "../SharedData/" and rejects every other ".." segment outright, so "../MyMaps/tiles.bin" never resolves at all. The page previously said only that paths are relative to the app root, which reads as though any sibling would work. The simulator makes that worse: it concatenates prefix and path with no whitelist, so an invented path works there and fails on the watch with no warning. Both are now stated. Corrections: - stride_deleted.json was cited as backup-before-replace. It is not. Treadmill writes it before an explicit user-initiated clear, never before a routine save. The calibrator overwrites stride.json in place with truncation, so a power loss during a save leaves a torn file and no backup at all. The .bak is written later, after a load has already found the store unparseable -- recovery evidence, not protection. RecordingMarker holds the pattern worth copying, and is now cited in its place. - stride.json.bak was missing from the file table. It is a fourth real file in the directory and anyone enumerating it will meet it. Answers to the five open questions, resolved against the kernel and confirmed on a watch over USB mass storage: 1. On device it is Apps/SharedData/, a sibling of Apps/Running/ and the rest. 2. It survives app removal by construction, being outside every app's root. Only a factory reset clears it, and nothing garbage-collects it otherwise -- which is free for calibration and a trap for anything map-sized. 3. Visible and writable over USB mass storage. 4. No atomic rename: rename onto an existing name fails, so write-temp-then- rename does not work. Locking does exist, in two layers -- the volume is serialised by a mutex, and a global table refuses a second open of a file already open for writing. That table is ten entries for the whole watch, shared with the syslog and activity recorder, which matters to any app wanting several files open at once. 5. The name is reserved by the resolver rule in (1). Also documents that IFile::open() flattens absent, locked, timed-out and too-many-open-files into one false, so the "every read is optional" rule can silently misread a concurrent writer as an empty file. --- Docs/shared-data.md | 96 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 91 insertions(+), 5 deletions(-) diff --git a/Docs/shared-data.md b/Docs/shared-data.md index 2f5581e9..0eb68e93 100644 --- a/Docs/shared-data.md +++ b/Docs/shared-data.md @@ -7,6 +7,10 @@ It exists because some data belongs to the user, not to an app. Stride calibrati case the SDK already ships: the Running app measures it outdoors with GNSS, and the Treadmill app needs it indoors where there is no GNSS. +On the watch the directory sits beside the app directories rather than inside any of them — +`Apps/SharedData/`, a sibling of `Apps/Running/` and the rest — and it is visible over USB +mass storage, so you can read a shared file from a desktop or seed one by hand. + For app-private files, see the [Files tutorial](Tutorials/Files/ARCHITECTURE.md). ## The path @@ -18,13 +22,28 @@ static constexpr const char *kPath = "../SharedData/stride.json"; Paths are relative to the app's own root. The whole path, filename included, must fit `IFileSystem::skMaxPathLen` (256 bytes). +`../SharedData/` is the only path that may leave your app's root, and it is a whitelist +rather than general parent traversal. The kernel resolves `../SharedData` and +`../SharedData/` to the shared directory, and rejects every other `..` segment +outright — leading, embedded or trailing. `../SomethingElse/file` does not fail when you +open it; it never resolves at all. The name is reserved by that rule, so an app cannot +create a sibling directory that collides with it. + +Nested paths inside it do work. `../SharedData/maps/uk.map` resolves, and +`mkdir("../SharedData/maps")` creates both levels. + ## Who uses it today | Path | Written by | Read by | | --- | --- | --- | | `../SharedData/stride.json` | `OutdoorStrideCalibrator::finalise()` ([`OutdoorStrideCalibrator.hpp:59`](../Libs/Header/SDK/Calibration/OutdoorStrideCalibrator.hpp)) | `StrideLut` ([`StrideLut.hpp:67`](../Libs/Header/SDK/Calibration/StrideLut.hpp)) | +| `../SharedData/stride.json.bak` | `OutdoorStrideCalibrator`, when the store will not parse ([`OutdoorStrideCalibrator.cpp:222-228`](../Libs/Source/Calibration/OutdoorStrideCalibrator.cpp)) | recovery only | | `../SharedData/stride_trace.csv` | Running app, when tracing is on ([`Running/.../Service.cpp:789`](../Examples/Apps/Running/Software/Libs/Sources/Service.cpp)) | diagnostic only | -| `../SharedData/stride_deleted.json` | Treadmill app, backing up the LUT before deleting it ([`Treadmill/.../Service.cpp:1177`](../Examples/Apps/Treadmill/Software/Libs/Sources/Service.cpp)) | recovery only | +| `../SharedData/stride_deleted.json` | Treadmill app, backing up the LUT before a user-initiated clear ([`Treadmill/.../Service.cpp:1177`](../Examples/Apps/Treadmill/Software/Libs/Sources/Service.cpp)) | recovery only | + +The last three are conditional — a watch that has never hit a corrupt store, never enabled +tracing and never cleared its calibration holds only `stride.json`. Expect the others to +appear, but do not require them. ## Create the directory before opening a file in it @@ -57,6 +76,17 @@ file is absent or will not open ([`StrideLut.cpp:163-175`](../Libs/Source/Calibration/StrideLut.cpp)). The caller gets an all-zero LUT and falls back to a default model. +One caveat, and it bites precisely where this rule tells you to stop looking. `IFile::open()` +returns a plain `bool`. File absent, file locked by another app's write, timed out waiting +for the volume, and too many files already open all arrive as the same `false`. A reader +that treats every failure as "nothing has written this yet" will quietly mistake *another +app is writing it right now* for *no data exists*, and fall back to defaults. + +`StrideLut::loadFromFile()` does exactly that, and for stride calibration the cost is one +session on a default model — a fair trade. If your shared file is larger or more expensive +to regenerate, decide deliberately whether a failed read means "empty" or "try again later". +The filesystem will not tell you which it was. + ## Share only what is shared App-specific data stays in the app's own root even when it is closely related to something @@ -72,17 +102,73 @@ Ask whether another app would be *right* to read the file, not just curious abou user's stride length is theirs and follows them between apps. A treadmill's calibration offset only means anything inside the model that produced it. +## Concurrent access + +Two apps writing the same shared file cannot corrupt it, and cannot both hold it open. + +Filesystem calls are serialised — every operation takes a mutex before touching the media, +so writes queue rather than interleave. On top of that the filesystem keeps a table of open +files: it refuses a second open of anything already open for writing, and refuses a +write-mode open of anything open at all. The loser simply gets `false` from `open()`. + +That table is small, and it is shared by the entire watch. The system log, the activity +recorder, settings, and every process of every running app draw on the same budget of ten +simultaneously open files. The eleventh open fails no matter who asks. An app that holds +several shared files open at once is competing with everything else the watch is doing, and +whether it fails depends on what that happens to be. Open what you need, use it, close it. + ## Expect interrupted writes -Apps are scheduled independently, and a watch can lose power mid-write. Write so that a torn -file is detectable: keep a backup copy before replacing (the Treadmill app does this with -`stride_deleted.json`), or use a format whose reader rejects a truncated file. The failure -to design against is a reader that silently accepts half a record. +Apps are scheduled independently, and a watch can lose power mid-write. + +The obvious defence is not available: **`rename()` will not replace an existing file.** +Renaming onto a name already in use fails, so the POSIX idiom of writing a temporary file +and renaming it over the original does not work here. You would have to remove the target +first, which reopens the very window the idiom exists to close. + +What the SDK does instead is rotate and fall back. `RecordingMarker` writes a temporary +file, moves the current good file aside to `.bak` (clearing any stale `.bak` first, because +of the rename rule above), renames the temporary into place, and has its reader fall back to +the `.bak` when the primary will not parse +([`RecordingMarker.cpp:82-140`](../Libs/Source/Fit/RecordingMarker.cpp)). A crash at any +single step leaves at least one intact copy. Copy that shape when the file matters. + +Be clear-eyed about what the calibration store does *not* do. `OutdoorStrideCalibrator` +overwrites `stride.json` in place, truncating on open, so a power loss mid-write leaves a +torn file and no backup. The `.bak` in the table above is written later — on a subsequent +load, once the store has already failed to parse. That is recovery evidence, not protection. +`stride_deleted.json` is not protection either: the Treadmill app writes it before an +explicit user-initiated *clear calibration*, never before a routine save. + +The failure to design against is a reader that silently accepts half a record. + +## Lifetime + +Shared files outlive the app that wrote them. `SharedData/` is a sibling of the app +directories rather than a child of any one of them, so removing an app cannot take it along. +That is the point — a user's stride calibration should survive reinstalling Running. + +Nothing ever collects the garbage. There is no owner, no reference count, and no screen that +lists shared files. A factory reset clears the directory along with everything else under +`Apps/`; short of that, whatever you write stays until some app deletes it. + +For a few kilobytes of calibration that costs nothing. For anything large — a downloaded map +set, say — it means leaving data on the watch with no owner and no way for the user to +reclaim the space. If your app writes something big, give the user a way to remove it, the +way Treadmill offers a clear-calibration action. ## In the simulator +Two differences, and the second one is the one that will catch you. + The simulator's filesystem root is `Output/` ([`Kernel.cpp:17`](../Libs/Source/Simulator/Kernel/Kernel.cpp)). It passes `..` through to the host filesystem instead of clamping it to that root, so `../SharedData/` lands beside `Output/` and not inside it. Apps behave correctly. The files are just not where you would first look for them. + +More importantly, the simulator does not enforce the whitelist at all. It concatenates its +prefix with your path and hands the result to the host filesystem, so `../anything/at/all` +works in the simulator and fails on the watch — and the simulator gives you no hint that you +have left the sandbox. If you invent a new shared path, check it against the rule in *The +path* above rather than against what the simulator accepts. From 2557a18c1dd8caef9ca952ef1ec1a3dcffd770a7 Mon Sep 17 00:00:00 2001 From: Ross Ryles Date: Sat, 8 Aug 2026 12:46:31 +0100 Subject: [PATCH 3/3] docs: correct rejected-path behaviour and the cleanup claim Two fixes to the previous commit, both from CodeRabbit. The rejected-path sentence was wrong. It said "../SomethingElse/file" does not fail when you open it, which is not what happens: the guard clears the path at construction, so fs.file() hands back a non-null object and open() returns false. Corrected, and extended with the part that actually matters -- the failure is indistinguishable from a missing file, and a null check on file() will not catch a bad path. The lifetime section claimed the user has no way to reclaim the space, which contradicts this page's own statement that the directory is visible and writable over USB mass storage. Reworded: no *automatic* cleanup, deletion only on an explicit request (an app's user-initiated clear, or a factory reset), and USB as a real but off-watch escape hatch rather than a substitute for an in-app one. --- Docs/shared-data.md | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/Docs/shared-data.md b/Docs/shared-data.md index 0eb68e93..466681d9 100644 --- a/Docs/shared-data.md +++ b/Docs/shared-data.md @@ -25,9 +25,14 @@ Paths are relative to the app's own root. The whole path, filename included, mus `../SharedData/` is the only path that may leave your app's root, and it is a whitelist rather than general parent traversal. The kernel resolves `../SharedData` and `../SharedData/` to the shared directory, and rejects every other `..` segment -outright — leading, embedded or trailing. `../SomethingElse/file` does not fail when you -open it; it never resolves at all. The name is reserved by that rule, so an app cannot -create a sibling directory that collides with it. +outright — leading, embedded or trailing. The name is reserved by that rule, so an app +cannot create a sibling directory that collides with it. + +`../SomethingElse/file` is rejected during path resolution, before any filesystem call is +made. Note where that leaves you: `fs.file()` still hands back a non-null object, and it is +`open()` that returns `false` — the same `false` you would get for a file that simply is not +there. A null check will not catch a bad path, and nothing distinguishes "you left the +sandbox" from "no such file". Nested paths inside it do work. `../SharedData/maps/uk.map` resolves, and `mkdir("../SharedData/maps")` creates both levels. @@ -148,14 +153,20 @@ Shared files outlive the app that wrote them. `SharedData/` is a sibling of the directories rather than a child of any one of them, so removing an app cannot take it along. That is the point — a user's stride calibration should survive reinstalling Running. -Nothing ever collects the garbage. There is no owner, no reference count, and no screen that -lists shared files. A factory reset clears the directory along with everything else under -`Apps/`; short of that, whatever you write stays until some app deletes it. +There is no automatic cleanup. No owner, no reference count, and no screen on the watch that +lists shared files. Deletion happens only when something explicitly asks for it — an app +acting on the user's instruction, the way Treadmill's clear-calibration action removes the +stride store, or a factory reset, which clears the directory along with everything else +under `Apps/`. Otherwise whatever you write stays. + +A user with a USB cable can delete shared files by hand, since the directory is visible over +mass storage. That is a reasonable escape hatch for a calibration gone bad. It is not a +substitute for being able to do it on the watch. -For a few kilobytes of calibration that costs nothing. For anything large — a downloaded map -set, say — it means leaving data on the watch with no owner and no way for the user to -reclaim the space. If your app writes something big, give the user a way to remove it, the -way Treadmill offers a clear-calibration action. +For a few kilobytes of calibration none of this costs anything. For anything large — a +downloaded map set, say — it means leaving data on the watch that nothing owns and no +on-device screen can remove. If your app writes something big, give the user a way to +remove it. ## In the simulator