Add Claude skills related to CTSM testing - #4193
Draft
samsrabin wants to merge 6 commits into
Draft
Conversation
How to build, run, and write a pFUnit test in CTSM, and what will bite you. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samsrabin
added
the
next
Test-driven development when a different agent writes the tests than writes the code. The arrangement: a dedicated test-writing agent runs before the implementing agent, writes and runs the tests, and commits them on their own. The implementer that follows is told it may not edit a test file — making the tests pass without touching them is the job. The supporting rules make that arrangement mean something. Tests are labelled by what kind of evidence they carry: a test of behaviour the change *alters* must fail at the test commit and pass at the end, and those two commits are the proof — anyone can check out the first one and watch it fail. A test of behaviour the change must *preserve* is green throughout, so red/green proves nothing about it and it owes a mutation instead: break the thing under test, confirm that specific assertion fails, restore it, report which assertion caught what. Test commits are standalone and never amended, because amending destroys exactly the commit a reviewer would check out. And at the end of a task, the diff of test files from the test commit to the final commit must be empty — that is the guarantee that no test was quietly reshaped to match the code that was eventually written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
How to add or change a CTSM system test, and who is allowed to run the suites. Covers what a system test can and cannot actually assess, and the resulting rule to prefer a unit test wherever the requirement can be reached that way; how to register a test and how test names work, since the full name is the key used for both baselines and for the expected-failures list; the mechanism for landing a test before the capability it tests exists, and the trap of doing that to a test that is also serving as a baseline reference; several testmod rules that fail silently when got wrong; which machine and compiler combinations catch which classes of bug; how to derive a wallclock limit from existing entries rather than guessing it; how to verify a change by building a case, including a cache refresh the build system does not perform for you when a source file is added; and when a commit may still be amended and when it may not. This skill also absorbs a rule about who runs the suites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What makes a test case actually pin something down, and how to make it readable. Two halves. **Fixtures that do not test what you think.** A test setup that holds some dimension constant, equal, or symmetric is blind to a whole class of bug along that dimension. Identical per-slot values hide indexing errors. Markers built as a product collide, so a transposition is invisible. A scale factor of one makes both legs of a scale-and-unscale operation the identity, so dropping either is invisible. A configuration already at capacity cannot demonstrate a cap. And the subtlest: a test setup that is a *fixed point* of the transformation being tested — where applying the transformation one step too far reproduces the test setup exactly — gives a guard against over-application zero coverage. **Tests a human can read.** Name a test for the condition it exercises and the code path that condition drives, never for an input value — a number in a name means nothing to a reader who has not memorised the threshold it is meant to exceed, and it goes stale when the threshold moves. Where two tests differ along one axis, name that axis in both. Open every test with a one-sentence summary that stands alone, contains no numbers, and says what the routine under test must do rather than what the test code does. Give the quantities a test depends on names instead of bare literals. State what an array's dimensions are. **Why it is separate from A:** the two trigger at different moments — A when structuring a task, B when actually writing a file — and separate descriptions are found more reliably than one merged one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samsrabin
force-pushed
the
ctsm-testing-skills
branch
from
September 2, 2026 23:25
4120ca1 to
3c4603c
Compare
samsrabin
commented
Sep 3, 2026
| today: these tests exist to constrain changes nobody has written yet, so "no current code path | ||
| does that" is a reason to expect the blind spot to matter later, not a reason to accept it. | ||
|
|
||
| ### Five shapes that go blind |
Member
Author
There was a problem hiding this comment.
- Review these to determine whether they're really worth including, or if they're not broadly applicable enough.
- Avoid "blind"
samsrabin
commented
Sep 3, 2026
| thickness", and never "sets up three layers and calls `DivideSnowLayers`". Name the | ||
| configuration the case exercises: that is usually the axis the file is organised around, and | ||
| it is not a number. | ||
| - **It contains no numbers.** No thicknesses, thresholds, layer indices or array bounds. In that |
samsrabin
marked this pull request as draft
September 3, 2026 16:11
samsrabin
removed
the
next
Member
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
Adds four "skills," which are folders of Markdown files with information relevant to certain tasks:
ctsm-unit-tests: How to build, run, and write a pFUnit test in CTSMwriting-tests-before-the-implementer: A different agent should write the tests than writes the code.ctsm-system-tests: How to add or change a CTSM system test; who is allowed to run the suitesdesigning-unit-test-cases: What makes a test actually do what it's supposed to; how to make it readablectsm-system-testscomes with a "hook":.claude/hooks/ctsm-suite-launch-guard.pyand the associated entry in.claude/settings.jsonshould make Claude ask if it tries to submit a system test suite, the idea being to prevent wasted compute time.Specific notes
Contributors other than yourself, if any:
CTSM issues resolved or otherwise addressed, if any:
Testing planned or performed, if any: None. However, these skills are built up from hard lessons learned during the last few weeks of co-development with Claude.
Requirements before merge: