From 23d75dfd86b7da8639bc5658c99617a0d15a46a8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 14:12:50 +0000 Subject: [PATCH 01/28] feat: make xcforge safe to share one Mac between agent sessions Route every xcodebuild call through one entry point that applies per-call options (DerivedData path, exact result bundle, extra args), holds an optional first-come, first-served build lock compatible with lockf(1), and warns on low disk or heavy swap. - simRecovery defaults to off; auto only reboots, erase must be asked for - never pkill every xcodebuild; hang snapshots match this run's bundle - WDA port cleanup only kills WDA runners on the configured local port - collision-free result bundle names; per-project last-result records used by build diagnose, test failures and test coverage - --isolated-sim runs on a throwaway simulator - xcforge lock status / build_lock_status Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01NVNakyntY4u9MvhsjbzRhZ --- CHANGELOG.md | 15 ++ Skills/xcforge/SKILL.md | 1 + Skills/xcforge/references/shared-mac.md | 47 ++++ .../Commands/Build/BuildCommand.swift | 23 +- .../Commands/Build/BuildTestCommand.swift | 20 +- .../Build/XcodebuildOptionGroup.swift | 50 ++++ .../Commands/Lock/LockCommand.swift | 62 +++++ .../Commands/Test/TestCommand.swift | 32 ++- .../Test/TestRerunFailedCommand.swift | 7 + Sources/XCForgeCLI/XCForgeCLI.swift | 2 +- Sources/XCForgeKit/Clients/AgentClient.swift | 37 ++- .../Persistence/DefaultsStore.swift | 44 +++- .../Persistence/LastResultStore.swift | 91 +++++++ Sources/XCForgeKit/ProviderRegistry.swift | 17 +- Sources/XCForgeKit/Support/BuildLock.swift | 236 ++++++++++++++++++ .../Support/DiagnosticSnapshot.swift | 11 +- Sources/XCForgeKit/Support/HangWatchdog.swift | 8 +- .../Support/IsolatedSimulator.swift | 91 +++++++ .../Support/SimulatorRecovery.swift | 40 ++- Sources/XCForgeKit/Support/Xcodebuild.swift | 165 ++++++++++++ .../Support/XcodebuildOptions.swift | 202 +++++++++++++++ Sources/XCForgeKit/Tools/BuildProvider.swift | 87 ++++--- Sources/XCForgeKit/Tools/TestProvider.swift | 205 +++++++++------ .../SharedMacSafetyTests.swift | 228 +++++++++++++++++ Tests/XCForgeKitTests/TestProviderTests.swift | 62 +++-- 25 files changed, 1622 insertions(+), 161 deletions(-) create mode 100644 Skills/xcforge/references/shared-mac.md create mode 100644 Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift create mode 100644 Sources/XCForgeCLI/Commands/Lock/LockCommand.swift create mode 100644 Sources/XCForgeKit/Persistence/LastResultStore.swift create mode 100644 Sources/XCForgeKit/Support/BuildLock.swift create mode 100644 Sources/XCForgeKit/Support/IsolatedSimulator.swift create mode 100644 Sources/XCForgeKit/Support/Xcodebuild.swift create mode 100644 Sources/XCForgeKit/Support/XcodebuildOptions.swift create mode 100644 Tests/XCForgeKitTests/SharedMacSafetyTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 535d256..5b7e51d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this ## [Unreleased] +### Added +- **Shared-Mac options for every xcodebuild call:** `--derived-data-path`, `--result-bundle-path`, repeatable `--xcodebuild-arg`, `--lock`, `--lock-wait` and `--min-free-gb` on `build compile|run|clean`, `build-test` and `test run|failures|list|rerun-failed`, with matching MCP arguments, env vars and `.xcforge.yaml` keys (`derivedDataPath`, `buildLock`, `artifactDir`, `minFreeGB`). See `Skills/xcforge/references/shared-mac.md` +- **Build lock with a first-come, first-served queue.** Takes the same `flock` as macOS `lockf(1)`, so it queues next to existing `lockf` wrappers. `xcforge lock status` / `build_lock_status` show the holder, the queue and wait times +- `--isolated-sim` / `isolatedSimulator` on `build-test`/`build_and_test` and `test run`/`test_sim`: run on a fresh simulator created for the run and deleted afterwards +- Preflight warnings for low disk and heavy swap before a build + +### Changed +- **`simRecovery` defaults to `off` everywhere** (was `auto` for `build_and_test`, `build-test` and `bless`). `auto` now only reboots; erasing a simulator needs the new `erase` mode. Invalid values are an error instead of silently becoming `auto`. `build-test` gains `--sim-recovery` +- Result bundles and diagnostic snapshots get collision-free names (`xcf----`), so two sessions starting in the same second no longer delete each other's bundles. `XCFORGE_ARTIFACT_DIR` / `artifactDir` moves them out of `/tmp` +- `build diagnose`, `test failures` and `test coverage` read this project's last recorded result bundle instead of the newest bundle any session left in `/tmp`. `test failures` without `--xcresult-path` reuses that bundle instead of re-running the whole suite + +### Fixed +- A hung build's fallback no longer runs `pkill -x xcodebuild`, which killed every session's build on the Mac. Hang snapshots sample the xcodebuild matched by this run's result bundle path instead of the newest xcodebuild on the system +- WDA port cleanup only kills WebDriverAgent runner processes, uses the port from `WDA_BASE_URL`, skips remote WDA hosts, and calls `lsof` at its real path (`/usr/sbin/lsof`) + ## [1.6.1] - 2026-05-19 ### Fixed diff --git a/Skills/xcforge/SKILL.md b/Skills/xcforge/SKILL.md index 4c3bc62..da62bb2 100644 --- a/Skills/xcforge/SKILL.md +++ b/Skills/xcforge/SKILL.md @@ -27,6 +27,7 @@ xcforge is a native Swift MCP server and CLI for iOS development. 109 MCP tools, |-----------|-----------| | **[Build Tools](references/build-tools.md)** | Building, running, cleaning, discovering projects, listing schemes | | **[Test Tools](references/test-tools.md)** | Running tests, analyzing failures, checking code coverage, build diagnostics | +| **[Shared Mac](references/shared-mac.md)** | DerivedData/result-bundle paths, extra xcodebuild args, build lock queue, isolated simulators, simulator recovery modes | | **[Simulator Tools](references/simulator-tools.md)** | Managing simulators — boot, shutdown, install, launch, clone, erase, delete, orientation, video recording, location, appearance, status bar | | **[Device Tools](references/device-tools.md)** | Physical iOS devices — list, info, install, uninstall, launch, terminate, list apps via devicectl | | **[SPM Tools](references/spm-tools.md)** | Swift packages — build, test, run, list dependencies, clean | diff --git a/Skills/xcforge/references/shared-mac.md b/Skills/xcforge/references/shared-mac.md new file mode 100644 index 0000000..2c06bf4 --- /dev/null +++ b/Skills/xcforge/references/shared-mac.md @@ -0,0 +1,47 @@ +# Sharing one Mac between sessions + +Every xcodebuild call xcforge makes goes through one entry point, which applies the options below. None of them are on by default; with nothing set, xcforge behaves as a single-user tool. + +| Option | CLI flag | MCP argument | Env var | `.xcforge.yaml` | +| --- | --- | --- | --- | --- | +| DerivedData folder | `--derived-data-path` | `derivedDataPath` | `XCFORGE_DERIVED_DATA_PATH` | `derivedDataPath` | +| Exact result bundle | `--result-bundle-path` | `resultBundlePath` | | | +| Extra xcodebuild args | `--xcodebuild-arg` (repeatable) | `xcodebuildArgs` | | | +| Build lock file | `--lock` | `buildLock` | `XCFORGE_BUILD_LOCK` | `buildLock` | +| Max wait for the lock | `--lock-wait` | `lockWaitSeconds` | `XCFORGE_LOCK_WAIT` | | +| Refuse below free disk (GB) | `--min-free-gb` | `minFreeGB` | `XCFORGE_MIN_FREE_GB` | `minFreeGB` | +| Artifact folder | | | `XCFORGE_ARTIFACT_DIR` | `artifactDir` | + +Precedence: flag/argument, then env var, then `.xcforge.yaml`. Relative yaml paths resolve against the yaml's folder. + +They apply to `build compile|run|clean`, `build-test`, `test run|failures|list|rerun-failed` and the MCP tools `build_sim`, `build_run_sim`, `build_compile`, `clean`, `test_sim`, `test_failures`, `test_coverage`, `build_and_diagnose`, `build_and_test`, `list_tests`, `bless`. + +## Build lock + +The lock is a plain `flock` on the file, the same lock macOS `lockf(1)` takes, so xcforge queues correctly next to shell wrappers such as `lockf /tmp/ios.lock xcodebuild ...`. xcforge waiters are served first-come, first-served through tickets in `.queue/`. The lock is held only while xcodebuild compiles or runs tests; `-showBuildSettings`, `-list` and WebDriverAgent are not locked. + +```bash +xcforge lock status --lock /tmp/ios.lock # holder, queue, wait times +xcforge build-test --lock /tmp/ios.lock # wait in line, then build and test +``` + +MCP: `build_lock_status`. + +## Isolated simulator + +`--isolated-sim` (CLI, `build-test` and `test run`) or `isolatedSimulator: true` (MCP, `build_and_test` and `test_sim`) creates a fresh simulator of the same model and OS, boots it, runs, then deletes it. Costs one cold boot; avoids "failed to launch / No such process" when another session shuts down or reconfigures the shared simulator. + +## Simulator recovery + +`simRecovery` / `--sim-recovery` defaults to `off` everywhere. `auto` reboots a simulator that isn't Booted. `erase` also erases it when a reboot didn't help. Unknown values are an error. + +## What xcforge never does on a shared Mac + +- Kill an xcodebuild it didn't start. Hang diagnostics sample the process matched by this run's result bundle path. +- Kill a non-WebDriverAgent process holding the WDA port. +- Read another session's result bundle. `build diagnose`, `test failures` and `test coverage` read this project's last recorded run (`~/.xcforge/last-results/`); `test failures` no longer re-runs the suite when a recorded test bundle exists. +- Delete a result bundle it didn't just create. Generated names are unique per process. + +## Preflight + +Before a build xcforge warns on stderr when the DerivedData volume has under 5 GB free or swap is over 85% used. `minFreeGB` turns low disk into a refusal. diff --git a/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift b/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift index 0efc7f4..51455b0 100644 --- a/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift +++ b/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift @@ -43,7 +43,14 @@ struct BuildCompile: AsyncParsableCommand { @Flag(help: "Emit the result as machine-readable JSON.") var json = false + @OptionGroup var xcodebuild: XcodebuildOptionGroup + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { let useJSON = shouldOutputJSON(flag: json) let env = Environment.live let configuration = await env.session.resolveConfiguration(self.configuration) @@ -96,7 +103,14 @@ struct BuildRun: AsyncParsableCommand { @Flag(help: "Emit the result as machine-readable JSON.") var json = false + @OptionGroup var xcodebuild: XcodebuildOptionGroup + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { let useJSON = shouldOutputJSON(flag: json) let env = Environment.live @@ -228,7 +242,7 @@ struct BuildDiagnose: AsyncParsableCommand { } else if let recent = await BuildTools.findRecentBuildXcresult() { path = recent } else { - let message = "No recent build results found in /tmp. Run `xcforge build` first." + let message = "No recorded build result for this project. Run `xcforge build compile` first." if useJSON { print(try WorkflowJSONRenderer.renderJSON(["error": message])) } else { @@ -269,7 +283,14 @@ struct BuildClean: AsyncParsableCommand { @Flag(help: "Emit the result as machine-readable JSON.") var json = false + @OptionGroup var xcodebuild: XcodebuildOptionGroup + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { let useJSON = shouldOutputJSON(flag: json) let execution = try await BuildTools.executeClean( diff --git a/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift b/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift index 9997430..d4b67fd 100644 --- a/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift +++ b/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift @@ -54,6 +54,15 @@ struct BuildTest: AsyncParsableCommand { ) var env: [String] = [] + @Option( + help: + "Simulator recovery: off (default), auto (reboot if not Booted), erase (also erase if a reboot didn't help)." + ) + var simRecovery: String = "off" + + @Flag(help: "Run on a fresh simulator of the same model and OS, deleted afterwards.") + var isolatedSim = false + @Flag(help: "Emit the result as machine-readable JSON.") var json = false @@ -69,7 +78,14 @@ struct BuildTest: AsyncParsableCommand { ) var gate = false + @OptionGroup var xcodebuild: XcodebuildOptionGroup + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { let useJSON = shouldOutputJSON(flag: json) || forMode == .agent let configuration = self.configuration ?? "Debug" let resolvedTimeout = timeoutSeconds.map { TimeInterval($0) } @@ -84,10 +100,12 @@ struct BuildTest: AsyncParsableCommand { coverage: coverage, long: long, diagnose: diagnose, + simRecovery: try SimRecoveryMode.parse(simRecovery), timeoutSeconds: resolvedTimeout, envEntries: env, gate: gate, - forMode: forMode + forMode: forMode, + isolatedSimulator: isolatedSim ) if useJSON { diff --git a/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift b/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift new file mode 100644 index 0000000..eb3ffe6 --- /dev/null +++ b/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift @@ -0,0 +1,50 @@ +import ArgumentParser +import Foundation +import XCForgeKit + +/// Flags shared by every command that runs xcodebuild. +struct XcodebuildOptionGroup: ParsableArguments { + @Option(help: "Pass -derivedDataPath to xcodebuild. Default: Xcode's shared DerivedData.") + var derivedDataPath: String? + + @Option(help: "Exact .xcresult path for the final build or test phase (replaced if it exists).") + var resultBundlePath: String? + + @Option( + name: .customLong("xcodebuild-arg"), parsing: .unconditionalSingleValue, + help: ArgumentHelp( + "Extra xcodebuild argument inserted before the action (repeatable), e.g. --xcodebuild-arg=-jobs.", + valueName: "ARG")) + var xcodebuildArgs: [String] = [] + + @Option( + name: .customLong("lock"), + help: + "Hold this lock file while xcodebuild runs (same flock as lockf(1)); waiters queue first-come, first-served. Also XCFORGE_BUILD_LOCK." + ) + var lockPath: String? + + @Option(name: .customLong("lock-wait"), help: "Seconds to wait for --lock before giving up. Default: 3600.") + var lockWait: Int? + + @Option(help: "Refuse to build below this much free disk (GB). Default: warn only.") + var minFreeGb: Double? + + var options: XcodebuildOptions { + XcodebuildOptions( + derivedDataPath: derivedDataPath.map { ($0 as NSString).expandingTildeInPath }, + resultBundlePath: resultBundlePath.map { ($0 as NSString).expandingTildeInPath }, + extraArgs: xcodebuildArgs, + lockPath: lockPath.map { ($0 as NSString).expandingTildeInPath }, + lockWaitSeconds: lockWait.map { TimeInterval($0) }, + minFreeGB: minFreeGb + ) + } + + /// Run `body` with these options applied to every xcodebuild call it makes. + func scoped(_ body: () async throws -> Void) async throws { + try await XcodebuildOptions.$current.withValue(options) { + try await body() + } + } +} diff --git a/Sources/XCForgeCLI/Commands/Lock/LockCommand.swift b/Sources/XCForgeCLI/Commands/Lock/LockCommand.swift new file mode 100644 index 0000000..fc8836b --- /dev/null +++ b/Sources/XCForgeCLI/Commands/Lock/LockCommand.swift @@ -0,0 +1,62 @@ +import ArgumentParser +import Foundation +import XCForgeKit + +struct Lock: AsyncParsableCommand { + static let configuration = CommandConfiguration( + commandName: "lock", + abstract: "Inspect the shared build lock.", + subcommands: [LockStatus.self], + defaultSubcommand: LockStatus.self + ) +} + +struct LockStatus: AsyncParsableCommand { + static let configuration = CommandConfiguration( + commandName: "status", + abstract: "Show who holds the build lock, who is queued, and for how long." + ) + + @Option(name: .customLong("lock"), help: "Lock file. Default: XCFORGE_BUILD_LOCK or .xcforge.yaml buildLock.") + var lockPath: String? + + @Flag(help: "Emit the result as machine-readable JSON.") + var json = false + + mutating func run() async throws { + let env = Environment.live + guard + let path = BuildLock.configuredPath( + explicit: lockPath.map { ($0 as NSString).expandingTildeInPath }, + cwd: env.currentDirectoryPath()) + else { + print("No build lock configured. Pass --lock, set XCFORGE_BUILD_LOCK, or add buildLock to .xcforge.yaml.") + throw ExitCode.validationFailure + } + let status = await BuildLock.status(path: path, env: env) + if shouldOutputJSON(flag: json) { + let now = Date() + let payload: [String: Any] = [ + "lock": status.lockPath, + "held": status.held, + "holder": status.holder.map { holder -> Any in + [ + "pid": Int(holder.pid), "command": holder.command, "cwd": holder.cwd, + "heldSeconds": Int(now.timeIntervalSince(holder.since)), + ] + } ?? NSNull(), + "externalHolders": status.externalHolders, + "queue": status.queue.map { entry -> [String: Any] in + [ + "pid": Int(entry.pid), "command": entry.command, + "waitingSeconds": Int(now.timeIntervalSince(entry.since)), + ] + }, + ] + let data = try JSONSerialization.data(withJSONObject: payload, options: [.prettyPrinted, .sortedKeys]) + print(String(decoding: data, as: UTF8.self)) + } else { + print(BuildLock.format(status)) + } + } +} diff --git a/Sources/XCForgeCLI/Commands/Test/TestCommand.swift b/Sources/XCForgeCLI/Commands/Test/TestCommand.swift index 45db7f9..4bb6412 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestCommand.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestCommand.swift @@ -82,9 +82,15 @@ struct TestRun: AsyncParsableCommand { @Flag(help: "Capture a diagnostic snapshot even when the test run succeeds.") var diagnose = false - @Option(help: "Simulator recovery mode (auto|off). Default: off.") + @Option( + help: + "Simulator recovery: off (default), auto (reboot if not Booted), erase (also erase if a reboot didn't help)." + ) var simRecovery: String = "off" + @Flag(help: "Run on a fresh simulator of the same model and OS, deleted afterwards.") + var isolatedSim = false + @Flag(help: "Emit the result as machine-readable JSON.") var json = false @@ -100,12 +106,19 @@ struct TestRun: AsyncParsableCommand { ) var gate = false + @OptionGroup var xcodebuild: XcodebuildOptionGroup + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { let useJSON = shouldOutputJSON(flag: json) || forMode == .agent let env = Environment.live let configuration = await env.session.resolveConfiguration(self.configuration) let resolvedTestplan = await env.session.resolveTestPlan(testplan) - let recoveryMode = SimRecoveryMode(rawValue: simRecovery) ?? .off + let recoveryMode = try SimRecoveryMode.parse(simRecovery) let execution = try await TestTools.executeTest( project: project, @@ -120,6 +133,7 @@ struct TestRun: AsyncParsableCommand { simRecovery: recoveryMode, gate: gate, forMode: forMode, + isolatedSimulator: isolatedSim, env: env ) @@ -159,7 +173,14 @@ struct TestFailures: AsyncParsableCommand { @Flag(help: "Emit the result as machine-readable JSON.") var json = false + @OptionGroup var xcodebuild: XcodebuildOptionGroup + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { let useJSON = shouldOutputJSON(flag: json) let result = try await TestTools.extractFailures( xcresultPath: xcresultPath, @@ -273,7 +294,14 @@ struct TestList: AsyncParsableCommand { @Flag(help: "Emit the result as machine-readable JSON.") var json = false + @OptionGroup var xcodebuild: XcodebuildOptionGroup + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { let useJSON = shouldOutputJSON(flag: json) let result = try await TestTools.executeListTests( project: project, diff --git a/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift b/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift index b733ca6..0bb7959 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift @@ -42,7 +42,14 @@ struct TestRerunFailed: AsyncParsableCommand { @Flag(help: "Apply known-failures gate to the rerun result.") var gate = false + @OptionGroup var xcodebuild: XcodebuildOptionGroup + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { let cwd = FileManager.default.currentDirectoryPath let repoRoot = RepoRoot.discover(from: cwd) ?? cwd guard let payload = LastFailuresStore.read(at: repoRoot) else { diff --git a/Sources/XCForgeCLI/XCForgeCLI.swift b/Sources/XCForgeCLI/XCForgeCLI.swift index a720303..b25e776 100644 --- a/Sources/XCForgeCLI/XCForgeCLI.swift +++ b/Sources/XCForgeCLI/XCForgeCLI.swift @@ -9,7 +9,7 @@ struct XCForgeCLI: AsyncParsableCommand { subcommands: [ Build.self, Test.self, BuildTest.self, Sim.self, Device.self, Diagnose.self, Defaults.self, Console.self, Git.self, Logs.self, Screenshot.self, UI.self, Accessibility.self, Plan.self, - SPM.self, Debug.self, Pose.self, Bless.self, WaitReady.self, Init.self, + SPM.self, Debug.self, Pose.self, Bless.self, WaitReady.self, Init.self, Lock.self, ] ) } diff --git a/Sources/XCForgeKit/Clients/AgentClient.swift b/Sources/XCForgeKit/Clients/AgentClient.swift index 20a47ba..9db57c0 100644 --- a/Sources/XCForgeKit/Clients/AgentClient.swift +++ b/Sources/XCForgeKit/Clients/AgentClient.swift @@ -185,20 +185,43 @@ public actor WDAClient { "\(backend.displayName) did not become ready within 10s after restart") } - /// Kill any process holding port 8100 to prevent binding conflicts. + /// Free the local WDA port, killing only WebDriverAgent runner processes listening on it. + /// The port comes from the configured base URL. Nothing is killed for a remote WDA + /// (a phone over the CoreDevice tunnel) or for an unrelated process on the port. private func cleanupPort8100() async { - if let result = try? await Shell.run("/usr/bin/lsof", arguments: ["-ti", ":8100"], timeout: 5), + guard let url = URL(string: baseURL), let host = url.host, + ["localhost", "127.0.0.1", "::1"].contains(host) + else { return } + let port = url.port ?? 8100 + guard + let result = try? await Shell.run( + "/usr/sbin/lsof", arguments: ["-ti", "tcp:\(port)", "-sTCP:LISTEN"], timeout: 5), result.succeeded, !result.stdout.isEmpty - { - for pidStr in result.stdout.split(separator: "\n") { - if let pid = Int32(pidStr.trimmingCharacters(in: .whitespaces)) { - kill(pid, SIGKILL) - } + else { return } + var killed = false + for pidStr in result.stdout.split(separator: "\n") { + guard let pid = Int32(pidStr.trimmingCharacters(in: .whitespaces)) else { continue } + let ps = try? await Shell.run( + "/bin/ps", arguments: ["-o", "comm=", "-p", String(pid)], timeout: 5) + let command = ps?.stdout ?? "" + guard Self.isWDARunnerCommand(command) else { + Log.warn("Port \(port) is held by pid \(pid) (\(command)), not a WDA runner — leaving it alone") + continue } + kill(pid, SIGKILL) + killed = true + } + if killed { try? await Task.sleep(nanoseconds: 300_000_000) // 0.3s for port release } } + /// True when a process command path looks like a WebDriverAgent test runner. + static func isWDARunnerCommand(_ command: String) -> Bool { + let name = (command.trimmingCharacters(in: .whitespacesAndNewlines) as NSString).lastPathComponent + return name.contains("WebDriverAgent") || name.contains("WDARunner") || name.hasSuffix("-Runner") + } + /// Kill any WDA process on port 8100 and terminate known runners. private func cleanupWDAProcesses(simulator: String) async { let _ = try? await Shell.xcrun( diff --git a/Sources/XCForgeKit/Persistence/DefaultsStore.swift b/Sources/XCForgeKit/Persistence/DefaultsStore.swift index e157e4d..c336abf 100644 --- a/Sources/XCForgeKit/Persistence/DefaultsStore.swift +++ b/Sources/XCForgeKit/Persistence/DefaultsStore.swift @@ -586,6 +586,14 @@ public enum RepoConfig { /// values to sticky session defaults. Defaults to `true` (legacy behavior) /// when the key is omitted. public var autoPromote: Bool? + /// `-derivedDataPath` for every xcodebuild call. Relative paths resolve against the yaml's folder. + public var derivedDataPath: String? + /// Lock file held around every xcodebuild call (same flock as `lockf(1)`). + public var buildLock: String? + /// Directory for generated result bundles and diagnostics. + public var artifactDir: String? + /// Refuse to build below this much free disk (GB). Absent means warn only. + public var minFreeGB: Double? public init( project: String? = nil, @@ -594,7 +602,11 @@ public enum RepoConfig { configuration: String? = nil, testPlan: String? = nil, testTimeout: Int? = nil, - autoPromote: Bool? = nil + autoPromote: Bool? = nil, + derivedDataPath: String? = nil, + buildLock: String? = nil, + artifactDir: String? = nil, + minFreeGB: Double? = nil ) { self.project = project self.scheme = scheme @@ -603,12 +615,17 @@ public enum RepoConfig { self.testPlan = testPlan self.testTimeout = testTimeout self.autoPromote = autoPromote + self.derivedDataPath = derivedDataPath + self.buildLock = buildLock + self.artifactDir = artifactDir + self.minFreeGB = minFreeGB } /// True when every field is nil (nothing to apply). public var isEmpty: Bool { project == nil && scheme == nil && simulator == nil && configuration == nil && testPlan == nil && testTimeout == nil && autoPromote == nil + && derivedDataPath == nil && buildLock == nil && artifactDir == nil && minFreeGB == nil } } @@ -661,7 +678,7 @@ public enum RepoConfig { let allowedKeys: Set = [ "project", "scheme", "simulator", "configuration", "testPlan", - "testTimeout", "autoPromote", + "testTimeout", "autoPromote", "derivedDataPath", "buildLock", "artifactDir", "minFreeGB", ] for key in dict.keys where !allowedKeys.contains(key) { Log.warn("\(RepoConfig.fileName): ignoring unknown key '\(key)'") @@ -706,6 +723,23 @@ public enum RepoConfig { } } + // Path keys resolve relative to the yaml's folder; they need not exist yet. + func resolvedPath(_ key: String) -> String? { + guard let raw = dict[key] else { return nil } + let expanded = (raw as NSString).expandingTildeInPath + if expanded.hasPrefix("/") { return expanded } + return ((configDir as NSString).appendingPathComponent(expanded) as NSString).standardizingPath + } + + var minFreeGB: Double? + if let raw = dict["minFreeGB"] { + if let parsed = Double(raw), parsed >= 0 { + minFreeGB = parsed + } else { + Log.warn("\(RepoConfig.fileName): ignoring non-numeric minFreeGB '\(raw)'") + } + } + let result = Values( project: project, scheme: dict["scheme"], @@ -713,7 +747,11 @@ public enum RepoConfig { configuration: dict["configuration"], testPlan: dict["testPlan"], testTimeout: testTimeout, - autoPromote: autoPromote + autoPromote: autoPromote, + derivedDataPath: resolvedPath("derivedDataPath"), + buildLock: resolvedPath("buildLock"), + artifactDir: resolvedPath("artifactDir"), + minFreeGB: minFreeGB ) return result.isEmpty ? nil : result } diff --git a/Sources/XCForgeKit/Persistence/LastResultStore.swift b/Sources/XCForgeKit/Persistence/LastResultStore.swift new file mode 100644 index 0000000..833cff4 --- /dev/null +++ b/Sources/XCForgeKit/Persistence/LastResultStore.swift @@ -0,0 +1,91 @@ +import Foundation + +/// Remembers the result bundle of the most recent build and test run per project, so +/// `build diagnose`, `test failures` and `test coverage` read this project's last run +/// instead of the newest bundle any session left in /tmp. +/// +/// One small JSON file per project under `~/.xcforge/last-results/` (or +/// `XCFORGE_LAST_RESULTS_DIR`). Writes are best-effort and never fail a build. +public enum LastResultStore { + public enum Kind: String, Codable, Sendable { + case build + case test + } + + public struct Record: Codable, Sendable, Equatable { + public var build: String? + public var test: String? + public var updatedAt: Date + } + + static func directory() -> String { + if let override = ProcessInfo.processInfo.environment["XCFORGE_LAST_RESULTS_DIR"], + !override.isEmpty + { + return (override as NSString).expandingTildeInPath + } + return NSHomeDirectory() + "/.xcforge/last-results" + } + + static func fileName(for project: String) -> String { + let standardized = (project as NSString).standardizingPath + let safe = standardized.map { ch -> Character in + ch.isLetter || ch.isNumber || ch == "." || ch == "-" ? ch : "_" + } + return String(safe) + ".json" + } + + static func filePath(for project: String) -> String { + (directory() as NSString).appendingPathComponent(fileName(for: project)) + } + + /// Record `bundlePath` as the latest result of `kind` for `project`. + public static func record(project: String, kind: Kind, bundlePath: String, now: Date = Date()) { + let path = filePath(for: project) + try? FileManager.default.createDirectory( + atPath: directory(), withIntermediateDirectories: true, attributes: nil) + var record = load(path) ?? Record(build: nil, test: nil, updatedAt: now) + switch kind { + case .build: record.build = bundlePath + case .test: record.test = bundlePath + } + record.updatedAt = now + let encoder = JSONEncoder() + encoder.outputFormatting = [.prettyPrinted, .sortedKeys] + encoder.dateEncodingStrategy = .iso8601 + guard let data = try? encoder.encode(record) else { return } + try? data.write(to: URL(fileURLWithPath: path), options: .atomic) + } + + /// The latest recorded bundle of `kind` for `project`, if it still exists on disk. + public static func latest(project: String, kind: Kind) -> String? { + guard let record = load(filePath(for: project)) else { return nil } + let path: String? + switch kind { + case .build: path = record.build + case .test: path = record.test + } + guard let path, FileManager.default.fileExists(atPath: path) else { return nil } + return path + } + + private static func load(_ path: String) -> Record? { + guard let data = FileManager.default.contents(atPath: path) else { return nil } + let decoder = JSONDecoder() + decoder.dateDecodingStrategy = .iso8601 + return try? decoder.decode(Record.self, from: data) + } + + /// Record the result bundle named in an xcodebuild argument list, if any. + static func recordFromArguments(_ args: [String]) { + guard let bundleIndex = args.firstIndex(of: "-resultBundlePath"), bundleIndex + 1 < args.count + else { return } + let bundle = args[bundleIndex + 1] + let projectIndex = args.firstIndex(of: "-workspace") ?? args.firstIndex(of: "-project") + guard let projectIndex, projectIndex + 1 < args.count else { return } + let project = args[projectIndex + 1] + let testActions: Set = ["test", "test-without-building"] + let kind: Kind = args.contains { testActions.contains($0) } ? .test : .build + record(project: project, kind: kind, bundlePath: bundle) + } +} diff --git a/Sources/XCForgeKit/ProviderRegistry.swift b/Sources/XCForgeKit/ProviderRegistry.swift index a6730d4..35b28c2 100644 --- a/Sources/XCForgeKit/ProviderRegistry.swift +++ b/Sources/XCForgeKit/ProviderRegistry.swift @@ -41,7 +41,7 @@ public enum ToolRegistry { } public static var allTools: [Tool] { - var tools = activeProviders.flatMap { $0.tools } + var tools = activeProviders.flatMap { $0.tools }.map(XcodebuildOptions.augment) // Always include the tool_groups management tool tools.append(toolGroupsTool) assert( @@ -56,11 +56,20 @@ public enum ToolRegistry { // Handle built-in tool_groups before checking providers if name == "tool_groups" { return handleToolGroups(args) } - for provider in activeProviders { - if let result = await provider.dispatch(name, args, env: env) { - return result + // Per-call xcodebuild options (derivedDataPath, buildLock, ...) ride in a task-local so + // every xcodebuild call made while serving this tool sees them. + let options = + XcodebuildOptions.mcpToolNames.contains(name) + ? XcodebuildOptions.fromMCPArguments(args) : XcodebuildOptions() + let result = await XcodebuildOptions.$current.withValue(options) { () async -> CallTool.Result? in + for provider in activeProviders { + if let result = await provider.dispatch(name, args, env: env) { + return result + } } + return nil } + if let result { return result } Log.warn("Unknown tool: \(name)") return .fail("Unknown tool: \(name)") } diff --git a/Sources/XCForgeKit/Support/BuildLock.swift b/Sources/XCForgeKit/Support/BuildLock.swift new file mode 100644 index 0000000..8101f34 --- /dev/null +++ b/Sources/XCForgeKit/Support/BuildLock.swift @@ -0,0 +1,236 @@ +import Foundation + +/// A first-come, first-served build lock shared between sessions on one Mac. +/// +/// The lock itself is a plain `flock(2)` on `path`, which is what macOS `lockf(1)` takes, +/// so xcforge queues correctly with shell wrappers such as `lockf /tmp/ios.lock xcodebuild ...`. +/// On top of that, xcforge waiters register a ticket in `.queue/` and only try the +/// lock when their ticket is the oldest live one, so they are served in arrival order. +/// The current xcforge holder is recorded in `.holder` for `xcforge lock status`. +public enum BuildLock { + + public struct Entry: Codable, Sendable, Equatable { + public let pid: Int32 + public let command: String + public let cwd: String + public let since: Date + } + + public struct Status: Sendable { + public let lockPath: String + /// True when someone holds the flock right now. + public let held: Bool + /// The xcforge process holding it, when the holder file is live. + public let holder: Entry? + /// Processes with the lock file open when the holder is not xcforge (e.g. a `lockf` wrapper). + public let externalHolders: [String] + /// xcforge waiters in queue order. + public let queue: [Entry] + } + + public struct Handle: Sendable { + let fd: Int32 + let holderPath: String + public let waitedSeconds: TimeInterval + + /// Release the lock. Safe to call once; the kernel also releases it if the process dies. + public func release() { + try? FileManager.default.removeItem(atPath: holderPath) + _ = flock(fd, LOCK_UN) + _ = close(fd) + } + } + + public enum LockError: Error, CustomStringConvertible { + case cannotOpen(String, Int32) + case timedOut(path: String, waited: TimeInterval, status: String) + + public var description: String { + switch self { + case .cannotOpen(let path, let code): + return "Cannot open build lock \(path): \(String(cString: strerror(code)))" + case .timedOut(let path, let waited, let status): + return "Gave up waiting \(Int(waited))s for build lock \(path).\n\(status)" + } + } + } + + static func queueDir(_ path: String) -> String { path + ".queue" } + static func holderPath(_ path: String) -> String { path + ".holder" } + + // MARK: - Acquire + + /// Wait in line for the lock, then take it. Throws `LockError.timedOut` after `maxWait`. + public static func acquire( + path: String, label: String, maxWait: TimeInterval, pollInterval: TimeInterval = 1.0 + ) async throws -> Handle { + let fm = FileManager.default + let parent = (path as NSString).deletingLastPathComponent + if !parent.isEmpty { + try? fm.createDirectory(atPath: parent, withIntermediateDirectories: true, attributes: nil) + } + let queue = queueDir(path) + try? fm.createDirectory(atPath: queue, withIntermediateDirectories: true, attributes: nil) + + let fd = open(path, O_RDWR | O_CREAT | O_CLOEXEC, 0o644) + guard fd >= 0 else { throw LockError.cannotOpen(path, errno) } + + let pid = ProcessInfo.processInfo.processIdentifier + let me = Entry(pid: pid, command: label, cwd: fm.currentDirectoryPath, since: Date()) + let nanos = UInt64(Date().timeIntervalSince1970 * 1_000_000_000) + let ticketName = String(format: "%020llu-%d.json", nanos, pid) + let ticketPath = (queue as NSString).appendingPathComponent(ticketName) + write(me, to: ticketPath) + + let start = Date() + var announced = false + while true { + let waiting = liveTickets(in: queue) + let first = waiting.first?.name + if first == nil || first == ticketName { + if flock(fd, LOCK_EX | LOCK_NB) == 0 { + try? fm.removeItem(atPath: ticketPath) + let waited = Date().timeIntervalSince(start) + let holder = Entry(pid: pid, command: label, cwd: me.cwd, since: Date()) + write(holder, to: holderPath(path)) + if announced { + Log.warn("Build lock \(path) acquired after \(Int(waited))s") + } + return Handle(fd: fd, holderPath: holderPath(path), waitedSeconds: waited) + } + } + if !announced { + let ahead = waiting.firstIndex { $0.name == ticketName } ?? 0 + Log.warn("Waiting for build lock \(path) (\(ahead) queued ahead)") + announced = true + } + let waited = Date().timeIntervalSince(start) + if waited >= maxWait || Task.isCancelled { + try? fm.removeItem(atPath: ticketPath) + _ = close(fd) + let snapshot = format(status(path: path, externalHolders: [])) + throw LockError.timedOut(path: path, waited: waited, status: snapshot) + } + try? await Task.sleep(nanoseconds: UInt64(pollInterval * 1_000_000_000)) + } + } + + // MARK: - Status + + /// Read the lock's holder and queue. `externalHolders` is filled in by the caller + /// (e.g. from `lsof`) since this function performs no subprocess calls. + public static func status(path: String, externalHolders: [String]) -> Status { + var held = false + let fd = open(path, O_RDONLY | O_CLOEXEC) + if fd >= 0 { + if flock(fd, LOCK_EX | LOCK_NB) == 0 { + _ = flock(fd, LOCK_UN) + } else { + held = true + } + _ = close(fd) + } + var holder: Entry? = read(holderPath(path)) + if let h = holder, !isAlive(h.pid) || !held { holder = nil } + let queue = liveTickets(in: queueDir(path)).map(\.entry) + return Status( + lockPath: path, held: held, holder: holder, + externalHolders: held && holder == nil ? externalHolders : [], queue: queue) + } + + /// Status including processes outside xcforge that have the lock file open (via `lsof`). + public static func status(path: String, env: Environment) async -> Status { + let first = status(path: path, externalHolders: []) + guard first.held, first.holder == nil else { return first } + var holders: [String] = [] + if let result = try? await env.shell.run( + "/usr/sbin/lsof", arguments: ["-F", "pc", path], timeout: 5) + { + var pid: String? + for line in result.stdout.split(separator: "\n") { + if line.hasPrefix("p") { + pid = String(line.dropFirst()) + } else if line.hasPrefix("c"), let p = pid { + holders.append("pid \(p) \(line.dropFirst())") + } + } + } + return status(path: path, externalHolders: holders) + } + + /// Resolve the lock path from explicit value, `XCFORGE_BUILD_LOCK` or `.xcforge.yaml`. + public static func configuredPath(explicit: String?, cwd: String) -> String? { + if let explicit, !explicit.isEmpty { return explicit } + return XcodebuildOptions.effective(cwd: cwd).lockPath + } + + /// Human-readable rendering of a status. + public static func format(_ status: Status, now: Date = Date()) -> String { + var lines = ["Build lock: \(status.lockPath)"] + if !status.held { + lines.append("Holder: none (free)") + } else if let h = status.holder { + let held = Int(now.timeIntervalSince(h.since)) + lines.append("Holder: xcforge pid \(h.pid), \(held)s — \(h.command)") + lines.append(" cwd: \(h.cwd)") + } else if !status.externalHolders.isEmpty { + lines.append("Holder: outside xcforge") + for p in status.externalHolders { lines.append(" \(p)") } + } else { + lines.append("Holder: outside xcforge (process unknown)") + } + if status.queue.isEmpty { + lines.append("Queue: empty") + } else { + lines.append("Queue (\(status.queue.count), first-come first-served):") + for (i, e) in status.queue.enumerated() { + let waited = Int(now.timeIntervalSince(e.since)) + lines.append(" \(i + 1). pid \(e.pid), waiting \(waited)s — \(e.command)") + } + } + return lines.joined(separator: "\n") + } + + // MARK: - Helpers + + struct Ticket { + let name: String + let entry: Entry + } + + /// Tickets of live waiters in arrival order. Tickets left by dead processes are removed. + static func liveTickets(in dir: String) -> [Ticket] { + let fm = FileManager.default + guard let names = try? fm.contentsOfDirectory(atPath: dir) else { return [] } + var tickets: [Ticket] = [] + for name in names.sorted() where name.hasSuffix(".json") { + let full = (dir as NSString).appendingPathComponent(name) + guard let entry: Entry = read(full) else { continue } + if isAlive(entry.pid) { + tickets.append(Ticket(name: name, entry: entry)) + } else { + try? fm.removeItem(atPath: full) + } + } + return tickets + } + + static func isAlive(_ pid: Int32) -> Bool { + if pid <= 0 { return false } + return kill(pid, 0) == 0 || errno == EPERM + } + + private static func write(_ entry: Entry, to path: String) { + let encoder = JSONEncoder() + encoder.dateEncodingStrategy = .iso8601 + guard let data = try? encoder.encode(entry) else { return } + try? data.write(to: URL(fileURLWithPath: path), options: .atomic) + } + + private static func read(_ path: String) -> Entry? { + guard let data = FileManager.default.contents(atPath: path) else { return nil } + let decoder = JSONDecoder() + decoder.dateDecodingStrategy = .iso8601 + return try? decoder.decode(Entry.self, from: data) + } +} diff --git a/Sources/XCForgeKit/Support/DiagnosticSnapshot.swift b/Sources/XCForgeKit/Support/DiagnosticSnapshot.swift index b3ef60b..cb5f83b 100644 --- a/Sources/XCForgeKit/Support/DiagnosticSnapshot.swift +++ b/Sources/XCForgeKit/Support/DiagnosticSnapshot.swift @@ -60,13 +60,20 @@ enum DiagnosticSnapshot { // MARK: - Capture - static func capture(udid: String?, snapshotPath: String, env: Environment) async -> Result { + /// - Parameter processMatch: A string unique to this invocation's command line (its result + /// bundle path). When set, only an xcodebuild started by this xcforge call is sampled, never + /// another session's build on the same Mac. + static func capture( + udid: String?, snapshotPath: String, processMatch: String? = nil, env: Environment + ) async -> Result { var sections: [String] = [] sections.append("=== xcforge diagnostic snapshot ===") sections.append("Timestamp: \(ISO8601DateFormatter().string(from: Date()))") + let pgrepArgs = processMatch.map { ["-n", "-f", NSRegularExpression.escapedPattern(for: $0)] } + ?? ["-n", "xcodebuild"] let pgrepResult = try? await env.shell.run( - "/usr/bin/pgrep", arguments: ["-n", "xcodebuild"], timeout: 5) + "/usr/bin/pgrep", arguments: pgrepArgs, timeout: 5) let pidStr = pgrepResult?.stdout .split(separator: "\n", omittingEmptySubsequences: true) diff --git a/Sources/XCForgeKit/Support/HangWatchdog.swift b/Sources/XCForgeKit/Support/HangWatchdog.swift index 83b28ae..d7bcebc 100644 --- a/Sources/XCForgeKit/Support/HangWatchdog.swift +++ b/Sources/XCForgeKit/Support/HangWatchdog.swift @@ -7,7 +7,10 @@ import Foundation final class HangWatchdog: Sendable { private let task: Task - init(udid: String?, snapshotPath: String, sampleAt: [TimeInterval], env: Environment) { + init( + udid: String?, snapshotPath: String, sampleAt: [TimeInterval], processMatch: String? = nil, + env: Environment + ) { task = Task { var last: DiagnosticSnapshot.Result? let start = Date() @@ -21,7 +24,8 @@ final class HangWatchdog: Sendable { } } guard !Task.isCancelled else { return last } - last = await DiagnosticSnapshot.capture(udid: udid, snapshotPath: snapshotPath, env: env) + last = await DiagnosticSnapshot.capture( + udid: udid, snapshotPath: snapshotPath, processMatch: processMatch, env: env) } return last } diff --git a/Sources/XCForgeKit/Support/IsolatedSimulator.swift b/Sources/XCForgeKit/Support/IsolatedSimulator.swift new file mode 100644 index 0000000..19b76ab --- /dev/null +++ b/Sources/XCForgeKit/Support/IsolatedSimulator.swift @@ -0,0 +1,91 @@ +import Foundation + +/// A throwaway simulator for one run, so sessions sharing a Mac never fight over one device. +/// +/// `create(from:)` makes a new simulator with the same device type and runtime as the source +/// (cloning needs the source shut down, which another session may not allow), boots it and +/// waits for boot to finish. `destroy` shuts it down and deletes it. +public enum IsolatedSimulator { + public struct Clone: Sendable, Equatable { + public let udid: String + public let name: String + } + + public struct IsolatedSimulatorError: Error, CustomStringConvertible { + public let description: String + } + + /// Find the device type and runtime of `source` (a UDID or a name) in `simctl list` JSON. + static func deviceSpec(source: String, listJSON: String) -> (deviceType: String, runtime: String)? { + guard let data = listJSON.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let devices = json["devices"] as? [String: Any] + else { return nil } + var nameMatch: (deviceType: String, runtime: String)? + for runtime in devices.keys.sorted(by: >) { + guard let list = devices[runtime] as? [[String: Any]] else { continue } + for device in list { + guard let type = device["deviceTypeIdentifier"] as? String else { continue } + if (device["udid"] as? String) == source { return (type, runtime) } + if nameMatch == nil, (device["name"] as? String) == source, + (device["isAvailable"] as? Bool) ?? true + { + nameMatch = (type, runtime) + } + } + } + return nameMatch + } + + /// Create and boot a fresh simulator matching `source`. + public static func create(from source: String, env: Environment) async throws -> Clone { + let list = try await env.shell.run( + "/usr/bin/xcrun", arguments: ["simctl", "list", "devices", "--json"], timeout: 30) + guard let spec = deviceSpec(source: source, listJSON: list.stdout) else { + throw IsolatedSimulatorError( + description: "Can't find simulator '\(source)' to copy for an isolated run.") + } + let name = "xcforge-isolated-\(UUID().uuidString.prefix(8).lowercased())" + let created = try await env.shell.run( + "/usr/bin/xcrun", arguments: ["simctl", "create", name, spec.deviceType, spec.runtime], + timeout: 60) + let udid = created.stdout.trimmingCharacters(in: .whitespacesAndNewlines) + guard created.succeeded, !udid.isEmpty else { + throw IsolatedSimulatorError( + description: "simctl create failed: \(created.stderr.isEmpty ? created.stdout : created.stderr)") + } + let clone = Clone(udid: udid, name: name) + let booted = try await env.shell.run( + "/usr/bin/xcrun", arguments: ["simctl", "bootstatus", udid, "-b"], timeout: 300) + guard booted.succeeded else { + await destroy(clone, env: env) + throw IsolatedSimulatorError( + description: "Isolated simulator \(name) failed to boot: \(booted.stderr)") + } + Log.warn("Isolated simulator \(name) (\(udid)) booted for this run") + return clone + } + + /// Shut down and delete an isolated simulator. Best-effort. + public static func destroy(_ clone: Clone, env: Environment) async { + _ = try? await env.shell.run( + "/usr/bin/xcrun", arguments: ["simctl", "shutdown", clone.udid], timeout: 30) + _ = try? await env.shell.run( + "/usr/bin/xcrun", arguments: ["simctl", "delete", clone.udid], timeout: 30) + } + + /// Run `body` against a fresh simulator copied from `source`, deleting it afterwards. + public static func with( + source: String, env: Environment, _ body: (String) async throws -> R + ) async throws -> R { + let clone = try await create(from: source, env: env) + do { + let result = try await body(clone.udid) + await destroy(clone, env: env) + return result + } catch { + await destroy(clone, env: env) + throw error + } + } +} diff --git a/Sources/XCForgeKit/Support/SimulatorRecovery.swift b/Sources/XCForgeKit/Support/SimulatorRecovery.swift index 29518eb..e8ef467 100644 --- a/Sources/XCForgeKit/Support/SimulatorRecovery.swift +++ b/Sources/XCForgeKit/Support/SimulatorRecovery.swift @@ -3,9 +3,31 @@ import Foundation // MARK: - Public types /// Controls whether SimulatorRecovery probes sim state before a build. +/// +/// - `off` (default): never touch the simulator. +/// - `auto`: if the simulator isn't Booted, shut it down and boot it again. Never erases. +/// - `erase`: like `auto`, then erase and reboot if a reboot didn't help. Destroys the +/// simulator's apps and data, so it only runs when asked for by name. public enum SimRecoveryMode: String, Sendable, CaseIterable { case auto + case erase case off + + /// Parse a user-supplied mode. Unknown values are an error, not a silent default. + public static func parse(_ raw: String?) throws -> SimRecoveryMode { + guard let raw, !raw.isEmpty else { return .off } + guard let mode = SimRecoveryMode(rawValue: raw.lowercased()) else { + throw SimRecoveryModeError(value: raw) + } + return mode + } +} + +public struct SimRecoveryModeError: Error, CustomStringConvertible { + public let value: String + public var description: String { + "Invalid simRecovery '\(value)'. Expected one of: off, auto, erase." + } } /// Result of a probeAndRecover call. @@ -27,7 +49,7 @@ public struct RecoveryOutcome: Sendable { /// Probes a simulator's state via `simctl list` and applies tiered recovery if unhealthy. /// /// Tier 1: shutdown + boot (non-destructive). -/// Tier 2: erase + boot (destructive) — only if Tier 1 re-probe fails. +/// Tier 2: erase + boot (destructive) — only in `.erase` mode, and only if Tier 1 re-probe fails. /// /// All operations are best-effort: failures surface in `RecoveryOutcome.failureReason` /// rather than being thrown. @@ -39,7 +61,10 @@ enum SimulatorRecovery { /// - udid: The simulator UDID to probe. /// - env: Execution environment. /// - Returns: A `RecoveryOutcome` describing what happened. Never throws. - static func probeAndRecover(udid: String, env: Environment) async -> RecoveryOutcome { + static func probeAndRecover( + udid: String, mode: SimRecoveryMode = .auto, env: Environment + ) async -> RecoveryOutcome { + if mode == .off { return .healthy } let probeResult = await probeSimState(udid: udid, env: env) switch probeResult { @@ -54,7 +79,8 @@ enum SimulatorRecovery { case .notBooted(let stateDetail): // Simulator is unhealthy — attempt tiered recovery. - let failureReason = await performTieredRecovery(udid: udid, env: env) + let failureReason = await performTieredRecovery( + udid: udid, allowErase: mode == .erase, env: env) return RecoveryOutcome( fired: true, reason: "sim_unhealthy", @@ -107,7 +133,9 @@ enum SimulatorRecovery { } /// Returns a failure description string on error, or nil on success. - private static func performTieredRecovery(udid: String, env: Environment) async -> String? { + private static func performTieredRecovery( + udid: String, allowErase: Bool, env: Environment + ) async -> String? { // Tier 1: non-destructive shutdown + boot _ = try? await env.shell.run( "/usr/bin/xcrun", arguments: ["simctl", "shutdown", udid], timeout: 15) @@ -118,6 +146,10 @@ enum SimulatorRecovery { if case .booted = tier1Probe { return nil } + guard allowErase else { + return "simulator did not reach Booted state after shutdown+boot" + + " (erase skipped; pass simRecovery: erase to allow it)" + } // Tier 2: destructive erase + boot let eraseResult = try? await env.shell.run( diff --git a/Sources/XCForgeKit/Support/Xcodebuild.swift b/Sources/XCForgeKit/Support/Xcodebuild.swift new file mode 100644 index 0000000..71223d9 --- /dev/null +++ b/Sources/XCForgeKit/Support/Xcodebuild.swift @@ -0,0 +1,165 @@ +import Foundation + +/// Single entry point for every `xcodebuild` invocation xcforge makes. +/// +/// Applies `XcodebuildOptions` (DerivedData path, extra arguments), checks free disk and +/// swap, and holds the shared build lock for the duration of the call when one is set. +public enum Xcodebuild { + public static let executable = "/usr/bin/xcodebuild" + + /// Actions and modes that accept `-derivedDataPath` and extra build arguments. + static let actionTokens: Set = [ + "build", "build-for-testing", "test", "test-without-building", "clean", "analyze", + "archive", "docbuild", "-showBuildSettings", "-enumerate-tests", + ] + + /// Run xcodebuild with the effective options applied. + public static func run( + _ arguments: [String], + workingDirectory: String? = nil, + environment: [String: String]? = nil, + timeout: TimeInterval, + env: Environment + ) async throws -> ShellResult { + let options = XcodebuildOptions.effective(cwd: env.currentDirectoryPath()) + let args = apply(options, to: arguments) + + if needsLock(args) { + try Preflight.check(options: options) + } + + var lock: BuildLock.Handle? + if let lockPath = options.lockPath, needsLock(args) { + lock = try await BuildLock.acquire( + path: lockPath, label: "xcodebuild " + summary(args), + maxWait: options.lockWaitSeconds ?? XcodebuildOptions.defaultLockWaitSeconds) + } + defer { lock?.release() } + + let result = try await env.shell.run( + executable, arguments: args, workingDirectory: workingDirectory, + environment: environment, timeout: timeout) + if result.exitCode != -2 { + LastResultStore.recordFromArguments(args) + } + return result + } + + /// True for invocations that build or run tests (as opposed to `-list` or `-version`). + static func isBuildInvocation(_ args: [String]) -> Bool { + args.contains { actionTokens.contains($0) } + } + + /// Actions that compile or run tests and therefore take the shared build lock. + /// `-showBuildSettings` alone does not, so it can run beside a build. + static let lockingActions: Set = [ + "build", "build-for-testing", "test", "test-without-building", "clean", "analyze", + "archive", "docbuild", + ] + + static func needsLock(_ args: [String]) -> Bool { + args.contains { lockingActions.contains($0) } + } + + /// Insert `-derivedDataPath` and extra arguments before the first action token. + /// Arguments the caller already passed are never duplicated. + static func apply(_ options: XcodebuildOptions, to arguments: [String]) -> [String] { + guard isBuildInvocation(arguments) else { return arguments } + var insert: [String] = [] + if let dd = options.derivedDataPath, !dd.isEmpty, !arguments.contains("-derivedDataPath"), + !arguments.contains("-xctestrun") + { + insert += ["-derivedDataPath", dd] + } + insert += options.extraArgs + guard !insert.isEmpty else { return arguments } + let index = arguments.firstIndex { actionTokens.contains($0) } ?? arguments.endIndex + var result = arguments + result.insert(contentsOf: insert, at: index) + return result + } + + /// Short description used for lock status: scheme and action. + static func summary(_ args: [String]) -> String { + var parts: [String] = [] + if let i = args.firstIndex(of: "-scheme"), i + 1 < args.count { + parts.append("-scheme \(args[i + 1])") + } + parts += args.filter { actionTokens.contains($0) } + return parts.joined(separator: " ") + } +} + +// MARK: - Preflight + +/// Disk and memory checks before a build. Warns by default; refuses only when the +/// caller set `minFreeGB`. +public enum Preflight { + public struct PreflightError: Error, CustomStringConvertible { + public let description: String + } + + /// Below this much free disk xcforge warns even without `minFreeGB`. + static let warnFreeGB: Double = 5 + /// Above this fraction of swap in use xcforge warns. + static let warnSwapFraction: Double = 0.85 + + static func check(options: XcodebuildOptions) throws { + let probePath = + options.derivedDataPath.map { existingAncestor(of: $0) } + ?? NSHomeDirectory() + "/Library/Developer" + if let freeGB = freeDiskGB(at: probePath) { + let rounded = String(format: "%.1f", freeGB) + if let minimum = options.minFreeGB, freeGB < minimum { + throw PreflightError( + description: + "Only \(rounded) GB free on the volume holding \(probePath); minFreeGB is \(minimum). " + + "Free disk space (e.g. delete old DerivedData) or lower minFreeGB.") + } + if freeGB < warnFreeGB { + Log.warn("Low disk: \(rounded) GB free on the volume holding \(probePath)") + } + } + if let swap = swapUsage(), swap.total > 0, swap.used / swap.total > warnSwapFraction { + let used = String(format: "%.1f", swap.used / 1_073_741_824) + let total = String(format: "%.1f", swap.total / 1_073_741_824) + Log.warn("Heavy swap: \(used) of \(total) GB in use; builds will be slow") + } + } + + /// Walk up to the nearest existing directory so a not-yet-created DerivedData path works. + static func existingAncestor(of path: String) -> String { + var current = (path as NSString).expandingTildeInPath + while !current.isEmpty && current != "/" && !FileManager.default.fileExists(atPath: current) { + current = (current as NSString).deletingLastPathComponent + } + return current.isEmpty ? "/" : current + } + + static func freeDiskGB(at path: String) -> Double? { + let url = URL(fileURLWithPath: path) + if let values = try? url.resourceValues(forKeys: [.volumeAvailableCapacityForImportantUsageKey]), + let bytes = values.volumeAvailableCapacityForImportantUsage + { + return Double(bytes) / 1_073_741_824 + } + if let attrs = try? FileManager.default.attributesOfFileSystem(forPath: path), + let bytes = attrs[.systemFreeSize] as? NSNumber + { + return bytes.doubleValue / 1_073_741_824 + } + return nil + } + + /// Swap usage in bytes from `vm.swapusage`, or nil when unavailable. + static func swapUsage() -> (used: Double, total: Double)? { + #if canImport(Darwin) + var usage = xsw_usage() + var size = MemoryLayout.size + guard sysctlbyname("vm.swapusage", &usage, &size, nil, 0) == 0 else { return nil } + return (Double(usage.xsu_used), Double(usage.xsu_total)) + #else + return nil + #endif + } +} diff --git a/Sources/XCForgeKit/Support/XcodebuildOptions.swift b/Sources/XCForgeKit/Support/XcodebuildOptions.swift new file mode 100644 index 0000000..4e3bfd5 --- /dev/null +++ b/Sources/XCForgeKit/Support/XcodebuildOptions.swift @@ -0,0 +1,202 @@ +import Foundation +import MCP + +/// Per-invocation knobs applied to every `xcodebuild` call xcforge makes. +/// +/// Values come from three layers, most specific first: +/// 1. The task-local `current` value, set by a CLI flag or MCP argument for one call. +/// 2. Environment variables (`XCFORGE_DERIVED_DATA_PATH`, `XCFORGE_BUILD_LOCK`, ...). +/// 3. `.xcforge.yaml` keys (`derivedDataPath`, `buildLock`, `artifactDir`, `minFreeGB`). +/// +/// Every field is optional: with nothing set, xcforge behaves exactly as before. +public struct XcodebuildOptions: Sendable, Equatable { + /// Passed as `-derivedDataPath` to every build/test action. + public var derivedDataPath: String? + /// Exact `-resultBundlePath` for the final phase of a build or test run. + public var resultBundlePath: String? + /// Extra arguments inserted before the xcodebuild action (flags, `KEY=VALUE` settings). + public var extraArgs: [String] + /// File to `flock` around every xcodebuild call. Same lock as macOS `lockf(1)`. + public var lockPath: String? + /// Maximum seconds to wait in the lock queue before giving up. + public var lockWaitSeconds: TimeInterval? + /// Refuse to start a build when the DerivedData volume has less free space than this. + public var minFreeGB: Double? + /// Directory for xcforge-generated result bundles and diagnostics. + public var artifactDir: String? + + public init( + derivedDataPath: String? = nil, + resultBundlePath: String? = nil, + extraArgs: [String] = [], + lockPath: String? = nil, + lockWaitSeconds: TimeInterval? = nil, + minFreeGB: Double? = nil, + artifactDir: String? = nil + ) { + self.derivedDataPath = derivedDataPath + self.resultBundlePath = resultBundlePath + self.extraArgs = extraArgs + self.lockPath = lockPath + self.lockWaitSeconds = lockWaitSeconds + self.minFreeGB = minFreeGB + self.artifactDir = artifactDir + } + + /// Options set for the current call (CLI flags or MCP arguments). + @TaskLocal public static var current = XcodebuildOptions() + + /// Default wait in the lock queue: one hour. + public static let defaultLockWaitSeconds: TimeInterval = 3600 + + /// Merge the task-local value over environment variables over `.xcforge.yaml`. + public static func effective( + cwd: String = FileManager.default.currentDirectoryPath, + environment: [String: String] = ProcessInfo.processInfo.environment + ) -> XcodebuildOptions { + let explicit = current + let repo = RepoConfig.discover(from: cwd) + + func nonEmpty(_ s: String?) -> String? { + guard let s, !s.isEmpty else { return nil } + return s + } + + var merged = XcodebuildOptions() + merged.derivedDataPath = + explicit.derivedDataPath ?? nonEmpty(environment["XCFORGE_DERIVED_DATA_PATH"]) + ?? repo?.derivedDataPath + merged.resultBundlePath = explicit.resultBundlePath + merged.lockPath = + explicit.lockPath ?? nonEmpty(environment["XCFORGE_BUILD_LOCK"]) ?? repo?.buildLock + merged.lockWaitSeconds = + explicit.lockWaitSeconds ?? environment["XCFORGE_LOCK_WAIT"].flatMap { TimeInterval($0) } + merged.minFreeGB = + explicit.minFreeGB ?? environment["XCFORGE_MIN_FREE_GB"].flatMap { Double($0) } + ?? repo?.minFreeGB + merged.artifactDir = + explicit.artifactDir ?? nonEmpty(environment["XCFORGE_ARTIFACT_DIR"]) ?? repo?.artifactDir + merged.extraArgs = explicit.extraArgs + return merged + } + + /// Directory for generated artifacts. Defaults to `/tmp`. + public static func artifactDirectory() -> String { + let dir = effective().artifactDir ?? "/tmp" + if dir != "/tmp" { + try? FileManager.default.createDirectory( + atPath: dir, withIntermediateDirectories: true, attributes: nil) + } + return dir + } + + /// A collision-free artifact path: `/xcf----.`. + /// Two runs started in the same second (or by two processes) never share a path. + public static func uniqueArtifactPath(prefix: String, extension ext: String) -> String { + let ts = Int(Date().timeIntervalSince1970) + let pid = ProcessInfo.processInfo.processIdentifier + let rand = UUID().uuidString.prefix(6).lowercased() + return "\(artifactDirectory())/xcf-\(prefix)-\(ts)-\(pid)-\(rand).\(ext)" + } + + /// Result bundle path for one phase. When the caller fixed `resultBundlePath`, the + /// final phases (`build`, `test`, `fail`) use it exactly and earlier phases get a + /// sibling path, so both bundles survive. + public static func resultBundlePath(prefix: String) -> String { + if let fixed = current.resultBundlePath, !fixed.isEmpty { + let finalPrefixes: Set = ["build", "test", "fail"] + if finalPrefixes.contains(prefix) { return fixed } + let base = fixed.hasSuffix(".xcresult") ? String(fixed.dropLast(".xcresult".count)) : fixed + return "\(base)-\(prefix).xcresult" + } + return uniqueArtifactPath(prefix: prefix, extension: "xcresult") + } + + // MARK: - MCP plumbing + + /// Tools whose xcodebuild calls honour these options. Their schemas advertise the keys. + public static let mcpToolNames: Set = [ + "build_sim", "build_run_sim", "build_compile", "clean", + "test_sim", "test_failures", "test_coverage", "build_and_diagnose", "build_and_test", + "list_tests", "bless", + ] + + /// JSON-schema properties added to each tool in `mcpToolNames`. + public static let mcpSchemaProperties: [String: Value] = [ + "derivedDataPath": .object([ + "type": .string("string"), + "description": .string("Passed to xcodebuild as -derivedDataPath. Default: Xcode's shared DerivedData."), + ]), + "resultBundlePath": .object([ + "type": .string("string"), + "description": .string( + "Exact .xcresult path for the final build or test phase. Replaced if it already exists."), + ]), + "xcodebuildArgs": .object([ + "type": .string("array"), + "items": .object(["type": .string("string")]), + "description": .string( + "Extra xcodebuild arguments inserted before the action, e.g. [\"-jobs\", \"6\"] or [\"SWIFT_TREAT_WARNINGS_AS_ERRORS=NO\"]." + ), + ]), + "buildLock": .object([ + "type": .string("string"), + "description": .string( + "Lock file to hold while xcodebuild runs. Same flock as macOS lockf(1); waiters queue first-come, first-served." + ), + ]), + "lockWaitSeconds": .object([ + "type": .string("integer"), + "description": .string("Give up after waiting this long for buildLock. Default: 3600."), + ]), + "minFreeGB": .object([ + "type": .string("number"), + "description": .string( + "Refuse to build when the DerivedData volume has less free space than this. Default: warn only."), + ]), + ] + + /// Parse the option keys out of an MCP argument dictionary. + public static func fromMCPArguments(_ args: [String: Value]?) -> XcodebuildOptions { + guard let args else { return XcodebuildOptions() } + func number(_ v: Value?) -> Double? { + guard let v else { return nil } + if let d = v.doubleValue { return d } + if let i = v.intValue { return Double(i) } + if let s = v.stringValue { return Double(s) } + return nil + } + let extra = args["xcodebuildArgs"]?.arrayValue?.compactMap { $0.stringValue } ?? [] + return XcodebuildOptions( + derivedDataPath: args["derivedDataPath"]?.stringValue, + resultBundlePath: args["resultBundlePath"]?.stringValue, + extraArgs: extra, + lockPath: args["buildLock"]?.stringValue, + lockWaitSeconds: number(args["lockWaitSeconds"]), + minFreeGB: number(args["minFreeGB"]), + artifactDir: nil + ) + } + + /// Return `tool` with the option properties merged into its input schema. + static func augment(_ tool: Tool) -> Tool { + guard mcpToolNames.contains(tool.name), + case .object(var schema) = tool.inputSchema + else { return tool } + var properties = schema["properties"]?.objectValue ?? [:] + for (key, value) in mcpSchemaProperties where properties[key] == nil { + properties[key] = value + } + schema["properties"] = .object(properties) + return Tool( + name: tool.name, + title: tool.title, + description: tool.description, + inputSchema: .object(schema), + annotations: tool.annotations, + outputSchema: tool.outputSchema, + icons: tool.icons, + _meta: tool._meta + ) + } +} diff --git a/Sources/XCForgeKit/Tools/BuildProvider.swift b/Sources/XCForgeKit/Tools/BuildProvider.swift index 43ef0d6..58e1192 100644 --- a/Sources/XCForgeKit/Tools/BuildProvider.swift +++ b/Sources/XCForgeKit/Tools/BuildProvider.swift @@ -292,6 +292,21 @@ public enum BuildTools { ]), ]) ), + Tool( + name: "build_lock_status", + description: """ + Show who holds the shared build lock and who is queued behind it, with wait times. The lock path comes from the 'lock' argument, XCFORGE_BUILD_LOCK, or .xcforge.yaml buildLock. + """, + inputSchema: .object([ + "type": .string("object"), + "properties": .object([ + "lock": .object([ + "type": .string("string"), + "description": .string("Lock file path. Defaults to the configured build lock."), + ]) + ]), + ]) + ), Tool( name: "discover_projects", description: "Find Xcode projects and workspaces in a directory.", @@ -386,9 +401,9 @@ public enum BuildTools { let buildTimeout = await TestTools.resolveTestTimeout(long: long, env: env) let snapshotPath = TestTools.diagnosticSnapshotPath() let watchdog = HangWatchdog( - udid: resolvedSimulator, snapshotPath: snapshotPath, sampleAt: [60, 120], env: env) - let result = try await env.shell.run( - "/usr/bin/xcodebuild", arguments: buildArgs, timeout: buildTimeout) + udid: resolvedSimulator, snapshotPath: snapshotPath, sampleAt: [60, 120], + processMatch: resultPath, env: env) + let result = try await Xcodebuild.run(buildArgs, timeout: buildTimeout, env: env) watchdog.cancel() let watchdogCapture = await watchdog.latestResult let diagResult: DiagnosticSnapshot.Result? @@ -397,14 +412,14 @@ public enum BuildTools { diagResult = captured } else { diagResult = await DiagnosticSnapshot.capture( - udid: resolvedSimulator, snapshotPath: snapshotPath, env: env) + udid: resolvedSimulator, snapshotPath: snapshotPath, processMatch: resultPath, env: env) } } else if diagnose { if let captured = watchdogCapture { diagResult = captured } else { diagResult = await DiagnosticSnapshot.capture( - udid: resolvedSimulator, snapshotPath: snapshotPath, env: env) + udid: resolvedSimulator, snapshotPath: snapshotPath, processMatch: resultPath, env: env) } } else { diagResult = watchdogCapture @@ -724,16 +739,22 @@ public enum BuildTools { return issue.message } - /// Find the most recent xcresult bundle from a build in /tmp. - public static func findRecentBuildXcresult(env: Environment = .live) async -> String? { + /// Find this project's most recent build result bundle. Falls back to the newest + /// xcforge build bundle in the artifact directory when no project can be resolved. + public static func findRecentBuildXcresult(project: String? = nil, env: Environment = .live) + async -> String? + { + if let resolved = try? await env.session.resolveProject(project) { + return LastResultStore.latest(project: resolved, kind: .build) + } + let dir = XcodebuildOptions.artifactDirectory() do { - let result = try await env.shell.run( - "/bin/ls", arguments: ["-1t", "/tmp/"], timeout: 5) + let result = try await env.shell.run("/bin/ls", arguments: ["-1t", dir], timeout: 5) guard result.succeeded else { return nil } let candidates = result.stdout.split(separator: "\n") .map(String.init) .filter { $0.hasPrefix("xcf-build-") && $0.hasSuffix(".xcresult") } - return candidates.first.map { "/tmp/\($0)" } + return candidates.first.map { (dir as NSString).appendingPathComponent($0) } } catch { return nil } @@ -777,11 +798,8 @@ public enum BuildTools { let projectFlag = isWorkspace ? "-workspace" : "-project" do { - let result = try await env.shell.run( - "/usr/bin/xcodebuild", - arguments: [ - projectFlag, project, "-scheme", scheme, "clean", - ], timeout: 60) + let result = try await Xcodebuild.run( + [projectFlag, project, "-scheme", scheme, "clean"], timeout: 60, env: env) return result.succeeded ? .ok("Clean succeeded") : .fail("Clean failed: \(result.stderr)") } catch { return .fail("Clean error: \(error)") @@ -877,10 +895,10 @@ public enum BuildTools { let buildTimeout = await TestTools.resolveTestTimeout(long: input.long ?? false, env: env) let buildSnapshotPath = TestTools.diagnosticSnapshotPath() let buildWatchdog = HangWatchdog( - udid: udid, snapshotPath: buildSnapshotPath, sampleAt: [60, 120], env: env) - async let buildTask = env.shell.run("/usr/bin/xcodebuild", arguments: buildArgs, timeout: buildTimeout) - async let settingsTask = env.shell.run( - "/usr/bin/xcodebuild", arguments: settingsArgs, timeout: 30) + udid: udid, snapshotPath: buildSnapshotPath, sampleAt: [60, 120], processMatch: resultPath, + env: env) + async let buildTask = Xcodebuild.run(buildArgs, timeout: buildTimeout, env: env) + async let settingsTask = Xcodebuild.run(settingsArgs, timeout: 30, env: env) async let bootTask = env.shell.run( "/usr/bin/xcrun", arguments: ["simctl", "boot", udid], timeout: 60) async let openTask = env.shell.run( @@ -900,7 +918,7 @@ public enum BuildTools { buildDiagResult = captured } else if input.diagnose ?? false { buildDiagResult = await DiagnosticSnapshot.capture( - udid: udid, snapshotPath: buildSnapshotPath, env: env) + udid: udid, snapshotPath: buildSnapshotPath, processMatch: resultPath, env: env) } else { buildDiagResult = nil } @@ -996,7 +1014,9 @@ public enum BuildTools { // Tier 3: Search DerivedData if appPath == nil { - let ddPath = NSHomeDirectory() + "/Library/Developer/Xcode/DerivedData" + let ddPath = + XcodebuildOptions.effective(cwd: env.currentDirectoryPath()).derivedDataPath + ?? NSHomeDirectory() + "/Library/Developer/Xcode/DerivedData" let findResult = try? await env.shell.run( "/usr/bin/find", arguments: [ @@ -1132,15 +1152,14 @@ public enum BuildTools { let projectFlag = isWorkspace ? "-workspace" : "-project" let destination = await AutoDetect.buildDestination(simulator) - let result = try await env.shell.run( - "/usr/bin/xcodebuild", - arguments: [ + let result = try await Xcodebuild.run( + [ projectFlag, project, "-scheme", scheme, "-configuration", configuration, "-destination", destination, "-showBuildSettings", - ], timeout: 30) + ], timeout: 30, env: env) guard result.succeeded else { let details = result.stderr.isEmpty ? result.stdout : result.stderr @@ -1259,11 +1278,8 @@ public enum BuildTools { let isWorkspace = resolvedProject.hasSuffix(".xcworkspace") let projectFlag = isWorkspace ? "-workspace" : "-project" - let result = try await Shell.run( - "/usr/bin/xcodebuild", - arguments: [ - projectFlag, resolvedProject, "-scheme", resolvedScheme, "clean", - ], timeout: 60) + let result = try await Xcodebuild.run( + [projectFlag, resolvedProject, "-scheme", resolvedScheme, "clean"], timeout: 60, env: env) return CleanExecution( succeeded: result.succeeded, @@ -1349,6 +1365,16 @@ public enum BuildTools { } extension BuildTools: ToolProvider { + static func buildLockStatus(_ args: [String: Value]?, env: Environment) async -> CallTool.Result { + let explicit = args?["lock"]?.stringValue + guard let path = BuildLock.configuredPath(explicit: explicit, cwd: env.currentDirectoryPath()) + else { + return .fail("No build lock configured. Pass 'lock', set XCFORGE_BUILD_LOCK, or add buildLock to .xcforge.yaml.") + } + let status = await BuildLock.status(path: path, env: env) + return .ok(BuildLock.format(status)) + } + public static func dispatch(_ name: String, _ args: [String: Value]?, env: Environment) async -> CallTool.Result? { @@ -1359,6 +1385,7 @@ extension BuildTools: ToolProvider { case "clean": return await clean(args, env: env) case "discover_projects": return await discoverProjects(args, env: env) case "list_schemes": return await listSchemes(args, env: env) + case "build_lock_status": return await buildLockStatus(args, env: env) default: return nil } } diff --git a/Sources/XCForgeKit/Tools/TestProvider.swift b/Sources/XCForgeKit/Tools/TestProvider.swift index e69af81..9c9b813 100644 --- a/Sources/XCForgeKit/Tools/TestProvider.swift +++ b/Sources/XCForgeKit/Tools/TestProvider.swift @@ -280,6 +280,7 @@ public enum TestTools { var simRecovery: String? var `for`: String? var gate: Bool? + var isolatedSimulator: Bool? } struct TestFailuresInput: Decodable { @@ -322,6 +323,7 @@ public enum TestTools { var env: [String]? var `for`: String? var gate: Bool? + var isolatedSimulator: Bool? } /// Error for malformed --env / env: entries. @@ -491,9 +493,17 @@ public enum TestTools { "simRecovery": .object([ "type": .string("string"), "description": .string( - "Simulator recovery mode: 'off' (default) skips probe; 'auto' probes bootstatus before run and erases+reboots if unhealthy." + "Simulator recovery before the run: 'off' (default) leaves the simulator alone; 'auto' reboots it if it isn't Booted;" + + " 'erase' also erases it if a reboot didn't help (destroys its apps and data)." + ), + "enum": .array([.string("off"), .string("auto"), .string("erase")]), + ]), + "isolatedSimulator": .object([ + "type": .string("boolean"), + "description": .string( + "Run on a fresh simulator of the same model and OS, created for this run and deleted afterwards." + + " Avoids collisions with other sessions on a shared simulator. Costs one cold boot." ), - "enum": .array([.string("auto"), .string("off")]), ]), "for": .object([ "type": .string("string"), @@ -678,10 +688,17 @@ public enum TestTools { "simRecovery": .object([ "type": .string("string"), "description": .string( - "Simulator recovery mode: 'auto' (default) probes sim state before run and applies tiered recovery if unhealthy; 'off' skips probe." - + " Result includes recoveryAttempts: Int, recoveryReason: String?, and simHealthCheckDetail: String? fields." + "Simulator recovery before the run: 'off' (default) leaves the simulator alone; 'auto' reboots it if it isn't Booted;" + + " 'erase' also erases it if a reboot didn't help (destroys its apps and data)." + ), + "enum": .array([.string("off"), .string("auto"), .string("erase")]), + ]), + "isolatedSimulator": .object([ + "type": .string("boolean"), + "description": .string( + "Run on a fresh simulator of the same model and OS, created for this run and deleted afterwards." + + " Avoids collisions with other sessions on a shared simulator. Costs one cold boot." ), - "enum": .array([.string("auto"), .string("off")]), ]), "timeoutSeconds": .object([ "type": .string("integer"), @@ -772,10 +789,9 @@ public enum TestTools { // MARK: - Shared helpers - /// Generate a unique xcresult path + /// Generate a unique xcresult path (or the caller's fixed `resultBundlePath`). static func xcresultPath(prefix: String) -> String { - let ts = Int(Date().timeIntervalSince1970) - return "/tmp/xcf-\(prefix)-\(ts).xcresult" + XcodebuildOptions.resultBundlePath(prefix: prefix) } /// Resolve the test watchdog timeout via the session actor. @@ -789,22 +805,18 @@ public enum TestTools { } static func diagnosticSnapshotPath() -> String { - let pid = ProcessInfo.processInfo.processIdentifier - let ts = Int(Date().timeIntervalSince1970) - return "/tmp/xcf-diag-\(pid)-\(ts).txt" + XcodebuildOptions.uniqueArtifactPath(prefix: "diag", extension: "txt") } private static func resolvedDiagResult( result: ShellResult, diagnose: Bool, watchdog: HangWatchdog, - udid: String?, snapshotPath: String, env: Environment + udid: String?, snapshotPath: String, processMatch: String? = nil, env: Environment ) async -> DiagnosticSnapshot.Result? { let watchdogCapture = await watchdog.latestResult - if result.exitCode == -1 { - if let captured = watchdogCapture { return captured } - return await DiagnosticSnapshot.capture(udid: udid, snapshotPath: snapshotPath, env: env) - } else if diagnose { + if result.exitCode == -1 || diagnose { if let captured = watchdogCapture { return captured } - return await DiagnosticSnapshot.capture(udid: udid, snapshotPath: snapshotPath, env: env) + return await DiagnosticSnapshot.capture( + udid: udid, snapshotPath: snapshotPath, processMatch: processMatch, env: env) } else { return watchdogCapture } @@ -815,30 +827,33 @@ public enum TestTools { return "\nDiagnostic snapshot: \(result.filePath)\nSummary: \(result.summaryLine)" } - /// Find the most recent xcresult bundle in /tmp that has coverage data. - /// Returns the path if one is found and contains valid coverage, nil otherwise. - private static func findRecentCoverageXcresult(env: Environment) async -> String? { + /// Find this project's most recent test bundle that has coverage data. + /// Uses the bundle recorded for the project's last test run; when the project can't be + /// resolved, falls back to the newest xcforge bundles in the artifact directory. + private static func findRecentCoverageXcresult(project: String? = nil, env: Environment) async + -> String? + { + if let resolved = try? await env.session.resolveProject(project) { + guard let recorded = LastResultStore.latest(project: resolved, kind: .test) else { + return nil + } + return await parseCoverage(recorded, env: env) != nil ? recorded : nil + } + let dir = XcodebuildOptions.artifactDirectory() do { - // List xcresult bundles created by xcforge (test or cov prefixed) - let result = try await env.shell.run( - "/bin/ls", - arguments: ["-1t", "/tmp/"], - timeout: 5 - ) + let result = try await env.shell.run("/bin/ls", arguments: ["-1t", dir], timeout: 5) guard result.succeeded else { return nil } let candidates = result.stdout.split(separator: "\n") .map(String.init) - .filter { $0.hasPrefix("xcf-") && $0.hasSuffix(".xcresult") } - - // Try each candidate (sorted newest-first by ls -t) for valid coverage - for candidate in candidates.prefix(5) { - let path = "/tmp/\(candidate)" - if let _ = await parseCoverage(path, env: env) { + .filter { $0.hasPrefix("xcf-test-") && $0.hasSuffix(".xcresult") } + for candidate in candidates.prefix(3) { + let path = (dir as NSString).appendingPathComponent(candidate) + if await parseCoverage(path, env: env) != nil { return path } } } catch { - // Ignore — fallback to running tests + // Ignore — caller reports that no coverage is available. } return nil } @@ -1076,14 +1091,14 @@ public enum TestTools { let timeout = await resolveTestTimeout(long: long, env: env) let snapshotPath = diagnosticSnapshotPath() - let watchdog = HangWatchdog(udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], env: env) - let result = try await env.shell.run( - "/usr/bin/xcodebuild", arguments: args, - environment: childEnvironment, timeout: timeout) + let watchdog = HangWatchdog( + udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) + let result = try await Xcodebuild.run( + args, environment: childEnvironment, timeout: timeout, env: env) watchdog.cancel() let diagResult = await resolvedDiagResult( result: result, diagnose: diagnose, watchdog: watchdog, - udid: udid, snapshotPath: snapshotPath, env: env) + udid: udid, snapshotPath: snapshotPath, processMatch: resultPath, env: env) return (result, resultPath, diagResult) } @@ -1109,12 +1124,13 @@ public enum TestTools { let timeout = await resolveTestTimeout(long: long, env: env) let snapshotPath = diagnosticSnapshotPath() - let watchdog = HangWatchdog(udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], env: env) - let result = try await env.shell.run("/usr/bin/xcodebuild", arguments: args, timeout: timeout) + let watchdog = HangWatchdog( + udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) + let result = try await Xcodebuild.run(args, timeout: timeout, env: env) watchdog.cancel() let diagResult = await resolvedDiagResult( result: result, diagnose: diagnose, watchdog: watchdog, - udid: udid, snapshotPath: snapshotPath, env: env) + udid: udid, snapshotPath: snapshotPath, processMatch: resultPath, env: env) return (result, resultPath, diagResult) } @@ -1147,14 +1163,13 @@ public enum TestTools { } let snapshotPath = diagnosticSnapshotPath() let watchdog = HangWatchdog( - udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], env: env) - let result = try await env.shell.run( - "/usr/bin/xcodebuild", arguments: args, - environment: childEnvironment, timeout: timeout) + udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) + let result = try await Xcodebuild.run( + args, environment: childEnvironment, timeout: timeout, env: env) watchdog.cancel() let diagResult = await resolvedDiagResult( result: result, diagnose: diagnose, watchdog: watchdog, - udid: udid, snapshotPath: snapshotPath, env: env) + udid: udid, snapshotPath: snapshotPath, processMatch: resultPath, env: env) return (result, resultPath, diagResult) } @@ -1196,14 +1211,13 @@ public enum TestTools { } let snapshotPath = diagnosticSnapshotPath() let watchdog = HangWatchdog( - udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], env: env) - let result = try await env.shell.run( - "/usr/bin/xcodebuild", arguments: args, - environment: childEnvironment, timeout: timeout) + udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) + let result = try await Xcodebuild.run( + args, environment: childEnvironment, timeout: timeout, env: env) watchdog.cancel() let diagResult = await resolvedDiagResult( result: result, diagnose: diagnose, watchdog: watchdog, - udid: udid, snapshotPath: snapshotPath, env: env) + udid: udid, snapshotPath: snapshotPath, processMatch: resultPath, env: env) return (result, resultPath, diagResult) } @@ -1433,7 +1447,7 @@ public enum TestTools { private static func exportFailureAttachments(_ xcresultPath: String, env: Environment) async -> [( test: String, path: String )] { - let outputDir = "/tmp/xcf-attachments-\(Int(Date().timeIntervalSince1970))" + let outputDir = XcodebuildOptions.uniqueArtifactPath(prefix: "attachments", extension: "d") do { _ = try await env.shell.run("/bin/mkdir", arguments: ["-p", outputDir], timeout: 5) } catch { @@ -1583,8 +1597,19 @@ public enum TestTools { simRecovery: SimRecoveryMode = .off, gate: Bool = false, forMode: OutputAudience = .human, + isolatedSimulator: Bool = false, env: Environment = .live ) async throws -> TestExecution { + if isolatedSimulator { + let source = try await env.session.resolveSimulator(simulator) + return try await IsolatedSimulator.with(source: source, env: env) { udid in + try await executeTest( + project: project, scheme: scheme, simulator: udid, configuration: configuration, + testplan: testplan, filter: filter, filterIDs: filterIDs, coverage: coverage, + long: long, diagnose: diagnose, simRecovery: simRecovery, gate: gate, + forMode: forMode, isolatedSimulator: false, env: env) + } + } let resolvedProject = try await env.session.resolveProject(project) let resolvedScheme = try await env.session.resolveScheme(scheme, project: resolvedProject) let resolvedSimulator = try await env.session.resolveSimulator(simulator) @@ -1614,8 +1639,9 @@ public enum TestTools { } // Run sim recovery if requested - if simRecovery == .auto { - let outcome = await SimulatorRecovery.probeAndRecover(udid: udidForDest, env: env) + if simRecovery != .off { + let outcome = await SimulatorRecovery.probeAndRecover( + udid: udidForDest, mode: simRecovery, env: env) if outcome.fired { Log.warn("Simulator recovery fired: \(outcome.reason ?? "unknown")") if let failure = outcome.failureReason { @@ -1917,6 +1943,11 @@ public enum TestTools { let resolvedPath: String if let provided = xcresultPath { resolvedPath = provided + } else if let resolvedProject = try? await env.session.resolveProject(project), + let recorded = LastResultStore.latest(project: resolvedProject, kind: .test) + { + // Read this project's last test run instead of re-running the whole suite. + resolvedPath = recorded } else { let resolvedProject = try await env.session.resolveProject(project) let resolvedScheme = try await env.session.resolveScheme(scheme, project: resolvedProject) @@ -1966,8 +1997,8 @@ public enum TestTools { let resolvedPath: String if let provided = xcresultPath { resolvedPath = provided - } else if let recent = await findRecentCoverageXcresult(env: env) { - // Reuse a recent xcresult that already has coverage data + } else if let recent = await findRecentCoverageXcresult(project: project, env: env) { + // Reuse this project's last test run when it has coverage data resolvedPath = recent } else { // No coverage data available — fail fast instead of silently running the entire test suite @@ -2112,13 +2143,24 @@ public enum TestTools { coverage: Bool = false, long: Bool = false, diagnose: Bool = false, - simRecovery: SimRecoveryMode = .auto, + simRecovery: SimRecoveryMode = .off, timeoutSeconds: TimeInterval? = nil, envEntries: [String] = [], gate: Bool = false, forMode: OutputAudience = .human, + isolatedSimulator: Bool = false, env: Environment = .live ) async throws -> BuildAndTestResult { + if isolatedSimulator { + let source = try await env.session.resolveSimulator(simulator) + return try await IsolatedSimulator.with(source: source, env: env) { udid in + try await executeBuildAndTest( + project: project, scheme: scheme, simulator: udid, configuration: configuration, + testplan: testplan, filter: filter, coverage: coverage, long: long, diagnose: diagnose, + simRecovery: simRecovery, timeoutSeconds: timeoutSeconds, envEntries: envEntries, + gate: gate, forMode: forMode, isolatedSimulator: false, env: env) + } + } let cwd = env.currentDirectoryPath() let repoRoot = AutoDetect.repoRoot(from: cwd) ?? cwd let childEnvironment = try buildTestRunnerEnvironment( @@ -2141,8 +2183,9 @@ public enum TestTools { var simHealthCheckDetail: String? // Pre-build simulator health probe (only when auto) - if simRecovery == .auto { - let outcome = await SimulatorRecovery.probeAndRecover(udid: udidForDest, env: env) + if simRecovery != .off { + let outcome = await SimulatorRecovery.probeAndRecover( + udid: udidForDest, mode: simRecovery, env: env) simHealthCheckDetail = outcome.healthCheckDetail if outcome.fired { recoveryAttempts += 1 @@ -2211,7 +2254,8 @@ public enum TestTools { await killXcodebuildTree(pid: xcodebuildPid, env: env) // Run sim recovery before retry - let retryOutcome = await SimulatorRecovery.probeAndRecover(udid: udidForDest, env: env) + let retryOutcome = await SimulatorRecovery.probeAndRecover( + udid: udidForDest, mode: simRecovery, env: env) recoveryAttempts += 1 recoveryReason = verdict?.rawValue ?? retryOutcome.reason recoveryFailureReason = retryOutcome.failureReason @@ -2318,7 +2362,8 @@ public enum TestTools { let xcodebuildPid = extractXcodebuildPid(from: testDiagResult) await killXcodebuildTree(pid: xcodebuildPid, env: env) - let retryOutcome = await SimulatorRecovery.probeAndRecover(udid: udidForDest, env: env) + let retryOutcome = await SimulatorRecovery.probeAndRecover( + udid: udidForDest, mode: simRecovery, env: env) if recoveryReason == nil { recoveryReason = verdict?.rawValue ?? retryOutcome.reason } if recoveryFailureReason == nil { recoveryFailureReason = retryOutcome.failureReason } if simHealthCheckDetail == nil { simHealthCheckDetail = retryOutcome.healthCheckDetail } @@ -2420,14 +2465,11 @@ public enum TestTools { /// Kill the xcodebuild process tree by PID. private static func killXcodebuildTree(pid: String?, env: Environment) async { - if let pid { - _ = try? await env.shell.run("/bin/kill", arguments: ["-9", pid], timeout: 5) - _ = try? await env.shell.run("/usr/bin/pkill", arguments: ["-P", pid], timeout: 5) - } else { - // Fallback: kill exact process name match (safe, does not match substrings) - _ = try? await env.shell.run( - "/usr/bin/pkill", arguments: ["-x", "xcodebuild"], timeout: 5) - } + // The PID comes from a snapshot matched on this run's own result bundle path, so it is + // never another session's xcodebuild. Without one there is nothing safe to kill. + guard let pid else { return } + _ = try? await env.shell.run("/usr/bin/pkill", arguments: ["-9", "-P", pid], timeout: 5) + _ = try? await env.shell.run("/bin/kill", arguments: ["-9", pid], timeout: 5) } /// Extract up to 10 slowest test cases from test-results JSON, sorted by duration descending. @@ -2646,16 +2688,15 @@ public enum TestTools { } // xcodebuild test -enumerate-tests builds for testing then lists tests without running them - let enumerateResult = try await env.shell.run( - "/usr/bin/xcodebuild", - arguments: [ + let enumerateResult = try await Xcodebuild.run( + [ projectFlag, resolvedProject, "-scheme", resolvedScheme, "-destination", destination, "-skipMacroValidation", "-enumerate-tests", "test", - ], timeout: 1800 + ], timeout: 1800, env: env ) guard enumerateResult.succeeded || !enumerateResult.stdout.isEmpty else { @@ -2808,10 +2849,10 @@ public enum TestTools { case .success(let input): // Resolve simRecovery mode (default: off for test_sim) let recoveryMode: SimRecoveryMode - if let raw = input.simRecovery, let mode = SimRecoveryMode(rawValue: raw) { - recoveryMode = mode - } else { - recoveryMode = .off + do { + recoveryMode = try SimRecoveryMode.parse(input.simRecovery) + } catch { + return .fail("\(error)") } // Build preamble: testplan visibility + filter/testplan conflict warning @@ -2844,6 +2885,7 @@ public enum TestTools { simRecovery: recoveryMode, gate: input.gate ?? false, forMode: forMode, + isolatedSimulator: input.isolatedSimulator ?? false, env: env ) @@ -3177,10 +3219,10 @@ public enum TestTools { let configuration = await env.session.resolveConfiguration(input.configuration) let testplan = await env.session.resolveTestPlan(input.testplan) let recoveryMode: SimRecoveryMode - if let raw = input.simRecovery, let mode = SimRecoveryMode(rawValue: raw) { - recoveryMode = mode - } else { - recoveryMode = .auto + do { + recoveryMode = try SimRecoveryMode.parse(input.simRecovery) + } catch { + return .fail("\(error)") } let forMode: OutputAudience = (input.for?.lowercased() == "agent") ? .agent : .human @@ -3200,6 +3242,7 @@ public enum TestTools { envEntries: input.env ?? [], gate: input.gate ?? false, forMode: forMode, + isolatedSimulator: input.isolatedSimulator ?? false, env: env ) if forMode == .agent { diff --git a/Tests/XCForgeKitTests/SharedMacSafetyTests.swift b/Tests/XCForgeKitTests/SharedMacSafetyTests.swift new file mode 100644 index 0000000..7fb96c7 --- /dev/null +++ b/Tests/XCForgeKitTests/SharedMacSafetyTests.swift @@ -0,0 +1,228 @@ +import Foundation +import MCP +import Testing + +@testable import XCForgeKit + +@Suite("Shared-Mac safety: xcodebuild options, build lock, last results", .serialized) +struct SharedMacSafetyTests { + + private func makeTempDir() -> String { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("xcforge-shared-\(UUID().uuidString)", isDirectory: true) + try! FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + return dir.path + } + + // MARK: - Xcodebuild.apply + + @Test("apply inserts -derivedDataPath and extra args before the action") + func applyInsertsBeforeAction() { + let options = XcodebuildOptions(derivedDataPath: "/dd", extraArgs: ["-jobs", "6"]) + let args = ["-project", "A.xcodeproj", "-scheme", "A", "build", "COMPILATION_CACHE_ENABLE_CACHING=YES"] + let result = Xcodebuild.apply(options, to: args) + #expect( + result == [ + "-project", "A.xcodeproj", "-scheme", "A", "-derivedDataPath", "/dd", "-jobs", "6", "build", + "COMPILATION_CACHE_ENABLE_CACHING=YES", + ]) + } + + @Test("apply leaves -list and an existing -derivedDataPath alone") + func applySkipsNonBuildAndDuplicates() { + let options = XcodebuildOptions(derivedDataPath: "/dd") + let list = ["-project", "A.xcodeproj", "-list", "-json"] + #expect(Xcodebuild.apply(options, to: list) == list) + let explicit = ["-derivedDataPath", "/mine", "build"] + #expect(Xcodebuild.apply(options, to: explicit) == explicit) + } + + @Test("-showBuildSettings gets DerivedData but does not take the lock") + func showBuildSettingsNoLock() { + let args = ["-scheme", "A", "-showBuildSettings"] + #expect(Xcodebuild.isBuildInvocation(args)) + #expect(!Xcodebuild.needsLock(args)) + #expect(Xcodebuild.needsLock(["-scheme", "A", "test-without-building"])) + } + + // MARK: - Result bundle paths + + @Test("generated result bundle paths never collide") + func uniquePaths() { + let a = XcodebuildOptions.resultBundlePath(prefix: "test") + let b = XcodebuildOptions.resultBundlePath(prefix: "test") + #expect(a != b) + #expect(a.hasSuffix(".xcresult")) + } + + @Test("a fixed resultBundlePath is used for the final phase; earlier phases get a sibling") + func fixedResultBundlePath() { + let options = XcodebuildOptions(resultBundlePath: "/out/run.xcresult") + XcodebuildOptions.$current.withValue(options) { + #expect(XcodebuildOptions.resultBundlePath(prefix: "test") == "/out/run.xcresult") + #expect( + XcodebuildOptions.resultBundlePath(prefix: "build-for-testing") + == "/out/run-build-for-testing.xcresult") + } + } + + // MARK: - Effective options + + @Test("task-local values win over environment variables") + func effectivePrecedence() { + let env = ["XCFORGE_DERIVED_DATA_PATH": "/env-dd", "XCFORGE_BUILD_LOCK": "/env.lock"] + let fromEnv = XcodebuildOptions.effective(cwd: "/", environment: env) + #expect(fromEnv.derivedDataPath == "/env-dd") + #expect(fromEnv.lockPath == "/env.lock") + + XcodebuildOptions.$current.withValue(XcodebuildOptions(derivedDataPath: "/explicit")) { + let merged = XcodebuildOptions.effective(cwd: "/", environment: env) + #expect(merged.derivedDataPath == "/explicit") + #expect(merged.lockPath == "/env.lock") + } + } + + @Test(".xcforge.yaml path keys resolve relative to the yaml's folder") + func repoConfigPathKeys() throws { + let dir = makeTempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let yaml = """ + derivedDataPath: build/dd + buildLock: /tmp/ios.lock + minFreeGB: 5 + """ + let path = (dir as NSString).appendingPathComponent(".xcforge.yaml") + try yaml.write(toFile: path, atomically: true, encoding: .utf8) + let values = RepoConfig.load(from: path, configDir: dir) + #expect(values?.derivedDataPath == (dir as NSString).appendingPathComponent("build/dd")) + #expect(values?.buildLock == "/tmp/ios.lock") + #expect(values?.minFreeGB == 5) + } + + // MARK: - MCP plumbing + + @Test("MCP arguments parse into options, and build tools advertise them") + func mcpOptions() { + let args: [String: Value] = [ + "derivedDataPath": .string("/dd"), + "xcodebuildArgs": .array([.string("-jobs"), .string("6")]), + "buildLock": .string("/tmp/l"), + "lockWaitSeconds": .int(30), + ] + let options = XcodebuildOptions.fromMCPArguments(args) + #expect(options.derivedDataPath == "/dd") + #expect(options.extraArgs == ["-jobs", "6"]) + #expect(options.lockPath == "/tmp/l") + #expect(options.lockWaitSeconds == 30) + + let tool = ToolRegistry.allTools.first { $0.name == "build_and_test" } + let properties = tool?.inputSchema.objectValue?["properties"]?.objectValue + #expect(properties?["derivedDataPath"] != nil) + #expect(properties?["buildLock"] != nil) + } + + // MARK: - Build lock + + @Test("lock is acquired, reported, and released") + func lockLifecycle() async throws { + let dir = makeTempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let path = (dir as NSString).appendingPathComponent("build.lock") + + let handle = try await BuildLock.acquire(path: path, label: "test build", maxWait: 5) + let held = BuildLock.status(path: path, externalHolders: []) + #expect(held.held) + #expect(held.holder?.pid == ProcessInfo.processInfo.processIdentifier) + #expect(held.holder?.command == "test build") + #expect(BuildLock.format(held).contains("test build")) + + handle.release() + let free = BuildLock.status(path: path, externalHolders: []) + #expect(!free.held) + #expect(free.holder == nil) + } + + @Test("a second waiter times out with the holder named in the error") + func lockTimeout() async throws { + let dir = makeTempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let path = (dir as NSString).appendingPathComponent("build.lock") + + let handle = try await BuildLock.acquire(path: path, label: "first", maxWait: 5) + defer { handle.release() } + do { + _ = try await BuildLock.acquire(path: path, label: "second", maxWait: 0.5, pollInterval: 0.1) + Issue.record("second acquire should have timed out") + } catch let error as BuildLock.LockError { + #expect("\(error)".contains("first")) + } + // The timed-out waiter's ticket is gone. + #expect(BuildLock.liveTickets(in: BuildLock.queueDir(path)).isEmpty) + } + + @Test("tickets from dead processes are pruned") + func deadTicketsPruned() throws { + let dir = makeTempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let queue = (dir as NSString).appendingPathComponent("q") + try FileManager.default.createDirectory(atPath: queue, withIntermediateDirectories: true) + let ticket = """ + {"pid": 2147483000, "command": "ghost", "cwd": "/", "since": "2026-01-01T00:00:00Z"} + """ + try ticket.write( + toFile: (queue as NSString).appendingPathComponent("00000000000000000001-1.json"), + atomically: true, encoding: .utf8) + #expect(BuildLock.liveTickets(in: queue).isEmpty) + #expect(try FileManager.default.contentsOfDirectory(atPath: queue).isEmpty) + } + + // MARK: - Last results + + @Test("last result is recorded per project from xcodebuild arguments") + func lastResultRecorded() throws { + let dir = makeTempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let bundle = (dir as NSString).appendingPathComponent("t.xcresult") + try FileManager.default.createDirectory(atPath: bundle, withIntermediateDirectories: true) + let project = (dir as NSString).appendingPathComponent("App-\(UUID().uuidString).xcodeproj") + + LastResultStore.recordFromArguments([ + "-project", project, "-scheme", "App", "-resultBundlePath", bundle, "test-without-building", + ]) + #expect(LastResultStore.latest(project: project, kind: .test) == bundle) + #expect(LastResultStore.latest(project: project, kind: .build) == nil) + try? FileManager.default.removeItem(atPath: LastResultStore.filePath(for: project)) + } + + // MARK: - Isolated simulator + + @Test("deviceSpec finds the device type and runtime by UDID or name") + func deviceSpecLookup() { + let json = """ + {"devices": { + "com.apple.CoreSimulator.SimRuntime.iOS-18-0": [ + {"udid": "AAA", "name": "iPhone 16", "deviceTypeIdentifier": "type.iPhone-16", "isAvailable": true} + ], + "com.apple.CoreSimulator.SimRuntime.iOS-26-0": [ + {"udid": "BBB", "name": "iPhone 17", "deviceTypeIdentifier": "type.iPhone-17", "isAvailable": true} + ] + }} + """ + let byUDID = IsolatedSimulator.deviceSpec(source: "AAA", listJSON: json) + #expect(byUDID?.deviceType == "type.iPhone-16") + #expect(byUDID?.runtime == "com.apple.CoreSimulator.SimRuntime.iOS-18-0") + let byName = IsolatedSimulator.deviceSpec(source: "iPhone 17", listJSON: json) + #expect(byName?.deviceType == "type.iPhone-17") + #expect(IsolatedSimulator.deviceSpec(source: "nope", listJSON: json) == nil) + } + + // MARK: - WDA port cleanup + + @Test("only WebDriverAgent runner processes are eligible for port cleanup") + func wdaRunnerDetection() { + #expect(WDAClient.isWDARunnerCommand("/path/xcforgeWDARunner-Runner")) + #expect(WDAClient.isWDARunnerCommand("WebDriverAgentRunner-Runner")) + #expect(!WDAClient.isWDARunnerCommand("/usr/bin/python3")) + #expect(!WDAClient.isWDARunnerCommand("node")) + } +} diff --git a/Tests/XCForgeKitTests/TestProviderTests.swift b/Tests/XCForgeKitTests/TestProviderTests.swift index 6be571c..3966d86 100644 --- a/Tests/XCForgeKitTests/TestProviderTests.swift +++ b/Tests/XCForgeKitTests/TestProviderTests.swift @@ -294,53 +294,69 @@ struct TestProviderTests { #expect(!eraseCalled) } - // MARK: 7. simRecovery .auto — unhealthy sim: tiered recovery; erase reached when tier 1 fails + // MARK: 7. simRecovery .auto — unhealthy sim: reboot only, never erase - @Test("simRecovery .auto unhealthy sim: probeAndRecover returns fired=true, erase called") - func simRecoveryAutoUnhealthy() async { - // State is Shutdown — health probe returns notBooted, triggering tiered recovery. - // After tier-1 shutdown+boot the list probe returns Shutdown again (mock is stateless), - // so tier 2 (erase) fires. + @Test("simRecovery .auto unhealthy sim: reboots but never erases") + func simRecoveryAutoUnhealthyNeverErases() async { + // State stays Shutdown after the reboot (mock is stateless). Auto mode must stop there. let shell = TestRecordingShell(simState: "Shutdown") let env = Environment(shell: shell) let outcome = await SimulatorRecovery.probeAndRecover( udid: "AAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE", + mode: .auto, env: env ) #expect(outcome.fired) #expect(outcome.reason == "sim_unhealthy") + #expect(outcome.failureReason?.contains("erase skipped") == true) + let eraseCalled = await shell.eraseCalled + #expect(!eraseCalled) + } + + @Test("simRecovery .erase unhealthy sim: erase runs when a reboot doesn't help") + func simRecoveryEraseUnhealthy() async { + let shell = TestRecordingShell(simState: "Shutdown") + let env = Environment(shell: shell) + + let outcome = await SimulatorRecovery.probeAndRecover( + udid: "AAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE", + mode: .erase, + env: env + ) + + #expect(outcome.fired) let eraseCalled = await shell.eraseCalled #expect(eraseCalled) } - // MARK: 8. simRecovery .off gate — no erase called for unhealthy sim + // MARK: 8. simRecovery .off — nothing touched - @Test("simRecovery .off skips probe: unhealthy sim, no erase invoked") - func simRecoveryOffSkipsProbeOnUnhealthySim() async { - // The gate in executeBuildAndTest is: `if simRecovery == .auto { probeAndRecover(...) }` - // With .off, probeAndRecover is never called. + @Test("simRecovery .off: unhealthy sim is left alone") + func simRecoveryOffLeavesSimAlone() async { let shell = TestRecordingShell(simState: "Shutdown") let env = Environment(shell: shell) - let recoveryMode: SimRecoveryMode = .off - var eraseWouldHaveFired = false - - if recoveryMode == .auto { - let outcome = await SimulatorRecovery.probeAndRecover( - udid: "AAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE", - env: env - ) - eraseWouldHaveFired = outcome.fired - } + let outcome = await SimulatorRecovery.probeAndRecover( + udid: "AAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE", + mode: .off, + env: env + ) - // Gate blocked the probe — neither fired nor erase called - #expect(!eraseWouldHaveFired) + #expect(!outcome.fired) let shellEraseCalled = await shell.eraseCalled #expect(!shellEraseCalled) } + @Test("SimRecoveryMode.parse defaults to off and rejects unknown values") + func simRecoveryModeParse() throws { + #expect(try SimRecoveryMode.parse(nil) == .off) + #expect(try SimRecoveryMode.parse("AUTO") == .auto) + #expect(try SimRecoveryMode.parse("erase") == .erase) + #expect(throws: SimRecoveryModeError.self) { try SimRecoveryMode.parse("yes") } + } + // MARK: 9. Child sections appear below legacy headers @Test("child process samples appear below legacy header lines in snapshot output") From 1fd77b0bd0085dc532448eecf9870b4b99a28861 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 14:24:03 +0000 Subject: [PATCH 02/28] feat: catch hung builds by silence, report every error, list Swift Testing tests Adds an idle timeout (600s of no output) to every build and test xcodebuild, raises the total limit to 1800s, retries test-without-building once when the runner never starts, continues building after errors and reads diagnostics from stdout, lists tests through JSON enumeration (Swift Testing included, with --testplan), and fixes test plan discovery and decoding. Also fixes the Bundle 1 test that failed to type-check and a lint nit. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01NVNakyntY4u9MvhsjbzRhZ --- CHANGELOG.md | 12 + CLAUDE.md | 5 +- Skills/xcforge/references/auto-detection.md | 2 +- Skills/xcforge/references/build-tools.md | 2 +- Skills/xcforge/references/cli-commands.md | 6 +- Skills/xcforge/references/test-tools.md | 6 +- .../Commands/Build/BuildCommand.swift | 2 +- .../Commands/Build/BuildTestCommand.swift | 10 +- .../Build/XcodebuildOptionGroup.swift | 12 +- .../Commands/Test/TestCommand.swift | 8 +- .../Commands/Test/TestRenderer.swift | 3 + .../Test/TestRerunFailedCommand.swift | 8 +- .../Persistence/DefaultsStore.swift | 24 +- .../Support/DiagnosticSnapshot.swift | 3 +- .../XCForgeKit/Support/ProcessRunner.swift | 91 ++++++- .../XCForgeKit/Support/WorkspaceContext.swift | 6 +- Sources/XCForgeKit/Support/Xcodebuild.swift | 56 +++- .../Support/XcodebuildOptions.swift | 33 ++- Sources/XCForgeKit/Tools/BuildProvider.swift | 50 ++-- .../XCForgeKit/Tools/TestPlanInspector.swift | 127 ++++++++- Sources/XCForgeKit/Tools/TestProvider.swift | 244 ++++++++++++++---- .../BuildTestReliabilityTests.swift | 129 +++++++++ Tests/XCForgeKitTests/RepoConfigTests.swift | 12 +- .../SharedMacSafetyTests.swift | 69 ++++- 24 files changed, 793 insertions(+), 127 deletions(-) create mode 100644 Tests/XCForgeKitTests/BuildTestReliabilityTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b7e51d..23d085a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,13 +11,25 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - **Build lock with a first-come, first-served queue.** Takes the same `flock` as macOS `lockf(1)`, so it queues next to existing `lockf` wrappers. `xcforge lock status` / `build_lock_status` show the holder, the queue and wait times - `--isolated-sim` / `isolatedSimulator` on `build-test`/`build_and_test` and `test run`/`test_sim`: run on a fresh simulator created for the run and deleted afterwards - Preflight warnings for low disk and heavy swap before a build +- **Idle timeout for builds and test runs.** xcodebuild is killed after 600s with no output (`--idle-timeout`, `idleTimeoutSeconds`, `XCFORGE_IDLE_TIMEOUT`, `.xcforge.yaml idleTimeout`; 0 disables), so a hung runner is caught without cutting off a slow cold build. Failures say whether the idle or total limit fired (`failureReason: timeout_idle|timeout_total`) +- `test-without-building` is retried once when the test runner never started ("Test runner hung before establishing connection", "Early unexpected exit…", "Failed to launch…" with no test begun) +- `test list --testplan` / `list_tests testplan`: lists via `-test-enumeration-format json`, which includes Swift Testing tests and reports how many the scheme or plan disables +- `test_plan_inspect` shows skipped tests, selected-only targets and every tag filter in the plan, and warns when a multi-tag filter may require all tags ### Changed +- **Default total time limit for builds and tests is 1800s** (was 180s); `--long` / `long: true` raises it to 7200s. Hangs are caught by the idle timeout instead +- **Builds continue after the first error** (`-IDEBuildingContinueBuildingAfterErrors=YES`), so one run reports every error. Opt out with `--no-continue-after-errors` / `continueAfterErrors: false` +- Build errors are read from stdout as well as stderr (xcodebuild prints compiler diagnostics on stdout), duplicates are dropped, and up to 50 are shown +- A run where zero tests executed is a failure even without a filter, when xcforge could read the result summary +- `build-test` and `test rerun-failed` honour `.xcforge.yaml` `testPlan` and `configuration` like `test run` does +- Coverage lookups use `xccov --only-targets` when only a yes/no is needed, and fall back to per-target coverage instead of truncated JSON for very large reports - **`simRecovery` defaults to `off` everywhere** (was `auto` for `build_and_test`, `build-test` and `bless`). `auto` now only reboots; erasing a simulator needs the new `erase` mode. Invalid values are an error instead of silently becoming `auto`. `build-test` gains `--sim-recovery` - Result bundles and diagnostic snapshots get collision-free names (`xcf----`), so two sessions starting in the same second no longer delete each other's bundles. `XCFORGE_ARTIFACT_DIR` / `artifactDir` moves them out of `/tmp` - `build diagnose`, `test failures` and `test coverage` read this project's last recorded result bundle instead of the newest bundle any session left in `/tmp`. `test failures` without `--xcresult-path` reuses that bundle instead of re-running the whole suite ### Fixed +- A timed-out `build-for-testing` is reported as a build failure instead of continuing to `test-without-building` against stale products +- `test_plan_inspect` finds plans inside `App.xcodeproj/xcshareddata/xctestplans` and decodes Xcode's string-form `skippedTests` and plans with missing sections - A hung build's fallback no longer runs `pkill -x xcodebuild`, which killed every session's build on the Mac. Hang snapshots sample the xcodebuild matched by this run's result bundle path instead of the newest xcodebuild on the system - WDA port cleanup only kills WebDriverAgent runner processes, uses the port from `WDA_BASE_URL`, skips remote WDA hosts, and calls `lsof` at its real path (`/usr/sbin/lsof`) diff --git a/CLAUDE.md b/CLAUDE.md index ecf174b..64379e3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,10 +14,11 @@ swift run xcforge build --help # CLI mode ``` **Test timeout defaults** (`build-test`, `test run`, `test_sim`, `build_and_test`): -- Default timeout: **180s**. Pass `--long` (CLI) or `long: true` (MCP) to use 1800s. +- Default total limit: **1800s** (`--long` / `long: true`: 7200s). Builds and test runs are also killed + after **600s with no output** (`--idle-timeout` / `idleTimeoutSeconds`, 0 disables). - Pass `--diagnose` / `diagnose: true` to capture a diagnostic snapshot even on success. - On timeout, the result includes a `/tmp/xcf-diag--.txt` path and a summary line. -- `build_sim` and `build_run_sim` share the same watchdog design; `build` (build-only) remains at 1800s. +- `build_sim` and `build_run_sim` share the same watchdog design. **Formatting** (CI enforces strict lint): ```bash diff --git a/Skills/xcforge/references/auto-detection.md b/Skills/xcforge/references/auto-detection.md index de1f882..1629de4 100644 --- a/Skills/xcforge/references/auto-detection.md +++ b/Skills/xcforge/references/auto-detection.md @@ -49,7 +49,7 @@ file is warned and skipped (never crashes). | `simulator` | Default simulator name or UDID. | shared with defaults model | | `configuration` | Build configuration for `build_sim`/`test_sim`/`build_and_test`/`build_and_diagnose` when no `--configuration`/`configuration` arg is given. Default `Debug`. | **repo-only — never written to `defaults.json` or profiles** | | `testPlan` | Default `.xctestplan` for `test_sim`/`build_and_test` when no `--testplan`/`testplan` arg is given. | **repo-only — never written to `defaults.json` or profiles** | -| `testTimeout` | Per-project default test timeout, positive integer seconds. Precedence: explicit `timeoutSeconds` > `testTimeout` > `--long`/180s default. Non-positive values rejected with warning. | **repo-only** | +| `testTimeout` | Per-project default test timeout, positive integer seconds. Precedence: explicit `timeoutSeconds` > `testTimeout` > `--long` (7200s) > 1800s default. Non-positive values rejected with warning. | **repo-only** | | `autoPromote` | `true` (default) keeps the 3-rep auto-promotion. `false` disables it (streak counter held at zero) so explicit values stay explicit across repeated iterative runs. | **repo-only** | `configuration`, `testPlan`, `testTimeout`, and `autoPromote` are repo-only by diff --git a/Skills/xcforge/references/build-tools.md b/Skills/xcforge/references/build-tools.md index a5ba9f4..7d6cc1d 100644 --- a/Skills/xcforge/references/build-tools.md +++ b/Skills/xcforge/references/build-tools.md @@ -16,7 +16,7 @@ Fast compile-only build without install or launch (~5s vs ~20s for full pipeline | `scheme` | No | Auto-detect | Scheme name | | `simulator` | No | Auto-detect (booted) | Simulator name or UDID (for SDK selection) | | `configuration` | No | Debug | Build configuration (Debug/Release) | -| `long` | No | false | Use 1800s timeout instead of default 180s | +| `long` | No | false | Raise the total time limit from 1800s to 7200s | **Returns:** Bundle ID, app path, build duration, warnings count. On failure: structured errors with file:line from xcresult. diff --git a/Skills/xcforge/references/cli-commands.md b/Skills/xcforge/references/cli-commands.md index 6ee64a6..a016774 100644 --- a/Skills/xcforge/references/cli-commands.md +++ b/Skills/xcforge/references/cli-commands.md @@ -83,7 +83,7 @@ Fast compile-only build without simulator boot/install/launch. Reuses the standa xcforge build compile # Compile-only, auto-detect everything xcforge build compile --project MyApp.xcodeproj --scheme MyApp xcforge build compile --configuration Release -xcforge build compile --long # 1800s timeout instead of 180s +xcforge build compile --long # 7200s total limit instead of 1800s xcforge build compile --json ``` @@ -93,7 +93,7 @@ xcforge build compile --json | `--scheme ` | Xcode scheme name. Auto-detected if omitted | | `--simulator ` | Simulator name or UDID (for SDK selection). Auto-detected if omitted | | `--configuration ` | Build configuration (Debug/Release). Default: Debug | -| `--long` | Use 1800s timeout instead of default 180s | +| `--long` | Raise the total time limit from 1800s to 7200s | | `--json` | Machine-readable JSON output | **Use case:** Rapid code iteration without deployment latency. Does not boot simulator or install app. @@ -186,7 +186,7 @@ xcforge test rerun-failed --json | `--simulator ` | Auto-detected if omitted | | `--configuration ` | Build configuration. Default: Debug | | `--testplan ` | Test plan name | -| `--long` | Use 1800s timeout instead of 180s | +| `--long` | Raise the total time limit from 1800s to 7200s | | `--for ` | `human` (default) or `agent` | | `--gate` | Apply known-failures gate to the rerun result | | `--json` | Machine-readable JSON output | diff --git a/Skills/xcforge/references/test-tools.md b/Skills/xcforge/references/test-tools.md index d287df1..f1e47e2 100644 --- a/Skills/xcforge/references/test-tools.md +++ b/Skills/xcforge/references/test-tools.md @@ -9,9 +9,11 @@ All test tools parse `.xcresult` bundles for structured results — no raw xcode > See [auto-detection.md](auto-detection.md). > > **Timeout note:** `timeoutSeconds` precedence is explicit arg > `.xcforge.yaml -> testTimeout` > `--long`/`long: true` (1800s) > default 180s. Non-positive +> testTimeout` > `--long`/`long: true` (7200s) > default 1800s. Non-positive > values are rejected on both the explicit and the YAML path and fall through to -> the next layer with a warning. +> the next layer with a warning. Separately, a build or test run that prints +> nothing for `idleTimeoutSeconds` (default 600, 0 disables) is killed and +> reported with `timeout: idle`. ## test_sim diff --git a/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift b/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift index 51455b0..4e1e18c 100644 --- a/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift +++ b/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift @@ -34,7 +34,7 @@ struct BuildCompile: AsyncParsableCommand { @Option(help: "Build configuration (Debug/Release). Default: Debug") var configuration: String? - @Flag(help: "Use 1800s timeout instead of the default 180s for large projects.") + @Flag(help: "Raise the total time limit from 1800s to 7200s. Hangs are caught by --idle-timeout either way.") var long = false @Flag(help: "Capture a diagnostic snapshot on completion even without a hang.") diff --git a/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift b/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift index d4b67fd..3b3b23f 100644 --- a/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift +++ b/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift @@ -33,12 +33,12 @@ struct BuildTest: AsyncParsableCommand { @Flag(help: "Enable code coverage collection.") var coverage = false - @Flag(help: "Use 1800s timeout instead of the default 180s (for long integration suites).") + @Flag(help: "Raise the total time limit from 1800s to 7200s. Hangs are caught by --idle-timeout either way.") var long = false @Option( help: - "Override timeout in seconds. Takes precedence over --long. Default: 180s (or 1800s with --long)." + "Override timeout in seconds. Takes precedence over --long. Default: 1800s (or 7200s with --long)." ) var timeoutSeconds: Int? @@ -87,7 +87,9 @@ struct BuildTest: AsyncParsableCommand { func execute() async throws { let useJSON = shouldOutputJSON(flag: json) || forMode == .agent - let configuration = self.configuration ?? "Debug" + let session = Environment.live.session + let configuration = await session.resolveConfiguration(self.configuration) + let resolvedTestplan = await session.resolveTestPlan(testplan) let resolvedTimeout = timeoutSeconds.map { TimeInterval($0) } let result = try await TestTools.executeBuildAndTest( @@ -95,7 +97,7 @@ struct BuildTest: AsyncParsableCommand { scheme: scheme, simulator: simulator, configuration: configuration, - testplan: testplan, + testplan: resolvedTestplan, filter: filter, coverage: coverage, long: long, diff --git a/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift b/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift index eb3ffe6..941cb8f 100644 --- a/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift +++ b/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift @@ -30,6 +30,14 @@ struct XcodebuildOptionGroup: ParsableArguments { @Option(help: "Refuse to build below this much free disk (GB). Default: warn only.") var minFreeGb: Double? + @Option(help: "Kill xcodebuild after this many seconds with no output. Default: 600. 0 disables.") + var idleTimeout: Int? + + @Flag( + inversion: .prefixedNo, + help: "Keep building after the first error so one run reports every error. Default: on.") + var continueAfterErrors: Bool? + var options: XcodebuildOptions { XcodebuildOptions( derivedDataPath: derivedDataPath.map { ($0 as NSString).expandingTildeInPath }, @@ -37,7 +45,9 @@ struct XcodebuildOptionGroup: ParsableArguments { extraArgs: xcodebuildArgs, lockPath: lockPath.map { ($0 as NSString).expandingTildeInPath }, lockWaitSeconds: lockWait.map { TimeInterval($0) }, - minFreeGB: minFreeGb + minFreeGB: minFreeGb, + idleTimeoutSeconds: idleTimeout.map { TimeInterval($0) }, + continueAfterErrors: continueAfterErrors ) } diff --git a/Sources/XCForgeCLI/Commands/Test/TestCommand.swift b/Sources/XCForgeCLI/Commands/Test/TestCommand.swift index 4bb6412..635e188 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestCommand.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestCommand.swift @@ -76,7 +76,7 @@ struct TestRun: AsyncParsableCommand { @Flag(help: "Enable code coverage collection.") var coverage = false - @Flag(help: "Use 1800s timeout instead of the default 180s (for long integration suites).") + @Flag(help: "Raise the total time limit from 1800s to 7200s. Hangs are caught by --idle-timeout either way.") var long = false @Flag(help: "Capture a diagnostic snapshot even when the test run succeeds.") @@ -291,6 +291,9 @@ struct TestList: AsyncParsableCommand { @Option(help: "Simulator name or UDID. Auto-detected from booted simulator if omitted.") var simulator: String? + @Option(help: "List the tests this test plan runs. Default: .xcforge.yaml testPlan, else the scheme's.") + var testplan: String? + @Flag(help: "Emit the result as machine-readable JSON.") var json = false @@ -306,7 +309,8 @@ struct TestList: AsyncParsableCommand { let result = try await TestTools.executeListTests( project: project, scheme: scheme, - simulator: simulator + simulator: simulator, + testplan: testplan ) if useJSON { diff --git a/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift b/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift index 87759e0..b2b8f46 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift @@ -120,6 +120,9 @@ enum TestRenderer { lines.append( "\(result.testCount) tests in \(result.targetCount) target(s), \(result.classCount) class(es)" ) + if let disabled = result.disabledTestCount, disabled > 0 { + lines.append("\(disabled) more disabled by the scheme or test plan") + } lines.append("") // Group by target/class diff --git a/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift b/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift index 0bb7959..00d0031 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift @@ -24,7 +24,7 @@ struct TestRerunFailed: AsyncParsableCommand { @Option(help: "Test plan name.") var testplan: String? - @Flag(help: "Use 1800s timeout instead of the default 180s.") + @Flag(help: "Raise the total time limit from 1800s to 7200s.") var long = false @Flag(help: "Capture a diagnostic snapshot even when the test run succeeds.") @@ -62,7 +62,9 @@ struct TestRerunFailed: AsyncParsableCommand { } let useJSON = shouldOutputJSON(flag: json) || forMode == .agent - let configuration = self.configuration ?? "Debug" + let session = Environment.live.session + let configuration = await session.resolveConfiguration(self.configuration) + let resolvedTestplan = await session.resolveTestPlan(testplan) // Pass failure IDs as a pre-split list so IDs containing commas (e.g. // parameterized Swift Testing arguments) survive the trip to xcodebuild @@ -72,7 +74,7 @@ struct TestRerunFailed: AsyncParsableCommand { scheme: scheme ?? payload.scheme, simulator: simulator ?? payload.simulator, configuration: configuration, - testplan: testplan, + testplan: resolvedTestplan, filter: nil, filterIDs: payload.failures, coverage: false, diff --git a/Sources/XCForgeKit/Persistence/DefaultsStore.swift b/Sources/XCForgeKit/Persistence/DefaultsStore.swift index c336abf..87b2aa5 100644 --- a/Sources/XCForgeKit/Persistence/DefaultsStore.swift +++ b/Sources/XCForgeKit/Persistence/DefaultsStore.swift @@ -580,7 +580,7 @@ public enum RepoConfig { public var configuration: String? public var testPlan: String? /// Default test watchdog timeout in seconds, applied when no explicit - /// `timeoutSeconds` is passed. Overrides the built-in 180s/1800s bimodal. + /// `timeoutSeconds` is passed. Overrides the built-in 1800s/7200s defaults. public var testTimeout: Int? /// When `false`, suppresses the 3-strikes silent promotion of explicit /// values to sticky session defaults. Defaults to `true` (legacy behavior) @@ -594,6 +594,8 @@ public enum RepoConfig { public var artifactDir: String? /// Refuse to build below this much free disk (GB). Absent means warn only. public var minFreeGB: Double? + /// Seconds of xcodebuild silence before it is treated as hung. 0 disables. + public var idleTimeout: TimeInterval? public init( project: String? = nil, @@ -606,7 +608,8 @@ public enum RepoConfig { derivedDataPath: String? = nil, buildLock: String? = nil, artifactDir: String? = nil, - minFreeGB: Double? = nil + minFreeGB: Double? = nil, + idleTimeout: TimeInterval? = nil ) { self.project = project self.scheme = scheme @@ -619,6 +622,7 @@ public enum RepoConfig { self.buildLock = buildLock self.artifactDir = artifactDir self.minFreeGB = minFreeGB + self.idleTimeout = idleTimeout } /// True when every field is nil (nothing to apply). @@ -626,6 +630,7 @@ public enum RepoConfig { project == nil && scheme == nil && simulator == nil && configuration == nil && testPlan == nil && testTimeout == nil && autoPromote == nil && derivedDataPath == nil && buildLock == nil && artifactDir == nil && minFreeGB == nil + && idleTimeout == nil } } @@ -679,6 +684,7 @@ public enum RepoConfig { let allowedKeys: Set = [ "project", "scheme", "simulator", "configuration", "testPlan", "testTimeout", "autoPromote", "derivedDataPath", "buildLock", "artifactDir", "minFreeGB", + "idleTimeout", ] for key in dict.keys where !allowedKeys.contains(key) { Log.warn("\(RepoConfig.fileName): ignoring unknown key '\(key)'") @@ -740,6 +746,15 @@ public enum RepoConfig { } } + var idleTimeout: TimeInterval? + if let raw = dict["idleTimeout"] { + if let parsed = TimeInterval(raw), parsed >= 0 { + idleTimeout = parsed + } else { + Log.warn("\(RepoConfig.fileName): ignoring non-numeric idleTimeout '\(raw)'") + } + } + let result = Values( project: project, scheme: dict["scheme"], @@ -751,7 +766,8 @@ public enum RepoConfig { derivedDataPath: resolvedPath("derivedDataPath"), buildLock: resolvedPath("buildLock"), artifactDir: resolvedPath("artifactDir"), - minFreeGB: minFreeGB + minFreeGB: minFreeGB, + idleTimeout: idleTimeout ) return result.isEmpty ? nil : result } @@ -810,7 +826,7 @@ public enum RepoConfig { lines.append( entry( "testTimeout", nil, - "Default test watchdog timeout in seconds (e.g. 600). Overrides 180s/1800s bimodal." + "Default test watchdog timeout in seconds (e.g. 600). Overrides the 1800s/7200s defaults." ).trimmingCharacters(in: .newlines)) lines.append("") lines.append( diff --git a/Sources/XCForgeKit/Support/DiagnosticSnapshot.swift b/Sources/XCForgeKit/Support/DiagnosticSnapshot.swift index cb5f83b..e69466a 100644 --- a/Sources/XCForgeKit/Support/DiagnosticSnapshot.swift +++ b/Sources/XCForgeKit/Support/DiagnosticSnapshot.swift @@ -70,7 +70,8 @@ enum DiagnosticSnapshot { sections.append("=== xcforge diagnostic snapshot ===") sections.append("Timestamp: \(ISO8601DateFormatter().string(from: Date()))") - let pgrepArgs = processMatch.map { ["-n", "-f", NSRegularExpression.escapedPattern(for: $0)] } + let pgrepArgs = + processMatch.map { ["-n", "-f", NSRegularExpression.escapedPattern(for: $0)] } ?? ["-n", "xcodebuild"] let pgrepResult = try? await env.shell.run( "/usr/bin/pgrep", arguments: pgrepArgs, timeout: 5) diff --git a/Sources/XCForgeKit/Support/ProcessRunner.swift b/Sources/XCForgeKit/Support/ProcessRunner.swift index 80b8722..1d1acc9 100644 --- a/Sources/XCForgeKit/Support/ProcessRunner.swift +++ b/Sources/XCForgeKit/Support/ProcessRunner.swift @@ -1,4 +1,5 @@ import Foundation +import os public struct ShellResult: Sendable { public let stdout: String @@ -22,6 +23,18 @@ public protocol ShellExecutor: Sendable { outputLimit: Int ) async throws -> ShellResult + /// Like `run`, but also kills the process after `idleTimeout` seconds without any output. + /// The default implementation ignores `idleTimeout`, so test doubles need not implement it. + func run( + _ executable: String, + arguments: [String], + workingDirectory: String?, + environment: [String: String]?, + timeout: TimeInterval, + idleTimeout: TimeInterval?, + outputLimit: Int + ) async throws -> ShellResult + func xcrun(timeout: TimeInterval, arguments: [String]) async throws -> ShellResult func git(_ arguments: [String], workingDirectory: String, timeout: TimeInterval) async throws @@ -29,6 +42,20 @@ public protocol ShellExecutor: Sendable { } extension ShellExecutor { + public func run( + _ executable: String, + arguments: [String], + workingDirectory: String?, + environment: [String: String]?, + timeout: TimeInterval, + idleTimeout: TimeInterval?, + outputLimit: Int + ) async throws -> ShellResult { + try await run( + executable, arguments: arguments, workingDirectory: workingDirectory, + environment: environment, timeout: timeout, outputLimit: outputLimit) + } + /// Convenience variadic wrapper matching Shell's original API. public func xcrun(timeout: TimeInterval = 300, _ args: String...) async throws -> ShellResult { try await xcrun(timeout: timeout, arguments: args) @@ -123,6 +150,21 @@ public struct LiveShell: ShellExecutor { environment: environment, timeout: timeout, outputLimit: outputLimit) } + public func run( + _ executable: String, + arguments: [String], + workingDirectory: String?, + environment: [String: String]?, + timeout: TimeInterval, + idleTimeout: TimeInterval?, + outputLimit: Int + ) async throws -> ShellResult { + try await Shell.run( + executable, arguments: arguments, workingDirectory: workingDirectory, + environment: environment, timeout: timeout, idleTimeout: idleTimeout, + outputLimit: outputLimit) + } + public func xcrun(timeout: TimeInterval, arguments: [String]) async throws -> ShellResult { try await Shell.xcrun(timeout: timeout, arguments: arguments) } @@ -147,6 +189,7 @@ public enum Shell { workingDirectory: String? = nil, environment: [String: String]? = nil, timeout: TimeInterval = 300, + idleTimeout: TimeInterval? = nil, outputLimit: Int = Shell.defaultOutputLimit ) async throws -> ShellResult { let process = Process() @@ -191,9 +234,35 @@ public enum Shell { kill(pid, SIGKILL) } + // Idle watchdog: kill the process after `idleTimeout` seconds with no output on either + // pipe. A slow but busy build keeps printing and is never killed by it. + let lastOutput = OSAllocatedUnfairLock(initialState: Date()) + let killedForIdle = OSAllocatedUnfairLock(initialState: false) + var idleTask: Task? + if let idle = idleTimeout, idle > 0 { + let tick = min(5.0, max(0.05, idle / 4)) + idleTask = Task.detached { + while !Task.isCancelled { + try? await Task.sleep(nanoseconds: UInt64(tick * 1_000_000_000)) + if Task.isCancelled { return } + let silent = Date().timeIntervalSince(lastOutput.withLock { $0 }) + if silent >= idle { + killedForIdle.withLock { $0 = true } + kill(pid, SIGTERM) + try? await Task.sleep(nanoseconds: 2_000_000_000) + kill(pid, SIGKILL) + return + } + } + } + } + let markOutput: @Sendable () -> Void = { lastOutput.withLock { $0 = Date() } } + // Read output concurrently, capping retained data to avoid OOM on large builds - async let stdoutData = stdoutPipe.fileHandleForReading.readTailAsync(limit: outputLimit) - async let stderrData = stderrPipe.fileHandleForReading.readTailAsync(limit: outputLimit) + async let stdoutData = stdoutPipe.fileHandleForReading.readTailAsync( + limit: outputLimit, onData: markOutput) + async let stderrData = stderrPipe.fileHandleForReading.readTailAsync( + limit: outputLimit, onData: markOutput) let (out, err) = await (stdoutData, stderrData) @@ -203,6 +272,7 @@ public enum Shell { // still-running process — that throws an uncatchable ObjC exception. for await _ in terminationContinuation.stream {} timeoutTask.cancel() + idleTask?.cancel() // If task was cancelled, the process may still be running if process.isRunning { @@ -213,12 +283,14 @@ public enum Shell { // Detect killed process (timeout or crash → uncaughtSignal) if process.terminationReason == .uncaughtSignal { let partialOut = String(data: out, encoding: .utf8) ?? "" - return ShellResult( - stdout: partialOut, - stderr: - "Process timed out after \(Int(timeout))s and was killed (signal \(process.terminationStatus))", - exitCode: -1 - ) + let reason: String + if killedForIdle.withLock({ $0 }), let idle = idleTimeout { + reason = "Process produced no output for \(Int(idle))s and was killed (idle timeout)" + } else { + reason = + "Process timed out after \(Int(timeout))s and was killed (signal \(process.terminationStatus))" + } + return ShellResult(stdout: partialOut, stderr: reason, exitCode: -1) } let stdout = @@ -254,7 +326,7 @@ extension FileHandle { /// Read pipe data incrementally, retaining only the last `limit` bytes. /// Prevents unbounded memory growth when subprocesses emit large output /// (e.g. xcodebuild builds that produce hundreds of MB). - func readTailAsync(limit: Int) async -> Data { + func readTailAsync(limit: Int, onData: (@Sendable () -> Void)? = nil) async -> Data { precondition(limit > 0, "outputLimit must be positive") return await withCheckedContinuation { continuation in DispatchQueue.global().async { @@ -262,6 +334,7 @@ extension FileHandle { while true { let chunk = self.availableData if chunk.isEmpty { break } + onData?() tail.append(chunk) if tail.count > limit { tail = Data(tail.suffix(limit)) diff --git a/Sources/XCForgeKit/Support/WorkspaceContext.swift b/Sources/XCForgeKit/Support/WorkspaceContext.swift index 6183d57..8781a3f 100644 --- a/Sources/XCForgeKit/Support/WorkspaceContext.swift +++ b/Sources/XCForgeKit/Support/WorkspaceContext.swift @@ -186,7 +186,9 @@ public actor SessionState { /// Resolve test watchdog timeout (seconds). Repo-only — never persisted. /// - /// Order: explicit > `.xcforge.yaml` `testTimeout` > `long ? 1800 : 180`. + /// Order: explicit > `.xcforge.yaml` `testTimeout` > `long ? 7200 : 1800`. + /// Hangs are caught separately by the idle timeout (no output for 600s), so the total + /// limit only has to bound a run that keeps printing. /// /// `explicit` must be a positive integer. A value `<= 0` is symmetric with the /// YAML path's rejection of non-positive values: it falls through to the @@ -201,7 +203,7 @@ public actor SessionState { ) } if let repo = repoDefaults?.testTimeout { return TimeInterval(repo) } - return long ? 1800 : 180 + return long ? 7200 : 1800 } /// Resolve simulator and return both its display name and UDID. diff --git a/Sources/XCForgeKit/Support/Xcodebuild.swift b/Sources/XCForgeKit/Support/Xcodebuild.swift index 71223d9..135cc49 100644 --- a/Sources/XCForgeKit/Support/Xcodebuild.swift +++ b/Sources/XCForgeKit/Support/Xcodebuild.swift @@ -19,6 +19,7 @@ public enum Xcodebuild { workingDirectory: String? = nil, environment: [String: String]? = nil, timeout: TimeInterval, + idleTimeout: TimeInterval? = nil, env: Environment ) async throws -> ShellResult { let options = XcodebuildOptions.effective(cwd: env.currentDirectoryPath()) @@ -36,9 +37,15 @@ public enum Xcodebuild { } defer { lock?.release() } + // Builds and test runs are killed after a stretch of silence rather than a fixed total, + // so a slow cold build is never cut off while a hung runner still is. + let idle = + idleTimeout ?? options.idleTimeoutSeconds + ?? (needsLock(args) ? XcodebuildOptions.defaultIdleTimeoutSeconds : nil) let result = try await env.shell.run( executable, arguments: args, workingDirectory: workingDirectory, - environment: environment, timeout: timeout) + environment: environment, timeout: timeout, + idleTimeout: (idle ?? 0) > 0 ? idle : nil, outputLimit: Shell.defaultOutputLimit) if result.exitCode != -2 { LastResultStore.recordFromArguments(args) } @@ -71,6 +78,12 @@ public enum Xcodebuild { { insert += ["-derivedDataPath", dd] } + if options.continueAfterErrors ?? true, + arguments.contains(where: { compileActions.contains($0) }), + !arguments.contains(where: { $0.hasPrefix(continueAfterErrorsDefault) }) + { + insert.append(continueAfterErrorsDefault + "=YES") + } insert += options.extraArgs guard !insert.isEmpty else { return arguments } let index = arguments.firstIndex { actionTokens.contains($0) } ?? arguments.endIndex @@ -79,6 +92,47 @@ public enum Xcodebuild { return result } + /// Xcode user default that keeps the build going after the first error, so every + /// error in the run is reported instead of only the first target's. + static let continueAfterErrorsDefault = "-IDEBuildingContinueBuildingAfterErrors" + + /// Actions that compile sources. + static let compileActions: Set = ["build", "build-for-testing", "test", "analyze"] + + /// How xcforge killed a process, or nil when it exited on its own. + public enum TimeoutKind: String, Sendable { + /// No output for the idle limit: the process was hung. + case idle + /// The total time limit ran out while the process was still printing. + case total + } + + public static func timeoutKind(_ result: ShellResult) -> TimeoutKind? { + guard result.exitCode == -1 else { return nil } + return result.stderr.contains("(idle timeout)") ? .idle : .total + } + + /// One line explaining a timeout and how to raise the limit, or nil. + public static func timeoutExplanation(_ result: ShellResult) -> String? { + switch timeoutKind(result) { + case .idle: + return result.stderr.trimmingCharacters(in: .whitespacesAndNewlines) + + ". Likely hung; raise idleTimeoutSeconds (--idle-timeout) if it was legitimately quiet." + case .total: + return result.stderr.trimmingCharacters(in: .whitespacesAndNewlines) + + ". It was still producing output; raise timeoutSeconds or pass long." + case nil: + return nil + } + } + + /// stdout and stderr together. xcodebuild prints compiler diagnostics to stdout. + public static func combinedOutput(_ result: ShellResult) -> String { + if result.stdout.isEmpty { return result.stderr } + if result.stderr.isEmpty { return result.stdout } + return result.stdout + "\n" + result.stderr + } + /// Short description used for lock status: scheme and action. static func summary(_ args: [String]) -> String { var parts: [String] = [] diff --git a/Sources/XCForgeKit/Support/XcodebuildOptions.swift b/Sources/XCForgeKit/Support/XcodebuildOptions.swift index 4e3bfd5..439f524 100644 --- a/Sources/XCForgeKit/Support/XcodebuildOptions.swift +++ b/Sources/XCForgeKit/Support/XcodebuildOptions.swift @@ -24,6 +24,11 @@ public struct XcodebuildOptions: Sendable, Equatable { public var minFreeGB: Double? /// Directory for xcforge-generated result bundles and diagnostics. public var artifactDir: String? + /// Kill xcodebuild after this many seconds without any output. 0 disables it. + public var idleTimeoutSeconds: TimeInterval? + /// Keep compiling other files and targets after the first error so one run reports them all. + /// Default: on. + public var continueAfterErrors: Bool? public init( derivedDataPath: String? = nil, @@ -32,7 +37,9 @@ public struct XcodebuildOptions: Sendable, Equatable { lockPath: String? = nil, lockWaitSeconds: TimeInterval? = nil, minFreeGB: Double? = nil, - artifactDir: String? = nil + artifactDir: String? = nil, + idleTimeoutSeconds: TimeInterval? = nil, + continueAfterErrors: Bool? = nil ) { self.derivedDataPath = derivedDataPath self.resultBundlePath = resultBundlePath @@ -41,6 +48,8 @@ public struct XcodebuildOptions: Sendable, Equatable { self.lockWaitSeconds = lockWaitSeconds self.minFreeGB = minFreeGB self.artifactDir = artifactDir + self.idleTimeoutSeconds = idleTimeoutSeconds + self.continueAfterErrors = continueAfterErrors } /// Options set for the current call (CLI flags or MCP arguments). @@ -49,6 +58,10 @@ public struct XcodebuildOptions: Sendable, Equatable { /// Default wait in the lock queue: one hour. public static let defaultLockWaitSeconds: TimeInterval = 3600 + /// Default silence before a build or test xcodebuild is treated as hung: ten minutes. + /// A cold build that keeps printing is never killed by it. + public static let defaultIdleTimeoutSeconds: TimeInterval = 600 + /// Merge the task-local value over environment variables over `.xcforge.yaml`. public static func effective( cwd: String = FileManager.default.currentDirectoryPath, @@ -76,6 +89,10 @@ public struct XcodebuildOptions: Sendable, Equatable { ?? repo?.minFreeGB merged.artifactDir = explicit.artifactDir ?? nonEmpty(environment["XCFORGE_ARTIFACT_DIR"]) ?? repo?.artifactDir + merged.idleTimeoutSeconds = + explicit.idleTimeoutSeconds ?? environment["XCFORGE_IDLE_TIMEOUT"].flatMap { TimeInterval($0) } + ?? repo?.idleTimeout + merged.continueAfterErrors = explicit.continueAfterErrors merged.extraArgs = explicit.extraArgs return merged } @@ -154,6 +171,16 @@ public struct XcodebuildOptions: Sendable, Equatable { "description": .string( "Refuse to build when the DerivedData volume has less free space than this. Default: warn only."), ]), + "idleTimeoutSeconds": .object([ + "type": .string("integer"), + "description": .string( + "Kill xcodebuild after this many seconds with no output. Default: 600. 0 disables it."), + ]), + "continueAfterErrors": .object([ + "type": .string("boolean"), + "description": .string( + "Keep building after the first error so one run reports every error. Default: true."), + ]), ] /// Parse the option keys out of an MCP argument dictionary. @@ -174,7 +201,9 @@ public struct XcodebuildOptions: Sendable, Equatable { lockPath: args["buildLock"]?.stringValue, lockWaitSeconds: number(args["lockWaitSeconds"]), minFreeGB: number(args["minFreeGB"]), - artifactDir: nil + artifactDir: nil, + idleTimeoutSeconds: number(args["idleTimeoutSeconds"]), + continueAfterErrors: args["continueAfterErrors"]?.boolValue ) } diff --git a/Sources/XCForgeKit/Tools/BuildProvider.swift b/Sources/XCForgeKit/Tools/BuildProvider.swift index 58e1192..6c1b61a 100644 --- a/Sources/XCForgeKit/Tools/BuildProvider.swift +++ b/Sources/XCForgeKit/Tools/BuildProvider.swift @@ -119,7 +119,7 @@ public enum BuildTools { let errorIssues = issues.filter { $0.severity == .error } if !errorIssues.isEmpty { return Array( - errorIssues.prefix(20).map { issue in + errorIssues.prefix(50).map { issue in if let loc = issue.location { var s = loc.filePath if let line = loc.line { s += ":\(line)" } @@ -168,7 +168,7 @@ public enum BuildTools { "long": .object([ "type": .string("boolean"), "description": .string( - "Use 1800s timeout instead of the default 180s for large projects." + "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." ), ]), "diagnose": .object([ @@ -214,7 +214,7 @@ public enum BuildTools { "long": .object([ "type": .string("boolean"), "description": .string( - "Use 1800s timeout instead of the default 180s for large projects that take longer to build." + "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." ), ]), "diagnose": .object([ @@ -260,7 +260,7 @@ public enum BuildTools { "long": .object([ "type": .string("boolean"), "description": .string( - "Use 1800s timeout instead of the default 180s for large projects." + "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." ), ]), "diagnose": .object([ @@ -466,21 +466,27 @@ public enum BuildTools { var warningCount = xcresultIssues.warningCount if issues.isEmpty { - issues = TestTools.fallbackBuildIssues(stderr: result.stderr) + issues = TestTools.fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(result)) errorCount = issues.filter { $0.severity == .error }.count warningCount = issues.filter { $0.severity == .warning }.count } // Classify failure from xcresult issues first, then stderr + let output = Xcodebuild.combinedOutput(result) let reason: String - if !issues.isEmpty { + if let timeout = Xcodebuild.timeoutKind(result) { + reason = "timeout_\(timeout.rawValue)" + } else if !issues.isEmpty { reason = classifyFailureFromIssues(issues) } else { - reason = classifyFailureReason(stderr: result.stderr) + reason = classifyFailureReason(stderr: output) } - let structured = extractStructuredErrors(stderr: result.stderr, failureReason: reason) - let errors = extractLegacyErrors(from: result.stderr) + var structured = extractStructuredErrors(stderr: output, failureReason: reason) + if let explanation = Xcodebuild.timeoutExplanation(result) { + structured.insert(explanation, at: 0) + } + let errors = extractLegacyErrors(from: output) return BuildExecution( succeeded: false, @@ -665,7 +671,8 @@ public enum BuildTools { private static func extractLegacyErrors(from stderr: String) -> [String] { let errorLines = stderr.split(separator: "\n") .filter { $0.contains(": error:") } - .prefix(20) + .reduce(into: [Substring]()) { if !$0.contains($1) { $0.append($1) } } + .prefix(50) .map(String.init) let stderrTail = String(stderr.suffix(2000)) return errorLines.isEmpty @@ -679,6 +686,11 @@ public enum BuildTools { if let reason = execution.failureReason { lines.append("Failure reason: \(reason)") + // The timeout explanation is always the first structured line; show it even when + // xcresult issues take over the error list below. + if reason.hasPrefix("timeout"), let note = execution.structuredErrors?.first { + lines.append(note) + } } lines.append("Scheme: \(execution.scheme)") @@ -692,9 +704,10 @@ public enum BuildTools { if !errors.isEmpty { lines.append("") lines.append("Errors (\(errors.count)):") - for issue in errors.prefix(20) { + for issue in errors.prefix(50) { lines.append(" \(formatIssue(issue))") } + if errors.count > 50 { lines.append(" ... \(errors.count - 50) more in the xcresult") } } if !warnings.isEmpty { lines.append("") @@ -932,16 +945,21 @@ public enum BuildTools { var warningCount = xcresultIssues.warningCount if issues.isEmpty { - issues = TestTools.fallbackBuildIssues(stderr: buildResult.stderr) + issues = TestTools.fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(buildResult)) errorCount = issues.filter { $0.severity == .error }.count warningCount = issues.filter { $0.severity == .warning }.count } + let output = Xcodebuild.combinedOutput(buildResult) let reason = - !issues.isEmpty - ? classifyFailureFromIssues(issues) : classifyFailureReason(stderr: buildResult.stderr) - let structured = extractStructuredErrors(stderr: buildResult.stderr, failureReason: reason) - let legacyErrors = extractLegacyErrors(from: buildResult.stderr) + Xcodebuild.timeoutKind(buildResult).map { "timeout_\($0.rawValue)" } + ?? (!issues.isEmpty + ? classifyFailureFromIssues(issues) : classifyFailureReason(stderr: output)) + var structured = extractStructuredErrors(stderr: output, failureReason: reason) + if let explanation = Xcodebuild.timeoutExplanation(buildResult) { + structured.insert(explanation, at: 0) + } + let legacyErrors = extractLegacyErrors(from: output) let execution = BuildExecution( succeeded: false, diff --git a/Sources/XCForgeKit/Tools/TestPlanInspector.swift b/Sources/XCForgeKit/Tools/TestPlanInspector.swift index 4851a66..558e353 100644 --- a/Sources/XCForgeKit/Tools/TestPlanInspector.swift +++ b/Sources/XCForgeKit/Tools/TestPlanInspector.swift @@ -7,11 +7,30 @@ public struct XCTestPlan: Codable { var configurations: [TestPlanConfig] var defaultOptions: TestPlanOptions var testTargets: [TestPlanTarget] + + // Xcode omits empty sections, so every key is optional on disk. + public init(from decoder: Decoder) throws { + let c = try decoder.container(keyedBy: CodingKeys.self) + version = try c.decodeIfPresent(Int.self, forKey: .version) ?? 1 + configurations = try c.decodeIfPresent([TestPlanConfig].self, forKey: .configurations) ?? [] + defaultOptions = + try c.decodeIfPresent(TestPlanOptions.self, forKey: .defaultOptions) + ?? TestPlanOptions(codeCoverage: nil, threadSanitizerEnabled: nil, addressSanitizerEnabled: nil) + testTargets = try c.decodeIfPresent([TestPlanTarget].self, forKey: .testTargets) ?? [] + } } public struct TestPlanConfig: Codable { public var name: String public var options: TestPlanOptions + + public init(from decoder: Decoder) throws { + let c = try decoder.container(keyedBy: CodingKeys.self) + name = try c.decode(String.self, forKey: .name) + options = + try c.decodeIfPresent(TestPlanOptions.self, forKey: .options) + ?? TestPlanOptions(codeCoverage: nil, threadSanitizerEnabled: nil, addressSanitizerEnabled: nil) + } } public struct TestPlanOptions: Codable { @@ -23,17 +42,30 @@ public struct TestPlanOptions: Codable { public struct TestPlanTarget: Codable { public var target: TestTargetRef public var parallelizable: Bool? + public var enabled: Bool? public var skippedTests: [SkippedTest]? + public var selectedTests: [SkippedTest]? } public struct TestTargetRef: Codable { public var name: String - public var containerPath: String - public var identifier: String + public var containerPath: String? + public var identifier: String? } +/// A test reference in a plan. Xcode writes these as plain strings (`"Class/test()"`); +/// older tools wrote `{ "identifier": ... }`. Both decode. public struct SkippedTest: Codable { public var identifier: String + + public init(from decoder: Decoder) throws { + if let single = try? decoder.singleValueContainer(), let value = try? single.decode(String.self) { + identifier = value + return + } + let c = try decoder.container(keyedBy: CodingKeys.self) + identifier = try c.decode(String.self, forKey: .identifier) + } } // MARK: - Inspector @@ -58,13 +90,18 @@ public enum TestPlanInspector { var searchedPaths: [String] = [] - let planDir = "\(projectDir)/xcshareddata/xctestplans" - searchedPaths.append("\(planDir)/\(baseName)") - - if let entries = try? FileManager.default.contentsOfDirectory(atPath: planDir), - let match = entries.first(where: { $0.lowercased() == baseName.lowercased() }) - { - return try parse(at: "\(planDir)/\(match)") + // Shared plans live inside the .xcodeproj/.xcworkspace bundle; loose plans sit beside it. + var planDirs = ["\(projectDir)/xcshareddata/xctestplans", projectDir] + if project.hasSuffix(".xcodeproj") || project.hasSuffix(".xcworkspace") { + planDirs.insert("\(project)/xcshareddata/xctestplans", at: 0) + } + for planDir in planDirs { + searchedPaths.append("\(planDir)/\(baseName)") + if let entries = try? FileManager.default.contentsOfDirectory(atPath: planDir), + let match = entries.first(where: { $0.lowercased() == baseName.lowercased() }) + { + return try parse(at: "\(planDir)/\(match)") + } } // Recursive find under projectDir (case-insensitive) @@ -100,10 +137,55 @@ public enum TestPlanInspector { private static func parse(at path: String) throws -> String { let data = try Data(contentsOf: URL(fileURLWithPath: path)) let plan = try JSONDecoder().decode(XCTestPlan.self, from: data) - return format(plan, path: path) + let raw = (try? JSONSerialization.jsonObject(with: data)) ?? [:] + return format(plan, path: path, tagSettings: tagSettings(in: raw)) + } + + /// Every tag-related setting in the plan, as `path: value` lines. Swift Testing tag + /// filters are stored under keys containing "tag"; the exact keys vary by Xcode version, + /// so they are reported verbatim rather than modelled. + static func tagSettings(in node: Any, path: String = "") -> [String] { + var found: [String] = [] + if let dict = node as? [String: Any] { + for key in dict.keys.sorted() { + let value = dict[key] as Any + let childPath = path.isEmpty ? key : "\(path).\(key)" + if key.lowercased().contains("tag") { + found.append("\(childPath): \(compactJSON(value))") + } else { + found += tagSettings(in: value, path: childPath) + } + } + } else if let array = node as? [Any] { + for (i, item) in array.enumerated() { + found += tagSettings(in: item, path: "\(path)[\(i)]") + } + } + return found + } + + private static func compactJSON(_ value: Any) -> String { + if JSONSerialization.isValidJSONObject(value), + let data = try? JSONSerialization.data(withJSONObject: value, options: [.sortedKeys]), + let text = String(data: data, encoding: .utf8) + { + return text + } + return "\(value)" } - private static func format(_ plan: XCTestPlan, path: String) -> String { + /// True when a tag setting lists more than one tag, which is when match-all versus + /// match-any changes which tests run. + static func hasMultiTagFilter(_ settings: [String]) -> Bool { + settings.contains { line in + guard let separator = line.range(of: ": ") else { return false } + // An array literal with a comma directly inside it: two or more tags. + let value = line[separator.upperBound...] + return value.range(of: #"\[[^\[\]]*,[^\[\]]*\]"#, options: .regularExpression) != nil + } + } + + private static func format(_ plan: XCTestPlan, path: String, tagSettings: [String] = []) -> String { var lines: [String] = [] lines.append("Test Plan: \(URL(fileURLWithPath: path).lastPathComponent)") lines.append("Version: \(plan.version)") @@ -125,15 +207,32 @@ public enum TestPlanInspector { for target in plan.testTargets { let skipped = target.skippedTests?.count ?? 0 var targetLine = " \(target.target.name)" + if target.enabled == false { targetLine += " (disabled)" } if let parallel = target.parallelizable { targetLine += " (parallelizable: \(parallel))" } if skipped > 0 { targetLine += " [\(skipped) skipped]" } + if let selected = target.selectedTests, !selected.isEmpty { + targetLine += " [only \(selected.count) selected]" + } lines.append(targetLine) + for test in (target.skippedTests ?? []).prefix(10) { + lines.append(" skip: \(test.identifier)") + } + if skipped > 10 { lines.append(" ... \(skipped - 10) more skipped") } + } + + if !tagSettings.isEmpty { + lines.append("") + lines.append("Tag filters:") + for setting in tagSettings { lines.append(" \(setting)") } + if hasMultiTagFilter(tagSettings) { + lines.append( + " Warning: with several included tags, a plan set to match all runs only tests that carry " + + "every one of them. Compare `xcforge test list --testplan ` with the scheme's count.") + } } lines.append("") - lines.append( - "Note: -enumerate-tests only discovers XCTest methods; Swift Testing @Test suites are not enumerated." - ) + lines.append("Run `xcforge test list --testplan ` to see exactly which tests this plan runs.") return lines.joined(separator: "\n") } diff --git a/Sources/XCForgeKit/Tools/TestProvider.swift b/Sources/XCForgeKit/Tools/TestProvider.swift index 9c9b813..66fff1e 100644 --- a/Sources/XCForgeKit/Tools/TestProvider.swift +++ b/Sources/XCForgeKit/Tools/TestProvider.swift @@ -367,6 +367,7 @@ public enum TestTools { var scheme: String? var simulator: String? var filter: String? + var testplan: String? } // MARK: - Public Result Types for build_and_test @@ -432,6 +433,19 @@ public enum TestTools { public let targetCount: Int public let classCount: Int public let testCount: Int + /// Tests the scheme or test plan disables. Nil when xcodebuild did not report them. + public let disabledTestCount: Int? + + public init( + tests: [TestIdentifier], targetCount: Int, classCount: Int, testCount: Int, + disabledTestCount: Int? = nil + ) { + self.tests = tests + self.targetCount = targetCount + self.classCount = classCount + self.testCount = testCount + self.disabledTestCount = disabledTestCount + } } public static let tools: [Tool] = [ @@ -481,7 +495,7 @@ public enum TestTools { "long": .object([ "type": .string("boolean"), "description": .string( - "Use 1800s timeout instead of the default 180s for suites known to run longer than 3 minutes." + "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." ), ]), "diagnose": .object([ @@ -676,7 +690,7 @@ public enum TestTools { "long": .object([ "type": .string("boolean"), "description": .string( - "Use 1800s timeout instead of the default 180s for suites known to run longer than 3 minutes." + "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." ), ]), "diagnose": .object([ @@ -703,7 +717,7 @@ public enum TestTools { "timeoutSeconds": .object([ "type": .string("integer"), "description": .string( - "Override the xcodebuild timeout in seconds. Takes precedence over 'long'. Default: 180 (or 1800 with long: true)." + "Override the xcodebuild timeout in seconds. Takes precedence over 'long'. Default: 1800 (or 7200 with long: true)." ), ]), "env": .object([ @@ -732,8 +746,8 @@ public enum TestTools { Tool( name: "list_tests", description: """ - List available test identifiers (Target/Class/method) for a scheme. \ - Use this to discover the correct filter format before running test_sim or build_and_test. \ + List test identifiers (Target/Class/method) for a scheme or test plan, XCTest and Swift Testing. \ + Identifiers are in the exact form test_sim and build_and_test accept as filter. \ Builds for testing first (does not run tests). \ Project, scheme, and simulator are auto-detected if omitted. """, @@ -758,6 +772,12 @@ public enum TestTools { "Substring filter on test identifiers. Returns only tests whose Target/Class/method contains this string. Use to verify filter format before running test_sim." ), ]), + "testplan": .object([ + "type": .string("string"), + "description": .string( + "List the tests this test plan runs (its tags and skips applied). Default: .xcforge.yaml testPlan, else the scheme's." + ), + ]), ]), ]) ), @@ -796,7 +816,7 @@ public enum TestTools { /// Resolve the test watchdog timeout via the session actor. /// Precedence: explicit `timeoutSeconds` > `.xcforge.yaml` `testTimeout` - /// > `long ? 1800 : 180`. Centralized so every test/build path picks up + /// > `long ? 7200 : 1800`. Centralized so every test/build path picks up /// the same configurable default. static func resolveTestTimeout( explicit: Int? = nil, long: Bool, env: Environment @@ -837,7 +857,7 @@ public enum TestTools { guard let recorded = LastResultStore.latest(project: resolved, kind: .test) else { return nil } - return await parseCoverage(recorded, env: env) != nil ? recorded : nil + return await parseCoverage(recorded, onlyTargets: true, env: env) != nil ? recorded : nil } let dir = XcodebuildOptions.artifactDirectory() do { @@ -848,7 +868,7 @@ public enum TestTools { .filter { $0.hasPrefix("xcf-test-") && $0.hasSuffix(".xcresult") } for candidate in candidates.prefix(3) { let path = (dir as NSString).appendingPathComponent(candidate) - if await parseCoverage(path, env: env) != nil { + if await parseCoverage(path, onlyTargets: true, env: env) != nil { return path } } @@ -1007,16 +1027,15 @@ public enum TestTools { guard let listResult = try? await executeListTests( - project: project, scheme: scheme, simulator: simulator, env: env + project: project, scheme: scheme, simulator: simulator, testplan: testplan, env: env ) else { hint += "\nCould not enumerate tests to suggest alternatives." return hint } - hint += "\n(found \(listResult.testCount) XCTest identifiers in bundle via -enumerate-tests)" + hint += "\n(found \(listResult.testCount) test identifiers via -enumerate-tests)" if listResult.testCount == 0 { - hint += - "\nThe test bundle appears to be empty. Note: -enumerate-tests only discovers XCTest methods, not Swift Testing @Test suites." + hint += "\nThe test bundle appears to be empty." return hint } let lowered = filter.lowercased() @@ -1212,8 +1231,18 @@ public enum TestTools { let snapshotPath = diagnosticSnapshotPath() let watchdog = HangWatchdog( udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) - let result = try await Xcodebuild.run( + var result = try await Xcodebuild.run( args, environment: childEnvironment, timeout: timeout, env: env) + // The test runner sometimes fails to launch or connect on a busy Mac. The build is + // already done, so one immediate retry is cheap and usually succeeds. + if !result.succeeded, Xcodebuild.timeoutKind(result) == nil, + let symptom = runnerLaunchFailure(Xcodebuild.combinedOutput(result)) + { + Log.warn("Test runner failed to start (\(symptom)); retrying test-without-building once") + _ = try? await env.shell.run("/bin/rm", arguments: ["-rf", resultPath], timeout: 5) + result = try await Xcodebuild.run( + args, environment: childEnvironment, timeout: timeout, env: env) + } watchdog.cancel() let diagResult = await resolvedDiagResult( result: result, diagnose: diagnose, watchdog: watchdog, @@ -1221,6 +1250,23 @@ public enum TestTools { return (result, resultPath, diagResult) } + /// Messages xcodebuild prints when the test runner never started, as opposed to tests + /// failing. Matched case-insensitively. + static let runnerLaunchFailureMarkers = [ + "test runner hung before establishing connection", + "early unexpected exit, operation never finished bootstrapping", + "failed to establish communication with the test runner", + "failed to launch", + ] + + /// The runner-launch symptom found in `output`, or nil. Output that shows any test + /// starting never counts, so a crash mid-suite is not re-run. + static func runnerLaunchFailure(_ output: String) -> String? { + let lower = output.lowercased() + if lower.contains("test case '") || lower.contains("◇ test ") { return nil } + return runnerLaunchFailureMarkers.first { lower.contains($0) } + } + /// Polls for `/Info.plist` existence up to `timeout` seconds in 500 ms increments. /// Returns `true` as soon as the file exists; `false` if timeout is reached. /// Handles the xcodebuild finalization race where the process exits before writing Info.plist. @@ -1334,7 +1380,7 @@ public enum TestTools { } if !buildResult.succeeded && issues.isEmpty { - issues = fallbackBuildIssues(stderr: buildResult.stderr) + issues = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(buildResult)) errorCount = issues.filter { $0.severity == .error }.count warningCount = issues.filter { $0.severity == .warning }.count analyzerWarningCount = issues.filter { $0.severity == .analyzerWarning }.count @@ -1562,18 +1608,30 @@ public enum TestTools { } /// Parse coverage report via xccov - private static func parseCoverage(_ path: String, env: Environment) async -> String? { + /// Coverage report JSON. `onlyTargets` skips per-file data, which is fast and small; use + /// it when only a yes/no or per-target numbers are needed. A full report too large to + /// hold falls back to the per-target report rather than returning truncated JSON. + private static func parseCoverage(_ path: String, onlyTargets: Bool = false, env: Environment) + async -> String? + { + let limit = 64 * 1024 * 1024 + var arguments = ["xccov", "view", "--report", "--json"] + if onlyTargets { arguments.append("--only-targets") } + arguments.append(path) do { let result = try await env.shell.run( - "/usr/bin/xcrun", - arguments: ["xccov", "view", "--report", "--json", path], - timeout: 30, - outputLimit: 20 * 1024 * 1024 // full coverage reports for large projects can exceed the 2 MB default - ) + "/usr/bin/xcrun", arguments: arguments, timeout: onlyTargets ? 60 : 180, outputLimit: limit) guard result.succeeded else { Log.warn("parseCoverage failed: \(result.stderr)") return nil } + if !onlyTargets, result.stdout.utf8.count >= limit - 8 { + Log.warn("Coverage report exceeds \(limit / 1024 / 1024) MB; reporting per target only") + return await parseCoverage(path, onlyTargets: true, env: env) + } + // --only-targets prints a bare array of targets; wrap it so callers see one shape. + let trimmed = result.stdout.trimmingCharacters(in: .whitespacesAndNewlines) + if trimmed.hasPrefix("[") { return "{\"targets\": \(trimmed)}" } return result.stdout } catch { Log.warn("parseCoverage error: \(error)") @@ -1662,8 +1720,9 @@ public enum TestTools { long: long, diagnose: diagnose, udid: udidForDest, env: env ) - // If build failed, surface it as a test failure - if !buildShellResult.succeeded && buildShellResult.exitCode != -1 { + // If build failed (or was killed by a timeout), surface it as a test failure rather than + // running test-without-building against stale or missing products. + if !buildShellResult.succeeded { let elapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - start) var buildDiagnostics: [BuildIssueObservation]? if let buildJSON = await parseBuildResults(buildResultPath, env: env), @@ -1674,10 +1733,10 @@ public enum TestTools { if !parsed.issues.isEmpty { buildDiagnostics = parsed.issues } } if buildDiagnostics == nil { - buildDiagnostics = fallbackBuildIssues(stderr: buildShellResult.stderr) + buildDiagnostics = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(buildShellResult)) if buildDiagnostics?.isEmpty == true { buildDiagnostics = nil } } - let failures: [TestFailureObservation] + var failures: [TestFailureObservation] if let diag = buildDiagnostics { failures = diag.filter { $0.severity == .error }.map { TestFailureObservation( @@ -1691,7 +1750,7 @@ public enum TestTools { let tail = String(buildShellResult.stderr.suffix(2000)).trimmingCharacters( in: .whitespacesAndNewlines) failures = - tail.isEmpty + tail.isEmpty || buildShellResult.exitCode == -1 ? [] : [ TestFailureObservation( @@ -1702,6 +1761,13 @@ public enum TestTools { ) ] } + if let explanation = Xcodebuild.timeoutExplanation(buildShellResult) { + failures.insert( + TestFailureObservation( + testName: "xcodebuild", testIdentifier: "xcodebuild", message: explanation, + source: "build-for-testing.timeout"), + at: 0) + } return TestExecution( succeeded: false, elapsed: elapsed, @@ -1778,7 +1844,7 @@ public enum TestTools { } } if buildDiagnostics == nil { - buildDiagnostics = fallbackBuildIssues(stderr: testShellResult.stderr) + buildDiagnostics = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(testShellResult)) if buildDiagnostics?.isEmpty == true { buildDiagnostics = nil } } } @@ -1821,7 +1887,9 @@ public enum TestTools { let expectedFailureCount = parsedSummary?.expectedFailureCount ?? 0 // Filter matched nothing → treat as failure so agents don't assume tests passed - let zeroMatchWithFilter = totalTestCount == 0 && filter != nil + // Zero tests ran (filter or test plan selected nothing) → failure, so a run that tested + // nothing never reads as a pass. Without a parsed summary the count is unknown, not zero. + let zeroMatchWithFilter = totalTestCount == 0 && (filter != nil || parsedSummary != nil) let cwd = env.currentDirectoryPath() let repoRoot = RepoRoot.discover(from: cwd) ?? cwd @@ -2283,7 +2351,7 @@ public enum TestTools { } if issues.isEmpty { - issues = fallbackBuildIssues(stderr: buildShellResult.stderr) + issues = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(buildShellResult)) } return BuildAndTestResult( @@ -2566,7 +2634,7 @@ public enum TestTools { if !parsed.issues.isEmpty { buildDiagnostics = parsed.issues } } if buildDiagnostics == nil { - buildDiagnostics = fallbackBuildIssues(stderr: shellResult.stderr) + buildDiagnostics = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(shellResult)) if buildDiagnostics?.isEmpty == true { buildDiagnostics = nil } } } @@ -2606,7 +2674,9 @@ public enum TestTools { let passedTestCount = parsedSummary?.passedTestCount ?? 0 let skippedTestCount = parsedSummary?.skippedTestCount ?? 0 let expectedFailureCount = parsedSummary?.expectedFailureCount ?? 0 - let zeroMatchWithFilter = totalTestCount == 0 && filter != nil + // Zero tests ran (filter or test plan selected nothing) → failure, so a run that tested + // nothing never reads as a pass. Without a parsed summary the count is unknown, not zero. + let zeroMatchWithFilter = totalTestCount == 0 && (filter != nil || parsedSummary != nil) let cwd = env.currentDirectoryPath() let repoRoot = RepoRoot.discover(from: cwd) ?? cwd @@ -2661,6 +2731,7 @@ public enum TestTools { project: String? = nil, scheme: String? = nil, simulator: String? = nil, + testplan: String? = nil, env: Environment = .live ) async throws -> ListTestsResult { let resolvedProject = try await env.session.resolveProject(project) @@ -2687,17 +2758,42 @@ public enum TestTools { Log.warn("Could not parse Xcode version from: \(xcodeVersionResult.stdout.prefix(100))") } - // xcodebuild test -enumerate-tests builds for testing then lists tests without running them + // xcodebuild test -enumerate-tests builds for testing then lists tests without running + // them. The JSON output covers XCTest and Swift Testing alike; the text output below is + // only a fallback for Xcode versions that ignore the format flags. + let jsonPath = XcodebuildOptions.uniqueArtifactPath(prefix: "tests", extension: "json") + defer { try? FileManager.default.removeItem(atPath: jsonPath) } + var enumerateArgs = [ + projectFlag, resolvedProject, + "-scheme", resolvedScheme, + "-destination", destination, + "-skipMacroValidation", + ] + if let plan = await env.session.resolveTestPlan(testplan) { + enumerateArgs += ["-testPlan", plan] + } + enumerateArgs += [ + "-enumerate-tests", + "-test-enumeration-style", "flat", + "-test-enumeration-format", "json", + "-test-enumeration-output-path", jsonPath, + "test", + ] let enumerateResult = try await Xcodebuild.run( - [ - projectFlag, resolvedProject, - "-scheme", resolvedScheme, - "-destination", destination, - "-skipMacroValidation", - "-enumerate-tests", - "test", - ], timeout: 1800, env: env - ) + enumerateArgs, timeout: await resolveTestTimeout(long: true, env: env), env: env) + + if let data = FileManager.default.contents(atPath: jsonPath), + let parsed = parseTestEnumerationJSON(data), !parsed.enabled.isEmpty || !parsed.disabled.isEmpty + { + let tests = parsed.enabled.compactMap(testIdentifier(from:)) + return ListTestsResult( + tests: tests, + targetCount: Set(tests.map(\.target)).count, + classCount: Set(tests.map { "\($0.target)/\($0.className)" }).count, + testCount: tests.count, + disabledTestCount: parsed.disabled.count + ) + } guard enumerateResult.succeeded || !enumerateResult.stdout.isEmpty else { let errorLines = enumerateResult.stderr.split(separator: "\n") @@ -2841,6 +2937,42 @@ public enum TestTools { ) } + /// Collect test identifiers from `-test-enumeration-format json` output. Tolerant of the + /// exact layout: any object with an `identifier` string counts, and it is disabled when + /// it sits under a key containing "disabled". + static func parseTestEnumerationJSON(_ data: Data) -> (enabled: [String], disabled: [String])? { + guard let root = try? JSONSerialization.jsonObject(with: data) else { return nil } + var enabled: [String] = [] + var disabled: [String] = [] + var seen = Set() + func walk(_ node: Any, disabledBranch: Bool) { + if let dict = node as? [String: Any] { + if let id = dict["identifier"] as? String, seen.insert(id + "|\(disabledBranch)").inserted { + if disabledBranch { disabled.append(id) } else { enabled.append(id) } + } + for (key, value) in dict where key != "identifier" { + walk(value, disabledBranch: disabledBranch || key.lowercased().contains("disabled")) + } + } else if let array = node as? [Any] { + for item in array { walk(item, disabledBranch: disabledBranch) } + } + } + walk(root, disabledBranch: false) + return (enabled, disabled) + } + + /// Split `Target/Suite[/Nested]/test()` into its parts. The full identifier is kept exactly + /// as xcodebuild printed it, which is the form `-only-testing` accepts. + static func testIdentifier(from id: String) -> TestIdentifier? { + let parts = id.split(separator: "/").map(String.init) + guard parts.count >= 2 else { return nil } + var method = parts.count >= 3 ? parts[parts.count - 1] : "" + if method.hasSuffix("()") { method = String(method.dropLast(2)) } + let className = parts.count >= 3 ? parts[1..<(parts.count - 1)].joined(separator: "/") : parts[1] + return TestIdentifier( + target: parts[0], className: className, methodName: method, fullIdentifier: id) + } + // MARK: - Tool Implementations static func testSim(_ args: [String: Value]?, env: Environment) async -> CallTool.Result { @@ -2926,7 +3058,7 @@ public enum TestTools { let icon = execution.succeeded ? "PASSED" : "FAILED" if execution.totalTestCount == 0 { - lines.append("No tests matched in \(execution.elapsed)s") + lines.append("No tests ran (\(execution.elapsed)s): the filter, scheme or test plan selected none") } else { lines.append("Tests \(icon) in \(execution.elapsed)s") lines.append( @@ -2970,7 +3102,8 @@ public enum TestTools { } } - let zeroMatchWithFilter = execution.totalTestCount == 0 && input.filter != nil + let zeroMatchWithFilter = + execution.totalTestCount == 0 && (input.filter != nil || execution.hasStructuredSummary) let text = lines.joined(separator: "\n") return (execution.succeeded && !zeroMatchWithFilter) ? .ok(text) : .fail(text) } catch { @@ -3303,6 +3436,7 @@ public enum TestTools { project: input.project, scheme: input.scheme, simulator: input.simulator, + testplan: input.testplan, env: env ) return formatListTests(result, filter: input.filter) @@ -3415,7 +3549,7 @@ public enum TestTools { } lines.append("Build: OK (\(result.buildElapsed)s)") if test.totalTestCount == 0 { - lines.append("No tests matched in \(test.elapsed)s") + lines.append("No tests ran (\(test.elapsed)s): the filter, scheme or test plan selected none") } else { let icon = test.succeeded ? "PASSED" : "FAILED" lines.append("Tests \(icon) in \(test.elapsed)s") @@ -3443,7 +3577,8 @@ public enum TestTools { } let text = lines.joined(separator: "\n") + suffix // Filter matched nothing → treat as failure so agents don't assume tests passed - let zeroMatchWithFilter = test.totalTestCount == 0 && filter != nil + let zeroMatchWithFilter = + test.totalTestCount == 0 && (filter != nil || test.hasStructuredSummary) return (test.succeeded && !zeroMatchWithFilter) ? .ok(text) : .fail(text) } @@ -3490,6 +3625,9 @@ public enum TestTools { ) } lines.append("") + if let disabled = result.disabledTestCount, disabled > 0 { + lines.insert("\(disabled) more disabled by the scheme or test plan", at: lines.count - 1) + } // Group by target/class for readability var grouped: [String: [String: [String]]] = [:] // target -> class -> methods @@ -3901,10 +4039,22 @@ public enum TestTools { ) } + /// Parse `file:line:col: error: message` lines from xcodebuild output (stdout and/or + /// stderr). The same diagnostic is often printed several times (once per architecture or + /// by both the compiler and the build summary), so duplicates are dropped, keeping order. static func fallbackBuildIssues(stderr: String) -> [BuildIssueObservation] { - stderr - .split(separator: "\n") - .compactMap { parseFallbackBuildIssue(String($0)) } + var seen = Set() + var issues: [BuildIssueObservation] = [] + for line in stderr.split(separator: "\n") { + guard let issue = parseFallbackBuildIssue(String(line)) else { continue } + let loc = issue.location + let key = [ + issue.severity == .error ? "e" : "w", loc?.filePath ?? "", loc?.line.map(String.init) ?? "", + loc?.column.map(String.init) ?? "", issue.message, + ].joined(separator: "|") + if seen.insert(key).inserted { issues.append(issue) } + } + return issues } private static func extractExecutionFailureMessage(stderr: String) -> String? { diff --git a/Tests/XCForgeKitTests/BuildTestReliabilityTests.swift b/Tests/XCForgeKitTests/BuildTestReliabilityTests.swift new file mode 100644 index 0000000..023f54c --- /dev/null +++ b/Tests/XCForgeKitTests/BuildTestReliabilityTests.swift @@ -0,0 +1,129 @@ +import Foundation +import Testing + +@testable import XCForgeKit + +@Suite("Build and test reliability: errors, runner retries, test listing, test plans") +struct BuildTestReliabilityTests { + + // MARK: - Error extraction + + @Test("duplicate compiler diagnostics collapse to one, keeping order") + func fallbackIssuesDeduplicate() { + let output = """ + /src/A.swift:3:5: error: cannot find 'x' in scope + x += 1 + ^ + /src/A.swift:3:5: error: cannot find 'x' in scope + /src/B.swift:9:1: error: missing return + /src/A.swift:3:5: warning: unused value + """ + let issues = TestTools.fallbackBuildIssues(stderr: output) + #expect(issues.map(\.message) == ["cannot find 'x' in scope", "missing return", "unused value"]) + #expect(issues.filter { $0.severity == .error }.count == 2) + } + + @Test("combined output reads compiler errors xcodebuild printed on stdout") + func combinedOutputIncludesStdout() { + let result = ShellResult( + stdout: "/src/A.swift:1:1: error: boom", stderr: "** BUILD FAILED **", exitCode: 65) + let issues = TestTools.fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(result)) + #expect(issues.count == 1) + #expect(issues.first?.location?.filePath == "/src/A.swift") + } + + // MARK: - Runner launch failures + + @Test("runner launch failures are recognised only when no test started") + func runnerLaunchFailureDetection() { + #expect( + TestTools.runnerLaunchFailure( + "Testing failed:\n\tTest runner hung before establishing connection.") != nil) + #expect( + TestTools.runnerLaunchFailure( + "Early unexpected exit, operation never finished bootstrapping - no restart will be attempted") + != nil) + #expect( + TestTools.runnerLaunchFailure( + "Test case 'A.b()' passed on 'iPhone'\nFailed to launch app with identifier") == nil) + #expect(TestTools.runnerLaunchFailure("Testing failed: XCTAssertEqual failed") == nil) + } + + // MARK: - Test enumeration JSON + + @Test("enumeration JSON yields enabled and disabled identifiers in any nesting") + func enumerationJSON() throws { + let json = """ + {"values": [{"testPlan": "Unit", + "enabledTests": [{"identifier": "AppTests/LoginTests/testLogin"}, + {"identifier": "AppTests/Parser/Nested/parsesDates()"}], + "disabledTests": [{"identifier": "AppTests/SlowTests/testSlow"}]}]} + """ + let parsed = try #require(TestTools.parseTestEnumerationJSON(Data(json.utf8))) + #expect(parsed.enabled == ["AppTests/LoginTests/testLogin", "AppTests/Parser/Nested/parsesDates()"]) + #expect(parsed.disabled == ["AppTests/SlowTests/testSlow"]) + } + + @Test("identifiers split into target, suite and test, keeping the exact form") + func identifierSplitting() throws { + let swiftTesting = try #require(TestTools.testIdentifier(from: "AppTests/Parser/Nested/parsesDates()")) + #expect(swiftTesting.target == "AppTests") + #expect(swiftTesting.className == "Parser/Nested") + #expect(swiftTesting.methodName == "parsesDates") + #expect(swiftTesting.fullIdentifier == "AppTests/Parser/Nested/parsesDates()") + let suiteOnly = try #require(TestTools.testIdentifier(from: "AppTests/Parser")) + #expect(suiteOnly.className == "Parser") + #expect(TestTools.testIdentifier(from: "AppTests") == nil) + } + + // MARK: - Test plans + + @Test("Xcode's string skippedTests and missing sections decode") + func testPlanXcodeFormat() throws { + let json = """ + { + "configurations": [{"id": "1", "name": "Config 1", "options": {}}], + "defaultOptions": {}, + "testTargets": [{ + "skippedTests": ["SlowTests", "LoginTests/testFlaky()"], + "target": {"containerPath": "container:App.xcodeproj", "identifier": "X", "name": "AppTests"} + }], + "version": 1 + } + """ + let plan = try JSONDecoder().decode(XCTestPlan.self, from: Data(json.utf8)) + #expect(plan.testTargets.first?.skippedTests?.map(\.identifier) == ["SlowTests", "LoginTests/testFlaky()"]) + let minimal = try JSONDecoder().decode(XCTestPlan.self, from: Data(#"{"version": 1}"#.utf8)) + #expect(minimal.testTargets.isEmpty) + } + + @Test("tag settings are reported verbatim and multi-tag filters are flagged") + func testPlanTags() throws { + let json = """ + {"defaultOptions": {"testTags": {"included": ["fast", "unit"], "matchAll": true}}, + "testTargets": [{"target": {"name": "AppTests"}, "skippedTags": ["slow"]}]} + """ + let raw = try JSONSerialization.jsonObject(with: Data(json.utf8)) + let settings = TestPlanInspector.tagSettings(in: raw) + #expect(settings.contains { $0.hasPrefix("defaultOptions.testTags:") && $0.contains("fast") }) + #expect(settings.contains { $0.hasPrefix("testTargets[0].skippedTags:") }) + #expect(TestPlanInspector.hasMultiTagFilter(settings)) + #expect(!TestPlanInspector.hasMultiTagFilter(["testTargets[0].skippedTags: [\"slow\"]"])) + } + + @Test("shared plans inside the .xcodeproj bundle are found") + func testPlanInsideProjectBundle() async throws { + let root = FileManager.default.temporaryDirectory + .appendingPathComponent("xcforge-plan-\(UUID().uuidString)") + let project = root.appendingPathComponent("App.xcodeproj") + let plans = project.appendingPathComponent("xcshareddata/xctestplans", isDirectory: true) + try FileManager.default.createDirectory(at: plans, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: root) } + try #"{"version": 1, "testTargets": [{"target": {"name": "AppTests"}}]}"#.write( + to: plans.appendingPathComponent("Unit.xctestplan"), atomically: true, encoding: .utf8) + + let summary = try await TestPlanInspector.inspectTestPlan( + name: "Unit", project: project.path, env: .live) + #expect(summary.contains("AppTests")) + } +} diff --git a/Tests/XCForgeKitTests/RepoConfigTests.swift b/Tests/XCForgeKitTests/RepoConfigTests.swift index 25bdf39..3d699cd 100644 --- a/Tests/XCForgeKitTests/RepoConfigTests.swift +++ b/Tests/XCForgeKitTests/RepoConfigTests.swift @@ -679,7 +679,7 @@ struct RepoConfigTests { #expect(result?.testTimeout == nil) } - @Test("resolveTestTimeout: explicit > testTimeout > long > 180") + @Test("resolveTestTimeout: explicit > testTimeout > long > 1800") func testTimeoutPrecedence() async { let root = makeTempDir() defer { cleanup(root) } @@ -699,7 +699,7 @@ struct RepoConfigTests { #expect(await session.resolveTestTimeout(explicit: nil, long: true) == 600) } - @Test("resolveTestTimeout falls back to 180/1800 when no repo testTimeout") + @Test("resolveTestTimeout falls back to 1800/7200 when no repo testTimeout") func testTimeoutBaseline() async { let root = makeTempDir() defer { cleanup(root) } @@ -711,8 +711,8 @@ struct RepoConfigTests { let store = DefaultsStore(baseDirectory: storeDir) let session = SessionState(defaultsStore: store, cwd: root.path) - #expect(await session.resolveTestTimeout(explicit: nil, long: false) == 180) - #expect(await session.resolveTestTimeout(explicit: nil, long: true) == 1800) + #expect(await session.resolveTestTimeout(explicit: nil, long: false) == 1800) + #expect(await session.resolveTestTimeout(explicit: nil, long: true) == 7200) } // MARK: - autoPromote parsing & opt-out @@ -881,8 +881,8 @@ struct RepoConfigTests { let store2 = DefaultsStore( baseDirectory: noTimeoutRoot.appendingPathComponent("store", isDirectory: true)) let session2 = SessionState(defaultsStore: store2, cwd: noTimeoutRoot.path) - #expect(await session2.resolveTestTimeout(explicit: 0, long: false) == 180) - #expect(await session2.resolveTestTimeout(explicit: 0, long: true) == 1800) + #expect(await session2.resolveTestTimeout(explicit: 0, long: false) == 1800) + #expect(await session2.resolveTestTimeout(explicit: 0, long: true) == 7200) } // MARK: - P9 setDefaults resets matching streak diff --git a/Tests/XCForgeKitTests/SharedMacSafetyTests.swift b/Tests/XCForgeKitTests/SharedMacSafetyTests.swift index 7fb96c7..1ac0f91 100644 --- a/Tests/XCForgeKitTests/SharedMacSafetyTests.swift +++ b/Tests/XCForgeKitTests/SharedMacSafetyTests.swift @@ -23,14 +23,15 @@ struct SharedMacSafetyTests { let result = Xcodebuild.apply(options, to: args) #expect( result == [ - "-project", "A.xcodeproj", "-scheme", "A", "-derivedDataPath", "/dd", "-jobs", "6", "build", + "-project", "A.xcodeproj", "-scheme", "A", "-derivedDataPath", "/dd", + "-IDEBuildingContinueBuildingAfterErrors=YES", "-jobs", "6", "build", "COMPILATION_CACHE_ENABLE_CACHING=YES", ]) } @Test("apply leaves -list and an existing -derivedDataPath alone") func applySkipsNonBuildAndDuplicates() { - let options = XcodebuildOptions(derivedDataPath: "/dd") + let options = XcodebuildOptions(derivedDataPath: "/dd", continueAfterErrors: false) let list = ["-project", "A.xcodeproj", "-list", "-json"] #expect(Xcodebuild.apply(options, to: list) == list) let explicit = ["-derivedDataPath", "/mine", "build"] @@ -45,6 +46,60 @@ struct SharedMacSafetyTests { #expect(Xcodebuild.needsLock(["-scheme", "A", "test-without-building"])) } + @Test("continue-after-errors is on for compiling actions only, and can be turned off") + func continueAfterErrors() { + let flag = "-IDEBuildingContinueBuildingAfterErrors=YES" + #expect(Xcodebuild.apply(XcodebuildOptions(), to: ["build"]) == [flag, "build"]) + #expect(Xcodebuild.apply(XcodebuildOptions(), to: ["build-for-testing"]).contains(flag)) + #expect(!Xcodebuild.apply(XcodebuildOptions(), to: ["test-without-building"]).contains(flag)) + #expect(!Xcodebuild.apply(XcodebuildOptions(), to: ["-showBuildSettings"]).contains(flag)) + #expect(Xcodebuild.apply(XcodebuildOptions(continueAfterErrors: false), to: ["build"]) == ["build"]) + let explicit = ["-IDEBuildingContinueBuildingAfterErrors=NO", "build"] + #expect(Xcodebuild.apply(XcodebuildOptions(), to: explicit) == explicit) + } + + // MARK: - Timeouts + + @Test("timeout kind distinguishes idle kills from total-limit kills") + func timeoutKinds() { + let idle = ShellResult( + stdout: "", stderr: "Process produced no output for 600s and was killed (idle timeout)", + exitCode: -1) + let total = ShellResult( + stdout: "", stderr: "Process timed out after 1800s and was killed (signal 15)", exitCode: -1) + let failed = ShellResult(stdout: "", stderr: "error", exitCode: 65) + #expect(Xcodebuild.timeoutKind(idle) == .idle) + #expect(Xcodebuild.timeoutKind(total) == .total) + #expect(Xcodebuild.timeoutKind(failed) == nil) + #expect(Xcodebuild.timeoutExplanation(idle)?.contains("idleTimeoutSeconds") == true) + #expect(Xcodebuild.timeoutExplanation(failed) == nil) + } + + @Test("idle timeout kills a silent process but not a chatty one") + func idleTimeoutKillsSilentProcess() async throws { + let silent = try await Shell.run( + "/bin/sleep", arguments: ["30"], timeout: 60, idleTimeout: 0.5) + #expect(silent.exitCode == -1) + #expect(silent.stderr.contains("idle timeout")) + + let chatty = try await Shell.run( + "/bin/sh", arguments: ["-c", "for i in 1 2 3 4 5 6; do echo $i; sleep 0.2; done"], + timeout: 60, idleTimeout: 0.8) + #expect(chatty.succeeded) + #expect(chatty.stdout.contains("6")) + } + + @Test("idle timeout options parse from MCP arguments and the environment") + func idleTimeoutOptions() { + let parsed = XcodebuildOptions.fromMCPArguments([ + "idleTimeoutSeconds": .int(120), "continueAfterErrors": .bool(false), + ]) + #expect(parsed.idleTimeoutSeconds == 120) + #expect(parsed.continueAfterErrors == false) + let fromEnv = XcodebuildOptions.effective(cwd: "/", environment: ["XCFORGE_IDLE_TIMEOUT": "0"]) + #expect(fromEnv.idleTimeoutSeconds == 0) + } + // MARK: - Result bundle paths @Test("generated result bundle paths never collide") @@ -116,9 +171,13 @@ struct SharedMacSafetyTests { #expect(options.lockWaitSeconds == 30) let tool = ToolRegistry.allTools.first { $0.name == "build_and_test" } - let properties = tool?.inputSchema.objectValue?["properties"]?.objectValue - #expect(properties?["derivedDataPath"] != nil) - #expect(properties?["buildLock"] != nil) + var properties: [String: Value] = [:] + if case .object(let schema)? = tool?.inputSchema, case .object(let props)? = schema["properties"] { + properties = props + } + #expect(properties["derivedDataPath"] != nil) + #expect(properties["buildLock"] != nil) + #expect(properties["idleTimeoutSeconds"] != nil) } // MARK: - Build lock From 5ff3a1005d3aee37f9521ecee040c998d4c24f2d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 14:30:30 +0000 Subject: [PATCH 03/28] feat: run WebDriverAgent on physical devices and support Xcode 27 Device Hub Adds wda start/status/stop (CLI and MCP) to build, sign, launch and reach xcforgeWDA on a device, device screenshots, deep links and env vars on device launch, clear messages for locked devices, UI Automation off and signing failures, and stops simulator self-heal from running against device targets. Opens Device Hub when Simulator.app is gone and accepts both apps in capture and accessibility code. Raises xcforgeWDA's deployment target to iOS 15. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01NVNakyntY4u9MvhsjbzRhZ --- CHANGELOG.md | 7 + README.md | 6 +- Skills/xcforge/references/device-tools.md | 46 ++- .../Commands/Device/DeviceCommand.swift | 53 ++- .../XCForgeCLI/Commands/WDA/WDACommand.swift | 131 +++++++ Sources/XCForgeCLI/XCForgeCLI.swift | 2 +- Sources/XCForgeKit/Clients/AXPBridge.swift | 4 +- Sources/XCForgeKit/Clients/AgentClient.swift | 77 ++-- .../Clients/FramebufferCapture.swift | 2 +- Sources/XCForgeKit/Support/DeviceWDA.swift | 338 ++++++++++++++++++ .../XCForgeKit/Support/ProjectResolver.swift | 2 +- Sources/XCForgeKit/Support/SimulatorApp.swift | 65 ++++ .../XCForgeKit/Support/WDASessionRepair.swift | 20 +- Sources/XCForgeKit/Tools/BuildProvider.swift | 7 +- Sources/XCForgeKit/Tools/DeviceProvider.swift | 221 +++++++++++- .../Tools/InteractionProvider.swift | 5 +- .../XCForgeKit/Tools/MultiDeviceTools.swift | 10 +- .../XCForgeKitTests/DeviceSupportTests.swift | 114 ++++++ .../WebDriverAgent.xcodeproj/project.pbxproj | 8 +- .../xcforgeWDA.xcodeproj/project.pbxproj | 8 +- 20 files changed, 1052 insertions(+), 74 deletions(-) create mode 100644 Sources/XCForgeCLI/Commands/WDA/WDACommand.swift create mode 100644 Sources/XCForgeKit/Support/DeviceWDA.swift create mode 100644 Sources/XCForgeKit/Support/SimulatorApp.swift create mode 100644 Tests/XCForgeKitTests/DeviceSupportTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 23d085a..a739b68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,8 +15,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - `test-without-building` is retried once when the test runner never started ("Test runner hung before establishing connection", "Early unexpected exit…", "Failed to launch…" with no test begun) - `test list --testplan` / `list_tests testplan`: lists via `-test-enumeration-format json`, which includes Swift Testing tests and reports how many the scheme or plan disables - `test_plan_inspect` shows skipped tests, selected-only targets and every tag filter in the plan, and warns when a multi-tag filter may require all tags +- **UI automation on physical devices.** `xcforge wda start|status|stop` and MCP `wda_start`/`wda_stop` build xcforgeWDA for the device, sign it with your team (`--team`, `XCFORGE_WDA_TEAM`), launch it with `test-without-building`, find it over the CoreDevice tunnel, and record its URL in `~/.xcforge/wda/`. `XCFORGE_DEVICE=` points `xcforge ui` commands at it. Locked device, UI Automation off, Developer Mode off, pairing and signing failures each get a one-line explanation +- `device screenshot` / `device_screenshot` (devicectl capture, falling back to the device's WDA) +- `device launch --url --env KEY=VALUE --arg A` and matching `url`/`env` on `device_launch` +- **Xcode 27 Device Hub support.** xcforge opens Device Hub when the selected Xcode has no Simulator.app, and screen capture, the accessibility bridge and window scripting accept either app ### Changed +- UI tools no longer run simulator recovery (terminate, rebuild, redeploy) when WDA is on a physical device; they report `device_runner_not_reachable` and point at `wda start` +- xcforgeWDA's iOS deployment target is 15.0 (Xcode 27 no longer builds for 13) +- devicectl output is read from Xcode 27's `properties` dictionary as well as the older split keys; `device info` adds tunnel, pairing and Developer Mode state - **Default total time limit for builds and tests is 1800s** (was 180s); `--long` / `long: true` raises it to 7200s. Hangs are caught by the idle timeout instead - **Builds continue after the first error** (`-IDEBuildingContinueBuildingAfterErrors=YES`), so one run reports every error. Opt out with `--no-continue-after-errors` / `continueAfterErrors: false` - Build errors are read from stdout as well as stderr (xcodebuild prints compiler diagnostics on stdout), duplicates are dropped, and up to 50 are shown diff --git a/README.md b/README.md index 4bd6b63..76ba981 100644 --- a/README.md +++ b/README.md @@ -219,7 +219,7 @@ Every tool available over MCP has a matching CLI command. Every CLI command supp | **Build** | 6 | `build_sim`, `build_compile` (fast compile-only), `build_run_sim`, `clean`, project/scheme discovery | | **Test** | 7 | `test_sim` with xcresult parsing, `test_failures`, `test_coverage`, `list_tests`, `test_plan_inspect`, agent output mode, known-failures gating | | **Simulator** | 18 | Full lifecycle + video recording, location simulation, dark mode toggle, status bar override, info | -| **Physical Devices** | 7 | Via `devicectl` — list, install, launch, screenshot, pair | +| **Physical Devices** | 10 | Via `devicectl` — list, install, launch (deep links, env), screenshot, WebDriverAgent on device | | **UI Automation** | 20 | WebDriverAgent + native AX bridge — find, tap (point or pixel), swipe, drag, type, alerts | | **Screenshots** | 3 | Framebuffer capture (0.3s), point-space coordinate alignment, optional grid overlay | | **Visual Regression** | 3 | Pixel-diff baselines, multi-device checks (Dark Mode, Landscape, iPad), `bless` workflow | @@ -285,7 +285,7 @@ Use alongside logs and screenshots for systematic root-cause analysis. ### Physical Device Support -7 tools wrapping Apple's `devicectl` for real devices — list connected devices, install/launch/terminate apps, take screenshots, and manage pairing. +Tools wrapping Apple's `devicectl` for real devices: list connected devices, install, launch (with deep links and environment variables) and terminate apps, and take screenshots. `xcforge wda start --device --team ` (MCP `wda_start`) builds, signs and starts WebDriverAgent on the device so the UI automation tools work there too; see [device-tools.md](Skills/xcforge/references/device-tools.md). --- @@ -394,7 +394,7 @@ Then add `XCFORGE_WDA_DIR` to your MCP config: } ``` -xcforge builds and deploys WDA automatically on the first UI tool call (~30–60s on first run, instant thereafter). See the [full UI Automation Setup guide](https://xcforge.dev/docs/guides/ui-automation-setup) for alternative configurations, custom ports, and troubleshooting. +xcforge builds and deploys WDA automatically on the first UI tool call on a simulator (~30–60s on first run, instant thereafter). For a physical device run `xcforge wda start --device --team ` (MCP `wda_start`) first, with the device unlocked and Settings > Developer > Enable UI Automation on. See the [full UI Automation Setup guide](https://xcforge.dev/docs/guides/ui-automation-setup) for alternative configurations, custom ports, and troubleshooting. --- diff --git a/Skills/xcforge/references/device-tools.md b/Skills/xcforge/references/device-tools.md index 4852c42..21ec0c0 100644 --- a/Skills/xcforge/references/device-tools.md +++ b/Skills/xcforge/references/device-tools.md @@ -1,4 +1,4 @@ -# Device Tools (7 tools) +# Device Tools (10 tools) Physical iOS/iPadOS device management via `xcrun devicectl`. All tools require a connected device (USB or WiFi). @@ -64,6 +64,10 @@ Launch an app on a connected physical device. | `terminate_existing` | No | true | Terminate existing instance before launching | | `timeout` | No | 30 | Console timeout in seconds | | `arguments` | No | — | Array of arguments passed to the app | +| `url` | No | — | Deep link or universal link opened in the app at launch | +| `env` | No | — | Object of environment variables for the app | + +CLI: `xcforge device launch --device [--url myapp://x] [--env KEY=VALUE] [--arg A]`. **Returns:** Launch confirmation and optional console output (if `console: true`). @@ -91,3 +95,43 @@ List apps installed on a connected physical device. | `bundle_id` | No | — | Filter to a specific bundle ID | **Returns:** Array of apps with bundle ID, name, and version. + +--- + +## device_screenshot + +Save a PNG of the device screen. Uses `devicectl device capture screenshot` (Xcode 26.6 and later), then the device's WebDriverAgent when `wda_start` has one running. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `device` | **Yes** | — | Device name or UDID | +| `path` | No | artifact dir | Where to write the PNG | + +CLI: `xcforge device screenshot --device [--output shot.png]`. + +--- + +## wda_start / wda_stop (UI automation on a real device) + +`wda_start` builds xcforgeWDA for the device, signs it with your team (`-allowProvisioningUpdates`), launches it with `xcodebuild test-without-building` in the background, and finds its URL (the CoreDevice tunnel address, else the URL WDA logs). The MCP session's UI tools (`find_element`, `click_element`, `type_text`, ...) then target the device. A runner that already answers is reused. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `device` | **Yes** | — | Device name or UDID | +| `team` | No | `XCFORGE_WDA_TEAM` | Development team ID for signing | +| `bundle_id` | No | `com.xcforge.wda..runner` | Runner bundle id | +| `port` | No | 8100 | Port WDA listens on | + +CLI: + +```bash +xcforge wda start --device --team ABCDE12345 +export XCFORGE_DEVICE= # ui commands now use the device's runner +xcforge ui tap --label "Sign In" +xcforge wda status +xcforge wda stop --device +``` + +Before starting, the device must be unlocked, in Developer Mode, and have **Settings > Developer > Enable UI Automation** on. Failures name which of these (or signing, pairing, an untrusted certificate) is the problem. UI tools never run simulator recovery against a device: when its runner stops answering they say so and point at `wda start`. + +State lives in `~/.xcforge/wda/.json` (URL, runner pid, log path), so other sessions on the Mac reuse the same runner. The build goes through the shared build lock when one is configured. diff --git a/Sources/XCForgeCLI/Commands/Device/DeviceCommand.swift b/Sources/XCForgeCLI/Commands/Device/DeviceCommand.swift index f72edf0..fdf8228 100644 --- a/Sources/XCForgeCLI/Commands/Device/DeviceCommand.swift +++ b/Sources/XCForgeCLI/Commands/Device/DeviceCommand.swift @@ -9,7 +9,7 @@ struct Device: AsyncParsableCommand { subcommands: [ DeviceList.self, DeviceInfo.self, DeviceInstall.self, DeviceUninstall.self, DeviceLaunch.self, DeviceTerminate.self, - DeviceApps.self, + DeviceApps.self, DeviceScreenshot.self, ], defaultSubcommand: DeviceList.self ) @@ -160,19 +160,41 @@ struct DeviceLaunch: AsyncParsableCommand { @Option(help: "Console timeout in seconds (default: 30). Only used with --console.") var timeout: Int = 30 + @Option(help: "Open this URL (deep link or universal link) in the app at launch.") + var url: String? + + @Option( + name: .customLong("env"), parsing: .unconditionalSingleValue, + help: ArgumentHelp("Environment variable for the app as KEY=VALUE (repeatable).", valueName: "KEY=VALUE")) + var environment: [String] = [] + + @Option( + name: .customLong("arg"), parsing: .unconditionalSingleValue, + help: ArgumentHelp("Launch argument for the app (repeatable).", valueName: "ARG")) + var arguments: [String] = [] + @Flag(help: "Emit the result as machine-readable JSON.") var json = false mutating func run() async throws { let useJSON = shouldOutputJSON(flag: json) let env = Environment.live + var variables: [String: String] = [:] + for entry in environment { + guard let eq = entry.firstIndex(of: "=") else { + throw ValidationError("--env expects KEY=VALUE, got '\(entry)'") + } + variables[String(entry[.. (or WDA_BASE_URL=). + The device must be unlocked, in Developer Mode, with Settings > Developer > Enable UI + Automation turned on. + """, + subcommands: [WDAStart.self, WDAStatus.self, WDAStop.self], + defaultSubcommand: WDAStatus.self + ) +} + +struct WDAStart: AsyncParsableCommand { + static let configuration = CommandConfiguration( + commandName: "start", + abstract: "Build, sign and start WebDriverAgent on a device (reuses a runner that already answers)." + ) + + @Option(help: "Device name or UDID.") + var device: String + + @Option(help: "Apple development team ID for signing. Default: XCFORGE_WDA_TEAM.") + var team: String? + + @Option(help: "Runner bundle id. Default: com.xcforge.wda..runner.") + var bundleId: String? + + @Option(help: "Port WDA listens on.") + var port: Int = DeviceWDA.defaultPort + + @Option(help: "Seconds to wait for the runner to answer after launch.") + var startTimeout: Int = 120 + + @Flag(help: "Emit the result as machine-readable JSON.") + var json = false + + @OptionGroup var xcodebuild: XcodebuildOptionGroup + + mutating func run() async throws { + let command = self + try await xcodebuild.scoped { try await command.execute() } + } + + func execute() async throws { + let state: DeviceWDA.State + do { + state = try await DeviceWDA.start( + device: device, team: team, bundleID: bundleId, port: port, + startTimeout: TimeInterval(startTimeout), env: Environment.live) + } catch { + fputs("\(error)\n", stderr) + throw ExitCode.failure + } + if shouldOutputJSON(flag: json) { + print(try WorkflowJSONRenderer.renderJSON(state)) + } else { + print("WebDriverAgent running on \(state.name ?? state.udid) at \(state.url)") + print("Use it from UI commands with:") + print(" export XCFORGE_DEVICE=\(state.udid)") + print("Runner log: \(state.logPath)") + } + } +} + +struct WDAStatus: AsyncParsableCommand { + static let configuration = CommandConfiguration( + commandName: "status", + abstract: "Show device runners started with `wda start` and whether they answer." + ) + + @Option(help: "Only this device (name or UDID).") + var device: String? + + @Flag(help: "Emit the result as machine-readable JSON.") + var json = false + + struct Entry: Codable { + let udid: String + let name: String? + let url: String + let pid: Int32 + let healthy: Bool + let logPath: String + } + + mutating func run() async throws { + var states = DeviceWDA.all() + if let device { states = states.filter { $0.udid == device || $0.name == device } } + var entries: [Entry] = [] + for state in states { + entries.append( + Entry( + udid: state.udid, name: state.name, url: state.url, pid: state.pid, + healthy: await DeviceWDA.isHealthy(state.url), logPath: state.logPath)) + } + if shouldOutputJSON(flag: json) { + print(try WorkflowJSONRenderer.renderJSON(entries)) + return + } + if entries.isEmpty { + print("No device runners recorded. Start one with `xcforge wda start --device `.") + return + } + for e in entries { + print("\(e.name ?? e.udid): \(e.healthy ? "answering" : "NOT answering") at \(e.url) (pid \(e.pid))") + } + } +} + +struct WDAStop: AsyncParsableCommand { + static let configuration = CommandConfiguration( + commandName: "stop", + abstract: "Stop the WebDriverAgent runner started on a device." + ) + + @Option(help: "Device name or UDID.") + var device: String + + mutating func run() async throws { + let known = DeviceWDA.load(device: device) != nil + await DeviceWDA.stop(device: device, env: Environment.live) + print(known ? "Stopped WebDriverAgent on \(device)" : "No WebDriverAgent recorded for \(device)") + } +} diff --git a/Sources/XCForgeCLI/XCForgeCLI.swift b/Sources/XCForgeCLI/XCForgeCLI.swift index b25e776..99093fe 100644 --- a/Sources/XCForgeCLI/XCForgeCLI.swift +++ b/Sources/XCForgeCLI/XCForgeCLI.swift @@ -9,7 +9,7 @@ struct XCForgeCLI: AsyncParsableCommand { subcommands: [ Build.self, Test.self, BuildTest.self, Sim.self, Device.self, Diagnose.self, Defaults.self, Console.self, Git.self, Logs.self, Screenshot.self, UI.self, Accessibility.self, Plan.self, - SPM.self, Debug.self, Pose.self, Bless.self, WaitReady.self, Init.self, Lock.self, + SPM.self, Debug.self, Pose.self, Bless.self, WaitReady.self, Init.self, Lock.self, WDA.self, ] ) } diff --git a/Sources/XCForgeKit/Clients/AXPBridge.swift b/Sources/XCForgeKit/Clients/AXPBridge.swift index be87053..b4c7c5a 100644 --- a/Sources/XCForgeKit/Clients/AXPBridge.swift +++ b/Sources/XCForgeKit/Clients/AXPBridge.swift @@ -300,7 +300,7 @@ public actor AXPBridge { let ws = NSWorkspace.shared guard let app = ws.runningApplications.first(where: { - $0.bundleIdentifier == "com.apple.iphonesimulator" + SimulatorApp.isSimulatorApp(bundleID: $0.bundleIdentifier) }) else { throw AXPError.simulatorNotRunning @@ -320,7 +320,7 @@ public actor AXPBridge { var description: String { switch self { case .simulatorNotRunning: - return "Simulator.app not running" + return "Simulator.app (Device Hub on Xcode 27) is not running" case .noBootedDevice: return "Simulator.app is running but no device is booted. Boot a device with: xcrun simctl boot " diff --git a/Sources/XCForgeKit/Clients/AgentClient.swift b/Sources/XCForgeKit/Clients/AgentClient.swift index 9db57c0..9b7df0b 100644 --- a/Sources/XCForgeKit/Clients/AgentClient.swift +++ b/Sources/XCForgeKit/Clients/AgentClient.swift @@ -30,8 +30,28 @@ public enum WDABackend: String, Sendable { public actor WDAClient { public init() {} - private var baseURL = - ProcessInfo.processInfo.environment["WDA_BASE_URL"] ?? "http://localhost:8100" + /// `WDA_BASE_URL` wins; otherwise `XCFORGE_DEVICE` selects the URL `xcforge wda start` + /// saved for that device; otherwise the simulator default. + private var baseURL = WDAClient.defaultBaseURL() + + static func defaultBaseURL( + environment: [String: String] = ProcessInfo.processInfo.environment + ) -> String { + if let explicit = environment["WDA_BASE_URL"], !explicit.isEmpty { return explicit } + if let device = environment["XCFORGE_DEVICE"], !device.isEmpty, + let state = DeviceWDA.load(device: device) + { + return state.url + } + return "http://localhost:8100" + } + + /// True when WDA runs somewhere other than this Mac (a physical device). Simulator + /// recovery (terminate, rebuild, redeploy) never applies there. + public var isRemote: Bool { + guard let host = URL(string: baseURL)?.host else { return false } + return !["localhost", "127.0.0.1", "::1"].contains(host) + } private var sessionId: String? private var knownSessionIds: [String] = [] // Track all created sessions @@ -258,29 +278,7 @@ public actor WDAClient { deployTask = nil await cleanupWDAProcesses(simulator: simulator) - // Find xcforgeWDA project — check env var first, then common relative location - let projectDir: String - if let envDir = ProcessInfo.processInfo.environment["XCFORGE_WDA_DIR"], - FileManager.default.fileExists(atPath: envDir + "/xcforgeWDA.xcodeproj") - { - projectDir = envDir - } else { - // Search order: CWD-relative (developer running from a clone) → Homebrew share dirs. - let candidates = [ - FileManager.default.currentDirectoryPath + "/xcforgeWDA", - FileManager.default.currentDirectoryPath + "/../xcforgeWDA", - "/opt/homebrew/share/xcforge/xcforgeWDA", - "/usr/local/share/xcforge/xcforgeWDA", - ] - guard - let found = candidates.first(where: { - FileManager.default.fileExists(atPath: $0 + "/xcforgeWDA.xcodeproj") - }) - else { - return false - } - projectDir = found - } + guard let projectDir = Self.locateXCForgeWDAProject() else { return false } // Resolve UDID for xcodebuild destination let udid = await resolveSimulatorUDID(simulator) @@ -337,6 +335,24 @@ public actor WDAClient { return false } + /// Folder containing `xcforgeWDA.xcodeproj`: `XCFORGE_WDA_DIR`, then a clone next to the + /// working directory, then the Homebrew share directories. + public static func locateXCForgeWDAProject() -> String? { + let fm = FileManager.default + if let envDir = ProcessInfo.processInfo.environment["XCFORGE_WDA_DIR"], + fm.fileExists(atPath: envDir + "/xcforgeWDA.xcodeproj") + { + return envDir + } + let candidates = [ + fm.currentDirectoryPath + "/xcforgeWDA", + fm.currentDirectoryPath + "/../xcforgeWDA", + "/opt/homebrew/share/xcforge/xcforgeWDA", + "/usr/local/share/xcforge/xcforgeWDA", + ] + return candidates.first { fm.fileExists(atPath: $0 + "/xcforgeWDA.xcodeproj") } + } + /// Find the xctestrun file in DerivedData/Build/Products. private func findXctestrun(derivedData: String) async -> String? { let productsDir = derivedData + "/Build/Products" @@ -388,6 +404,9 @@ public actor WDAClient { // 1. Already healthy? Done — whatever backend is active, it works. if await isHealthy() { return } + // A device runner can't be restarted from here; say what to do instead. + if isRemote { throw WDAError.remoteNotResponding(baseURL) } + // 2. Try restarting current backend do { try await restartWDA(simulator: simulator) @@ -1122,6 +1141,7 @@ public enum WDASessionCause: String, Codable, Sendable { case noBootedSimulator = "no_booted_simulator" case bundleNotInstalled = "bundle_not_installed" case sessionBindRejected = "session_bind_rejected" + case deviceRunnerNotReachable = "device_runner_not_reachable" case unknown = "unknown" /// A copy-pasteable remediation command for this cause. @@ -1139,6 +1159,8 @@ public enum WDASessionCause: String, Codable, Sendable { case .sessionBindRejected: return "Verify the bundle id is installed and runnable on the booted simulator, then retry" + case .deviceRunnerNotReachable: + return "xcforge wda start --device # unlock the device and enable UI Automation first" case .unknown: return "xcforge ui status # inspect WDA; attach the raw detail when filing an issue" } @@ -1172,6 +1194,7 @@ enum WDAError: Error, CustomStringConvertible { case wdaRestart(String) case wdaNotResponding case noBackendAvailable + case remoteNotResponding(String) var description: String { switch self { @@ -1186,6 +1209,10 @@ enum WDAError: Error, CustomStringConvertible { case .noBackendAvailable: return "No WDA backend available. Neither xcforgeWDA nor Original WDA could be started. Install xcforgeWDA or start WebDriverAgent." + case .remoteNotResponding(let url): + return "WebDriverAgent at \(url) is not responding. Start it with `xcforge wda start --device ` " + + "(MCP: wda_start). If it was running: unlock the device, check Settings > Developer > " + + "Enable UI Automation, and check the device is still connected." } } } diff --git a/Sources/XCForgeKit/Clients/FramebufferCapture.swift b/Sources/XCForgeKit/Clients/FramebufferCapture.swift index 53e4e07..07c6d93 100644 --- a/Sources/XCForgeKit/Clients/FramebufferCapture.swift +++ b/Sources/XCForgeKit/Clients/FramebufferCapture.swift @@ -143,7 +143,7 @@ enum FramebufferCapture { static func findSimulatorWindow(simulator: String = "booted") async throws -> SCWindow { let content = try await SCShareableContent.current let simWindows = content.windows.filter { - $0.owningApplication?.bundleIdentifier == "com.apple.iphonesimulator" + SimulatorApp.isSimulatorApp(bundleID: $0.owningApplication?.bundleIdentifier) && $0.frame.width > 100 && $0.frame.height > 100 } diff --git a/Sources/XCForgeKit/Support/DeviceWDA.swift b/Sources/XCForgeKit/Support/DeviceWDA.swift new file mode 100644 index 0000000..cadf4fa --- /dev/null +++ b/Sources/XCForgeKit/Support/DeviceWDA.swift @@ -0,0 +1,338 @@ +import Foundation + +#if canImport(FoundationNetworking) + import FoundationNetworking +#endif + +/// WebDriverAgent on a physical device: build, sign, launch, find its URL, stop. +/// +/// The runner is built with the caller's development team, launched with +/// `xcodebuild test-without-building` in the background (devicectl cannot reliably launch +/// an XCTest runner), and reached over the CoreDevice tunnel address or the URL WDA logs. +/// One state file per device in `~/.xcforge/wda/` records the URL and the runner pid so +/// later CLI calls and other sessions can reuse it. +public enum DeviceWDA { + + public struct State: Codable, Sendable, Equatable { + public let udid: String + public let name: String? + public let url: String + public let pid: Int32 + public let logPath: String + public let bundleID: String + public let startedAt: Date + } + + public struct StartError: Error, CustomStringConvertible { + public let description: String + } + + public static let defaultPort = 8100 + + // MARK: - State + + static func stateDirectory() -> String { + let base = + ProcessInfo.processInfo.environment["XCFORGE_WDA_STATE_DIR"] + ?? NSHomeDirectory() + "/.xcforge/wda" + try? FileManager.default.createDirectory( + atPath: base, withIntermediateDirectories: true, attributes: nil) + return base + } + + static func statePath(udid: String) -> String { + (stateDirectory() as NSString).appendingPathComponent("\(udid).json") + } + + /// Saved state for a device, matched by UDID or name. Nil when none is recorded. + public static func load(device: String) -> State? { + let direct = statePath(udid: device) + if let state = read(direct) { return state } + return all().first { $0.name == device || $0.udid == device } + } + + /// Every recorded device runner. + public static func all() -> [State] { + let dir = stateDirectory() + let names = (try? FileManager.default.contentsOfDirectory(atPath: dir)) ?? [] + return names.sorted().filter { $0.hasSuffix(".json") }.compactMap { + read((dir as NSString).appendingPathComponent($0)) + } + } + + static func save(_ state: State) { + let encoder = JSONEncoder() + encoder.dateEncodingStrategy = .iso8601 + encoder.outputFormatting = [.prettyPrinted, .sortedKeys] + guard let data = try? encoder.encode(state) else { return } + try? data.write(to: URL(fileURLWithPath: statePath(udid: state.udid)), options: .atomic) + } + + private static func read(_ path: String) -> State? { + guard let data = FileManager.default.contents(atPath: path) else { return nil } + let decoder = JSONDecoder() + decoder.dateDecodingStrategy = .iso8601 + return try? decoder.decode(State.self, from: data) + } + + // MARK: - Device lookup + + public struct DeviceInfo: Sendable, Equatable { + public let udid: String + public let name: String + public let tunnelIP: String? + public let pairingState: String? + public let developerMode: String? + } + + /// Find `device` (UDID, CoreDevice identifier or name) in `devicectl list devices` JSON. + static func findDevice(_ device: String, listJSON: [String: Any]) -> DeviceInfo? { + guard let result = listJSON["result"] as? [String: Any], + let devices = result["devices"] as? [[String: Any]] + else { return nil } + for entry in devices { + let udid = DeviceTools.property(entry, "udid") as? String + let identifier = entry["identifier"] as? String + let name = DeviceTools.property(entry, "name") as? String + guard [udid, identifier, name].contains(where: { $0 == device }) else { continue } + return DeviceInfo( + udid: udid ?? identifier ?? device, + name: name ?? device, + tunnelIP: DeviceTools.property(entry, "tunnelIPAddress") as? String, + pairingState: DeviceTools.property(entry, "pairingState") as? String, + developerMode: DeviceTools.property(entry, "developerModeStatus") as? String) + } + return nil + } + + static func lookup(_ device: String, env: Environment) async -> DeviceInfo? { + let jsonPath = NSTemporaryDirectory() + "xcforge-devicectl-\(UUID().uuidString).json" + defer { try? FileManager.default.removeItem(atPath: jsonPath) } + _ = try? await env.shell.run( + "/usr/bin/xcrun", arguments: ["devicectl", "list", "devices", "--json-output", jsonPath], + timeout: 30) + guard let data = FileManager.default.contents(atPath: jsonPath), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] + else { return nil } + return findDevice(device, listJSON: json) + } + + // MARK: - Failure explanations + + /// Turn a known device or signing failure in xcodebuild/devicectl output into one + /// actionable sentence. Nil when nothing recognisable is found. + public static func explainFailure(_ output: String) -> String? { + let lower = output.lowercased() + let rules: [(markers: [String], message: String)] = [ + ( + ["ui automation", "automation mode", "enable ui automation"], + "UI Automation is off on the device. Turn on Settings > Developer > Enable UI Automation, then retry." + ), + ( + ["passcode", "device is locked", "is locked", "unlock"], + "The device is locked. Unlock it (and keep it unlocked while WebDriverAgent starts), then retry." + ), + ( + ["developer mode"], + "Developer Mode is off. Turn on Settings > Privacy & Security > Developer Mode and restart the device." + ), + ( + ["not paired", "pairing", "trust this computer"], + "The device is not paired with this Mac. Unlock it, tap Trust, and check `xcrun devicectl list devices`." + ), + ( + ["requires a development team", "no account for team", "signing for", "no profiles for"], + "Code signing failed. Pass your team with --team (or XCFORGE_WDA_TEAM) and sign in to that team in Xcode." + ), + ( + ["untrusted developer", "not trusted", "verify the app"], + "The runner's developer certificate is not trusted on the device. Trust it in Settings > General > VPN & Device Management." + ), + ( + ["device is busy", "copying shared cache"], + "The device is still preparing for development. Wait for Xcode to finish, then retry." + ), + ] + for rule in rules where rule.markers.contains(where: { lower.contains($0) }) { + return rule.message + } + return nil + } + + // MARK: - URL discovery + + /// The URL WDA prints as `ServerURLHere->http://...<-ServerURLHere`. + static func loggedServerURL(_ log: String) -> String? { + guard let start = log.range(of: "ServerURLHere->"), + let end = log.range(of: "<-ServerURLHere", range: start.upperBound.. [String] { + var urls: [String] = [] + if let ip = tunnelIP, !ip.isEmpty { + urls.append(ip.contains(":") ? "http://[\(ip)]:\(port)" : "http://\(ip):\(port)") + } + if let logged = loggedURL { + urls.append(logged.hasSuffix("/") ? String(logged.dropLast()) : logged) + } + return urls + } + + /// True when `GET /status` answers. + public static func isHealthy(_ url: String, timeout: TimeInterval = 3) async -> Bool { + guard let statusURL = URL(string: url + "/status") else { return false } + var request = URLRequest(url: statusURL) + request.timeoutInterval = timeout + guard let (_, response) = try? await URLSession.shared.data(for: request) else { return false } + return (response as? HTTPURLResponse)?.statusCode == 200 + } + + /// PNG bytes from WDA's `/screenshot`, or nil. + public static func screenshot(baseURL: String) async -> Data? { + guard let url = URL(string: baseURL + "/screenshot") else { return nil } + var request = URLRequest(url: url) + request.timeoutInterval = 20 + guard let (data, _) = try? await URLSession.shared.data(for: request), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let base64 = json["value"] as? String + else { return nil } + return Data(base64Encoded: base64, options: .ignoreUnknownCharacters) + } + + // MARK: - Start / stop + + /// Bundle id for the runner. A team-specific default avoids colliding with an id + /// another team already registered. + static func runnerBundleID(team: String?, explicit: String?) -> String { + if let explicit, !explicit.isEmpty { return explicit } + if let team, !team.isEmpty { return "com.xcforge.wda.\(team.lowercased()).runner" } + return "com.xcforge.wda.runner" + } + + /// Build, sign and launch WDA on `device`, wait for it to answer, and save its URL. + public static func start( + device: String, team: String?, bundleID: String? = nil, port: Int = defaultPort, + startTimeout: TimeInterval = 120, env: Environment + ) async throws -> State { + guard let info = await lookup(device, env: env) else { + throw StartError( + description: + "No connected device matches '\(device)'. Check `xcrun devicectl list devices` (it must be paired and unlocked)." + ) + } + if let mode = info.developerMode, mode.lowercased() != "enabled" { + throw StartError(description: explainFailure("developer mode") ?? "Developer Mode is off.") + } + + if let existing = load(device: info.udid), BuildLock.isAlive(existing.pid), + await isHealthy(existing.url) + { + return existing + } + await stop(device: info.udid, env: env) + + guard let projectDir = WDAClient.locateXCForgeWDAProject() else { + throw StartError( + description: + "Can't find the xcforgeWDA project. Set XCFORGE_WDA_DIR to the folder containing xcforgeWDA.xcodeproj." + ) + } + let team = team ?? ProcessInfo.processInfo.environment["XCFORGE_WDA_TEAM"] + let runnerID = runnerBundleID(team: team, explicit: bundleID) + let derivedData = (stateDirectory() as NSString).appendingPathComponent("build-\(info.udid)") + + var buildArgs = [ + "-project", "\(projectDir)/xcforgeWDA.xcodeproj", + "-scheme", "xcforgeWDARunner", + "-destination", "id=\(info.udid)", + "-derivedDataPath", derivedData, + "-allowProvisioningUpdates", + "-allowProvisioningDeviceRegistration", + "CODE_SIGN_STYLE=Automatic", + "PRODUCT_BUNDLE_IDENTIFIER=\(runnerID)", + ] + if let team, !team.isEmpty { buildArgs.append("DEVELOPMENT_TEAM=\(team)") } + buildArgs.append("build-for-testing") + let build = try await Xcodebuild.run(buildArgs, timeout: 1800, env: env) + guard build.succeeded else { + let output = Xcodebuild.combinedOutput(build) + let errors = TestTools.fallbackBuildIssues(stderr: output).filter { $0.severity == .error } + let detail = + errors.isEmpty ? String(output.suffix(1500)) : errors.prefix(5).map(\.message).joined(separator: "\n") + throw StartError( + description: "Building WebDriverAgent for \(info.name) failed.\n" + + (explainFailure(output).map { $0 + "\n" } ?? "") + detail) + } + + let products = (derivedData as NSString).appendingPathComponent("Build/Products") + let xctestrun = (try? FileManager.default.contentsOfDirectory(atPath: products))? + .first { $0.hasSuffix(".xctestrun") } + .map { (products as NSString).appendingPathComponent($0) } + guard let xctestrun else { + throw StartError(description: "WebDriverAgent built but no .xctestrun was found in \(products).") + } + + // Launch detached so the runner outlives this command. TEST_RUNNER_ variables reach + // the runner without the prefix, so WDA sees USE_PORT. + let logPath = (stateDirectory() as NSString).appendingPathComponent("\(info.udid).log") + let command = + "nohup /usr/bin/xcrun xcodebuild test-without-building -xctestrun \(shellQuote(xctestrun)) " + + "-destination \(shellQuote("id=\(info.udid)")) > \(shellQuote(logPath)) 2>&1 & echo $!" + let launched = try await env.shell.run( + "/bin/sh", arguments: ["-c", command], workingDirectory: nil, + environment: ["TEST_RUNNER_USE_PORT": "\(port)"], timeout: 15) + guard let pid = Int32(launched.stdout.trimmingCharacters(in: .whitespacesAndNewlines)) else { + throw StartError(description: "Could not launch the WebDriverAgent runner: \(launched.stderr)") + } + + let deadline = Date().addingTimeInterval(startTimeout) + var tunnelIP = info.tunnelIP + while Date() < deadline { + let log = (try? String(contentsOfFile: logPath, encoding: .utf8)) ?? "" + if !BuildLock.isAlive(pid) { + throw StartError( + description: "The WebDriverAgent runner exited before it was reachable.\n" + + (explainFailure(log).map { $0 + "\n" } ?? "") + "Log: \(logPath)") + } + if tunnelIP == nil { tunnelIP = await lookup(info.udid, env: env)?.tunnelIP } + for url in candidateURLs(tunnelIP: tunnelIP, loggedURL: loggedServerURL(log), port: port) { + if await isHealthy(url) { + let state = State( + udid: info.udid, name: info.name, url: url, pid: pid, logPath: logPath, + bundleID: runnerID + ".xctrunner", startedAt: Date()) + save(state) + return state + } + } + try? await Task.sleep(nanoseconds: 2_000_000_000) + } + let log = (try? String(contentsOfFile: logPath, encoding: .utf8)) ?? "" + kill(pid, SIGTERM) + let tried = candidateURLs(tunnelIP: tunnelIP, loggedURL: loggedServerURL(log), port: port) + throw StartError( + description: "WebDriverAgent did not answer within \(Int(startTimeout))s.\n" + + (explainFailure(log).map { $0 + "\n" } ?? "") + + "Tried: \(tried.isEmpty ? "no address yet" : tried.joined(separator: ", "))\n" + + "Log: \(logPath)") + } + + /// Stop the recorded runner for `device` and forget its URL. Best-effort. + public static func stop(device: String, env: Environment) async { + guard let state = load(device: device) else { return } + if BuildLock.isAlive(state.pid) { + // Children first: once xcodebuild exits they are reparented and -P no longer finds them. + _ = try? await env.shell.run( + "/usr/bin/pkill", arguments: ["-TERM", "-P", "\(state.pid)"], timeout: 5) + kill(state.pid, SIGTERM) + } + try? FileManager.default.removeItem(atPath: statePath(udid: state.udid)) + } + + static func shellQuote(_ s: String) -> String { + "'" + s.replacingOccurrences(of: "'", with: "'\\''") + "'" + } +} diff --git a/Sources/XCForgeKit/Support/ProjectResolver.swift b/Sources/XCForgeKit/Support/ProjectResolver.swift index 3fea4c0..6c3a3b7 100644 --- a/Sources/XCForgeKit/Support/ProjectResolver.swift +++ b/Sources/XCForgeKit/Support/ProjectResolver.swift @@ -512,7 +512,7 @@ enum AutoDetect { _ device: SimulatorDevice, requested: String ) async throws -> SimulatorDevice { - _ = try? await Shell.run("/usr/bin/open", arguments: ["-a", "Simulator"], timeout: 5) + await SimulatorApp.open(shell: LiveShell()) do { _ = try await Shell.xcrun(timeout: 30, "simctl", "bootstatus", device.udid, "-b") diff --git a/Sources/XCForgeKit/Support/SimulatorApp.swift b/Sources/XCForgeKit/Support/SimulatorApp.swift new file mode 100644 index 0000000..1b56c95 --- /dev/null +++ b/Sources/XCForgeKit/Support/SimulatorApp.swift @@ -0,0 +1,65 @@ +import Foundation + +/// The app that shows simulator windows. Xcode 26 and earlier ship Simulator.app; Xcode 27 +/// replaces it with Device Hub (`DeviceHub.app`, bundle id `com.apple.dt.Devices`), and +/// `open -a Simulator` no longer works there. +public enum SimulatorApp { + public static let simulatorBundleID = "com.apple.iphonesimulator" + public static let deviceHubBundleID = "com.apple.dt.Devices" + + /// Bundle ids whose windows show simulator screens. + public static let bundleIDs: Set = [simulatorBundleID, deviceHubBundleID] + + /// Process names for System Events scripting, in the order they are tried. + public static let processNames = ["Simulator", "Device Hub", "DeviceHub"] + + public static func isSimulatorApp(bundleID: String?) -> Bool { + guard let bundleID else { return false } + return bundleIDs.contains(bundleID) + } + + /// Candidate app paths inside the selected Xcode, newest layout first. + /// `developerDir` is `xcode-select -p`, e.g. `/Applications/Xcode.app/Contents/Developer`. + static func candidatePaths(developerDir: String) -> [String] { + let contents = (developerDir as NSString).deletingLastPathComponent + return [ + "\(contents)/Applications/DeviceHub.app", + "\(developerDir)/Applications/DeviceHub.app", + "\(developerDir)/Applications/Simulator.app", + ] + } + + /// Path of the simulator UI app for the selected Xcode, or nil when none is found. + static func installedPath( + developerDir: String, exists: (String) -> Bool = { FileManager.default.fileExists(atPath: $0) } + ) -> String? { + candidatePaths(developerDir: developerDir).first(where: exists) + } + + /// Bring up the simulator UI of the selected Xcode. Best-effort: simulators run headless + /// without it, so failure is logged and ignored. + public static func open(shell: ShellExecutor) async { + var developerDir = "/Applications/Xcode.app/Contents/Developer" + if let selected = try? await shell.run("/usr/bin/xcode-select", arguments: ["-p"], timeout: 5), + selected.succeeded, !selected.stdout.isEmpty + { + developerDir = selected.stdout.trimmingCharacters(in: .whitespacesAndNewlines) + } + if let path = installedPath(developerDir: developerDir) { + if let result = try? await shell.run("/usr/bin/open", arguments: ["-g", path], timeout: 10), + result.succeeded + { + return + } + } + for bundleID in [simulatorBundleID, deviceHubBundleID] { + if let result = try? await shell.run( + "/usr/bin/open", arguments: ["-g", "-b", bundleID], timeout: 10), + result.succeeded + { + return + } + } + Log.warn("Could not open Simulator or Device Hub; simulators keep running without a window") + } +} diff --git a/Sources/XCForgeKit/Support/WDASessionRepair.swift b/Sources/XCForgeKit/Support/WDASessionRepair.swift index 084f67c..fd9a40a 100644 --- a/Sources/XCForgeKit/Support/WDASessionRepair.swift +++ b/Sources/XCForgeKit/Support/WDASessionRepair.swift @@ -43,7 +43,22 @@ public enum WDASessionRepair { ) } - // 2. No booted simulator → nothing WDA can attach to. Only assert this + // 2. WDA on a physical device: simulator probes and rebuilds don't apply. + if await env.wdaClient.isRemote { + let url = await env.wdaClient.getBaseURL() + if await env.wdaClient.isHealthy() { + return WDASessionCreateError( + cause: .sessionBindRejected, + detail: "WDA on the device (\(url)) rejected the session" + + (bundleId.map { " for bundleId=\($0)" } ?? "") + ". Raw: \(raw)") + } + return WDASessionCreateError( + cause: .deviceRunnerNotReachable, + detail: "WDA on the device (\(url)) is not responding. The device may be locked, UI Automation " + + "may be off, or the runner stopped. Raw: \(raw)") + } + + // 3. No booted simulator → nothing WDA can attach to. Only assert this // non-recoverable cause when the probe DEFINITIVELY saw no booted // device. On JSON shape drift / parse failure the state is `.unknown` // and we must NOT flip a possibly-booted machine to no_booted_simulator @@ -104,7 +119,8 @@ public enum WDASessionRepair { switch cause { case .wdaRunnerNotRunning, .wdaRunnerBuildFailed: return true - case .noBootedSimulator, .bundleNotInstalled, .sessionBindRejected, .unknown: + case .noBootedSimulator, .bundleNotInstalled, .sessionBindRejected, .deviceRunnerNotReachable, + .unknown: return false } } diff --git a/Sources/XCForgeKit/Tools/BuildProvider.swift b/Sources/XCForgeKit/Tools/BuildProvider.swift index 6c1b61a..69a5760 100644 --- a/Sources/XCForgeKit/Tools/BuildProvider.swift +++ b/Sources/XCForgeKit/Tools/BuildProvider.swift @@ -184,7 +184,7 @@ public enum BuildTools { name: "build_run_sim", description: """ Build, install, and launch an iOS app on a simulator in one call. \ - Runs build, settings extraction, simulator boot, and Simulator.app \ + Runs build, settings extraction, simulator boot, and Simulator/Device Hub \ in parallel for maximum speed. Equivalent to Xcode's Cmd+R. \ Project, scheme, and simulator are auto-detected if omitted. """, @@ -914,8 +914,7 @@ public enum BuildTools { async let settingsTask = Xcodebuild.run(settingsArgs, timeout: 30, env: env) async let bootTask = env.shell.run( "/usr/bin/xcrun", arguments: ["simctl", "boot", udid], timeout: 60) - async let openTask = env.shell.run( - "/usr/bin/open", arguments: ["-a", "Simulator"], timeout: 10) + async let openTask: Void = SimulatorApp.open(shell: env.shell) // Await build first (critical — abort if it fails) let buildResult: ShellResult @@ -1079,7 +1078,7 @@ public enum BuildTools { } // Await Simulator.app (fire and forget) - _ = try? await openTask + await openTask // ── Phase 2: Sequential (needs build artifacts + booted simulator) ── diff --git a/Sources/XCForgeKit/Tools/DeviceProvider.swift b/Sources/XCForgeKit/Tools/DeviceProvider.swift index dc4ad15..61d9d8c 100644 --- a/Sources/XCForgeKit/Tools/DeviceProvider.swift +++ b/Sources/XCForgeKit/Tools/DeviceProvider.swift @@ -129,6 +129,15 @@ public enum DeviceTools { "items": .object(["type": .string("string")]), "description": .string("Arguments to pass to the launched app"), ]), + "url": .object([ + "type": .string("string"), + "description": .string("Open this URL (deep link or universal link) in the app at launch."), + ]), + "env": .object([ + "type": .string("object"), + "additionalProperties": .object(["type": .string("string")]), + "description": .string("Environment variables for the launched app."), + ]), ]), "required": .array([.string("device"), .string("bundle_id")]), ]) @@ -171,6 +180,63 @@ public enum DeviceTools { "required": .array([.string("device")]), ]) ), + Tool( + name: "device_screenshot", + description: + "Save a screenshot of a connected physical device as PNG. Uses devicectl, or the device's WebDriverAgent when devicectl can't capture.", + inputSchema: .object([ + "type": .string("object"), + "properties": .object([ + "device": .object([ + "type": .string("string"), "description": .string("Device name or UDID"), + ]), + "path": .object([ + "type": .string("string"), + "description": .string("Where to write the PNG. Default: a new file in the artifact directory."), + ]), + ]), + "required": .array([.string("device")]), + ]) + ), + Tool( + name: "wda_start", + description: + "Build, sign and start WebDriverAgent on a connected physical device, then point this server's UI tools (find_element, click_element, ...) at it. Reuses a runner that is already answering.", + inputSchema: .object([ + "type": .string("object"), + "properties": .object([ + "device": .object([ + "type": .string("string"), "description": .string("Device name or UDID"), + ]), + "team": .object([ + "type": .string("string"), + "description": .string("Apple development team ID for signing. Default: XCFORGE_WDA_TEAM."), + ]), + "bundle_id": .object([ + "type": .string("string"), + "description": .string( + "Runner bundle id. Default: com.xcforge.wda..runner (or com.xcforge.wda.runner without a team)."), + ]), + "port": .object([ + "type": .string("integer"), "description": .string("Port WDA listens on. Default: 8100."), + ]), + ]), + "required": .array([.string("device")]), + ]) + ), + Tool( + name: "wda_stop", + description: "Stop WebDriverAgent started on a physical device with wda_start.", + inputSchema: .object([ + "type": .string("object"), + "properties": .object([ + "device": .object([ + "type": .string("string"), "description": .string("Device name or UDID"), + ]) + ]), + "required": .array([.string("device")]), + ]) + ), ] // MARK: - Input Structs @@ -200,6 +266,20 @@ public enum DeviceTools { let terminate_existing: Bool? let timeout: Int? let arguments: [String]? + let url: String? + let env: [String: String]? + } + + private struct WDAStartInput: Decodable { + let device: String + let team: String? + let bundle_id: String? + let port: Int? + } + + private struct ScreenshotInput: Decodable { + let device: String + let path: String? } private struct TerminateInput: Decodable { @@ -260,17 +340,14 @@ public enum DeviceTools { var devices: [DeviceEntry] = [] for device in deviceList { - let properties = device["deviceProperties"] as? [String: Any] ?? [:] - let connectionProperties = device["connectionProperties"] as? [String: Any] ?? [:] - let name = properties["name"] as? String ?? "Unknown" - let osVersion = - (device["deviceProperties"] as? [String: Any])?["osVersionNumber"] as? String ?? "Unknown" + let name = property(device, "name") as? String ?? "Unknown" + let osVersion = property(device, "osVersionNumber") as? String ?? "Unknown" let udid = - (device["hardwareProperties"] as? [String: Any])?["udid"] as? String + property(device, "udid") as? String ?? (device["identifier"] as? String) ?? "Unknown" let state = (device["visibilityClass"] as? String) ?? "available" - let connectionType = connectionProperties["transportType"] as? String ?? "unknown" + let connectionType = property(device, "transportType") as? String ?? "unknown" devices.append( DeviceEntry( @@ -322,24 +399,35 @@ public enum DeviceTools { ) } - let deviceInfo = resultObj["deviceProperties"] as? [String: Any] ?? resultObj var lines: [String] = [] - if let name = deviceInfo["name"] as? String { lines.append("Name: \(name)") } - if let osVersion = deviceInfo["osVersionNumber"] as? String { + if let name = property(resultObj, "name") as? String { lines.append("Name: \(name)") } + if let osVersion = property(resultObj, "osVersionNumber") as? String { lines.append("OS: \(osVersion)") } - - let hw = resultObj["hardwareProperties"] as? [String: Any] ?? [:] - if let udid = hw["udid"] as? String { lines.append("UDID: \(udid)") } - if let model = hw["marketingName"] as? String ?? hw["productType"] as? String { + if let udid = property(resultObj, "udid") as? String { lines.append("UDID: \(udid)") } + if let model = property(resultObj, "marketingName") as? String + ?? property(resultObj, "productType") as? String + { lines.append("Model: \(model)") } - if let platform = hw["platform"] as? String { lines.append("Platform: \(platform)") } - - let conn = resultObj["connectionProperties"] as? [String: Any] ?? [:] - if let transport = conn["transportType"] as? String { + if let platform = property(resultObj, "platform") as? String { + lines.append("Platform: \(platform)") + } + if let transport = property(resultObj, "transportType") as? String { lines.append("Connection: \(transport)") } + if let tunnel = property(resultObj, "tunnelState") as? String { + lines.append("Tunnel: \(tunnel)") + } + if let address = property(resultObj, "tunnelIPAddress") as? String { + lines.append("Tunnel IP: \(address)") + } + if let pairing = property(resultObj, "pairingState") as? String { + lines.append("Pairing: \(pairing)") + } + if let devMode = property(resultObj, "developerModeStatus") as? String { + lines.append("Developer Mode: \(devMode)") + } return DeviceResult( succeeded: true, @@ -411,6 +499,8 @@ public enum DeviceTools { terminateExisting: Bool, timeout: Int, arguments: [String]?, + url: String? = nil, + environment: [String: String]? = nil, env: Environment ) async -> DeviceResult { do { @@ -422,6 +512,15 @@ public enum DeviceTools { if terminateExisting { args.append("--terminate-existing") } + if let url, !url.isEmpty { + args += ["--payload-url", url] + } + if let environment, !environment.isEmpty, + let data = try? JSONSerialization.data(withJSONObject: environment, options: [.sortedKeys]), + let json = String(data: data, encoding: .utf8) + { + args += ["--environment-variables", json] + } args.append(bundleId) @@ -538,11 +637,59 @@ public enum DeviceTools { // MARK: - Physical Device UDID Detection + /// Capture the device screen to `path` (PNG). devicectl first (Xcode 26.6 and later), then + /// the device's WebDriverAgent when `xcforge wda start` has one running. + public static func executeDeviceScreenshot(device: String, path: String?, env: Environment) async + -> DeviceResult + { + let output = path ?? XcodebuildOptions.uniqueArtifactPath(prefix: "device-shot", extension: "png") + var devicectlError = "" + do { + let (result, json) = try await runDevicectl( + arguments: ["device", "capture", "screenshot", "--device", device, "--destination", output], + timeout: 60, env: env) + if result.succeeded, FileManager.default.fileExists(atPath: output) { + return DeviceResult(succeeded: true, message: "Screenshot: \(output)") + } + devicectlError = json.flatMap { extractError(from: $0) } ?? result.stderr + } catch { + devicectlError = "\(error)" + } + + if let state = DeviceWDA.load(device: device), + let data = await DeviceWDA.screenshot(baseURL: state.url) + { + do { + try data.write(to: URL(fileURLWithPath: output)) + return DeviceResult(succeeded: true, message: "Screenshot (via WebDriverAgent): \(output)") + } catch { + return DeviceResult(succeeded: false, message: "Could not write \(output): \(error)") + } + } + let reason = DeviceWDA.explainFailure(devicectlError) ?? devicectlError + return DeviceResult( + succeeded: false, + message: + "Screenshot failed: \(reason.isEmpty ? "devicectl returned no image" : reason)\n" + + "devicectl screen capture needs Xcode 26.6 or later. Alternatively start WebDriverAgent " + + "on the device with `xcforge wda start --device \(device)` and retry.") + } + public static func isConnectedPhysicalDevice(_ identifier: String, env: Environment) async -> Bool { let list = await executeListDevices(filter: nil, env: env) return list.devices.contains { $0.udid == identifier || $0.name == identifier } } + /// Read a device field from devicectl JSON. Xcode 27 groups fields under `properties`; + /// earlier versions split them across `deviceProperties`, `hardwareProperties` and + /// `connectionProperties`. Both layouts are read. + static func property(_ device: [String: Any], _ key: String) -> Any? { + for container in ["properties", "deviceProperties", "hardwareProperties", "connectionProperties"] { + if let dict = device[container] as? [String: Any], let value = dict[key] { return value } + } + return device[key] + } + // MARK: - Formatting Helpers private static func formatDeviceList(_ devices: [DeviceEntry]) -> String { @@ -625,9 +772,47 @@ extension DeviceTools: ToolProvider { terminateExisting: input.terminate_existing ?? true, timeout: input.timeout ?? 30, arguments: input.arguments, + url: input.url, + environment: input.env, env: env )) } + case "wda_start": + switch ToolInput.decode(WDAStartInput.self, from: args) { + case .failure(let err): return err + case .success(let input): + do { + let state = try await DeviceWDA.start( + device: input.device, team: input.team, bundleID: input.bundle_id, + port: input.port ?? DeviceWDA.defaultPort, env: env) + await env.wdaClient.setBaseURL(state.url) + return .ok( + "WebDriverAgent running on \(state.name ?? state.udid) at \(state.url)\n" + + "UI tools in this session now target the device. Runner log: \(state.logPath)") + } catch { + return .fail("\(error)") + } + } + case "wda_stop": + switch ToolInput.decode(DeviceInput.self, from: args) { + case .failure(let err): return err + case .success(let input): + let state = DeviceWDA.load(device: input.device) + await DeviceWDA.stop(device: input.device, env: env) + if let state, await env.wdaClient.getBaseURL() == state.url { + await env.wdaClient.setBaseURL("http://localhost:8100") + } + return .ok( + state == nil + ? "No WebDriverAgent recorded for \(input.device)" : "Stopped WebDriverAgent on \(input.device)") + } + case "device_screenshot": + switch ToolInput.decode(ScreenshotInput.self, from: args) { + case .failure(let err): return err + case .success(let input): + return dispatchResult( + await executeDeviceScreenshot(device: input.device, path: input.path, env: env)) + } case "device_terminate": switch ToolInput.decode(TerminateInput.self, from: args) { case .failure(let err): return err diff --git a/Sources/XCForgeKit/Tools/InteractionProvider.swift b/Sources/XCForgeKit/Tools/InteractionProvider.swift index 64001d6..8225c89 100644 --- a/Sources/XCForgeKit/Tools/InteractionProvider.swift +++ b/Sources/XCForgeKit/Tools/InteractionProvider.swift @@ -6,7 +6,7 @@ enum UITools { Tool( name: "wda_status", description: - "Check if WebDriverAgent (WDA) is running and reachable. Call this first to verify the UI automation backend is available before using find_element, click_element, or other WDA-dependent tools.", + "Check whether WebDriverAgent (WDA) is running and reachable, and which URL UI tools use. UI tools start WDA on a simulator by themselves; for a physical device use wda_start.", inputSchema: .object(["type": .string("object"), "properties": .object([:])]) ), Tool( @@ -653,6 +653,7 @@ enum UITools { let sessionInfo = "Sessions tracked: \(await wdaClient.sessionCount)" var msg = "WDA Status: \(status.ready ? "READY" : "NOT READY")\nBackend: \(backendName)\nBundle: \(status.bundleId)\n\(sessionInfo)" + msg += "\nURL: \(await wdaClient.getBaseURL())" if let info = fallback { msg += "\nInfo: \(info)" } @@ -660,6 +661,8 @@ enum UITools { } catch { return .ok("WDA reachable but status parse failed: \(error)") } + } else if await wdaClient.isRemote { + return .fail(WDAError.remoteNotResponding(await wdaClient.getBaseURL()).description) } else { return .fail( "WDA not responding (health check timeout 2s). Backend: \(backendName). Try restarting WDA or the simulator." diff --git a/Sources/XCForgeKit/Tools/MultiDeviceTools.swift b/Sources/XCForgeKit/Tools/MultiDeviceTools.swift index b41a60e..dd4f1bc 100644 --- a/Sources/XCForgeKit/Tools/MultiDeviceTools.swift +++ b/Sources/XCForgeKit/Tools/MultiDeviceTools.swift @@ -282,7 +282,7 @@ enum MultiDeviceTools { if !r.success { devices[idx].error = "Boot: \(r.message)" } } } - _ = try? await env.shell.run("/usr/bin/open", arguments: ["-a", "Simulator"], timeout: 5) + await SimulatorApp.open(shell: env.shell) await waitForAllBooted(udids: needsBoot.map(\.udid), timeout: 30, env: env) } for i in devices.indices where devices[i].wasAlreadyBooted { @@ -610,7 +610,7 @@ enum MultiDeviceTools { let script = """ set output to "" tell application "System Events" - tell process "Simulator" + tell (first process whose bundle identifier is "com.apple.iphonesimulator" or bundle identifier is "com.apple.dt.Devices") repeat with w in windows set output to output & (id of w) & "|||" & (name of w) & linefeed end repeat @@ -653,10 +653,10 @@ enum MultiDeviceTools { { let menuItem = toPortrait ? "Rotate Right" : "Rotate Left" let script = """ - tell application "Simulator" to activate - delay 0.2 tell application "System Events" - tell process "Simulator" + tell (first process whose bundle identifier is "com.apple.iphonesimulator" or bundle identifier is "com.apple.dt.Devices") + set frontmost to true + delay 0.2 perform action "AXRaise" of (first window whose id is \(windowID)) delay 0.3 click menu item "\(menuItem)" of menu "Device" of menu bar 1 diff --git a/Tests/XCForgeKitTests/DeviceSupportTests.swift b/Tests/XCForgeKitTests/DeviceSupportTests.swift new file mode 100644 index 0000000..08b0f40 --- /dev/null +++ b/Tests/XCForgeKitTests/DeviceSupportTests.swift @@ -0,0 +1,114 @@ +import Foundation +import Testing + +@testable import XCForgeKit + +@Suite("Physical devices and Device Hub", .serialized) +struct DeviceSupportTests { + + // MARK: - Device Hub + + @Test("Device Hub is preferred over Simulator.app when the selected Xcode has it") + func simulatorAppPaths() { + let dev = "/Applications/Xcode.app/Contents/Developer" + let hub = "/Applications/Xcode.app/Contents/Applications/DeviceHub.app" + let sim = "/Applications/Xcode.app/Contents/Developer/Applications/Simulator.app" + #expect(SimulatorApp.installedPath(developerDir: dev, exists: { $0 == hub || $0 == sim }) == hub) + #expect(SimulatorApp.installedPath(developerDir: dev, exists: { $0 == sim }) == sim) + #expect(SimulatorApp.installedPath(developerDir: dev, exists: { _ in false }) == nil) + #expect(SimulatorApp.isSimulatorApp(bundleID: "com.apple.dt.Devices")) + #expect(SimulatorApp.isSimulatorApp(bundleID: "com.apple.iphonesimulator")) + #expect(!SimulatorApp.isSimulatorApp(bundleID: "com.apple.dt.Xcode")) + } + + // MARK: - devicectl JSON + + @Test("device fields are read from Xcode 27 `properties` and the older split layout") + func devicectlLayouts() { + let legacy: [String: Any] = [ + "identifier": "CORE-1", + "deviceProperties": ["name": "Justin's iPhone", "developerModeStatus": "enabled"], + "hardwareProperties": ["udid": "00008110-AAA"], + "connectionProperties": ["tunnelIPAddress": "fd00::1", "pairingState": "paired"], + ] + let modern: [String: Any] = [ + "identifier": "CORE-2", + "properties": ["name": "Test iPad", "udid": "00008120-BBB", "tunnelIPAddress": "fd00::2"], + ] + let list: [String: Any] = ["result": ["devices": [legacy, modern]]] + + let byName = DeviceWDA.findDevice("Justin's iPhone", listJSON: list) + #expect(byName?.udid == "00008110-AAA") + #expect(byName?.tunnelIP == "fd00::1") + #expect(byName?.developerMode == "enabled") + let byUDID = DeviceWDA.findDevice("00008120-BBB", listJSON: list) + #expect(byUDID?.name == "Test iPad") + #expect(DeviceWDA.findDevice("CORE-1", listJSON: list)?.udid == "00008110-AAA") + #expect(DeviceWDA.findDevice("nope", listJSON: list) == nil) + } + + // MARK: - WDA URL discovery + + @Test("the URL WDA logs is extracted, and the tunnel address is tried first") + func wdaURLs() { + let log = "noise\nServerURLHere->http://192.168.1.20:8100<-ServerURLHere\nmore" + #expect(DeviceWDA.loggedServerURL(log) == "http://192.168.1.20:8100") + #expect(DeviceWDA.loggedServerURL("nothing") == nil) + #expect( + DeviceWDA.candidateURLs(tunnelIP: "fd00::1", loggedURL: "http://192.168.1.20:8100/", port: 8100) + == ["http://[fd00::1]:8100", "http://192.168.1.20:8100"]) + #expect(DeviceWDA.candidateURLs(tunnelIP: "10.0.0.2", loggedURL: nil, port: 9000) == ["http://10.0.0.2:9000"]) + } + + @Test("known device failures become one actionable sentence") + func failureExplanations() { + #expect(DeviceWDA.explainFailure("Enable UI Automation in Settings")?.contains("UI Automation") == true) + #expect(DeviceWDA.explainFailure("The device is passcode protected")?.contains("locked") == true) + #expect(DeviceWDA.explainFailure("Developer Mode disabled")?.contains("Developer Mode") == true) + #expect( + DeviceWDA.explainFailure("Signing for \"WebDriverAgentRunner\" requires a development team")? + .contains("--team") == true) + #expect(DeviceWDA.explainFailure("Build succeeded") == nil) + } + + @Test("runner bundle id is team-specific by default") + func runnerBundleID() { + #expect(DeviceWDA.runnerBundleID(team: "ABCDE12345", explicit: nil) == "com.xcforge.wda.abcde12345.runner") + #expect(DeviceWDA.runnerBundleID(team: nil, explicit: nil) == "com.xcforge.wda.runner") + #expect(DeviceWDA.runnerBundleID(team: "X", explicit: "com.me.wda") == "com.me.wda") + } + + // MARK: - State and base URL + + @Test("a saved device runner is found by name or UDID and selects the WDA URL") + func savedStateSelectsURL() throws { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("xcforge-wda-\(UUID().uuidString)").path + setenv("XCFORGE_WDA_STATE_DIR", dir, 1) + defer { + unsetenv("XCFORGE_WDA_STATE_DIR") + try? FileManager.default.removeItem(atPath: dir) + } + let state = DeviceWDA.State( + udid: "00008110-AAA", name: "Phone", url: "http://[fd00::1]:8100", pid: 1, logPath: "/tmp/x.log", + bundleID: "com.xcforge.wda.runner.xctrunner", startedAt: Date()) + DeviceWDA.save(state) + + #expect(DeviceWDA.load(device: "00008110-AAA")?.url == state.url) + #expect(DeviceWDA.load(device: "Phone")?.udid == "00008110-AAA") + #expect(WDAClient.defaultBaseURL(environment: ["XCFORGE_DEVICE": "Phone"]) == state.url) + #expect( + WDAClient.defaultBaseURL(environment: ["XCFORGE_DEVICE": "Phone", "WDA_BASE_URL": "http://h:1"]) + == "http://h:1") + #expect(WDAClient.defaultBaseURL(environment: [:]) == "http://localhost:8100") + } + + @Test("a device WDA URL is remote; localhost is not") + func remoteDetection() async { + let client = WDAClient() + await client.setBaseURL("http://[fd00::1]:8100") + #expect(await client.isRemote) + await client.setBaseURL("http://localhost:8100") + #expect(!(await client.isRemote)) + } +} diff --git a/xcforgeWDA/WebDriverAgent.xcodeproj/project.pbxproj b/xcforgeWDA/WebDriverAgent.xcodeproj/project.pbxproj index 2d35326..9b9b8c0 100644 --- a/xcforgeWDA/WebDriverAgent.xcodeproj/project.pbxproj +++ b/xcforgeWDA/WebDriverAgent.xcodeproj/project.pbxproj @@ -3952,7 +3952,7 @@ "$(SDKROOT)/usr/include/libxml2", "$(SRCROOT)/Modules", ); - IPHONEOS_DEPLOYMENT_TARGET = 13.0; + IPHONEOS_DEPLOYMENT_TARGET = 15.0; MTL_ENABLE_DEBUG_INFO = YES; ONLY_ACTIVE_ARCH = YES; SDKROOT = iphoneos; @@ -4015,7 +4015,7 @@ "$(SDKROOT)/usr/include/libxml2", "$(SRCROOT)/Modules", ); - IPHONEOS_DEPLOYMENT_TARGET = 13.0; + IPHONEOS_DEPLOYMENT_TARGET = 15.0; MTL_ENABLE_DEBUG_INFO = NO; SDKROOT = iphoneos; STRING_CATALOG_GENERATE_SYMBOLS = YES; @@ -4276,7 +4276,7 @@ CLANG_ANALYZER_NONNULL = YES; DEBUG_INFORMATION_FORMAT = dwarf; INFOPLIST_FILE = WebDriverAgentTests/IntegrationApp/Info.plist; - IPHONEOS_DEPLOYMENT_TARGET = 13.0; + IPHONEOS_DEPLOYMENT_TARGET = 15.0; LD_RUNPATH_SEARCH_PATHS = ( "$(inherited)", "@executable_path/Frameworks", @@ -4293,7 +4293,7 @@ ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; CLANG_ANALYZER_NONNULL = YES; INFOPLIST_FILE = WebDriverAgentTests/IntegrationApp/Info.plist; - IPHONEOS_DEPLOYMENT_TARGET = 13.0; + IPHONEOS_DEPLOYMENT_TARGET = 15.0; LD_RUNPATH_SEARCH_PATHS = ( "$(inherited)", "@executable_path/Frameworks", diff --git a/xcforgeWDA/xcforgeWDA.xcodeproj/project.pbxproj b/xcforgeWDA/xcforgeWDA.xcodeproj/project.pbxproj index 077eb23..1981ad9 100644 --- a/xcforgeWDA/xcforgeWDA.xcodeproj/project.pbxproj +++ b/xcforgeWDA/xcforgeWDA.xcodeproj/project.pbxproj @@ -3952,7 +3952,7 @@ "$(SDKROOT)/usr/include/libxml2", "$(SRCROOT)/Modules", ); - IPHONEOS_DEPLOYMENT_TARGET = 13.0; + IPHONEOS_DEPLOYMENT_TARGET = 15.0; MTL_ENABLE_DEBUG_INFO = YES; ONLY_ACTIVE_ARCH = YES; SDKROOT = iphoneos; @@ -4015,7 +4015,7 @@ "$(SDKROOT)/usr/include/libxml2", "$(SRCROOT)/Modules", ); - IPHONEOS_DEPLOYMENT_TARGET = 13.0; + IPHONEOS_DEPLOYMENT_TARGET = 15.0; MTL_ENABLE_DEBUG_INFO = NO; SDKROOT = iphoneos; STRING_CATALOG_GENERATE_SYMBOLS = YES; @@ -4276,7 +4276,7 @@ CLANG_ANALYZER_NONNULL = YES; DEBUG_INFORMATION_FORMAT = dwarf; INFOPLIST_FILE = WebDriverAgentTests/IntegrationApp/Info.plist; - IPHONEOS_DEPLOYMENT_TARGET = 13.0; + IPHONEOS_DEPLOYMENT_TARGET = 15.0; LD_RUNPATH_SEARCH_PATHS = ( "$(inherited)", "@executable_path/Frameworks", @@ -4293,7 +4293,7 @@ ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; CLANG_ANALYZER_NONNULL = YES; INFOPLIST_FILE = WebDriverAgentTests/IntegrationApp/Info.plist; - IPHONEOS_DEPLOYMENT_TARGET = 13.0; + IPHONEOS_DEPLOYMENT_TARGET = 15.0; LD_RUNPATH_SEARCH_PATHS = ( "$(inherited)", "@executable_path/Frameworks", From 83b0a74c224c807a7ce6d32087db966a24bb6e06 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 14:33:24 +0000 Subject: [PATCH 04/28] refactor: drop opinionated defaults and wording Auto-promotion becomes opt-in, xcforge's own build flags can be turned off, lldb continue and simulator launch timeouts are longer and configurable, bless stops suggesting commit messages, and tool text and the skill lose preference claims and marketing copy. Fixes stale paths in CLAUDE.md. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01NVNakyntY4u9MvhsjbzRhZ --- CHANGELOG.md | 5 ++ CLAUDE.md | 4 +- Skills/xcforge/SKILL.md | 46 +++++++------------ Skills/xcforge/references/auto-detection.md | 10 ++-- Skills/xcforge/references/cli-commands.md | 2 +- .../xcforge/references/screenshot-visual.md | 3 +- .../Build/XcodebuildOptionGroup.swift | 8 +++- .../Commands/Debug/DebugCommand.swift | 7 ++- .../Persistence/DefaultsStore.swift | 20 ++++---- .../XCForgeKit/Support/WorkspaceContext.swift | 5 +- Sources/XCForgeKit/Support/Xcodebuild.swift | 11 ++++- .../Support/XcodebuildOptions.swift | 27 ++++++++++- Sources/XCForgeKit/Tools/BlessProvider.swift | 16 ------- Sources/XCForgeKit/Tools/BuildProvider.swift | 4 +- .../XCForgeKit/Tools/DebuggerProvider.swift | 10 +++- .../XCForgeKit/Tools/SimulatorProvider.swift | 11 +++-- Sources/XCForgeKit/Tools/TestProvider.swift | 2 +- Tests/XCForgeKitTests/RepoConfigTests.swift | 21 +++++++-- .../SharedMacSafetyTests.swift | 9 ++++ .../content/docs/guides/workflow-patterns.mdx | 1 - 20 files changed, 136 insertions(+), 86 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a739b68..ae95f41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - **Xcode 27 Device Hub support.** xcforge opens Device Hub when the selected Xcode has no Simulator.app, and screen capture, the accessibility bridge and window scripting accept either app ### Changed +- **Auto-promotion is opt-in.** Repeated explicit values no longer silently become session defaults unless `.xcforge.yaml` sets `autoPromote: true` +- `--no-default-flags` / `defaultFlags: false` / `XCFORGE_DEFAULT_FLAGS=0` / `.xcforge.yaml defaultFlags: false` drop the flags xcforge adds to builds (`-skipMacroValidation`, `-parallelizeTargets`, `COMPILATION_CACHE_ENABLE_CACHING=YES`) +- `lldb_continue` / `debug continue` wait 30s by default and take `timeoutSeconds` / `--timeout` (was a fixed 10s); simulator app launch waits up to 60s (was 15s) +- `bless` no longer prints a suggested commit message +- Tool descriptions and the skill drop "preferred", "call this first" and marketing copy; pitfalls are stated as facts rather than rules - UI tools no longer run simulator recovery (terminate, rebuild, redeploy) when WDA is on a physical device; they report `device_runner_not_reachable` and point at `wda start` - xcforgeWDA's iOS deployment target is 15.0 (Xcode 27 no longer builds for 13) - devicectl output is read from Xcode 27's `properties` dictionary as well as the older split keys; `device info` adds tunnel, pairing and Developer Mode state diff --git a/CLAUDE.md b/CLAUDE.md index 64379e3..a39ea8c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ bun run lint && bun run format # Biome lint/format **Dual-mode single binary**: no args → MCP server (stdio JSON-RPC); with args → CLI (ArgumentParser). Mode detection in `Sources/XCForgeCLI/Main.swift`. **Two targets** (see `Package.swift`): -- `XCForgeKit` — shared library containing all 102+ MCP tools +- `XCForgeKit` — shared library containing all 110+ MCP tools - `XCForgeCLI` — CLI layer wrapping the library with ArgumentParser commands ### Tool Provider System @@ -74,7 +74,7 @@ MCP tools use snake_case (`build_sim`, `ui_tap`). CLI commands mirror them as su 2. Register it in `ProviderRegistry` 3. Add a matching CLI command in `Sources/XCForgeCLI/Commands/` 4. Add tests in `Tests/XCForgeKitTests/` -5. Update skill reference in `.claude/skills/xcforge/references/` +5. Update skill reference in `Skills/xcforge/references/` ## Dependencies diff --git a/Skills/xcforge/SKILL.md b/Skills/xcforge/SKILL.md index da62bb2..ac7bc00 100644 --- a/Skills/xcforge/SKILL.md +++ b/Skills/xcforge/SKILL.md @@ -7,21 +7,9 @@ description: Complete reference for xcforge — 109 MCP tools + 113 CLI commands xcforge is a native Swift MCP server and CLI for iOS development. 109 MCP tools, 113 CLI commands across 19 groups, zero runtime dependencies. It provides build, test, simulator management, physical device support via devicectl, Swift package workflows, UI automation via WebDriverAgent with native HID fallback, ultra-fast screenshots, clipboard access, video recording, location simulation, appearance control, status bar overrides, smart log filtering, visual regression, multi-device checks, accessibility/localization layout checks, session profiles, structured diagnosis workflows, multi-step plan execution, visual pose iteration, bless (baseline+test+diff in one call), agent-optimized test output with known-failures gating, and test plan inspection. Every MCP tool has a CLI equivalent. -**Key advantages over alternatives:** -- Screenshots in 0.3s (44x faster) via CoreSimulator IOSurface API -- Native HID taps/swipes in <5ms (bypasses WDA when available) -- Structured xcresult parsing — pass/fail counts, failure file:line, failure screenshots -- One-call alert batching (`accept_all`) handles 3+ permission dialogs -- Topic-filtered logs — 90% fewer tokens, 8 topics with line counts -- Element-to-element drag & drop in 1 call -- Auto-scroll to off-screen elements (3-tier fallback) -- View hierarchy in ~20ms (750x faster) -- Physical device support — install, launch, terminate, list apps via devicectl -- Session profiles — save/switch named default sets for quick context switching - -## Reference Loading Guide - -**Load reference files when using or advising on any xcforge tool.** Each reference covers one tool category with exact parameters, return values, and usage patterns. +## References + +Each reference covers one tool category with exact parameters, return values, and examples. Open the one you need. | Reference | Load When | |-----------|-----------| @@ -29,7 +17,7 @@ xcforge is a native Swift MCP server and CLI for iOS development. 109 MCP tools, | **[Test Tools](references/test-tools.md)** | Running tests, analyzing failures, checking code coverage, build diagnostics | | **[Shared Mac](references/shared-mac.md)** | DerivedData/result-bundle paths, extra xcodebuild args, build lock queue, isolated simulators, simulator recovery modes | | **[Simulator Tools](references/simulator-tools.md)** | Managing simulators — boot, shutdown, install, launch, clone, erase, delete, orientation, video recording, location, appearance, status bar | -| **[Device Tools](references/device-tools.md)** | Physical iOS devices — list, info, install, uninstall, launch, terminate, list apps via devicectl | +| **[Device Tools](references/device-tools.md)** | Physical iOS devices — list, info, install, launch (deep links, env), screenshot, and WebDriverAgent on the device for UI automation | | **[SPM Tools](references/spm-tools.md)** | Swift packages — build, test, run, list dependencies, clean | | **[UI Automation](references/ui-automation.md)** | Finding/clicking elements, alerts, typing, gestures, drag & drop, view hierarchy, clipboard, native HID taps/swipes | | **[Screenshot & Visual](references/screenshot-visual.md)** | Taking screenshots, saving baselines, comparing visual regressions, multi-device checks, bless workflow | @@ -56,7 +44,7 @@ All tools that accept `project`, `scheme`, `simulator`, `bundle_id`, or `app_pat **Repo config beats persisted defaults** (committed `.xcforge.yaml` outranks the personal machine-global file — like git `local` > `global`). `.xcforge.yaml` also supports repo-only `configuration` and `testPlan` keys. -**Auto-promotion:** 3 consecutive calls with the same explicit value auto-promotes it to a session default. +**Auto-promotion** (opt-in with `.xcforge.yaml autoPromote: true`): 3 consecutive calls with the same explicit value save it as a session default. Most tools work with zero parameters for single-project repos with one booted simulator. @@ -69,7 +57,7 @@ build_run_sim() → builds + boots + installs + launches in parallel (~9s faste ### Test → Fix → Verify (MCP) ``` -build_and_test() → build + test in one call (preferred) +build_and_test() → build + test in one call build_and_test(filter: "testFoo") → build + run specific test (auto-resolves target) test_failures(include_console: true) → error messages + file:line + console # fix the code... @@ -85,7 +73,7 @@ test_sim(filter: "MyClass/testFoo") → run specific test (auto-resolv ### TDD Loop (CLI) ```bash xcforge build run → Cmd+R equivalent: build + boot + install + launch -xcforge build-test → build + test in one step (preferred for TDD) +xcforge build-test → build + test in one step xcforge build-test --filter "testFoo" → build + run specific test xcforge build-test --env BLESS_BASELINE=1 → inject TEST_RUNNER_BLESS_BASELINE=1 into test process xcforge build-test --for agent → slim ≤10-field JSON (agent-safe output) @@ -200,6 +188,8 @@ device_install(device: "iPhone", app_path: "/path/to/App.app") device_launch(device: "iPhone", bundle_id: "com.app.id") device_apps(device: "iPhone") → list installed apps device_terminate(device: "iPhone", identifier: "com.app.id") +device_screenshot(device: "iPhone") +wda_start(device: "iPhone", team: "ABCDE12345") → UI tools now drive the phone ``` ### Swift Package Workflow (MCP) @@ -224,7 +214,7 @@ xcforge spm clean → clean artifacts ``` lldb_attach(bundleId: "com.example.App") → { sessionId, pid, status: "stopped" } lldb_set_breakpoint(sessionId: "...", file: "Foo.swift", line: 42) -lldb_continue(sessionId: "...", mode: "continue") → runs until breakpoint; 10s timeout +lldb_continue(sessionId: "...", mode: "continue") → runs until breakpoint lldb_backtrace(sessionId: "...", threadIndex: 0) → structured frames lldb_inspect_variable(sessionId: "...", expression: "self.count") lldb_run_command(sessionId: "...", command: "thread list") @@ -247,11 +237,11 @@ clipboard_set(text: "test data") → write to pasteboard clipboard_get() → read pasteboard content ``` -## Common Mistakes +## Pitfalls 1. **Starting log capture after the event** — `start_log_capture` only captures from the moment it's called. Start capture BEFORE reproducing the issue. -2. **Not using `accept_all` after fresh install** — First app launch shows 2-3 permission dialogs. Call `handle_alert(action: "accept_all")` right after `launch_app` or `build_run_sim`. +2. **Permission dialogs after a fresh install** — the first launch can show several. `handle_alert(action: "accept_all")` clears them in one call. 3. **Manual swipe loops to find elements** — Use `find_element(scroll: true)` instead. It handles auto-scrolling with 3-tier fallback (scrollToVisible, calculated drag, iterative with stall detection). @@ -259,11 +249,9 @@ clipboard_get() → read pasteboard content 5. **Ignoring topic menu in `read_logs`** — The topic menu shows line counts. Use `include` to add specific topics rather than reading everything. -6. **Not setting defaults for repeated operations** — Call `set_defaults(project: "...", scheme: "...", simulator: "...")` at the start of a session to avoid repeating parameters. - -7. **Using `test_sim` then parsing output for failures** — Use `test_failures` for detailed failure info with file:line and screenshots. `test_sim` gives the summary; `test_failures` gives the details. +6. **Using `test_sim` then parsing output for failures** — Use `test_failures` for detailed failure info with file:line and screenshots. `test_sim` gives the summary; `test_failures` gives the details. -8. **Forgetting WDA session** — UI automation tools require WebDriverAgent running on the simulator. If `wda_status` fails, WDA needs to be started. `build_run_sim` does NOT start WDA automatically. +7. **WebDriverAgent** — UI automation tools start WDA on a simulator themselves on first use. On a physical device run `wda_start` (CLI `xcforge wda start`) first; the device must be unlocked with Settings > Developer > Enable UI Automation on. **Sub-pitfall: WDA can't see SwiftUI sheets / alerts / `fullScreenCover`.** Those mount in a secondary window owned by the app. Queries reach them only when the WDA session is bound to the app's bundle id. Run `xcforge ui session --bundle-id ` once — `WDAClient` persists the binding across recreates from then on, and the CLI verifies the binding via `GET /session/`, exiting non-zero on mismatch (no more silent no-binds). If `ui find` / `tap-by-id` 404s on an id you can see in `ui ls`, this is almost always why. @@ -271,8 +259,8 @@ clipboard_get() → read pasteboard content **Sub-pitfall: `xcforge pose --screenshot` capturing the launch zoom animation.** `pose` now waits 1.5s by default before capturing so the iOS launch zoom can settle. Override with `--screenshot-delay ` (use `0` for legacy immediate-capture). -9. **Not clearing status bar overrides** — `sim_statusbar` overrides persist until cleared. Always call `sim_statusbar_clear()` after capturing screenshots with overrides. +8. **Status bar overrides persist** — `sim_statusbar` overrides stay until `sim_statusbar_clear()`. -10. **Confusing simulator vs device tools** — Simulator tools (`sim`, `build_sim`, etc.) don't work with physical devices. Use `device_*` tools (`list_devices`, `device_install`, etc.) for physical iOS devices connected via USB/WiFi. +9. **Simulator vs device tools** — simulator tools (`sim`, `build_sim`, etc.) don't target physical devices. Use the `device_*` tools and `wda_start` for devices connected over USB or Wi-Fi. -11. **Using `build_sim` for Swift packages** — Swift packages don't have .xcodeproj files. Use `swift_package_build` / `swift_package_test` instead. +10. **Using `build_sim` for Swift packages** — Swift packages don't have .xcodeproj files. Use `swift_package_build` / `swift_package_test` instead. diff --git a/Skills/xcforge/references/auto-detection.md b/Skills/xcforge/references/auto-detection.md index 1629de4..b970415 100644 --- a/Skills/xcforge/references/auto-detection.md +++ b/Skills/xcforge/references/auto-detection.md @@ -5,7 +5,7 @@ All tools that accept `project`, `scheme`, or `simulator` resolve values in this order: 1. **Explicit parameter** — value passed directly in the tool call (highest priority) -2. **In-session value** — set this session via `set_defaults`/`profile_switch`, auto-promoted (3× rule, unless `autoPromote: false`), or already resolved earlier this session +2. **In-session value** — set this session via `set_defaults`/`profile_switch`, auto-promoted (3× rule, only with `autoPromote: true`), or already resolved earlier this session 3. **Repo config** — `.xcforge.yaml` at the repo root (committed team config; see below) 4. **Persisted defaults (per active project)** — the active project's record in `~/.xcforge/defaults.json` 5. **Auto-detect** — runtime detection (see below) @@ -50,7 +50,7 @@ file is warned and skipped (never crashes). | `configuration` | Build configuration for `build_sim`/`test_sim`/`build_and_test`/`build_and_diagnose` when no `--configuration`/`configuration` arg is given. Default `Debug`. | **repo-only — never written to `defaults.json` or profiles** | | `testPlan` | Default `.xctestplan` for `test_sim`/`build_and_test` when no `--testplan`/`testplan` arg is given. | **repo-only — never written to `defaults.json` or profiles** | | `testTimeout` | Per-project default test timeout, positive integer seconds. Precedence: explicit `timeoutSeconds` > `testTimeout` > `--long` (7200s) > 1800s default. Non-positive values rejected with warning. | **repo-only** | -| `autoPromote` | `true` (default) keeps the 3-rep auto-promotion. `false` disables it (streak counter held at zero) so explicit values stay explicit across repeated iterative runs. | **repo-only** | +| `autoPromote` | `true` turns on 3-rep auto-promotion of explicit values to session defaults. Off (`false`) by default. | **repo-only** | `configuration`, `testPlan`, `testTimeout`, and `autoPromote` are repo-only by design and never flow into the machine-global persisted JSON or named profiles. @@ -100,11 +100,9 @@ xcforge init --force # overwrite an existing file ## Auto-Promotion -When the same explicit value is passed **3 consecutive times** for a parameter, xcforge auto-promotes it to a session default. This avoids the need to call `set_defaults` explicitly for repeated workflows. +Off by default. With `autoPromote: true` in `.xcforge.yaml`, passing the same explicit value **3 consecutive times** for a parameter saves it as a session default, so later calls can omit it. -Example: calling `build_sim(scheme: "MyApp")` 3 times in a row auto-saves "MyApp" as the default scheme. - -**Opt out:** set `autoPromote: false` in `.xcforge.yaml`. The streak counter then stays at zero, so explicit values never get promoted — useful for iterative test loops where you want each call to stand on its own. Auto-promotion is also no longer sticky after the fact: `set_defaults` resets the matching field's streak so an old explicit value can't ambush a later run after you've overridden it. +Example: with `autoPromote: true`, calling `build_sim(scheme: "MyApp")` 3 times in a row saves "MyApp" as the default scheme. `defaults show` notes when a value was auto-promoted. Auto-promotion is also no longer sticky after the fact: `set_defaults` resets the matching field's streak so an old explicit value can't ambush a later run after you've overridden it. ## set_defaults (MCP tool) diff --git a/Skills/xcforge/references/cli-commands.md b/Skills/xcforge/references/cli-commands.md index a016774..0b61a3b 100644 --- a/Skills/xcforge/references/cli-commands.md +++ b/Skills/xcforge/references/cli-commands.md @@ -931,7 +931,7 @@ xcforge bless --baseline home-dark --tests "SnapshotTests/HomeTests" --project M | `--scheme ` | Auto-detected if omitted | | `--simulator ` | Auto-detected if omitted | -**Steps performed:** save baseline → run tests → compare visual → suggest `[bless] ` commit message. +**Steps performed:** save baseline → run tests → compare visual. **Exit code:** 0 when all tests pass and diff is within threshold, 1 otherwise. diff --git a/Skills/xcforge/references/screenshot-visual.md b/Skills/xcforge/references/screenshot-visual.md index ed4ce9e..eb01a20 100644 --- a/Skills/xcforge/references/screenshot-visual.md +++ b/Skills/xcforge/references/screenshot-visual.md @@ -92,8 +92,7 @@ Codify the baseline-write → test → diff → commit cycle in one call. Saves 1. Saves a visual baseline via `save_visual_baseline` 2. Runs tests via `build_and_test` with the provided filter 3. Compares current screenshot against the saved baseline -4. Suggests a commit message: `[bless] ` -**Returns:** Pass/fail status, baseline path, test result summary, visual diff result, and suggested commit message. Returns failure if any test fails. +**Returns:** Pass/fail status, baseline path, test result summary, and visual diff result. Returns failure if any test fails. **CLI:** `xcforge bless --baseline --tests ` diff --git a/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift b/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift index 941cb8f..8175915 100644 --- a/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift +++ b/Sources/XCForgeCLI/Commands/Build/XcodebuildOptionGroup.swift @@ -38,6 +38,11 @@ struct XcodebuildOptionGroup: ParsableArguments { help: "Keep building after the first error so one run reports every error. Default: on.") var continueAfterErrors: Bool? + @Flag( + inversion: .prefixedNo, + help: "Add -skipMacroValidation, -parallelizeTargets and compilation caching to builds. Default: on.") + var defaultFlags: Bool? + var options: XcodebuildOptions { XcodebuildOptions( derivedDataPath: derivedDataPath.map { ($0 as NSString).expandingTildeInPath }, @@ -47,7 +52,8 @@ struct XcodebuildOptionGroup: ParsableArguments { lockWaitSeconds: lockWait.map { TimeInterval($0) }, minFreeGB: minFreeGb, idleTimeoutSeconds: idleTimeout.map { TimeInterval($0) }, - continueAfterErrors: continueAfterErrors + continueAfterErrors: continueAfterErrors, + defaultFlags: defaultFlags ) } diff --git a/Sources/XCForgeCLI/Commands/Debug/DebugCommand.swift b/Sources/XCForgeCLI/Commands/Debug/DebugCommand.swift index 4863be7..10f8eab 100644 --- a/Sources/XCForgeCLI/Commands/Debug/DebugCommand.swift +++ b/Sources/XCForgeCLI/Commands/Debug/DebugCommand.swift @@ -327,6 +327,9 @@ struct DebugContinue: AsyncParsableCommand { @Option(help: "Execution mode: continue, step-over, step-into, step-out (default: continue).") var mode: String = "continue" + @Option(help: "Seconds to wait for the process to stop again (breakpoint, step done).") + var timeout: Int = 30 + @Flag(help: "Emit the result as machine-readable JSON.") var json = false @@ -341,11 +344,11 @@ struct DebugContinue: AsyncParsableCommand { case "step_out": command = "finish" default: command = "continue" } + let wait = TimeInterval(max(1, timeout)) do { let pid = try await target.resolvePID() let result = try await DebuggerOneShot.withSession(pid: pid) { session in - // 10-second timeout per spec I/O matrix - let output = try await session.sendCommand(command, timeout: 10) + let output = try await session.sendCommand(command, timeout: wait) return DebuggerProvider.parseContinueOutput(output) } if useJSON { diff --git a/Sources/XCForgeKit/Persistence/DefaultsStore.swift b/Sources/XCForgeKit/Persistence/DefaultsStore.swift index 87b2aa5..d62272c 100644 --- a/Sources/XCForgeKit/Persistence/DefaultsStore.swift +++ b/Sources/XCForgeKit/Persistence/DefaultsStore.swift @@ -582,9 +582,8 @@ public enum RepoConfig { /// Default test watchdog timeout in seconds, applied when no explicit /// `timeoutSeconds` is passed. Overrides the built-in 1800s/7200s defaults. public var testTimeout: Int? - /// When `false`, suppresses the 3-strikes silent promotion of explicit - /// values to sticky session defaults. Defaults to `true` (legacy behavior) - /// when the key is omitted. + /// When `true`, three consecutive uses of the same explicit value promote it to a + /// sticky session default. Off when the key is omitted. public var autoPromote: Bool? /// `-derivedDataPath` for every xcodebuild call. Relative paths resolve against the yaml's folder. public var derivedDataPath: String? @@ -596,6 +595,8 @@ public enum RepoConfig { public var minFreeGB: Double? /// Seconds of xcodebuild silence before it is treated as hung. 0 disables. public var idleTimeout: TimeInterval? + /// False drops the flags xcforge adds to every build. + public var defaultFlags: Bool? public init( project: String? = nil, @@ -609,7 +610,8 @@ public enum RepoConfig { buildLock: String? = nil, artifactDir: String? = nil, minFreeGB: Double? = nil, - idleTimeout: TimeInterval? = nil + idleTimeout: TimeInterval? = nil, + defaultFlags: Bool? = nil ) { self.project = project self.scheme = scheme @@ -623,6 +625,7 @@ public enum RepoConfig { self.artifactDir = artifactDir self.minFreeGB = minFreeGB self.idleTimeout = idleTimeout + self.defaultFlags = defaultFlags } /// True when every field is nil (nothing to apply). @@ -630,7 +633,7 @@ public enum RepoConfig { project == nil && scheme == nil && simulator == nil && configuration == nil && testPlan == nil && testTimeout == nil && autoPromote == nil && derivedDataPath == nil && buildLock == nil && artifactDir == nil && minFreeGB == nil - && idleTimeout == nil + && idleTimeout == nil && defaultFlags == nil } } @@ -684,7 +687,7 @@ public enum RepoConfig { let allowedKeys: Set = [ "project", "scheme", "simulator", "configuration", "testPlan", "testTimeout", "autoPromote", "derivedDataPath", "buildLock", "artifactDir", "minFreeGB", - "idleTimeout", + "idleTimeout", "defaultFlags", ] for key in dict.keys where !allowedKeys.contains(key) { Log.warn("\(RepoConfig.fileName): ignoring unknown key '\(key)'") @@ -767,7 +770,8 @@ public enum RepoConfig { buildLock: resolvedPath("buildLock"), artifactDir: resolvedPath("artifactDir"), minFreeGB: minFreeGB, - idleTimeout: idleTimeout + idleTimeout: idleTimeout, + defaultFlags: dict["defaultFlags"].flatMap(XcodebuildOptions.parseBool) ) return result.isEmpty ? nil : result } @@ -832,7 +836,7 @@ public enum RepoConfig { lines.append( entry( "autoPromote", nil, - "Set to false to disable 3-strikes auto-promotion of explicit values. Default: true." + "Set to true to save a value as a session default after 3 consecutive explicit uses. Default: false." ).trimmingCharacters(in: .newlines)) return lines.joined(separator: "\n") + "\n" } diff --git a/Sources/XCForgeKit/Support/WorkspaceContext.swift b/Sources/XCForgeKit/Support/WorkspaceContext.swift index 8781a3f..64f5ff2 100644 --- a/Sources/XCForgeKit/Support/WorkspaceContext.swift +++ b/Sources/XCForgeKit/Support/WorkspaceContext.swift @@ -558,11 +558,12 @@ public actor SessionState { value: String, streak: inout (value: String, count: Int), stored: inout String?, source: inout DefaultsSource ) { - // `.xcforge.yaml autoPromote: false` opts out of the 3-strikes promotion. + // Promotion is opt-in (`.xcforge.yaml autoPromote: true`): silently turning repeated + // explicit values into sticky defaults surprised agents that switch targets on purpose. // When disabled, do not touch the streak counter at all (P10): toggling // autoPromote on later — or persisting/reloading state — would otherwise // trip the threshold from a single use. - let autoPromoteEnabled = repoDefaults?.autoPromote ?? true + let autoPromoteEnabled = repoDefaults?.autoPromote ?? false guard autoPromoteEnabled else { streak = ("", 0) return diff --git a/Sources/XCForgeKit/Support/Xcodebuild.swift b/Sources/XCForgeKit/Support/Xcodebuild.swift index 135cc49..02564ff 100644 --- a/Sources/XCForgeKit/Support/Xcodebuild.swift +++ b/Sources/XCForgeKit/Support/Xcodebuild.swift @@ -70,8 +70,10 @@ public enum Xcodebuild { /// Insert `-derivedDataPath` and extra arguments before the first action token. /// Arguments the caller already passed are never duplicated. - static func apply(_ options: XcodebuildOptions, to arguments: [String]) -> [String] { - guard isBuildInvocation(arguments) else { return arguments } + static func apply(_ options: XcodebuildOptions, to original: [String]) -> [String] { + guard isBuildInvocation(original) else { return original } + let arguments = + options.defaultFlags == false ? original.filter { !xcforgeDefaultFlags.contains($0) } : original var insert: [String] = [] if let dd = options.derivedDataPath, !dd.isEmpty, !arguments.contains("-derivedDataPath"), !arguments.contains("-xctestrun") @@ -92,6 +94,11 @@ public enum Xcodebuild { return result } + /// Flags xcforge adds to builds on its own. `defaultFlags: false` removes them. + static let xcforgeDefaultFlags: Set = [ + "-skipMacroValidation", "-parallelizeTargets", "COMPILATION_CACHE_ENABLE_CACHING=YES", + ] + /// Xcode user default that keeps the build going after the first error, so every /// error in the run is reported instead of only the first target's. static let continueAfterErrorsDefault = "-IDEBuildingContinueBuildingAfterErrors" diff --git a/Sources/XCForgeKit/Support/XcodebuildOptions.swift b/Sources/XCForgeKit/Support/XcodebuildOptions.swift index 439f524..7cd11d0 100644 --- a/Sources/XCForgeKit/Support/XcodebuildOptions.swift +++ b/Sources/XCForgeKit/Support/XcodebuildOptions.swift @@ -29,6 +29,10 @@ public struct XcodebuildOptions: Sendable, Equatable { /// Keep compiling other files and targets after the first error so one run reports them all. /// Default: on. public var continueAfterErrors: Bool? + /// When false, drop the flags xcforge adds on its own (`-skipMacroValidation`, + /// `-parallelizeTargets`, `COMPILATION_CACHE_ENABLE_CACHING=YES`) so the build matches a + /// plain xcodebuild invocation. Default: true. + public var defaultFlags: Bool? public init( derivedDataPath: String? = nil, @@ -39,7 +43,8 @@ public struct XcodebuildOptions: Sendable, Equatable { minFreeGB: Double? = nil, artifactDir: String? = nil, idleTimeoutSeconds: TimeInterval? = nil, - continueAfterErrors: Bool? = nil + continueAfterErrors: Bool? = nil, + defaultFlags: Bool? = nil ) { self.derivedDataPath = derivedDataPath self.resultBundlePath = resultBundlePath @@ -50,6 +55,7 @@ public struct XcodebuildOptions: Sendable, Equatable { self.artifactDir = artifactDir self.idleTimeoutSeconds = idleTimeoutSeconds self.continueAfterErrors = continueAfterErrors + self.defaultFlags = defaultFlags } /// Options set for the current call (CLI flags or MCP arguments). @@ -93,10 +99,20 @@ public struct XcodebuildOptions: Sendable, Equatable { explicit.idleTimeoutSeconds ?? environment["XCFORGE_IDLE_TIMEOUT"].flatMap { TimeInterval($0) } ?? repo?.idleTimeout merged.continueAfterErrors = explicit.continueAfterErrors + merged.defaultFlags = + explicit.defaultFlags ?? environment["XCFORGE_DEFAULT_FLAGS"].flatMap(parseBool) ?? repo?.defaultFlags merged.extraArgs = explicit.extraArgs return merged } + static func parseBool(_ raw: String) -> Bool? { + switch raw.lowercased() { + case "1", "true", "yes", "on": return true + case "0", "false", "no", "off": return false + default: return nil + } + } + /// Directory for generated artifacts. Defaults to `/tmp`. public static func artifactDirectory() -> String { let dir = effective().artifactDir ?? "/tmp" @@ -181,6 +197,12 @@ public struct XcodebuildOptions: Sendable, Equatable { "description": .string( "Keep building after the first error so one run reports every error. Default: true."), ]), + "defaultFlags": .object([ + "type": .string("boolean"), + "description": .string( + "Set false to drop the flags xcforge adds (-skipMacroValidation, -parallelizeTargets, COMPILATION_CACHE_ENABLE_CACHING=YES). Default: true." + ), + ]), ] /// Parse the option keys out of an MCP argument dictionary. @@ -203,7 +225,8 @@ public struct XcodebuildOptions: Sendable, Equatable { minFreeGB: number(args["minFreeGB"]), artifactDir: nil, idleTimeoutSeconds: number(args["idleTimeoutSeconds"]), - continueAfterErrors: args["continueAfterErrors"]?.boolValue + continueAfterErrors: args["continueAfterErrors"]?.boolValue, + defaultFlags: args["defaultFlags"]?.boolValue ) } diff --git a/Sources/XCForgeKit/Tools/BlessProvider.swift b/Sources/XCForgeKit/Tools/BlessProvider.swift index f8fbd49..17d9666 100644 --- a/Sources/XCForgeKit/Tools/BlessProvider.swift +++ b/Sources/XCForgeKit/Tools/BlessProvider.swift @@ -115,12 +115,6 @@ public enum BlessTools: ToolProvider { lines.append("Visual diff: skipped (baseline path not captured from save output)") } - // Step 4: Suggest commit message - let slug = commitSlug(from: input.tests) - lines.append("") - lines.append("Suggested commit:") - lines.append(" git commit -m \"[bless] \(slug)\"") - return testPassed ? .ok(lines.joined(separator: "\n")) : .fail(lines.joined(separator: "\n")) } @@ -142,14 +136,4 @@ public enum BlessTools: ToolProvider { } return nil } - - static func commitSlug(from filter: String) -> String { - let slug = - filter - .lowercased() - .replacingOccurrences(of: "/", with: "-") - .replacingOccurrences(of: " ", with: "-") - let trimmed = String(slug.prefix(40)) - return trimmed.trimmingCharacters(in: .init(charactersIn: "-")) - } } diff --git a/Sources/XCForgeKit/Tools/BuildProvider.swift b/Sources/XCForgeKit/Tools/BuildProvider.swift index 69a5760..db7c1f0 100644 --- a/Sources/XCForgeKit/Tools/BuildProvider.swift +++ b/Sources/XCForgeKit/Tools/BuildProvider.swift @@ -1106,14 +1106,14 @@ public enum BuildTools { launchResult = try await env.shell.run( "/usr/bin/xcrun", arguments: ["simctl", "launch", "--terminate-running-process", udid, bundleId], - timeout: 15) + timeout: SimTools.launchTimeout) } catch { return .fail("Build + Install succeeded\nLaunch error: \(error)") } guard launchResult.succeeded else { if launchResult.exitCode == -1 { - return .fail("Build + Install succeeded\nLaunch timed out after 15s") + return .fail("Build + Install succeeded\nLaunch timed out after \(Int(SimTools.launchTimeout))s") } return .fail("Build + Install succeeded\nLaunch FAILED: \(launchResult.stderr)") } diff --git a/Sources/XCForgeKit/Tools/DebuggerProvider.swift b/Sources/XCForgeKit/Tools/DebuggerProvider.swift index 043c054..223b0da 100644 --- a/Sources/XCForgeKit/Tools/DebuggerProvider.swift +++ b/Sources/XCForgeKit/Tools/DebuggerProvider.swift @@ -235,6 +235,11 @@ extension DebuggerProvider { "type": .string("string"), "description": .string("Execution mode: continue, step_over, step_into, step_out (default: continue)."), ]), + "timeoutSeconds": .object([ + "type": .string("integer"), + "description": .string( + "How long to wait for the process to stop again (breakpoint, step done). Default: 30."), + ]), ]), "required": .array([.string("sessionId")]), ]) @@ -295,6 +300,7 @@ extension DebuggerProvider { struct ContinueInput: Decodable { let sessionId: String let mode: String? + let timeoutSeconds: Int? } struct RunCommandInput: Decodable { @@ -551,8 +557,8 @@ extension DebuggerProvider { } do { - // 10-second timeout per spec I/O matrix - let output = try await session.sendCommand(command, timeout: 10) + let timeout = TimeInterval(max(1, input.timeoutSeconds ?? 30)) + let output = try await session.sendCommand(command, timeout: timeout) let result = parseContinueOutput(output) let data = try JSONEncoder().encode(result) return .ok(String(data: data, encoding: .utf8) ?? "{}") diff --git a/Sources/XCForgeKit/Tools/SimulatorProvider.swift b/Sources/XCForgeKit/Tools/SimulatorProvider.swift index 4762ca1..69adfb8 100644 --- a/Sources/XCForgeKit/Tools/SimulatorProvider.swift +++ b/Sources/XCForgeKit/Tools/SimulatorProvider.swift @@ -346,7 +346,7 @@ public enum SimTools { name: "set_orientation", description: """ Set device orientation (portrait/landscape) via WDA. \ - Uses XCUIDevice.shared.orientation — the only reliable programmatic method. \ + Uses XCUIDevice.shared.orientation. \ Returns the confirmed orientation after change. """, inputSchema: .object([ @@ -602,7 +602,8 @@ public enum SimTools { if launch.timedOut { return SimResult( - succeeded: false, message: "Launch timed out after 15s. The simulator may need a restart." + succeeded: false, + message: "Launch timed out after \(Int(launchTimeout))s. The simulator may be overloaded or need a restart." ) } @@ -994,6 +995,10 @@ public enum SimTools { // MARK: - Internal Helpers + /// Seconds `simctl launch` may take. Generous because a busy shared Mac can be slow to + /// spawn the first process after boot. + static let launchTimeout: TimeInterval = 60 + static func launchAppStructured( simulatorUDID: String, bundleId: String, args: [String]? = nil, env: Environment ) async throws -> StructuredAppLaunch { @@ -1010,7 +1015,7 @@ public enum SimTools { let result = try await env.shell.run( "/usr/bin/xcrun", arguments: launchArgs, - timeout: 15 + timeout: launchTimeout ) return StructuredAppLaunch( diff --git a/Sources/XCForgeKit/Tools/TestProvider.swift b/Sources/XCForgeKit/Tools/TestProvider.swift index 66fff1e..f6a3c5e 100644 --- a/Sources/XCForgeKit/Tools/TestProvider.swift +++ b/Sources/XCForgeKit/Tools/TestProvider.swift @@ -649,7 +649,7 @@ public enum TestTools { description: """ Build an iOS app then run tests in one call. Short-circuits on build failure \ with structured diagnostics (errors with file:line). If build succeeds, runs \ - tests and returns pass/fail summary. Preferred over separate build_and_diagnose + test_sim calls. \ + tests and returns pass/fail summary. \ Project, scheme, and simulator are auto-detected if omitted. """, inputSchema: .object([ diff --git a/Tests/XCForgeKitTests/RepoConfigTests.swift b/Tests/XCForgeKitTests/RepoConfigTests.swift index 3d699cd..72863b0 100644 --- a/Tests/XCForgeKitTests/RepoConfigTests.swift +++ b/Tests/XCForgeKitTests/RepoConfigTests.swift @@ -766,12 +766,25 @@ struct RepoConfigTests { #expect(!shown.contains("auto-promoted")) } - @Test("autoPromote defaults to true when key omitted (legacy behavior)") - func autoPromoteDefaultsTrue() async { + @Test("autoPromote is off when the key is omitted and on with autoPromote: true") + func autoPromoteOptIn() async { + let offRoot = makeTempDir() + defer { cleanup(offRoot) } + createDir(offRoot.appendingPathComponent(".git")) + writeFile("scheme: AnyScheme\n", at: offRoot) + let offStoreDir = offRoot.appendingPathComponent("store", isDirectory: true) + createDir(offStoreDir) + let offStore = DefaultsStore(baseDirectory: offStoreDir) + let offSession = SessionState(defaultsStore: offStore, cwd: offRoot.path) + for _ in 0..<5 { + _ = try? await offSession.resolveScheme("PromoteMe", project: "/dummy.xcodeproj") + } + #expect(!(await offSession.showDefaults()).contains("auto-promoted")) + let root = makeTempDir() defer { cleanup(root) } createDir(root.appendingPathComponent(".git")) - writeFile("scheme: AnyScheme\n", at: root) + writeFile("scheme: AnyScheme\nautoPromote: true\n", at: root) let storeDir = root.appendingPathComponent("store", isDirectory: true) createDir(storeDir) @@ -892,7 +905,7 @@ struct RepoConfigTests { let root = makeTempDir() defer { cleanup(root) } createDir(root.appendingPathComponent(".git")) - // autoPromote defaults to true. + writeFile("autoPromote: true\n", at: root) let storeDir = root.appendingPathComponent("store", isDirectory: true) createDir(storeDir) diff --git a/Tests/XCForgeKitTests/SharedMacSafetyTests.swift b/Tests/XCForgeKitTests/SharedMacSafetyTests.swift index 1ac0f91..422be35 100644 --- a/Tests/XCForgeKitTests/SharedMacSafetyTests.swift +++ b/Tests/XCForgeKitTests/SharedMacSafetyTests.swift @@ -58,6 +58,15 @@ struct SharedMacSafetyTests { #expect(Xcodebuild.apply(XcodebuildOptions(), to: explicit) == explicit) } + @Test("defaultFlags: false strips the flags xcforge adds on its own") + func defaultFlagsOff() { + let args = ["-skipMacroValidation", "-parallelizeTargets", "build", "COMPILATION_CACHE_ENABLE_CACHING=YES"] + let off = XcodebuildOptions(continueAfterErrors: false, defaultFlags: false) + #expect(Xcodebuild.apply(off, to: args) == ["build"]) + #expect(Xcodebuild.apply(XcodebuildOptions(continueAfterErrors: false), to: args) == args) + #expect(XcodebuildOptions.effective(cwd: "/", environment: ["XCFORGE_DEFAULT_FLAGS": "0"]).defaultFlags == false) + } + // MARK: - Timeouts @Test("timeout kind distinguishes idle kills from total-limit kills") diff --git a/Web/src/content/docs/guides/workflow-patterns.mdx b/Web/src/content/docs/guides/workflow-patterns.mdx index 83fba82..a52b1da 100644 --- a/Web/src/content/docs/guides/workflow-patterns.mdx +++ b/Web/src/content/docs/guides/workflow-patterns.mdx @@ -159,7 +159,6 @@ What it does: 1. Saves the current simulator screenshot as a named visual baseline. 2. Runs the specified tests via `build-test`. 3. Compares the post-test screenshot against the saved baseline. -4. Outputs a suggested commit message: `[bless] uitests-logintests`. Why this pattern works: From 737c7fe02942e1ea1169f9cda8b2e2eab9be3556 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 14:35:31 +0000 Subject: [PATCH 05/28] test: serialize WDA URLProtocol stubs across suites Suites that stub WDA's localhost URL ran in parallel and could answer each other's requests. A shared async gate now holds each stubbed region. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01NVNakyntY4u9MvhsjbzRhZ --- .../AgentClientMultiWindowTests.swift | 4 +++ .../XCForgeKitTests/UIInteractionTests.swift | 2 ++ .../XCForgeKitTests/WDABundleHintTests.swift | 2 ++ .../WDASessionRetryTests.swift | 2 ++ Tests/XCForgeKitTests/WDAStubGate.swift | 34 +++++++++++++++++++ 5 files changed, 44 insertions(+) create mode 100644 Tests/XCForgeKitTests/WDAStubGate.swift diff --git a/Tests/XCForgeKitTests/AgentClientMultiWindowTests.swift b/Tests/XCForgeKitTests/AgentClientMultiWindowTests.swift index 97fffde..e136a1c 100644 --- a/Tests/XCForgeKitTests/AgentClientMultiWindowTests.swift +++ b/Tests/XCForgeKitTests/AgentClientMultiWindowTests.swift @@ -15,6 +15,8 @@ struct AgentClientMultiWindowTests { @Test("getSource returns a payload that contains both the keyWindow and a secondary window subtree") func sourceReturnsBothWindows() async throws { + await WDAStubGate.shared.acquire() + defer { WDAStubGate.shared.releaseSoon() } StubWDAProtocol.reset() URLProtocol.registerClass(StubWDAProtocol.self) defer { @@ -53,6 +55,8 @@ struct AgentClientMultiWindowTests { @Test("findElement returns a sheet-window element id when WDA's per-window retry resolved it") func findElementResolvesSheetMember() async throws { + await WDAStubGate.shared.acquire() + defer { WDAStubGate.shared.releaseSoon() } StubWDAProtocol.reset() URLProtocol.registerClass(StubWDAProtocol.self) defer { diff --git a/Tests/XCForgeKitTests/UIInteractionTests.swift b/Tests/XCForgeKitTests/UIInteractionTests.swift index 3b38ce8..1cbef6b 100644 --- a/Tests/XCForgeKitTests/UIInteractionTests.swift +++ b/Tests/XCForgeKitTests/UIInteractionTests.swift @@ -89,6 +89,8 @@ private func makeStubbedClient() async -> WDAClient { } private func withStubbedWDA(_ body: () async throws -> T) async rethrows -> T { + await WDAStubGate.shared.acquire() + defer { WDAStubGate.shared.releaseSoon() } StubWDAUITap.reset() URLProtocol.registerClass(StubWDAUITap.self) defer { diff --git a/Tests/XCForgeKitTests/WDABundleHintTests.swift b/Tests/XCForgeKitTests/WDABundleHintTests.swift index 60c1988..18e30a3 100644 --- a/Tests/XCForgeKitTests/WDABundleHintTests.swift +++ b/Tests/XCForgeKitTests/WDABundleHintTests.swift @@ -108,6 +108,8 @@ private final class StubWDABundleHint: URLProtocol, @unchecked Sendable { } private func withStubbedBundleHintWDA(_ body: () async throws -> T) async rethrows -> T { + await WDAStubGate.shared.acquire() + defer { WDAStubGate.shared.releaseSoon() } StubWDABundleHint.reset() URLProtocol.registerClass(StubWDABundleHint.self) defer { diff --git a/Tests/XCForgeKitTests/WDASessionRetryTests.swift b/Tests/XCForgeKitTests/WDASessionRetryTests.swift index c6b5ada..ccf506e 100644 --- a/Tests/XCForgeKitTests/WDASessionRetryTests.swift +++ b/Tests/XCForgeKitTests/WDASessionRetryTests.swift @@ -84,6 +84,8 @@ final class StubWDAProtocol: URLProtocol, @unchecked Sendable { /// Run `body` with `StubWDAProtocol` registered globally; reset state on entry/exit. private func withStubbedWDA(_ body: () async throws -> T) async rethrows -> T { + await WDAStubGate.shared.acquire() + defer { WDAStubGate.shared.releaseSoon() } StubWDAProtocol.reset() URLProtocol.registerClass(StubWDAProtocol.self) defer { diff --git a/Tests/XCForgeKitTests/WDAStubGate.swift b/Tests/XCForgeKitTests/WDAStubGate.swift new file mode 100644 index 0000000..ff83818 --- /dev/null +++ b/Tests/XCForgeKitTests/WDAStubGate.swift @@ -0,0 +1,34 @@ +import Foundation + +/// Serializes every test that registers a URLProtocol stub for WDA's localhost URL. +/// +/// Each WDA suite is `.serialized` internally, but suites run in parallel and all their +/// stubs answer `localhost`, so one suite's stub (or its `reset()`) could swallow another +/// suite's requests. Holding this gate for the whole stubbed region keeps them apart. +actor WDAStubGate { + static let shared = WDAStubGate() + + private var busy = false + private var waiters: [CheckedContinuation] = [] + + func acquire() async { + if !busy { + busy = true + return + } + await withCheckedContinuation { waiters.append($0) } + } + + func release() { + if waiters.isEmpty { + busy = false + } else { + waiters.removeFirst().resume() + } + } + + /// Release from a synchronous `defer`. + nonisolated func releaseSoon() { + Task { await release() } + } +} From 7e0a793f4dbf544e9251585aa913f407d8f40d7e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 15:34:50 +0000 Subject: [PATCH 06/28] feat: stop orphaned builds, report progress, scope config per project - Cancelling an MCP call, a timeout, or xcforge exiting now kills the whole child process tree, so xcodebuild no longer keeps DerivedData locked. - MCP calls with a progress token get a notification every 10s with the latest output line, so clients don't time out long builds. - xcodebuild calls on the same DerivedData or project queue in-process. - An explicit project uses its own repo's .xcforge.yaml; results name the project used; yaml quotes and comments are handled and problems warned. - Unknown MCP arguments are rejected with the closest real name. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EtxtFPs6UBAWXqEpkj2W2N --- CHANGELOG.md | 8 + Package.swift | 2 +- Skills/xcforge/references/shared-mac.md | 10 + Sources/XCForgeCLI/Main.swift | 62 +++++- Sources/XCForgeKit/Clients/AgentClient.swift | 5 +- .../Persistence/DefaultsStore.swift | 65 +++++- Sources/XCForgeKit/ProviderRegistry.swift | 89 +++++++- .../XCForgeKit/Support/ProcessRunner.swift | 149 +++++++++++-- .../XCForgeKit/Support/WorkspaceContext.swift | 42 ++++ Sources/XCForgeKit/Support/Xcodebuild.swift | 101 ++++++++- .../Tools/InteractionProvider.swift | 7 +- .../AgentCallSafetyTests.swift | 208 ++++++++++++++++++ 12 files changed, 706 insertions(+), 42 deletions(-) create mode 100644 Tests/XCForgeKitTests/AgentCallSafetyTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index ae95f41..b45cffb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,9 +18,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - **UI automation on physical devices.** `xcforge wda start|status|stop` and MCP `wda_start`/`wda_stop` build xcforgeWDA for the device, sign it with your team (`--team`, `XCFORGE_WDA_TEAM`), launch it with `test-without-building`, find it over the CoreDevice tunnel, and record its URL in `~/.xcforge/wda/`. `XCFORGE_DEVICE=` points `xcforge ui` commands at it. Locked device, UI Automation off, Developer Mode off, pairing and signing failures each get a one-line explanation - `device screenshot` / `device_screenshot` (devicectl capture, falling back to the device's WDA) - `device launch --url --env KEY=VALUE --arg A` and matching `url`/`env` on `device_launch` +- **MCP progress notifications.** When a `tools/call` carries a `progressToken`, xcforge sends `notifications/progress` every 10s with the elapsed time and the latest output line, so clients that time out silent requests keep waiting for long builds and tests +- Each build and test tool result ends with `project: `, so a build of the wrong checkout or worktree is visible +- `.xcforge.yaml` problems (unknown keys with a "did you mean", unparseable values) are listed by `set_defaults show` / `defaults show` and once at the end of the first tool result that uses the file - **Xcode 27 Device Hub support.** xcforge opens Device Hub when the selected Xcode has no Simulator.app, and screen capture, the accessibility bridge and window scripting accept either app ### Changed +- **Cancelling stops the build.** An MCP cancel, a client disconnect, Ctrl-C, SIGTERM or SIGHUP now stops the xcodebuild (or other command) xcforge started, together with every process it spawned. Timeouts kill the whole process tree too, so compiler and test-runner children no longer outlive the call and hold DerivedData ("database is locked" on the next run) +- Two MCP calls that would run xcodebuild on the same DerivedData folder (or the same project's default one) at once now run one after the other, first come first served +- An explicit `project` in another repo or worktree uses the `.xcforge.yaml` next to it, for both session defaults (scheme, simulator, test plan) and xcodebuild options (DerivedData, lock, idle timeout) +- `.xcforge.yaml` values may be quoted and may carry a trailing `# comment` +- **Unknown MCP arguments are an error** naming the closest real argument (`derived_data_path` → `derivedDataPath`) instead of being ignored - **Auto-promotion is opt-in.** Repeated explicit values no longer silently become session defaults unless `.xcforge.yaml` sets `autoPromote: true` - `--no-default-flags` / `defaultFlags: false` / `XCFORGE_DEFAULT_FLAGS=0` / `.xcforge.yaml defaultFlags: false` drop the flags xcforge adds to builds (`-skipMacroValidation`, `-parallelizeTargets`, `COMPILATION_CACHE_ENABLE_CACHING=YES`) - `lldb_continue` / `debug continue` wait 30s by default and take `timeoutSeconds` / `--timeout` (was a fixed 10s); simulator app launch waits up to 60s (was 15s) diff --git a/Package.swift b/Package.swift index cd52bce..253002e 100644 --- a/Package.swift +++ b/Package.swift @@ -5,7 +5,7 @@ let package = Package( name: "xcforge", platforms: [.macOS(.v13)], dependencies: [ - .package(url: "https://github.com/modelcontextprotocol/swift-sdk.git", from: "0.11.0"), + .package(url: "https://github.com/modelcontextprotocol/swift-sdk.git", from: "0.12.0"), .package(url: "https://github.com/apple/swift-log.git", from: "1.0.0"), .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.5.0"), ], diff --git a/Skills/xcforge/references/shared-mac.md b/Skills/xcforge/references/shared-mac.md index 2c06bf4..4c52037 100644 --- a/Skills/xcforge/references/shared-mac.md +++ b/Skills/xcforge/references/shared-mac.md @@ -35,6 +35,16 @@ MCP: `build_lock_status`. `simRecovery` / `--sim-recovery` defaults to `off` everywhere. `auto` reboots a simulator that isn't Booted. `erase` also erases it when a reboot didn't help. Unknown values are an error. +## Long calls, cancellation and parallel calls + +- MCP clients that send a `progressToken` get a progress notification every 10s with the latest output line. +- Cancelling a call (MCP cancel, client disconnect, Ctrl-C, SIGTERM) stops the xcodebuild xcforge started and every process it spawned. A CLI agent whose shell tool kills `xcforge` after its own timeout no longer leaves an orphaned build behind. +- Within one MCP server, xcodebuild calls on the same DerivedData folder (or the same project's default one) run one at a time, in arrival order. This is always on; `buildLock` adds the same guarantee across processes. + +## Which `.xcforge.yaml` applies + +The one nearest the project being built, walking up to its repo root. Passing `project` from another worktree uses that worktree's file. Values may be quoted and may end in a `# comment`. Unknown keys and bad values are listed in `defaults show` and noted once in the first tool result that uses the file. + ## What xcforge never does on a shared Mac - Kill an xcodebuild it didn't start. Hang diagnostics sample the process matched by this run's result bundle path. diff --git a/Sources/XCForgeCLI/Main.swift b/Sources/XCForgeCLI/Main.swift index ed4bb5c..94d98cc 100644 --- a/Sources/XCForgeCLI/Main.swift +++ b/Sources/XCForgeCLI/Main.swift @@ -1,4 +1,6 @@ import ArgumentParser +import Dispatch +import Foundation import Logging import MCP import XCForgeKit @@ -6,6 +8,11 @@ import XCForgeKit @main struct Main { static func main() async throws { + // Ctrl-C, a shell tool's timeout kill or a client hang-up must not leave xcodebuild + // running with DerivedData locked. Stop our children before exiting. + let signalSources = installTerminationHandlers() + defer { signalSources.forEach { $0.cancel() } } + if CommandLine.arguments.count > 1 { await XCForgeCLI.main() } else { @@ -22,12 +29,65 @@ struct Main { } await server.withMethodHandler(CallTool.self) { params in - await ToolRegistry.dispatch(params.name, params.arguments) + guard let token = params._meta?.progressToken else { + return await ToolRegistry.dispatch(params.name, params.arguments) + } + return await ProgressHeartbeat.run(token: token, server: server) { + await ToolRegistry.dispatch(params.name, params.arguments) + } } let transport = StdioTransport(logger: logger) try await server.start(transport: transport) await server.waitUntilCompleted() + // The client closed stdin: nothing will read the results of calls still running. + ChildProcesses.terminateAll() + } + } + + /// Forward SIGINT, SIGTERM and SIGHUP to running children, then exit with the usual + /// 128 + signal status. + static func installTerminationHandlers() -> [DispatchSourceSignal] { + [SIGINT, SIGTERM, SIGHUP].map { sig in + signal(sig, SIG_IGN) + let source = DispatchSource.makeSignalSource(signal: sig, queue: .global()) + source.setEventHandler { + ChildProcesses.terminateAll(grace: 1) + // Give SIGTERM'd children a moment to exit before we go. + Thread.sleep(forTimeInterval: ChildProcesses.running.isEmpty ? 0 : 1.2) + exit(128 + sig) + } + source.resume() + return source + } + } +} + +/// Sends MCP progress notifications while a tool call runs, so clients that time out +/// silent requests keep waiting for long builds and tests. +enum ProgressHeartbeat { + /// Seconds between notifications. + static let interval: UInt64 = 10 + + static func run( + token: ProgressToken, server: Server, _ body: @Sendable () async -> CallTool.Result + ) async -> CallTool.Result { + let activity = ToolActivity() + let started = Date() + let heartbeat = Task { + var beats = 0.0 + while !Task.isCancelled { + try? await Task.sleep(nanoseconds: interval * 1_000_000_000) + if Task.isCancelled { break } + beats += 1 + let elapsed = Int(Date().timeIntervalSince(started)) + let message = activity.lastLine.map { "\(elapsed)s: \($0)" } ?? "\(elapsed)s elapsed" + try? await server.notify( + ProgressNotification.message( + .init(progressToken: token, progress: beats, message: message))) + } } + defer { heartbeat.cancel() } + return await ToolActivity.$current.withValue(activity) { await body() } } } diff --git a/Sources/XCForgeKit/Clients/AgentClient.swift b/Sources/XCForgeKit/Clients/AgentClient.swift index 9b7df0b..37866f5 100644 --- a/Sources/XCForgeKit/Clients/AgentClient.swift +++ b/Sources/XCForgeKit/Clients/AgentClient.swift @@ -322,7 +322,10 @@ public actor WDAClient { "-destination", "id=\(udid)", ] deployTask = Task.detached { - _ = try? await Shell.run("/usr/bin/xcrun", arguments: testArgs, timeout: 3600) + // The runner outlives this call by design; don't stop it when xcforge exits. + _ = try? await ChildProcesses.$untracked.withValue(true) { + try await Shell.run("/usr/bin/xcrun", arguments: testArgs, timeout: 3600) + } } // Step 3: Poll for server readiness (up to 30s) diff --git a/Sources/XCForgeKit/Persistence/DefaultsStore.swift b/Sources/XCForgeKit/Persistence/DefaultsStore.swift index d62272c..8906104 100644 --- a/Sources/XCForgeKit/Persistence/DefaultsStore.swift +++ b/Sources/XCForgeKit/Persistence/DefaultsStore.swift @@ -597,6 +597,10 @@ public enum RepoConfig { public var idleTimeout: TimeInterval? /// False drops the flags xcforge adds to every build. public var defaultFlags: Bool? + /// Problems found while reading the file (unknown keys, bad values). Shown to agents. + public var warnings: [String] = [] + /// The `.xcforge.yaml` these values came from. + public var sourcePath: String? public init( project: String? = nil, @@ -668,6 +672,12 @@ public enum RepoConfig { return nil } + var warnings: [String] = [] + func warn(_ message: String) { + warnings.append(message) + Log.warn(message) + } + var dict: [String: String] = [:] let normalized = contents.replacingOccurrences(of: "\r\n", with: "\n") .replacingOccurrences(of: "\r", with: "\n") @@ -676,21 +686,28 @@ public enum RepoConfig { if trimmed.isEmpty || trimmed.hasPrefix("#") { continue } guard let colonIdx = trimmed.firstIndex(of: ":") else { continue } let key = trimmed[trimmed.startIndex.. = [ "project", "scheme", "simulator", "configuration", "testPlan", "testTimeout", "autoPromote", "derivedDataPath", "buildLock", "artifactDir", "minFreeGB", "idleTimeout", "defaultFlags", ] - for key in dict.keys where !allowedKeys.contains(key) { - Log.warn("\(RepoConfig.fileName): ignoring unknown key '\(key)'") + for key in dict.keys.sorted() where !allowedKeys.contains(key) { + let suggestion = + allowedKeys.first { $0.lowercased() == key.lowercased() } + ?? FuzzyMatch.fuzzyRank(needle: key, candidates: Array(allowedKeys), maxResults: 1).first? + .candidate + let hint = suggestion.map { " (did you mean '\($0)'?)" } ?? "" + warn("\(RepoConfig.fileName): ignoring unknown key '\(key)'\(hint)") } // Resolve relative project path against config file directory @@ -701,7 +718,7 @@ public enum RepoConfig { if FileManager.default.fileExists(atPath: normalized) { project = normalized } else { - Log.warn( + warn( "\(RepoConfig.fileName): project path '\(p)' resolved to '\(normalized)' which does not exist — ignoring" ) project = nil @@ -715,8 +732,7 @@ public enum RepoConfig { if let parsed = Int(raw), parsed > 0 { testTimeout = parsed } else { - Log.warn( - "\(RepoConfig.fileName): ignoring non-numeric or non-positive testTimeout '\(raw)'") + warn("\(RepoConfig.fileName): ignoring non-numeric or non-positive testTimeout '\(raw)'") } } @@ -727,8 +743,7 @@ public enum RepoConfig { case "true": autoPromote = true case "false": autoPromote = false default: - Log.warn( - "\(RepoConfig.fileName): ignoring non-boolean autoPromote '\(raw)' (expected true/false)") + warn("\(RepoConfig.fileName): ignoring non-boolean autoPromote '\(raw)' (expected true/false)") } } @@ -745,7 +760,7 @@ public enum RepoConfig { if let parsed = Double(raw), parsed >= 0 { minFreeGB = parsed } else { - Log.warn("\(RepoConfig.fileName): ignoring non-numeric minFreeGB '\(raw)'") + warn("\(RepoConfig.fileName): ignoring non-numeric minFreeGB '\(raw)'") } } @@ -754,7 +769,7 @@ public enum RepoConfig { if let parsed = TimeInterval(raw), parsed >= 0 { idleTimeout = parsed } else { - Log.warn("\(RepoConfig.fileName): ignoring non-numeric idleTimeout '\(raw)'") + warn("\(RepoConfig.fileName): ignoring non-numeric idleTimeout '\(raw)'") } } @@ -773,7 +788,35 @@ public enum RepoConfig { idleTimeout: idleTimeout, defaultFlags: dict["defaultFlags"].flatMap(XcodebuildOptions.parseBool) ) - return result.isEmpty ? nil : result + var withWarnings = result + withWarnings.warnings = warnings + withWarnings.sourcePath = fileLabel + return result.isEmpty && warnings.isEmpty ? nil : withWarnings + } + + /// A yaml scalar without surrounding quotes or a trailing `# comment`. + /// `"My App" # dev` → `My App`; `iPhone 16 # dev` → `iPhone 16`; `a#b` stays `a#b`. + static func cleanValue(_ raw: String) -> String { + let value = raw.trimmingCharacters(in: .whitespaces) + if let quote = value.first, quote == "\"" || quote == "'" { + let afterOpen = value.index(after: value.startIndex) + if let close = value[afterOpen...].firstIndex(of: quote) { + return String(value[afterOpen.. value.startIndex, + value[value.index(before: index)].isWhitespace + { + cut = index + break + } + index = value.index(after: index) + } + return value[.. = [] + /// Groups currently disabled. MCP requests run concurrently, so access goes through a lock. + private static let disabledGroupsLock = OSAllocatedUnfairLock(initialState: Set()) + private static var disabledGroups: Set { disabledGroupsLock.withLock { $0 } } /// All valid group names derived from registered providers. public static var allGroups: [String] { @@ -37,7 +40,8 @@ public enum ToolRegistry { /// Active (non-disabled) providers. private static var activeProviders: [any ToolProvider.Type] { - providers.filter { !disabledGroups.contains($0.group) } + let disabled = disabledGroups + return providers.filter { !disabled.contains($0.group) } } public static var allTools: [Tool] { @@ -56,6 +60,9 @@ public enum ToolRegistry { // Handle built-in tool_groups before checking providers if name == "tool_groups" { return handleToolGroups(args) } + // A misspelled argument would otherwise be ignored and the call would quietly use a default. + if let rejection = unknownArgumentsError(name, args) { return rejection } + // Per-call xcodebuild options (derivedDataPath, buildLock, ...) ride in a task-local so // every xcodebuild call made while serving this tool sees them. let options = @@ -69,11 +76,78 @@ public enum ToolRegistry { } return nil } - if let result { return result } + if let result { return await withNotes(result, tool: name, env: env) } Log.warn("Unknown tool: \(name)") return .fail("Unknown tool: \(name)") } + /// `.xcforge.yaml` files whose warnings this process already showed. + private static let shownConfigWarnings = OSAllocatedUnfairLock(initialState: Set()) + + /// Add short trailing notes: which project a build or test ran against (so work in a second + /// worktree is visible), and `.xcforge.yaml` problems the first time each file is used. + static func withNotes(_ result: CallTool.Result, tool: String, env: Environment) async -> CallTool.Result { + var notes: [String] = [] + if XcodebuildOptions.mcpToolNames.contains(tool), let project = await env.session.activeProject { + notes.append("project: \(absolutePath(project))") + } + let warnings = await env.session.configWarnings + if !warnings.isEmpty, let path = await env.session.configPath, + shownConfigWarnings.withLock({ $0.insert(path).inserted }) + { + notes += warnings.map { "config warning: \($0)" } + } + guard !notes.isEmpty else { return result } + let note = Tool.Content.text(text: notes.joined(separator: "\n"), annotations: nil, _meta: nil) + return CallTool.Result( + content: result.content + [note], structuredContent: result.structuredContent, + isError: result.isError, _meta: result._meta) + } + + private static func absolutePath(_ path: String) -> String { + let expanded = (path as NSString).expandingTildeInPath + guard !expanded.hasPrefix("/") else { return (expanded as NSString).standardizingPath } + let cwd = FileManager.default.currentDirectoryPath as NSString + return (cwd.appendingPathComponent(expanded) as NSString).standardizingPath + } + + // MARK: - Argument validation + + /// An error naming arguments the tool doesn't declare, with close matches, or nil when all + /// are known. Tools whose schema allows extra properties are not checked. + static func unknownArgumentsError(_ name: String, _ args: [String: Value]?) -> CallTool.Result? { + guard let args, !args.isEmpty, + let tool = allTools.first(where: { $0.name == name }), + case .object(let schema) = tool.inputSchema, + schema["additionalProperties"]?.boolValue != true, + let properties = schema["properties"]?.objectValue + else { return nil } + let known = Array(properties.keys) + let unknown = args.keys.filter { properties[$0] == nil }.sorted() + guard !unknown.isEmpty else { return nil } + var lines: [String] = [] + for key in unknown { + if let close = closestArgument(key, in: known) { + lines.append("Unknown argument '\(key)'; did you mean '\(close)'?") + } else { + lines.append("Unknown argument '\(key)'.") + } + } + let valid = known.isEmpty ? "(no arguments)" : known.sorted().joined(separator: ", ") + lines.append("\(name) accepts: \(valid)") + return .fail(lines.joined(separator: "\n")) + } + + /// `derived_data_path` → `derivedDataPath` (same letters ignoring case and underscores), + /// else the nearest name by edit distance. + private static func closestArgument(_ key: String, in known: [String]) -> String? { + let folded = key.replacingOccurrences(of: "_", with: "").lowercased() + if let same = known.first(where: { $0.replacingOccurrences(of: "_", with: "").lowercased() == folded }) { + return same + } + return FuzzyMatch.fuzzyRank(needle: key, candidates: known, maxResults: 1).first?.candidate + } + // MARK: - tool_groups Tool private static let toolGroupsTool = Tool( @@ -121,7 +195,7 @@ public enum ToolRegistry { "Unknown group(s): \(unknown.joined(separator: ", ")). Valid: \(allGroups.joined(separator: ", "))" ) } - for g in toEnable { disabledGroups.remove(g) } + disabledGroupsLock.withLock { $0.subtract(toEnable) } } // Process disable (protect session-state group — it provides set_defaults and profile tools) @@ -139,13 +213,14 @@ public enum ToolRegistry { "Cannot disable protected group(s): \(blocked.joined(separator: ", ")). These provide essential session management tools." ) } - for g in toDisable { disabledGroups.insert(g) } + disabledGroupsLock.withLock { $0.formUnion(toDisable) } } // Always return current status var lines = ["Tool groups:"] + let disabled = disabledGroups for group in allGroups { - let status = disabledGroups.contains(group) ? "disabled" : "enabled" + let status = disabled.contains(group) ? "disabled" : "enabled" let toolCount = providers.filter { $0.group == group }.flatMap { $0.tools }.count lines.append(" \(group): \(status) (\(toolCount) tools)") } diff --git a/Sources/XCForgeKit/Support/ProcessRunner.swift b/Sources/XCForgeKit/Support/ProcessRunner.swift index 1d1acc9..e580bd7 100644 --- a/Sources/XCForgeKit/Support/ProcessRunner.swift +++ b/Sources/XCForgeKit/Support/ProcessRunner.swift @@ -224,14 +224,18 @@ public enum Shell { try process.run() - // Timeout watchdog + // Track the child so a cancelled call, a client disconnect or Ctrl-C can stop it + // (and everything it spawned) instead of leaving it holding DerivedData. let pid = process.processIdentifier + let tracked = !ChildProcesses.untracked + if tracked { ChildProcesses.register(pid) } + defer { if tracked { ChildProcesses.unregister(pid) } } + + // Timeout watchdog let timeoutNanos = UInt64(timeout * 1_000_000_000) let timeoutTask = Task.detached { try? await Task.sleep(nanoseconds: timeoutNanos) - kill(pid, SIGTERM) - try? await Task.sleep(nanoseconds: 2_000_000_000) - kill(pid, SIGKILL) + ProcessTree.terminate(pid) } // Idle watchdog: kill the process after `idleTimeout` seconds with no output on either @@ -248,23 +252,31 @@ public enum Shell { let silent = Date().timeIntervalSince(lastOutput.withLock { $0 }) if silent >= idle { killedForIdle.withLock { $0 = true } - kill(pid, SIGTERM) - try? await Task.sleep(nanoseconds: 2_000_000_000) - kill(pid, SIGKILL) + ProcessTree.terminate(pid) return } } } } - let markOutput: @Sendable () -> Void = { lastOutput.withLock { $0 = Date() } } - - // Read output concurrently, capping retained data to avoid OOM on large builds - async let stdoutData = stdoutPipe.fileHandleForReading.readTailAsync( - limit: outputLimit, onData: markOutput) - async let stderrData = stderrPipe.fileHandleForReading.readTailAsync( - limit: outputLimit, onData: markOutput) + // Feed the latest output line to the tool's progress reporter, when there is one. + let activity = ToolActivity.current + let markOutput: @Sendable (Data) -> Void = { chunk in + lastOutput.withLock { $0 = Date() } + activity?.note(chunk) + } - let (out, err) = await (stdoutData, stderrData) + // Read output concurrently, capping retained data to avoid OOM on large builds. + // Cancelling the calling task (MCP cancel, client gone) kills the process tree, which + // closes the pipes and ends the reads. + let (out, err) = await withTaskCancellationHandler { + async let stdoutData = stdoutPipe.fileHandleForReading.readTailAsync( + limit: outputLimit, onData: markOutput) + async let stderrData = stderrPipe.fileHandleForReading.readTailAsync( + limit: outputLimit, onData: markOutput) + return await (stdoutData, stderrData) + } onCancel: { + ProcessTree.terminate(pid) + } // Await process exit without blocking the cooperative thread pool. // Note: for-await on AsyncStream breaks early if the Task is cancelled @@ -275,8 +287,8 @@ public enum Shell { idleTask?.cancel() // If task was cancelled, the process may still be running - if process.isRunning { - kill(process.processIdentifier, SIGTERM) + if process.isRunning || Task.isCancelled { + ProcessTree.terminate(pid) return ShellResult(stdout: "", stderr: "Task cancelled", exitCode: -2) } @@ -326,7 +338,7 @@ extension FileHandle { /// Read pipe data incrementally, retaining only the last `limit` bytes. /// Prevents unbounded memory growth when subprocesses emit large output /// (e.g. xcodebuild builds that produce hundreds of MB). - func readTailAsync(limit: Int, onData: (@Sendable () -> Void)? = nil) async -> Data { + func readTailAsync(limit: Int, onData: (@Sendable (Data) -> Void)? = nil) async -> Data { precondition(limit > 0, "outputLimit must be positive") return await withCheckedContinuation { continuation in DispatchQueue.global().async { @@ -334,7 +346,7 @@ extension FileHandle { while true { let chunk = self.availableData if chunk.isEmpty { break } - onData?() + onData?(chunk) tail.append(chunk) if tail.count > limit { tail = Data(tail.suffix(limit)) @@ -358,3 +370,102 @@ extension FileHandle { } } } + +// MARK: - Child process tracking + +/// Process IDs of children xcforge started and is still waiting on. +/// +/// Used to stop them when the CLI gets Ctrl-C or SIGTERM, or when the MCP client goes away, +/// so no orphaned xcodebuild keeps running and holding DerivedData. +public enum ChildProcesses { + private static let pids = OSAllocatedUnfairLock(initialState: Set()) + + /// Set for long-lived helpers (the simulator WebDriverAgent runner) that should keep + /// running when xcforge exits. Cancellation and timeouts still stop them. + @TaskLocal public static var untracked = false + + static func register(_ pid: pid_t) { _ = pids.withLock { $0.insert(pid) } } + static func unregister(_ pid: pid_t) { _ = pids.withLock { $0.remove(pid) } } + + /// Children currently running. + public static var running: [pid_t] { pids.withLock { Array($0) } } + + /// Stop every tracked child and its descendants. Safe to call from a signal handler queue. + public static func terminateAll(grace: TimeInterval = 2) { + for pid in running { ProcessTree.terminate(pid, grace: grace) } + } +} + +/// Signals a process together with everything it spawned (compilers, test runners). +public enum ProcessTree { + /// The process and all its descendants, parents first. Capped so a runaway tree can't stall us. + static func members(of root: pid_t, cap: Int = 512) -> [pid_t] { + var result: [pid_t] = [root] + var index = 0 + while index < result.count && result.count < cap { + result += children(of: result[index]).filter { !result.contains($0) } + index += 1 + } + return Array(result.prefix(cap)) + } + + static func children(of pid: pid_t) -> [pid_t] { + #if canImport(Darwin) + let count = proc_listchildpids(pid, nil, 0) + guard count > 0 else { return [] } + var buffer = [pid_t](repeating: 0, count: Int(count) + 16) + let filled = buffer.withUnsafeMutableBytes { raw in + proc_listchildpids(pid, raw.baseAddress, Int32(raw.count)) + } + guard filled > 0 else { return [] } + return Array(buffer.prefix(Int(filled))).filter { $0 > 0 } + #else + return [] + #endif + } + + /// SIGTERM the whole tree now, then SIGKILL whatever is still alive after `grace` seconds. + /// The member list is taken once, before signalling, so children re-parented to launchd + /// when their parent exits are still reached. + public static func terminate(_ root: pid_t, grace: TimeInterval = 2) { + guard root > 0 else { return } + let targets = members(of: root) + for pid in targets.reversed() { kill(pid, SIGTERM) } + let nanos = UInt64(max(0, grace) * 1_000_000_000) + Task.detached { + try? await Task.sleep(nanoseconds: nanos) + for pid in targets.reversed() where kill(pid, 0) == 0 { kill(pid, SIGKILL) } + } + } +} + +// MARK: - Tool activity (progress) + +/// Latest output line of the tool call being served, for MCP progress notifications. +/// +/// The MCP server sets `current` for calls that asked for progress; `Shell.run` feeds it +/// every chunk of child output. Reading it never blocks the child. +public final class ToolActivity: Sendable { + @TaskLocal public static var current: ToolActivity? + + private let state = OSAllocatedUnfairLock(initialState: nil) + + public init() {} + + /// The most recent non-empty output line, trimmed to 200 characters. + public var lastLine: String? { state.withLock { $0 } } + + func note(_ chunk: Data) { + guard let text = String(data: chunk.suffix(4096), encoding: .utf8) else { return } + guard let line = Self.lastLine(in: text) else { return } + state.withLock { $0 = line } + } + + static func lastLine(in text: String) -> String? { + let line = text.split(whereSeparator: \.isNewline).reversed() + .lazy.map { $0.trimmingCharacters(in: .whitespaces) } + .first { !$0.isEmpty } + guard let line else { return nil } + return line.count > 200 ? String(line.prefix(200)) + "…" : line + } +} diff --git a/Sources/XCForgeKit/Support/WorkspaceContext.swift b/Sources/XCForgeKit/Support/WorkspaceContext.swift index 64f5ff2..6fe5351 100644 --- a/Sources/XCForgeKit/Support/WorkspaceContext.swift +++ b/Sources/XCForgeKit/Support/WorkspaceContext.swift @@ -99,7 +99,14 @@ public actor SessionState { /// There is no longer a "global persisted project" fallback — project /// identity must come from a source the user controls. public func resolveProject(_ explicit: String?) async throws -> String { + let resolved = try await resolveProjectUncached(explicit) + lastResolvedProject = resolved + return resolved + } + + private func resolveProjectUncached(_ explicit: String?) async throws -> String { if let explicit { + refreshRepoDefaults(forProject: explicit) trackUsage(value: explicit, streak: &projectStreak, stored: &project, source: &projectSource) loadRecordIfNeeded(forProject: explicit) return explicit @@ -123,6 +130,35 @@ public actor SessionState { return detected } + /// Use the `.xcforge.yaml` that governs `project`, not the one next to the server's + /// working directory. A project in another worktree or repo gets its own scheme, test + /// plan and simulator settings; values cached from the previous config are dropped. + private func refreshRepoDefaults(forProject project: String) { + let expanded = (project as NSString).expandingTildeInPath + let absolute = + expanded.hasPrefix("/") + ? expanded + : (FileManager.default.currentDirectoryPath as NSString).appendingPathComponent(expanded) + let directory = ((absolute as NSString).standardizingPath as NSString).deletingLastPathComponent + let values = RepoConfig.discover(from: directory) + guard values?.sourcePath != repoDefaults?.sourcePath else { return } + repoDefaults = values + if schemeSource == .repoConfig { scheme = nil } + if simulatorSource == .repoConfig { simulator = nil } + } + + /// Problems found in the `.xcforge.yaml` in effect (unknown keys, bad values). + public var configWarnings: [String] { repoDefaults?.warnings ?? [] } + + /// The project resolved for this session so far, if any. + public var activeProject: String? { lastResolvedProject } + + /// Whatever `resolveProject` returned last, explicit values included. + private var lastResolvedProject: String? + + /// Path of the `.xcforge.yaml` in effect, if any. + public var configPath: String? { repoDefaults?.sourcePath } + /// Resolve scheme name. Caches auto-detected result for the session. public func resolveScheme(_ explicit: String?, project: String) async throws -> String { if let explicit { @@ -346,6 +382,12 @@ public actor SessionState { lines.append( " note: one or more defaults were auto-promoted from repeated explicit use.") } + if let path = repoDefaults?.sourcePath { + lines.append(" config: \(path)") + } + for warning in configWarnings { + lines.append(" config warning: \(warning)") + } return lines.joined(separator: "\n") } diff --git a/Sources/XCForgeKit/Support/Xcodebuild.swift b/Sources/XCForgeKit/Support/Xcodebuild.swift index 02564ff..c1590c7 100644 --- a/Sources/XCForgeKit/Support/Xcodebuild.swift +++ b/Sources/XCForgeKit/Support/Xcodebuild.swift @@ -22,9 +22,25 @@ public enum Xcodebuild { idleTimeout: TimeInterval? = nil, env: Environment ) async throws -> ShellResult { - let options = XcodebuildOptions.effective(cwd: env.currentDirectoryPath()) + // Settings come from the .xcforge.yaml next to the project being built, so a build of + // another worktree doesn't pick up this checkout's DerivedData or lock. + let configDir = projectDirectory(arguments, workingDirectory: workingDirectory) + let options = XcodebuildOptions.effective(cwd: configDir ?? env.currentDirectoryPath()) let args = apply(options, to: arguments) + // Two tool calls in one server (parallel MCP calls) never run xcodebuild against the same + // DerivedData at once; the second waits its turn instead of hitting "database is locked". + var slot: XcodebuildQueue.Ticket? + if needsLock(args) { + let key = queueKey(args, workingDirectory: workingDirectory) + slot = await XcodebuildQueue.shared.enter(key: key) + } + defer { + if let slot { + Task { await XcodebuildQueue.shared.leave(slot) } + } + } + if needsLock(args) { try Preflight.check(options: options) } @@ -52,6 +68,40 @@ public enum Xcodebuild { return result } + /// Folder of the `-project` or `-workspace` argument, made absolute, or nil when neither is given. + static func projectDirectory(_ args: [String], workingDirectory: String?) -> String? { + guard let path = projectPath(args, workingDirectory: workingDirectory) else { return nil } + return (path as NSString).deletingLastPathComponent + } + + /// Absolute `-project`/`-workspace` path, or nil. + static func projectPath(_ args: [String], workingDirectory: String?) -> String? { + for flag in ["-workspace", "-project"] { + if let i = args.firstIndex(of: flag), i + 1 < args.count { + let raw = (args[i + 1] as NSString).expandingTildeInPath + let base = workingDirectory ?? FileManager.default.currentDirectoryPath + let absolute = raw.hasPrefix("/") ? raw : (base as NSString).appendingPathComponent(raw) + return (absolute as NSString).standardizingPath + } + } + return nil + } + + /// What two xcodebuild calls must not share at the same time: the DerivedData folder when + /// one is given, else the project (whose default DerivedData folder is derived from it). + static func queueKey(_ args: [String], workingDirectory: String?) -> String { + if let i = args.firstIndex(of: "-derivedDataPath"), i + 1 < args.count { + return "dd:" + ((args[i + 1] as NSString).standardizingPath) + } + if let project = projectPath(args, workingDirectory: workingDirectory) { + return "project:" + project + } + if let i = args.firstIndex(of: "-xctestrun"), i + 1 < args.count { + return "xctestrun:" + args[i + 1] + } + return "cwd:" + (workingDirectory ?? FileManager.default.currentDirectoryPath) + } + /// True for invocations that build or run tests (as opposed to `-list` or `-version`). static func isBuildInvocation(_ args: [String]) -> Bool { args.contains { actionTokens.contains($0) } @@ -151,6 +201,55 @@ public enum Xcodebuild { } } +// MARK: - In-process queue + +/// First-come, first-served turns per DerivedData folder within one xcforge process. +/// +/// The cross-process build lock is opt-in; this one is always on, because an MCP client can +/// send two build or test calls at once and both would otherwise run on the same DerivedData. +actor XcodebuildQueue { + static let shared = XcodebuildQueue() + + struct Ticket: Sendable, Hashable { + let key: String + let id: UInt64 + } + + private var nextID: UInt64 = 0 + private var holders: [String: UInt64] = [:] + private var waiters: [String: [(id: UInt64, resume: CheckedContinuation)]] = [:] + + /// Wait for this key's turn. Returns at once when nobody holds it. + func enter(key: String) async -> Ticket { + nextID += 1 + let id = nextID + if holders[key] == nil { + holders[key] = id + return Ticket(key: key, id: id) + } + await withCheckedContinuation { continuation in + waiters[key, default: []].append((id, continuation)) + } + return Ticket(key: key, id: id) + } + + /// Hand the key to the next waiter, or free it. + func leave(_ ticket: Ticket) { + guard holders[ticket.key] == ticket.id else { return } + if var queue = waiters[ticket.key], !queue.isEmpty { + let next = queue.removeFirst() + waiters[ticket.key] = queue.isEmpty ? nil : queue + holders[ticket.key] = next.id + next.resume.resume() + } else { + holders[ticket.key] = nil + } + } + + /// Number of calls waiting for a key (for tests and status). + func waiting(for key: String) -> Int { waiters[key]?.count ?? 0 } +} + // MARK: - Preflight /// Disk and memory checks before a build. Warns by default; refuses only when the diff --git a/Sources/XCForgeKit/Tools/InteractionProvider.swift b/Sources/XCForgeKit/Tools/InteractionProvider.swift index 8225c89..404206f 100644 --- a/Sources/XCForgeKit/Tools/InteractionProvider.swift +++ b/Sources/XCForgeKit/Tools/InteractionProvider.swift @@ -359,7 +359,12 @@ enum UITools { "description": .string( "Optional accessibility id. Restrict the listing to that element and its descendants." ), - ]) + ]), + "source": .object([ + "type": .string("string"), + "enum": .array([.string("auto"), .string("wda"), .string("axp")]), + "description": .string("Tree to read: auto (default), wda or axp."), + ]), ]), ]) ), diff --git a/Tests/XCForgeKitTests/AgentCallSafetyTests.swift b/Tests/XCForgeKitTests/AgentCallSafetyTests.swift new file mode 100644 index 0000000..afb7615 --- /dev/null +++ b/Tests/XCForgeKitTests/AgentCallSafetyTests.swift @@ -0,0 +1,208 @@ +import Foundation +import MCP +import Testing + +@testable import XCForgeKit + +@Suite("Agent call safety: cancellation, queueing, config scoping, argument checks", .serialized) +struct AgentCallSafetyTests { + + private func makeRepo(yaml: String?) -> String { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("xcforge-callsafety-\(UUID().uuidString)", isDirectory: true) + try! FileManager.default.createDirectory( + at: dir.appendingPathComponent(".git"), withIntermediateDirectories: true) + if let yaml { + try! yaml.write( + to: dir.appendingPathComponent(".xcforge.yaml"), atomically: true, encoding: .utf8) + } + return dir.path + } + + private func isAlive(_ pid: pid_t) -> Bool { kill(pid, 0) == 0 } + + private func waitUntil(seconds: Double, _ condition: () -> Bool) async -> Bool { + let deadline = Date().addingTimeInterval(seconds) + while Date() < deadline { + if condition() { return true } + try? await Task.sleep(nanoseconds: 50_000_000) + } + return condition() + } + + // MARK: - Cancellation stops the process tree + + @Test("cancelling a shell call kills the child and its children", .timeLimit(.minutes(1))) + func cancelKillsTree() async throws { + let before = Set(ChildProcesses.running) + // `; true` keeps sh from exec'ing sleep, so sleep is a grandchild of xcforge. + let task = Task { + try await Shell.run("/bin/sh", arguments: ["-c", "/bin/sleep 30; true"], timeout: 60) + } + + var shell: pid_t = 0 + let started = await waitUntil(seconds: 10) { + guard let pid = Set(ChildProcesses.running).subtracting(before).first else { return false } + shell = pid + return !ProcessTree.children(of: pid).isEmpty + } + #expect(started, "sh and its sleep child should be running") + let sleeper = ProcessTree.children(of: shell) + + task.cancel() + let result = try await task.value + #expect(result.exitCode == -2) + + let gone = await waitUntil(seconds: 5) { !isAlive(shell) && sleeper.allSatisfy { !isAlive($0) } } + #expect(gone, "cancel must not leave sh or sleep running") + #expect(!ChildProcesses.running.contains(shell)) + } + + @Test("a timeout kills grandchildren too", .timeLimit(.minutes(1))) + func timeoutKillsTree() async throws { + let marker = FileManager.default.temporaryDirectory + .appendingPathComponent("xcforge-tree-\(UUID().uuidString)").path + // The grandchild records its pid so we can check it after the parent is gone. + let script = "/bin/sh -c 'echo $$ > \(marker); exec /bin/sleep 30' ; true" + let result = try await Shell.run("/bin/sh", arguments: ["-c", script], timeout: 1) + #expect(result.exitCode == -1) + let pid = (try? String(contentsOfFile: marker, encoding: .utf8)) + .flatMap { pid_t($0.trimmingCharacters(in: .whitespacesAndNewlines)) } + #expect(pid != nil) + if let pid { + let gone = await waitUntil(seconds: 5) { !isAlive(pid) } + #expect(gone, "the sleeping grandchild must be killed with its parent") + } + try? FileManager.default.removeItem(atPath: marker) + } + + @Test("tool activity keeps the last non-empty output line") + func activityLastLine() async throws { + #expect(ToolActivity.lastLine(in: "CompileSwift a.swift\nLd App\n\n") == "Ld App") + #expect(ToolActivity.lastLine(in: "\n \n") == nil) + let long = String(repeating: "x", count: 300) + #expect(ToolActivity.lastLine(in: long)?.count == 201) + + let activity = ToolActivity() + _ = try await ToolActivity.$current.withValue(activity) { + try await Shell.run("/bin/sh", arguments: ["-c", "echo first; echo second"], timeout: 10) + } + #expect(activity.lastLine == "second") + } + + // MARK: - In-process xcodebuild queue + + @Test("a second xcodebuild on the same DerivedData waits for the first", .timeLimit(.minutes(1))) + func queueSerializesSameKey() async { + let queue = XcodebuildQueue() + let first = await queue.enter(key: "dd:/a") + let second = Task { await queue.enter(key: "dd:/a") } + let other = await queue.enter(key: "dd:/b") // a different folder never waits + #expect(other.key == "dd:/b") + + let queued = await waitUntilAsync { await queue.waiting(for: "dd:/a") == 1 } + #expect(queued) + await queue.leave(first) + let ticket = await second.value + #expect(ticket.key == "dd:/a") + await queue.leave(ticket) + await queue.leave(other) + #expect(await queue.waiting(for: "dd:/a") == 0) + } + + private func waitUntilAsync(_ condition: () async -> Bool) async -> Bool { + for _ in 0..<100 { + if await condition() { return true } + try? await Task.sleep(nanoseconds: 20_000_000) + } + return await condition() + } + + @Test("queue key prefers DerivedData, then the absolute project path") + func queueKeys() { + #expect(Xcodebuild.queueKey(["-derivedDataPath", "/dd/../dd2", "build"], workingDirectory: nil) == "dd:/dd2") + #expect( + Xcodebuild.queueKey(["-project", "App.xcodeproj", "build"], workingDirectory: "/repo") + == "project:/repo/App.xcodeproj") + #expect(Xcodebuild.projectDirectory(["-workspace", "/w/App.xcworkspace"], workingDirectory: nil) == "/w") + #expect(Xcodebuild.projectDirectory(["build"], workingDirectory: nil) == nil) + } + + // MARK: - .xcforge.yaml parsing and scoping + + @Test("yaml values lose quotes and trailing comments") + func yamlCleanValue() { + #expect(RepoConfig.cleanValue(" \"My App\" ") == "My App") + #expect(RepoConfig.cleanValue("'My App' # dev") == "My App") + #expect(RepoConfig.cleanValue("iPhone 16 # dev sim") == "iPhone 16") + #expect(RepoConfig.cleanValue("a#b") == "a#b") + #expect(RepoConfig.cleanValue("Plain") == "Plain") + } + + @Test("yaml problems are collected as warnings with suggestions") + func yamlWarnings() { + let root = makeRepo(yaml: "scheme: \"App\" # main\ntestplan: quick\nminFreeGB: lots\n") + defer { try? FileManager.default.removeItem(atPath: root) } + let values = RepoConfig.discover(from: root) + #expect(values?.scheme == "App") + #expect(values?.minFreeGB == nil) + let warnings = values?.warnings ?? [] + #expect(warnings.contains { $0.contains("'testplan'") && $0.contains("did you mean 'testPlan'") }) + #expect(warnings.contains { $0.contains("minFreeGB") }) + #expect(values?.sourcePath?.hasSuffix(".xcforge.yaml") == true) + } + + @Test("a yaml with only bad keys still reports them") + func yamlOnlyBadKeys() { + let root = makeRepo(yaml: "shceme: App\n") + defer { try? FileManager.default.removeItem(atPath: root) } + let values = RepoConfig.discover(from: root) + #expect(values?.scheme == nil) + #expect(values?.warnings.first?.contains("did you mean 'scheme'") == true) + } + + @Test("an explicit project in another repo uses that repo's .xcforge.yaml") + func explicitProjectUsesItsOwnConfig() async throws { + let repoA = makeRepo(yaml: "scheme: SchemeA\ntestPlan: planA\n") + let repoB = makeRepo(yaml: "scheme: SchemeB\ntestPlan: planB\n") + defer { + try? FileManager.default.removeItem(atPath: repoA) + try? FileManager.default.removeItem(atPath: repoB) + } + let store = DefaultsStore( + baseDirectory: URL(fileURLWithPath: repoA).appendingPathComponent("store", isDirectory: true)) + let session = SessionState(defaultsStore: store, cwd: repoA) + #expect(await session.resolveTestPlan(nil) == "planA") + + let projectB = (repoB as NSString).appendingPathComponent("B.xcodeproj") + _ = try await session.resolveProject(projectB) + #expect(try await session.resolveScheme(nil, project: projectB) == "SchemeB") + #expect(await session.resolveTestPlan(nil) == "planB") + #expect(await session.activeProject == projectB) + #expect(await session.showDefaults().contains("config: ")) + } + + // MARK: - Unknown MCP arguments + + @Test("unknown arguments are rejected with the closest real name") + func unknownArguments() { + let snake = ToolRegistry.unknownArgumentsError("build_sim", ["derived_data_path": .string("/dd")]) + #expect(snake?.isError == true) + #expect(text(snake).contains("did you mean 'derivedDataPath'")) + + let typo = ToolRegistry.unknownArgumentsError("build_sim", ["shceme": .string("A")]) + #expect(text(typo).contains("did you mean 'scheme'")) + + #expect(ToolRegistry.unknownArgumentsError("build_sim", ["scheme": .string("A")]) == nil) + #expect(ToolRegistry.unknownArgumentsError("build_sim", nil) == nil) + #expect(ToolRegistry.unknownArgumentsError("list_elements", ["source": .string("wda")]) == nil) + } + + private func text(_ result: CallTool.Result?) -> String { + guard let result else { return "" } + return result.content.compactMap { + if case .text(let text, _, _) = $0 { return text } + return nil + }.joined(separator: "\n") + } +} From 9abc0a238a561ae032eb658e4789ae3ef7a62eb9 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 16:03:15 +0000 Subject: [PATCH 07/28] feat: test results agents can act on (bundle 8) - One test ID format (Target/Suite/test()) in results, list_tests, last-failures and known-failures; filters accept it as printed, and short IDs get the target from the scheme or test plan - Every failure message with file and line, argument/repetition label, attachments and optional console tail; identical messages grouped, text output capped with a count of the rest - Run options on test_sim/build_and_test and the CLI: retries, iterations, until-failure, parallel, per-test timeout, skip build - Flaky tests (failed then passed on retry) reported - rerun-failed reuses recorded run settings, skips the build when no source changed, and refuses when the last run failed before any test - Test failures are no longer reported as a test target build failure; agent JSON gives a reason when no test reported Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EtxtFPs6UBAWXqEpkj2W2N --- CHANGELOG.md | 7 + Skills/xcforge/references/test-tools.md | 43 +- .../Commands/Build/BuildTestCommand.swift | 41 +- .../Commands/Test/TestCommand.swift | 9 +- .../Commands/Test/TestRenderer.swift | 58 +- .../Test/TestRerunFailedCommand.swift | 47 +- .../Commands/Test/TestRunOptionGroup.swift | 55 + .../Persistence/LastFailuresStore.swift | 35 +- .../Persistence/LastResultStore.swift | 35 + .../Support/AgentResultProjection.swift | 87 +- .../XCForgeKit/Support/ProjectResolver.swift | 115 +- Sources/XCForgeKit/Support/TestIDs.swift | 199 ++ .../XCForgeKit/Tools/TestPlanInspector.swift | 33 +- Sources/XCForgeKit/Tools/TestProvider.swift | 1711 ++++++++--------- .../AgentResultProjectionTests.swift | 4 +- Tests/XCForgeKitTests/TestResultsTests.swift | 350 ++++ 16 files changed, 1854 insertions(+), 975 deletions(-) create mode 100644 Sources/XCForgeCLI/Commands/Test/TestRunOptionGroup.swift create mode 100644 Sources/XCForgeKit/Support/TestIDs.swift create mode 100644 Tests/XCForgeKitTests/TestResultsTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index b45cffb..68150bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,8 +22,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - Each build and test tool result ends with `project: `, so a build of the wrong checkout or worktree is visible - `.xcforge.yaml` problems (unknown keys with a "did you mean", unparseable values) are listed by `set_defaults show` / `defaults show` and once at the end of the first tool result that uses the file - **Xcode 27 Device Hub support.** xcforge opens Device Hub when the selected Xcode has no Simulator.app, and screen capture, the accessibility bridge and window scripting accept either app +- **Test run options:** `retries`, `iterations`, `untilFailure`, `parallel`, `testTimeoutSeconds`, `skipBuild`, `includeConsole`, `env` and `timeoutSeconds` on `test_sim` and `build_and_test`, with matching flags (`--retries`, `--iterations`, `--until-failure`, `--parallel/--no-parallel`, `--test-timeout`, `--no-build`, `--include-console`, `--env`, `--timeout-seconds`) on `test run`, `build-test` and `test rerun-failed` +- Tests that fail and then pass on retry are listed as flaky (`flaky` in agent JSON) +- `test rerun-failed` reuses the recorded project, test plan, configuration and env, and skips build-for-testing when no source file changed since the last one (`--build` forces it) ### Changed +- **One test ID format.** Results, `list_tests`, last-failures and known-failures all use `Target/Suite/test()` (Swift Testing) or `Target/Class/testMethod` (XCTest), and filters accept it as printed. The target is added to short IDs from the scheme's or test plan's test targets instead of guessed, and the `test()()` spelling xcodebuild needs is written for you +- Failures carry every message with file and line, the argument or repetition it came from, and their attachments; identical messages are grouped, and text output is capped at 20 failures with a count of the rest +- A test failure is no longer reported as "test target build failed". When the build or runner fails before any test reports, agent JSON says why (`reason`) and `rerun-failed` refuses instead of running nothing +- Agent JSON `failed` counts every failure, not only the ones listed - **Cancelling stops the build.** An MCP cancel, a client disconnect, Ctrl-C, SIGTERM or SIGHUP now stops the xcodebuild (or other command) xcforge started, together with every process it spawned. Timeouts kill the whole process tree too, so compiler and test-runner children no longer outlive the call and hold DerivedData ("database is locked" on the next run) - Two MCP calls that would run xcodebuild on the same DerivedData folder (or the same project's default one) at once now run one after the other, first come first served - An explicit `project` in another repo or worktree uses the `.xcforge.yaml` next to it, for both session defaults (scheme, simulator, test plan) and xcodebuild options (DerivedData, lock, idle timeout) diff --git a/Skills/xcforge/references/test-tools.md b/Skills/xcforge/references/test-tools.md index f1e47e2..c73b1a4 100644 --- a/Skills/xcforge/references/test-tools.md +++ b/Skills/xcforge/references/test-tools.md @@ -31,19 +31,45 @@ Run tests and return structured xcresult summary. | `for` | No | `human` | Output audience: `human` preserves full JSON; `agent` returns a slim ≤10-field projection | | `gate` | No | false | Subtract IDs in `.xcforge/known-failures.yaml` when computing `succeeded`. Raw failure list unchanged | -**Filter auto-resolution:** The test target prefix is auto-resolved when omitted. All these formats work: -- `testMethodName` → auto-prefixes `TestTarget/testMethodName` -- `TestClass/testMethodName` → auto-prefixes `TestTarget/TestClass/testMethodName` -- `TestTarget/TestClass/testMethodName` → passed through as-is +**Test IDs:** one format everywhere: `Target/Suite/test()` for Swift Testing, +`Target/Class/testMethod` for XCTest. `list_tests`, failure lists, `.xcforge/last-failures.json` +and `known-failures.yaml` all use it, and `filter` accepts it as is. Shorter forms work too: +- `Suite/test()` or `Suite` → the target is added when the scheme or test plan has one test target + (with several, pass the target) +- `Target/Suite/test()` → passed through. xcforge writes the `test()()` spelling xcodebuild needs + for Swift Testing; pass the plain form **Filter typo suggestions:** If the filter doesn't match any test ID, xcforge runs a Levenshtein fuzzy match and surfaces "Did you mean: ...?" suggestions in the result. Use `list_tests` to discover available identifiers if unsure. +**Run options** (`test_sim` and `build_and_test`; CLI flags on `test run`, `build-test`, `test rerun-failed`): + +| Parameter | CLI | Description | +|-----------|-----|-------------| +| `timeoutSeconds` | `--timeout-seconds` | Total limit for the run | +| `env` | `--env KEY=VALUE` | Environment for the tests. Keys are prefixed with `TEST_RUNNER_` (Xcode strips it inside tests) | +| `skipBuild` | `--no-build` | Test the last build-for-testing without building. `rerun-failed` does this on its own when no file changed since that build; `--build` forces a build | +| `retries` | `--retries N` | Retry each failing test up to N times. A test that fails then passes is reported under `flaky` | +| `iterations` | `--iterations N` | Run each test N times | +| `untilFailure` | `--until-failure` | With `iterations`, stop at the first failure | +| `parallel` | `--parallel` / `--no-parallel` | Turn parallel testing on or off (default: the scheme's setting) | +| `testTimeoutSeconds` | `--test-timeout` | Per-test time allowance; a test running longer fails | +| `includeConsole` | `--include-console` | Add each failing test's last 40 console lines to its failure | + +`retries` can't be combined with `iterations` or `untilFailure`. + +**Failure output:** each failure lists every message with `file:line` and, for parameterized or +repeated tests, the argument or repetition it came from. Failures with the same message are grouped +("N tests, same message"). Text output shows 20 failures and says how many more there are; +`test_failures` lists them all. When the build or the test runner fails before any test reports, +the result says so (`reason` in agent JSON) and `rerun-failed` refuses instead of rerunning nothing. + **Returns:** - Total/passed/failed/skipped counts - Duration -- Per-failure: test name, error message, file:line +- Per-failure: full test ID, each message with file:line, attachments +- Flaky tests (failed, then passed on retry) - Failure screenshot paths (auto-exported from xcresult) - xcresult path (reusable with `test_failures` and `test_coverage`) - Device info (name, OS version) @@ -110,7 +136,7 @@ Build and return structured errors/warnings from xcresult. Unlike `build_sim`, t ## build_and_test -Build then test in one call. Short-circuits on build failure with structured diagnostics. **Preferred over separate `build_and_diagnose` + `test_sim` calls.** +Build then test in one call. Stops on a build failure with structured diagnostics. Takes the same run options as `test_sim`. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| @@ -121,7 +147,6 @@ Build then test in one call. Short-circuits on build failure with structured dia | `testplan` | No | — | Test plan name | | `filter` | No | — | Test filter — accepts relaxed formats (auto-resolves target prefix) | | `coverage` | No | false | Enable code coverage collection | -| `env` | No | — | Array of `KEY=VALUE` strings. Each key is auto-prefixed with `TEST_RUNNER_` before xcodebuild runs (Xcode strips the prefix inside tests). `TEST_RUNNER_XCFORGE_REPO_ROOT` is always injected. | | `for` | No | `human` | Output audience: `human` preserves full JSON; `agent` returns a slim ≤10-field projection | | `gate` | No | false | Subtract IDs in `.xcforge/known-failures.yaml` when computing `succeeded`. Raw failure list unchanged | @@ -132,7 +157,7 @@ Build then test in one call. Short-circuits on build failure with structured dia **Returns:** Phase indicator (`build` or `test`), build elapsed time, build diagnostics (on failure), test execution result (on success). -**Failure persistence:** On a failing run, failure IDs are saved to `.xcforge/last-failures.json` (cleared on green). Use `xcforge test rerun-failed` to replay exactly those IDs. +**Failure persistence:** On a failing run, failure IDs and the run settings (project, test plan, configuration, env) are saved to `.xcforge/last-failures.json` (cleared on green). `xcforge test rerun-failed` replays exactly those IDs with the same settings, and skips the build when nothing changed. --- @@ -146,7 +171,7 @@ List available test identifiers for a scheme. Use to discover the correct filter | `scheme` | No | Auto-detect | Scheme name | | `simulator` | No | Auto-detect (booted) | Simulator name or UDID | -**Returns:** List of test identifiers in `Target/Class/method` format, grouped by target and class. Includes counts of targets, classes, and test methods. +**Returns:** List of test identifiers in `Target/Suite/test()` / `Target/Class/method` format, grouped by target and class. Includes counts of targets, classes, and test methods. **Note:** Requires a build-for-testing step (does not run tests). First call may take time to build. diff --git a/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift b/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift index 3b3b23f..8515a08 100644 --- a/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift +++ b/Sources/XCForgeCLI/Commands/Build/BuildTestCommand.swift @@ -26,7 +26,7 @@ struct BuildTest: AsyncParsableCommand { @Option( help: - "Test filter. Accepts: 'testMethod', 'TestClass/testMethod', or 'TestTarget/TestClass/testMethod'. Target prefix is auto-resolved." + "Tests to run, comma-separated: 'Target/Suite/test()', 'Suite/test()' or 'Suite'. The target is added when missing." ) var filter: String? @@ -36,24 +36,9 @@ struct BuildTest: AsyncParsableCommand { @Flag(help: "Raise the total time limit from 1800s to 7200s. Hangs are caught by --idle-timeout either way.") var long = false - @Option( - help: - "Override timeout in seconds. Takes precedence over --long. Default: 1800s (or 7200s with --long)." - ) - var timeoutSeconds: Int? - @Flag(help: "Capture a diagnostic snapshot even when the build/test succeeds.") var diagnose = false - @Option( - name: .long, parsing: .singleValue, - help: ArgumentHelp( - "Environment variable for the test runner (repeatable). Format: KEY=VALUE. The key is auto-prefixed with TEST_RUNNER_; Xcode strips the prefix inside the test process, so 'BLESS_BASELINE=1' surfaces as ProcessInfo.environment[\"BLESS_BASELINE\"]. TEST_RUNNER_XCFORGE_REPO_ROOT is always injected (override with --env XCFORGE_REPO_ROOT=...).", - valueName: "KEY=VALUE" - ) - ) - var env: [String] = [] - @Option( help: "Simulator recovery: off (default), auto (reboot if not Booted), erase (also erase if a reboot didn't help)." @@ -78,6 +63,8 @@ struct BuildTest: AsyncParsableCommand { ) var gate = false + @OptionGroup var testRun: TestRunOptionGroup + @OptionGroup var xcodebuild: XcodebuildOptionGroup mutating func run() async throws { @@ -90,7 +77,6 @@ struct BuildTest: AsyncParsableCommand { let session = Environment.live.session let configuration = await session.resolveConfiguration(self.configuration) let resolvedTestplan = await session.resolveTestPlan(testplan) - let resolvedTimeout = timeoutSeconds.map { TimeInterval($0) } let result = try await TestTools.executeBuildAndTest( project: project, @@ -103,11 +89,14 @@ struct BuildTest: AsyncParsableCommand { long: long, diagnose: diagnose, simRecovery: try SimRecoveryMode.parse(simRecovery), - timeoutSeconds: resolvedTimeout, - envEntries: env, + timeoutSeconds: testRun.timeout, + envEntries: testRun.env, gate: gate, forMode: forMode, - isolatedSimulator: isolatedSim + isolatedSimulator: isolatedSim, + skipBuild: testRun.build == false, + testOptions: testRun.testOptions, + includeConsole: testRun.includeConsole ) if useJSON { @@ -158,7 +147,7 @@ enum BuildTestRenderer { return lines.joined(separator: "\n") } - lines.append("Build OK (\(result.buildElapsed)s)") + lines.append(result.skippedBuild ? "Build skipped (tested the last build)" : "Build OK (\(result.buildElapsed)s)") if let test = result.testResult { let icon = test.succeeded ? "PASSED" : "FAILED" @@ -170,12 +159,10 @@ enum BuildTestRenderer { if !test.failures.isEmpty { lines.append("") lines.append("Failures:") - for failure in test.failures { - lines.append(" \(failure.testName): \(failure.message)") - if !failure.source.isEmpty && failure.source != "stderr" { - lines.append(" at \(failure.source)") - } - } + lines += TestFailureText.lines(test.failures) + } + if !test.flakyTests.isEmpty { + lines.append("Flaky (failed, then passed on retry): \(test.flakyTests.joined(separator: ", "))") } if !test.screenshotPaths.isEmpty { diff --git a/Sources/XCForgeCLI/Commands/Test/TestCommand.swift b/Sources/XCForgeCLI/Commands/Test/TestCommand.swift index 635e188..98a82f1 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestCommand.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestCommand.swift @@ -70,7 +70,7 @@ struct TestRun: AsyncParsableCommand { @Option(help: "Test plan name.") var testplan: String? - @Option(help: "Test filter, e.g. 'MyTests/testFoo' or 'MyTests'.") + @Option(help: "Tests to run, comma-separated: 'Target/Suite/test()', 'Suite/test()' or 'Suite'.") var filter: String? @Flag(help: "Enable code coverage collection.") @@ -106,6 +106,8 @@ struct TestRun: AsyncParsableCommand { ) var gate = false + @OptionGroup var testRun: TestRunOptionGroup + @OptionGroup var xcodebuild: XcodebuildOptionGroup mutating func run() async throws { @@ -131,9 +133,14 @@ struct TestRun: AsyncParsableCommand { long: long, diagnose: diagnose, simRecovery: recoveryMode, + timeoutSeconds: testRun.timeout, + envEntries: testRun.env, gate: gate, forMode: forMode, isolatedSimulator: isolatedSim, + skipBuild: testRun.build == false, + testOptions: testRun.testOptions, + includeConsole: testRun.includeConsole, env: env ) diff --git a/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift b/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift index b2b8f46..b7739f9 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift @@ -19,12 +19,9 @@ enum TestRenderer { } lines.append(statParts.joined(separator: ", ")) - // Inline failure summaries - for failure in execution.failures.prefix(20) { - lines.append("FAIL: \(failure.testName)") - if !failure.message.isEmpty { - lines.append(" \(failure.message)") - } + lines += TestFailureText.lines(execution.failures, indent: "") + if !execution.flakyTests.isEmpty { + lines.append("Flaky (failed, then passed on retry): \(execution.flakyTests.joined(separator: ", "))") } // Device info @@ -57,42 +54,18 @@ enum TestRenderer { var lines: [String] = [] lines.append("\(result.failures.count) test failure(s):") lines.append("") + lines += TestFailureText.lines(result.failures, indent: "") - for failure in result.failures { - lines.append("FAIL: \(failure.testName) [\(failure.testIdentifier)]") - if !failure.message.isEmpty { - lines.append(" " + failure.message.replacingOccurrences(of: "\n", with: "\n ")) - } - - // Matching screenshots - let matchingScreenshots = result.screenshots.filter { - $0.testName.contains(failure.testIdentifier) || failure.testIdentifier.contains($0.testName) - } - for screenshot in matchingScreenshots { - lines.append(" Screenshot: \(screenshot.path)") - } - - // Console output - let funcName = - failure.testIdentifier.split(separator: "/").last.map(String.init) ?? failure.testIdentifier - if let console = result.consoleByTest[funcName] - ?? result.consoleByTest[failure.testIdentifier] - { - lines.append(" Console:") - lines.append(" " + console.replacingOccurrences(of: "\n", with: "\n ")) - } - + // Attachments that couldn't be tied to one failure + let attached = Set(result.failures.flatMap { $0.attachments ?? [] }) + let loose = result.screenshots.map(\.path).filter { !attached.contains($0) } + if !loose.isEmpty { lines.append("") + lines.append("Other failure attachments (\(loose.count)):") + lines += loose.prefix(20).map { " \($0)" } } - // All screenshots at the end - if !result.screenshots.isEmpty { - lines.append("Failure screenshots (\(result.screenshots.count)):") - for att in result.screenshots { - lines.append(" \(att.path)") - } - } - + lines.append("") lines.append("xcresult: \(result.xcresultPath)") return lines.joined(separator: "\n") } @@ -128,7 +101,8 @@ enum TestRenderer { // Group by target/class var grouped: [String: [String: [String]]] = [:] for test in result.tests { - grouped[test.target, default: [:]][test.className, default: []].append(test.methodName) + let method = TestIDs.components(test.fullIdentifier).last ?? test.methodName + grouped[test.target, default: [:]][test.className, default: []].append(method) } for (target, classes) in grouped.sorted(by: { $0.key < $1.key }) { @@ -143,9 +117,9 @@ enum TestRenderer { lines.append("") lines.append("Filter examples:") - lines.append(" xcforge test --filter \"Target/Class/method\"") - lines.append(" xcforge test --filter \"Class/method\" (auto-resolves target)") - lines.append(" xcforge test --filter \"Class\" (all tests in class)") + lines.append(" xcforge test --filter \"Target/Suite/test()\"") + lines.append(" xcforge test --filter \"Suite/test()\" (the target is added)") + lines.append(" xcforge test --filter \"Suite\" (every test in the suite)") return lines.joined(separator: "\n") } diff --git a/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift b/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift index 00d0031..128a532 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift @@ -42,6 +42,8 @@ struct TestRerunFailed: AsyncParsableCommand { @Flag(help: "Apply known-failures gate to the rerun result.") var gate = false + @OptionGroup var testRun: TestRunOptionGroup + @OptionGroup var xcodebuild: XcodebuildOptionGroup mutating func run() async throws { @@ -50,27 +52,51 @@ struct TestRerunFailed: AsyncParsableCommand { } func execute() async throws { + let env = Environment.live + let session = env.session + // Failures are recorded in the repo of the project that was tested. let cwd = FileManager.default.currentDirectoryPath - let repoRoot = RepoRoot.discover(from: cwd) ?? cwd + var repoRoot = RepoRoot.discover(from: cwd) ?? cwd + if let project, let resolved = try? await session.resolveProject(project) { + repoRoot = RepoRoot.discover(from: (resolved as NSString).deletingLastPathComponent) ?? repoRoot + } guard let payload = LastFailuresStore.read(at: repoRoot) else { fputs("no prior failures recorded — run `test run` first\n", stderr) throw ExitCode(2) } + if let reason = payload.infraFailure { + fputs( + "the last run failed before any test reported a result (\(reason)); there are no failures to rerun." + + " Fix that and run the tests again.\n", stderr) + throw ExitCode(2) + } if payload.failures.isEmpty { print("no failures to rerun") return } let useJSON = shouldOutputJSON(flag: json) || forMode == .agent - let session = Environment.live.session - let configuration = await session.resolveConfiguration(self.configuration) - let resolvedTestplan = await session.resolveTestPlan(testplan) + let recorded = payload.run + let resolvedProject = project ?? recorded?.project + let configuration = await session.resolveConfiguration(self.configuration ?? recorded?.configuration) + let resolvedTestplan = await session.resolveTestPlan(testplan ?? recorded?.testPlan) + let envEntries = testRun.env.isEmpty ? (recorded?.env ?? []) : testRun.env + + // Without --build/--no-build, skip the build when nothing changed since the last one. + let skipBuild: Bool + if let build = testRun.build { + skipBuild = !build + } else { + skipBuild = await TestTools.lastTestBuildIsCurrent( + project: resolvedProject, scheme: scheme ?? payload.scheme, simulator: simulator ?? payload.simulator, + configuration: configuration, env: env) + } // Pass failure IDs as a pre-split list so IDs containing commas (e.g. // parameterized Swift Testing arguments) survive the trip to xcodebuild // intact — no comma-join + re-split round-trip. let execution = try await TestTools.executeTest( - project: project, + project: resolvedProject, scheme: scheme ?? payload.scheme, simulator: simulator ?? payload.simulator, configuration: configuration, @@ -81,13 +107,22 @@ struct TestRerunFailed: AsyncParsableCommand { long: long, diagnose: diagnose, simRecovery: .off, + timeoutSeconds: testRun.timeout, + envEntries: envEntries, gate: gate, - forMode: forMode + forMode: forMode, + skipBuild: skipBuild, + testOptions: testRun.testOptions, + includeConsole: testRun.includeConsole, + env: env ) if useJSON { print(try WorkflowJSONRenderer.renderTestJSON(execution, forAgent: forMode == .agent)) } else { + if skipBuild && testRun.build == nil { + print("Build skipped: no source changed since the last build (pass --build to force one).") + } print(TestRenderer.renderTest(execution)) } diff --git a/Sources/XCForgeCLI/Commands/Test/TestRunOptionGroup.swift b/Sources/XCForgeCLI/Commands/Test/TestRunOptionGroup.swift new file mode 100644 index 0000000..0c38899 --- /dev/null +++ b/Sources/XCForgeCLI/Commands/Test/TestRunOptionGroup.swift @@ -0,0 +1,55 @@ +import ArgumentParser +import Foundation +import XCForgeKit + +/// Options shared by `test run`, `test rerun-failed` and `build-test`, so a rerun can be +/// invoked exactly like the run it repeats. +struct TestRunOptionGroup: ParsableArguments { + @Option( + help: + "Total time limit in seconds for each xcodebuild step. Takes precedence over --long. Default: 1800s (7200s with --long)." + ) + var timeoutSeconds: Int? + + @Option( + name: .long, parsing: .singleValue, + help: ArgumentHelp( + "Environment variable for the test runner (repeatable). Format: KEY=VALUE. The key is auto-prefixed with TEST_RUNNER_; Xcode strips the prefix inside the test process, so 'BLESS_BASELINE=1' surfaces as ProcessInfo.environment[\"BLESS_BASELINE\"]. TEST_RUNNER_XCFORGE_REPO_ROOT is always injected (override with --env XCFORGE_REPO_ROOT=...).", + valueName: "KEY=VALUE" + ) + ) + var env: [String] = [] + + @Flag( + inversion: .prefixedNo, + help: + "Build for testing first. --no-build tests the last build's products. Default: build (rerun-failed: skip when no source changed since the last build)." + ) + var build: Bool? + + @Option(help: "Rerun a failing test up to this many more times; tests that then pass are reported as flaky.") + var retries: Int? + + @Option(help: "Run every test this many times.") + var iterations: Int? + + @Flag(help: "Repeat the tests until one fails (capped by --iterations when given).") + var untilFailure = false + + @Flag(inversion: .prefixedNo, help: "Turn parallel testing on or off. Default: the scheme's or test plan's setting.") + var parallel: Bool? + + @Option(help: "Time allowance per test in seconds (XCTest; Swift Testing uses .timeLimit in code).") + var testTimeout: Int? + + @Flag(help: "Attach the last lines each failing test printed to its failure.") + var includeConsole = false + + var testOptions: TestTools.TestRunOptions { + TestTools.TestRunOptions( + retries: retries, iterations: iterations, untilFailure: untilFailure, parallel: parallel, + testTimeoutSeconds: testTimeout) + } + + var timeout: TimeInterval? { timeoutSeconds.map { TimeInterval($0) } } +} diff --git a/Sources/XCForgeKit/Persistence/LastFailuresStore.swift b/Sources/XCForgeKit/Persistence/LastFailuresStore.swift index 83e53d8..b104094 100644 --- a/Sources/XCForgeKit/Persistence/LastFailuresStore.swift +++ b/Sources/XCForgeKit/Persistence/LastFailuresStore.swift @@ -9,17 +9,45 @@ public enum LastFailuresStore { public static let fileName = "last-failures.json" public static let directoryName = ".xcforge" + /// How the recorded run was invoked, so a rerun reproduces it instead of failing or + /// passing for a different reason. + public struct RunSettings: Codable, Sendable, Equatable { + public var project: String? + public var testPlan: String? + public var configuration: String? + /// The `KEY=VALUE` test-runner environment entries, as given. + public var env: [String]? + + public init( + project: String? = nil, testPlan: String? = nil, configuration: String? = nil, env: [String]? = nil + ) { + self.project = project + self.testPlan = testPlan + self.configuration = configuration + self.env = env + } + } + public struct Payload: Codable, Sendable, Equatable { public let failures: [String] public let scheme: String? public let simulator: String? public let recordedAt: Date + public let run: RunSettings? + /// Set when the run failed before any test reported (build error, runner crash). + /// `failures` is empty then, and a rerun has nothing it can safely replay. + public let infraFailure: String? - public init(failures: [String], scheme: String?, simulator: String?, recordedAt: Date = Date()) { + public init( + failures: [String], scheme: String?, simulator: String?, recordedAt: Date = Date(), + run: RunSettings? = nil, infraFailure: String? = nil + ) { self.failures = failures self.scheme = scheme self.simulator = simulator self.recordedAt = recordedAt + self.run = run + self.infraFailure = infraFailure } } @@ -36,6 +64,8 @@ public enum LastFailuresStore { failures: [String], scheme: String?, simulator: String?, + run: RunSettings? = nil, + infraFailure: String? = nil, at repoRoot: String, now: Date = Date() ) -> Bool { @@ -50,7 +80,8 @@ public enum LastFailuresStore { let target = URL(fileURLWithPath: path(at: repoRoot)) let tmp = URL(fileURLWithPath: target.path + ".tmp") let payload = Payload( - failures: failures, scheme: scheme, simulator: simulator, recordedAt: now) + failures: failures, scheme: scheme, simulator: simulator, recordedAt: now, run: run, + infraFailure: infraFailure) let encoder = JSONEncoder() encoder.outputFormatting = [.prettyPrinted, .sortedKeys] diff --git a/Sources/XCForgeKit/Persistence/LastResultStore.swift b/Sources/XCForgeKit/Persistence/LastResultStore.swift index 833cff4..926413f 100644 --- a/Sources/XCForgeKit/Persistence/LastResultStore.swift +++ b/Sources/XCForgeKit/Persistence/LastResultStore.swift @@ -16,6 +16,9 @@ public enum LastResultStore { public var build: String? public var test: String? public var updatedAt: Date + /// When build-for-testing last succeeded, and for which scheme, configuration and platform. + public var testBuiltAt: Date? + public var testBuildKey: String? } static func directory() -> String { @@ -50,6 +53,38 @@ public enum LastResultStore { case .test: record.test = bundlePath } record.updatedAt = now + save(record, to: path) + } + + /// What a test build depends on besides the sources. + static func testBuildKey(scheme: String, configuration: String, coverage: Bool, physicalDevice: Bool) + -> String + { + [scheme, configuration, coverage ? "coverage" : "", physicalDevice ? "device" : "simulator"] + .joined(separator: "|") + } + + /// Note that build-for-testing just succeeded for `project` with `key`. + static func recordTestBuild(project: String, key: String, now: Date = Date()) { + let path = filePath(for: project) + try? FileManager.default.createDirectory( + atPath: directory(), withIntermediateDirectories: true, attributes: nil) + var record = load(path) ?? Record(build: nil, test: nil, updatedAt: now) + record.testBuiltAt = now + record.testBuildKey = key + save(record, to: path) + } + + /// True when the last successful build-for-testing for `project` used `key` and nothing + /// under `sourceRoot` changed after it, so its products can be tested as they are. + static func testBuildIsCurrent(project: String, key: String, sourceRoot: String) -> Bool { + guard let record = load(filePath(for: project)), record.testBuildKey == key, + let builtAt = record.testBuiltAt + else { return false } + return !SourceChanges.anyModified(under: sourceRoot, after: builtAt) + } + + private static func save(_ record: Record, to path: String) { let encoder = JSONEncoder() encoder.outputFormatting = [.prettyPrinted, .sortedKeys] encoder.dateEncodingStrategy = .iso8601 diff --git a/Sources/XCForgeKit/Support/AgentResultProjection.swift b/Sources/XCForgeKit/Support/AgentResultProjection.swift index 4198321..b9e5e9d 100644 --- a/Sources/XCForgeKit/Support/AgentResultProjection.swift +++ b/Sources/XCForgeKit/Support/AgentResultProjection.swift @@ -1,9 +1,9 @@ import Foundation -/// Slim 10-field projection of a test run designed for agent consumption. -/// Encoded JSON omits screenshots, slowest-test arrays, build diagnostics, -/// xcresult paths, and other context-burning fields. `knownFailures` is -/// omitted when nil or empty to keep the wire shape tight. +/// Slim projection of a test run designed for agent consumption. +/// Encoded JSON omits slowest-test arrays, build diagnostics, xcresult paths, +/// and other context-burning fields. `knownFailures`, `flaky` and `reason` are +/// omitted when empty to keep the wire shape tight. public struct AgentTestResult: Codable, Sendable, Equatable { public let succeeded: Bool public let buildOk: Bool @@ -14,14 +14,36 @@ public struct AgentTestResult: Codable, Sendable, Equatable { public let timedOut: Bool public let knownFailures: [String]? public let failures: [AgentFailure] + /// Tests that failed and then passed on a retry. Omitted when empty. + public let flaky: [String]? + /// Why the run failed when `failures` can't say: nothing ran, or the runner died. + public let reason: String? + /// One failing test. `message` is the first line of its first message; `file` and + /// `line` locate it, `moreMessages` counts the rest. public struct AgentFailure: Codable, Sendable, Equatable { public let id: String public let message: String + public let file: String? + public let line: Int? + public let moreMessages: Int? + public let attachments: [String]? + + public init( + id: String, message: String, file: String? = nil, line: Int? = nil, moreMessages: Int? = nil, + attachments: [String]? = nil + ) { + self.id = id + self.message = message + self.file = file + self.line = line + self.moreMessages = moreMessages + self.attachments = attachments + } } enum CodingKeys: String, CodingKey { - case succeeded, buildOk, total, passed, failed, skipped, timedOut, knownFailures, failures + case succeeded, buildOk, total, passed, failed, skipped, timedOut, knownFailures, failures, flaky, reason } public func encode(to encoder: Encoder) throws { @@ -37,6 +59,10 @@ public struct AgentTestResult: Codable, Sendable, Equatable { try c.encode(known, forKey: .knownFailures) } try c.encode(failures, forKey: .failures) + if let flaky, !flaky.isEmpty { + try c.encode(flaky, forKey: .flaky) + } + try c.encodeIfPresent(reason, forKey: .reason) } public init( @@ -48,7 +74,9 @@ public struct AgentTestResult: Codable, Sendable, Equatable { skipped: Int, timedOut: Bool, knownFailures: [String]?, - failures: [AgentFailure] + failures: [AgentFailure], + flaky: [String]? = nil, + reason: String? = nil ) { self.succeeded = succeeded self.buildOk = buildOk @@ -59,6 +87,8 @@ public struct AgentTestResult: Codable, Sendable, Equatable { self.timedOut = timedOut self.knownFailures = knownFailures self.failures = failures + self.flaky = flaky + self.reason = reason } } @@ -78,21 +108,47 @@ public enum AgentResultProjection { } public static func project(_ e: TestTools.TestExecution) -> AgentTestResult { - let knownSet = Set(e.knownFailures ?? []) - let visibleFailures = e.failures.filter { !knownSet.contains($0.testIdentifier) } - let agentFailures = visibleFailures.map { - AgentTestResult.AgentFailure(id: $0.testIdentifier, message: firstLine($0.message)) + let known = e.knownFailures ?? [] + func isKnown(_ id: String) -> Bool { known.contains { TestIDs.same($0, id) } } + let visibleFailures = e.failures.filter { !isKnown($0.testIdentifier) } + let agentFailures = visibleFailures.map { failure -> AgentTestResult.AgentFailure in + let messages = failure.messages ?? [] + let first = messages.first + return AgentTestResult.AgentFailure( + id: failure.testIdentifier, + message: firstLine(first.map { m in m.label.map { "[\($0)] \(m.text)" } ?? m.text } ?? failure.message), + file: first?.file, + line: first?.line, + moreMessages: messages.count > 1 ? messages.count - 1 : nil, + attachments: (failure.attachments ?? []).isEmpty ? nil : failure.attachments + ) + } + // The real count of failing tests, less the gated ones, even when parsing lost some of them. + let failed = max(agentFailures.count, e.failedTestCount - known.count) + var reason: String? + if !e.succeeded && agentFailures.isEmpty { + if e.xcforgeTimedOut { + reason = "timed out before the tests finished" + } else if e.totalTestCount == 0 { + reason = "no tests ran: the filter, scheme or test plan selected none" + } else if failed > 0 { + reason = "\(failed) tests failed but their details couldn't be read from \(e.xcresultPath)" + } else { + reason = e.xcresultParseError ?? "xcodebuild failed without a test failure; see \(e.xcresultPath)" + } } return AgentTestResult( succeeded: e.succeeded, buildOk: !e.buildFailed, total: e.totalTestCount, passed: e.passedTestCount, - failed: agentFailures.count, + failed: failed, skipped: e.skippedTestCount, timedOut: e.xcforgeTimedOut, knownFailures: e.knownFailures, - failures: agentFailures + failures: agentFailures, + flaky: e.flakyTests.isEmpty ? nil : e.flakyTests, + reason: reason ) } @@ -109,7 +165,9 @@ public enum AgentResultProjection { skipped: projected.skipped, timedOut: b.xcforgeTimedOut || projected.timedOut, knownFailures: projected.knownFailures, - failures: projected.failures + failures: projected.failures, + flaky: projected.flaky, + reason: projected.reason ) } return projected @@ -127,7 +185,8 @@ public enum AgentResultProjection { skipped: 0, timedOut: b.xcforgeTimedOut, knownFailures: known, - failures: [] + failures: [], + reason: b.buildSucceeded ? "the tests didn't run" : "the build failed" ) } } diff --git a/Sources/XCForgeKit/Support/ProjectResolver.swift b/Sources/XCForgeKit/Support/ProjectResolver.swift index 6c3a3b7..05d58e4 100644 --- a/Sources/XCForgeKit/Support/ProjectResolver.swift +++ b/Sources/XCForgeKit/Support/ProjectResolver.swift @@ -257,6 +257,31 @@ enum AutoDetect { return [] } + /// Test target names and how they were found. + struct TestTargets: Sendable, Equatable { + let names: [String] + /// True when read from the test plan or scheme, false when guessed from target names. + let exact: Bool + } + + /// Test targets for filter resolution: the test plan's, else the scheme's test action's, + /// else targets whose names end in Tests. Workspaces have no target list in `-list`, so the + /// scheme is the only reliable source there. + static func testTargetNames( + project: String, scheme: String?, testplan: String?, env: Environment = .live + ) async -> TestTargets { + if let testplan, let names = TestPlanInspector.testTargetNames(plan: testplan, project: project), + !names.isEmpty + { + return TestTargets(names: names, exact: true) + } + if let scheme, let names = SchemeFile.testTargets(scheme: scheme, project: project), !names.isEmpty { + return TestTargets(names: names, exact: true) + } + let guessed = (try? await testTargets(project: project, env: env)) ?? [] + return TestTargets(names: guessed, exact: false) + } + // MARK: - Destination builder /// Build xcodebuild destination string from a simulator name, UDID, or physical device identifier. @@ -338,7 +363,7 @@ enum AutoDetect { /// Returns true if the string looks like a physical device UDID. /// Supports both legacy 40-char hex format and newer 8-16 hex format (e.g. 00008101-001A2B3C4D5E6F78). - private static func isPhysicalDeviceUDID(_ s: String) -> Bool { + static func isPhysicalDeviceUDID(_ s: String) -> Bool { let legacy = #"^[0-9a-fA-F]{40}$"# let modern = #"^[0-9a-fA-F]{8}-[0-9a-fA-F]{16}$"# return s.range(of: legacy, options: .regularExpression) != nil @@ -590,3 +615,91 @@ enum AutoDetect { return exactMatch ?? caseInsensitive ?? prefixBooted ?? prefixMatch } } + +/// Reads a scheme's test action from its `.xcscheme` file. +enum SchemeFile { + /// The `.xcscheme` for `scheme`, looked for in the project (or the workspace and the + /// projects beside it), shared schemes first. + static func locate(scheme: String, project: String) -> String? { + let fm = FileManager.default + var containers = [project] + if project.hasSuffix(".xcworkspace") { + containers += projects(near: (project as NSString).deletingLastPathComponent) + } + for container in containers { + let shared = "\(container)/xcshareddata/xcschemes/\(scheme).xcscheme" + if fm.fileExists(atPath: shared) { return shared } + } + for container in containers { + let userData = "\(container)/xcuserdata" + for user in (try? fm.contentsOfDirectory(atPath: userData)) ?? [] { + let path = "\(userData)/\(user)/xcschemes/\(scheme).xcscheme" + if fm.fileExists(atPath: path) { return path } + } + } + return nil + } + + /// `.xcodeproj` bundles within three levels of `directory`, skipping build output. + static func projects(near directory: String) -> [String] { + guard + let enumerator = FileManager.default.enumerator( + at: URL(fileURLWithPath: directory), includingPropertiesForKeys: [.isDirectoryKey], + options: [.skipsHiddenFiles, .skipsPackageDescendants]) + else { return [] } + var found: [String] = [] + for case let url as URL in enumerator { + if SourceChanges.skippedDirectories.contains(url.lastPathComponent) { + enumerator.skipDescendants() + continue + } + if url.pathExtension == "xcodeproj" { found.append(url.path) } + if enumerator.level >= 3 { enumerator.skipDescendants() } + } + return found.sorted() + } + + /// Test target names the scheme's test action runs: those of its test plans when it uses + /// them, else its testables. Nil when the scheme file can't be found. + static func testTargets(scheme: String, project: String) -> [String]? { + guard let path = locate(scheme: scheme, project: project), + let xml = try? String(contentsOfFile: path, encoding: .utf8) + else { return nil } + // `container:` paths are relative to the folder holding the project or workspace that + // owns the scheme, which for a workspace may be a project in a subfolder. + var container = (path as NSString).deletingLastPathComponent + while !container.hasSuffix(".xcodeproj") && !container.hasSuffix(".xcworkspace") && container.count > 1 { + container = (container as NSString).deletingLastPathComponent + } + return testTargets(schemeXML: xml, projectDirectory: (container as NSString).deletingLastPathComponent) + } + + static func testTargets(schemeXML xml: String, projectDirectory: String) -> [String] { + guard let start = xml.range(of: "", range: start.upperBound..]*?skipped\s*=\s*"NO"[\s\S]*?"# + let testables = matches(enabled, in: action, group: 0) + for testable in testables { + names += matches(#"BlueprintName\s*=\s*"([^"]+)""#, in: testable).prefix(1) + } + } + var seen = Set() + return names.filter { seen.insert($0).inserted } + } + + private static func matches(_ pattern: String, in text: String, group: Int = 1) -> [String] { + guard let regex = try? NSRegularExpression(pattern: pattern) else { return [] } + return regex.matches(in: text, range: NSRange(text.startIndex..., in: text)).compactMap { + Range($0.range(at: group), in: text).map { String(text[$0]) } + } + } +} diff --git a/Sources/XCForgeKit/Support/TestIDs.swift b/Sources/XCForgeKit/Support/TestIDs.swift new file mode 100644 index 0000000..e64ad44 --- /dev/null +++ b/Sources/XCForgeKit/Support/TestIDs.swift @@ -0,0 +1,199 @@ +import Foundation + +/// One test ID format everywhere: `Target/Suite/test()` for Swift Testing, +/// `Target/Class/testMethod` for XCTest, the form `list_tests` prints and +/// `-only-testing` accepts. Shorter forms (`Suite/test()`, `test()`) are accepted as +/// input and compared with `same(_:_:)`. +public enum TestIDs { + /// Slash-separated components, ignoring slashes inside `[...]` and `(...)`. + public static func components(_ id: String) -> [String] { + var parts: [String] = [] + var current = "" + var depth = 0 + for ch in id { + if ch == "[" || ch == "(" { depth += 1 } + if (ch == "]" || ch == ")") && depth > 0 { depth -= 1 } + if ch == "/" && depth == 0 { + parts.append(current) + current = "" + continue + } + current.append(ch) + } + parts.append(current) + return parts + } + + /// The full ID for a test the result bundle reports as `identifier` (which lacks the + /// target) inside the test bundle named `target`. + public static func canonical(target: String?, identifier: String) -> String { + let id = collapseDoubleParens(identifier) + guard let target, !target.isEmpty else { return id } + if id == target || id.hasPrefix(target + "/") { return id } + return target + "/" + id + } + + /// `test()()` back to `test()`: the doubled form is only an `-only-testing` workaround. + static func collapseDoubleParens(_ id: String) -> String { + id.hasSuffix("()()") ? String(id.dropLast(2)) : id + } + + /// Comparison key: doubled parens collapsed and a trailing `()` dropped, so the XCTest + /// and Swift Testing spellings of a name compare equal. + static func key(_ id: String) -> String { + let collapsed = collapseDoubleParens(id.trimmingCharacters(in: .whitespaces)) + return collapsed.hasSuffix("()") ? String(collapsed.dropLast(2)) : collapsed + } + + /// True when `a` and `b` name the same test, allowing either to leave out leading + /// components (the target, or outer suites). `testFoo` never matches `testFoo2`. + public static func same(_ a: String, _ b: String) -> Bool { + let x = key(a) + let y = key(b) + guard !x.isEmpty, !y.isEmpty else { return false } + return x == y || x.hasSuffix("/" + y) || y.hasSuffix("/" + x) + } + + /// The argument to pass to `-only-testing` for a target-qualified ID. xcodebuild strips a + /// trailing `()` from the last component, while Swift Testing names its tests with one, + /// so a Swift Testing test needs it doubled to be found. + static func onlyTestingArgument(_ id: String) -> String { + guard components(id).count >= 3, id.hasSuffix("()"), !id.hasSuffix("()()") else { return id } + return id + "()" + } +} + +/// A failure message with the source location xcresult put in front of it +/// (`File.swift:42: message`) split out. +public struct FailureMessage: Codable, Sendable, Equatable { + public let text: String + public let file: String? + public let line: Int? + /// The argument, repetition or device the message came from, when the test ran more than once. + public let label: String? + + public init(text: String, file: String? = nil, line: Int? = nil, label: String? = nil) { + self.text = text + self.file = file + self.line = line + self.label = label + } + + /// Split `File.swift:42: message` into its parts. Text without a location is kept whole. + public static func parse(_ raw: String, label: String? = nil) -> FailureMessage { + let pattern = #"^([^\s:][^:\n]*\.(?:swift|m|mm|c|cc|cpp|h|hpp)):(\d+):\s?(.*)$"# + guard let regex = try? NSRegularExpression(pattern: pattern, options: [.dotMatchesLineSeparators]), + let match = regex.firstMatch(in: raw, range: NSRange(raw.startIndex..., in: raw)), + let fileRange = Range(match.range(at: 1), in: raw), + let lineRange = Range(match.range(at: 2), in: raw), + let textRange = Range(match.range(at: 3), in: raw) + else { + return FailureMessage(text: raw, label: label) + } + return FailureMessage( + text: String(raw[textRange]), file: String(raw[fileRange]), line: Int(raw[lineRange]), label: label) + } + + /// `[label] File.swift:42: text`, as a person reads it. + public var display: String { + var out = "" + if let label { out += "[\(label)] " } + if let file { out += "\(file):\(line ?? 0): " } + return out + text + } +} + +/// Shared text rendering of failure lists: identical messages grouped, the list capped +/// with a count of what was left out, every ID in its full form. +public enum TestFailureText { + public static let defaultLimit = 20 + + public static func lines( + _ failures: [TestTools.TestFailureObservation], limit: Int = defaultLimit, indent: String = " " + ) -> [String] { + // Group failures whose whole message is identical (a shared setUp failing, say). + var groups: [(ids: [String], failure: TestTools.TestFailureObservation)] = [] + var indexByMessage: [String: Int] = [:] + for failure in failures { + if !failure.message.isEmpty, let i = indexByMessage[failure.message] { + groups[i].ids.append(failure.testIdentifier) + } else { + indexByMessage[failure.message] = groups.count + groups.append((ids: [failure.testIdentifier], failure: failure)) + } + } + + var out: [String] = [] + var shownTests = 0 + for group in groups.prefix(limit) { + shownTests += group.ids.count + let names = + group.ids.count == 1 + ? group.ids[0] + : group.ids.prefix(5).joined(separator: ", ") + + (group.ids.count > 5 ? " and \(group.ids.count - 5) more" : "") + + " (\(group.ids.count) tests, same message)" + out.append("\(indent)FAIL: \(names)") + let messages = group.failure.messages ?? [] + if messages.isEmpty { + if !group.failure.message.isEmpty { + out.append(indent + " " + group.failure.message.replacingOccurrences(of: "\n", with: "\n\(indent) ")) + } + } else { + for message in messages.prefix(5) { + out.append(indent + " " + message.display.replacingOccurrences(of: "\n", with: "\n\(indent) ")) + } + if messages.count > 5 { out.append("\(indent) +\(messages.count - 5) more messages") } + } + for path in group.failure.attachments ?? [] { + out.append("\(indent) Attachment: \(path)") + } + if let console = group.failure.console, !console.isEmpty { + out.append("\(indent) Console (last lines):") + out.append(indent + " " + console.replacingOccurrences(of: "\n", with: "\n\(indent) ")) + } + } + let hidden = failures.count - shownTests + if hidden > 0 { + out.append("\(indent)+\(hidden) more failures (test_failures or `xcforge test failures` lists them all)") + } + return out + } +} + +/// Whether any source under a folder changed after a given time, so a rerun can skip +/// build-for-testing when nothing was edited. +enum SourceChanges { + static let skippedDirectories: Set = [ + ".git", ".build", ".swiftpm", ".xcforge", "DerivedData", "build", "Build", "node_modules", "Pods", + "Carthage", "xcuserdata", + ] + + /// More files than this and the walk gives up and reports a change. + static let fileCap = 100_000 + + static func anyModified(under root: String, after date: Date) -> Bool { + let url = URL(fileURLWithPath: root) + guard + let enumerator = FileManager.default.enumerator( + at: url, includingPropertiesForKeys: [.contentModificationDateKey, .isDirectoryKey], + options: [.skipsHiddenFiles]) + else { return true } + var count = 0 + for case let file as URL in enumerator { + let name = file.lastPathComponent + if skippedDirectories.contains(name) || name.hasSuffix(".xcresult") { + enumerator.skipDescendants() + continue + } + count += 1 + if count > fileCap { return true } + guard let values = try? file.resourceValues(forKeys: [.contentModificationDateKey, .isDirectoryKey]), + values.isDirectory != true, + let modified = values.contentModificationDate + else { continue } + if modified > date { return true } + } + return false + } +} diff --git a/Sources/XCForgeKit/Tools/TestPlanInspector.swift b/Sources/XCForgeKit/Tools/TestPlanInspector.swift index 558e353..0035563 100644 --- a/Sources/XCForgeKit/Tools/TestPlanInspector.swift +++ b/Sources/XCForgeKit/Tools/TestPlanInspector.swift @@ -78,8 +78,17 @@ public enum TestPlanInspector { public static func inspectTestPlan(name: String, project: String, env: Environment) async throws -> String { + let located = locate(name: name, project: project) + if let path = located.path { + return try parse(at: path) + } + throw InspectError.notFound(name: name, searched: located.searched) + } + + /// Find the `.xctestplan` named `name` for `project`, and the places looked in. + static func locate(name: String, project: String) -> (path: String?, searched: [String]) { let normalized = name.hasSuffix(".xctestplan") ? name : "\(name).xctestplan" - let baseName = normalized + let baseName = (normalized as NSString).lastPathComponent let projectDir: String if project.hasSuffix(".xcodeproj") || project.hasSuffix(".xcworkspace") { @@ -100,18 +109,30 @@ public enum TestPlanInspector { if let entries = try? FileManager.default.contentsOfDirectory(atPath: planDir), let match = entries.first(where: { $0.lowercased() == baseName.lowercased() }) { - return try parse(at: "\(planDir)/\(match)") + return ("\(planDir)/\(match)", searchedPaths) } } // Recursive find under projectDir (case-insensitive) - let found = findTestPlan(named: baseName, under: projectDir) - if let found { - return try parse(at: found) + if let found = findTestPlan(named: baseName, under: projectDir) { + return (found, searchedPaths) } searchedPaths.append("\(projectDir)/**/*.xctestplan (recursive, case-insensitive)") + return (nil, searchedPaths) + } - throw InspectError.notFound(name: name, searched: searchedPaths) + /// Names of the test targets a plan runs (disabled targets left out), or nil when the plan + /// can't be found or read. + static func testTargetNames(plan: String, project: String) -> [String]? { + guard let path = locate(name: plan, project: project).path else { return nil } + return testTargetNames(atPath: path) + } + + static func testTargetNames(atPath path: String) -> [String]? { + guard let data = FileManager.default.contents(atPath: path), + let decoded = try? JSONDecoder().decode(XCTestPlan.self, from: data) + else { return nil } + return decoded.testTargets.filter { $0.enabled != false }.map(\.target.name) } // MARK: - Private diff --git a/Sources/XCForgeKit/Tools/TestProvider.swift b/Sources/XCForgeKit/Tools/TestProvider.swift index f6a3c5e..8b10c6b 100644 --- a/Sources/XCForgeKit/Tools/TestProvider.swift +++ b/Sources/XCForgeKit/Tools/TestProvider.swift @@ -16,9 +16,17 @@ struct TestDiscoveryError: Error, CustomStringConvertible { public enum TestTools { public struct TestFailureObservation: Codable, Sendable, Equatable { public let testName: String + /// Full ID, `Target/Suite/test()`, the form `list_tests` prints and filters accept. public let testIdentifier: String + /// Every message joined, for display. `messages` has them one by one with location. public let message: String public let source: String + /// Each failure message with its file and line, and the argument or repetition it came from. + public var messages: [FailureMessage]? + /// Files xcresult attached to this test (failure screenshots). + public var attachments: [String]? + /// The last lines the test printed, when console output was requested. + public var console: String? } public struct BuildIssueObservation: Codable, Sendable, Equatable { @@ -150,6 +158,8 @@ public enum TestTools { /// Test IDs the run treated as known-failing via `.xcforge/known-failures.yaml`. /// Non-nil only when the gate was requested for this run. public let knownFailures: [String]? + /// Tests that failed at least once and then passed on a retry or repetition. + public let flakyTests: [String] init( succeeded: Bool, @@ -174,7 +184,8 @@ public enum TestTools { xcresultParseError: String? = nil, xcforgeTimedOut: Bool = false, slowestTests: [SlowTest] = [], - knownFailures: [String]? = nil + knownFailures: [String]? = nil, + flakyTests: [String] = [] ) { self.succeeded = succeeded self.elapsed = elapsed @@ -199,6 +210,7 @@ public enum TestTools { self.xcforgeTimedOut = xcforgeTimedOut self.slowestTests = slowestTests self.knownFailures = knownFailures + self.flakyTests = flakyTests } } @@ -281,6 +293,15 @@ public enum TestTools { var `for`: String? var gate: Bool? var isolatedSimulator: Bool? + var timeoutSeconds: Int? + var env: [String]? + var skipBuild: Bool? + var retries: Int? + var iterations: Int? + var untilFailure: Bool? + var parallel: Bool? + var testTimeoutSeconds: Int? + var includeConsole: Bool? } struct TestFailuresInput: Decodable { @@ -324,6 +345,13 @@ public enum TestTools { var `for`: String? var gate: Bool? var isolatedSimulator: Bool? + var skipBuild: Bool? + var retries: Int? + var iterations: Int? + var untilFailure: Bool? + var parallel: Bool? + var testTimeoutSeconds: Int? + var includeConsole: Bool? } /// Error for malformed --env / env: entries. @@ -391,6 +419,17 @@ public enum TestTools { public let xcforgeTimedOut: Bool /// Test IDs gated via `.xcforge/known-failures.yaml`. Non-nil only when gating was requested. public let knownFailures: [String]? + /// What was built and tested. + public let project: String? + public let scheme: String? + public let simulator: String? + /// True when build-for-testing was skipped and the last build's products were tested. + public let skippedBuild: Bool + /// Result bundle of a failed build-for-testing. + public let buildXcresultPath: String? + public let hangDiagnosticSummary: String? + /// Timeout explanation, or the end of xcodebuild's output when no diagnostics were parsed. + public let buildFailureDetail: String? init( phase: String, @@ -404,7 +443,14 @@ public enum TestTools { recoveryFailureReason: String? = nil, simHealthCheckDetail: String? = nil, xcforgeTimedOut: Bool = false, - knownFailures: [String]? = nil + knownFailures: [String]? = nil, + project: String? = nil, + scheme: String? = nil, + simulator: String? = nil, + skippedBuild: Bool = false, + buildXcresultPath: String? = nil, + hangDiagnosticSummary: String? = nil, + buildFailureDetail: String? = nil ) { self.phase = phase self.buildSucceeded = buildSucceeded @@ -418,6 +464,13 @@ public enum TestTools { self.simHealthCheckDetail = simHealthCheckDetail self.xcforgeTimedOut = xcforgeTimedOut self.knownFailures = knownFailures + self.project = project + self.scheme = scheme + self.simulator = simulator + self.skippedBuild = skippedBuild + self.buildXcresultPath = buildXcresultPath + self.hangDiagnosticSummary = hangDiagnosticSummary + self.buildFailureDetail = buildFailureDetail } } @@ -448,91 +501,143 @@ public enum TestTools { } } + /// Options every test-running tool takes, so a rerun can reproduce the run it repeats. + static let testRunSchemaProperties: [String: Value] = [ + "timeoutSeconds": .object([ + "type": .string("integer"), + "description": .string( + "Total time limit in seconds for each xcodebuild step. Takes precedence over 'long'. Default: 1800 (7200 with long)." + ), + ]), + "env": .object([ + "type": .string("array"), + "items": .object(["type": .string("string")]), + "description": .string( + "Environment variables for the test runner, each KEY=VALUE. The key is prefixed with TEST_RUNNER_ for" + + " xcodebuild, which strips it inside the test process, so 'BLESS_BASELINE=1' reads as" + + " ProcessInfo.environment[\"BLESS_BASELINE\"]. TEST_RUNNER_XCFORGE_REPO_ROOT is always set." + ), + ]), + "skipBuild": .object([ + "type": .string("boolean"), + "description": .string( + "Skip build-for-testing and test the last build's products. Only valid when no source changed since."), + ]), + "retries": .object([ + "type": .string("integer"), + "description": .string( + "Rerun a failing test up to this many more times. Tests that then pass are listed as flaky."), + ]), + "iterations": .object([ + "type": .string("integer"), + "description": .string("Run every test this many times."), + ]), + "untilFailure": .object([ + "type": .string("boolean"), + "description": .string("Repeat the tests until one fails (capped by iterations when given)."), + ]), + "parallel": .object([ + "type": .string("boolean"), + "description": .string("Turn parallel testing on or off. Default: the scheme's or test plan's setting."), + ]), + "testTimeoutSeconds": .object([ + "type": .string("integer"), + "description": .string( + "Time allowance per test in seconds (XCTest; Swift Testing uses .timeLimit in code)."), + ]), + "includeConsole": .object([ + "type": .string("boolean"), + "description": .string("Attach the last lines each failing test printed to its failure."), + ]), + ] + public static let tools: [Tool] = [ Tool( name: "test_sim", description: """ - Run xcodebuild test on simulator and return structured xcresult summary. \ - Shows passed/failed/skipped/expected-failure counts and duration. \ + Build for testing, then run tests on a simulator and return the result: counts, and \ + each failure with its full test ID, every message with file:line, and attachments. \ Project, scheme, and simulator are auto-detected if omitted. """, inputSchema: .object([ "type": .string("object"), - "properties": .object([ - "project": .object([ - "type": .string("string"), - "description": .string("Path to .xcodeproj or .xcworkspace. Auto-detected if omitted."), - ]), - "scheme": .object([ - "type": .string("string"), - "description": .string("Xcode scheme name. Auto-detected if omitted."), - ]), - "simulator": .object([ - "type": .string("string"), - "description": .string( - "Simulator name or UDID. Auto-detected from booted simulator if omitted."), - ]), - "configuration": .object([ - "type": .string("string"), - "description": .string("Build configuration (Debug/Release). Default: Debug"), - ]), - "testplan": .object([ - "type": .string("string"), - "description": .string( - "Test plan name. Required when filtering Swift Testing @Test suites (they are not discoverable by -only-testing without a testplan). Optional for XCTest." - ), - ]), - "filter": .object([ - "type": .string("string"), - "description": .string( - "Test filter. Accepts: 'testMethod', 'TestClass/testMethod', or full 'TestTarget/TestClass/testMethod'. Target prefix auto-resolution works for XCTest only. Swift Testing @Test suites require the full Target/Suite path AND a testplan to be discoverable." - ), - ]), - "coverage": .object([ - "type": .string("boolean"), - "description": .string("Enable code coverage collection. Default: false"), - ]), - "long": .object([ - "type": .string("boolean"), - "description": .string( - "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." - ), - ]), - "diagnose": .object([ - "type": .string("boolean"), - "description": .string( - "Capture a diagnostic snapshot on completion even without a hang, for baseline inspection." - ), - ]), - "simRecovery": .object([ - "type": .string("string"), - "description": .string( - "Simulator recovery before the run: 'off' (default) leaves the simulator alone; 'auto' reboots it if it isn't Booted;" - + " 'erase' also erases it if a reboot didn't help (destroys its apps and data)." - ), - "enum": .array([.string("off"), .string("auto"), .string("erase")]), - ]), - "isolatedSimulator": .object([ - "type": .string("boolean"), - "description": .string( - "Run on a fresh simulator of the same model and OS, created for this run and deleted afterwards." - + " Avoids collisions with other sessions on a shared simulator. Costs one cold boot." - ), - ]), - "for": .object([ - "type": .string("string"), - "description": .string( - "Output audience. 'human' (default) preserves the full JSON shape; 'agent' returns a slim ≤10-field projection." - ), - "enum": .array([.string("human"), .string("agent")]), - ]), - "gate": .object([ - "type": .string("boolean"), - "description": .string( - "Subtract IDs listed in .xcforge/known-failures.yaml when computing succeeded:. Opt-in; raw failure list is unchanged." - ), - ]), - ]), + "properties": .object( + testRunSchemaProperties.merging([ + "project": .object([ + "type": .string("string"), + "description": .string("Path to .xcodeproj or .xcworkspace. Auto-detected if omitted."), + ]), + "scheme": .object([ + "type": .string("string"), + "description": .string("Xcode scheme name. Auto-detected if omitted."), + ]), + "simulator": .object([ + "type": .string("string"), + "description": .string( + "Simulator name or UDID. Auto-detected from booted simulator if omitted."), + ]), + "configuration": .object([ + "type": .string("string"), + "description": .string("Build configuration (Debug/Release). Default: Debug"), + ]), + "testplan": .object([ + "type": .string("string"), + "description": .string("Test plan name. Default: .xcforge.yaml testPlan, else the scheme's."), + ]), + "filter": .object([ + "type": .string("string"), + "description": .string( + "Tests to run, comma-separated. Full IDs as list_tests prints them ('Target/Suite/test()'," + + " 'Target/Class/testMethod'), or without the target ('Suite/test()', 'Suite'), which is" + + " added from the scheme or test plan." + ), + ]), + "coverage": .object([ + "type": .string("boolean"), + "description": .string("Enable code coverage collection. Default: false"), + ]), + "long": .object([ + "type": .string("boolean"), + "description": .string( + "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." + ), + ]), + "diagnose": .object([ + "type": .string("boolean"), + "description": .string( + "Capture a diagnostic snapshot on completion even without a hang, for baseline inspection." + ), + ]), + "simRecovery": .object([ + "type": .string("string"), + "description": .string( + "Simulator recovery before the run: 'off' (default) leaves the simulator alone; 'auto' reboots it if it isn't Booted;" + + " 'erase' also erases it if a reboot didn't help (destroys its apps and data)." + ), + "enum": .array([.string("off"), .string("auto"), .string("erase")]), + ]), + "isolatedSimulator": .object([ + "type": .string("boolean"), + "description": .string( + "Run on a fresh simulator of the same model and OS, created for this run and deleted afterwards." + + " Avoids collisions with other sessions on a shared simulator. Costs one cold boot." + ), + ]), + "for": .object([ + "type": .string("string"), + "description": .string( + "Output audience. 'human' (default) preserves the full JSON shape; 'agent' returns a slim ≤10-field projection." + ), + "enum": .array([.string("human"), .string("agent")]), + ]), + "gate": .object([ + "type": .string("boolean"), + "description": .string( + "Subtract IDs listed in .xcforge/known-failures.yaml when computing succeeded:. Opt-in; raw failure list is unchanged." + ), + ]), + ]) { current, _ in current } + ), ]) ), Tool( @@ -654,93 +759,82 @@ public enum TestTools { """, inputSchema: .object([ "type": .string("object"), - "properties": .object([ - "project": .object([ - "type": .string("string"), - "description": .string("Path to .xcodeproj or .xcworkspace. Auto-detected if omitted."), - ]), - "scheme": .object([ - "type": .string("string"), - "description": .string("Xcode scheme name. Auto-detected if omitted."), - ]), - "simulator": .object([ - "type": .string("string"), - "description": .string("Simulator name or UDID. Auto-detected if omitted."), - ]), - "configuration": .object([ - "type": .string("string"), - "description": .string("Build configuration (Debug/Release). Default: Debug"), - ]), - "testplan": .object([ - "type": .string("string"), - "description": .string( - "Test plan name. Required when filtering Swift Testing @Test suites (they are not discoverable by -only-testing without a testplan). Optional for XCTest." - ), - ]), - "filter": .object([ - "type": .string("string"), - "description": .string( - "Test filter. Accepts: 'testMethod', 'TestClass/testMethod', or full 'TestTarget/TestClass/testMethod'. Target prefix auto-resolution works for XCTest only. Swift Testing @Test suites require the full Target/Suite path AND a testplan to be discoverable." - ), - ]), - "coverage": .object([ - "type": .string("boolean"), - "description": .string("Enable code coverage collection. Default: false"), - ]), - "long": .object([ - "type": .string("boolean"), - "description": .string( - "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." - ), - ]), - "diagnose": .object([ - "type": .string("boolean"), - "description": .string( - "Capture a diagnostic snapshot on completion even without a hang, for baseline inspection." - ), - ]), - "simRecovery": .object([ - "type": .string("string"), - "description": .string( - "Simulator recovery before the run: 'off' (default) leaves the simulator alone; 'auto' reboots it if it isn't Booted;" - + " 'erase' also erases it if a reboot didn't help (destroys its apps and data)." - ), - "enum": .array([.string("off"), .string("auto"), .string("erase")]), - ]), - "isolatedSimulator": .object([ - "type": .string("boolean"), - "description": .string( - "Run on a fresh simulator of the same model and OS, created for this run and deleted afterwards." - + " Avoids collisions with other sessions on a shared simulator. Costs one cold boot." - ), - ]), - "timeoutSeconds": .object([ - "type": .string("integer"), - "description": .string( - "Override the xcodebuild timeout in seconds. Takes precedence over 'long'. Default: 1800 (or 7200 with long: true)." - ), - ]), - "env": .object([ - "type": .string("array"), - "items": .object(["type": .string("string")]), - "description": .string( - "Environment variables for the test runner. Each entry is KEY=VALUE; the key is auto-prefixed with TEST_RUNNER_ before xcodebuild runs (Xcode strips the prefix inside the test process). Example: 'BLESS_BASELINE=1' surfaces as ProcessInfo.environment[\"BLESS_BASELINE\"] in tests. TEST_RUNNER_XCFORGE_REPO_ROOT is always injected; override by supplying 'XCFORGE_REPO_ROOT=...'." - ), - ]), - "for": .object([ - "type": .string("string"), - "description": .string( - "Output audience. 'human' (default) preserves the full JSON shape; 'agent' returns a slim ≤10-field projection." - ), - "enum": .array([.string("human"), .string("agent")]), - ]), - "gate": .object([ - "type": .string("boolean"), - "description": .string( - "Subtract IDs listed in .xcforge/known-failures.yaml when computing succeeded:. Opt-in; raw failure list is unchanged." - ), - ]), - ]), + "properties": .object( + testRunSchemaProperties.merging([ + "project": .object([ + "type": .string("string"), + "description": .string("Path to .xcodeproj or .xcworkspace. Auto-detected if omitted."), + ]), + "scheme": .object([ + "type": .string("string"), + "description": .string("Xcode scheme name. Auto-detected if omitted."), + ]), + "simulator": .object([ + "type": .string("string"), + "description": .string("Simulator name or UDID. Auto-detected if omitted."), + ]), + "configuration": .object([ + "type": .string("string"), + "description": .string("Build configuration (Debug/Release). Default: Debug"), + ]), + "testplan": .object([ + "type": .string("string"), + "description": .string("Test plan name. Default: .xcforge.yaml testPlan, else the scheme's."), + ]), + "filter": .object([ + "type": .string("string"), + "description": .string( + "Tests to run, comma-separated. Full IDs as list_tests prints them ('Target/Suite/test()'," + + " 'Target/Class/testMethod'), or without the target ('Suite/test()', 'Suite'), which is" + + " added from the scheme or test plan." + ), + ]), + "coverage": .object([ + "type": .string("boolean"), + "description": .string("Enable code coverage collection. Default: false"), + ]), + "long": .object([ + "type": .string("boolean"), + "description": .string( + "Raise the total time limit from 1800s to 7200s. Hangs are caught by idleTimeoutSeconds either way." + ), + ]), + "diagnose": .object([ + "type": .string("boolean"), + "description": .string( + "Capture a diagnostic snapshot on completion even without a hang, for baseline inspection." + ), + ]), + "simRecovery": .object([ + "type": .string("string"), + "description": .string( + "Simulator recovery before the run: 'off' (default) leaves the simulator alone; 'auto' reboots it if it isn't Booted;" + + " 'erase' also erases it if a reboot didn't help (destroys its apps and data)." + ), + "enum": .array([.string("off"), .string("auto"), .string("erase")]), + ]), + "isolatedSimulator": .object([ + "type": .string("boolean"), + "description": .string( + "Run on a fresh simulator of the same model and OS, created for this run and deleted afterwards." + + " Avoids collisions with other sessions on a shared simulator. Costs one cold boot." + ), + ]), + "for": .object([ + "type": .string("string"), + "description": .string( + "Output audience. 'human' (default) preserves the full JSON shape; 'agent' returns a slim ≤10-field projection." + ), + "enum": .array([.string("human"), .string("agent")]), + ]), + "gate": .object([ + "type": .string("boolean"), + "description": .string( + "Subtract IDs listed in .xcforge/known-failures.yaml when computing succeeded:. Opt-in; raw failure list is unchanged." + ), + ]), + ]) { current, _ in current } + ), ]) ), Tool( @@ -910,61 +1004,51 @@ public enum TestTools { /// Agents often pass `ClassName/testMethod` or just `testMethod` without the test target prefix. /// This discovers the test target(s) and prepends when missing. static func resolveFilter( - _ filter: String, project: String, env: Environment + _ filter: String, project: String, scheme: String? = nil, testplan: String? = nil, env: Environment ) async -> String { let trimmedFilter = filter.trimmingCharacters(in: .whitespacesAndNewlines) guard !trimmedFilter.isEmpty else { return filter } + let resolved = await resolveFilters( + splitFilterList(trimmedFilter), project: project, scheme: scheme, testplan: testplan, env: env) + return resolved.joined(separator: ",") + } - // Comma-separated list: resolve each entry independently and rejoin. - let parts = splitFilterList(trimmedFilter) - if parts.count > 1 { - var resolved: [String] = [] - for part in parts { - resolved.append(await resolveFilter(part, project: project, env: env)) - } - return resolved.joined(separator: ",") - } - - let componentCount = slashComponentCount(trimmedFilter) - - // 2+ unbracketed slashes means Target/Class/method — assume already complete - if componentCount >= 3 { - // Swift Testing workaround: xcodebuild strips a trailing "()" from the last - // component of -only-testing identifiers. Swift Testing methods are internally - // identified with "()", so we must append "()" again to survive the stripping. - // e.g. "Target/Suite/testFoo()" → "Target/Suite/testFoo()()" so xcodebuild - // strips the outer "()" and the inner "()" matches the Swift Testing identifier. - if trimmedFilter.hasSuffix("()") && !trimmedFilter.hasSuffix("()()") { - return trimmedFilter + "()" + /// Resolve each ID to `Target/Suite/test()` and its `-only-testing` spelling. Test targets + /// are looked up once, and only when an ID needs one. + static func resolveFilters( + _ ids: [String], project: String, scheme: String? = nil, testplan: String? = nil, env: Environment + ) async -> [String] { + var targets: AutoDetect.TestTargets? + var resolved: [String] = [] + for raw in ids { + let id = TestIDs.collapseDoubleParens(raw.trimmingCharacters(in: .whitespacesAndNewlines)) + guard !id.isEmpty else { continue } + if targets == nil { + targets = await AutoDetect.testTargetNames( + project: project, scheme: scheme, testplan: testplan, env: env) } - return trimmedFilter - } - - // Check if the first component already matches a known test target. - // This prevents double-prepending (e.g. "Target/Suite" → "Target/Target/Suite"). - let firstComponent = extractFirstComponent(trimmedFilter) - - // 0 or 1 unbracketed slash — may need target prefix - guard let targets = try? await AutoDetect.testTargets(project: project, env: env), - !targets.isEmpty - else { - return trimmedFilter - } - - // If the first component already IS a known test target, don't prepend - if let firstComponent, targets.contains(firstComponent) { - return trimmedFilter - } - - if targets.count == 1 { - return "\(targets[0])/\(trimmedFilter)" + let known = targets ?? AutoDetect.TestTargets(names: [], exact: false) + resolved.append(TestIDs.onlyTestingArgument(qualify(id, targets: known.names, exact: known.exact))) } + return resolved + } - // Ambiguous — log available targets for diagnostics - Log.warn( - "Multiple test targets found: \(targets.joined(separator: ", ")). Cannot auto-resolve filter '\(trimmedFilter)'. Prefix with target name, e.g. '\(targets[0])/\(trimmedFilter)'" - ) - return trimmedFilter + /// Prefix `id` with its test target when it lacks one and the target is known. `exact` says + /// the target list came from the scheme or test plan rather than a naming guess; only then + /// is a three-part ID such as `Outer/Inner/test()` taken to be missing its target. + static func qualify(_ id: String, targets: [String], exact: Bool = true) -> String { + let components = TestIDs.components(id) + // Already starts with a known target (also prevents "Target/Suite" → "Target/Target/Suite"). + if let first = components.first, targets.contains(first) { return id } + if !exact && components.count >= 3 { return id } + if targets.count == 1 { return "\(targets[0])/\(id)" } + if targets.count > 1 && components.count < 3 { + Log.warn( + "Multiple test targets found: \(targets.joined(separator: ", ")). Cannot auto-resolve filter '\(id)'." + + " Prefix with target name, e.g. '\(targets[0])/\(id)'") + } + // Unknown targets: three or more components is taken to be complete already. + return id } /// Count slash-separated components, ignoring slashes inside `[...]` brackets. @@ -984,26 +1068,6 @@ public enum TestTools { return count } - /// Extract the first slash-separated component (before the first unbracketed slash). - private static func extractFirstComponent(_ filter: String) -> String? { - var result = "" - var bracketDepth = 0 - for char in filter { - if char == "[" { - bracketDepth += 1 - result.append(char) - } else if char == "]", bracketDepth > 0 { - bracketDepth -= 1 - result.append(char) - } else if char == "/", bracketDepth == 0 { - return result.isEmpty ? nil : result - } else { - result.append(char) - } - } - return result.isEmpty ? nil : result - } - /// Discover test targets for error messages when filter resolution fails. static func availableTestTargets(project: String, env: Environment) async -> [String] { (try? await AutoDetect.testTargets(project: project, env: env)) ?? [] @@ -1017,14 +1081,6 @@ public enum TestTools { ) async -> String { var hint = "\n\n⚠️ 0 tests matched filter \"\(filter)\"" - // Swift Testing hint: testplan is required for -only-testing to discover Swift Testing suites - if testplan == nil { - hint += - "\nHint: Swift Testing @Test suites require a testplan to be discoverable by -only-testing." - hint += - "\nTry adding testplan, e.g.: test_sim(testplan: \"\", filter: \"\(filter)\")" - } - guard let listResult = try? await executeListTests( project: project, scheme: scheme, simulator: simulator, testplan: testplan, env: env @@ -1038,8 +1094,10 @@ public enum TestTools { hint += "\nThe test bundle appears to be empty." return hint } - let lowered = filter.lowercased() - let matches = listResult.tests.filter { $0.fullIdentifier.lowercased().contains(lowered) } + let lowered = TestIDs.key(filter).lowercased() + let matches = listResult.tests.filter { + TestIDs.same($0.fullIdentifier, filter) || $0.fullIdentifier.lowercased().contains(lowered) + } if !matches.isEmpty { hint += "\nDid you mean:" for m in matches.prefix(10) { @@ -1200,6 +1258,7 @@ public enum TestTools { long: Bool = false, diagnose: Bool = false, udid: String? = nil, timeoutOverride: TimeInterval? = nil, childEnvironment: [String: String]? = nil, + extraTestArguments: [String] = [], env: Environment ) async throws -> (ShellResult, String, DiagnosticSnapshot.Result?) { _ = try? await env.shell.run("/bin/rm", arguments: ["-rf", resultPath], timeout: 5) @@ -1220,6 +1279,7 @@ public enum TestTools { args += ["-only-testing", id] } } + args += extraTestArguments args += ["test-without-building"] let timeout: TimeInterval @@ -1549,11 +1609,12 @@ public enum TestTools { return attachments } - /// Extract console output per failed test from xcresult action log - /// Returns dict: testName → emittedOutput - private static func extractFailedTestConsole(_ xcresultPath: String, env: Environment) async - -> [String: String] - { + /// Lines of console output kept per test: the end of the output, where a failure is. + static let consoleTailLines = 40 + + /// Console output per test from the xcresult action log, keyed by the test's identifier + /// when the log gives one and by its name otherwise. Only the last lines are kept. + static func extractTestConsole(_ xcresultPath: String, env: Environment) async -> [String: String] { let shellResult: ShellResult do { shellResult = try await env.shell.run( @@ -1564,42 +1625,38 @@ public enum TestTools { timeout: 30 ) } catch { - Log.warn("extractFailedTestConsole error: \(error)") + Log.warn("extractTestConsole error: \(error)") return [:] } - guard shellResult.succeeded, - let data = shellResult.stdout.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] - else { + guard shellResult.succeeded, let data = shellResult.stdout.data(using: .utf8) else { if !shellResult.succeeded { - Log.warn("extractFailedTestConsole failed: \(shellResult.stderr)") + Log.warn("extractTestConsole failed: \(shellResult.stderr)") } return [:] } + return parseTestConsole(data) + } + static func parseTestConsole(_ data: Data) -> [String: String] { + guard let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { return [:] } var consoleByTest: [String: String] = [:] - // Recursively find test case subsections with testDetails.emittedOutput func findTestOutput(in node: [String: Any]) { if let testDetails = node["testDetails"] as? [String: Any], - let testName = testDetails["testName"] as? String, let emitted = testDetails["emittedOutput"] as? String { - // Only keep if it looks like a failure - if emitted.contains("failed") || emitted.contains("issue") { - // Trim to just the useful parts (skip "Test started" boilerplate) - let lines = emitted.split(separator: "\n", omittingEmptySubsequences: false) - let useful = lines.filter { !$0.hasPrefix("◇ Test") && !$0.isEmpty } - if !useful.isEmpty { - consoleByTest[testName] = useful.joined(separator: "\n") - } + let key = + (testDetails["testIdentifier"] as? String) ?? (testDetails["testIdentifierString"] as? String) + ?? (testDetails["testName"] as? String) + // Drop Swift Testing's own progress lines; keep everything the test printed. + let useful = emitted.split(separator: "\n", omittingEmptySubsequences: true) + .filter { !$0.hasPrefix("◇ Test") && !$0.hasPrefix("↳") } + if let key, !useful.isEmpty { + consoleByTest[key] = useful.suffix(consoleTailLines).joined(separator: "\n") } } - if let subsections = node["subsections"] as? [[String: Any]] { - for sub in subsections { - findTestOutput(in: sub) - } + for sub in subsections { findTestOutput(in: sub) } } } @@ -1607,6 +1664,32 @@ public enum TestTools { return consoleByTest } + /// The console entry for `testIdentifier`: by ID first, then by bare test name. + static func console(for testIdentifier: String, in consoleByTest: [String: String]) -> String? { + if let exact = consoleByTest[testIdentifier] { return exact } + if let match = consoleByTest.first(where: { TestIDs.same($0.key, testIdentifier) }) { + return match.value + } + let name = TestIDs.components(testIdentifier).last ?? testIdentifier + return consoleByTest.first { TestIDs.key($0.key) == TestIDs.key(name) }?.value + } + + /// Put each exported attachment and console tail on the failure it belongs to. + static func attach( + attachments: [(test: String, path: String)], console consoleByTest: [String: String], + to failures: [TestFailureObservation] + ) -> [TestFailureObservation] { + failures.map { failure in + var updated = failure + let paths = attachments.filter { TestIDs.same($0.test, failure.testIdentifier) }.map(\.path) + if !paths.isEmpty { updated.attachments = paths } + if !consoleByTest.isEmpty { + updated.console = console(for: failure.testIdentifier, in: consoleByTest) + } + return updated + } + } + /// Parse coverage report via xccov /// Coverage report JSON. `onlyTargets` skips per-file data, which is fast and small; use /// it when only a yes/no or per-target numbers are needed. A full report too large to @@ -1641,297 +1724,170 @@ public enum TestTools { // MARK: - Public Execution Methods - public static func executeTest( - project: String? = nil, - scheme: String? = nil, - simulator: String? = nil, - configuration: String = "Debug", - testplan: String? = nil, - filter: String? = nil, - filterIDs: [String]? = nil, - coverage: Bool = false, - long: Bool = false, - diagnose: Bool = false, - simRecovery: SimRecoveryMode = .off, - gate: Bool = false, - forMode: OutputAudience = .human, - isolatedSimulator: Bool = false, - env: Environment = .live - ) async throws -> TestExecution { - if isolatedSimulator { - let source = try await env.session.resolveSimulator(simulator) - return try await IsolatedSimulator.with(source: source, env: env) { udid in - try await executeTest( - project: project, scheme: scheme, simulator: udid, configuration: configuration, - testplan: testplan, filter: filter, filterIDs: filterIDs, coverage: coverage, - long: long, diagnose: diagnose, simRecovery: simRecovery, gate: gate, - forMode: forMode, isolatedSimulator: false, env: env) - } - } - let resolvedProject = try await env.session.resolveProject(project) - let resolvedScheme = try await env.session.resolveScheme(scheme, project: resolvedProject) - let resolvedSimulator = try await env.session.resolveSimulator(simulator) - let resolvedFilter: String? - if let ids = filterIDs, !ids.isEmpty { - // Pre-split list path: avoid `splitFilterList` rejoin/re-split which mangles IDs - // containing commas inside parens (e.g. parameterized Swift Testing arguments). - // Rejoin with comma so existing downstream consumers see one filter string; - // `splitFilterList` is paren-aware so the round-trip is lossless. - var resolved: [String] = [] - for id in ids { - resolved.append(await resolveFilter(id, project: resolvedProject, env: env)) - } - resolvedFilter = resolved.joined(separator: ",") - } else if let filter { - resolvedFilter = await resolveFilter(filter, project: resolvedProject, env: env) - } else { - resolvedFilter = nil - } - - // Resolve simulator to UDID for precise id= destination - let udidForDest: String - if let resolved = try? await AutoDetect.resolveSimulatorNameAndUDID(resolvedSimulator) { - udidForDest = resolved.udid - } else { - udidForDest = resolvedSimulator - } - - // Run sim recovery if requested - if simRecovery != .off { - let outcome = await SimulatorRecovery.probeAndRecover( - udid: udidForDest, mode: simRecovery, env: env) - if outcome.fired { - Log.warn("Simulator recovery fired: \(outcome.reason ?? "unknown")") - if let failure = outcome.failureReason { - Log.warn("Simulator recovery failed: \(failure)") - } - } - } + /// Options that change how the test step runs. They are passed to + /// `test-without-building` only, never to `build-for-testing`. + public struct TestRunOptions: Sendable, Equatable { + /// Rerun a failing test up to this many more times; one that then passes is reported as flaky. + public var retries: Int? + /// Run every test this many times. + public var iterations: Int? + /// Repeat until a test fails (capped by `iterations` when given). + public var untilFailure: Bool + /// Force parallel testing on or off. Nil keeps the scheme's or test plan's setting. + public var parallel: Bool? + /// Per-test time allowance in seconds (XCTest; Swift Testing uses `.timeLimit`). + public var testTimeoutSeconds: Int? - let buildResultPath = xcresultPath(prefix: "build-for-testing") - let testResultPath = xcresultPath(prefix: "test") - let destination = await AutoDetect.buildDestination(udidForDest) - - // Phase 1: build-for-testing - let start = CFAbsoluteTimeGetCurrent() - let (buildShellResult, _, buildDiagResult) = try await runBuildForTesting( - project: resolvedProject, scheme: resolvedScheme, destination: destination, - configuration: configuration, coverage: coverage, resultPath: buildResultPath, - long: long, diagnose: diagnose, udid: udidForDest, env: env - ) - - // If build failed (or was killed by a timeout), surface it as a test failure rather than - // running test-without-building against stale or missing products. - if !buildShellResult.succeeded { - let elapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - start) - var buildDiagnostics: [BuildIssueObservation]? - if let buildJSON = await parseBuildResults(buildResultPath, env: env), - let data = buildJSON.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] - { - let parsed = parseBuildIssues(json) - if !parsed.issues.isEmpty { buildDiagnostics = parsed.issues } - } - if buildDiagnostics == nil { - buildDiagnostics = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(buildShellResult)) - if buildDiagnostics?.isEmpty == true { buildDiagnostics = nil } - } - var failures: [TestFailureObservation] - if let diag = buildDiagnostics { - failures = diag.filter { $0.severity == .error }.map { - TestFailureObservation( - testName: "xcodebuild", - testIdentifier: "xcodebuild", - message: $0.message, - source: "build-for-testing.stderr" - ) - } - } else { - let tail = String(buildShellResult.stderr.suffix(2000)).trimmingCharacters( - in: .whitespacesAndNewlines) - failures = - tail.isEmpty || buildShellResult.exitCode == -1 - ? [] - : [ - TestFailureObservation( - testName: "xcodebuild", - testIdentifier: "xcodebuild", - message: tail, - source: "build-for-testing.stderr" - ) - ] - } - if let explanation = Xcodebuild.timeoutExplanation(buildShellResult) { - failures.insert( - TestFailureObservation( - testName: "xcodebuild", testIdentifier: "xcodebuild", message: explanation, - source: "build-for-testing.timeout"), - at: 0) - } - return TestExecution( - succeeded: false, - elapsed: elapsed, - xcresultPath: buildResultPath, - scheme: resolvedScheme, - simulator: resolvedSimulator, - totalTestCount: max(failures.count, 1), - passedTestCount: 0, - failedTestCount: failures.count, - skippedTestCount: 0, - expectedFailureCount: 0, - failures: failures, - deviceName: nil, - osVersion: nil, - screenshotPaths: [], - hasStructuredSummary: false, - buildFailed: true, - buildDiagnostics: buildDiagnostics, - hangDiagnosticPath: buildDiagResult?.filePath, - hangDiagnosticSummary: buildDiagResult?.summaryLine - ) - } - - // Phase 2: test-without-building - let (testShellResult, path, diagResult) = try await runTestWithoutBuilding( - project: resolvedProject, scheme: resolvedScheme, destination: destination, - configuration: configuration, testplan: testplan, - filter: resolvedFilter, coverage: coverage, resultPath: testResultPath, - long: long, diagnose: diagnose, udid: udidForDest, - env: env - ) - let elapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - start) - - // Parse xcresult summary - var parsedSummary: ParsedTestSummary? - if let summaryJSON = await parseTestSummary(path, env: env), - let data = summaryJSON.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] - { - parsedSummary = Self.parseTestSummary(json) - } - - // Parse failures from test details - var failures: [TestFailureObservation] = [] - if let detailsJSON = await parseTestDetails(path, env: env), - let data = detailsJSON.data(using: .utf8), - let parsedFailures = parseTestFailures(data) - { - failures = parsedFailures - } - if failures.isEmpty { - failures = parsedSummary?.failures ?? [] - } - - // Export failure screenshots - let hasFailures = (parsedSummary?.failedTestCount ?? failures.count) > 0 - var screenshots: [ScreenshotAttachment] = [] - if hasFailures { - let attachments = await exportFailureAttachments(path, env: env) - screenshots = attachments.map { ScreenshotAttachment(testName: $0.test, path: $0.path) } - } - - // Parse build diagnostics from xcresult when the test run failed - var buildDiagnostics: [BuildIssueObservation]? - let buildFailed = !testShellResult.succeeded - if buildFailed { - if let buildJSON = await parseBuildResults(path, env: env), - let data = buildJSON.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] - { - let parsed = parseBuildIssues(json) - if !parsed.issues.isEmpty { - buildDiagnostics = parsed.issues - } - } - if buildDiagnostics == nil { - buildDiagnostics = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(testShellResult)) - if buildDiagnostics?.isEmpty == true { buildDiagnostics = nil } - } + public init( + retries: Int? = nil, iterations: Int? = nil, untilFailure: Bool = false, parallel: Bool? = nil, + testTimeoutSeconds: Int? = nil + ) { + self.retries = retries + self.iterations = iterations + self.untilFailure = untilFailure + self.parallel = parallel + self.testTimeoutSeconds = testTimeoutSeconds } - // Fallback error extraction for test failures list - if buildFailed && failures.isEmpty { - let errorLines = testShellResult.stderr.split(separator: "\n") - .filter { $0.contains(": error:") || $0.contains(" failed") || $0.contains("FAILED") } - .prefix(20) - if errorLines.isEmpty { - let tail = String(testShellResult.stderr.suffix(2000)).trimmingCharacters( - in: .whitespacesAndNewlines) - if !tail.isEmpty { - failures = [ - TestFailureObservation( - testName: "xcodebuild", - testIdentifier: "xcodebuild", - message: tail, - source: "stderr" - ) - ] - } - } else { - failures = errorLines.map { - TestFailureObservation( - testName: "xcodebuild", - testIdentifier: "xcodebuild", - message: String($0), - source: "stderr" - ) - } - } + public struct InvalidError: Error, CustomStringConvertible { + public let description: String } - let totalTestCount = - parsedSummary?.totalTestCount ?? max(failures.count, testShellResult.succeeded ? 0 : 1) - let failedTestCount = parsedSummary?.failedTestCount ?? failures.count - let passedTestCount = parsedSummary?.passedTestCount ?? 0 - let skippedTestCount = parsedSummary?.skippedTestCount ?? 0 - let expectedFailureCount = parsedSummary?.expectedFailureCount ?? 0 - - // Filter matched nothing → treat as failure so agents don't assume tests passed - // Zero tests ran (filter or test plan selected nothing) → failure, so a run that tested - // nothing never reads as a pass. Without a parsed summary the count is unknown, not zero. - let zeroMatchWithFilter = totalTestCount == 0 && (filter != nil || parsedSummary != nil) - - let cwd = env.currentDirectoryPath() - let repoRoot = RepoRoot.discover(from: cwd) ?? cwd - let gating = applyGating( - gate: gate, failures: failures, failedTestCount: failedTestCount, repoRoot: repoRoot) - let rawSucceeded = testShellResult.succeeded && failedTestCount == 0 && !zeroMatchWithFilter - let gatedSucceeded = gate ? (rawSucceeded || gating.allKnown) : rawSucceeded - - if gate, let warning = gating.warning, forMode == .human { - FileHandle.standardError.write(Data("xcforge: \(warning)\n".utf8)) + /// The xcodebuild arguments for these options. + public func arguments() throws -> [String] { + var args: [String] = [] + if let retries, retries > 0 { + if iterations != nil || untilFailure { + throw InvalidError(description: "retries can't be combined with iterations or untilFailure") + } + args += ["-retry-tests-on-failure", "-test-iterations", String(retries + 1)] + } + if untilFailure { args.append("-run-tests-until-failure") } + if let iterations { + guard iterations > 0 else { throw InvalidError(description: "iterations must be at least 1") } + if iterations > 1 { args += ["-test-iterations", String(iterations)] } + } + if let parallel { args += ["-parallel-testing-enabled", parallel ? "YES" : "NO"] } + if let testTimeoutSeconds { + guard testTimeoutSeconds > 0 else { + throw InvalidError(description: "testTimeoutSeconds must be positive") + } + args += [ + "-test-timeouts-enabled", "YES", + "-default-test-execution-time-allowance", String(testTimeoutSeconds), + "-maximum-test-execution-time-allowance", String(testTimeoutSeconds), + ] + } + return args } + } - persistLastFailures( - failures: failures, - succeeded: rawSucceeded, - scheme: resolvedScheme, - simulator: resolvedSimulator, - repoRoot: repoRoot, - gateAllKnown: gate && gating.allKnown, - forMode: forMode) + /// Build for testing, then run the tests: the one pipeline behind `test_sim`, `test run`, + /// `test rerun-failed` and `build_and_test`. A build failure comes back as a failed run with + /// `buildFailed` set. + public static func executeTest( + project: String? = nil, + scheme: String? = nil, + simulator: String? = nil, + configuration: String = "Debug", + testplan: String? = nil, + filter: String? = nil, + filterIDs: [String]? = nil, + coverage: Bool = false, + long: Bool = false, + diagnose: Bool = false, + simRecovery: SimRecoveryMode = .off, + timeoutSeconds: TimeInterval? = nil, + envEntries: [String] = [], + gate: Bool = false, + forMode: OutputAudience = .human, + isolatedSimulator: Bool = false, + skipBuild: Bool = false, + testOptions: TestRunOptions = TestRunOptions(), + includeConsole: Bool = false, + env: Environment = .live + ) async throws -> TestExecution { + let result = try await executeBuildAndTest( + project: project, scheme: scheme, simulator: simulator, configuration: configuration, + testplan: testplan, filter: filter, filterIDs: filterIDs, coverage: coverage, long: long, + diagnose: diagnose, simRecovery: simRecovery, timeoutSeconds: timeoutSeconds, + envEntries: envEntries, gate: gate, forMode: forMode, isolatedSimulator: isolatedSimulator, + skipBuild: skipBuild, testOptions: testOptions, includeConsole: includeConsole, env: env) + if let test = result.testResult { return test } + return testExecution(fromBuildFailure: result) + } + /// True when the last build-for-testing of this project used the same scheme, configuration + /// and platform and no file in its repo changed since, so its products can be tested as they are. + public static func lastTestBuildIsCurrent( + project: String?, scheme: String?, simulator: String?, configuration: String, coverage: Bool = false, + env: Environment = .live + ) async -> Bool { + guard let resolvedProject = try? await env.session.resolveProject(project), + let resolvedScheme = try? await env.session.resolveScheme(scheme, project: resolvedProject) + else { return false } + var destination = (try? await env.session.resolveSimulator(simulator)) ?? simulator ?? "" + if let resolved = try? await AutoDetect.resolveSimulatorNameAndUDID(destination) { + destination = resolved.udid + } + let key = LastResultStore.testBuildKey( + scheme: resolvedScheme, configuration: configuration, coverage: coverage, + physicalDevice: AutoDetect.isPhysicalDeviceUDID(destination)) + let projectDirectory = (resolvedProject as NSString).deletingLastPathComponent + let root = RepoRoot.discover(from: projectDirectory) ?? projectDirectory + return LastResultStore.testBuildIsCurrent(project: resolvedProject, key: key, sourceRoot: root) + } + + /// A failed `TestExecution` describing a build-for-testing failure. + static func testExecution(fromBuildFailure result: BuildAndTestResult) -> TestExecution { + var failures: [TestFailureObservation] = (result.buildDiagnostics ?? []) + .filter { $0.severity == .error } + .map { issue in + TestFailureObservation( + testName: "xcodebuild", + testIdentifier: "xcodebuild", + message: issue.message, + source: "build-for-testing.stderr", + messages: [ + FailureMessage(text: issue.message, file: issue.location?.filePath, line: issue.location?.line) + ] + ) + } + if let detail = result.buildFailureDetail, !detail.isEmpty { + if result.xcforgeTimedOut { + failures.insert( + TestFailureObservation( + testName: "xcodebuild", testIdentifier: "xcodebuild", message: detail, + source: "build-for-testing.timeout"), + at: 0) + } else if failures.isEmpty { + failures.append( + TestFailureObservation( + testName: "xcodebuild", testIdentifier: "xcodebuild", message: detail, + source: "build-for-testing.stderr")) + } + } return TestExecution( - succeeded: gatedSucceeded, - elapsed: elapsed, - xcresultPath: path, - scheme: resolvedScheme, - simulator: resolvedSimulator, - totalTestCount: totalTestCount, - passedTestCount: passedTestCount, - failedTestCount: failedTestCount, - skippedTestCount: skippedTestCount, - expectedFailureCount: expectedFailureCount, + succeeded: false, + elapsed: result.buildElapsed, + xcresultPath: result.buildXcresultPath ?? "", + scheme: result.scheme ?? "", + simulator: result.simulator ?? "", + totalTestCount: max(failures.count, 1), + passedTestCount: 0, + failedTestCount: failures.count, + skippedTestCount: 0, + expectedFailureCount: 0, failures: failures, - deviceName: parsedSummary?.destinationDeviceName, - osVersion: parsedSummary?.destinationOSVersion, - screenshotPaths: screenshots, - hasStructuredSummary: parsedSummary != nil, - buildFailed: buildFailed, - buildDiagnostics: buildDiagnostics, - hangDiagnosticPath: diagResult?.filePath, - hangDiagnosticSummary: diagResult?.summaryLine, - knownFailures: (gate && !gating.matchedIDs.isEmpty) ? gating.matchedIDs : nil + deviceName: nil, + osVersion: nil, + screenshotPaths: [], + hasStructuredSummary: false, + buildFailed: true, + buildDiagnostics: result.buildDiagnostics, + hangDiagnosticPath: result.hangDiagnosticPath, + hangDiagnosticSummary: result.hangDiagnosticSummary, + xcforgeTimedOut: result.xcforgeTimedOut ) } @@ -1951,8 +1907,10 @@ public enum TestTools { guard gate else { return GatingOutcome(matchedIDs: [], allKnown: false, warning: nil) } let loaded = KnownFailuresStore.load(repoRoot: repoRoot) let testFailures = failures.filter { $0.testIdentifier != "xcodebuild" } - let matched = testFailures.map { $0.testIdentifier }.filter { loaded.ids.contains($0) } - let unmatched = testFailures.contains { !loaded.ids.contains($0.testIdentifier) } + // Registry entries may be written with or without the target; compare by test, not by string. + func known(_ id: String) -> Bool { loaded.ids.contains { TestIDs.same($0, id) } } + let matched = testFailures.map { $0.testIdentifier }.filter(known) + let unmatched = testFailures.contains { !known($0.testIdentifier) } // `allKnown` requires: every observed test-level failure is in the registry AND // the parsed failure list covers the reported failure count. The second clause // prevents falsely rescuing a run whose xcresult parse dropped some failures @@ -1973,11 +1931,12 @@ public enum TestTools { simulator: String, repoRoot: String, gateAllKnown: Bool = false, - forMode: OutputAudience = .human + forMode: OutputAudience = .human, + run: LastFailuresStore.RunSettings? = nil ) { let testFailureIDs = failures - .filter { $0.testIdentifier != "xcodebuild" } + .filter { $0.testIdentifier != "xcodebuild" && $0.testIdentifier != "test_infrastructure" } .map { $0.testIdentifier } // When gate is on and every observed failure is in the registry, treat the // run as green for persistence purposes — otherwise `rerun-failed` would @@ -1990,13 +1949,22 @@ public enum TestTools { LastFailuresStore.clear(at: repoRoot) return } + var wrote = true if !testFailureIDs.isEmpty { - let wrote = LastFailuresStore.write( - failures: testFailureIDs, scheme: scheme, simulator: simulator, at: repoRoot) - if !wrote && forMode == .human { - FileHandle.standardError.write( - Data("xcforge: failed to write .xcforge/last-failures.json\n".utf8)) - } + wrote = LastFailuresStore.write( + failures: testFailureIDs, scheme: scheme, simulator: simulator, run: run, at: repoRoot) + } else { + // The run failed before any test reported (build error, runner crash, timeout). Record + // that, so `rerun-failed` refuses instead of replaying an older run's failures. + let reason = + failures.first.map { AgentResultProjection.firstLine($0.message) } + ?? "the run failed before any test reported a result" + wrote = LastFailuresStore.write( + failures: [], scheme: scheme, simulator: simulator, run: run, infraFailure: reason, at: repoRoot) + } + if !wrote && forMode == .human { + FileHandle.standardError.write( + Data("xcforge: failed to write .xcforge/last-failures.json\n".utf8)) } } @@ -2043,7 +2011,8 @@ public enum TestTools { let screenshots = attachments.map { ScreenshotAttachment(testName: $0.test, path: $0.path) } let consoleByTest = (includeConsole && !failures.isEmpty) - ? await extractFailedTestConsole(resolvedPath, env: env) : [:] + ? await extractTestConsole(resolvedPath, env: env) : [:] + failures = attach(attachments: attachments, console: consoleByTest, to: failures) return TestFailuresResult( failures: failures, @@ -2208,6 +2177,7 @@ public enum TestTools { configuration: String = "Debug", testplan: String? = nil, filter: String? = nil, + filterIDs: [String]? = nil, coverage: Bool = false, long: Bool = false, diagnose: Bool = false, @@ -2217,6 +2187,9 @@ public enum TestTools { gate: Bool = false, forMode: OutputAudience = .human, isolatedSimulator: Bool = false, + skipBuild: Bool = false, + testOptions: TestRunOptions = TestRunOptions(), + includeConsole: Bool = false, env: Environment = .live ) async throws -> BuildAndTestResult { if isolatedSimulator { @@ -2224,18 +2197,27 @@ public enum TestTools { return try await IsolatedSimulator.with(source: source, env: env) { udid in try await executeBuildAndTest( project: project, scheme: scheme, simulator: udid, configuration: configuration, - testplan: testplan, filter: filter, coverage: coverage, long: long, diagnose: diagnose, - simRecovery: simRecovery, timeoutSeconds: timeoutSeconds, envEntries: envEntries, - gate: gate, forMode: forMode, isolatedSimulator: false, env: env) + testplan: testplan, filter: filter, filterIDs: filterIDs, coverage: coverage, long: long, + diagnose: diagnose, simRecovery: simRecovery, timeoutSeconds: timeoutSeconds, + envEntries: envEntries, gate: gate, forMode: forMode, isolatedSimulator: false, + skipBuild: skipBuild, testOptions: testOptions, includeConsole: includeConsole, env: env) } } - let cwd = env.currentDirectoryPath() - let repoRoot = AutoDetect.repoRoot(from: cwd) ?? cwd - let childEnvironment = try buildTestRunnerEnvironment( - userEntries: envEntries, repoRoot: repoRoot) + let testArguments = try testOptions.arguments() let resolvedProject = try await env.session.resolveProject(project) let resolvedScheme = try await env.session.resolveScheme(scheme, project: resolvedProject) let resolvedSimulator = try await env.session.resolveSimulator(simulator) + // Last failures and the known-failures registry live in the repo of the project under + // test, which is not necessarily the one the server was started in. + let cwd = env.currentDirectoryPath() + let projectDirectory = (resolvedProject as NSString).deletingLastPathComponent + let repoRoot = + RepoRoot.discover(from: projectDirectory) ?? AutoDetect.repoRoot(from: cwd) ?? cwd + let childEnvironment = try buildTestRunnerEnvironment( + userEntries: envEntries, repoRoot: repoRoot) + let runSettings = LastFailuresStore.RunSettings( + project: resolvedProject, testPlan: testplan, configuration: configuration, + env: envEntries.isEmpty ? nil : envEntries) // Resolve simulator to UDID for precise id= destination let udidForDest: String @@ -2266,14 +2248,23 @@ public enum TestTools { } } + // A pre-split list (rerun-failed) is kept as is, so IDs with commas inside their + // arguments are never re-split. + var requestedIDs = filterIDs ?? [] + if requestedIDs.isEmpty, let filter { requestedIDs = splitFilterList(filter) } let resolvedFilter: String? - if let filter { - resolvedFilter = await resolveFilter(filter, project: resolvedProject, env: env) + if !requestedIDs.isEmpty { + let resolved = await resolveFilters( + requestedIDs, project: resolvedProject, scheme: resolvedScheme, testplan: testplan, env: env) + resolvedFilter = resolved.joined(separator: ",") } else { resolvedFilter = nil } let destination = await AutoDetect.buildDestination(udidForDest) + let testBuildKey = LastResultStore.testBuildKey( + scheme: resolvedScheme, configuration: configuration, coverage: coverage, + physicalDevice: AutoDetect.isPhysicalDeviceUDID(udidForDest)) // --- Inner helper: run the build-for-testing phase --- func runBuildPhase(diagnose: Bool) async throws -> ( @@ -2299,74 +2290,91 @@ public enum TestTools { configuration: configuration, testplan: testplan, filter: resolvedFilter, coverage: coverage, resultPath: resultPath, long: long, diagnose: diagnose, udid: udidForDest, timeoutOverride: timeoutSeconds, - childEnvironment: childEnvironment, + childEnvironment: childEnvironment, extraTestArguments: testArguments, env: env ) } // --- Build phase --- - var buildStart = CFAbsoluteTimeGetCurrent() - var (buildShellResult, buildResultPath, buildDiagResult) = try await runBuildPhase( - diagnose: diagnose) - var buildElapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - buildStart) - - // Auto-retry on deadlock: only when watchdog fired (diagResult != nil) AND shell timeout (-1) - if buildDiagResult != nil && buildShellResult.exitCode == -1 && recoveryAttempts < 1 { - let buildVerdict = buildDiagResult.map { - DiagnosticSnapshot.classifyVerdict(snapshotPath: $0.filePath) + var buildElapsed = "0.0" + if !skipBuild { + var buildStart = CFAbsoluteTimeGetCurrent() + var (buildShellResult, buildResultPath, buildDiagResult) = try await runBuildPhase( + diagnose: diagnose) + buildElapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - buildStart) + + // Auto-retry on deadlock: only when watchdog fired (diagResult != nil) AND shell timeout (-1) + if buildDiagResult != nil && buildShellResult.exitCode == -1 && recoveryAttempts < 1 { + let buildVerdict = buildDiagResult.map { + DiagnosticSnapshot.classifyVerdict(snapshotPath: $0.filePath) + } + let verdict = buildVerdict == .timeout ? nil : buildVerdict + if verdict != nil { + // Kill the stuck xcodebuild tree by PID before retrying + let xcodebuildPid = extractXcodebuildPid(from: buildDiagResult) + await killXcodebuildTree(pid: xcodebuildPid, env: env) + + // Run sim recovery before retry + let retryOutcome = await SimulatorRecovery.probeAndRecover( + udid: udidForDest, mode: simRecovery, env: env) + recoveryAttempts += 1 + recoveryReason = verdict?.rawValue ?? retryOutcome.reason + recoveryFailureReason = retryOutcome.failureReason + if simHealthCheckDetail == nil { simHealthCheckDetail = retryOutcome.healthCheckDetail } + + buildStart = CFAbsoluteTimeGetCurrent() + let retried = try await runBuildPhase(diagnose: true) + buildShellResult = retried.0 + buildResultPath = retried.1 + buildDiagResult = retried.2 + buildElapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - buildStart) + } } - let verdict = buildVerdict == .timeout ? nil : buildVerdict - if verdict != nil { - // Kill the stuck xcodebuild tree by PID before retrying - let xcodebuildPid = extractXcodebuildPid(from: buildDiagResult) - await killXcodebuildTree(pid: xcodebuildPid, env: env) - - // Run sim recovery before retry - let retryOutcome = await SimulatorRecovery.probeAndRecover( - udid: udidForDest, mode: simRecovery, env: env) - recoveryAttempts += 1 - recoveryReason = verdict?.rawValue ?? retryOutcome.reason - recoveryFailureReason = retryOutcome.failureReason - if simHealthCheckDetail == nil { simHealthCheckDetail = retryOutcome.healthCheckDetail } - buildStart = CFAbsoluteTimeGetCurrent() - let retried = try await runBuildPhase(diagnose: true) - buildShellResult = retried.0 - buildResultPath = retried.1 - buildDiagResult = retried.2 - buildElapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - buildStart) - } - } + // Build failure (real, not a timeout hang) + if !buildShellResult.succeeded { + var issues: [BuildIssueObservation] = [] - // Build failure (real, not a timeout hang) - if !buildShellResult.succeeded { - var issues: [BuildIssueObservation] = [] + if let buildJSON = await parseBuildResults(buildResultPath, env: env), + let data = buildJSON.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] + { + let parsed = parseBuildIssues(json) + issues = parsed.issues + } - if let buildJSON = await parseBuildResults(buildResultPath, env: env), - let data = buildJSON.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] - { - let parsed = parseBuildIssues(json) - issues = parsed.issues - } + if issues.isEmpty { + issues = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(buildShellResult)) + } - if issues.isEmpty { - issues = fallbackBuildIssues(stderr: Xcodebuild.combinedOutput(buildShellResult)) + let tail = String(buildShellResult.stderr.suffix(2000)).trimmingCharacters( + in: .whitespacesAndNewlines) + let detail = Xcodebuild.timeoutExplanation(buildShellResult) ?? (tail.isEmpty ? nil : tail) + persistLastFailures( + failures: [], succeeded: false, scheme: resolvedScheme, simulator: resolvedSimulator, + repoRoot: repoRoot, forMode: forMode, run: runSettings) + + return BuildAndTestResult( + phase: "build", + buildSucceeded: false, + buildElapsed: buildElapsed, + buildDiagnostics: issues.isEmpty ? nil : issues, + testResult: nil, + hangDiagnosticPath: buildDiagResult?.filePath, + recoveryAttempts: recoveryAttempts, + recoveryReason: recoveryReason, + recoveryFailureReason: recoveryFailureReason, + simHealthCheckDetail: simHealthCheckDetail, + xcforgeTimedOut: buildShellResult.exitCode == -1, + project: resolvedProject, + scheme: resolvedScheme, + simulator: resolvedSimulator, + buildXcresultPath: buildResultPath, + hangDiagnosticSummary: buildDiagResult?.summaryLine, + buildFailureDetail: detail + ) } - - return BuildAndTestResult( - phase: "build", - buildSucceeded: false, - buildElapsed: buildElapsed, - buildDiagnostics: issues.isEmpty ? nil : issues, - testResult: nil, - hangDiagnosticPath: buildDiagResult?.filePath, - recoveryAttempts: recoveryAttempts, - recoveryReason: recoveryReason, - recoveryFailureReason: recoveryFailureReason, - simHealthCheckDetail: simHealthCheckDetail, - xcforgeTimedOut: buildShellResult.exitCode == -1 - ) + LastResultStore.recordTestBuild(project: resolvedProject, key: testBuildKey) } // --- Test phase --- @@ -2381,6 +2389,15 @@ public enum TestTools { testResultPath = result.1 testDiagResult = result.2 } catch { + let failure = TestFailureObservation( + testName: "test_infrastructure", + testIdentifier: "test_infrastructure", + message: "Test execution failed: \(error)", + source: "xcforge" + ) + persistLastFailures( + failures: [failure], succeeded: false, scheme: resolvedScheme, simulator: resolvedSimulator, + repoRoot: repoRoot, forMode: forMode, run: runSettings) return BuildAndTestResult( phase: "test", buildSucceeded: true, @@ -2397,14 +2414,7 @@ public enum TestTools { failedTestCount: 0, skippedTestCount: 0, expectedFailureCount: 0, - failures: [ - TestFailureObservation( - testName: "test_infrastructure", - testIdentifier: "test_infrastructure", - message: "Test execution failed: \(error)", - source: "xcforge" - ) - ], + failures: [failure], deviceName: nil, osVersion: nil, screenshotPaths: [], @@ -2416,7 +2426,10 @@ public enum TestTools { recoveryAttempts: recoveryAttempts, recoveryReason: recoveryReason, recoveryFailureReason: recoveryFailureReason, - simHealthCheckDetail: simHealthCheckDetail + simHealthCheckDetail: simHealthCheckDetail, + project: resolvedProject, + scheme: resolvedScheme, + simulator: resolvedSimulator ) } @@ -2448,73 +2461,39 @@ public enum TestTools { let testElapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - testStart) - do { - let testExecution = try await buildTestExecutionResult( - shellResult: testShellResult, - resultPath: testResultPath, - diagResult: testDiagResult, - resolvedScheme: resolvedScheme, - resolvedSimulator: resolvedSimulator, - elapsed: testElapsed, - filter: filter, - gate: gate, - forMode: forMode, - env: env - ) - return BuildAndTestResult( - phase: "test", - buildSucceeded: true, - buildElapsed: buildElapsed, - buildDiagnostics: nil, - testResult: testExecution, - hangDiagnosticPath: testExecution.hangDiagnosticPath, - recoveryAttempts: recoveryAttempts, - recoveryReason: recoveryReason, - recoveryFailureReason: recoveryFailureReason, - simHealthCheckDetail: simHealthCheckDetail, - knownFailures: testExecution.knownFailures - ) - } catch { - return BuildAndTestResult( - phase: "test", - buildSucceeded: true, - buildElapsed: buildElapsed, - buildDiagnostics: nil, - testResult: TestExecution( - succeeded: false, - elapsed: testElapsed, - xcresultPath: testResultPath, - scheme: resolvedScheme, - simulator: resolvedSimulator, - totalTestCount: 0, - passedTestCount: 0, - failedTestCount: 0, - skippedTestCount: 0, - expectedFailureCount: 0, - failures: [ - TestFailureObservation( - testName: "test_infrastructure", - testIdentifier: "test_infrastructure", - message: "Test result parsing failed: \(error)", - source: "xcforge" - ) - ], - deviceName: nil, - osVersion: nil, - screenshotPaths: [], - hasStructuredSummary: false, - buildFailed: false, - buildDiagnostics: nil, - hangDiagnosticPath: testDiagResult?.filePath, - hangDiagnosticSummary: testDiagResult?.summaryLine - ), - hangDiagnosticPath: testDiagResult?.filePath, - recoveryAttempts: recoveryAttempts, - recoveryReason: recoveryReason, - recoveryFailureReason: recoveryFailureReason, - simHealthCheckDetail: simHealthCheckDetail - ) - } + let testExecution = await buildTestExecutionResult( + shellResult: testShellResult, + resultPath: testResultPath, + diagResult: testDiagResult, + resolvedScheme: resolvedScheme, + resolvedSimulator: resolvedSimulator, + elapsed: testElapsed, + filterRequested: resolvedFilter != nil, + gate: gate, + forMode: forMode, + repoRoot: repoRoot, + runSettings: runSettings, + skippedBuild: skipBuild, + includeConsole: includeConsole, + env: env + ) + return BuildAndTestResult( + phase: "test", + buildSucceeded: true, + buildElapsed: buildElapsed, + buildDiagnostics: nil, + testResult: testExecution, + hangDiagnosticPath: testExecution.hangDiagnosticPath, + recoveryAttempts: recoveryAttempts, + recoveryReason: recoveryReason, + recoveryFailureReason: recoveryFailureReason, + simHealthCheckDetail: simHealthCheckDetail, + knownFailures: testExecution.knownFailures, + project: resolvedProject, + scheme: resolvedScheme, + simulator: resolvedSimulator, + skippedBuild: skipBuild + ) } /// Extract the xcodebuild PID from a diagnostic snapshot result (reads the snapshot file). @@ -2579,11 +2558,15 @@ public enum TestTools { resolvedScheme: String, resolvedSimulator: String, elapsed: String, - filter: String?, + filterRequested: Bool, gate: Bool = false, forMode: OutputAudience = .human, + repoRoot: String, + runSettings: LastFailuresStore.RunSettings? = nil, + skippedBuild: Bool = false, + includeConsole: Bool = false, env: Environment - ) async throws -> TestExecution { + ) async -> TestExecution { let xcforgeTimedOut = shellResult.exitCode == -1 var parsedSummary: ParsedTestSummary? @@ -2595,22 +2578,20 @@ public enum TestTools { parsedSummary = Self.parseTestSummary(json) } else if !xcforgeTimedOut { // Parse failed and this wasn't a timeout kill — surface the exit code for caller diagnosis - let exitDesc = xcforgeTimedOut ? "-1 (timed out)" : "\(shellResult.exitCode)" xcresultParseError = - "xcresulttool failed to parse \(resultPath) (xcodebuild exit: \(exitDesc))" + "xcresulttool failed to parse \(resultPath) (xcodebuild exit: \(shellResult.exitCode))" } var failures: [TestFailureObservation] = [] var slowestTests: [SlowTest] = [] + var flakyTests: [String] = [] if let detailsJSON = await parseTestDetails(resultPath, env: env), - let data = detailsJSON.data(using: .utf8) + let data = detailsJSON.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] { - if let parsedFailures = parseTestFailures(data) { - failures = parsedFailures - } - if let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] { - slowestTests = parseSlowTests(json) - } + failures = parseTestFailures(json) + slowestTests = parseSlowTests(json) + flakyTests = parseFlakyTests(json) } if failures.isEmpty { failures = parsedSummary?.failures ?? [] @@ -2621,11 +2602,14 @@ public enum TestTools { if hasFailures { let attachments = await exportFailureAttachments(resultPath, env: env) screenshots = attachments.map { ScreenshotAttachment(testName: $0.test, path: $0.path) } + let consoleByTest = includeConsole ? await extractTestConsole(resultPath, env: env) : [:] + failures = attach(attachments: attachments, console: consoleByTest, to: failures) } + // Failing tests also exit non-zero; only a run that produced no test results and + // reports build errors failed to build. var buildDiagnostics: [BuildIssueObservation]? - let buildFailed = !shellResult.succeeded - if buildFailed { + if !shellResult.succeeded && parsedSummary == nil { if let buildJSON = await parseBuildResults(resultPath, env: env), let data = buildJSON.data(using: .utf8), let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] @@ -2638,8 +2622,9 @@ public enum TestTools { if buildDiagnostics?.isEmpty == true { buildDiagnostics = nil } } } + let buildFailed = buildDiagnostics?.contains { $0.severity == .error } ?? false - if buildFailed && failures.isEmpty { + if !shellResult.succeeded && failures.isEmpty { let errorLines = shellResult.stderr.split(separator: "\n") .filter { $0.contains(": error:") || $0.contains(" failed") || $0.contains("FAILED") } .prefix(20) @@ -2666,6 +2651,16 @@ public enum TestTools { ) } } + if skippedBuild && parsedSummary == nil { + failures.insert( + TestFailureObservation( + testName: "xcodebuild", testIdentifier: "xcodebuild", + message: + "The build was skipped and the tests didn't run; the last build's products may be missing." + + " Run again with the build.", + source: "xcforge"), + at: 0) + } } let totalTestCount = @@ -2676,10 +2671,8 @@ public enum TestTools { let expectedFailureCount = parsedSummary?.expectedFailureCount ?? 0 // Zero tests ran (filter or test plan selected nothing) → failure, so a run that tested // nothing never reads as a pass. Without a parsed summary the count is unknown, not zero. - let zeroMatchWithFilter = totalTestCount == 0 && (filter != nil || parsedSummary != nil) + let zeroMatchWithFilter = totalTestCount == 0 && (filterRequested || parsedSummary != nil) - let cwd = env.currentDirectoryPath() - let repoRoot = RepoRoot.discover(from: cwd) ?? cwd let gating = applyGating( gate: gate, failures: failures, failedTestCount: failedTestCount, repoRoot: repoRoot) let rawSucceeded = shellResult.succeeded && failedTestCount == 0 && !zeroMatchWithFilter @@ -2696,7 +2689,8 @@ public enum TestTools { simulator: resolvedSimulator, repoRoot: repoRoot, gateAllKnown: gate && gating.allKnown, - forMode: forMode) + forMode: forMode, + run: runSettings) return TestExecution( succeeded: gatedSucceeded, @@ -2721,7 +2715,8 @@ public enum TestTools { xcresultParseError: xcresultParseError, xcforgeTimedOut: xcforgeTimedOut, slowestTests: slowestTests, - knownFailures: (gate && !gating.matchedIDs.isEmpty) ? gating.matchedIDs : nil + knownFailures: (gate && !gating.matchedIDs.isEmpty) ? gating.matchedIDs : nil, + flakyTests: flakyTests ) } @@ -2916,17 +2911,11 @@ public enum TestTools { let testTargets = try await AutoDetect.testTargets(project: resolvedProject, env: env) if !testTargets.isEmpty { throw TestDiscoveryError( - "No XCTest methods found, but found test targets: \(testTargets.joined(separator: ", ")). " - + "Note: -enumerate-tests only discovers XCTest methods. Swift Testing @Test suites are not " - + "discoverable this way — use test_sim with a testplan to run them. " - + "Use these as filter prefixes, e.g. filter: \"\(testTargets[0])/YourTestClass/testMethodName\"" + "xcodebuild listed no tests, but the project has test targets: \(testTargets.joined(separator: ", ")). " + + "Check that the scheme's test action or test plan includes them." ) } - throw TestDiscoveryError( - "No tests found for scheme '\(resolvedScheme)'. " - + "Note: Swift Testing @Test suites are not discoverable by -enumerate-tests. " - + "Use test_sim with a testplan to run Swift Testing suites." - ) + throw TestDiscoveryError("No tests found for scheme '\(resolvedScheme)'.") } return ListTestsResult( @@ -3015,9 +3004,16 @@ public enum TestTools { long: input.long ?? false, diagnose: input.diagnose ?? false, simRecovery: recoveryMode, + timeoutSeconds: input.timeoutSeconds.map { TimeInterval($0) }, + envEntries: input.env ?? [], gate: input.gate ?? false, forMode: forMode, isolatedSimulator: input.isolatedSimulator ?? false, + skipBuild: input.skipBuild ?? false, + testOptions: TestRunOptions( + retries: input.retries, iterations: input.iterations, untilFailure: input.untilFailure ?? false, + parallel: input.parallel, testTimeoutSeconds: input.testTimeoutSeconds), + includeConsole: input.includeConsole ?? false, env: env ) @@ -3069,17 +3065,10 @@ public enum TestTools { if !execution.failures.isEmpty { lines.append("\nFailures:") - for f in execution.failures { - lines.append(" FAIL: \(f.testName)") - if !f.message.isEmpty { lines.append(" \(f.message)") } - } + lines += TestFailureText.lines(execution.failures) } - - if !execution.screenshotPaths.isEmpty { - lines.append("\nFailure screenshots (\(execution.screenshotPaths.count)):") - for ss in execution.screenshotPaths { - lines.append(" \(ss.path)") - } + if !execution.flakyTests.isEmpty { + lines.append("Flaky (failed, then passed on retry): \(execution.flakyTests.joined(separator: ", "))") } if execution.totalTestCount == 0, let f = input.filter { @@ -3116,47 +3105,14 @@ public enum TestTools { switch ToolInput.decode(TestFailuresInput.self, from: args) { case .failure(let err): return err case .success(let input): - let xcresultPath: String - - if let provided = input.xcresult_path { - xcresultPath = provided - } else { - do { - let project = try await env.session.resolveProject(input.project) - let scheme = try await env.session.resolveScheme(input.scheme, project: project) - let simulator = try await env.session.resolveSimulator(input.simulator) - let destination = await AutoDetect.buildDestination(simulator) - let path = Self.xcresultPath(prefix: "fail") - let (_, p, _) = try await runTests( - project: project, scheme: scheme, destination: destination, - configuration: "Debug", testplan: nil, filter: nil, - coverage: false, resultPath: path, - env: env - ) - xcresultPath = p - } catch { - return .fail("\(error)") - } - } - - let includeConsole = input.include_console ?? false - - // Parse test details and extract failures - guard let detailsJSON = await parseTestDetails(xcresultPath, env: env), - let data = detailsJSON.data(using: .utf8) - else { - return .fail("Failed to parse xcresult at \(xcresultPath)") + do { + let result = try await extractFailures( + xcresultPath: input.xcresult_path, project: input.project, scheme: input.scheme, + simulator: input.simulator, includeConsole: input.include_console ?? false, env: env) + return formatTestFailures(result) + } catch { + return .fail("\(error)") } - - // Export failure screenshots if available - let attachments = await exportFailureAttachments(xcresultPath, env: env) - - // Extract console output per failed test if requested - let consoleByTest = - includeConsole ? await extractFailedTestConsole(xcresultPath, env: env) : [:] - - return formatTestFailures( - data, xcresultPath: xcresultPath, attachments: attachments, consoleByTest: consoleByTest) } } @@ -3376,6 +3332,11 @@ public enum TestTools { gate: input.gate ?? false, forMode: forMode, isolatedSimulator: input.isolatedSimulator ?? false, + skipBuild: input.skipBuild ?? false, + testOptions: TestRunOptions( + retries: input.retries, iterations: input.iterations, untilFailure: input.untilFailure ?? false, + parallel: input.parallel, testTimeoutSeconds: input.testTimeoutSeconds), + includeConsole: input.includeConsole ?? false, env: env ) if forMode == .agent { @@ -3547,7 +3508,8 @@ public enum TestTools { } return .fail(lines.joined(separator: "\n") + suffix) } - lines.append("Build: OK (\(result.buildElapsed)s)") + lines.append( + result.skippedBuild ? "Build: skipped (tested the last build)" : "Build: OK (\(result.buildElapsed)s)") if test.totalTestCount == 0 { lines.append("No tests ran (\(test.elapsed)s): the filter, scheme or test plan selected none") } else { @@ -3560,12 +3522,10 @@ public enum TestTools { if !test.failures.isEmpty { lines.append("") lines.append("Failures:") - for failure in test.failures { - lines.append(" \(failure.testName): \(failure.message)") - if !failure.source.isEmpty && failure.source != "stderr" { - lines.append(" at \(failure.source)") - } - } + lines += TestFailureText.lines(test.failures) + } + if !test.flakyTests.isEmpty { + lines.append("Flaky (failed, then passed on retry): \(test.flakyTests.joined(separator: ", "))") } lines.append("") lines.append("xcresult: \(test.xcresultPath)") @@ -3632,7 +3592,9 @@ public enum TestTools { // Group by target/class for readability var grouped: [String: [String: [String]]] = [:] // target -> class -> methods for test in tests { - grouped[test.target, default: [:]][test.className, default: []].append(test.methodName) + // The last component as xcodebuild names it, `()` included, so it can be pasted back. + let method = TestIDs.components(test.fullIdentifier).last ?? test.methodName + grouped[test.target, default: [:]][test.className, default: []].append(method) } for (target, classes) in grouped.sorted(by: { $0.key < $1.key }) { @@ -3647,12 +3609,8 @@ public enum TestTools { lines.append("") lines.append("Use these identifiers with the filter parameter:") - lines.append(" Full: filter: \"Target/Class/method\"") - lines.append(" Class: filter: \"Class\" (auto-resolves target — XCTest only)") - lines.append(" Method: filter: \"method\" (auto-resolves target — XCTest only)") - lines.append("") - lines.append( - "Note: Swift Testing @Test suites require the full Target/Suite path and a testplan.") + lines.append(" Full: filter: \"Target/Suite/test()\"") + lines.append(" Suite: filter: \"Suite\" or \"Suite/test()\" (the target is added)") return .ok(lines.joined(separator: "\n")) } @@ -3721,66 +3679,37 @@ public enum TestTools { return lines.joined(separator: "\n") } - private static func formatTestFailures( - _ data: Data, xcresultPath: String, - attachments: [(test: String, path: String)] = [], - consoleByTest: [String: String] = [:] - ) -> CallTool.Result { - guard let observedFailures = parseTestFailures(data) else { - return .fail("Failed to parse test details JSON") - } - - if observedFailures.isEmpty { - return .ok("No test failures found.\nxcresult: \(xcresultPath)") - } - - let failures = observedFailures.map { failure -> String in - var failLine = "FAIL: \(failure.testName) [\(failure.testIdentifier)]" - if !failure.message.isEmpty { - failLine += "\n " + failure.message.replacingOccurrences(of: "\n", with: "\n ") - } - - let matchingScreenshots = attachments.filter { - $0.test.contains(failure.testIdentifier) || failure.testIdentifier.contains($0.test) - } - for screenshot in matchingScreenshots { - failLine += "\n Screenshot: \(screenshot.path)" - } - - let funcName = - failure.testIdentifier.split(separator: "/").last.map(String.init) ?? failure.testIdentifier - if let console = consoleByTest[funcName] ?? consoleByTest[failure.testIdentifier] { - failLine += "\n Console:\n " + console.replacingOccurrences(of: "\n", with: "\n ") - } - - return failLine + static func formatTestFailures(_ result: TestFailuresResult) -> CallTool.Result { + if result.failures.isEmpty { + return .ok("No test failures found.\nxcresult: \(result.xcresultPath)") } - - var output = "\(failures.count) test failure(s):\n\n" + failures.joined(separator: "\n\n") - - // List all screenshots at the end for easy access - if !attachments.isEmpty { - output += "\n\nFailure screenshots (\(attachments.count)):" - for att in attachments { - output += "\n \(att.path)" - } + var lines = ["\(result.failures.count) test failure(s):", ""] + lines += TestFailureText.lines(result.failures, indent: "") + // Attachments that couldn't be tied to a failure are still worth a path. + let attached = Set(result.failures.flatMap { $0.attachments ?? [] }) + let loose = result.screenshots.map(\.path).filter { !attached.contains($0) } + if !loose.isEmpty { + lines.append("") + lines.append("Other failure attachments (\(loose.count)):") + lines += loose.prefix(20).map { " \($0)" } } - - output += "\n\nxcresult: \(xcresultPath)" - let truncated = - output.count > 50000 ? String(output.prefix(50000)) + "\n... [truncated]" : output - return .fail(truncated) + lines.append("") + lines.append("xcresult: \(result.xcresultPath)") + return .fail(lines.joined(separator: "\n")) } private static func parseTestSummary(_ json: [String: Any]) -> ParsedTestSummary { let failures = ((json["testFailures"] as? [[String: Any]]) ?? []).map { failure in - TestFailureObservation( + let text = failure["failureText"] as? String + return TestFailureObservation( testName: (failure["testName"] as? String) ?? "?", - testIdentifier: (failure["testIdentifierString"] as? String) - ?? (failure["testName"] as? String) ?? "?", - message: (failure["failureText"] as? String) - ?? "Test failed without a captured failure message.", - source: "xcresult.test-summary" + testIdentifier: TestIDs.canonical( + target: failure["targetName"] as? String, + identifier: (failure["testIdentifierString"] as? String) + ?? (failure["testName"] as? String) ?? "?"), + message: text ?? "Test failed without a captured failure message.", + source: "xcresult.test-summary", + messages: text.map { [FailureMessage.parse($0)] } ) } @@ -3807,34 +3736,34 @@ public enum TestTools { return parseTestFailures(json) } - private static func parseTestFailures(_ json: [String: Any]) -> [TestFailureObservation] { + /// Node types that hold one run of a test: a parameterized argument, a repetition or + /// retry, a device or a test plan configuration. Their names label the messages below them. + static let runNodeTypes: Set = [ + "Arguments", "Repetition", "Device", "Test Plan Configuration", "Test Case Run", + ] + + static func parseTestFailures(_ json: [String: Any]) -> [TestFailureObservation] { var failures: [TestFailureObservation] = [] - func collectFailure(from node: [String: Any]) { + func collectFailure(from node: [String: Any], target: String?) { let nodeType = (node["nodeType"] as? String) ?? "" let result = (node["result"] as? String) ?? "" let name = (node["name"] as? String) ?? "?" let identifier = (node["nodeIdentifier"] as? String) ?? name + var target = target + if nodeType.hasSuffix("test bundle") { target = name } if nodeType == "Test Case" && result == "Failed" { - var messages: [String] = [] - if let children = node["children"] as? [[String: Any]] { - for child in children where (child["nodeType"] as? String) == "Failure Message" { - let message = (child["name"] as? String) ?? "" - if !message.isEmpty { - messages.append(message) - } - } - } - + let messages = failureMessages(in: node, label: nil) failures.append( TestFailureObservation( testName: name, - testIdentifier: identifier, + testIdentifier: TestIDs.canonical(target: target, identifier: identifier), message: messages.isEmpty ? "Test failed without a captured failure message." - : messages.joined(separator: "\n"), - source: "xcresult.test-details" + : messages.map(\.display).joined(separator: "\n"), + source: "xcresult.test-details", + messages: messages.isEmpty ? nil : messages ) ) return @@ -3842,20 +3771,72 @@ public enum TestTools { if let children = node["children"] as? [[String: Any]] { for child in children { - collectFailure(from: child) + collectFailure(from: child, target: target) } } } if let testNodes = json["testNodes"] as? [[String: Any]] { for node in testNodes { - collectFailure(from: node) + collectFailure(from: node, target: nil) } } return failures } + /// Every failure message under a test case, at any depth, labelled with the run it came from. + static func failureMessages(in node: [String: Any], label: String?) -> [FailureMessage] { + var out: [FailureMessage] = [] + for child in (node["children"] as? [[String: Any]]) ?? [] { + let type = (child["nodeType"] as? String) ?? "" + let name = (child["name"] as? String) ?? "" + if type == "Failure Message" { + if !name.isEmpty { out.append(FailureMessage.parse(name, label: label)) } + } else if runNodeTypes.contains(type), !name.isEmpty { + // A run that passed has nothing to report; one that failed labels its messages. + if (child["result"] as? String) == "Passed" { continue } + out += failureMessages(in: child, label: label.map { "\($0) · \(name)" } ?? name) + } else { + out += failureMessages(in: child, label: label) + } + } + return out + } + + /// Tests that passed in the end but failed in at least one retry or repetition. + static func parseFlakyTests(_ json: [String: Any]) -> [String] { + var flaky: [String] = [] + + func failedRun(_ node: [String: Any]) -> Bool { + for child in (node["children"] as? [[String: Any]]) ?? [] { + let type = (child["nodeType"] as? String) ?? "" + if runNodeTypes.contains(type), (child["result"] as? String) == "Failed" { return true } + if failedRun(child) { return true } + } + return false + } + + func walk(_ node: [String: Any], target: String?) { + let nodeType = (node["nodeType"] as? String) ?? "" + let name = (node["name"] as? String) ?? "?" + var target = target + if nodeType.hasSuffix("test bundle") { target = name } + if nodeType == "Test Case" { + let result = (node["result"] as? String) ?? "" + if result != "Failed" && result != "Skipped" && failedRun(node) { + let identifier = (node["nodeIdentifier"] as? String) ?? name + flaky.append(TestIDs.canonical(target: target, identifier: identifier)) + } + return + } + for child in (node["children"] as? [[String: Any]]) ?? [] { walk(child, target: target) } + } + + for node in (json["testNodes"] as? [[String: Any]]) ?? [] { walk(node, target: nil) } + return flaky + } + private static func formatCoverageReport( _ json: [String: Any], minCoverage: Double, xcresultPath: String ) -> CallTool.Result { diff --git a/Tests/XCForgeKitTests/AgentResultProjectionTests.swift b/Tests/XCForgeKitTests/AgentResultProjectionTests.swift index 748b25c..eed75fb 100644 --- a/Tests/XCForgeKitTests/AgentResultProjectionTests.swift +++ b/Tests/XCForgeKitTests/AgentResultProjectionTests.swift @@ -59,10 +59,10 @@ struct AgentResultProjectionTests { @Test("encoded JSON has only the locked agent-shape keys at the top level") func slimKeys() throws { - // Locked allowed set per spec — `knownFailures` is optional so the ceiling is 9. + // `knownFailures`, `flaky` and `reason` appear only when they have something to say. let allowed: Set = [ "succeeded", "buildOk", "total", "passed", "failed", "skipped", - "timedOut", "knownFailures", "failures", + "timedOut", "knownFailures", "failures", "flaky", "reason", ] let exec = makeExecution(knownFailures: ["Suite/k"]) let json = try WorkflowJSONRenderer.renderTestJSON(exec, forAgent: true) diff --git a/Tests/XCForgeKitTests/TestResultsTests.swift b/Tests/XCForgeKitTests/TestResultsTests.swift new file mode 100644 index 0000000..4f43978 --- /dev/null +++ b/Tests/XCForgeKitTests/TestResultsTests.swift @@ -0,0 +1,350 @@ +import Foundation +import Testing + +@testable import XCForgeKit + +@Suite("Test results: IDs, messages, reruns and output", .serialized) +struct TestResultsTests { + + private func json(_ text: String) -> [String: Any] { + (try? JSONSerialization.jsonObject(with: Data(text.utf8)) as? [String: Any]) ?? [:] + } + + private func tempDir() -> String { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("xcforge-results-\(UUID().uuidString)", isDirectory: true).path + try? FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + return dir + } + + // MARK: - One ID format + + @Test("canonical IDs carry the target once, and doubled parens are collapsed") + func canonicalIDs() { + #expect(TestIDs.canonical(target: "AppTests", identifier: "Parser/parses()") == "AppTests/Parser/parses()") + #expect(TestIDs.canonical(target: "AppTests", identifier: "AppTests/Parser/parses()") == "AppTests/Parser/parses()") + #expect(TestIDs.canonical(target: nil, identifier: "Parser/parses()()") == "Parser/parses()") + } + + @Test("same() accepts short forms but never a different test with a common prefix") + func sameTest() { + #expect(TestIDs.same("AppTests/Parser/parses()", "Parser/parses()")) + #expect(TestIDs.same("AppTests/Parser/parses()", "Parser/parses")) + #expect(TestIDs.same("AppTests/Parser/parses()()", "AppTests/Parser/parses()")) + #expect(!TestIDs.same("AppTests/Parser/testFoo", "AppTests/Parser/testFoo2")) + #expect(!TestIDs.same("AppTests/Parser/testFoo", "Foo")) + #expect(!TestIDs.same("", "Foo")) + } + + @Test("the -only-testing spelling doubles () once, only for target-qualified IDs") + func onlyTestingSpelling() { + #expect(TestIDs.onlyTestingArgument("AppTests/Parser/parses()") == "AppTests/Parser/parses()()") + #expect(TestIDs.onlyTestingArgument("AppTests/Parser/parses()()") == "AppTests/Parser/parses()()") + #expect(TestIDs.onlyTestingArgument("Parser/parses()") == "Parser/parses()") + #expect(TestIDs.onlyTestingArgument("AppTests/Parser/testFoo") == "AppTests/Parser/testFoo") + } + + @Test("components ignore slashes inside arguments") + func idComponents() { + #expect(TestIDs.components("T/Suite/test(path: \"a/b\")").count == 3) + #expect(TestIDs.components("T/Suite/test[a/b]").count == 3) + } + + @Test("short filters get the target; nested suites too when the target list is exact") + func qualifyFilters() { + #expect(TestTools.qualify("Parser/parses()", targets: ["AppTests"]) == "AppTests/Parser/parses()") + #expect(TestTools.qualify("AppTests/Parser", targets: ["AppTests"]) == "AppTests/Parser") + #expect(TestTools.qualify("Outer/Inner/test()", targets: ["AppTests"]) == "AppTests/Outer/Inner/test()") + // A guessed target list never rewrites an ID that may already be complete. + #expect(TestTools.qualify("Other/Suite/test()", targets: ["AppTests"], exact: false) == "Other/Suite/test()") + #expect(TestTools.qualify("Parser/parses()", targets: ["A", "B"]) == "Parser/parses()") + #expect(TestTools.qualify("Parser/parses()", targets: []) == "Parser/parses()") + } + + // MARK: - Test targets from the scheme + + @Test("scheme test targets come from enabled testables") + func schemeTestables() { + let xml = """ + + + + + + + + + + + + + + + + + + + """ + #expect(SchemeFile.testTargets(schemeXML: xml, projectDirectory: "/tmp") == ["AppTests", "AppUITests"]) + } + + @Test("scheme test targets come from its test plan when it has one") + func schemeTestPlan() throws { + let dir = tempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let plan = """ + {"version": 1, "testTargets": [ + {"target": {"name": "AppTests", "containerPath": "container:App.xcodeproj"}}, + {"enabled": false, "target": {"name": "OffTests"}} + ]} + """ + try plan.write(toFile: "\(dir)/Quick.xctestplan", atomically: true, encoding: .utf8) + let xml = """ + + + + """ + #expect(SchemeFile.testTargets(schemeXML: xml, projectDirectory: dir) == ["AppTests"]) + } + + // MARK: - Every message, with location + + private let detailsJSON = """ + {"testNodes": [{"nodeType": "Test Plan", "name": "Quick", "children": [ + {"nodeType": "Unit test bundle", "name": "AppTests", "children": [ + {"nodeType": "Test Suite", "name": "Outer", "children": [ + {"nodeType": "Test Suite", "name": "Inner", "children": [ + {"nodeType": "Test Case", "name": "parses(input:)", "nodeIdentifier": "Outer/Inner/parses(input:)", + "result": "Failed", "children": [ + {"nodeType": "Arguments", "name": "\\"a\\"", "result": "Passed", "children": []}, + {"nodeType": "Arguments", "name": "\\"b\\"", "result": "Failed", "children": [ + {"nodeType": "Failure Message", "name": "ParserTests.swift:42: Expectation failed: 1 == 2"} + ]}, + {"nodeType": "Arguments", "name": "\\"c\\"", "result": "Failed", "children": [ + {"nodeType": "Failure Message", "name": "ParserTests.swift:43: Expectation failed: x"} + ]} + ]} + ]}, + {"nodeType": "Test Case", "name": "retried()", "nodeIdentifier": "Outer/retried()", "result": "Passed", + "children": [ + {"nodeType": "Repetition", "name": "Retry 1", "result": "Failed", "children": []}, + {"nodeType": "Repetition", "name": "Retry 2", "result": "Passed", "children": []} + ]}, + {"nodeType": "Test Case", "name": "noMessage()", "nodeIdentifier": "Outer/noMessage()", "result": "Failed"} + ]} + ]} + ]}]} + """ + + @Test("failures carry the full ID and every message with file, line and argument") + func failureMessages() { + let failures = TestTools.parseTestFailures(json(detailsJSON)) + #expect(failures.count == 2) + let parameterized = failures[0] + #expect(parameterized.testIdentifier == "AppTests/Outer/Inner/parses(input:)") + let messages = parameterized.messages ?? [] + #expect(messages.count == 2) + #expect(messages.first?.file == "ParserTests.swift") + #expect(messages.first?.line == 42) + #expect(messages.first?.label == "\"b\"") + #expect(messages.first?.text == "Expectation failed: 1 == 2") + #expect(parameterized.message.contains("[\"c\"] ParserTests.swift:43: Expectation failed: x")) + #expect(failures[1].testIdentifier == "AppTests/Outer/noMessage()") + #expect(failures[1].messages == nil) + } + + @Test("tests that passed on a retry are reported as flaky") + func flakyTests() { + #expect(TestTools.parseFlakyTests(json(detailsJSON)) == ["AppTests/Outer/retried()"]) + } + + @Test("message locations are split out only when present") + func messageParsing() { + let located = FailureMessage.parse("Sources/Thing.swift:7: XCTAssertEqual failed: (\"1\") is not equal to (\"2\")") + #expect(located.file == "Sources/Thing.swift") + #expect(located.line == 7) + #expect(located.text.hasPrefix("XCTAssertEqual failed")) + let plain = FailureMessage.parse("Test crashed with signal SEGV") + #expect(plain.file == nil) + #expect(plain.text == "Test crashed with signal SEGV") + #expect(FailureMessage(text: "boom", file: "A.swift", line: 3, label: "x").display == "[x] A.swift:3: boom") + } + + // MARK: - Test step options + + @Test("test options become test-step arguments") + func testOptionArguments() throws { + #expect( + try TestTools.TestRunOptions(retries: 2).arguments() + == ["-retry-tests-on-failure", "-test-iterations", "3"]) + #expect(try TestTools.TestRunOptions(iterations: 5).arguments() == ["-test-iterations", "5"]) + #expect( + try TestTools.TestRunOptions(parallel: false, testTimeoutSeconds: 30).arguments() + == [ + "-parallel-testing-enabled", "NO", "-test-timeouts-enabled", "YES", + "-default-test-execution-time-allowance", "30", "-maximum-test-execution-time-allowance", "30", + ]) + #expect(try TestTools.TestRunOptions().arguments().isEmpty) + #expect(throws: TestTools.TestRunOptions.InvalidError.self) { + try TestTools.TestRunOptions(retries: 1, iterations: 3).arguments() + } + } + + // MARK: - Console and attachments per failure + + @Test("console output is keyed by test and keeps the last lines whatever they say") + func consoleTail() { + let lines = (1...60).map { "line \($0)" }.joined(separator: "\\n") + let log = """ + {"subsections": [ + {"testDetails": {"testIdentifier": "Parser/parses()", "testName": "parses()", + "emittedOutput": "◇ Test parses() started.\\n\(lines)"}}, + {"testDetails": {"testName": "testFoo2", "emittedOutput": "other test"}} + ]} + """ + let console = TestTools.parseTestConsole(Data(log.utf8)) + let parses = TestTools.console(for: "AppTests/Parser/parses()", in: console) + #expect(parses?.hasSuffix("line 60") == true) + #expect(parses?.contains("line 20") == false) + #expect(parses?.contains("◇ Test") == false) + #expect(TestTools.console(for: "AppTests/Suite/testFoo", in: console) == nil) + } + + @Test("attachments go to their own failure, not to a test with a longer name") + func attachmentsPerFailure() { + let failures = [ + TestTools.TestFailureObservation( + testName: "testFoo", testIdentifier: "AppTests/Suite/testFoo", message: "a", source: "x"), + TestTools.TestFailureObservation( + testName: "testFoo2", testIdentifier: "AppTests/Suite/testFoo2", message: "b", source: "x"), + ] + let attached = TestTools.attach( + attachments: [(test: "Suite/testFoo2", path: "/tmp/b.png")], console: [:], to: failures) + #expect(attached[0].attachments == nil) + #expect(attached[1].attachments == ["/tmp/b.png"]) + } + + // MARK: - Bounded output + + @Test("text output groups identical messages and caps the list") + func cappedOutput() { + var failures = (1...30).map { + TestTools.TestFailureObservation( + testName: "t\($0)", testIdentifier: "AppTests/Suite/t\($0)()", message: "distinct \($0)", source: "x") + } + failures += (1...3).map { + TestTools.TestFailureObservation( + testName: "s\($0)", testIdentifier: "AppTests/Setup/s\($0)()", message: "setUp failed", source: "x") + } + let lines = TestFailureText.lines(failures, limit: 20) + #expect(lines.filter { $0.contains("FAIL:") }.count == 20) + #expect(lines.last?.contains("+13 more failures") == true) + + let grouped = TestFailureText.lines(Array(failures.suffix(3))) + #expect(grouped.first?.contains("3 tests, same message") == true) + #expect(grouped.filter { $0.contains("FAIL:") }.count == 1) + } + + // MARK: - Agent JSON + + @Test("agent failures carry file and line; counts agree with the summary") + func agentProjection() throws { + var failure = TestTools.TestFailureObservation( + testName: "parses()", testIdentifier: "AppTests/Parser/parses()", + message: "P.swift:4: first\nP.swift:9: second", source: "x") + failure.messages = [ + FailureMessage(text: "first", file: "P.swift", line: 4), + FailureMessage(text: "second", file: "P.swift", line: 9), + ] + let execution = TestTools.TestExecution( + succeeded: false, elapsed: "1", xcresultPath: "/tmp/r.xcresult", scheme: "App", simulator: "iPhone", + totalTestCount: 5, passedTestCount: 2, failedTestCount: 3, skippedTestCount: 0, expectedFailureCount: 0, + failures: [failure], deviceName: nil, osVersion: nil, screenshotPaths: [], hasStructuredSummary: true, + buildFailed: false, buildDiagnostics: nil, flakyTests: ["AppTests/Parser/flaky()"]) + let projected = AgentResultProjection.project(execution) + #expect(projected.failed == 3) + #expect(projected.failures[0].file == "P.swift") + #expect(projected.failures[0].line == 4) + #expect(projected.failures[0].message == "first") + #expect(projected.failures[0].moreMessages == 1) + #expect(projected.flaky == ["AppTests/Parser/flaky()"]) + #expect(projected.reason == nil) + } + + @Test("a failed run with no failures says why") + func agentReason() { + let execution = TestTools.TestExecution( + succeeded: false, elapsed: "1", xcresultPath: "/tmp/r.xcresult", scheme: "App", simulator: "iPhone", + totalTestCount: 0, passedTestCount: 0, failedTestCount: 0, skippedTestCount: 0, expectedFailureCount: 0, + failures: [], deviceName: nil, osVersion: nil, screenshotPaths: [], hasStructuredSummary: true, + buildFailed: false, buildDiagnostics: nil) + let projected = AgentResultProjection.project(execution) + #expect(projected.reason?.contains("no tests ran") == true) + } + + // MARK: - Reruns + + @Test("a run that failed before any test reported records that, so rerun refuses") + func infraFailureRecord() { + let root = tempDir() + defer { try? FileManager.default.removeItem(atPath: root) } + _ = LastFailuresStore.write(failures: ["AppTests/Old/test()"], scheme: "S", simulator: "Sim", at: root) + let crash = TestTools.TestFailureObservation( + testName: "xcodebuild", testIdentifier: "xcodebuild", message: "Test runner crashed\nmore", source: "stderr") + TestTools.persistLastFailures( + failures: [crash], succeeded: false, scheme: "S", simulator: "Sim", repoRoot: root, + run: LastFailuresStore.RunSettings(project: "/p/App.xcodeproj", testPlan: "Quick", env: ["A=1"])) + let payload = LastFailuresStore.read(at: root) + #expect(payload?.failures.isEmpty == true) + #expect(payload?.infraFailure == "Test runner crashed") + #expect(payload?.run?.testPlan == "Quick") + #expect(payload?.run?.env == ["A=1"]) + } + + @Test("an old last-failures file without run settings still reads") + func legacyPayload() throws { + let root = tempDir() + defer { try? FileManager.default.removeItem(atPath: root) } + let dir = (root as NSString).appendingPathComponent(".xcforge") + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + try #"{"failures": ["A/b"], "scheme": "S", "recordedAt": "2026-10-01T00:00:00Z"}"# + .write(toFile: LastFailuresStore.path(at: root), atomically: true, encoding: .utf8) + let payload = LastFailuresStore.read(at: root) + #expect(payload?.failures == ["A/b"]) + #expect(payload?.run == nil) + #expect(payload?.infraFailure == nil) + } + + @Test("a test build is current until a source changes or the settings differ") + func testBuildFreshness() throws { + let root = tempDir() + defer { try? FileManager.default.removeItem(atPath: root) } + let source = (root as NSString).appendingPathComponent("Thing.swift") + try "let a = 1".write(toFile: source, atomically: true, encoding: .utf8) + let old = Date().addingTimeInterval(-120) + try FileManager.default.setAttributes([.modificationDate: old], ofItemAtPath: source) + let project = (root as NSString).appendingPathComponent("App-\(UUID().uuidString).xcodeproj") + defer { try? FileManager.default.removeItem(atPath: LastResultStore.filePath(for: project)) } + + let key = LastResultStore.testBuildKey( + scheme: "App", configuration: "Debug", coverage: false, physicalDevice: false) + LastResultStore.recordTestBuild(project: project, key: key, now: Date().addingTimeInterval(-60)) + #expect(LastResultStore.testBuildIsCurrent(project: project, key: key, sourceRoot: root)) + + let other = LastResultStore.testBuildKey( + scheme: "App", configuration: "Release", coverage: false, physicalDevice: false) + #expect(!LastResultStore.testBuildIsCurrent(project: project, key: other, sourceRoot: root)) + + try "let a = 2".write(toFile: source, atomically: true, encoding: .utf8) + #expect(!LastResultStore.testBuildIsCurrent(project: project, key: key, sourceRoot: root)) + } +} From 14f6c1f53a5e216c20d79baccf318ac2ffc73ac8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 16:14:00 +0000 Subject: [PATCH 08/28] fix: build and test reporting agents can trust (bundle 6) - CLI test run shows a failed test build as compile errors with file and line; test_sim uses the parsed build errors too - test failures never runs tests: it reads the last test run, says when there is none, and returns compile errors when that run's build-for-testing failed instead of an older run's failures - build-for-testing passes the test plan and builds only the targets a filter names; the last-build record tracks plan and targets - build compile, build-for-testing, test and test list share build flags - hang watchdog samples at 300s/540s and only reports on timeout or diagnose - build compile skips -showBuildSettings and compiles without a booted simulator - one simulator name lookup everywhere (booted first, newest OS, no prefix matches) - a zero-match filter lists suggestions without rebuilding - agent JSON carries the result bundle and the timeout reason - build run exits non-zero when the app didn't launch Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EtxtFPs6UBAWXqEpkj2W2N --- CHANGELOG.md | 10 + CLAUDE.md | 3 +- Skills/xcforge/references/build-tools.md | 2 +- Skills/xcforge/references/test-tools.md | 10 +- .../Commands/Build/BuildCommand.swift | 9 +- .../Commands/Test/TestRenderer.swift | 18 ++ .../Test/TestRerunFailedCommand.swift | 2 +- .../Persistence/LastResultStore.swift | 43 +++- .../Support/AgentResultProjection.swift | 37 ++- Sources/XCForgeKit/Support/HangWatchdog.swift | 5 + .../XCForgeKit/Support/ProjectResolver.swift | 120 +++++----- Sources/XCForgeKit/Support/TestIDs.swift | 18 ++ Sources/XCForgeKit/Support/Xcodebuild.swift | 2 +- Sources/XCForgeKit/Tools/BuildProvider.swift | 58 +++-- .../XCForgeKit/Tools/SimulatorProvider.swift | 24 +- Sources/XCForgeKit/Tools/TestProvider.swift | 223 ++++++++++++------ .../AgentResultProjectionTests.swift | 11 +- Tests/XCForgeKitTests/BuildLoopTests.swift | 165 +++++++++++++ 18 files changed, 534 insertions(+), 226 deletions(-) create mode 100644 Tests/XCForgeKitTests/BuildLoopTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 68150bb..1e18cfb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - Failures carry every message with file and line, the argument or repetition it came from, and their attachments; identical messages are grouped, and text output is capped at 20 failures with a count of the rest - A test failure is no longer reported as "test target build failed". When the build or runner fails before any test reports, agent JSON says why (`reason`) and `rerun-failed` refuses instead of running nothing - Agent JSON `failed` counts every failure, not only the ones listed +- Agent JSON includes the result bundle (`xcresult`) and, for a timed-out run, which limit fired and how to raise it (`reason`), even when some failures were read +- CLI `test run` shows a failed test build as compile errors with file and line, not as failing tests +- `test failures` / `test_failures` never run tests: without `xcresultPath` they read the project's last test run, say so when there is none, and return the compile errors when that run's build-for-testing failed (instead of the previous run's failures) +- build-for-testing uses the test plan and, for a filter whose IDs all name a test target, builds only those targets +- `build compile`, build-for-testing, `test` and `test list` pass the same build flags, so switching between them doesn't rebuild everything +- The hang watchdog samples at 300s and 540s instead of 60s and 120s, and a snapshot is only reported for a timeout or `diagnose` +- `build compile` / `build_compile` skip the `-showBuildSettings` call and no longer need a booted simulator (they compile for a booted one if any, else the newest iPhone simulator) +- Simulator names resolve the same way everywhere: exact name, the booted match first, else the newest OS; two matches on the same OS ask for a UDID. Prefix matches (`iPhone 16` → `iPhone 16 Pro`) are gone +- A filter that matches no test lists suggestions from the last build instead of building again +- `build run` exits non-zero when the app wasn't launched - **Cancelling stops the build.** An MCP cancel, a client disconnect, Ctrl-C, SIGTERM or SIGHUP now stops the xcodebuild (or other command) xcforge started, together with every process it spawned. Timeouts kill the whole process tree too, so compiler and test-runner children no longer outlive the call and hold DerivedData ("database is locked" on the next run) - Two MCP calls that would run xcodebuild on the same DerivedData folder (or the same project's default one) at once now run one after the other, first come first served - An explicit `project` in another repo or worktree uses the `.xcforge.yaml` next to it, for both session defaults (scheme, simulator, test plan) and xcodebuild options (DerivedData, lock, idle timeout) diff --git a/CLAUDE.md b/CLAUDE.md index a39ea8c..cb2002d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,7 +18,8 @@ swift run xcforge build --help # CLI mode after **600s with no output** (`--idle-timeout` / `idleTimeoutSeconds`, 0 disables). - Pass `--diagnose` / `diagnose: true` to capture a diagnostic snapshot even on success. - On timeout, the result includes a `/tmp/xcf-diag--.txt` path and a summary line. -- `build_sim` and `build_run_sim` share the same watchdog design. +- `build_sim` and `build_run_sim` share the same watchdog design. The watchdog samples at 300s and 540s and + only reports a snapshot for a timeout or `diagnose`. **Formatting** (CI enforces strict lint): ```bash diff --git a/Skills/xcforge/references/build-tools.md b/Skills/xcforge/references/build-tools.md index 7d6cc1d..02df7db 100644 --- a/Skills/xcforge/references/build-tools.md +++ b/Skills/xcforge/references/build-tools.md @@ -8,7 +8,7 @@ ## build_compile -Fast compile-only build without install or launch (~5s vs ~20s for full pipeline). Reuses the standard build infrastructure, skips the simulator boot/install/launch chain. +Compile-only build: no boot, install or launch, and no extra build-settings lookup. Needs no booted simulator: without one configured or booted it compiles for the newest available iPhone simulator. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| diff --git a/Skills/xcforge/references/test-tools.md b/Skills/xcforge/references/test-tools.md index c73b1a4..becc1fd 100644 --- a/Skills/xcforge/references/test-tools.md +++ b/Skills/xcforge/references/test-tools.md @@ -59,6 +59,10 @@ Use `list_tests` to discover available identifiers if unsure. `retries` can't be combined with `iterations` or `untilFailure`. +**Test build scope:** build-for-testing uses the test plan, and a filter whose IDs all name a +test target builds only those targets. The build uses the same flags as `build_compile`, so +switching between them doesn't rebuild everything. + **Failure output:** each failure lists every message with `file:line` and, for parameterized or repeated tests, the argument or repetition it came from. Failures with the same message are grouped ("N tests, same message"). Text output shows 20 failures and says how many more there are; @@ -83,12 +87,10 @@ Get detailed failure information with optional console output per failed test. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `xcresult_path` | No* | — | Path to .xcresult bundle | -| `project` | No* | Auto-detect | Alternative: re-derive from project | -| `scheme` | No* | Auto-detect | Used with project | -| `simulator` | No* | Auto-detect | Used with project | +| `project` | No* | Auto-detect | Read this project's last test run | | `include_console` | No | false | Include console output captured during each failed test | -*Provide either `xcresult_path` OR `project`/`scheme`/`simulator`. The xcresult_path from a previous `test_sim` call is preferred. +*Without `xcresult_path`, reads the project's last test run (`test_sim`, `build_and_test`, `xcforge test run`). It never runs tests; with no recorded run it says so. When that run's build-for-testing failed, it returns the compile errors with file and line instead of failures. **Returns per failure:** - Test class and method name diff --git a/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift b/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift index 4e1e18c..e7aeeec 100644 --- a/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift +++ b/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift @@ -62,6 +62,7 @@ struct BuildCompile: AsyncParsableCommand { configuration: configuration, long: long, diagnose: diagnose, + compileOnly: true, env: env ) @@ -191,7 +192,7 @@ struct BuildRun: AsyncParsableCommand { installStatus = "skipped (boot failed)" launchStatus = "skipped (boot failed)" } else { - installStatus = "skipped (no app path)" + installStatus = "skipped (couldn't find the built app)" } if useJSON { @@ -206,10 +207,8 @@ struct BuildRun: AsyncParsableCommand { launchStatus: launchStatus, appPid: appPid, appRunning: appRunning)) } - let failed = - bootStatus.hasPrefix("failed") || installStatus.hasPrefix("failed") - || launchStatus.hasPrefix("failed") - if failed { + // `build run` promises a running app; anything short of that is a failure. + if !appRunning { throw ExitCode.failure } } diff --git a/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift b/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift index b7739f9..1ebeaa8 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestRenderer.swift @@ -5,6 +5,18 @@ enum TestRenderer { static func renderTest(_ execution: TestTools.TestExecution) -> String { var lines: [String] = [] + // A test build that failed is compile errors, not failing tests. + if execution.buildFailed { + lines.append("TEST BUILD FAILED in \(execution.elapsed)s, no test ran") + lines += TestFailureText.buildErrorLines(execution.failures, indent: "") + if let path = execution.hangDiagnosticPath { + lines.append("Diagnostic snapshot: \(path)") + } + lines.append("Scheme: \(execution.scheme)") + if !execution.xcresultPath.isEmpty { lines.append("xcresult: \(execution.xcresultPath)") } + return lines.joined(separator: "\n") + } + let icon = execution.succeeded ? "PASSED" : "FAILED" lines.append("Tests \(icon) in \(execution.elapsed)s") @@ -50,6 +62,12 @@ enum TestRenderer { if result.failures.isEmpty { return "No test failures found.\nxcresult: \(result.xcresultPath)" } + if result.buildFailed { + var lines = ["The last test run failed to build, so no test ran. Build errors:"] + lines += TestFailureText.buildErrorLines(result.failures, indent: "") + lines.append("xcresult: \(result.xcresultPath)") + return lines.joined(separator: "\n") + } var lines: [String] = [] lines.append("\(result.failures.count) test failure(s):") diff --git a/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift b/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift index 128a532..64d1ca1 100644 --- a/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift +++ b/Sources/XCForgeCLI/Commands/Test/TestRerunFailedCommand.swift @@ -89,7 +89,7 @@ struct TestRerunFailed: AsyncParsableCommand { } else { skipBuild = await TestTools.lastTestBuildIsCurrent( project: resolvedProject, scheme: scheme ?? payload.scheme, simulator: simulator ?? payload.simulator, - configuration: configuration, env: env) + configuration: configuration, testplan: resolvedTestplan, testIDs: payload.failures, env: env) } // Pass failure IDs as a pre-split list so IDs containing commas (e.g. diff --git a/Sources/XCForgeKit/Persistence/LastResultStore.swift b/Sources/XCForgeKit/Persistence/LastResultStore.swift index 926413f..fa89de2 100644 --- a/Sources/XCForgeKit/Persistence/LastResultStore.swift +++ b/Sources/XCForgeKit/Persistence/LastResultStore.swift @@ -19,6 +19,8 @@ public enum LastResultStore { /// When build-for-testing last succeeded, and for which scheme, configuration and platform. public var testBuiltAt: Date? public var testBuildKey: String? + /// The test targets that build covered; nil when it built every target in the scheme or plan. + public var testBuildTargets: [String]? } static func directory() -> String { @@ -56,31 +58,42 @@ public enum LastResultStore { save(record, to: path) } - /// What a test build depends on besides the sources. - static func testBuildKey(scheme: String, configuration: String, coverage: Bool, physicalDevice: Bool) - -> String - { - [scheme, configuration, coverage ? "coverage" : "", physicalDevice ? "device" : "simulator"] - .joined(separator: "|") + /// What a test build depends on besides the sources and the test targets. + static func testBuildKey( + scheme: String, configuration: String, coverage: Bool, physicalDevice: Bool, testPlan: String? = nil + ) -> String { + [ + scheme, configuration, coverage ? "coverage" : "", physicalDevice ? "device" : "simulator", + testPlan ?? "", + ] + .joined(separator: "|") } - /// Note that build-for-testing just succeeded for `project` with `key`. - static func recordTestBuild(project: String, key: String, now: Date = Date()) { + /// Note that build-for-testing just succeeded for `project` with `key`, covering `targets` + /// (nil: every test target). + static func recordTestBuild(project: String, key: String, targets: [String]? = nil, now: Date = Date()) { let path = filePath(for: project) try? FileManager.default.createDirectory( atPath: directory(), withIntermediateDirectories: true, attributes: nil) var record = load(path) ?? Record(build: nil, test: nil, updatedAt: now) record.testBuiltAt = now record.testBuildKey = key + record.testBuildTargets = targets?.sorted() save(record, to: path) } - /// True when the last successful build-for-testing for `project` used `key` and nothing - /// under `sourceRoot` changed after it, so its products can be tested as they are. - static func testBuildIsCurrent(project: String, key: String, sourceRoot: String) -> Bool { + /// True when the last successful build-for-testing for `project` used `key`, built every + /// target in `targets` (nil: needs every target), and nothing under `sourceRoot` changed + /// after it, so its products can be tested as they are. + static func testBuildIsCurrent( + project: String, key: String, targets: [String]? = nil, sourceRoot: String + ) -> Bool { guard let record = load(filePath(for: project)), record.testBuildKey == key, let builtAt = record.testBuiltAt else { return false } + if let built = record.testBuildTargets { + guard let targets, Set(targets).isSubset(of: Set(built)) else { return false } + } return !SourceChanges.anyModified(under: sourceRoot, after: builtAt) } @@ -112,7 +125,10 @@ public enum LastResultStore { } /// Record the result bundle named in an xcodebuild argument list, if any. - static func recordFromArguments(_ args: [String]) { + /// + /// A failed build-for-testing counts as the latest test run, so `test failures` reports + /// its errors instead of an earlier run's failures. + static func recordFromArguments(_ args: [String], succeeded: Bool = true) { guard let bundleIndex = args.firstIndex(of: "-resultBundlePath"), bundleIndex + 1 < args.count else { return } let bundle = args[bundleIndex + 1] @@ -120,7 +136,8 @@ public enum LastResultStore { guard let projectIndex, projectIndex + 1 < args.count else { return } let project = args[projectIndex + 1] let testActions: Set = ["test", "test-without-building"] - let kind: Kind = args.contains { testActions.contains($0) } ? .test : .build + let failedTestBuild = !succeeded && args.contains("build-for-testing") + let kind: Kind = (failedTestBuild || args.contains { testActions.contains($0) }) ? .test : .build record(project: project, kind: kind, bundlePath: bundle) } } diff --git a/Sources/XCForgeKit/Support/AgentResultProjection.swift b/Sources/XCForgeKit/Support/AgentResultProjection.swift index b9e5e9d..2a222d0 100644 --- a/Sources/XCForgeKit/Support/AgentResultProjection.swift +++ b/Sources/XCForgeKit/Support/AgentResultProjection.swift @@ -1,9 +1,9 @@ import Foundation /// Slim projection of a test run designed for agent consumption. -/// Encoded JSON omits slowest-test arrays, build diagnostics, xcresult paths, -/// and other context-burning fields. `knownFailures`, `flaky` and `reason` are -/// omitted when empty to keep the wire shape tight. +/// Encoded JSON omits slowest-test arrays, build diagnostics and other context-burning +/// fields. `knownFailures`, `flaky`, `reason` and `xcresult` are omitted when empty to +/// keep the wire shape tight. public struct AgentTestResult: Codable, Sendable, Equatable { public let succeeded: Bool public let buildOk: Bool @@ -16,8 +16,11 @@ public struct AgentTestResult: Codable, Sendable, Equatable { public let failures: [AgentFailure] /// Tests that failed and then passed on a retry. Omitted when empty. public let flaky: [String]? - /// Why the run failed when `failures` can't say: nothing ran, or the runner died. + /// Why the run failed when `failures` can't say: nothing ran, the runner died, or a + /// time limit killed it (which limit, and how to raise it). public let reason: String? + /// The result bundle, for `test_failures` or xcresulttool. + public let xcresult: String? /// One failing test. `message` is the first line of its first message; `file` and /// `line` locate it, `moreMessages` counts the rest. @@ -44,6 +47,7 @@ public struct AgentTestResult: Codable, Sendable, Equatable { enum CodingKeys: String, CodingKey { case succeeded, buildOk, total, passed, failed, skipped, timedOut, knownFailures, failures, flaky, reason + case xcresult } public func encode(to encoder: Encoder) throws { @@ -63,6 +67,9 @@ public struct AgentTestResult: Codable, Sendable, Equatable { try c.encode(flaky, forKey: .flaky) } try c.encodeIfPresent(reason, forKey: .reason) + if let xcresult, !xcresult.isEmpty { + try c.encode(xcresult, forKey: .xcresult) + } } public init( @@ -76,7 +83,8 @@ public struct AgentTestResult: Codable, Sendable, Equatable { knownFailures: [String]?, failures: [AgentFailure], flaky: [String]? = nil, - reason: String? = nil + reason: String? = nil, + xcresult: String? = nil ) { self.succeeded = succeeded self.buildOk = buildOk @@ -89,6 +97,7 @@ public struct AgentTestResult: Codable, Sendable, Equatable { self.failures = failures self.flaky = flaky self.reason = reason + self.xcresult = xcresult } } @@ -126,10 +135,11 @@ public enum AgentResultProjection { // The real count of failing tests, less the gated ones, even when parsing lost some of them. let failed = max(agentFailures.count, e.failedTestCount - known.count) var reason: String? - if !e.succeeded && agentFailures.isEmpty { - if e.xcforgeTimedOut { - reason = "timed out before the tests finished" - } else if e.totalTestCount == 0 { + if e.xcforgeTimedOut { + // Reported even when some failures were read: the run didn't finish. + reason = e.timeoutDetail.map(firstLine) ?? "timed out before the tests finished" + } else if !e.succeeded && agentFailures.isEmpty { + if e.totalTestCount == 0 { reason = "no tests ran: the filter, scheme or test plan selected none" } else if failed > 0 { reason = "\(failed) tests failed but their details couldn't be read from \(e.xcresultPath)" @@ -148,7 +158,8 @@ public enum AgentResultProjection { knownFailures: e.knownFailures, failures: agentFailures, flaky: e.flakyTests.isEmpty ? nil : e.flakyTests, - reason: reason + reason: reason, + xcresult: e.xcresultPath ) } @@ -167,7 +178,8 @@ public enum AgentResultProjection { knownFailures: projected.knownFailures, failures: projected.failures, flaky: projected.flaky, - reason: projected.reason + reason: projected.reason, + xcresult: projected.xcresult ) } return projected @@ -186,7 +198,8 @@ public enum AgentResultProjection { timedOut: b.xcforgeTimedOut, knownFailures: known, failures: [], - reason: b.buildSucceeded ? "the tests didn't run" : "the build failed" + reason: b.buildSucceeded ? "the tests didn't run" : "the build failed", + xcresult: b.buildXcresultPath ) } } diff --git a/Sources/XCForgeKit/Support/HangWatchdog.swift b/Sources/XCForgeKit/Support/HangWatchdog.swift index d7bcebc..2555512 100644 --- a/Sources/XCForgeKit/Support/HangWatchdog.swift +++ b/Sources/XCForgeKit/Support/HangWatchdog.swift @@ -5,6 +5,11 @@ import Foundation /// Spawn one watchdog per xcodebuild invocation; call `cancel()` when the process finishes. /// Cancelling before the first deadline fires leaves zero side effects — no snapshot files written. final class HangWatchdog: Sendable { + /// When to sample: late enough that a healthy build is usually done, and before the + /// default 600s idle limit kills a hung one. Samples are only reported for a timeout + /// or when a diagnosis was asked for. + static let defaultSampleAt: [TimeInterval] = [300, 540] + private let task: Task init( diff --git a/Sources/XCForgeKit/Support/ProjectResolver.swift b/Sources/XCForgeKit/Support/ProjectResolver.swift index 05d58e4..8f254fa 100644 --- a/Sources/XCForgeKit/Support/ProjectResolver.swift +++ b/Sources/XCForgeKit/Support/ProjectResolver.swift @@ -16,7 +16,7 @@ struct MultipleSimulatorMatchError: Error, CustomStringConvertible { /// Throws ResolverError with rich messages when ambiguous. enum AutoDetect { - private struct SimulatorDevice: Sendable { + struct SimulatorDevice: Sendable, Equatable { let name: String let udid: String let runtime: String @@ -330,35 +330,59 @@ enum AutoDetect { return (nameOrUDID, nameOrUDID) } - let devices = try await loadSimulatorDevices() + let winner = try pickSimulator(named: nameOrUDID, from: try await loadSimulatorDevices()) + return (winner.name, winner.udid) + } - // Filter by name, case-insensitive + /// The one simulator every name lookup resolves `name` to: an exact, case-insensitive + /// name match that is available or booted; a booted one when exactly one match is booted, + /// else the one with the newest OS. Two matches on the same newest OS are an error that + /// lists their UDIDs. + static func pickSimulator(named name: String, from devices: [SimulatorDevice]) throws -> SimulatorDevice { let matches = devices.filter { - $0.isAvailable && $0.name.caseInsensitiveCompare(nameOrUDID) == .orderedSame - } - - if matches.isEmpty { - throw ResolverError("No available simulator found with name '\(nameOrUDID)'.") + ($0.isAvailable || $0.state == "Booted") && $0.name.caseInsensitiveCompare(name) == .orderedSame } - - // Sort by OS version descending to pick highest-OS match - let sorted = matches.sorted { lhs, rhs in - lhs.runtime > rhs.runtime + guard !matches.isEmpty else { + throw ResolverError("No available simulator found with name '\(name)'.") } - - // If multiple match the same (highest) runtime, that's an ambiguity error - let highestRuntime = sorted[0].runtime - let topTier = sorted.filter { $0.runtime == highestRuntime } + if matches.count == 1 { return matches[0] } + let booted = matches.filter { $0.state == "Booted" } + if booted.count == 1 { return booted[0] } + let candidates = booted.isEmpty ? matches : booted + let newest = candidates.map { runtimeVersion($0.runtime) }.max { $0.lexicographicallyPrecedes($1) } ?? [] + let topTier = candidates.filter { runtimeVersion($0.runtime) == newest } if topTier.count > 1 { let descriptions = topTier.map { describe($0) }.joined(separator: "\n ") throw MultipleSimulatorMatchError( - "Simulator '\(nameOrUDID)' matches \(topTier.count) devices with the same OS version" - + " (\(highestRuntime)). Specify a UDID instead:\n \(descriptions)" + "Simulator '\(name)' matches \(topTier.count) devices with the same OS version" + + " (\(topTier[0].runtime)). Specify a UDID instead:\n \(descriptions)" ) } + return topTier[0] + } - let winner = sorted[0] - return (winner.name, winner.udid) + /// `iOS-18-10` → [18, 10], so 18.10 sorts after 18.2. + static func runtimeVersion(_ runtime: String) -> [Int] { + runtime.split { !$0.isNumber }.compactMap { Int($0) } + } + + /// A simulator to compile against when none is configured or booted: the booted one + /// with the newest OS, else the available iPhone with the newest OS. Compiling only + /// needs the platform and architecture, so any of them gives the same build. + static func simulatorForCompile() async -> String? { + guard let devices = try? await loadSimulatorDevices() else { return nil } + return simulatorForCompile(from: devices) + } + + static func simulatorForCompile(from devices: [SimulatorDevice]) -> String? { + let ios = devices.filter { $0.isAvailable && $0.runtime.lowercased().hasPrefix("ios") } + let booted = ios.filter { $0.state == "Booted" } + let pool = booted.isEmpty ? ios.filter { $0.name.hasPrefix("iPhone") } : booted + let sorted = pool.sorted { + let (l, r) = (runtimeVersion($0.runtime), runtimeVersion($1.runtime)) + return l == r ? $0.name < $1.name : r.lexicographicallyPrecedes(l) + } + return sorted.first?.udid } /// Returns true if the string looks like a physical device UDID. @@ -462,13 +486,18 @@ enum AutoDetect { throw ResolverError("Simulator validation failed: \(error)") } - guard shellResult.succeeded, - let data = shellResult.stdout.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], - let devices = json["devices"] as? [String: [[String: Any]]] - else { + guard shellResult.succeeded, let results = parseSimulatorDevices(shellResult.stdout) else { throw ResolverError("Failed to parse simulator list") } + return results + } + + /// Devices from `simctl list devices -j` output, or nil when it doesn't parse. + static func parseSimulatorDevices(_ output: String) -> [SimulatorDevice]? { + guard let data = output.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let devices = json["devices"] as? [String: [[String: Any]]] + else { return nil } var results: [SimulatorDevice] = [] for (runtime, deviceList) in devices { @@ -495,7 +524,7 @@ enum AutoDetect { return results } - private static func describe(_ device: SimulatorDevice) -> String { + static func describe(_ device: SimulatorDevice) -> String { let availabilityLabel = device.isAvailable ? device.state : "Unavailable" return "\(device.name) (\(device.runtime)) — \(availabilityLabel) — \(device.udid)" } @@ -576,43 +605,8 @@ enum AutoDetect { /// Resolve simulator name to UDID via simctl private static func resolveNameToUDID(_ name: String) async -> String? { - guard let result = try? await Shell.xcrun(timeout: 15, "simctl", "list", "devices", "-j"), - result.succeeded, - let data = result.stdout.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], - let devices = json["devices"] as? [String: [[String: Any]]] - else { - return nil - } - - let nameLower = name.lowercased() - var exactMatch: String? - var caseInsensitive: String? - var prefixBooted: String? - var prefixMatch: String? - - for (_, deviceList) in devices { - for device in deviceList { - guard let deviceName = device["name"] as? String, - let udid = device["udid"] as? String - else { continue } - let usable = - (device["isAvailable"] as? Bool ?? false) || (device["state"] as? String) == "Booted" - guard usable else { continue } - - let isBooted = (device["state"] as? String) == "Booted" - - if deviceName == name { - exactMatch = udid - } else if exactMatch == nil && deviceName.lowercased() == nameLower { - caseInsensitive = udid - } else if deviceName.lowercased().hasPrefix(nameLower) { - if isBooted { prefixBooted = udid } else if prefixMatch == nil { prefixMatch = udid } - } - } - } - - return exactMatch ?? caseInsensitive ?? prefixBooted ?? prefixMatch + guard let devices = try? await loadSimulatorDevices() else { return nil } + return try? pickSimulator(named: name, from: devices).udid } } diff --git a/Sources/XCForgeKit/Support/TestIDs.swift b/Sources/XCForgeKit/Support/TestIDs.swift index e64ad44..0ce2436 100644 --- a/Sources/XCForgeKit/Support/TestIDs.swift +++ b/Sources/XCForgeKit/Support/TestIDs.swift @@ -161,6 +161,24 @@ public enum TestFailureText { } } +extension TestFailureText { + /// Compile errors from a failed test build, one per line with file and line, capped. + public static func buildErrorLines( + _ failures: [TestTools.TestFailureObservation], limit: Int = defaultLimit, indent: String = " " + ) -> [String] { + let messages = failures.flatMap { failure in + failure.messages ?? [FailureMessage(text: failure.message)] + } + var out = messages.prefix(limit).map { + indent + $0.display.replacingOccurrences(of: "\n", with: "\n\(indent) ") + } + if messages.count > limit { + out.append("\(indent)+\(messages.count - limit) more errors") + } + return out + } +} + /// Whether any source under a folder changed after a given time, so a rerun can skip /// build-for-testing when nothing was edited. enum SourceChanges { diff --git a/Sources/XCForgeKit/Support/Xcodebuild.swift b/Sources/XCForgeKit/Support/Xcodebuild.swift index c1590c7..e2145c3 100644 --- a/Sources/XCForgeKit/Support/Xcodebuild.swift +++ b/Sources/XCForgeKit/Support/Xcodebuild.swift @@ -63,7 +63,7 @@ public enum Xcodebuild { environment: environment, timeout: timeout, idleTimeout: (idle ?? 0) > 0 ? idle : nil, outputLimit: Shell.defaultOutputLimit) if result.exitCode != -2 { - LastResultStore.recordFromArguments(args) + LastResultStore.recordFromArguments(args, succeeded: result.succeeded) } return result } diff --git a/Sources/XCForgeKit/Tools/BuildProvider.swift b/Sources/XCForgeKit/Tools/BuildProvider.swift index db7c1f0..271122c 100644 --- a/Sources/XCForgeKit/Tools/BuildProvider.swift +++ b/Sources/XCForgeKit/Tools/BuildProvider.swift @@ -371,11 +371,21 @@ public enum BuildTools { configuration: String = "Debug", long: Bool = false, diagnose: Bool = false, + compileOnly: Bool = false, env: Environment = .live ) async throws -> BuildExecution { let resolvedProject = try await env.session.resolveProject(project) let resolvedScheme = try await env.session.resolveScheme(scheme, project: resolvedProject) - let resolvedSimulator = try await env.session.resolveSimulator(simulator) + let resolvedSimulator: String + do { + resolvedSimulator = try await env.session.resolveSimulator(simulator) + } catch { + // Compiling needs a simulator platform, not a particular booted device. + guard compileOnly, simulator == nil, let fallback = await AutoDetect.simulatorForCompile() else { + throw error + } + resolvedSimulator = fallback + } let isWorkspace = resolvedProject.hasSuffix(".xcworkspace") let projectFlag = isWorkspace ? "-workspace" : "-project" @@ -391,17 +401,15 @@ public enum BuildTools { "-configuration", configuration, "-destination", destination, "-skipMacroValidation", - "-parallelizeTargets", - "-resultBundlePath", resultPath, - "build", ] - buildArgs += ["COMPILATION_CACHE_ENABLE_CACHING=YES"] + buildArgs += TestTools.compileFlags + buildArgs += ["-resultBundlePath", resultPath, "build"] + TestTools.compileSettings let start = CFAbsoluteTimeGetCurrent() let buildTimeout = await TestTools.resolveTestTimeout(long: long, env: env) let snapshotPath = TestTools.diagnosticSnapshotPath() let watchdog = HangWatchdog( - udid: resolvedSimulator, snapshotPath: snapshotPath, sampleAt: [60, 120], + udid: resolvedSimulator, snapshotPath: snapshotPath, sampleAt: HangWatchdog.defaultSampleAt, processMatch: resultPath, env: env) let result = try await Xcodebuild.run(buildArgs, timeout: buildTimeout, env: env) watchdog.cancel() @@ -422,7 +430,8 @@ public enum BuildTools { udid: resolvedSimulator, snapshotPath: snapshotPath, processMatch: resultPath, env: env) } } else { - diagResult = watchdogCapture + // A sample of a healthy build is noise; only a timeout or `diagnose` reports one. + diagResult = nil } let elapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - start) @@ -430,11 +439,15 @@ public enum BuildTools { let xcresultIssues = await extractIssuesFromXcresult(resultPath, env: env) if result.succeeded { - let buildInfo = await extractBuildInfo( - project: resolvedProject, scheme: resolvedScheme, - simulator: resolvedSimulator, configuration: configuration, - env: env - ) + // Finding the app costs another xcodebuild call; a compile check doesn't need it. + var buildInfo: (bundleId: String?, appPath: String?) = (nil, nil) + if !compileOnly { + buildInfo = await extractBuildInfo( + project: resolvedProject, scheme: resolvedScheme, + simulator: resolvedSimulator, configuration: configuration, + env: env + ) + } if let bid = buildInfo.bundleId { await env.session.setBuildInfo( @@ -562,6 +575,7 @@ public enum BuildTools { configuration: await env.session.resolveConfiguration(input.configuration), long: input.long ?? false, diagnose: input.diagnose ?? false, + compileOnly: true, env: env ) @@ -881,17 +895,16 @@ public enum BuildTools { let resultPath = TestTools.xcresultPath(prefix: "build") _ = try? await env.shell.run("/bin/rm", arguments: ["-rf", resultPath], timeout: 5) - let buildArgs = [ + var args = [ projectFlag, project, "-scheme", scheme, "-configuration", configuration, "-destination", destination, "-skipMacroValidation", - "-parallelizeTargets", - "-resultBundlePath", resultPath, - "build", - "COMPILATION_CACHE_ENABLE_CACHING=YES", ] + args += TestTools.compileFlags + args += ["-resultBundlePath", resultPath, "build"] + TestTools.compileSettings + let buildArgs = args let settingsArgs = [ projectFlag, project, @@ -908,7 +921,7 @@ public enum BuildTools { let buildTimeout = await TestTools.resolveTestTimeout(long: input.long ?? false, env: env) let buildSnapshotPath = TestTools.diagnosticSnapshotPath() let buildWatchdog = HangWatchdog( - udid: udid, snapshotPath: buildSnapshotPath, sampleAt: [60, 120], processMatch: resultPath, + udid: udid, snapshotPath: buildSnapshotPath, sampleAt: HangWatchdog.defaultSampleAt, processMatch: resultPath, env: env) async let buildTask = Xcodebuild.run(buildArgs, timeout: buildTimeout, env: env) async let settingsTask = Xcodebuild.run(settingsArgs, timeout: 30, env: env) @@ -926,13 +939,14 @@ public enum BuildTools { } buildWatchdog.cancel() let buildDiagResult: DiagnosticSnapshot.Result? - if let captured = await buildWatchdog.latestResult { + // A sample of a healthy build is noise; only a timeout or `diagnose` reports one. + if buildResult.exitCode != -1 && !(input.diagnose ?? false) { + buildDiagResult = nil + } else if let captured = await buildWatchdog.latestResult { buildDiagResult = captured - } else if input.diagnose ?? false { + } else { buildDiagResult = await DiagnosticSnapshot.capture( udid: udid, snapshotPath: buildSnapshotPath, processMatch: resultPath, env: env) - } else { - buildDiagResult = nil } let buildElapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - totalStart) diff --git a/Sources/XCForgeKit/Tools/SimulatorProvider.swift b/Sources/XCForgeKit/Tools/SimulatorProvider.swift index 69adfb8..d8c10df 100644 --- a/Sources/XCForgeKit/Tools/SimulatorProvider.swift +++ b/Sources/XCForgeKit/Tools/SimulatorProvider.swift @@ -444,31 +444,13 @@ public enum SimTools { if nameOrUDID == "booted" { return "booted" } let result = try await env.shell.xcrun(timeout: 15, "simctl", "list", "devices", "-j") - guard let data = result.stdout.data(using: .utf8), - let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], - let deviceGroups = json["devices"] as? [String: [[String: Any]]] - else { + guard let devices = AutoDetect.parseSimulatorDevices(result.stdout) else { throw NSError( domain: "SimTools", code: 1, userInfo: [NSLocalizedDescriptionKey: "Failed to parse simulator list"]) } - - // Exact match only — prefix/substring matching causes false positives on destructive ops - // (e.g. "iPhone 16 Pro" would match "iPhone 16 Pro Max" with hasPrefix) - let needle = nameOrUDID.lowercased() - for (_, devices) in deviceGroups { - for device in devices { - guard let name = device["name"] as? String, - let udid = device["udid"] as? String - else { continue } - if name.lowercased() == needle { return udid } - } - } - throw NSError( - domain: "SimTools", code: 2, - userInfo: [ - NSLocalizedDescriptionKey: "Simulator '\(nameOrUDID)' not found. Use exact name or UDID." - ]) + // The same pick as every other name lookup, so a name never means two devices. + return try AutoDetect.pickSimulator(named: nameOrUDID, from: devices).udid } // MARK: - Typed Execute Methods diff --git a/Sources/XCForgeKit/Tools/TestProvider.swift b/Sources/XCForgeKit/Tools/TestProvider.swift index 8b10c6b..8e09653 100644 --- a/Sources/XCForgeKit/Tools/TestProvider.swift +++ b/Sources/XCForgeKit/Tools/TestProvider.swift @@ -160,6 +160,8 @@ public enum TestTools { public let knownFailures: [String]? /// Tests that failed at least once and then passed on a retry or repetition. public let flakyTests: [String] + /// Which limit killed the run and how to raise it, when `xcforgeTimedOut`. + public let timeoutDetail: String? init( succeeded: Bool, @@ -185,7 +187,8 @@ public enum TestTools { xcforgeTimedOut: Bool = false, slowestTests: [SlowTest] = [], knownFailures: [String]? = nil, - flakyTests: [String] = [] + flakyTests: [String] = [], + timeoutDetail: String? = nil ) { self.succeeded = succeeded self.elapsed = elapsed @@ -211,6 +214,7 @@ public enum TestTools { self.slowestTests = slowestTests self.knownFailures = knownFailures self.flakyTests = flakyTests + self.timeoutDetail = timeoutDetail } } @@ -229,6 +233,8 @@ public enum TestTools { public let screenshots: [ScreenshotAttachment] public let consoleByTest: [String: String] public let xcresultPath: String + /// The bundle is a test build that failed: `failures` are its compile errors. + public var buildFailed = false } public struct CoverageResult: Codable, Sendable { @@ -643,10 +649,9 @@ public enum TestTools { Tool( name: "test_failures", description: """ - Get only failed tests with their error messages from an xcresult bundle. \ - Either provide an xcresult_path from a previous test_sim run, \ - or provide project/scheme to run tests first (auto-detected if omitted). \ - Returns test name + failure message for each failed test. + Failed tests with their messages from an xcresult bundle: xcresult_path, or else \ + the project's last test run (test_sim, build_and_test). Never runs tests. \ + When that run failed to build, returns its compile errors instead. """, inputSchema: .object([ "type": .string("object"), @@ -927,13 +932,11 @@ public enum TestTools { udid: String?, snapshotPath: String, processMatch: String? = nil, env: Environment ) async -> DiagnosticSnapshot.Result? { let watchdogCapture = await watchdog.latestResult - if result.exitCode == -1 || diagnose { - if let captured = watchdogCapture { return captured } - return await DiagnosticSnapshot.capture( - udid: udid, snapshotPath: snapshotPath, processMatch: processMatch, env: env) - } else { - return watchdogCapture - } + // A sample of a healthy build is noise: report one only for a timeout or when asked. + guard result.exitCode == -1 || diagnose else { return nil } + if let captured = watchdogCapture { return captured } + return await DiagnosticSnapshot.capture( + udid: udid, snapshotPath: snapshotPath, processMatch: processMatch, env: env) } private static func formatDiagnosticSuffix(_ result: DiagnosticSnapshot.Result?) -> String { @@ -1083,7 +1086,7 @@ public enum TestTools { guard let listResult = try? await executeListTests( - project: project, scheme: scheme, simulator: simulator, testplan: testplan, env: env + project: project, scheme: scheme, simulator: simulator, testplan: testplan, withoutBuilding: true, env: env ) else { hint += "\nCould not enumerate tests to suggest alternatives." @@ -1132,6 +1135,11 @@ public enum TestTools { ] } + /// Flags every compiling call passes, so `build compile`, build-for-testing and `test` + /// share one set of build settings and don't rebuild each other's products. + static let compileFlags = ["-parallelizeTargets"] + static let compileSettings = ["COMPILATION_CACHE_ENABLE_CACHING=YES"] + /// Run xcodebuild test and return the xcresult path plus any diagnostic snapshot. private static func runTests( project: String, scheme: String, destination: String, @@ -1148,6 +1156,7 @@ public enum TestTools { project: project, scheme: scheme, destination: destination, configuration: configuration ) + args += compileFlags args += ["-resultBundlePath", resultPath] if coverage { @@ -1164,12 +1173,13 @@ public enum TestTools { } } - args += ["test"] + args += ["test"] + compileSettings let timeout = await resolveTestTimeout(long: long, env: env) let snapshotPath = diagnosticSnapshotPath() let watchdog = HangWatchdog( - udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) + udid: udid, snapshotPath: snapshotPath, sampleAt: HangWatchdog.defaultSampleAt, processMatch: resultPath, + env: env) let result = try await Xcodebuild.run( args, environment: childEnvironment, timeout: timeout, env: env) watchdog.cancel() @@ -1192,17 +1202,14 @@ public enum TestTools { project: project, scheme: scheme, destination: destination, configuration: configuration ) - args += [ - "-parallelizeTargets", - "-resultBundlePath", resultPath, - "build", - ] - args += ["COMPILATION_CACHE_ENABLE_CACHING=YES"] + args += compileFlags + args += ["-resultBundlePath", resultPath, "build"] + compileSettings let timeout = await resolveTestTimeout(long: long, env: env) let snapshotPath = diagnosticSnapshotPath() let watchdog = HangWatchdog( - udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) + udid: udid, snapshotPath: snapshotPath, sampleAt: HangWatchdog.defaultSampleAt, processMatch: resultPath, + env: env) let result = try await Xcodebuild.run(args, timeout: timeout, env: env) watchdog.cancel() let diagResult = await resolvedDiagResult( @@ -1212,9 +1219,13 @@ public enum TestTools { } /// Run `xcodebuild build-for-testing` and return the xcresult path plus any diagnostic snapshot. + /// + /// `testplan` and `targets` limit the build to the test targets the run needs; nil + /// `targets` builds every test target in the scheme or plan. static func runBuildForTesting( project: String, scheme: String, destination: String, configuration: String, coverage: Bool, resultPath: String, + testplan: String? = nil, targets: [String]? = nil, long: Bool = false, diagnose: Bool = false, udid: String? = nil, timeoutOverride: TimeInterval? = nil, childEnvironment: [String: String]? = nil, @@ -1226,11 +1237,18 @@ public enum TestTools { project: project, scheme: scheme, destination: destination, configuration: configuration ) + args += compileFlags args += ["-resultBundlePath", resultPath] if coverage { args += ["-enableCodeCoverage", "YES"] } - args += ["build-for-testing"] + if let testplan { + args += ["-testPlan", testplan] + } + for target in targets ?? [] { + args += ["-only-testing", target] + } + args += ["build-for-testing"] + compileSettings let timeout: TimeInterval if let override = timeoutOverride { @@ -1240,7 +1258,8 @@ public enum TestTools { } let snapshotPath = diagnosticSnapshotPath() let watchdog = HangWatchdog( - udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) + udid: udid, snapshotPath: snapshotPath, sampleAt: HangWatchdog.defaultSampleAt, processMatch: resultPath, + env: env) let result = try await Xcodebuild.run( args, environment: childEnvironment, timeout: timeout, env: env) watchdog.cancel() @@ -1290,7 +1309,8 @@ public enum TestTools { } let snapshotPath = diagnosticSnapshotPath() let watchdog = HangWatchdog( - udid: udid, snapshotPath: snapshotPath, sampleAt: [60, 120], processMatch: resultPath, env: env) + udid: udid, snapshotPath: snapshotPath, sampleAt: HangWatchdog.defaultSampleAt, processMatch: resultPath, + env: env) var result = try await Xcodebuild.run( args, environment: childEnvironment, timeout: timeout, env: env) // The test runner sometimes fails to launch or connect on a busy Mac. The build is @@ -1821,7 +1841,7 @@ public enum TestTools { /// and platform and no file in its repo changed since, so its products can be tested as they are. public static func lastTestBuildIsCurrent( project: String?, scheme: String?, simulator: String?, configuration: String, coverage: Bool = false, - env: Environment = .live + testplan: String? = nil, testIDs: [String] = [], env: Environment = .live ) async -> Bool { guard let resolvedProject = try? await env.session.resolveProject(project), let resolvedScheme = try? await env.session.resolveScheme(scheme, project: resolvedProject) @@ -1832,27 +1852,59 @@ public enum TestTools { } let key = LastResultStore.testBuildKey( scheme: resolvedScheme, configuration: configuration, coverage: coverage, - physicalDevice: AutoDetect.isPhysicalDeviceUDID(destination)) + physicalDevice: AutoDetect.isPhysicalDeviceUDID(destination), testPlan: testplan) let projectDirectory = (resolvedProject as NSString).deletingLastPathComponent let root = RepoRoot.discover(from: projectDirectory) ?? projectDirectory - return LastResultStore.testBuildIsCurrent(project: resolvedProject, key: key, sourceRoot: root) + var targets: [String]? + if !testIDs.isEmpty { + let resolved = await resolveFilters( + testIDs, project: resolvedProject, scheme: resolvedScheme, testplan: testplan, env: env) + targets = await buildTargets( + forFilter: resolved, project: resolvedProject, scheme: resolvedScheme, testplan: testplan, env: env) + } + return LastResultStore.testBuildIsCurrent( + project: resolvedProject, key: key, targets: targets, sourceRoot: root) + } + + /// The test targets a filter needs built, or nil (build them all) when an ID doesn't + /// start with one of the scheme's or plan's `known` test targets. + static func buildTargetsForFilter(_ ids: [String], known: [String]) -> [String]? { + var targets: [String] = [] + for id in ids { + guard let target = TestIDs.components(id).first, known.contains(target) else { return nil } + if !targets.contains(target) { targets.append(target) } + } + return targets.isEmpty ? nil : targets + } + + /// `buildTargetsForFilter` with the known targets read from the scheme or test plan. + static func buildTargets( + forFilter ids: [String], project: String, scheme: String, testplan: String?, env: Environment + ) async -> [String]? { + guard !ids.isEmpty else { return nil } + let known = await AutoDetect.testTargetNames(project: project, scheme: scheme, testplan: testplan, env: env) + guard known.exact else { return nil } + return buildTargetsForFilter(ids, known: known.names) + } + + /// Compile errors as "xcodebuild" failures, each with its file and line. + static func buildErrorFailures(_ issues: [BuildIssueObservation]) -> [TestFailureObservation] { + issues.filter { $0.severity == .error }.map { issue in + TestFailureObservation( + testName: "xcodebuild", + testIdentifier: "xcodebuild", + message: issue.message, + source: "build-for-testing", + messages: [ + FailureMessage(text: issue.message, file: issue.location?.filePath, line: issue.location?.line) + ] + ) + } } /// A failed `TestExecution` describing a build-for-testing failure. static func testExecution(fromBuildFailure result: BuildAndTestResult) -> TestExecution { - var failures: [TestFailureObservation] = (result.buildDiagnostics ?? []) - .filter { $0.severity == .error } - .map { issue in - TestFailureObservation( - testName: "xcodebuild", - testIdentifier: "xcodebuild", - message: issue.message, - source: "build-for-testing.stderr", - messages: [ - FailureMessage(text: issue.message, file: issue.location?.filePath, line: issue.location?.line) - ] - ) - } + var failures = buildErrorFailures(result.buildDiagnostics ?? []) if let detail = result.buildFailureDetail, !detail.isEmpty { if result.xcforgeTimedOut { failures.insert( @@ -1887,7 +1939,8 @@ public enum TestTools { buildDiagnostics: result.buildDiagnostics, hangDiagnosticPath: result.hangDiagnosticPath, hangDiagnosticSummary: result.hangDiagnosticSummary, - xcforgeTimedOut: result.xcforgeTimedOut + xcforgeTimedOut: result.xcforgeTimedOut, + timeoutDetail: result.xcforgeTimedOut ? result.buildFailureDetail : nil ) } @@ -1979,24 +2032,15 @@ public enum TestTools { let resolvedPath: String if let provided = xcresultPath { resolvedPath = provided - } else if let resolvedProject = try? await env.session.resolveProject(project), - let recorded = LastResultStore.latest(project: resolvedProject, kind: .test) - { - // Read this project's last test run instead of re-running the whole suite. - resolvedPath = recorded } else { + // This project's last test run (or failed test build). Never runs the tests itself. let resolvedProject = try await env.session.resolveProject(project) - let resolvedScheme = try await env.session.resolveScheme(scheme, project: resolvedProject) - let resolvedSimulator = try await env.session.resolveSimulator(simulator) - let destination = await AutoDetect.buildDestination(resolvedSimulator) - let path = Self.xcresultPath(prefix: "fail") - let (_, p, _) = try await runTests( - project: resolvedProject, scheme: resolvedScheme, destination: destination, - configuration: "Debug", testplan: nil, filter: nil, - coverage: false, resultPath: path, - env: env - ) - resolvedPath = p + guard let recorded = LastResultStore.latest(project: resolvedProject, kind: .test) else { + throw TestDiscoveryError( + "No test run recorded for \(resolvedProject). Run the tests first (test_sim or `xcforge test run`)," + + " or pass xcresultPath.") + } + resolvedPath = recorded } var failures: [TestFailureObservation] = [] @@ -2007,6 +2051,19 @@ public enum TestTools { failures = parsed } + // A bundle without test failures may be a test build that failed: report its errors. + var buildFailed = false + if failures.isEmpty, let buildJSON = await parseBuildResults(resolvedPath, env: env), + let data = buildJSON.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] + { + let errors = buildErrorFailures(parseBuildIssues(json).issues) + if !errors.isEmpty { + buildFailed = true + failures = errors + } + } + let attachments = failures.isEmpty ? [] : await exportFailureAttachments(resolvedPath, env: env) let screenshots = attachments.map { ScreenshotAttachment(testName: $0.test, path: $0.path) } let consoleByTest = @@ -2018,7 +2075,8 @@ public enum TestTools { failures: failures, screenshots: screenshots, consoleByTest: consoleByTest, - xcresultPath: resolvedPath + xcresultPath: resolvedPath, + buildFailed: buildFailed ) } @@ -2264,7 +2322,15 @@ public enum TestTools { let destination = await AutoDetect.buildDestination(udidForDest) let testBuildKey = LastResultStore.testBuildKey( scheme: resolvedScheme, configuration: configuration, coverage: coverage, - physicalDevice: AutoDetect.isPhysicalDeviceUDID(udidForDest)) + physicalDevice: AutoDetect.isPhysicalDeviceUDID(udidForDest), testPlan: testplan) + // A filtered run builds only the test targets it needs. + var filterTargets: [String]? + if let resolvedFilter { + filterTargets = await buildTargets( + forFilter: splitFilterList(resolvedFilter), project: resolvedProject, scheme: resolvedScheme, + testplan: testplan, env: env) + } + let targetsToBuild = filterTargets // --- Inner helper: run the build-for-testing phase --- func runBuildPhase(diagnose: Bool) async throws -> ( @@ -2274,6 +2340,7 @@ public enum TestTools { return try await runBuildForTesting( project: resolvedProject, scheme: resolvedScheme, destination: destination, configuration: configuration, coverage: coverage, resultPath: resultPath, + testplan: testplan, targets: targetsToBuild, long: long, diagnose: diagnose, udid: udidForDest, timeoutOverride: timeoutSeconds, childEnvironment: childEnvironment, env: env @@ -2374,7 +2441,7 @@ public enum TestTools { buildFailureDetail: detail ) } - LastResultStore.recordTestBuild(project: resolvedProject, key: testBuildKey) + LastResultStore.recordTestBuild(project: resolvedProject, key: testBuildKey, targets: targetsToBuild) } // --- Test phase --- @@ -2623,6 +2690,7 @@ public enum TestTools { } } let buildFailed = buildDiagnostics?.contains { $0.severity == .error } ?? false + if buildFailed { failures = buildErrorFailures(buildDiagnostics ?? []) } if !shellResult.succeeded && failures.isEmpty { let errorLines = shellResult.stderr.split(separator: "\n") @@ -2716,7 +2784,8 @@ public enum TestTools { xcforgeTimedOut: xcforgeTimedOut, slowestTests: slowestTests, knownFailures: (gate && !gating.matchedIDs.isEmpty) ? gating.matchedIDs : nil, - flakyTests: flakyTests + flakyTests: flakyTests, + timeoutDetail: Xcodebuild.timeoutExplanation(shellResult) ) } @@ -2727,6 +2796,7 @@ public enum TestTools { scheme: String? = nil, simulator: String? = nil, testplan: String? = nil, + withoutBuilding: Bool = false, env: Environment = .live ) async throws -> ListTestsResult { let resolvedProject = try await env.session.resolveProject(project) @@ -2767,13 +2837,17 @@ public enum TestTools { if let plan = await env.session.resolveTestPlan(testplan) { enumerateArgs += ["-testPlan", plan] } + // `withoutBuilding` lists from the products of the last build-for-testing instead of + // building again. Building uses the same flags as every other build, so it doesn't + // invalidate them. + if !withoutBuilding { enumerateArgs += compileFlags } enumerateArgs += [ "-enumerate-tests", "-test-enumeration-style", "flat", "-test-enumeration-format", "json", "-test-enumeration-output-path", jsonPath, - "test", ] + enumerateArgs += withoutBuilding ? ["test-without-building"] : ["test"] + compileSettings let enumerateResult = try await Xcodebuild.run( enumerateArgs, timeout: await resolveTestTimeout(long: true, env: env), env: env) @@ -3027,20 +3101,9 @@ public enum TestTools { if execution.buildFailed { lines.append("TEST TARGET BUILD FAILED in \(execution.elapsed)s") - if let diagnostics = execution.buildDiagnostics, !diagnostics.isEmpty { - let errors = diagnostics.filter { $0.severity == .error } - if !errors.isEmpty { - lines.append( - "Build errors (\(errors.count)):\n" - + errors.map { issue in - if let loc = issue.location { - return " \(loc.filePath):\(loc.line ?? 0): \(issue.message)" - } - return " \(issue.message)" - }.joined(separator: "\n")) - } - } else if !execution.failures.isEmpty { - lines.append(execution.failures.map { " \($0.message)" }.joined(separator: "\n")) + if !execution.failures.isEmpty { + lines.append("Build errors:") + lines += TestFailureText.buildErrorLines(execution.failures) } lines.append("xcresult: \(execution.xcresultPath)") if let diagPath = execution.hangDiagnosticPath { @@ -3683,6 +3746,12 @@ public enum TestTools { if result.failures.isEmpty { return .ok("No test failures found.\nxcresult: \(result.xcresultPath)") } + if result.buildFailed { + var lines = ["The last test run failed to build, so no test ran. Build errors:"] + lines += TestFailureText.buildErrorLines(result.failures) + lines.append("xcresult: \(result.xcresultPath)") + return .fail(lines.joined(separator: "\n")) + } var lines = ["\(result.failures.count) test failure(s):", ""] lines += TestFailureText.lines(result.failures, indent: "") // Attachments that couldn't be tied to a failure are still worth a path. diff --git a/Tests/XCForgeKitTests/AgentResultProjectionTests.swift b/Tests/XCForgeKitTests/AgentResultProjectionTests.swift index eed75fb..ee36077 100644 --- a/Tests/XCForgeKitTests/AgentResultProjectionTests.swift +++ b/Tests/XCForgeKitTests/AgentResultProjectionTests.swift @@ -46,30 +46,31 @@ struct AgentResultProjectionTests { ) } - @Test("green run projects to the locked 8-key shape (no knownFailures)") + @Test("green run projects to the locked 9-key shape (no knownFailures)") func greenShape() throws { let exec = makeExecution() let projected = AgentResultProjection.project(exec) let json = try WorkflowJSONRenderer.renderJSON(projected) let obj = try JSONSerialization.jsonObject(with: Data(json.utf8)) as! [String: Any] let keys = Set(obj.keys) - #expect(keys == ["succeeded", "buildOk", "total", "passed", "failed", "skipped", "timedOut", "failures"]) + #expect( + keys == ["succeeded", "buildOk", "total", "passed", "failed", "skipped", "timedOut", "failures", "xcresult"]) #expect(keys.count <= 10) } @Test("encoded JSON has only the locked agent-shape keys at the top level") func slimKeys() throws { - // `knownFailures`, `flaky` and `reason` appear only when they have something to say. + // `knownFailures`, `flaky`, `reason` and `xcresult` appear only when they have something to say. let allowed: Set = [ "succeeded", "buildOk", "total", "passed", "failed", "skipped", - "timedOut", "knownFailures", "failures", "flaky", "reason", + "timedOut", "knownFailures", "failures", "flaky", "reason", "xcresult", ] let exec = makeExecution(knownFailures: ["Suite/k"]) let json = try WorkflowJSONRenderer.renderTestJSON(exec, forAgent: true) let obj = try JSONSerialization.jsonObject(with: Data(json.utf8)) as! [String: Any] let keys = Set(obj.keys) #expect(keys.isSubset(of: allowed)) - #expect(keys.count <= 9) + #expect(keys.count <= 10) } @Test("all failures gated → succeeded preserved, failed:0, knownFailures populated") diff --git a/Tests/XCForgeKitTests/BuildLoopTests.swift b/Tests/XCForgeKitTests/BuildLoopTests.swift new file mode 100644 index 0000000..eca7577 --- /dev/null +++ b/Tests/XCForgeKitTests/BuildLoopTests.swift @@ -0,0 +1,165 @@ +import Foundation +import Testing + +@testable import XCForgeKit + +@Suite("Build loop: simulator lookup, test build scope, reporting", .serialized) +struct BuildLoopTests { + + private typealias Device = AutoDetect.SimulatorDevice + + private func device( + _ name: String, _ udid: String, runtime: String = "iOS-18-0", state: String = "Shutdown", + available: Bool = true + ) -> Device { + Device(name: name, udid: udid, runtime: runtime, state: state, isAvailable: available) + } + + private func tempDir() -> String { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("xcforge-loop-\(UUID().uuidString)", isDirectory: true).path + try? FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + return dir + } + + // MARK: - One simulator lookup + + @Test("a name resolves to the booted match, else the newest OS, never a prefix match") + func pickSimulator() throws { + let devices = [ + device("iPhone 16", "OLD", runtime: "iOS-18-2"), + device("iPhone 16", "NEW", runtime: "iOS-18-10"), + device("iPhone 16 Pro", "PRO", runtime: "iOS-26-0", state: "Booted"), + device("iPhone 16", "GONE", runtime: "iOS-27-0", available: false), + ] + #expect(try AutoDetect.pickSimulator(named: "iphone 16", from: devices).udid == "NEW") + #expect(try AutoDetect.pickSimulator(named: "iPhone 16 Pro", from: devices).udid == "PRO") + + let withBooted = devices + [device("iPhone 16", "BOOTED", runtime: "iOS-17-0", state: "Booted")] + #expect(try AutoDetect.pickSimulator(named: "iPhone 16", from: withBooted).udid == "BOOTED") + + #expect(throws: (any Error).self) { try AutoDetect.pickSimulator(named: "iPhone", from: devices) } + let tie = [device("iPad", "A"), device("iPad", "B")] + #expect(throws: MultipleSimulatorMatchError.self) { try AutoDetect.pickSimulator(named: "iPad", from: tie) } + } + + @Test("runtime versions compare numerically") + func runtimeVersions() { + #expect(AutoDetect.runtimeVersion("iOS-18-10") == [18, 10]) + #expect(AutoDetect.runtimeVersion("iOS-18-2").lexicographicallyPrecedes(AutoDetect.runtimeVersion("iOS-18-10"))) + } + + @Test("compiling picks a booted iOS simulator, else the newest iPhone, without needing one booted") + func compileSimulator() { + let none = [ + device("iPad Air", "IPAD", runtime: "iOS-26-0"), + device("iPhone 15", "P15", runtime: "iOS-17-5"), + device("iPhone 17", "P17", runtime: "iOS-26-0"), + device("Apple Watch", "WATCH", runtime: "watchOS-11-0"), + ] + #expect(AutoDetect.simulatorForCompile(from: none) == "P17") + let booted = none + [ + device("iPhone 15", "B1", runtime: "iOS-17-5", state: "Booted"), + device("iPad Air", "B2", runtime: "iOS-26-0", state: "Booted"), + ] + #expect(AutoDetect.simulatorForCompile(from: booted) == "B2") + #expect(AutoDetect.simulatorForCompile(from: []) == nil) + } + + // MARK: - Test build scope + + @Test("a filtered run builds only its targets, and only when every ID names a known target") + func filterTargets() { + let known = ["AppTests", "AppUITests"] + let one = TestTools.buildTargetsForFilter(["AppTests/A/a()", "AppTests/B/b()"], known: known) + #expect(one == ["AppTests"]) + let both = TestTools.buildTargetsForFilter(["AppTests/A/a()", "AppUITests/U"], known: known) + #expect(both == ["AppTests", "AppUITests"]) + #expect(TestTools.buildTargetsForFilter(["Suite/test()"], known: known) == nil) + #expect(TestTools.buildTargetsForFilter([], known: known) == nil) + } + + @Test("a test build covers a rerun only when it built the targets the rerun needs") + func testBuildScope() throws { + let root = tempDir() + defer { try? FileManager.default.removeItem(atPath: root) } + let project = (root as NSString).appendingPathComponent("App-\(UUID().uuidString).xcodeproj") + defer { try? FileManager.default.removeItem(atPath: LastResultStore.filePath(for: project)) } + let key = LastResultStore.testBuildKey( + scheme: "App", configuration: "Debug", coverage: false, physicalDevice: false, testPlan: "quick") + let past = Date().addingTimeInterval(-60) + + func current(_ targets: [String]?, plan: String? = "quick") -> Bool { + let key = LastResultStore.testBuildKey( + scheme: "App", configuration: "Debug", coverage: false, physicalDevice: false, testPlan: plan) + return LastResultStore.testBuildIsCurrent(project: project, key: key, targets: targets, sourceRoot: root) + } + + LastResultStore.recordTestBuild(project: project, key: key, targets: ["AppTests"], now: past) + #expect(current(["AppTests"])) + #expect(!current(["AppUITests"])) + #expect(!current(nil)) + + LastResultStore.recordTestBuild(project: project, key: key, targets: nil, now: past) + #expect(current(["AppUITests"])) + #expect(!current(nil, plan: "ci")) + } + + // MARK: - Reporting + + @Test("a failed test build becomes the latest test result, so test failures can't go stale") + func failedTestBuildRecorded() throws { + let dir = tempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let project = (dir as NSString).appendingPathComponent("App-\(UUID().uuidString).xcodeproj") + defer { try? FileManager.default.removeItem(atPath: LastResultStore.filePath(for: project)) } + let testBundle = (dir as NSString).appendingPathComponent("t.xcresult") + let buildBundle = (dir as NSString).appendingPathComponent("b.xcresult") + try FileManager.default.createDirectory(atPath: testBundle, withIntermediateDirectories: true) + try FileManager.default.createDirectory(atPath: buildBundle, withIntermediateDirectories: true) + + LastResultStore.recordFromArguments( + ["-project", project, "-resultBundlePath", testBundle, "test-without-building"], succeeded: false) + LastResultStore.recordFromArguments( + ["-project", project, "-resultBundlePath", buildBundle, "build-for-testing"], succeeded: true) + #expect(LastResultStore.latest(project: project, kind: .test) == testBundle) + + LastResultStore.recordFromArguments( + ["-project", project, "-resultBundlePath", buildBundle, "build-for-testing"], succeeded: false) + #expect(LastResultStore.latest(project: project, kind: .test) == buildBundle) + } + + @Test("compile errors render with file and line, capped") + func buildErrorLines() { + let errors = TestTools.buildErrorFailures([ + TestTools.BuildIssueObservation( + severity: .error, message: "cannot find 'x' in scope", + location: SourceLocation(filePath: "AppTests/ATests.swift", line: 12), source: "xcresult"), + TestTools.BuildIssueObservation(severity: .warning, message: "unused", location: nil, source: "xcresult"), + ]) + #expect(errors.count == 1) + #expect(TestFailureText.buildErrorLines(errors) == [" AppTests/ATests.swift:12: cannot find 'x' in scope"]) + let many = Array(repeating: errors[0], count: 25) + let lines = TestFailureText.buildErrorLines(many, limit: 20) + #expect(lines.count == 21) + #expect(lines.last == " +5 more errors") + } + + @Test("agent JSON names the result bundle and why a timed-out run stopped") + func agentTimeoutAndBundle() throws { + let failure = TestTools.TestFailureObservation( + testName: "a()", testIdentifier: "AppTests/A/a()", message: "boom", source: "xcresult") + let execution = TestTools.TestExecution( + succeeded: false, elapsed: "600", xcresultPath: "/tmp/run.xcresult", scheme: "App", simulator: "iPhone", + totalTestCount: 3, passedTestCount: 1, failedTestCount: 1, skippedTestCount: 0, expectedFailureCount: 0, + failures: [failure], deviceName: nil, osVersion: nil, screenshotPaths: [], hasStructuredSummary: true, + buildFailed: false, buildDiagnostics: nil, xcforgeTimedOut: true, + timeoutDetail: "Killed after 600s with no output (idle timeout). Likely hung.") + let projected = AgentResultProjection.project(execution) + #expect(projected.timedOut) + #expect(projected.reason?.contains("idle timeout") == true) + #expect(projected.xcresult == "/tmp/run.xcresult") + let json = try WorkflowJSONRenderer.renderJSON(projected) + #expect(json.contains("\"xcresult\"")) + } +} From a77c500cab8d7f8eccc04beb2ef7dead564a14ec Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 16:15:00 +0000 Subject: [PATCH 09/28] test: format compile-simulator test for swift-format Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EtxtFPs6UBAWXqEpkj2W2N --- Tests/XCForgeKitTests/BuildLoopTests.swift | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Tests/XCForgeKitTests/BuildLoopTests.swift b/Tests/XCForgeKitTests/BuildLoopTests.swift index eca7577..5f4fb9a 100644 --- a/Tests/XCForgeKitTests/BuildLoopTests.swift +++ b/Tests/XCForgeKitTests/BuildLoopTests.swift @@ -58,7 +58,8 @@ struct BuildLoopTests { device("Apple Watch", "WATCH", runtime: "watchOS-11-0"), ] #expect(AutoDetect.simulatorForCompile(from: none) == "P17") - let booted = none + [ + var booted = none + booted += [ device("iPhone 15", "B1", runtime: "iOS-17-5", state: "Booted"), device("iPad Air", "B2", runtime: "iOS-26-0", state: "Booted"), ] From 8419c69fdd471fc836f1c7059196a32320760f7c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 16:30:16 +0000 Subject: [PATCH 10/28] feat: run the app and see what happened (bundle 9) - launch_app, build_run_sim, sim launch and build run check the app is still running 2s after launch and report the crash (exception, reason, top frames, .ips path) when it isn't - Launch args, env, deep link and terminate options; new open_url and sim openurl - build_run_sim boots after a successful build, waits for boot, and installs the scheme's application target from -showBuildSettings -json - CLI log and console capture run in the background and survive the start command; reads default to the newest 200 lines and say what was left out; wait_for_log no longer clears the buffer - crashes topic matches error/fault lines instead of default-level ones - Auto-detect ignores the workspace inside .xcodeproj and picks the project-named scheme; clean takes a configuration and can delete this project's DerivedData - A new process loads the last build's bundle ID and app path Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EtxtFPs6UBAWXqEpkj2W2N --- CHANGELOG.md | 12 + Skills/xcforge/references/auto-detection.md | 5 +- Skills/xcforge/references/build-tools.md | 13 +- Skills/xcforge/references/cli-commands.md | 18 +- Skills/xcforge/references/log-console.md | 12 +- Skills/xcforge/references/simulator-tools.md | 17 +- .../Commands/Build/BuildCommand.swift | 30 +- .../Commands/Build/BuildRenderer.swift | 3 + .../Commands/Console/ConsoleCommand.swift | 66 ++-- .../Commands/Console/ConsoleRenderer.swift | 3 + .../XCForgeCLI/Commands/Log/LogCommand.swift | 71 ++-- .../XCForgeCLI/Commands/Sim/SimCommand.swift | 47 ++- Sources/XCForgeKit/Support/AppLiveness.swift | 155 +++++++++ .../XCForgeKit/Support/DetachedCapture.swift | 180 ++++++++++ .../XCForgeKit/Support/ProjectResolver.swift | 21 +- .../XCForgeKit/Support/WorkspaceContext.swift | 14 +- Sources/XCForgeKit/Tools/BuildProvider.swift | 323 +++++++++--------- .../XCForgeKit/Tools/ConsoleProvider.swift | 38 +-- Sources/XCForgeKit/Tools/LogProvider.swift | 82 +++-- .../XCForgeKit/Tools/SimulatorProvider.swift | 140 +++++++- Tests/XCForgeKitTests/LogToolsTests.swift | 16 +- Tests/XCForgeKitTests/RunLoopTests.swift | 186 ++++++++++ 22 files changed, 1127 insertions(+), 325 deletions(-) create mode 100644 Sources/XCForgeKit/Support/AppLiveness.swift create mode 100644 Sources/XCForgeKit/Support/DetachedCapture.swift create mode 100644 Tests/XCForgeKitTests/RunLoopTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e18cfb..050275e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - **Test run options:** `retries`, `iterations`, `untilFailure`, `parallel`, `testTimeoutSeconds`, `skipBuild`, `includeConsole`, `env` and `timeoutSeconds` on `test_sim` and `build_and_test`, with matching flags (`--retries`, `--iterations`, `--until-failure`, `--parallel/--no-parallel`, `--test-timeout`, `--no-build`, `--include-console`, `--env`, `--timeout-seconds`) on `test run`, `build-test` and `test rerun-failed` - Tests that fail and then pass on retry are listed as flaky (`flaky` in agent JSON) - `test rerun-failed` reuses the recorded project, test plan, configuration and env, and skips build-for-testing when no source file changed since the last one (`--build` forces it) +- **Launches report a crash at startup.** `launch_app`, `build_run_sim`, `sim launch` and `build run` check the app is still running 2s after launch; when it isn't, they fail with the exception, reason, crashed thread's top frames and the `.ips` report path +- Launch arguments, environment and a deep link: `args`, `env` (KEY=VALUE), `url` and `terminate` on `launch_app` and `build_run_sim`; `--arg`, `--env`, `--url` and `--no-terminate` on `sim launch` and `build run`; `--env` on `console launch` +- `open_url` / `sim openurl` open a URL or deep link on a simulator +- `clean` / `build clean` take `configuration` and `derivedData: true` / `--derived-data`, which deletes this project's DerivedData folder (and only that one) ### Changed - **One test ID format.** Results, `list_tests`, last-failures and known-failures all use `Target/Suite/test()` (Swift Testing) or `Target/Class/testMethod` (XCTest), and filters accept it as printed. The target is added to short IDs from the scheme's or test plan's test targets instead of guessed, and the `test()()` spelling xcodebuild needs is written for you @@ -41,6 +45,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - Simulator names resolve the same way everywhere: exact name, the booted match first, else the newest OS; two matches on the same OS ask for a UDID. Prefix matches (`iPhone 16` → `iPhone 16 Pro`) are gone - A filter that matches no test lists suggestions from the last build instead of building again - `build run` exits non-zero when the app wasn't launched +- `build_run_sim` boots the simulator once the build succeeds (not alongside it), waits for the boot to finish before installing, and installs the scheme's application target from `-showBuildSettings -json` instead of whichever product was listed last or a `.app` found anywhere in DerivedData +- **CLI log and console capture run in the background.** `log start` and `console launch` keep streaming to `~/.xcforge/capture/` after the command exits (`XCFORGE_CAPTURE_DIR` moves it), so `log read`, `log wait`, `console read` and the `stop` commands in later invocations see it. Before, the capture died with the `start` command +- `read_logs`, `read_app_console`, `log read` and `console read` return the newest 200 lines by default (`last: 0` for all) and say how many earlier lines were left out; over-long output keeps the end instead of the start +- `wait_for_log` no longer clears the buffer `read_logs` reads +- The `crashes` log topic matches error and fault lines; it used to match default-level (`Df`) lines, which is most of the log +- Auto-detect ignores the `project.xcworkspace` inside every `.xcodeproj`, picks the scheme named after the project (or the only non-test scheme) when there are several, and gives `xcodebuild -list` 60s instead of 15s +- `clean` cleans the configuration and simulator build folder xcforge builds into, and is limited by the idle timeout instead of a fixed 60s +- A new CLI process picks up the last build's bundle ID and app path, so `sim launch` and `sim install` work right after `build run` without passing them - **Cancelling stops the build.** An MCP cancel, a client disconnect, Ctrl-C, SIGTERM or SIGHUP now stops the xcodebuild (or other command) xcforge started, together with every process it spawned. Timeouts kill the whole process tree too, so compiler and test-runner children no longer outlive the call and hold DerivedData ("database is locked" on the next run) - Two MCP calls that would run xcodebuild on the same DerivedData folder (or the same project's default one) at once now run one after the other, first come first served - An explicit `project` in another repo or worktree uses the `.xcforge.yaml` next to it, for both session defaults (scheme, simulator, test plan) and xcodebuild options (DerivedData, lock, idle timeout) diff --git a/Skills/xcforge/references/auto-detection.md b/Skills/xcforge/references/auto-detection.md index b970415..8f43991 100644 --- a/Skills/xcforge/references/auto-detection.md +++ b/Skills/xcforge/references/auto-detection.md @@ -77,12 +77,13 @@ xcforge init --force # overwrite an existing file - Scans working directory for `.xcodeproj` and `.xcworkspace` files - If exactly one found, uses it automatically - If multiple found, returns the list and asks for selection -- Prefers `.xcworkspace` over `.xcodeproj` when both exist (CocoaPods, SPM workspace) +- Prefers `.xcworkspace` over `.xcodeproj` when both exist (CocoaPods, SPM workspace). The `project.xcworkspace` inside every `.xcodeproj` doesn't count ### scheme - Queries `xcodebuild -list` for the resolved project - If exactly one scheme, uses it -- If multiple, returns the list +- If several, uses the one named after the project, else the only one that isn't a test or `Pods-` scheme +- Otherwise returns the list ### simulator - Finds currently booted simulator via `simctl list devices` diff --git a/Skills/xcforge/references/build-tools.md b/Skills/xcforge/references/build-tools.md index 02df7db..ee6ca96 100644 --- a/Skills/xcforge/references/build-tools.md +++ b/Skills/xcforge/references/build-tools.md @@ -45,7 +45,7 @@ Build for iOS Simulator. Returns structured errors from xcresult (not raw xcodeb ## build_run_sim -Build + boot + install + launch in one call. Runs a parallel 2-phase pipeline: build and boot happen simultaneously, then install and launch. ~9s faster than calling each tool sequentially. +Build, boot, install and launch in one call (Xcode's Cmd+R). The simulator boots once the build succeeds; the app installed is the scheme's application target (from `-showBuildSettings -json`), not an extension or framework. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| @@ -53,21 +53,24 @@ Build + boot + install + launch in one call. Runs a parallel 2-phase pipeline: b | `scheme` | No | Auto-detect | Scheme name | | `simulator` | No | Auto-detect (booted) | Simulator name or UDID | | `configuration` | No | Debug | Build configuration | +| `args` | No | — | Launch arguments for the app | +| `env` | No | — | Environment for the app, `KEY=VALUE` strings | +| `url` | No | — | URL or deep link to open once the app is running | -**Returns:** Bundle ID, app path, simulator UDID, build duration. On failure: structured build errors. - -This is the **Cmd+R equivalent** — the single most common tool call for iOS development. +**Returns:** Bundle ID, app path, PID, timings. On failure: structured build errors, or, when the app dies within 2s of launch, `App running: false` with the exception, reason, top frames of the crashed thread and the crash report path. --- ## clean -Clean build artifacts (DerivedData for the project). +Clean the scheme's simulator build products for a configuration. `derivedData: true` also deletes this project's DerivedData folder, found from its `BUILD_ROOT` (never the whole DerivedData directory); that is the fix for "database is locked" or a stale index. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `project` | No | Auto-detect | Path to .xcodeproj or .xcworkspace | | `scheme` | No | Auto-detect | Scheme name | +| `configuration` | No | Debug | Build configuration | +| `derivedData` | No | false | Also delete this project's DerivedData folder | --- diff --git a/Skills/xcforge/references/cli-commands.md b/Skills/xcforge/references/cli-commands.md index 0b61a3b..4e3c85f 100644 --- a/Skills/xcforge/references/cli-commands.md +++ b/Skills/xcforge/references/cli-commands.md @@ -52,9 +52,12 @@ xcforge build --json # Machine-readable JSON output | `--simulator ` | Simulator name or UDID. Auto-detected from booted simulator | | `--configuration ` | Build configuration (Debug/Release). Default: Debug | | `--diagnose` | Build-only with structured diagnostics (skips boot/install/launch) | +| `--arg ` | Launch argument for the app (repeatable) | +| `--env KEY=VALUE` | Environment variable for the app (repeatable) | +| `--url ` | URL or deep link to open once the app is running | | `--json` | Machine-readable JSON output | -**Pipeline behavior:** On build success, automatically boots the simulator (if not already booted), installs the app, and launches it. On build failure, stops immediately with build errors. Persists `bundleId` and `appPath` to `defaults.json` so subsequent `sim install` / `sim launch` calls auto-detect across process boundaries. +**Pipeline behavior:** On build success, boots the simulator (if not already booted) and waits for it, installs the scheme's application target, launches it, and checks it is still running 2s later. A crash at launch fails the command with the crash reason and frames. On build failure, stops immediately with build errors. Persists `bundleId` and `appPath` to `defaults.json` so subsequent `sim install` / `sim launch` calls auto-detect across process boundaries. **JSON output:** With `--json`, emits a `BuildRunResult` with `build`, `boot`, `install`, `launch` phase statuses plus `appPid` and `appRunning` fields. @@ -105,6 +108,7 @@ Clean Xcode build artifacts for a project/scheme. ```bash xcforge build clean xcforge build clean --project MyApp.xcodeproj --scheme MyApp +xcforge build clean --derived-data # Also delete this project's DerivedData folder xcforge build clean --json ``` @@ -327,7 +331,9 @@ xcforge sim info --simulator "iPhone 16 Pro" # Get metrics for specific simu xcforge sim boot "iPhone 16 Pro" # Boot a simulator xcforge sim shutdown "iPhone 16 Pro" # Shutdown (or "all") xcforge sim install --app-path /path/to/App.app # Install app (auto-detects sim) -xcforge sim launch --bundle-id com.app.id # Launch app (auto-detects sim) +xcforge sim launch --bundle-id com.app.id # Launch app; fails with the crash if it dies within 2s +xcforge sim launch --arg -UITest --env API=stub --url myapp://screen # Args, env, deep link +xcforge sim openurl myapp://settings # Open a URL or deep link xcforge sim terminate --bundle-id com.app.id # Terminate app xcforge sim clone "iPhone 16 Pro" --name "Clone" # Clone simulator xcforge sim erase "iPhone 16 Pro" # Erase to factory state @@ -395,7 +401,7 @@ All subcommands support `--json`. `--path` defaults to current directory. Stream, read, and wait on simulator logs. Four subcommands — `read` is the default. ```bash -xcforge log start # Start capture (smart mode, debug level) +xcforge log start # Start a background capture (smart mode, debug level) xcforge log start --mode app # App-only logs + crashes xcforge log start --mode verbose # Unfiltered system logs xcforge log start --process MyApp # Filter by process name @@ -403,13 +409,14 @@ xcforge log start --subsystem com.myapp # Filter by subsystem xcforge log read # Read with topic filtering (app + crashes) xcforge log read --include network # Add network topic xcforge log read --include lifecycle --last 50 # Last 50 lifecycle + app lines +xcforge log read --last 0 # Everything (default: newest 200 lines) xcforge log read --clear # Clear buffer after reading xcforge log stop # Stop capture xcforge log wait --pattern "error.*timeout" # Wait for regex pattern xcforge log wait --pattern "launched" --timeout 10 ``` -All subcommands support `--json`. +All subcommands support `--json`. The capture keeps running in the background after `log start` returns, writing to `~/.xcforge/capture/`, until `log stop`. --- @@ -421,13 +428,14 @@ Launch, read, and stop app console output capture (print/NSLog). Three subcomman xcforge console launch # Launch app with console capture xcforge console launch --bundle-id com.app.id # Explicit bundle ID xcforge console launch --args "--verbose" # Pass launch args to app +xcforge console launch --env API=stub # Environment for the app (repeatable) xcforge console read # Read stdout + stderr xcforge console read --stream stdout --last 20 # Last 20 stdout lines xcforge console read --clear # Clear buffer after reading xcforge console stop # Stop capture and terminate app ``` -All subcommands support `--json`. Auto-detects simulator and bundle ID from session state. +All subcommands support `--json`. Auto-detects simulator and bundle ID from session state. The console keeps streaming in the background after `console launch` returns, until `console stop`; reads return the newest 200 lines per stream by default. --- diff --git a/Skills/xcforge/references/log-console.md b/Skills/xcforge/references/log-console.md index fccd01a..0c8e132 100644 --- a/Skills/xcforge/references/log-console.md +++ b/Skills/xcforge/references/log-console.md @@ -32,7 +32,9 @@ Start capturing OS logs from the simulator. start_log_capture(subsystem: "com.apple.SwiftUI") ``` -**Important:** Start capture BEFORE reproducing the issue. Logs are not retroactive. +Logs are not retroactive: only what happens after the capture starts is recorded. + +From the CLI, `xcforge log start` runs the capture in the background, writing to `~/.xcforge/capture/` (`XCFORGE_CAPTURE_DIR` moves it), until `xcforge log stop`; `log read` and `log wait` in later commands read it. --- @@ -51,7 +53,7 @@ Read buffered logs with topic filtering. Returns a topic menu showing available | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `include` | No | `["app", "crashes"]` | Array of topic names to include | -| `last` | No | — | Only return last N lines (applied after topic filtering) | +| `last` | No | 200 | Return the last N lines, after topic filtering; 0 returns all. The header says how many earlier lines were left out | | `clear` | No | false | Clear buffer after reading | **Response format:** @@ -68,7 +70,7 @@ Hint: include=["network"] to add SSL/TLS + background transfer logs | Topic | Always On | Matches | Use Case | |-------|-----------|---------|----------| | `app` | Yes | subsystem == bundleId OR process == appName | Your app's os_log, print(), NSLog() | -| `crashes` | Yes | Fault-level logs from any process | Crash detection | +| `crashes` | Yes | Error and fault lines from any process | Crash detection | | `network` | No | trustd, nsurlsessiond | SSL/TLS certificates, background transfers | | `lifecycle` | No | runningboardd, com.apple.runningboard.* | Jetsam, memory pressure, app kills | | `springboard` | No | SpringBoard process | Push notifications, app state changes | @@ -91,7 +93,7 @@ Wait for a specific log pattern with timeout. Eliminates sleep() hacks. | `simulator` | No | Auto-detect | Simulator (used if log capture not already running) | | `subsystem` | No | — | Filter by subsystem (used if log capture not already running) | -**Returns:** The matching log line, or timeout error. +**Returns:** The matching lines that arrived after the call started, or a timeout error. It doesn't clear what `read_logs` returns. **Use case:** Wait for "app launched" signal, wait for network request completion, wait for crash after specific action. @@ -121,7 +123,7 @@ Read buffered console output. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| -| `last` | No | — | Only return last N lines per stream | +| `last` | No | 200 | Return the last N lines per stream; 0 returns all | | `clear` | No | false | Clear buffer after reading | | `stream` | No | `both` | Which stream: `stdout`, `stderr`, or `both` | diff --git a/Skills/xcforge/references/simulator-tools.md b/Skills/xcforge/references/simulator-tools.md index f33109c..5f4c8c9 100644 --- a/Skills/xcforge/references/simulator-tools.md +++ b/Skills/xcforge/references/simulator-tools.md @@ -49,12 +49,27 @@ Install a .app bundle on a simulator. ## launch_app -Launch an installed app. +Launch an installed app, then check it is still running 2s later. When it isn't, the call fails with the exception, reason, crashed thread's top frames and the `.ips` crash report path. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `simulator` | No | Auto-detect (booted) | Simulator name or UDID | | `bundle_id` | No | Auto-detect (last build) | App bundle identifier | +| `args` | No | — | Launch arguments for the app | +| `env` | No | — | Environment for the app, `KEY=VALUE` strings | +| `url` | No | — | URL or deep link to open once the app is running | +| `terminate` | No | true | Terminate a running copy first | + +--- + +## open_url + +Open a URL or deep link on a simulator. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `url` | **Yes** | — | URL or deep link | +| `simulator` | No | Auto-detect (booted) | Simulator name or UDID | --- diff --git a/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift b/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift index e7aeeec..f26236c 100644 --- a/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift +++ b/Sources/XCForgeCLI/Commands/Build/BuildCommand.swift @@ -101,6 +101,15 @@ struct BuildRun: AsyncParsableCommand { @Flag(help: "Extract structured diagnostics (errors, warnings with file locations).") var diagnose = false + @Option(name: .customLong("arg"), help: "Launch argument for the app (repeatable).") + var launchArgs: [String] = [] + + @Option(name: .customLong("env"), help: "Environment variable for the app, KEY=VALUE (repeatable).") + var launchEnv: [String] = [] + + @Option(help: "URL or deep link to open once the app is running.") + var url: String? + @Flag(help: "Emit the result as machine-readable JSON.") var json = false @@ -174,15 +183,14 @@ struct BuildRun: AsyncParsableCommand { installStatus = installResult.succeeded ? "ok" : "failed: \(installResult.message)" if installResult.succeeded, let bundleId = execution.bundleId { + // The launch reports whether the app survived its first seconds, and why not. let launchResult = await SimTools.executeLaunchApp( - simulator: resolvedSimulator, bundleId: bundleId, env: env) + simulator: resolvedSimulator, bundleId: bundleId, args: launchArgs, environment: launchEnv, url: url, + env: env) launchStatus = launchResult.succeeded ? "ok" : "failed: \(launchResult.message)" if launchResult.succeeded { - // simctl launch prints ": " — extract PID from message - appPid = launchResult.message - .split(separator: "\n").last - .flatMap { $0.split(separator: ":").last } - .map { String($0).trimmingCharacters(in: .whitespaces) } + appPid = launchResult.message.split(separator: "\n").lazy + .compactMap { AppLiveness.pid(fromLaunchOutput: String($0)) }.first.map { String($0) } appRunning = true } } else if !installResult.succeeded { @@ -279,6 +287,12 @@ struct BuildClean: AsyncParsableCommand { @Option(help: "Xcode scheme name. Auto-detected if omitted.") var scheme: String? + @Option(help: "Build configuration (Debug/Release). Default: Debug") + var configuration: String? + + @Flag(help: "Also delete this project's DerivedData folder (fixes \"database is locked\" and stale indexes).") + var derivedData = false + @Flag(help: "Emit the result as machine-readable JSON.") var json = false @@ -294,7 +308,9 @@ struct BuildClean: AsyncParsableCommand { let execution = try await BuildTools.executeClean( project: project, - scheme: scheme + scheme: scheme, + configuration: configuration, + deleteDerivedData: derivedData ) if useJSON { diff --git a/Sources/XCForgeCLI/Commands/Build/BuildRenderer.swift b/Sources/XCForgeCLI/Commands/Build/BuildRenderer.swift index 6b12698..ab8675a 100644 --- a/Sources/XCForgeCLI/Commands/Build/BuildRenderer.swift +++ b/Sources/XCForgeCLI/Commands/Build/BuildRenderer.swift @@ -174,6 +174,9 @@ enum BuildRenderer { lines.append("Project: \(execution.project)") lines.append("Scheme: \(execution.scheme)") + if let removed = execution.removedDerivedData { + lines.append("Deleted DerivedData: \(removed)") + } if let error = execution.error, !error.isEmpty { lines.append("") diff --git a/Sources/XCForgeCLI/Commands/Console/ConsoleCommand.swift b/Sources/XCForgeCLI/Commands/Console/ConsoleCommand.swift index be0f422..13b9969 100644 --- a/Sources/XCForgeCLI/Commands/Console/ConsoleCommand.swift +++ b/Sources/XCForgeCLI/Commands/Console/ConsoleCommand.swift @@ -35,13 +35,16 @@ struct ConsoleLaunch: AsyncParsableCommand { @Option(help: "Space-separated launch arguments for the app.") var args: String? + @Option(help: "Environment variable for the app, KEY=VALUE (repeatable).") + var env: [String] = [] + @Flag(help: "Emit the result as machine-readable JSON.") var json = false mutating func run() async throws { let useJSON = shouldOutputJSON(flag: json) - let env = Environment.live - guard let resolvedBundleId = await env.session.resolveBundleId(bundleId) else { + let environment = Environment.live + guard let resolvedBundleId = await environment.session.resolveBundleId(bundleId) else { let result = ConsoleResult( succeeded: false, message: "Missing bundle_id — provide --bundle-id or run build first", @@ -58,8 +61,10 @@ struct ConsoleLaunch: AsyncParsableCommand { let launchArgs = args?.split(separator: " ").map(String.init) ?? [] let sim: String + let childEnvironment: [String: String] do { - sim = try await env.session.resolveSimulator(simulator) + sim = try await environment.session.resolveSimulator(simulator) + childEnvironment = try SimTools.parseEnvironment(env) } catch { let result = ConsoleResult( succeeded: false, @@ -76,9 +81,17 @@ struct ConsoleLaunch: AsyncParsableCommand { do { let udid = try await SimTools.resolveSimulator(sim) - let msg = try await AppConsole.shared.launch( - simulator: udid, bundleId: resolvedBundleId, args: launchArgs - ) + // The console streams to files in the background, so `console read` in a later + // command still sees it. + var simctlArgs = ["simctl", "launch", "--console", "--terminate-running-process", udid, resolvedBundleId] + simctlArgs += launchArgs + var simctlEnvironment: [String: String] = [:] + for (key, value) in childEnvironment { + simctlEnvironment["SIMCTL_CHILD_" + key] = value + } + try DetachedCapture.console.start( + arguments: simctlArgs, environment: simctlEnvironment, captureStderr: true, bundleId: resolvedBundleId) + let msg = "Console capture started for \(resolvedBundleId). `xcforge console read` reads it." let result = ConsoleResult( succeeded: true, message: msg, @@ -113,7 +126,7 @@ struct ConsoleRead: AsyncParsableCommand { abstract: "Read captured console output from a running app." ) - @Option(help: "Only return last N lines per stream.") + @Option(help: "Return the last N lines per stream. Default: 200; 0 returns all.") var last: Int? @Flag(help: "Clear buffer after reading.") @@ -127,30 +140,23 @@ struct ConsoleRead: AsyncParsableCommand { mutating func run() async throws { let useJSON = shouldOutputJSON(flag: json) - let output = await AppConsole.shared.read(last: last, clear: clear) - - let filteredStdout: [String]? - let filteredStderr: [String]? - - switch stream { - case "stdout": - filteredStdout = output.stdout - filteredStderr = nil - case "stderr": - filteredStdout = nil - filteredStderr = output.stderr - default: - filteredStdout = output.stdout - filteredStderr = output.stderr - } + let capture = DetachedCapture.console + let stdoutTail = CaptureTail.tail(capture.lines(.stdout), last: last) + let stderrTail = CaptureTail.tail(capture.lines(.stderr), last: last) + if clear { capture.clear() } + + var message = clear ? "Buffer cleared after reading" : "Console output read" + if !capture.exists { message = "No console capture. Start one with `xcforge console launch`." } + let omitted = (stream == "stderr" ? 0 : stdoutTail.omitted) + (stream == "stdout" ? 0 : stderrTail.omitted) + if let note = CaptureTail.omittedNote(omitted) { message += " \(note)" } let result = ConsoleResult( succeeded: true, - message: clear ? "Buffer cleared after reading" : "Console output read", - stdout: filteredStdout, - stderr: filteredStderr, - isRunning: output.isRunning, - bundleId: output.bundleId + message: message, + stdout: stream == "stderr" ? nil : stdoutTail.lines, + stderr: stream == "stdout" ? nil : stderrTail.lines, + isRunning: capture.isRunning, + bundleId: capture.state()?.bundleId ) if useJSON { @@ -172,11 +178,11 @@ struct ConsoleStop: AsyncParsableCommand { mutating func run() async throws { let useJSON = shouldOutputJSON(flag: json) - await AppConsole.shared.stop() + let stopped = DetachedCapture.console.stop() let result = ConsoleResult( succeeded: true, - message: "App console stopped", + message: stopped ? "App console stopped" : "No console capture was running", stdout: nil, stderr: nil, isRunning: false, bundleId: nil ) diff --git a/Sources/XCForgeCLI/Commands/Console/ConsoleRenderer.swift b/Sources/XCForgeCLI/Commands/Console/ConsoleRenderer.swift index ed57488..5d19bdf 100644 --- a/Sources/XCForgeCLI/Commands/Console/ConsoleRenderer.swift +++ b/Sources/XCForgeCLI/Commands/Console/ConsoleRenderer.swift @@ -23,6 +23,9 @@ enum ConsoleRenderer { let status = (result.isRunning ?? false) ? "running" : "stopped" let header = "App: \(result.bundleId ?? "?") [\(status)]" lines.append(header) + if result.message != "Console output read" && result.message != "Buffer cleared after reading" { + lines.append(result.message) + } if stream == "stdout" || stream == "both" { let stdoutLines = result.stdout ?? [] diff --git a/Sources/XCForgeCLI/Commands/Log/LogCommand.swift b/Sources/XCForgeCLI/Commands/Log/LogCommand.swift index 651780b..13748ba 100644 --- a/Sources/XCForgeCLI/Commands/Log/LogCommand.swift +++ b/Sources/XCForgeCLI/Commands/Log/LogCommand.swift @@ -75,8 +75,10 @@ struct LogStart: AsyncParsableCommand { ) do { - try await LogCapture.shared.start(arguments: logArgs, mode: mode) - let msg = "Log capture started (\(note))" + // Each CLI command is its own process: the capture runs on in the background, + // writing to a file, until `log stop`. + try DetachedCapture.log.start(arguments: logArgs, mode: mode) + let msg = "Log capture started in the background (\(note)). `xcforge log read` reads it; `log stop` ends it." if useJSON { print( try WorkflowJSONRenderer.renderJSON( @@ -111,8 +113,7 @@ struct LogStop: AsyncParsableCommand { mutating func run() async throws { let useJSON = shouldOutputJSON(flag: json) - await LogCapture.shared.stop() - let msg = "Log capture stopped" + let msg = DetachedCapture.log.stop() ? "Log capture stopped" : "No log capture was running" if useJSON { print( try WorkflowJSONRenderer.renderJSON( @@ -138,7 +139,7 @@ struct LogRead: AsyncParsableCommand { ) var include: [String] = [] - @Option(help: "Only return last N lines (applied after topic filtering).") + @Option(help: "Return the last N lines, after topic filtering. Default: 200; 0 returns all.") var last: Int? @Flag(help: "Clear buffer after reading.") @@ -154,17 +155,15 @@ struct LogRead: AsyncParsableCommand { topics.insert(t) } - let isRunning = await LogCapture.shared.isRunning - let captureMode = await LogCapture.shared.captureMode - let allLines: [String] - if clear { - allLines = await LogCapture.shared.readAndClear() - } else { - allLines = await LogCapture.shared.read(last: nil) - } + let capture = DetachedCapture.log + let isRunning = capture.isRunning + let captureMode = capture.state()?.mode ?? "smart" + let allLines = LogTools.collapseRepeats(capture.lines(.stdout)) + if clear { capture.clear() } if allLines.isEmpty { - let statusNote = isRunning ? " (capture is running)" : " (capture not running)" + let statusNote = + isRunning ? " (capture is running)" : " (capture not running; start it with `xcforge log start`)" let msg = "No log lines captured\(statusNote)" if useJSON { print( @@ -188,30 +187,21 @@ struct LogRead: AsyncParsableCommand { bundleId: bundleId, processName: processName ) - // Apply `last` AFTER filtering - let finalLines: [String] - if let n = last { - finalLines = Array(filterResult.filteredLines.suffix(n)) - } else { - finalLines = filterResult.filteredLines - } + // Apply `last` AFTER filtering; the newest lines are the ones that matter. + let (finalLines, omitted) = CaptureTail.tail(filterResult.filteredLines, last: last) + let output = CaptureTail.keepEnd(finalLines.joined(separator: "\n"), limit: 50000) if useJSON { - let output = finalLines.joined(separator: "\n") - let truncated = - output.count > 50000 ? String(output.prefix(50000)) + "\n... [truncated]" : output print( try WorkflowJSONRenderer.renderJSON( - LogResult(succeeded: true, message: truncated, lineCount: finalLines.count))) + LogResult(succeeded: true, message: output, lineCount: finalLines.count))) } else { - let summary = LogTools.buildTopicSummary( + var summary = LogTools.buildTopicSummary( result: filterResult, include: topics, captureMode: captureMode, bundleId: bundleId ) - let output = finalLines.joined(separator: "\n") - let truncated = - output.count > 50000 ? String(output.prefix(50000)) + "\n... [truncated]" : output - print(LogRenderer.renderRead(summary: summary, logs: truncated, lineCount: finalLines.count)) + if let note = CaptureTail.omittedNote(omitted) { summary += "\n\(note)" } + print(LogRenderer.renderRead(summary: summary, logs: output, lineCount: finalLines.count)) } } } @@ -274,9 +264,11 @@ struct LogWait: AsyncParsableCommand { throw ExitCode.failure } - // Start log capture if not running - let wasRunning = await LogCapture.shared.isRunning - if !wasRunning { + // Read new lines from the background capture when one runs; otherwise stream for + // the length of this command. + let capture = DetachedCapture.log + let useBackground = capture.isRunning + if !useBackground { let (logArgs, _) = await LogTools.buildLogArgs( simulator: sim, mode: "smart", level: "debug", process: nil, subsystem: subsystem, predicate: nil @@ -296,8 +288,9 @@ struct LogWait: AsyncParsableCommand { } } - // Clear existing buffer to only match new lines - _ = await LogCapture.shared.readAndClear() + // Match only lines that arrive from now on. + var byteOffset = capture.byteCount(.stdout) + var position = await LogCapture.shared.position let startTime = CFAbsoluteTimeGetCurrent() let deadline = startTime + timeout @@ -305,7 +298,13 @@ struct LogWait: AsyncParsableCommand { // Poll for matches while CFAbsoluteTimeGetCurrent() < deadline { - let lines = await LogCapture.shared.readAndClear() + let lines: [String] + if useBackground { + (lines, byteOffset) = capture.newLines(.stdout, after: byteOffset) + } else { + lines = await LogCapture.shared.lines(since: position) + position += lines.count + } for line in lines { let range = NSRange(line.startIndex..., in: line) if regex.firstMatch(in: line, range: range) != nil { diff --git a/Sources/XCForgeCLI/Commands/Sim/SimCommand.swift b/Sources/XCForgeCLI/Commands/Sim/SimCommand.swift index b07d8f7..e5621e7 100644 --- a/Sources/XCForgeCLI/Commands/Sim/SimCommand.swift +++ b/Sources/XCForgeCLI/Commands/Sim/SimCommand.swift @@ -13,7 +13,7 @@ struct Sim: ParsableCommand { SimOrientation.self, SimRecordStart.self, SimRecordStop.self, SimLocation.self, SimLocationReset.self, SimAppearance.self, SimStatusBar.self, SimStatusBarClear.self, - SimInfo.self, + SimInfo.self, SimOpenURL.self, ], defaultSubcommand: SimList.self ) @@ -151,13 +151,26 @@ struct SimLaunch: AsyncParsableCommand { @Option(help: "App bundle identifier. Auto-detected from last build if omitted.") var bundleId: String? + @Option(name: .customLong("arg"), help: "Launch argument for the app (repeatable).") + var args: [String] = [] + + @Option(help: "Environment variable for the app, KEY=VALUE (repeatable).") + var env: [String] = [] + + @Option(help: "URL or deep link to open once the app is running.") + var url: String? + + @Flag(inversion: .prefixedNo, help: "Terminate a running copy first. Default: on.") + var terminate = true + @Flag(help: "Emit the result as machine-readable JSON.") var json = false mutating func run() async throws { let useJSON = shouldOutputJSON(flag: json) - let env = Environment.live - let result = await SimTools.executeLaunchApp(simulator: simulator, bundleId: bundleId, env: env) + let result = await SimTools.executeLaunchApp( + simulator: simulator, bundleId: bundleId, args: args, environment: env, url: url, + terminateFirst: terminate, env: Environment.live) if useJSON { print(try WorkflowJSONRenderer.renderJSON(result)) @@ -617,3 +630,31 @@ struct SimInfo: AsyncParsableCommand { } } } + +struct SimOpenURL: AsyncParsableCommand { + static let configuration = CommandConfiguration( + commandName: "openurl", + abstract: "Open a URL or deep link on a simulator." + ) + + @Argument(help: "URL or deep link to open.") + var url: String + + @Option(help: "Simulator name or UDID. Auto-detected from booted simulator if omitted.") + var simulator: String? + + @Flag(help: "Emit the result as machine-readable JSON.") + var json = false + + mutating func run() async throws { + let result = await SimTools.executeOpenURL(simulator: simulator, url: url, env: Environment.live) + if shouldOutputJSON(flag: json) { + print(try WorkflowJSONRenderer.renderJSON(result)) + } else { + print(SimRenderer.render(result)) + } + if !result.succeeded { + throw ExitCode.failure + } + } +} diff --git a/Sources/XCForgeKit/Support/AppLiveness.swift b/Sources/XCForgeKit/Support/AppLiveness.swift new file mode 100644 index 0000000..a1ec082 --- /dev/null +++ b/Sources/XCForgeKit/Support/AppLiveness.swift @@ -0,0 +1,155 @@ +import Foundation + +/// Whether an app launched on a simulator is still running a moment later, and why not. +/// +/// `simctl launch` returns as soon as the process exists, so an app that crashes during +/// startup still "launches". Simulator apps are host processes, so their pid can be +/// checked with `ps`, and their crash reports land in the host's DiagnosticReports. +public enum AppLiveness { + /// How long a launched app must survive to count as running. + static let settleSeconds: TimeInterval = 2 + + enum Outcome: Equatable, Sendable { + case running + case exited(CrashSummary?) + } + + /// The pid in `simctl launch` output (`com.example.app: 1234`). + public static func pid(fromLaunchOutput output: String) -> Int32? { + for line in output.split(whereSeparator: \.isNewline).reversed() { + let parts = line.split(separator: ":", omittingEmptySubsequences: false) + // Only `: `; a URL with a port mustn't read as a pid. + guard parts.count == 2, !parts[0].isEmpty, !parts[0].contains(where: \.isWhitespace) else { continue } + if let pid = Int32(parts[1].trimmingCharacters(in: .whitespaces)) { return pid } + } + return nil + } + + /// True when `pid` exists and isn't a zombie. + static func isRunning(pid: Int32, env: Environment) async -> Bool { + let arguments = ["-p", String(pid), "-o", "stat="] + // When ps can't run, say running: never report a crash that wasn't seen. + guard let result = try? await env.shell.run("/bin/ps", arguments: arguments, timeout: 5) else { return true } + let state = result.stdout.trimmingCharacters(in: .whitespacesAndNewlines) + return result.succeeded && !state.isEmpty && !state.hasPrefix("Z") + } + + /// Wait `settleSeconds`, then report whether the app is still up. When it isn't, look + /// for the crash report it left. + static func check( + pid: Int32, bundleId: String, launchedAt: Date, settle: TimeInterval = settleSeconds, env: Environment + ) async -> Outcome { + try? await Task.sleep(nanoseconds: UInt64(settle * 1_000_000_000)) + if await isRunning(pid: pid, env: env) { return .running } + // The report is written a moment after the process dies. + for _ in 0..<10 { + if let path = CrashReports.newest(bundleId: bundleId, after: launchedAt), + let text = try? String(contentsOfFile: path, encoding: .utf8), + let summary = CrashReports.summarize(text, path: path) + { + return .exited(summary) + } + try? await Task.sleep(nanoseconds: 500_000_000) + } + return .exited(nil) + } + + /// One line per fact, for tool output. + static func describe(_ outcome: Outcome, bundleId: String) -> String { + switch outcome { + case .running: + return "App running: true" + case .exited(nil): + return "App running: false (\(bundleId) exited within \(Int(settleSeconds))s of launch; no crash report found)" + case .exited(let summary?): + var lines = ["App running: false (\(bundleId) crashed at launch)", "Crash: \(summary.headline)"] + lines += summary.frames.map { " \($0)" } + lines.append("Crash report: \(summary.path)") + return lines.joined(separator: "\n") + } + } +} + +/// The parts of a crash report an agent needs: what happened and where. +struct CrashSummary: Equatable, Sendable, Codable { + let exception: String + let reason: String? + let frames: [String] + let path: String + + var headline: String { + reason.map { "\(exception): \($0)" } ?? exception + } +} + +enum CrashReports { + static func directory() -> String { + if let override = ProcessInfo.processInfo.environment["XCFORGE_CRASH_REPORTS_DIR"], !override.isEmpty { + return override + } + return NSHomeDirectory() + "/Library/Logs/DiagnosticReports" + } + + /// The newest `.ips` report for `bundleId` written after `date`. + static func newest(bundleId: String, after date: Date, in directory: String = directory()) -> String? { + let fm = FileManager.default + guard let names = try? fm.contentsOfDirectory(atPath: directory) else { return nil } + var best: (path: String, date: Date)? + for name in names where name.hasSuffix(".ips") { + let path = (directory as NSString).appendingPathComponent(name) + guard let modified = (try? fm.attributesOfItem(atPath: path))?[.modificationDate] as? Date, + modified >= date.addingTimeInterval(-1), + best.map({ modified > $0.date }) ?? true, + let header = headerLine(path), + reportBundleID(header) == bundleId + else { continue } + best = (path, modified) + } + return best?.path + } + + private static func headerLine(_ path: String) -> String? { + guard let handle = FileHandle(forReadingAtPath: path) else { return nil } + defer { try? handle.close() } + let data = (try? handle.read(upToCount: 4096)) ?? Data() + return String(data: data, encoding: .utf8)?.split(whereSeparator: \.isNewline).first.map(String.init) + } + + static func reportBundleID(_ header: String) -> String? { + guard let data = header.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] + else { return nil } + return json["bundleID"] as? String + } + + /// Exception, termination reason and the crashed thread's top frames from an `.ips` + /// report (a JSON header line, then the JSON body). + static func summarize(_ text: String, path: String, frameLimit: Int = 8) -> CrashSummary? { + guard let newline = text.firstIndex(where: \.isNewline), + let data = String(text[text.index(after: newline)...]).data(using: .utf8), + let body = try? JSONSerialization.jsonObject(with: data) as? [String: Any] + else { return nil } + + let exception = body["exception"] as? [String: Any] + var name = (exception?["type"] as? String) ?? "crash" + if let signal = exception?["signal"] as? String { name += " (\(signal))" } + + var reason: String? + if let asi = body["asi"] as? [String: [String]], let first = asi.values.flatMap({ $0 }).first { + reason = first + } else if let termination = body["termination"] as? [String: Any] { + reason = termination["indicator"] as? String + } + + let images = (body["usedImages"] as? [[String: Any]]) ?? [] + let threads = (body["threads"] as? [[String: Any]]) ?? [] + let crashed = threads.first { ($0["triggered"] as? Bool) == true } ?? threads.first + let frames = ((crashed?["frames"] as? [[String: Any]]) ?? []).prefix(frameLimit).map { frame -> String in + let index = frame["imageIndex"] as? Int + let image = index.flatMap { $0 < images.count ? images[$0]["name"] as? String : nil } ?? "?" + if let symbol = frame["symbol"] as? String { return "\(image) \(symbol)" } + return "\(image) +\(frame["imageOffset"] as? Int ?? 0)" + } + return CrashSummary(exception: name, reason: reason, frames: Array(frames), path: path) + } +} diff --git a/Sources/XCForgeKit/Support/DetachedCapture.swift b/Sources/XCForgeKit/Support/DetachedCapture.swift new file mode 100644 index 0000000..01185e2 --- /dev/null +++ b/Sources/XCForgeKit/Support/DetachedCapture.swift @@ -0,0 +1,180 @@ +import Foundation + +/// A capture process (a log stream, an app's console) that outlives the CLI command that +/// started it. Each CLI command is its own process, so an in-memory buffer is gone by the +/// time `read` runs; this writes the stream to files and keeps the pid in a state file, so +/// `start`, `read` and `stop` can be separate commands. +public struct DetachedCapture: Sendable { + public struct State: Codable, Sendable { + public let pid: Int32 + public let mode: String? + public let bundleId: String? + public let startedAt: Date + /// Byte offsets reads start from, per stream: `clear` moves them to the end. + public var readFrom: [String: UInt64] + } + + public enum Stream: String, Sendable, CaseIterable { + case stdout, stderr + } + + public let name: String + public let directory: String + + public init(name: String, directory: String = DetachedCapture.defaultDirectory()) { + self.name = name + self.directory = directory + } + + /// The simulator log stream started by `xcforge log start`. + public static var log: DetachedCapture { DetachedCapture(name: "log") } + /// The app console started by `xcforge console launch`. + public static var console: DetachedCapture { DetachedCapture(name: "console") } + + public static func defaultDirectory() -> String { + if let override = ProcessInfo.processInfo.environment["XCFORGE_CAPTURE_DIR"], !override.isEmpty { + return override + } + return NSHomeDirectory() + "/.xcforge/capture" + } + + var statePath: String { (directory as NSString).appendingPathComponent("\(name).json") } + + public func outputPath(_ stream: Stream) -> String { + (directory as NSString).appendingPathComponent("\(name).\(stream.rawValue).log") + } + + /// Start `executable arguments` under nohup with its output going to files, replacing + /// any capture of the same name. Environment entries are added to this process's own. + @discardableResult + public func start( + executable: String = "/usr/bin/xcrun", arguments: [String], environment: [String: String] = [:], + captureStderr: Bool = false, mode: String? = nil, bundleId: String? = nil + ) throws -> State { + stop() + try FileManager.default.createDirectory(atPath: directory, withIntermediateDirectories: true) + for stream in Stream.allCases { + FileManager.default.createFile(atPath: outputPath(stream), contents: nil) + } + guard let out = FileHandle(forWritingAtPath: outputPath(.stdout)), + let err = FileHandle(forWritingAtPath: outputPath(.stderr)) + else { throw CocoaError(.fileWriteUnknown) } + defer { + try? out.close() + try? err.close() + } + + let process = Process() + // nohup keeps the capture alive when the shell that ran the CLI goes away. + process.executableURL = URL(fileURLWithPath: "/usr/bin/nohup") + process.arguments = [executable] + arguments + if !environment.isEmpty { + process.environment = ProcessInfo.processInfo.environment.merging(environment) { $1 } + } + process.standardInput = FileHandle.nullDevice + process.standardOutput = out + process.standardError = captureStderr ? err : FileHandle.nullDevice + try process.run() + + let state = State( + pid: process.processIdentifier, mode: mode, bundleId: bundleId, startedAt: Date(), readFrom: [:]) + try save(state) + return state + } + + /// Stop the capture. Returns false when none was running. + @discardableResult + public func stop() -> Bool { + guard let state = state() else { return false } + let running = Self.isAlive(state.pid) + if running { kill(state.pid, SIGTERM) } + try? FileManager.default.removeItem(atPath: statePath) + return running + } + + public func state() -> State? { + guard let data = FileManager.default.contents(atPath: statePath) else { return nil } + let decoder = JSONDecoder() + decoder.dateDecodingStrategy = .iso8601 + return try? decoder.decode(State.self, from: data) + } + + public var isRunning: Bool { + state().map { Self.isAlive($0.pid) } ?? false + } + + /// Whether a capture was started and its output is still on disk, running or not. + public var exists: Bool { state() != nil } + + static func isAlive(_ pid: Int32) -> Bool { + pid > 0 && kill(pid, 0) == 0 + } + + /// Lines written since the last `clear` (or since `fromByte`, when given). + public func lines(_ stream: Stream, fromByte: UInt64? = nil) -> [String] { + let start = fromByte ?? state()?.readFrom[stream.rawValue] ?? 0 + guard let handle = FileHandle(forReadingAtPath: outputPath(stream)) else { return [] } + defer { try? handle.close() } + try? handle.seek(toOffset: start) + let data = (try? handle.readToEnd()) ?? Data() + return String(decoding: data, as: UTF8.self).split(separator: "\n").map(String.init) + } + + /// Complete lines written after byte `offset`, and the offset to read from next. A line + /// still being written stays for the next call. + public func newLines(_ stream: Stream, after offset: UInt64) -> (lines: [String], next: UInt64) { + guard let handle = FileHandle(forReadingAtPath: outputPath(stream)) else { return ([], offset) } + defer { try? handle.close() } + try? handle.seek(toOffset: offset) + let data = (try? handle.readToEnd()) ?? Data() + guard let lastNewline = data.lastIndex(of: UInt8(ascii: "\n")) else { return ([], offset) } + let complete = data[data.startIndex...lastNewline] + let lines = String(decoding: complete, as: UTF8.self).split(separator: "\n").map(String.init) + return (lines, offset + UInt64(complete.count)) + } + + /// The current end of a stream's output, for reading only what comes after it. + public func byteCount(_ stream: Stream) -> UInt64 { + let size = (try? FileManager.default.attributesOfItem(atPath: outputPath(stream)))?[.size] + return (size as? NSNumber)?.uint64Value ?? 0 + } + + /// Skip everything captured so far on later reads. + public func clear() { + guard var state = state() else { return } + for stream in Stream.allCases { + state.readFrom[stream.rawValue] = byteCount(stream) + } + try? save(state) + } + + private func save(_ state: State) throws { + let encoder = JSONEncoder() + encoder.dateEncodingStrategy = .iso8601 + try encoder.encode(state).write(to: URL(fileURLWithPath: statePath), options: .atomic) + } +} + +/// Reading a capture: the newest lines by default, with a count of what was left out. +public enum CaptureTail { + /// How many lines a read returns when the caller doesn't say. + public static let defaultLines = 200 + + /// The last `last` lines (`defaultLines` when nil, everything when 0) and how many + /// earlier lines were omitted. + public static func tail(_ lines: [String], last: Int?) -> (lines: [String], omitted: Int) { + let limit = last ?? defaultLines + guard limit > 0, lines.count > limit else { return (lines, 0) } + return (Array(lines.suffix(limit)), lines.count - limit) + } + + /// Keep the end of `text` when it's over `limit` characters: the newest output matters most. + public static func keepEnd(_ text: String, limit: Int) -> String { + guard text.count > limit else { return text } + return "... [earlier output truncated]\n" + String(text.suffix(limit)) + } + + public static func omittedNote(_ omitted: Int) -> String? { + omitted > 0 ? "(\(omitted) earlier lines omitted; last 0 returns all)" : nil + } +} diff --git a/Sources/XCForgeKit/Support/ProjectResolver.swift b/Sources/XCForgeKit/Support/ProjectResolver.swift index 8f254fa..8cbdb7c 100644 --- a/Sources/XCForgeKit/Support/ProjectResolver.swift +++ b/Sources/XCForgeKit/Support/ProjectResolver.swift @@ -133,12 +133,14 @@ enum AutoDetect { "-not", "-path", "*/.build/*", "-not", "-path", "*/DerivedData/*", "-not", "-path", "*/.swiftpm/*", + // Every .xcodeproj holds a project.xcworkspace; it isn't a separate workspace. + "-not", "-path", "*.xcodeproj/*", ], timeout: 10) let paths = result.stdout .split(separator: "\n") .map(String.init) - .filter { !$0.isEmpty } + .filter { !$0.isEmpty && !$0.contains(".xcodeproj/") } // Prefer .xcworkspace over .xcodeproj when both exist let workspaces = paths.filter { $0.hasSuffix(".xcworkspace") } @@ -165,7 +167,11 @@ enum AutoDetect { /// Detect scheme for a project. Returns name if exactly one scheme exists. static func scheme(project: String) async throws -> String { let schemes = try await availableSchemes(project: project) + let projectName = ((project as NSString).lastPathComponent as NSString).deletingPathExtension + if let preferred = preferredScheme(schemes, projectName: projectName) { + return preferred + } switch schemes.count { case 1: return schemes[0] @@ -176,6 +182,15 @@ enum AutoDetect { } } + /// The scheme to use without asking: the only one, the one named after the project + /// (Xcode's app scheme), or the only one that isn't a test or CocoaPods scheme. + static func preferredScheme(_ schemes: [String], projectName: String) -> String? { + if schemes.count == 1 { return schemes[0] } + if schemes.contains(projectName) { return projectName } + let apps = schemes.filter { !$0.hasSuffix("Tests") && !$0.hasPrefix("Pods-") } + return apps.count == 1 ? apps[0] : nil + } + static func availableSchemes(project: String) async throws -> [String] { let isWorkspace = project.hasSuffix(".xcworkspace") let projectFlag = isWorkspace ? "-workspace" : "-project" @@ -184,7 +199,7 @@ enum AutoDetect { "/usr/bin/xcodebuild", arguments: [ projectFlag, project, "-list", "-json", - ], timeout: 15) + ], timeout: 60) // Package resolution on a cold checkout can take most of a minute. guard result.succeeded, let data = result.stdout.data(using: .utf8), @@ -238,7 +253,7 @@ enum AutoDetect { "/usr/bin/xcodebuild", arguments: [ projectFlag, project, "-list", "-json", - ], timeout: 15) + ], timeout: 60) // Package resolution on a cold checkout can take most of a minute. guard result.succeeded, let data = result.stdout.data(using: .utf8), diff --git a/Sources/XCForgeKit/Support/WorkspaceContext.swift b/Sources/XCForgeKit/Support/WorkspaceContext.swift index 6fe5351..d5ef7de 100644 --- a/Sources/XCForgeKit/Support/WorkspaceContext.swift +++ b/Sources/XCForgeKit/Support/WorkspaceContext.swift @@ -274,8 +274,9 @@ public actor SessionState { ) } - public func resolveBundleId(_ explicit: String?) -> String? { + public func resolveBundleId(_ explicit: String?) async -> String? { if let explicit { return explicit } + await loadBuildInfoIfNeeded() // scheme is no longer eagerly cached from persisted at init, so the // build-scheme mismatch guard must consult the effective scheme // (session cache → repo config → persisted) to keep stale bundle ids @@ -285,13 +286,22 @@ public actor SessionState { return bundleId } - func resolveAppPath(_ explicit: String?) -> String? { + func resolveAppPath(_ explicit: String?) async -> String? { if let explicit { return explicit } + await loadBuildInfoIfNeeded() let effScheme = scheme ?? repoDefaults?.scheme ?? persistedScheme if let buildScheme, let effScheme, buildScheme != effScheme { return nil } return appPath } + /// A new process (the CLI, or a restarted MCP server) has no project resolved yet, so + /// the last build's bundle ID and app path, saved per project, aren't loaded. Resolve + /// the project to load them. + private func loadBuildInfoIfNeeded() async { + guard loadedRecordForProject == nil else { return } + _ = try? await resolveProject(nil) + } + func clearBuildInfo() { bundleId = nil appPath = nil diff --git a/Sources/XCForgeKit/Tools/BuildProvider.swift b/Sources/XCForgeKit/Tools/BuildProvider.swift index 271122c..8703a86 100644 --- a/Sources/XCForgeKit/Tools/BuildProvider.swift +++ b/Sources/XCForgeKit/Tools/BuildProvider.swift @@ -183,9 +183,9 @@ public enum BuildTools { Tool( name: "build_run_sim", description: """ - Build, install, and launch an iOS app on a simulator in one call. \ - Runs build, settings extraction, simulator boot, and Simulator/Device Hub \ - in parallel for maximum speed. Equivalent to Xcode's Cmd+R. \ + Build, install, and launch an iOS app on a simulator in one call (Xcode's Cmd+R). \ + Boots the simulator once the build succeeds, then reports whether the app is still \ + running 2s after launch, with the crash reason and top frames when it isn't. \ Project, scheme, and simulator are auto-detected if omitted. """, inputSchema: .object([ @@ -223,6 +223,18 @@ public enum BuildTools { "Capture a diagnostic snapshot on completion even without a hang, for baseline inspection." ), ]), + "args": .object([ + "type": .string("array"), "items": .object(["type": .string("string")]), + "description": .string("Launch arguments passed to the app."), + ]), + "env": .object([ + "type": .string("array"), "items": .object(["type": .string("string")]), + "description": .string("Environment for the app, as KEY=VALUE strings."), + ]), + "url": .object([ + "type": .string("string"), + "description": .string("URL or deep link to open once the app is running."), + ]), ]), ]) ), @@ -275,8 +287,9 @@ public enum BuildTools { Tool( name: "clean", description: """ - Clean Xcode build artifacts for a project/scheme. \ - Project and scheme are auto-detected if omitted. + Clean the scheme's simulator build products. derivedData: true also deletes this \ + project's DerivedData folder (only this project's), the fix for "database is locked" \ + or a stale index. Project and scheme are auto-detected if omitted. """, inputSchema: .object([ "type": .string("object"), @@ -289,6 +302,14 @@ public enum BuildTools { "type": .string("string"), "description": .string("Xcode scheme name. Auto-detected if omitted."), ]), + "configuration": .object([ + "type": .string("string"), + "description": .string("Build configuration (Debug/Release). Default: Debug"), + ]), + "derivedData": .object([ + "type": .string("boolean"), + "description": .string("Also delete this project's DerivedData folder. Default: false"), + ]), ]), ]) ), @@ -347,11 +368,16 @@ public enum BuildTools { let configuration: String? let long: Bool? let diagnose: Bool? + var args: [String]? = nil + var env: [String]? = nil + var url: String? = nil } struct CleanInput: Decodable { let project: String? let scheme: String? + var configuration: String? = nil + var derivedData: Bool? = nil } struct DiscoverInput: Decodable { @@ -812,25 +838,17 @@ public enum BuildTools { switch ToolInput.decode(CleanInput.self, from: args) { case .failure(let err): return err case .success(let input): - let project: String - let scheme: String do { - project = try await env.session.resolveProject(input.project) - scheme = try await env.session.resolveScheme(input.scheme, project: project) + let execution = try await executeClean( + project: input.project, scheme: input.scheme, configuration: input.configuration, + deleteDerivedData: input.derivedData ?? false, env: env) + guard execution.succeeded else { return .fail("Clean failed: \(execution.error ?? "unknown error")") } + var message = "Clean succeeded (\(execution.scheme))" + if let removed = execution.removedDerivedData { message += "\nDeleted DerivedData: \(removed)" } + return .ok(message) } catch { return .fail("\(error)") } - - let isWorkspace = project.hasSuffix(".xcworkspace") - let projectFlag = isWorkspace ? "-workspace" : "-project" - - do { - let result = try await Xcodebuild.run( - [projectFlag, project, "-scheme", scheme, "clean"], timeout: 60, env: env) - return result.succeeded ? .ok("Clean succeeded") : .fail("Clean failed: \(result.stderr)") - } catch { - return .fail("Clean error: \(error)") - } } } @@ -846,6 +864,7 @@ public enum BuildTools { "(", "-name", "*.xcodeproj", "-o", "-name", "*.xcworkspace", ")", "-not", "-path", "*/Pods/*", "-not", "-path", "*/.build/*", + "-not", "-path", "*.xcodeproj/*", ], timeout: 15) return .ok(result.stdout.isEmpty ? "No projects found" : result.stdout) } catch { @@ -911,23 +930,19 @@ public enum BuildTools { "-scheme", scheme, "-configuration", configuration, "-destination", destination, - "-showBuildSettings", + "-showBuildSettings", "-json", ] - // ── Phase 1: Parallel ── - // Build is the critical path (~10-60s). Settings extraction, simulator boot, - // and Simulator.app launch run concurrently — they complete while the build - // is still compiling, adding zero wall-clock overhead. + // ── Phase 1: build, with settings extraction alongside ── + // The simulator boots only once the build succeeds: a failed build shouldn't cost a + // booted simulator's memory on a shared Mac. let buildTimeout = await TestTools.resolveTestTimeout(long: input.long ?? false, env: env) let buildSnapshotPath = TestTools.diagnosticSnapshotPath() let buildWatchdog = HangWatchdog( udid: udid, snapshotPath: buildSnapshotPath, sampleAt: HangWatchdog.defaultSampleAt, processMatch: resultPath, env: env) async let buildTask = Xcodebuild.run(buildArgs, timeout: buildTimeout, env: env) - async let settingsTask = Xcodebuild.run(settingsArgs, timeout: 30, env: env) - async let bootTask = env.shell.run( - "/usr/bin/xcrun", arguments: ["simctl", "boot", udid], timeout: 60) - async let openTask: Void = SimulatorApp.open(shell: env.shell) + async let settingsTask = Xcodebuild.run(settingsArgs, timeout: 60, env: env) // Await build first (critical — abort if it fails) let buildResult: ShellResult @@ -997,102 +1012,48 @@ public enum BuildTools { return .fail(failMsg) } - // Await settings — 3-tier fallback for app path + bundle ID: - // 1. -showBuildSettings (parallel, fastest when it works) - // 2. Parse build stdout for .app paths - // 3. Search DerivedData with find - // Bundle ID always via PlistBuddy once we have the .app path. - var appPath: String? - var infoSource = "showBuildSettings" - - // Tier 1: -showBuildSettings - if let settingsResult = try? await settingsTask, settingsResult.succeeded { - var builtProductsDir: String? - var fullProductName: String? - - for line in settingsResult.stdout.split(separator: "\n") { - let trimmed = line.trimmingCharacters(in: .whitespaces) - if trimmed.hasPrefix("BUILT_PRODUCTS_DIR = ") { - builtProductsDir = String(trimmed.dropFirst("BUILT_PRODUCTS_DIR = ".count)) - } else if trimmed.hasPrefix("FULL_PRODUCT_NAME = ") { - fullProductName = String(trimmed.dropFirst("FULL_PRODUCT_NAME = ".count)) - } - } - - if let dir = builtProductsDir, let name = fullProductName { - appPath = "\(dir)/\(name)" - } + // The scheme's application target, from the JSON settings. Schemes also build + // extensions and frameworks, so the last product listed isn't necessarily the app. + var product: BuildProductInfo? + if let settings = try? await settingsTask, settings.succeeded { + product = appProduct(fromSettings: settings.stdout) } - - // Tier 2: Parse build stdout for .app path - if appPath == nil { - let suffix = "/\(configuration)-iphonesimulator/" - for line in buildResult.stdout.split(separator: "\n").reversed() { - let s = String(line) - if let range = s.range(of: suffix) { - let afterConfig = s[range.upperBound...] - if let appEnd = afterConfig.range(of: ".app") { - let fullLine = String(s[s.startIndex..: " on stdout - let appPid = launchResult.stdout.trimmingCharacters(in: .whitespacesAndNewlines) - .split(separator: ":").last - .map { String($0).trimmingCharacters(in: .whitespaces) } + let appPid = launch.message.split(separator: "\n").lazy + .compactMap { AppLiveness.pid(fromLaunchOutput: String($0)) }.first let totalElapsed = String(format: "%.1f", CFAbsoluteTimeGetCurrent() - totalStart) @@ -1143,17 +1090,14 @@ public enum BuildTools { output += "\nScheme: \(scheme) | Simulator: \(simulator)" output += "\nBundle ID: \(bundleId)" output += "\nApp path: \(finalAppPath)" - if let pid = appPid, !pid.isEmpty { - output += "\nApp PID: \(pid)" - } + if let appPid { output += "\nApp PID: \(appPid)" } output += "\nApp running: true" + if let url = input.url { output += "\nOpened: \(url)" } output += "\n" output += "\n Build: \(buildElapsed)s" - output += "\n App info: \(infoSource) (parallel)" - output += "\n Boot: \(bootStatus) (parallel)" + output += "\n Boot: \(bootStatus) (\(bootElapsed)s)" output += "\n Install: \(installElapsed)s" output += "\n Launch: OK" - output += "\n Simulator: opened" // Surface build warnings from xcresult if any let xcresultIssues = await extractIssuesFromXcresult(resultPath, env: env) @@ -1189,8 +1133,8 @@ public enum BuildTools { "-scheme", scheme, "-configuration", configuration, "-destination", destination, - "-showBuildSettings", - ], timeout: 30, env: env) + "-showBuildSettings", "-json", + ], timeout: 60, env: env) guard result.succeeded else { let details = result.stderr.isEmpty ? result.stdout : result.stderr @@ -1198,31 +1142,29 @@ public enum BuildTools { "Unable to resolve app context for \(scheme): \(details.trimmingCharacters(in: .whitespacesAndNewlines))" ) } - - var bundleId: String? - var builtProductsDir: String? - var fullProductName: String? - - for line in result.stdout.split(separator: "\n") { - let trimmed = line.trimmingCharacters(in: .whitespaces) - if trimmed.hasPrefix("PRODUCT_BUNDLE_IDENTIFIER = ") { - bundleId = String(trimmed.dropFirst("PRODUCT_BUNDLE_IDENTIFIER = ".count)) - } else if trimmed.hasPrefix("BUILT_PRODUCTS_DIR = ") { - builtProductsDir = String(trimmed.dropFirst("BUILT_PRODUCTS_DIR = ".count)) - } else if trimmed.hasPrefix("FULL_PRODUCT_NAME = ") { - fullProductName = String(trimmed.dropFirst("FULL_PRODUCT_NAME = ".count)) - } + guard let product = appProduct(fromSettings: result.stdout) else { + throw BuildSettingsError("Scheme \(scheme) builds no application target, so there is no app to install") } + return product + } - guard let bundleId else { - throw BuildSettingsError( - "Build settings did not contain PRODUCT_BUNDLE_IDENTIFIER for \(scheme)") + /// The application target's product in `-showBuildSettings -json` output. Schemes also + /// build extensions, frameworks and test bundles; only the app can be installed and + /// launched, whichever target the output lists last. + static func appProduct(fromSettings output: String) -> BuildProductInfo? { + guard let data = output.data(using: .utf8), + let entries = try? JSONSerialization.jsonObject(with: data) as? [[String: Any]] + else { return nil } + for entry in entries { + guard let settings = entry["buildSettings"] as? [String: String], + settings["PRODUCT_TYPE"] == "com.apple.product-type.application" || settings["WRAPPER_EXTENSION"] == "app", + let bundleId = settings["PRODUCT_BUNDLE_IDENTIFIER"], + let directory = settings["BUILT_PRODUCTS_DIR"], + let name = settings["FULL_PRODUCT_NAME"] + else { continue } + return BuildProductInfo(bundleId: bundleId, appPath: "\(directory)/\(name)") } - guard let builtProductsDir, let fullProductName else { - throw BuildSettingsError("Build settings did not contain an app product path for \(scheme)") - } - - return BuildProductInfo(bundleId: bundleId, appPath: "\(builtProductsDir)/\(fullProductName)") + return nil } private static func extractBuildInfo( @@ -1296,28 +1238,84 @@ public enum BuildTools { public let project: String public let scheme: String public let error: String? + /// The project's DerivedData folder, when `deleteDerivedData` removed it. + public var removedDerivedData: String? = nil } + /// Clean the scheme's products for the simulator build xcforge runs (same configuration, + /// same build folder), and optionally delete this project's DerivedData folder, the fix + /// for a corrupted build database or index. public static func executeClean( project: String? = nil, scheme: String? = nil, + configuration: String? = nil, + deleteDerivedData: Bool = false, env: Environment = .live ) async throws -> CleanExecution { let resolvedProject = try await env.session.resolveProject(project) let resolvedScheme = try await env.session.resolveScheme(scheme, project: resolvedProject) + let resolvedConfiguration = await env.session.resolveConfiguration(configuration) let isWorkspace = resolvedProject.hasSuffix(".xcworkspace") let projectFlag = isWorkspace ? "-workspace" : "-project" + let base = [ + projectFlag, resolvedProject, "-scheme", resolvedScheme, "-configuration", resolvedConfiguration, + "-destination", "generic/platform=iOS Simulator", + ] - let result = try await Xcodebuild.run( - [projectFlag, resolvedProject, "-scheme", resolvedScheme, "clean"], timeout: 60, env: env) + // Look the folder up before cleaning: the settings call needs a working project. + var derivedData: String? + if deleteDerivedData { + let settings = try await Xcodebuild.run(base + ["-showBuildSettings", "-json"], timeout: 120, env: env) + derivedData = settings.succeeded ? derivedDataFolder(fromSettings: settings.stdout) : nil + } - return CleanExecution( + // A clean is killed after the usual stretch of silence, not a fixed 60s. + let result = try await Xcodebuild.run(base + ["clean"], timeout: 1800, env: env) + var execution = CleanExecution( succeeded: result.succeeded, project: resolvedProject, scheme: resolvedScheme, - error: result.succeeded ? nil : result.stderr + error: result.succeeded ? nil : Xcodebuild.combinedOutput(result) ) + guard deleteDerivedData else { return execution } + guard let derivedData else { + return CleanExecution( + succeeded: false, project: resolvedProject, scheme: resolvedScheme, + error: "Couldn't find this project's DerivedData folder from its build settings; nothing deleted.") + } + do { + if FileManager.default.fileExists(atPath: derivedData) { + try FileManager.default.removeItem(atPath: derivedData) + } + execution.removedDerivedData = derivedData + } catch { + return CleanExecution( + succeeded: false, project: resolvedProject, scheme: resolvedScheme, + error: "Deleting \(derivedData) failed: \(error)") + } + return execution + } + + /// This project's DerivedData folder: BUILD_ROOT is `/Build/Products`. Anything + /// that doesn't look like that, or is too short a path to be one project's folder, is + /// refused rather than deleted. + static func derivedDataFolder(fromSettings output: String) -> String? { + guard let data = output.data(using: .utf8), + let entries = try? JSONSerialization.jsonObject(with: data) as? [[String: Any]], + let settings = entries.first?["buildSettings"] as? [String: String], + let root = settings["BUILD_ROOT"] + else { return nil } + return derivedDataFolder(fromBuildRoot: root) + } + + static func derivedDataFolder(fromBuildRoot root: String) -> String? { + let suffix = "/Build/Products" + guard root.hasPrefix("/"), root.hasSuffix(suffix) else { return nil } + let folder = String(root.dropLast(suffix.count)) + let depth = folder.split(separator: "/").count + guard depth >= 3, folder != NSHomeDirectory() else { return nil } + return folder } public struct DiscoverExecution: Codable, Sendable { @@ -1333,6 +1331,7 @@ public enum BuildTools { "(", "-name", "*.xcodeproj", "-o", "-name", "*.xcworkspace", ")", "-not", "-path", "*/Pods/*", "-not", "-path", "*/.build/*", + "-not", "-path", "*.xcodeproj/*", ], timeout: 15) let projects = result.stdout diff --git a/Sources/XCForgeKit/Tools/ConsoleProvider.swift b/Sources/XCForgeKit/Tools/ConsoleProvider.swift index da97dec..6f50ce6 100644 --- a/Sources/XCForgeKit/Tools/ConsoleProvider.swift +++ b/Sources/XCForgeKit/Tools/ConsoleProvider.swift @@ -173,7 +173,7 @@ enum ConsoleTools { "properties": .object([ "last": .object([ "type": .string("number"), - "description": .string("Only return last N lines per stream"), + "description": .string("Return the last N lines per stream. Default: 200; 0 returns all."), ]), "clear": .object([ "type": .string("boolean"), @@ -247,28 +247,14 @@ enum ConsoleTools { let clear = input.clear ?? false let stream = input.stream ?? "both" - let output = await AppConsole.shared.read(last: last, clear: clear) + let output = await AppConsole.shared.read(last: nil, clear: clear) var sections: [String] = [] - - if stream == "stdout" || stream == "both" { - if output.stdout.isEmpty { - sections.append("=== STDOUT (empty) ===") - } else { - let text = output.stdout.joined(separator: "\n") - let truncated = text.count > 30000 ? String(text.prefix(30000)) + "\n... [truncated]" : text - sections.append("=== STDOUT (\(output.stdout.count) lines) ===\n\(truncated)") - } - } - - if stream == "stderr" || stream == "both" { - if output.stderr.isEmpty { - sections.append("=== STDERR (empty) ===") - } else { - let text = output.stderr.joined(separator: "\n") - let truncated = text.count > 30000 ? String(text.prefix(30000)) + "\n... [truncated]" : text - sections.append("=== STDERR (\(output.stderr.count) lines) ===\n\(truncated)") - } + for (name, lines, wanted) in [ + ("STDOUT", output.stdout, stream == "stdout" || stream == "both"), + ("STDERR", output.stderr, stream == "stderr" || stream == "both"), + ] where wanted { + sections.append(ConsoleTools.section(name, lines: lines, last: last)) } let status = output.isRunning ? "running" : "stopped" @@ -277,6 +263,16 @@ enum ConsoleTools { return .ok(header + "\n\n" + sections.joined(separator: "\n\n")) } + /// One stream's newest lines, with how many earlier ones were left out. + static func section(_ name: String, lines: [String], last: Int?) -> String { + let lines = lines.filter { !$0.isEmpty } + guard !lines.isEmpty else { return "=== \(name) (empty) ===" } + let (kept, omitted) = CaptureTail.tail(lines, last: last) + var header = "=== \(name) (\(lines.count) lines) ===" + if let note = CaptureTail.omittedNote(omitted) { header += " \(note)" } + return header + "\n" + CaptureTail.keepEnd(kept.joined(separator: "\n"), limit: 30000) + } + static func stopAppConsole(_ args: [String: Value]?) async -> CallTool.Result { await AppConsole.shared.stop() return .ok("App console stopped") diff --git a/Sources/XCForgeKit/Tools/LogProvider.swift b/Sources/XCForgeKit/Tools/LogProvider.swift index 293dc47..6e63f1e 100644 --- a/Sources/XCForgeKit/Tools/LogProvider.swift +++ b/Sources/XCForgeKit/Tools/LogProvider.swift @@ -10,6 +10,9 @@ public actor LogCapture { private var buffer: [String] = [] private var pipe: Pipe? private let maxLines = 5000 + /// Lines dropped from the front of `buffer` (trimmed or cleared), so a position taken + /// earlier still points at the same line. + private var dropped = 0 // Deduplication state private var lastMessageContent: String? @@ -49,7 +52,7 @@ public actor LogCapture { let lines = text.split(separator: "\n", omittingEmptySubsequences: false).map(String.init) for line in lines { guard !line.isEmpty else { continue } - let content = stripTimestamp(line) + let content = Self.stripTimestamp(line) if content == lastMessageContent { repeatCount += 1 @@ -61,6 +64,7 @@ public actor LogCapture { } } if buffer.count > maxLines { + dropped += buffer.count - maxLines buffer.removeFirst(buffer.count - maxLines) } } @@ -68,7 +72,7 @@ public actor LogCapture { /// Strip the compact-format timestamp prefix for dedup comparison. /// Compact format: "2026-03-29 16:45:12.345 Df processName message..." /// We strip everything up to and including the first tab or double-space after the timestamp. - private func stripTimestamp(_ line: String) -> String { + static func stripTimestamp(_ line: String) -> String { // Find the third space (after "date time level") to get to the message content var spaceCount = 0 for (i, ch) in line.enumerated() { @@ -102,11 +106,21 @@ public actor LogCapture { public func readAndClear() -> [String] { flushRepeat() let result = buffer + dropped += buffer.count buffer.removeAll() lastMessageContent = nil return result } + /// A marker for "now": `lines(since:)` returns what arrives after it, without clearing + /// lines another reader still wants. + public var position: Int { dropped + buffer.count } + + public func lines(since position: Int) -> [String] { + let start = max(0, position - dropped) + return start < buffer.count ? Array(buffer[start...]) : [] + } + public func read(last: Int?) -> [String] { flushRepeat() if let n = last { @@ -143,7 +157,7 @@ public enum LogTools { struct ParsedLogLine { let processName: String // "locationd", "SpringBoard", "MyApp" - let logType: String // "Db", "Df", "De", "Di" + let logType: String // "Db", "I", "Df", "E", "F" let subsystem: String? // "com.apple.locationd.Motion" (nil if missing) let category: String? // "Motion", "endpoint" (nil if missing) } @@ -171,9 +185,9 @@ public enum LogTools { let chars = Array(line.unicodeScalars) - // Position 24-25: log type (Db, Df, De, Di) + // Position 24-25: log type (Db debug, I info, Df default, E error, F fault) guard chars.count > 25 else { return nil } - let logType = String(chars[24...25].map { Character($0) }) + let logType = String(chars[24...25].map { Character($0) }).trimmingCharacters(in: .whitespaces) // After log type + whitespace: process name until '[' or '(' var idx = 26 @@ -222,6 +236,31 @@ public enum LogTools { subsystem: subsystem, category: category) } + static func isErrorOrFault(_ logType: String) -> Bool { + let code = logType.trimmingCharacters(in: .whitespaces) + return code == "E" || code == "F" || code == "Er" || code == "Fa" + } + + /// Collapse consecutive repeats the way live capture does, for lines read from a file. + public static func collapseRepeats(_ lines: [String]) -> [String] { + var result: [String] = [] + var last: String? + var repeats = 0 + for line in lines where !line.isEmpty { + let content = LogCapture.stripTimestamp(line) + if content == last { + repeats += 1 + continue + } + if repeats > 0 { result.append(" ... repeated \(repeats)x") } + repeats = 0 + last = content + result.append(line) + } + if repeats > 0 { result.append(" ... repeated \(repeats)x") } + return result + } + /// Categorize a parsed log line into topic(s). /// Always includes `app` (if matching) and `crashes` (if fault). Returns `system` as fallback. static func categorize(_ parsed: ParsedLogLine, bundleId: String?, processName: String?) -> Set< @@ -237,8 +276,9 @@ public enum LogTools { topics.insert("app") } - // crashes: fault level - if parsed.logType == "Df" { + // crashes: error and fault levels. Compact style codes: Db debug, I info, Df default, + // E error, F fault (some versions spell them Er/Fa). Df is ordinary output. + if isErrorOrFault(parsed.logType) { topics.insert("crashes") } @@ -555,7 +595,8 @@ public enum LogTools { ]), "last": .object([ "type": .string("number"), - "description": .string("Only return last N lines (applied after topic filtering)"), + "description": .string( + "Return the last N lines, after topic filtering. Default: 200; 0 returns all."), ]), "clear": .object([ "type": .string("boolean"), @@ -719,24 +760,18 @@ public enum LogTools { bundleId: bundleId, processName: processName ) - // Apply `last` AFTER filtering - let finalLines: [String] - if let n = last { - finalLines = Array(filterResult.filteredLines.suffix(n)) - } else { - finalLines = filterResult.filteredLines - } + // Apply `last` AFTER filtering; the newest lines are the ones that matter. + let (finalLines, omitted) = CaptureTail.tail(filterResult.filteredLines, last: last) // Build summary header - let summary = buildTopicSummary( + var summary = buildTopicSummary( result: filterResult, include: includeTopics, captureMode: captureMode, bundleId: bundleId ) + if let note = CaptureTail.omittedNote(omitted) { summary += "\n\(note)" } - let output = finalLines.joined(separator: "\n") - let truncated = - output.count > 50000 ? String(output.prefix(50000)) + "\n... [truncated]" : output - return .ok("\(summary)\n\(truncated)") + let output = CaptureTail.keepEnd(finalLines.joined(separator: "\n"), limit: 50000) + return .ok("\(summary)\n\(output)") } static func waitForLog(_ args: [String: Value]?, env: Environment) async -> CallTool.Result { @@ -779,8 +814,8 @@ public enum LogTools { } } - // Clear existing buffer to only match new lines - _ = await LogCapture.shared.readAndClear() + // Match only lines that arrive from now on, without clearing what read_logs would show. + var position = await LogCapture.shared.position let startTime = CFAbsoluteTimeGetCurrent() let deadline = startTime + timeout @@ -788,7 +823,8 @@ public enum LogTools { // Poll for matches while CFAbsoluteTimeGetCurrent() < deadline { - let lines = await LogCapture.shared.readAndClear() + let lines = await LogCapture.shared.lines(since: position) + position += lines.count for line in lines { let range = NSRange(line.startIndex..., in: line) if regex.firstMatch(in: line, range: range) != nil { diff --git a/Sources/XCForgeKit/Tools/SimulatorProvider.swift b/Sources/XCForgeKit/Tools/SimulatorProvider.swift index d8c10df..c079df1 100644 --- a/Sources/XCForgeKit/Tools/SimulatorProvider.swift +++ b/Sources/XCForgeKit/Tools/SimulatorProvider.swift @@ -103,8 +103,11 @@ public enum SimTools { ), Tool( name: "launch_app", - description: - "Launch an app on a booted simulator. Bundle ID is auto-detected from last build if omitted.", + description: """ + Launch an app on a booted simulator and check it is still running 2s later. \ + An app that crashed at launch fails with its crash report's exception and top frames. \ + Bundle ID is auto-detected from last build if omitted. + """, inputSchema: .object([ "type": .string("object"), "properties": .object([ @@ -118,9 +121,42 @@ public enum SimTools { "description": .string( "App bundle identifier. Auto-detected from last build_sim if omitted."), ]), + "args": .object([ + "type": .string("array"), "items": .object(["type": .string("string")]), + "description": .string("Launch arguments passed to the app."), + ]), + "env": .object([ + "type": .string("array"), "items": .object(["type": .string("string")]), + "description": .string("Environment for the app, as KEY=VALUE strings."), + ]), + "url": .object([ + "type": .string("string"), + "description": .string("URL or deep link to open once the app is running."), + ]), + "terminate": .object([ + "type": .string("boolean"), + "description": .string("Terminate a running copy first. Default: true."), + ]), ]), ]) ), + Tool( + name: "open_url", + description: "Open a URL or deep link on a simulator (simctl openurl).", + inputSchema: .object([ + "type": .string("object"), + "properties": .object([ + "simulator": .object([ + "type": .string("string"), + "description": .string("Simulator name or UDID. Auto-detected from booted simulator if omitted."), + ]), + "url": .object([ + "type": .string("string"), "description": .string("URL or deep link to open."), + ]), + ]), + "required": .array([.string("url")]), + ]) + ), Tool( name: "terminate_app", description: @@ -387,6 +423,20 @@ public enum SimTools { let bundle_id: String? } + private struct LaunchInput: Decodable { + let simulator: String? + let bundle_id: String? + let args: [String]? + let env: [String]? + let url: String? + let terminate: Bool? + } + + private struct OpenURLInput: Decodable { + let simulator: String? + let url: String + } + private struct CloneInput: Decodable { let simulator: String let name: String @@ -503,6 +553,8 @@ public enum SimTools { let udid = try await resolveSimulator(simulator, env: env) let result = try await env.shell.xcrun(timeout: 60, "simctl", "boot", udid) if result.succeeded || result.stderr.contains("current state: Booted") { + // `simctl boot` returns before the simulator can take installs and launches. + _ = try? await env.shell.xcrun(timeout: 120, "simctl", "bootstatus", udid, "-b") return SimResult(succeeded: true, message: "Simulator booted: \(udid)") } return SimResult(succeeded: false, message: "Boot failed: \(result.stderr)") @@ -553,8 +605,12 @@ public enum SimTools { } } + /// Launch the app and check it is still running `AppLiveness.settleSeconds` later; an + /// app that died is a failure that carries its crash report. `environment` entries are + /// `KEY=VALUE`; `url` is opened in the app after it starts. public static func executeLaunchApp( - simulator: String?, bundleId: String?, args: [String]? = nil, env: Environment + simulator: String?, bundleId: String?, args: [String]? = nil, environment: [String] = [], + url: String? = nil, terminateFirst: Bool = true, env: Environment ) async -> SimResult { guard let resolvedBundleId = await env.session.resolveBundleId(bundleId) else { return SimResult( @@ -566,10 +622,18 @@ public enum SimTools { } catch { return SimResult(succeeded: false, message: "\(error)") } + let childEnvironment: [String: String] + do { + childEnvironment = try parseEnvironment(environment) + } catch { + return SimResult(succeeded: false, message: "\(error)") + } do { let udid = try await resolveSimulator(sim, env: env) + let launchedAt = Date() let launch = try await launchAppStructured( - simulatorUDID: udid, bundleId: resolvedBundleId, args: args, env: env) + simulatorUDID: udid, bundleId: resolvedBundleId, args: args, environment: childEnvironment, + terminateFirst: terminateFirst, env: env) if launch.succeeded { // Hint WDA so any subsequent implicit `ensureSession()` (e.g. from `ui ls @@ -577,9 +641,18 @@ public enum SimTools { // Springboard. await env.wdaClient.recordLaunchedApp(bundleId: resolvedBundleId) let note = launch.wasRunning ? " (was running, relaunched)" : "" - return SimResult( - succeeded: true, - message: "Launched \(resolvedBundleId) on \(udid)\(note)\n\(launch.stdout)") + var message = "Launched \(resolvedBundleId) on \(udid)\(note)\n\(launch.stdout)" + if let url { + let opened = try await env.shell.xcrun(timeout: 30, "simctl", "openurl", udid, url) + message += opened.succeeded ? "\nOpened \(url)" : "\nOpening \(url) failed: \(opened.stderr)" + } + if let pid = AppLiveness.pid(fromLaunchOutput: launch.stdout) { + let outcome = await AppLiveness.check( + pid: pid, bundleId: resolvedBundleId, launchedAt: launchedAt, env: env) + message += "\n" + AppLiveness.describe(outcome, bundleId: resolvedBundleId) + if outcome != .running { return SimResult(succeeded: false, message: message) } + } + return SimResult(succeeded: true, message: message) } if launch.timedOut { @@ -977,15 +1050,45 @@ public enum SimTools { // MARK: - Internal Helpers + /// `KEY=VALUE` entries as a dictionary. + public static func parseEnvironment(_ entries: [String]) throws -> [String: String] { + var result: [String: String] = [:] + for entry in entries { + guard let equals = entry.firstIndex(of: "="), equals != entry.startIndex else { + throw ResolverError("Environment entry '\(entry)' must be KEY=VALUE.") + } + result[String(entry[.. SimResult { + do { + let sim = try await env.session.resolveSimulator(simulator) + let udid = try await resolveSimulator(sim, env: env) + let result = try await env.shell.xcrun(timeout: 30, "simctl", "openurl", udid, url) + return result.succeeded + ? SimResult(succeeded: true, message: "Opened \(url) on \(udid)") + : SimResult(succeeded: false, message: "Opening \(url) failed: \(result.stderr)") + } catch { + return SimResult(succeeded: false, message: "\(error)") + } + } + /// Seconds `simctl launch` may take. Generous because a busy shared Mac can be slow to /// spawn the first process after boot. static let launchTimeout: TimeInterval = 60 + /// `environment` reaches the app (simctl passes on `SIMCTL_CHILD_`-prefixed variables). + /// `terminateFirst: false` brings a running app to the front instead of relaunching it. static func launchAppStructured( - simulatorUDID: String, bundleId: String, args: [String]? = nil, env: Environment + simulatorUDID: String, bundleId: String, args: [String]? = nil, environment: [String: String] = [:], + terminateFirst: Bool = true, env: Environment ) async throws -> StructuredAppLaunch { - let wasRunning = try await terminateAppIfRunning( - simulatorUDID: simulatorUDID, bundleId: bundleId, env: env) + var wasRunning = false + if terminateFirst { + wasRunning = try await terminateAppIfRunning(simulatorUDID: simulatorUDID, bundleId: bundleId, env: env) + } if wasRunning { try? await Task.sleep(nanoseconds: 500_000_000) } @@ -994,9 +1097,14 @@ public enum SimTools { if let args, !args.isEmpty { launchArgs.append(contentsOf: args) } + var childEnvironment: [String: String] = [:] + for (key, value) in environment { + childEnvironment["SIMCTL_CHILD_" + key] = value + } let result = try await env.shell.run( "/usr/bin/xcrun", arguments: launchArgs, + environment: childEnvironment.isEmpty ? nil : childEnvironment, timeout: launchTimeout ) @@ -1072,11 +1180,19 @@ extension SimTools: ToolProvider { await executeInstallApp(simulator: input.simulator, appPath: input.app_path, env: env)) } case "launch_app": - switch ToolInput.decode(AppInput.self, from: args) { + switch ToolInput.decode(LaunchInput.self, from: args) { case .failure(let err): return err case .success(let input): return dispatchResult( - await executeLaunchApp(simulator: input.simulator, bundleId: input.bundle_id, env: env)) + await executeLaunchApp( + simulator: input.simulator, bundleId: input.bundle_id, args: input.args, environment: input.env ?? [], + url: input.url, terminateFirst: input.terminate ?? true, env: env)) + } + case "open_url": + switch ToolInput.decode(OpenURLInput.self, from: args) { + case .failure(let err): return err + case .success(let input): + return dispatchResult(await executeOpenURL(simulator: input.simulator, url: input.url, env: env)) } case "terminate_app": switch ToolInput.decode(AppInput.self, from: args) { diff --git a/Tests/XCForgeKitTests/LogToolsTests.swift b/Tests/XCForgeKitTests/LogToolsTests.swift index 3372a4c..caa155d 100644 --- a/Tests/XCForgeKitTests/LogToolsTests.swift +++ b/Tests/XCForgeKitTests/LogToolsTests.swift @@ -359,17 +359,17 @@ struct ParseLineTests { @Test("parses fault level") func faultLevel() { let line = - "2026-03-29 22:26:04.200 Df MyTestApp[12345:67890] [com.test.myapp:default] Fatal error" + "2026-03-29 22:26:04.200 F MyTestApp[12345:67890] [com.test.myapp:default] Fatal error" let parsed = LogTools.parseLine(line) - #expect(parsed?.logType == "Df") + #expect(parsed?.logType == "F") } @Test("parses error level") func errorLevel() { let line = - "2026-03-29 22:26:04.200 De MyTestApp[12345:67890] [com.test.myapp:default] Error occurred" + "2026-03-29 22:26:04.200 E MyTestApp[12345:67890] [com.test.myapp:default] Error occurred" let parsed = LogTools.parseLine(line) - #expect(parsed?.logType == "De") + #expect(parsed?.logType == "E") } @Test("returns nil for dedup line") @@ -425,7 +425,7 @@ struct CategorizeTests { @Test("crashes topic for fault level") func crashesFault() { let parsed = LogTools.ParsedLogLine( - processName: "storekitd", logType: "Df", + processName: "storekitd", logType: "F", subsystem: "com.apple.storekit", category: "default") let topics = LogTools.categorize(parsed, bundleId: "com.test.myapp", processName: "MyApp") #expect(topics.contains("crashes")) @@ -506,7 +506,7 @@ struct CategorizeTests { @Test("multi-topic: app + crashes for own app fault") func multiTopic() { let parsed = LogTools.ParsedLogLine( - processName: "MyApp", logType: "Df", + processName: "MyApp", logType: "F", subsystem: "com.test.myapp", category: "default") let topics = LogTools.categorize(parsed, bundleId: "com.test.myapp", processName: "MyApp") #expect(topics.contains("app")) @@ -524,7 +524,7 @@ struct FilterByTopicsTests { static let appLog = "2026-03-29 22:26:04.112 Db MyTestApp[12345:67890] [com.test.myapp:default] viewDidLoad" static let appFault = - "2026-03-29 22:26:04.200 Df MyTestApp[12345:67890] [com.test.myapp:default] Fatal error" + "2026-03-29 22:26:04.200 F MyTestApp[12345:67890] [com.test.myapp:default] Fatal error" static let trustdLog = "2026-03-29 22:26:04.300 Db trustd[88:99] [com.apple.trust:policy] evaluating cert" static let lifecycleLog = @@ -568,7 +568,7 @@ struct FilterByTopicsTests { bundleId: "com.test.myapp", processName: "MyTestApp" ) #expect(result.topicCounts["app"] == 2, "appLog + appFault") - #expect(result.topicCounts["crashes"] == 1, "only appFault is Df") + #expect(result.topicCounts["crashes"] == 1, "only appFault is a fault") #expect(result.topicCounts["network"] == 1, "trustd") #expect(result.topicCounts["lifecycle"] == 1, "runningboardd") #expect(result.topicCounts["springboard"] == 1, "SpringBoard") diff --git a/Tests/XCForgeKitTests/RunLoopTests.swift b/Tests/XCForgeKitTests/RunLoopTests.swift new file mode 100644 index 0000000..5462905 --- /dev/null +++ b/Tests/XCForgeKitTests/RunLoopTests.swift @@ -0,0 +1,186 @@ +import Foundation +import Testing + +@testable import XCForgeKit + +@Suite("Run loop: launch checks, the right app, captures, clean", .serialized) +struct RunLoopTests { + + private func tempDir() -> String { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("xcforge-run-\(UUID().uuidString)", isDirectory: true).path + try? FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + return dir + } + + // MARK: - Launch + + @Test("the pid comes from the `bundle: pid` line, never from a URL's port") + func launchPid() { + #expect(AppLiveness.pid(fromLaunchOutput: "com.example.app: 4242") == 4242) + let message = "Launched com.example.app on ABC\ncom.example.app: 77\nOpened https://example.com:8080" + #expect(AppLiveness.pid(fromLaunchOutput: message) == 77) + #expect(AppLiveness.pid(fromLaunchOutput: "App running: true") == nil) + } + + @Test("launch environment entries must be KEY=VALUE") + func launchEnvironment() throws { + let parsed = try SimTools.parseEnvironment(["A=1", "URL=https://x?a=b"]) + #expect(parsed == ["A": "1", "URL": "https://x?a=b"]) + #expect(throws: (any Error).self) { try SimTools.parseEnvironment(["NOVALUE"]) } + } + + static let crashReport = """ + {"app_name":"App","bundleID":"com.example.app","bug_type":"309"} + { + "exception": {"type": "EXC_CRASH", "signal": "SIGABRT"}, + "asi": {"libsystem_c.dylib": ["abort() called"]}, + "usedImages": [{"name": "App"}, {"name": "libsystem_c.dylib"}], + "threads": [ + {"frames": [{"imageIndex": 0, "imageOffset": 10}]}, + {"triggered": true, "frames": [ + {"imageIndex": 1, "symbol": "abort"}, + {"imageIndex": 0, "symbol": "AppDelegate.setUp()"}, + {"imageIndex": 0, "imageOffset": 512} + ]} + ] + } + """ + + @Test("a crash report becomes the exception, reason and crashed thread's frames") + func crashSummary() throws { + let summary = try #require(CrashReports.summarize(Self.crashReport, path: "/r.ips")) + #expect(summary.headline == "EXC_CRASH (SIGABRT): abort() called") + #expect(summary.frames == ["libsystem_c.dylib abort", "App AppDelegate.setUp()", "App +512"]) + let text = AppLiveness.describe(.exited(summary), bundleId: "com.example.app") + #expect(text.contains("App running: false")) + #expect(text.contains("Crash report: /r.ips")) + } + + @Test("the newest report for the app since launch is the one picked") + func newestCrashReport() throws { + let dir = tempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let launchedAt = Date() + let mine = (dir as NSString).appendingPathComponent("App-1.ips") + let other = (dir as NSString).appendingPathComponent("Other-1.ips") + let old = (dir as NSString).appendingPathComponent("App-0.ips") + try Self.crashReport.write(toFile: mine, atomically: true, encoding: .utf8) + try Self.crashReport.replacingOccurrences(of: "com.example.app", with: "com.other") + .write(toFile: other, atomically: true, encoding: .utf8) + try Self.crashReport.write(toFile: old, atomically: true, encoding: .utf8) + try FileManager.default.setAttributes( + [.modificationDate: launchedAt.addingTimeInterval(-3600)], ofItemAtPath: old) + + #expect(CrashReports.newest(bundleId: "com.example.app", after: launchedAt, in: dir) == mine) + #expect(CrashReports.newest(bundleId: "com.missing", after: launchedAt, in: dir) == nil) + } + + // MARK: - The right app + + @Test("the scheme's application target is the product, not the last target listed") + func appProduct() throws { + let settings: [[String: Any]] = [ + [ + "target": "App", + "buildSettings": [ + "PRODUCT_TYPE": "com.apple.product-type.application", "PRODUCT_BUNDLE_IDENTIFIER": "com.example.app", + "BUILT_PRODUCTS_DIR": "/dd/Build/Products/Debug-iphonesimulator", "FULL_PRODUCT_NAME": "App.app", + ], + ], + [ + "target": "Widget", + "buildSettings": [ + "PRODUCT_TYPE": "com.apple.product-type.app-extension", "WRAPPER_EXTENSION": "appex", + "PRODUCT_BUNDLE_IDENTIFIER": "com.example.app.widget", + "BUILT_PRODUCTS_DIR": "/dd/Build/Products/Debug-iphonesimulator", "FULL_PRODUCT_NAME": "Widget.appex", + ], + ], + ] + let json = String(decoding: try JSONSerialization.data(withJSONObject: settings), as: UTF8.self) + let product = try #require(BuildTools.appProduct(fromSettings: json)) + #expect(product.bundleId == "com.example.app") + #expect(product.appPath == "/dd/Build/Products/Debug-iphonesimulator/App.app") + + let extensionData = try JSONSerialization.data(withJSONObject: [settings[1]]) + let onlyExtension = String(decoding: extensionData, as: UTF8.self) + #expect(BuildTools.appProduct(fromSettings: onlyExtension) == nil) + } + + @Test("with several schemes, the one named after the project or the only app scheme wins") + func preferredScheme() { + #expect(AutoDetect.preferredScheme(["App", "AppDev", "AppTests"], projectName: "App") == "App") + #expect(AutoDetect.preferredScheme(["Main", "MainTests", "Pods-Main"], projectName: "App") == "Main") + #expect(AutoDetect.preferredScheme(["One", "Two"], projectName: "App") == nil) + #expect(AutoDetect.preferredScheme(["Only"], projectName: "App") == "Only") + } + + // MARK: - Clean + + @Test("only a path shaped like one project's DerivedData folder can be deleted") + func derivedDataFolder() { + let folder = "/Users/me/Library/Developer/Xcode/DerivedData/App-abc" + #expect(BuildTools.derivedDataFolder(fromBuildRoot: folder + "/Build/Products") == folder) + #expect(BuildTools.derivedDataFolder(fromBuildRoot: "/tmp/Build/Products") == nil) + #expect(BuildTools.derivedDataFolder(fromBuildRoot: "/a/b/c/Build") == nil) + #expect(BuildTools.derivedDataFolder(fromBuildRoot: "relative/x/y/Build/Products") == nil) + } + + // MARK: - Logs and console + + @Test("reads return the newest 200 lines by default and say how many were left out") + func captureTail() { + let lines = (1...250).map { "line \($0)" } + let (kept, omitted) = CaptureTail.tail(lines, last: nil) + #expect(kept.count == 200) + #expect(kept.last == "line 250") + #expect(omitted == 50) + #expect(CaptureTail.tail(lines, last: 0).lines.count == 250) + #expect(CaptureTail.tail(lines, last: 5).lines == Array(lines.suffix(5))) + #expect(CaptureTail.keepEnd("abcdef", limit: 3).hasSuffix("def")) + } + + @Test("errors and faults are crashes; default-level lines are not") + func crashTopic() { + #expect(LogTools.isErrorOrFault("E")) + #expect(LogTools.isErrorOrFault("F")) + #expect(LogTools.isErrorOrFault("Fa")) + #expect(!LogTools.isErrorOrFault("Df")) + let line = "2026-03-29 22:26:04.618 Df log[43371:3a7ccf] (LoggingSupport) Sending stream request" + let topics = LogTools.parseLine(line).map { LogTools.categorize($0, bundleId: nil, processName: nil) } + #expect(topics?.contains("crashes") == false) + } + + @Test("lines read from a capture file collapse repeats like live capture") + func collapseRepeats() { + let a = "2026-03-29 22:26:04.112 Db App[1:2] [com.app:x] hello" + let a2 = "2026-03-29 22:26:05.112 Db App[1:2] [com.app:x] hello" + let b = "2026-03-29 22:26:06.112 Db App[1:2] [com.app:x] bye" + #expect(LogTools.collapseRepeats([a, a2, a2, b]) == [a, " ... repeated 2x", b]) + } + + @Test("a background capture's output survives this process and reads by stream") + func detachedCapture() async throws { + let dir = tempDir() + defer { try? FileManager.default.removeItem(atPath: dir) } + let capture = DetachedCapture(name: "test", directory: dir) + try capture.start( + executable: "/bin/sh", arguments: ["-c", "echo one; echo two >&2; echo \"$XCF_TEST\""], + environment: ["XCF_TEST": "three"], captureStderr: true, bundleId: "com.example.app") + for _ in 0..<50 where capture.isRunning { + try await Task.sleep(nanoseconds: 100_000_000) + } + #expect(capture.lines(.stdout) == ["one", "three"]) + #expect(capture.lines(.stderr) == ["two"]) + #expect(capture.state()?.bundleId == "com.example.app") + + let (lines, next) = capture.newLines(.stdout, after: 0) + #expect(lines == ["one", "three"]) + #expect(capture.newLines(.stdout, after: next).lines.isEmpty) + + capture.clear() + #expect(capture.lines(.stdout).isEmpty) + capture.stop() + #expect(!capture.exists) + } +} From 384d97ac858de25ff1605ccbd3fed7b4edcb8f15 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 9 Oct 2026 16:49:08 +0000 Subject: [PATCH 11/28] feat: UI driving that hits the right screen (bundle 10) - Every UI tool takes simulator; each simulator gets its own WDA port (saved in ~/.xcforge/wda/simulator-ports.json), XCFORGE_SIMULATOR for the CLI - Recreated WDA sessions send forceAppLaunch:false; restart reruns the cached xctestrun with test-without-building and the runner has no one-hour cap - The AXP fast path runs only with one simulator booted and no phone, reads that simulator's windows, and get_source/list_elements start WDA instead of falling back to Simulator.app - Smaller trees: empty wrappers, hidden and off-screen nodes dropped; value, disabled and selected shown - find_element reports every match and takes index/timeout/until_gone; actions take wait_for, until_gone and timeout - type_text types into the focused field, with key, dismiss_keyboard and secure (redacted) options - alert_action / XCFORGE_ALERT_ACTION and a visible-alert hint in not-found errors - Screenshots report device points, take crop and max_dimension, and never share a fixed /tmp file - sim_content_size, sim_locale, app_container, sim_push, sim_privacy (and sim content-size|locale|container|push|privacy) - Accessibility, localization and multi-device checks restore the text size, language and appearance they found - Coordinate taps and swipes account for interface orientation Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EtxtFPs6UBAWXqEpkj2W2N --- CHANGELOG.md | 15 + Skills/xcforge/references/cli-commands.md | 17 +- .../xcforge/references/screenshot-visual.md | 13 +- Skills/xcforge/references/simulator-tools.md | 72 ++++ Skills/xcforge/references/ui-automation.md | 52 ++- .../Screenshot/ScreenshotCommand.swift | 85 ++-- .../XCForgeCLI/Commands/Sim/SimCommand.swift | 143 ++++++- .../XCForgeCLI/Commands/UI/UICommand.swift | 59 ++- Sources/XCForgeKit/Clients/AXPBridge.swift | 13 +- Sources/XCForgeKit/Clients/AgentClient.swift | 183 +++++++-- .../XCForgeKit/Clients/IndigoHIDClient.swift | 76 +++- Sources/XCForgeKit/Plan/RunEngine.swift | 27 +- Sources/XCForgeKit/ProviderRegistry.swift | 9 +- .../XCForgeKit/Support/ReadinessProbe.swift | 3 +- .../Support/ScreenshotShaping.swift | 109 ++++++ Sources/XCForgeKit/Support/UITarget.swift | 45 +++ Sources/XCForgeKit/Support/UIWait.swift | 117 ++++++ Sources/XCForgeKit/Support/WDAPorts.swift | 47 +++ .../XCForgeKit/Tools/AccessibilityTools.swift | 27 +- .../XCForgeKit/Tools/CaptureProvider.swift | 121 ++++-- Sources/XCForgeKit/Tools/DeviceProvider.swift | 2 +- .../Tools/InteractionProvider.swift | 323 ++++++++++++---- .../XCForgeKit/Tools/MultiDeviceTools.swift | 15 +- Sources/XCForgeKit/Tools/SimSetupTools.swift | 364 ++++++++++++++++++ .../XCForgeKit/Tools/SimulatorProvider.swift | 6 +- Tests/XCForgeKitTests/UIDrivingTests.swift | 265 +++++++++++++ 26 files changed, 1928 insertions(+), 280 deletions(-) create mode 100644 Sources/XCForgeKit/Support/ScreenshotShaping.swift create mode 100644 Sources/XCForgeKit/Support/UITarget.swift create mode 100644 Sources/XCForgeKit/Support/UIWait.swift create mode 100644 Sources/XCForgeKit/Support/WDAPorts.swift create mode 100644 Sources/XCForgeKit/Tools/SimSetupTools.swift create mode 100644 Tests/XCForgeKitTests/UIDrivingTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 050275e..bf55104 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - Launch arguments, environment and a deep link: `args`, `env` (KEY=VALUE), `url` and `terminate` on `launch_app` and `build_run_sim`; `--arg`, `--env`, `--url` and `--no-terminate` on `sim launch` and `build run`; `--env` on `console launch` - `open_url` / `sim openurl` open a URL or deep link on a simulator - `clean` / `build clean` take `configuration` and `derivedData: true` / `--derived-data`, which deletes this project's DerivedData folder (and only that one) +- **Every UI tool takes `simulator`, and each simulator has its own WebDriverAgent** on its own port (8100 up, saved in `~/.xcforge/wda/simulator-ports.json`), so with two simulators booted the screenshot and the taps hit the same one. `XCFORGE_SIMULATOR` does the same for CLI `ui` commands +- **Actions can wait for the result:** `wait_for`, `until_gone` and `timeout` on taps, swipes, typing and `find_element`, instead of sleeps +- `find_element` reports how many elements matched, with each one's label and frame, and takes `index` +- `type_text` types into the focused field; `key` sends return, delete, tab and other named keys, `dismiss_keyboard` hides the keyboard, and `secure` keeps the text out of the result (`ui type --key`, `--dismiss-keyboard`, `--secure`) +- `alert_action` (accept/dismiss) on `wda_create_session` and `XCFORGE_ALERT_ACTION` let permission alerts be handled automatically; "not found" errors mention a visible alert +- `screenshot` takes `crop` (x,y,width,height in device points) and `max_dimension` (longest side in pixels, with how many points one pixel covers); `screenshot capture --crop`, `--max-dimension` +- Simulator setup tools: `sim_content_size`, `sim_locale`, `app_container`, `sim_push` and `sim_privacy` (grant/revoke/reset), and `sim content-size|locale|container|push|privacy`. Setters report the value they replaced ### Changed - **One test ID format.** Results, `list_tests`, last-failures and known-failures all use `Target/Suite/test()` (Swift Testing) or `Target/Class/testMethod` (XCTest), and filters accept it as printed. The target is added to short IDs from the scheme's or test plan's test targets instead of guessed, and the `test()()` spelling xcodebuild needs is written for you @@ -75,12 +82,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this - **`simRecovery` defaults to `off` everywhere** (was `auto` for `build_and_test`, `build-test` and `bless`). `auto` now only reboots; erasing a simulator needs the new `erase` mode. Invalid values are an error instead of silently becoming `auto`. `build-test` gains `--sim-recovery` - Result bundles and diagnostic snapshots get collision-free names (`xcf----`), so two sessions starting in the same second no longer delete each other's bundles. `XCFORGE_ARTIFACT_DIR` / `artifactDir` moves them out of `/tmp` - `build diagnose`, `test failures` and `test coverage` read this project's last recorded result bundle instead of the newest bundle any session left in `/tmp`. `test failures` without `--xcresult-path` reuses that bundle instead of re-running the whole suite +- `list_elements` and `get_source` drop wrapper containers, hidden and off-screen elements, and add `value`, `disabled` and `selected` +- `get_source` and `list_elements` start WebDriverAgent when it isn't running, and fail clearly instead of reading the Simulator app's own accessibility tree +- Screenshots report the device's point size, not the Simulator window's, and the fallback writes a new file per capture instead of one shared `/tmp` file; CLI `screenshot capture` writes a new file unless `--output` is given ### Fixed - A timed-out `build-for-testing` is reported as a build failure instead of continuing to `test-without-building` against stale products - `test_plan_inspect` finds plans inside `App.xcodeproj/xcshareddata/xctestplans` and decodes Xcode's string-form `skippedTests` and plans with missing sections - A hung build's fallback no longer runs `pkill -x xcodebuild`, which killed every session's build on the Mac. Hang snapshots sample the xcodebuild matched by this run's result bundle path instead of the newest xcodebuild on the system - WDA port cleanup only kills WebDriverAgent runner processes, uses the port from `WDA_BASE_URL`, skips remote WDA hosts, and calls `lsof` at its real path (`/usr/sbin/lsof`) +- A WebDriverAgent session recreated after a hiccup no longer relaunches the app (`forceAppLaunch: false`) +- Restarting xcforgeWDA reruns its last build with `test-without-building` instead of opening the runner app (which never starts the server), so recovery no longer falls through to a full rebuild; the runner is no longer stopped after an hour +- The accessibility-tree fast path is used only with exactly one simulator booted and no device attached, and reads that simulator's windows, not Simulator's menus +- Coordinate taps and swipes (`indigo_tap`, `indigo_swipe`, plan steps) account for landscape and upside-down orientation +- `accessibility_check`, `localization_check` and `multi_device_check` put back the text size, language and appearance they found instead of resetting to large, the first locale and light ## [1.6.1] - 2026-05-19 diff --git a/Skills/xcforge/references/cli-commands.md b/Skills/xcforge/references/cli-commands.md index 4e3c85f..9f3e168 100644 --- a/Skills/xcforge/references/cli-commands.md +++ b/Skills/xcforge/references/cli-commands.md @@ -347,6 +347,11 @@ xcforge sim location-reset # Clear GPS override xcforge sim appearance --appearance dark # Set light/dark mode xcforge sim statusbar --time "9:41" --battery-level 100 # Override status bar xcforge sim statusbar-clear # Restore default status bar +xcforge sim content-size accessibility-large # Dynamic Type size (prints the old one) +xcforge sim locale fr_FR # Locale + language (relaunch the app to apply) +xcforge sim container --container data # App container path (last built app) +xcforge sim push '{"aps":{"alert":"Hi"}}' # Deliver a push notification +xcforge sim privacy grant photos # Grant a permission before the app asks ``` All subcommands support `--json`. `install`, `launch`, `terminate` auto-detect simulator and bundle ID from session state. @@ -445,7 +450,9 @@ Capture simulator screenshots and manage visual baselines. Three subcommands — ```bash xcforge screenshot # Same as `xcforge screenshot capture` -xcforge screenshot capture # Capture to /tmp/xcforge-screenshot.png +xcforge screenshot capture # Capture to a new temp file (path printed) +xcforge screenshot capture --max-dimension 800 # Smaller image for a quick look +xcforge screenshot capture --crop 0,100,390,200 # One area, in device points xcforge screenshot capture --format jpeg --output /path/to/file.jpeg xcforge screenshot capture --grid # Include point-coordinate grid overlay xcforge screenshot baseline --name login-screen # Save as named baseline @@ -497,9 +504,8 @@ UI automation via WebDriverAgent. 17 subcommands — `status` is the default. xcforge ui status # Check WDA health xcforge ui session # Create WDA session xcforge ui session --bundle-id com.app.id # Bind session to app — verifies CFBundleIdentifier -xcforge ui ls # Flat element list — auto picks WDA when sim is booted -xcforge ui ls --source wda # Force WDA (iOS app tree); use this if sheets aren't visible -xcforge ui ls --source axp # Force macOS Accessibility (Simulator.app chrome) +xcforge ui ls # Flat element list (WDA; started when needed) +xcforge ui ls --source axp # macOS Accessibility tree of the one booted simulator xcforge ui ls --scope home.drawer.root # Restrict to a11y-id and its descendants xcforge ui find --using "accessibility id" --value "Save" xcforge ui find --using "accessibility id" --value "Save" --scroll @@ -516,6 +522,9 @@ xcforge ui pinch --center-x 200 --center-y 400 --scale 2.0 xcforge ui drag --from-x 100 --from-y 200 --to-x 300 --to-y 400 xcforge ui type --text "hello world" xcforge ui type --text "hello" --element-id --clear-first +xcforge ui type --text "secret" --secure --key return # Redacted, then press return +xcforge ui type --dismiss-keyboard +XCFORGE_SIMULATOR="iPhone 16" xcforge ui tap --x 200 --y 400 # Drive a specific simulator xcforge ui get-text --element-id xcforge ui source # Full view hierarchy (JSON) xcforge ui source --format xml diff --git a/Skills/xcforge/references/screenshot-visual.md b/Skills/xcforge/references/screenshot-visual.md index eb01a20..20100ce 100644 --- a/Skills/xcforge/references/screenshot-visual.md +++ b/Skills/xcforge/references/screenshot-visual.md @@ -9,6 +9,14 @@ Take a simulator screenshot. 0.3s latency — 44x faster than alternatives. | `simulator` | No | Auto-detect (booted) | Simulator name or UDID | | `format` | No | jpeg | Image format: `png` or `jpeg` | | `grid` | No | false | Overlay a point-coordinate grid on the image | +| `crop` | No | — | `x,y,width,height` in device points (the coordinates taps use) | +| `max_dimension` | No | — | Shrink so the longer side is at most this many pixels | +| `waitFor` / `timeout` | No | — | Readiness signal to wait for before capturing | + +**Token budget.** A full-resolution iPhone screenshot costs several thousand tokens. Pass +`max_dimension: 800` for a look at the screen, and `crop` to see one area at full detail. The +result line gives the device's point size, the crop, the output pixels and, when scaled, how +many points one pixel covers, so coordinates read off the image map back to taps. **3-tier capture strategy:** 1. **Burst** — native CoreSimulator IOSurface framebuffer access (~10ms) @@ -17,7 +25,10 @@ Take a simulator screenshot. 0.3s latency — 44x faster than alternatives. **Grid overlay:** When `grid: true`, overlays a transparent grid with 50pt minor lines and 100pt labeled divisions. Displays both X and Y axis labels at 100pt intervals. Falls back to ungridded image with a warning if overlay allocation fails. -**Returns:** Inline base64 image + metadata (resolution, byte size, capture method). +**Returns:** Inline base64 image + metadata: device size in points (from the simulator's +screen, not the Simulator window), pixel size, byte size, capture method. The CLI +(`xcforge screenshot capture`, with `--crop` and `--max-dimension`) writes a new file per capture +unless `--output` is given. Use `jpeg` (default) for fastest transfer. Use `png` for pixel-perfect visual regression baselines. Use `grid: true` for coordinate verification during layout work or UI automation scripting. diff --git a/Skills/xcforge/references/simulator-tools.md b/Skills/xcforge/references/simulator-tools.md index 5f4c8c9..5836d84 100644 --- a/Skills/xcforge/references/simulator-tools.md +++ b/Skills/xcforge/references/simulator-tools.md @@ -179,6 +179,78 @@ Set simulator appearance to light or dark mode. --- +## sim_content_size + +Read or set the Dynamic Type text size. Returns the size it replaced, so you can put it back. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `simulator` | No | Auto-detect (booted) | Simulator name or UDID | +| `size` | No | — (read) | `extra-small` … `extra-extra-extra-large`, `accessibility-medium` … `accessibility-extra-extra-extra-large`, `increment`, `decrement` | + +CLI: `xcforge sim content-size [size]` + +--- + +## sim_locale + +Read or set the simulator's locale (`AppleLocale`) and preferred language (`AppleLanguages`). +Returns the previous values. Apps pick up a change on their next launch. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `simulator` | No | Auto-detect (booted) | Simulator name or UDID | +| `locale` | No | — (read) | e.g. `fr_FR`, `ar_SA` | +| `language` | No | From the locale | e.g. `fr`, `pt-BR` | + +CLI: `xcforge sim locale [locale] --language ` + +--- + +## app_container + +Path of an installed app's container. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `simulator` | No | Auto-detect (booted) | Simulator name or UDID | +| `bundle_id` | No | Last built app | App bundle ID | +| `container` | No | `data` | `app`, `data`, `groups`, or an app group identifier | + +CLI: `xcforge sim container --bundle-id --container data` + +--- + +## sim_push + +Deliver a push notification. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `simulator` | No | Auto-detect (booted) | Simulator name or UDID | +| `bundle_id` | No | Last built app | App bundle ID | +| `payload` | **Yes** | — | APNs JSON (`{"aps":{"alert":"Hi"}}`) or plain alert text | + +CLI: `xcforge sim push ''` (or a path to a `.json`/`.apns` file) + +--- + +## sim_privacy + +Grant, revoke or reset a privacy permission so permission alerts don't block a flow. Changing a +permission may terminate the app. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `simulator` | No | Auto-detect (booted) | Simulator name or UDID | +| `action` | **Yes** | — | `grant`, `revoke`, `reset` | +| `service` | **Yes** | — | `all`, `calendar`, `contacts`, `contacts-limited`, `location`, `location-always`, `photos`, `photos-add`, `media-library`, `microphone`, `motion`, `reminders`, `siri` | +| `bundle_id` | No | Last built app | App to change; `reset` without one resets every app | + +CLI: `xcforge sim privacy grant photos --bundle-id ` + +--- + ## sim_statusbar Override simulator status bar values for clean screenshots. diff --git a/Skills/xcforge/references/ui-automation.md b/Skills/xcforge/references/ui-automation.md index 3c77128..ac620ef 100644 --- a/Skills/xcforge/references/ui-automation.md +++ b/Skills/xcforge/references/ui-automation.md @@ -4,6 +4,20 @@ All UI tools communicate directly with WebDriverAgent via HTTP — no Appium, no **Prerequisite:** WebDriverAgent must be running on the target simulator. Check with `wda_status`. +**Which screen.** Every UI tool takes `simulator` (name or UDID). Each simulator gets its own +WebDriverAgent on its own port (8100 up, saved in `~/.xcforge/wda/simulator-ports.json`), so with +two simulators booted the screenshot and the taps go to the same one. The last simulator named is +used when a call names none; until then calls go to the booted simulator. From the CLI, set +`XCFORGE_SIMULATOR=` (`XCFORGE_DEVICE` for a phone). + +**Waiting instead of sleeping.** Taps, swipes, typing and `handle_alert` take `wait_for` (an +accessibility id or label to appear), `until_gone` (one to disappear) and `timeout` (seconds, +default 10). The result says whether the wait held; a wait that times out marks the call as an error. + +**Permission alerts.** `alert_action: accept|dismiss` on `wda_create_session` (or +`XCFORGE_ALERT_ACTION`) has WDA answer system alerts by itself. `sim_privacy grant` grants a +permission before the app asks. A "not found" error says when an alert is in the way. + ## wda_status Check if WebDriverAgent is running and reachable. @@ -22,6 +36,10 @@ Create a new WDA session, optionally activating an app. |-----------|----------|---------|-------------| | `bundle_id` | No | — | App to activate (optional) | | `wda_url` | No | http://localhost:8100 | Custom WDA URL | +| `alert_action` | No | `none` | `accept` or `dismiss` system alerts automatically for this session | + +A session xcforge recreates by itself (after a WDA hiccup or restart) never relaunches the app +(`forceAppLaunch: false`), so screen state and launch arguments survive. **Bundle id binding (v1.4.1+).** When `bundle_id` is provided, `WDAClient` persists it as the active bundle id and threads it through every subsequent recreate @@ -81,18 +99,18 @@ additive (`encodeIfPresent`; old consumers ignore unknown keys). ## list_elements (CLI: `xcforge ui ls`) -Flat one-line-per-element listing. Format: ` |