fix(ios): nightly cleanup skips while any other CI job runs on the host (cron can fire at peak), and never kills Xcode's own build services - #403
Open
patrickserrano wants to merge 1 commit into
Conversation
…st (cron can fire at peak), and never kills Xcode's own build services GitHub fired the 07:00Z schedule five hours late on 2026-09-18, at peak CI, so the cron time says nothing about whether the host is idle. - Quiet-window gate: the pass runs only when no Runner.Worker other than this job's own (found by walking the parent chain from $$) is executing anywhere on the host, across every runner and OS user. Another job, or no own worker found, skips without stamping; a dry run reports the skip and shows the selection anyway. ps failing or an unreadable count fails the run, never proceeds. - Shared Xcode services are never killed: nothing executing from /Applications/Xcode*.app/, matched on the executable at the start of the command. The xcodebuild client is the one exception: its orphan under this runner's _work is the hung job the cleanup exists for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens #395's nightly cleanup (
profiles/ios/workflows/cleanup-ci.yml). The cron time is not the run time: on 2026-09-18 GitHub fired the 07:00Z schedule about five hours late. The runs landed at 12:04–12:18Z, at peak CI, and the old per-repository cleanups caused 11 CoreSimulatorService restarts. #395 is already safe by construction, because a running job's processes have a live parent. This PR adds exactly two changes on top.1. Quiet-window gate
The whole kill and delete pass is skipped if any other GitHub Actions job is executing on this host.
Runner.Workerprocess except this job's own, across every runner and every OS user (ps -A, no uid filter). The host runsgithub-user andpatrickserrano-user runners.$$to the first process whose executable isRunner.Workerand excludes exactly that PID. The walk and the count read onepssnapshot, so they can't race each other.grep Runner.Workeror atailof_diag/Runner.Worker_*.logis not a job.skipped: N other job(s) running on this host (<runner dirs>).A self-updated runner'sbin.<version>/is stripped from the dir. It exits 0 and does not stamp, so the next scheduled run retries.::warning::. Apsfailure or an unreadable count fails the run and nothing proceeds.would be skipped: …and then shows the selection anyway, the same way the existing busy-host gate does.aggressive, which deletes bootedCI-iPhone-*devices. This OS user's other runners boot devices under the same names.2. Shared Xcode services are never killed
Nothing whose executable is under
/Applications/Xcode*.app/is selected, even when it is orphaned under_work. That covers SWBBuildService, XCBBuildService, SourceKit and the XPC helpers. The match is anchored at the start of the command, so a non-Xcode tool that merely names an Xcode path is still selected.The xcodebuild client is killed, when orphaned under this runner's
_work. That is the hung job the cleanup exists for. The exception is/Applications/Xcode*.app/Contents/Developer/usr/bin/xcodebuildas the executable. The PM decided this after I flagged a conflict: read literally, the brief would also have excluded xcodebuild, flipping #395's pid-104 "must be killed" test. The yml comment states the rule in these words.Tests (
internal/shipped/cleanup_ci_test.go)The fake host now puts the shell running
cleanup_mainunder this runner'sRunner.Worker, the way a realrun:step's bash is. The ps shim substitutes the shell's real$$for an@SELF@placeholder, so the walk runs from the real$$.TestCleanupCIQuietWindowcovers:bin.2.335.1) → skip, nothing touched, exact log line, no stamp;would be skippedplus the selection;grep Runner.Workerand a_diagtail → proceeds;TestCleanupCINeverKillsSharedXcodeServicesuses six shared services, all orphaned ≥3h under_workby argv or cwd, and none is selected:Xcode-beta.app;Xcode Service.app, a path with a space;Xcode_26.1.app;Its controls are all selected: the xcodebuild client from
Xcode.appand fromXcode-beta.app,/usr/bin/log stream, and/usr/bin/env /Applications/Xcode.app/…/actool.Mutation table
Each mutation was applied to the workflow, the cleanup tests were run, and the file was restored with a verified
cmp. All 15 were killed by named tests.QuietWindow/another_Runner.Worker_skips_the_pass,/no_worker_in_the_parent_chain…,/a_manual_aggressive_run…QuietWindow/only_its_own_worker,_up_the_parent_chain,_proceeds(+ every #395 test that expects a pass)$$, never its parentsQuietWindow/only_its_own_worker…(+ every #395 pass test)QuietWindow/no_worker_in_the_parent_chain_skips_with_a_warningQuietWindow/another_OS_user's_worker_alone_skips_the_passRunner.Workermatched anywhere in the commandQuietWindow/only_its_own_worker…(thegrep/_diagcontrols)bin.<version>not stripped from the runner dirQuietWindow/another_Runner.Worker_skips_the_passQuietWindow/another_Runner.Worker…,/no_worker_in_the_parent_chain…QuietWindow/an_unreadable_count_fails_safeQuietWindow/ps_failing_fails_the_run_without_actingNeverKillsSharedXcodeServicesNeverKillsSharedXcodeServices(the/usr/bin/env … actoolcontrol)NeverKillsSharedXcodeServices,SelectsOnlyThisRunnersOrphans(#395's pid 104)NeverKillsSharedXcodeServices(pid 1106)Xcode.app, notXcode-beta/Xcode_26.1NeverKillsSharedXcodeServicesTwo mutations were wrong on the first pass and were rerun. G5's uid came through an env var the harness does not pass through, and G6 left the
$anchor in place. Neither first run tested what it claimed. The table shows the corrected runs.Locally green:
gofmt -l .empty,go vet ./...,go build ./...,go test <all but shipped> -race(41 packages) andgo test ./internal/shipped/(133s). The cleanup tests also pass under macOS/bin/bash3.2.Proven on
This shared Mac, 2026-09-18, read-only (dry run). The library was extracted from this branch's workflow with the documented
sedand run with/bin/bashascleanup_main standard true schedulefor each of the threepatrickserranorunners. Nothing was killed, deleted, locked or written./tmp/lacquer-ci-cleanup.lockand thepixelfox-dedicated-2stamp were already there from #395's proof, before this run, and were unchanged after it. No workflow ran in any consumer repository.(a) Another job running: the live host at 12:33Z.
ps -Ashowed two live workers,pixelfox-2/bin/Runner.Workerandpixelfox/bin.2.335.1/Runner.Worker:pixelfox-dedicated-2andmac-minireported the same line:would be skipped: 2 other job(s) running on this host (…pixelfox-2, …pixelfox). They also hit the existing busy gate on a bootedCI-iPhone-35343054205. The warning is expected in a local shell, which has no worker above it. That is the fail-safe firing on real input.(b) Idle: simulated with a ps shim, because the host was never idle and a local shell has no worker. The shim returned the host's real
pslisting with the other liveRunner.Workerrows removed. It also made the shell's parent a syntheticpixelfox/bin.2.335.1/Runner.Worker. The gate stayed quiet and the pass proceeded to the selection:The first try of the shim had a printf bug, and
psexited non-zero. The gate printed::error::ps failed, so the other jobs on this host cannot be counted.and the run exited 1 having done nothing. That is the fail-closed path on the real host, by accident.The walk's one assumption, checked live. A real step's shell is a direct child of its Runner.Worker. On this host, worker
65966's child was76102 /opt/homebrew/bin/bash -e …/pixelfox/_work/_temp/90c65f71-….sh, which is arun:step. In a job, the walk therefore finds its own worker in one hop.The operator's long-lived Xcode daemons. Each has parent 1 and is hours to days old:
Xcode Service(8d), Xcode'sPython(7d),com.apple.dt.SKAgentandSourceKitService(5d), andDeviceHub(10h). Their cwds are/,~/Developer/pixelfox/semistrategyand~/Library/Containers/com.apple.dt.Devices/Data, all outside every runner's_work, so neither #395 nor this branch selects them. That matches the "none" above. As a counterfactual, anlsofshim put every process's cwd underpixelfox/_work. With that, #395's selection took all five of these Xcode daemons (pids 7723, 36159, 95541, 95543, 96068), and this branch's took none of them.Until the fleet sync, consumer repositories keep whatever cleanup they last synced.