Skip to content

fix(ios): nightly cleanup skips while any other CI job runs on the host (cron can fire at peak), and never kills Xcode's own build services - #403

Open
patrickserrano wants to merge 1 commit into
mainfrom
fix/ios-cleanup-quiet-window
Open

patrickserrano wants to merge 1 commit into
mainfrom
fix/ios-cleanup-quiet-window

Conversation

@patrickserrano

Copy link
Copy Markdown
Owner

Hardens #395's nightly cleanup (profiles/ios/workflows/cleanup-ci.yml). The cron time is not the run time: on 2026-09-18 GitHub fired the 07:00Z schedule about five hours late. The runs landed at 12:04–12:18Z, at peak CI, and the old per-repository cleanups caused 11 CoreSimulatorService restarts. #395 is already safe by construction, because a running job's processes have a live parent. This PR adds exactly two changes on top.

1. Quiet-window gate

The whole kill and delete pass is skipped if any other GitHub Actions job is executing on this host.

  • Which jobs count: any Runner.Worker process except this job's own, across every runner and every OS user (ps -A, no uid filter). The host runs github-user and patrickserrano-user runners.
  • Finding its own worker: it walks the parent chain from $$ to the first process whose executable is Runner.Worker and excludes exactly that PID. The walk and the count read one ps snapshot, so they can't race each other.
  • Matching: on the executable, the first word of the command. grep Runner.Worker or a tail of _diag/Runner.Worker_*.log is not a job.
  • Log line: skipped: N other job(s) running on this host (<runner dirs>). A self-updated runner's bin.<version>/ is stripped from the dir. It exits 0 and does not stamp, so the next scheduled run retries.
  • Fail-safe: if the walk finds no worker of its own, the pass skips with a ::warning::. A ps failure or an unreadable count fails the run and nothing proceeds.
  • Dry run: reports would be skipped: … and then shows the selection anyway, the same way the existing busy-host gate does.
  • Levels: it applies to every level. It matters most for manual aggressive, which deletes booted CI-iPhone-* devices. This OS user's other runners boot devices under the same names.
  • No load-average gate, per the operator's decision.

2. Shared Xcode services are never killed

Nothing whose executable is under /Applications/Xcode*.app/ is selected, even when it is orphaned under _work. That covers SWBBuildService, XCBBuildService, SourceKit and the XPC helpers. The match is anchored at the start of the command, so a non-Xcode tool that merely names an Xcode path is still selected.

The xcodebuild client is killed, when orphaned under this runner's _work. That is the hung job the cleanup exists for. The exception is /Applications/Xcode*.app/Contents/Developer/usr/bin/xcodebuild as the executable. The PM decided this after I flagged a conflict: read literally, the brief would also have excluded xcodebuild, flipping #395's pid-104 "must be killed" test. The yml comment states the rule in these words.

Tests (internal/shipped/cleanup_ci_test.go)

The fake host now puts the shell running cleanup_main under this runner's Runner.Worker, the way a real run: step's bash is. The ps shim substitutes the shell's real $$ for an @SELF@ placeholder, so the walk runs from the real $$.

TestCleanupCIQuietWindow covers:

  • two other workers (another OS user's, and a same-user runner on bin.2.335.1) → skip, nothing touched, exact log line, no stamp;
  • another OS user's worker alone → skip;
  • manual aggressive → held back;
  • dry run → would be skipped plus the selection;
  • only its own worker, two hops up, alongside a grep Runner.Worker and a _diag tail → proceeds;
  • no worker in the chain → skip with a warning, no stamp;
  • a worker that exists but is not in the chain → counted as another job;
  • ps fails → non-zero exit, nothing acted on;
  • unreadable count → non-zero exit, nothing acted on.

TestCleanupCINeverKillsSharedXcodeServices uses six shared services, all orphaned ≥3h under _work by argv or cwd, and none is selected:

  • SWBBuildService and XCBBuildService;
  • SourceKitService in Xcode-beta.app;
  • Xcode Service.app, a path with a space;
  • SKAgent in Xcode_26.1.app;
  • an XCBBuildService with the xcodebuild path as an argument.

Its controls are all selected: the xcodebuild client from Xcode.app and from Xcode-beta.app, /usr/bin/log stream, and /usr/bin/env /Applications/Xcode.app/…/actool.

Mutation table

Each mutation was applied to the workflow, the cleanup tests were run, and the file was restored with a verified cmp. All 15 were killed by named tests.

# Mutation Killed by (among others)
G1 gate never called QuietWindow/another_Runner.Worker_skips_the_pass, /no_worker_in_the_parent_chain…, /a_manual_aggressive_run…
G2 own worker not excluded QuietWindow/only_its_own_worker,_up_the_parent_chain,_proceeds (+ every #395 test that expects a pass)
G3 walk checks only $$, never its parents QuietWindow/only_its_own_worker… (+ every #395 pass test)
G4 no own worker → proceeds (fail open) QuietWindow/no_worker_in_the_parent_chain_skips_with_a_warning
G5 counts only this OS user's workers QuietWindow/another_OS_user's_worker_alone_skips_the_pass
G6 Runner.Worker matched anywhere in the command QuietWindow/only_its_own_worker… (the grep/_diag controls)
G7 bin.<version> not stripped from the runner dir QuietWindow/another_Runner.Worker_skips_the_pass
G8 real skip does not return QuietWindow/another_Runner.Worker…, /no_worker_in_the_parent_chain…
G9 unreadable count not validated QuietWindow/an_unreadable_count_fails_safe
G10 ps failure reads as a skip (exit 0) QuietWindow/ps_failing_fails_the_run_without_acting
X1 Xcode exclusion removed NeverKillsSharedXcodeServices
X2 Xcode matched anywhere in the command NeverKillsSharedXcodeServices (the /usr/bin/env … actool control)
X3 xcodebuild exception removed NeverKillsSharedXcodeServices, SelectsOnlyThisRunnersOrphans (#395's pid 104)
X4 xcodebuild exception matched anywhere NeverKillsSharedXcodeServices (pid 1106)
X5 only Xcode.app, not Xcode-beta/Xcode_26.1 NeverKillsSharedXcodeServices

Two mutations were wrong on the first pass and were rerun. G5's uid came through an env var the harness does not pass through, and G6 left the $ anchor in place. Neither first run tested what it claimed. The table shows the corrected runs.

Locally green: gofmt -l . empty, go vet ./..., go build ./..., go test <all but shipped> -race (41 packages) and go test ./internal/shipped/ (133s). The cleanup tests also pass under macOS /bin/bash 3.2.

Proven on

This shared Mac, 2026-09-18, read-only (dry run). The library was extracted from this branch's workflow with the documented sed and run with /bin/bash as cleanup_main standard true schedule for each of the three patrickserrano runners. Nothing was killed, deleted, locked or written. /tmp/lacquer-ci-cleanup.lock and the pixelfox-dedicated-2 stamp were already there from #395's proof, before this run, and were unchanged after it. No workflow ran in any consumer repository.

(a) Another job running: the live host at 12:33Z. ps -A showed two live workers, pixelfox-2/bin/Runner.Worker and pixelfox/bin.2.335.1/Runner.Worker:

======== runner mac-mini-2 (pixelfox) ========
Level standard, dry run true, event schedule.
Runner mac-mini-2 as patrickserrano, work directory /Users/patrickserrano/Developer/actions-runner/pixelfox/_work.
DRY RUN: nothing will be killed, deleted or written.
::warning::This shell (pid 26263) has no Runner.Worker in its parent chain, so this job cannot be told apart from another job on the host; would be skipped rather than guess.
would be skipped: 2 other job(s) running on this host (/Users/patrickserrano/Developer/actions-runner/pixelfox-2, /Users/patrickserrano/Developer/actions-runner/pixelfox).
A real run would skip (above). Showing the selection anyway.
Orphaned processes under /Users/patrickserrano/Developer/actions-runner/pixelfox/_work, older than 3h00m:
  none
CI simulators (CI-iPhone-*, CI-Watch-*):
  none to delete

pixelfox-dedicated-2 and mac-mini reported the same line: would be skipped: 2 other job(s) running on this host (…pixelfox-2, …pixelfox). They also hit the existing busy gate on a booted CI-iPhone-35343054205. The warning is expected in a local shell, which has no worker above it. That is the fail-safe firing on real input.

(b) Idle: simulated with a ps shim, because the host was never idle and a local shell has no worker. The shim returned the host's real ps listing with the other live Runner.Worker rows removed. It also made the shell's parent a synthetic pixelfox/bin.2.335.1/Runner.Worker. The gate stayed quiet and the pass proceeded to the selection:

Level standard, dry run true, event workflow_dispatch.
Runner mac-mini-2 as patrickserrano, work directory /Users/patrickserrano/Developer/actions-runner/pixelfox/_work.
DRY RUN: nothing will be killed, deleted or written.
Orphaned processes under /Users/patrickserrano/Developer/actions-runner/pixelfox/_work, older than 3h00m:
  none
CI simulators (CI-iPhone-*, CI-Watch-*):
  none to delete

The first try of the shim had a printf bug, and ps exited non-zero. The gate printed ::error::ps failed, so the other jobs on this host cannot be counted. and the run exited 1 having done nothing. That is the fail-closed path on the real host, by accident.

The walk's one assumption, checked live. A real step's shell is a direct child of its Runner.Worker. On this host, worker 65966's child was 76102 /opt/homebrew/bin/bash -e …/pixelfox/_work/_temp/90c65f71-….sh, which is a run: step. In a job, the walk therefore finds its own worker in one hop.

The operator's long-lived Xcode daemons. Each has parent 1 and is hours to days old: Xcode Service (8d), Xcode's Python (7d), com.apple.dt.SKAgent and SourceKitService (5d), and DeviceHub (10h). Their cwds are /, ~/Developer/pixelfox/semistrategy and ~/Library/Containers/com.apple.dt.Devices/Data, all outside every runner's _work, so neither #395 nor this branch selects them. That matches the "none" above. As a counterfactual, an lsof shim put every process's cwd under pixelfox/_work. With that, #395's selection took all five of these Xcode daemons (pids 7723, 36159, 95541, 95543, 96068), and this branch's took none of them.

Until the fleet sync, consumer repositories keep whatever cleanup they last synced.

…st (cron can fire at peak), and never kills Xcode's own build services

GitHub fired the 07:00Z schedule five hours late on 2026-09-18, at peak CI,
so the cron time says nothing about whether the host is idle.

- Quiet-window gate: the pass runs only when no Runner.Worker other than this
  job's own (found by walking the parent chain from $$) is executing anywhere
  on the host, across every runner and OS user. Another job, or no own worker
  found, skips without stamping; a dry run reports the skip and shows the
  selection anyway. ps failing or an unreadable count fails the run, never
  proceeds.
- Shared Xcode services are never killed: nothing executing from
  /Applications/Xcode*.app/, matched on the executable at the start of the
  command. The xcodebuild client is the one exception: its orphan under this
  runner's _work is the hung job the cleanup exists for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant