feat(audit): declare a test suite intentionally not run in CI (reason + expiry), so deliberate gaps stop showing as uncovered

[patrickserrano]

Summary

#400 made lacquer audit report local-package suites no CI job runs. One real case is deliberate: momfriend's MomFriendCoreTests needs on-device models, so CI builds it (swift build --build-tests) and never runs it. None of the report's fixes fit (a selector, a swift test step, covered_elsewhere), so the entry stayed reported forever, which teaches people to skip the report.

This adds an honest, reasoned and time-boxed declaration:

[[project.not_run_in_ci]]
target = "MomFriendCoreTests"
reason = "needs on-device models; built in CI, run on device before release"
until  = "2026-12-31"

Spelling. [[project.not_run_in_ci]] sits beside [[project.covered_elsewhere]], which makes the opposite claim ("a workflow I own runs it"). I kept the suggested spelling. It uses the same array-of-tables form, under [project], where the audit's test-target inputs already live.

Behaviour

• In term: the target leaves "test targets no selector covers" (and "could not check") and gets its own printed line: deliberately not run in CI: <target> — <reason> (until <date>).
• Past until (the whole day is inclusive, same construction as depignore/exclusion): the target stays reported, a "declarations that have EXPIRED" block names it, and audit exits 4, the code an expired dependabot_ignore/exclude uses.
  • It still fires when the .xcodeproj is absent (multimeter's state), so an expiry can't silently stop being checked.
• Stale (reported, never gates): the target doesn't exist, or something now covers it: a selector, a verified covered_elsewhere, or a workflow the audit sees running it. The message names which one.
  • A target missing while a referenced package is unreadable is not called stale ("could not look" ≠ "not there").
• Native targets and package suites alike.
• The uncovered report's advice gains two lines pointing at the new declaration. Otherwise nobody discovers it, and this is the only output change in the fleet (below).

Validation (internal/config)

• All three fields are required. Rejected at load:
  • an empty or blank reason
  • a missing, malformed or impossible until (e.g. 2026-02-30), and an unquoted TOML date
  • a duplicate target
  • an unknown key. This is a plain struct, so the manifestTables reflection path names the table's accepted keys.
• Two further rejections:
  • A target also declared in covered_elsewhere. At most one of the two can be true.
  • A manifest without [project].xcodeproj. The audit reads targets from it, so the declaration and its date would never be evaluated.
• An already-expired entry still loads, so sync/fix keep working. Expiry is the audit's job, as it is for [baseline.relax].

lacquer fleet: Report.Blocking() now blocks on an expired declaration, using the same testtargets.Expired the audit uses, so the two can't disagree about the day it lapses. It also appears in the notes and on the expiry horizon.

• Refs fix(fleet): Blocking() claims to mirror audit's exit codes and does not check orphans #358, which tracks Blocking() drifting from audit's exit codes. This PR adds no new drift and leaves fix(fleet): Blocking() claims to mirror audit's exit codes and does not check orphans #358's orphan part alone.

cmd/lacquer/main.go: audit section only (passes the declarations in, and adds them to the exit-4 case). No console code touched.

Docs: site/src/content/docs/reference/commands.md has a new "A suite deliberately not run in CI" section, and the audit exit-code row and the package-suite paragraph now mention it. guides/ios-rules.md is a docsmirror of profiles/ios/CLAUDE.ios.md and is outside this change.

Rollout note: a lacquer older than this release rejects a manifest carrying the key ("unknown key", exit 1), as measured below. So momfriend should add its declaration only once its CI's pinned lacquer includes this release.

Tests (written first, watched fail)

• internal/config/not_run_in_ci_test.go: loads; the reflected table is reason,target,until; each validation error; past-until still loads.
• internal/testtargets/notrun_test.go: a momfriend-shaped fixture (ios/MomFriend.xcodeproj + local package ios/MomFriendCore, CI builds it and never runs it).
  • the no-declaration control
  • the declared suite moves to its own line, exit unchanged
  • expired: comes back and blocks
  • the day boundary at ±1ns
  • an unreadable until fails closed
  • stale because covered by a selector, and by a workflow run
  • native target, including stale because covered by a verified covered_elsewhere
  • stale because missing, and not stale beside an unreadable package
  • an "could not check" suite moves
  • stale + expired still blocks
  • a declaration alone is still printed
  • no declarations leaves the output byte-identical
• cmd/lacquer/not_run_in_ci_test.go (wiring through run([]string{"audit"})): undeclared → exit 0 and uncovered; in term → its own line, exit 0; expired → exit 4; expired with the .xcodeproj absent → exit 4.
• internal/fleet/fleet_test.go: expired → Blocking(), named in notes; in term (the until day itself) → not blocking, and on the horizon.

Local run of this repo's CI bar: gofmt clean, go vet ./..., go build ./..., go test -race on all packages except internal/shipped (all ok), and go test ./internal/shipped/ (ok, 100s).

Mutation testing (CLAUDE.md rule 2)

This ran on a separate copy of the tree. Each mutation was applied, the package's tests were run, and the file was restored and byte-verified. 33 mutations; every one is killed by a named test. The first pass had one survivor: dropping len(r.NotRun) == 0 from Format's empty check. A report whose only content was a deliberate line printed nothing, so the exception was invisible. TestADeclarationAloneIsStillPrinted was added and kills it. Six first-pass mutations didn't compile (Go unused-variable errors), so they proved nothing. They were rewritten to compile and re-run, and all six are killed.

Mutation	Killed by
date: boundary at midnight (now.After(d))	TestUntilIsInclusiveOfTheWholeDay
date: drop the -1ns	TestUntilIsInclusiveOfTheWholeDay
date: +2 days	TestUntilIsInclusiveOfTheWholeDay, TestExpiredDeclarationComesBackAndBlocks, TestStaleAndExpiredStillBlocks, TestDeclaredUncheckedSuiteMoves
date: After→Before	TestUntilIsInclusiveOfTheWholeDay (+7 more)
date: unparseable date treated as in term	TestUnreadableUntilIsExpired
stale: drop the "is a gap" case	TestDeclaredSuiteMovesToItsOwnLine, TestExpiredDeclarationComesBackAndBlocks, TestNativeTargetsAreDeclaredTheSameWay, TestDeclaredUncheckedSuiteMoves
stale: drop the "package unreadable" case	TestMissingTargetBesideAnUnreadablePackageIsNotStale
stale: drop the "missing" case	TestDeclarationForAMissingTargetIsStale
stale: drop the "verified covered_elsewhere" case	TestNativeTargetsAreDeclaredTheSameWay
stale: drop the "a workflow runs it" case	TestDeclarationForACoveredSuiteIsStale/a_workflow_runs_it
stale: a stale declaration still applied	TestDeclarationForACoveredSuiteIsStale (both)
apply: ignore Unchecked suites	TestDeclaredUncheckedSuiteMoves
apply: an expired declaration still applied	TestExpiredDeclarationComesBackAndBlocks, TestDeclaredUncheckedSuiteMoves, TestUnreadableUntilIsExpired
apply: Uncovered not filtered	TestDeclaredSuiteMovesToItsOwnLine, TestNativeTargetsAreDeclaredTheSameWay
apply: Unchecked not filtered	TestDeclaredUncheckedSuiteMoves
Blocking: counts everything	TestDeclaredSuiteMovesToItsOwnLine, TestDeclarationForACoveredSuiteIsStale, TestUntilIsInclusiveOfTheWholeDay
Blocking: counts nothing	TestExpiredDeclarationComesBackAndBlocks, TestStaleAndExpiredStillBlocks, TestUnreadableUntilIsExpired, …
Format: NotRun missing from the empty check	TestADeclarationAloneIsStillPrinted (added after it survived)
Format: formatNotRun not called	7 tests
main: no exit-4 gate	TestAuditHonoursNotRunInCI/expired…, /expired, project absent…
main: Deliberate not called	TestAuditHonoursNotRunInCI/in term…, /expired…
main: the absent-project branch dropped	TestAuditHonoursNotRunInCI/expired, project absent…
main: Blocking not captured when the project was read	TestAuditHonoursNotRunInCI/expired…
fleet: Blocking() ignores it	TestExpiredNotRunInCIBlocks
fleet: always expired	TestInTermNotRunInCIDoesNotBlock
fleet: never recorded	TestExpiredNotRunInCIBlocks, TestInTermNotRunInCIDoesNotBlock
config: each of the 7 checks removed (duplicate, covered_elsewhere conflict, xcodeproj, blank reason, empty until, unparsed until, target charset)	the matching TestNotRunInCIRejectsMalformedEntries subtest

Fleet dry-run

• Binaries: lacquer-before built from origin/main (654e0c8, v1.41.1); lacquer-after built from this branch.
• Root: the same LACQUER_ROOT (a git archive of origin/main) for both.
• Repos: fresh read-only --depth 1 clones of all 15 projects in fleet-ops/fleet.toml that have a repo, cloned today. git status --porcelain was empty in every clone afterwards. No live checkout was touched, and nothing was pushed anywhere.
• Result: no exit code changed anywhere. Nobody declares not_run_in_ci, so no finding moved. The only change is the two new advice lines, in the 7 repos that have an uncovered-targets section. Zero lines were removed in any repo.

project	before exit	after exit	diff
steps	4	4	+2 advice lines
a-bible-verse-each-day	4	4	none
port-of-entry	4	4	none
flare	0	0	+2 advice lines
kit	4	4	+2 advice lines
sleevetap	4	4	none
OutOfTheMouths	4	4	none
pixelfoxstudio.com	0	0	none
rail	0	0	none
dailybread	4	4	+2 advice lines
ShelfLife	4	4	+2 advice lines
Skein	4	4	none
momfriend	0	0	+2 advice lines
Windsock	4	4	+2 advice lines
multimeter	0	0	none

Every changed line, all 15 repos (each line appears in 7 repos: dailybread, flare, kit, momfriend, ShelfLife, steps, Windsock):

+ (x7)     A suite deliberately run only outside CI (on a device, by hand) can say so
+ (x7)     in [[project.not_run_in_ci]], with a reason and an until date.

Positive control: momfriend, with the declaration added locally

This was a scratch copy of the momfriend clone with the exact [[project.not_run_in_ci]] block above appended to .lacquer.toml. Today is 2026-09-18.

run	exit
lacquer-before, declaration present	1: load manifest: … has unknown key(s) (hence the rollout note above)
lacquer-after, until = "2026-12-31" (in term)	0
lacquer-after, until = "2026-09-17" (expired)	4
lacquer-after fleet over the expired copy + the pristine clone	4: FAIL momfriend-positive EXPIRED not_run_in_ci MomFriendCoreTests (2026-09-17)

In term, compared with the undeclared audit of the same commit (the lines that differ only by the temp-dir path are omitted):

-  MomFriendCoreTests  (unit tests, local package ios/MomFriendCore)
-    A local-package suite is also run by `swift test` in its package, and no
-    workflow here that a pull request starts does that either. Name it in
-    extra_test_targets (the scheme's TestAction must list it, or its selector
-    matches nothing and exits 0), add a step that runs
-    `swift test --package-path <package>` to a workflow a pull request starts,
-    or declare [[project.covered_elsewhere]] for the workflow that does run it
-    (checked against that file, not believed).
+deliberately not run in CI: MomFriendCoreTests — needs on-device models; built in CI, run on device before release (until 2026-12-31)
+    Declared in [[project.not_run_in_ci]]: nothing in CI runs these, on purpose,
+    and the reason says where they are run instead. Not a finding while in term.
+    Past its until date a declaration expires, the suite is reported again, and
+    this audit fails (exit 4) — so the gap comes back for review.

(momfriend's other uncovered target stays listed. Only the package paragraph goes, because the declared suite was the only package suite left.)

Expired (until = "2026-09-17"): the uncovered list is unchanged from undeclared, plus:

+not_run_in_ci declarations that have EXPIRED:
+  MomFriendCoreTests — EXPIRED 2026-09-17: needs on-device models; built in CI, run on device before release
+    The term ran out, so the suite is reported again and this audit fails
+    (exit 4). Run it in CI, delete it, or — if the reason still holds — review
+    it and set a new until date.