Skip to content

Commit 7afbc24

Browse files
authored
chore: implement phase 1 (#2779)
* chore: implement phase 1 Strict parsers for the state and ruler endpoints (H1), a Prometheus-style duration parser, and fixtures sliced from real Grafana 13.1.0 payloads covering every required/optional-field and must-error case, including the "Normal (NoData)"/"Normal (Error)" composite reason states found live in the current fleet capture (not in the original plan's vocabulary). * chore: apply code review comments * chore: implement phase 2 (#2780) * chore: implement phase 2 Fix retry-error conflation, measure full poll latency, and harden Source test doubles for concurrency. * chore: enhance unit tests * chore: address code review comments * chore: implement phase 3 (#2781) * chore: implement phase 3 Add Resolve() for alert name resolution (uid:/Title/Folder/Title/ Folder/Group/Title forms, UID collapse, no-match suggestions) and the grafana-alertcheck CLI's list subcommand, the first runnable piece of the gate. Incorporates review fixes: reject empty path segments in classifyForm, guard uid: against an empty suffix, scope the no-match rule count and suggestions to supported rule kinds only, and exit 0 on -h/--help. * chore: enhance unit tests * chore: implement phase 4 (#2782) * chore: implement phase 4 Add per-rule poll timings, scheduler, and budget check (P4). - schedule.go: DeriveTimings, Scheduler, CheckBudget (§5) - Address review: add Folder/Title resolve test, rename CheckBudget's minPollEvery to tightestUID * chore: enhance unit tests * chore: address code review comments * chore: implement phase 5 (#2783) * chore: implement phase 5 Add the JSONL evidence log (P5). - log.go: Header/Poll records, reduction, H2 transition markers, §3.2 verification, append-only Writer with flock, ReadLog - flock_unix.go: non-blocking exclusive lock, unix only - schedule.go: DeriveTimingsFromLog — log-mode cadence comes from the header, never from the definitions * chore: remove unix build tags * chore: address code review comments * chore: implement phase 6 (#2784) * chore: implement phase 6 Invariant defended: H2. The one question: can watch return success over a window that nothing is recording? Watch() records the first observation of each non-skipped rule, then detaches a child that polls at the cadence in the header. The parent returns only after the child reports ready on an inherited pipe, and writes the pidfile after that. A clean stop writes the sentinel; a hard error does not. * chore: add a unit test, remove build tags * chore: address code review comments * chore: implement phase 7 (#2785) * chore: implement phase 7 Invariant defended: H3. The one question: can a rule be called alive because it looked alive one poll ago? proveCoverage (grafana-alertcheck/internal/gate/coverage.go) is the pure coverage function: nine checks over one rule's polls — sentinel, from-bounds, heartbeat continuity, health error/nodata, liveness, in-window pause, rule absence, KeepLast. Liveness is absolute, never a delta. Cross-domain comparisons translate by each poll's own skew and widen boundary segments by its skew bound, fail-closed. * chore: enhance unit tests * chore: address code review comments * chore: implement phase 8 (#2786) * chore: implement phase 8 Invariant defended: H6/H7. The one question: can a violation ever outrank an unobservable rule, or can a pass happen without Violations empty and err nil? Adds classify.go: the pure per-instance classifier (outcome table, preexisting policy, BadFor) and decide(), the seam combining proveCoverage with those timelines under one Policy. Consolidates rule-poll filtering and skew translation onto pollsForRule/runnerTime, shared with coverage.go. * chore: rename some vars + add unit tests * chore: address code review comments * chore: implement phase 9 (#2787) * chore: implement phase 9 Invariant defended: H5/H7. The one question: can check report a pass over a window it did not prove? Check() is the I/O shell around the pure decide(). Single-step synthesizes the header and its own sentinel, so no mode flag reaches the pure layer. Log mode stops the recorder before the one full read. The header, not a definition re-resolved after the window closed, is the authority for what was paused when the window opened — it decides `skipped`, the drain set, and the transitionGrace max. The flock, not the pidfile, is the authority for whether a writer still exists. * chore: remove unix build tag * chore: implement phase 10 (#2788) * Wire watch/check subcommands to the gate library, with a table+JSON renderer and H6/H7 exit-code mapping. Extend Result with per-rule/global thresholds and a real skew bound; export SkewHardLimit; reject --states normal. * chore: fix goreleaser.yaml and add version command * chore: implement phase 11 (#2789) * chore: implement phase 11 Add coverage.go's declared-KeepLast check (no_data_state/exec_err_state, not just an observed reason) and close the remaining §22 gaps: newly_bad's no-early-exit clock assertion, a recorder-mode gap right after the deploy, a rule's own coverage gap overriding its own recovery, a genuinely skew-discriminating staleness test, exit-2 consequences on two Reason-only coverage tests, and end-to-end checks for log-name collapse, a truncated log, and the real watch-written state histogram. * chore: address code review comments * chore: more concise comments (#2790) * chore: more concise comments * fix: merge conflict * chore: shorten comments * fix: resolve conflict * chore: use testify's require in tests (#2792) * chore: use testify's require in tests * chore: move remaining assumptions to testify * chore: address code review comments * chore: fix logging and std out printing (#2798) * chore: fix logging and std out printing * chore: truncate to seconds when comparing from time * chore: add centralized docs (#2802) * chore: add centralized docs * chore: further update docs * chore: address code review comments (#2807) * chore: address code review comments * chore: get rid of goreleaser
1 parent 2945092 commit 7afbc24

77 files changed

Lines changed: 14832 additions & 77 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/grafana-alertcheck-release.yml

Lines changed: 0 additions & 34 deletions
This file was deleted.

grafana-alertcheck/.goreleaser.yaml

Lines changed: 0 additions & 33 deletions
This file was deleted.

grafana-alertcheck/.tool-versions

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# golangci-lint: keep in sync with devbox.json (used by CI in .github/workflows/linters.yml via `devbox run -- just lint`).
2+
golang 1.26.6
3+
golangci-lint 2.12.2

grafana-alertcheck/README.md

Lines changed: 43 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,46 @@
11
# grafana-alertcheck
22

3-
A CD quality gate for Grafana alerts: bookend a release with `watch` (record) and `check` (classify) to
4-
answer whether any watched alert was in a bad state during the release window.
3+
A CD quality gate for Grafana alerts. It bookends a release with two commands — `watch` (record) and
4+
`check` (classify) — and answers whether any watched alert was in a bad state during the release window.
55

6-
Under construction.
6+
```
7+
watch → your work → check
8+
```
9+
10+
`watch` starts a background recorder that polls each named alert into a JSONL log. After the work emits a
11+
`from`/`to` pair, `check` proves continuous coverage of that window, classifies each alert's state
12+
timeline, and exits `0`, `1`, or `2`.
13+
14+
It **fails closed**: if it cannot get an answer, it stops the release — never a pass on an unproven window.
15+
16+
## Quickstart
17+
18+
```bash
19+
export GRAFANA_URL=https://grafana.example.com
20+
export GRAFANA_TOKEN=…
21+
22+
grafana-alertcheck watch --out /tmp/run.jsonl --alerts alerts.txt
23+
./deploy.sh # emits deployed_at=<RFC3339>
24+
./verify.sh # emits finished_at=<RFC3339>
25+
grafana-alertcheck check --in /tmp/run.jsonl --from "$deployed_at" --to "$finished_at"
26+
```
27+
28+
Requires Grafana >= 13.0.0 and < 14.0.0. Connection details come from the environment only — the token is
29+
never a flag.
30+
31+
## Documentation
32+
33+
| Doc | Covers |
34+
| --- | ------ |
35+
| [`docs/index.md`](./docs/index.md) | Overview, quickstarts, exit codes, common surprises |
36+
| [`docs/how-alerts-are-evaluated.md`](./docs/how-alerts-are-evaluated.md) | Verdict model, coverage proof, health/liveness |
37+
| [`docs/advanced.md`](./docs/advanced.md) | Check budget, scheduling, why history isn't queried |
38+
| [`docs/architecture.md`](./docs/architecture.md) | Design invariants, the pure-function seam, recorder lifecycle |
39+
| [`docs/reference/cli.md`](./docs/reference/cli.md) | Full CLI reference — subcommands, flags, naming |
40+
| [`docs/reference/log-format.md`](./docs/reference/log-format.md) | The JSONL log schema, for debugging artifacts |
41+
42+
## Build
43+
44+
```bash
45+
go build ./... && go test ./...
46+
```

grafana-alertcheck/cmd/check.go

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
package main
2+
3+
import (
4+
"context"
5+
"encoding/json"
6+
"errors"
7+
"flag"
8+
"fmt"
9+
"io"
10+
"os/signal"
11+
"syscall"
12+
"time"
13+
14+
"github.com/smartcontractkit/chainlink-testing-framework/grafana-alertcheck/internal/gate"
15+
)
16+
17+
const checkUsage = "usage: grafana-alertcheck check [--in <file>] [--pidfile F] --from RFC3339 --to RFC3339 " +
18+
"[--alerts ...] [--folder F] [--states ...] [--preexisting ...] [--min-observed N] [--allow-paused] " +
19+
"[--nodata-is-unobservable] [--concurrency N] [--output json]"
20+
21+
// runCheck is the classify step's CLI surface: parse flags into a gate.Config,
22+
// run gate.Check, and translate its (Result, error) into output and an exit
23+
// code. All of the correctness lives in the gate package — this file's only job
24+
// is presentation and the exit-code mapping, which exitCode below keeps as one
25+
// pure function so it can be tested without a network.
26+
func runCheck(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
27+
fs := flag.NewFlagSet("check", flag.ContinueOnError)
28+
fs.SetOutput(stderr)
29+
fs.Usage = func() { fmt.Fprintln(stderr, checkUsage) }
30+
31+
common := registerCommon(fs)
32+
in := fs.String("in", "", "path of a log recorded by watch; empty selects single-step mode")
33+
pidfile := fs.String("pidfile", "", "pidfile of the recorder to stop before reading --in (default <in>.pid)")
34+
from := fs.String("from", "", "the moment the deploy finished, RFC3339 (required with --in)")
35+
to := fs.String("to", "", "the end of the window to classify, RFC3339 (required)")
36+
states := fs.String("states", "", "comma-separated bad states to classify against (default: firing)")
37+
preexisting := fs.String("preexisting", "", "how to judge an instance already bad at `from` (default: fail-unless-recovered)")
38+
minObserved := fs.Int("min-observed", 0, "minimum rules that must be observed (default: every resolved rule)")
39+
allowPaused := fs.Bool("allow-paused", false, "do not count a rule paused before the window against --min-observed")
40+
nodataIsUnobservable := fs.Bool("nodata-is-unobservable", false, "treat a sustained health=nodata as unobservable rather than a note")
41+
output := fs.String("output", "", `"json" writes the machine-readable Result to stdout in addition to the table; default is the table alone`)
42+
43+
if err := fs.Parse(args); err != nil {
44+
if errors.Is(err, flag.ErrHelp) {
45+
return 0
46+
}
47+
return 2
48+
}
49+
if fs.NArg() != 0 {
50+
fmt.Fprintf(stderr, "check: unexpected arguments %v\n", fs.Args())
51+
return 2
52+
}
53+
if *output != "" && *output != "json" {
54+
fmt.Fprintf(stderr, "--output: unknown value %q (only \"json\" is supported)\n", *output)
55+
return 2
56+
}
57+
58+
url, token, err := grafanaEnv()
59+
if err != nil {
60+
fmt.Fprintln(stderr, err)
61+
return 2
62+
}
63+
alerts, err := readAlerts(stdin, *common.alerts)
64+
if err != nil {
65+
fmt.Fprintln(stderr, err)
66+
return 2
67+
}
68+
stateList, err := parseStates(*states)
69+
if err != nil {
70+
fmt.Fprintln(stderr, err)
71+
return 2
72+
}
73+
preexistingPolicy, err := parsePreexisting(*preexisting)
74+
if err != nil {
75+
fmt.Fprintln(stderr, err)
76+
return 2
77+
}
78+
79+
cfg := gate.Config{
80+
URL: url,
81+
Token: token,
82+
Alerts: alerts,
83+
Folder: *common.folder,
84+
States: stateList,
85+
Preexisting: preexistingPolicy,
86+
MinObserved: *minObserved,
87+
AllowPaused: *allowPaused,
88+
NodataIsUnobservable: *nodataIsUnobservable,
89+
Log: *in,
90+
PidFile: *pidfile,
91+
Concurrency: *common.concurrency,
92+
Clock: gate.SystemClock{},
93+
Notes: newNoteStyler(stderr),
94+
}
95+
if *to == "" {
96+
fmt.Fprintln(stderr, "check: --to is required")
97+
return 2
98+
}
99+
t, err := time.Parse(time.RFC3339, *to)
100+
if err != nil {
101+
fmt.Fprintf(stderr, "--to: %v\n", err)
102+
return 2
103+
}
104+
cfg.To = t
105+
if *from != "" {
106+
f, err := time.Parse(time.RFC3339, *from)
107+
if err != nil {
108+
fmt.Fprintf(stderr, "--from: %v\n", err)
109+
return 2
110+
}
111+
cfg.From = f
112+
}
113+
114+
// SIGINT/SIGTERM cancel the run cleanly rather than leaving an operator's
115+
// Ctrl-C to kill the process mid-collection: Check's collection loop and
116+
// drain wait both already select on ctx.Done() (check.go), so this makes
117+
// an interrupted run fail the way every other could-not-check path does
118+
// — exit 2, never a silently truncated pass.
119+
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
120+
defer stop()
121+
122+
result, checkErr := gate.Check(ctx, cfg)
123+
124+
if checkErr != nil {
125+
fmt.Fprintln(stderr, checkErr)
126+
} else if err := renderTable(stderr, result); err != nil {
127+
fmt.Fprintln(stderr, err)
128+
}
129+
if *output == "json" {
130+
enc := json.NewEncoder(stdout)
131+
enc.SetIndent("", " ")
132+
if err := enc.Encode(result); err != nil {
133+
fmt.Fprintf(stderr, "encode --output json: %v\n", err)
134+
return 2
135+
}
136+
}
137+
return exitCode(result, checkErr)
138+
}
139+
140+
// exitCode is the whole exit-code mapping, kept as one pure function of
141+
// exactly what Check returns so it is testable without a network: err != nil
142+
// is exit 2 UNCONDITIONALLY — never 0 and never 1, even alongside real
143+
// violations, because an inability to check beats a violation and an error is
144+
// never a pass. Violations without an error is exit 1. Neither is exit 0.
145+
func exitCode(res gate.Result, err error) int {
146+
switch {
147+
case err != nil:
148+
return 2
149+
case len(res.Violations) > 0:
150+
return 1
151+
default:
152+
return 0
153+
}
154+
}

0 commit comments

Comments
 (0)