Report
The interceptor's /readyz probe is backed by a single health check: the routing
table's HasSynced(). That is a latch that can never go false once it has passed —
the code acknowledges this itself:
// pkg/routing/table.go
func (t *table) HealthCheck(_ context.Context) error {
// TODO: HasSynced never fails after passing once, it is not testing health over time
if !t.HasSynced() {
return errNotSyncedTable
}
return nil
}
If the refresh loop stops making progress — a wedged informer, a signaler that
stopped firing, an error path in refreshMemory — the interceptor keeps serving an
increasingly stale routing table while reporting ready.
Expected Behavior
/readyz reflects health over time: when the routing table has not been
successfully refreshed for longer than a multiple of the informer resync period
(KEDA_HTTP_SCALER_CONFIG_MAP_INFORMER_RSYNC_PERIOD, default 60m), the probe
should fail so the pod is pulled out of rotation.
Actual Behavior
Once the initial sync has passed, /readyz returns 200 forever, no matter how
stale the routing table becomes. The user-facing symptom is 404s, or traffic sent
to deleted backends, on one interceptor pod out of many — exceptionally hard to
diagnose without this signal.
Steps to Reproduce the Problem
- Deploy the interceptor with at least one
InterceptorRoute/HTTPScaledObject.
- Disrupt the refresh path (e.g. block informer events) without killing the process.
- Observe that
/readyz keeps returning 200 while the served routing table
diverges from the actual route objects.
Note: identified by code review of main @ 2ca9930c, not reproduced at runtime.
Logs from KEDA HTTP Add-on
n/a — the defect is the absence of a failure signal
HTTP Add-on Version
main @ 2ca9930c (post-0.15.0)
Kubernetes Version
Any
Platform
Any
Would you be open to contributing a fix?
Yes
Anything else?
Related closed issues for the opposite failure mode of the same check (failing at
startup): #1452, #1459.
A proposed fix exists: record lastRefreshTime and route count on each successful
refreshMemory, re-sync the table on the resync-period cadence (an informer
watching zero objects delivers no resync events, so a periodic refresh is needed
to avoid false positives on empty tables), and fail HealthCheck when the last
successful refresh is older than a multiple of that period.
Report
The interceptor's
/readyzprobe is backed by a single health check: the routingtable's
HasSynced(). That is a latch that can never go false once it has passed —the code acknowledges this itself:
If the refresh loop stops making progress — a wedged informer, a signaler that
stopped firing, an error path in
refreshMemory— the interceptor keeps serving anincreasingly stale routing table while reporting ready.
Expected Behavior
/readyzreflects health over time: when the routing table has not beensuccessfully refreshed for longer than a multiple of the informer resync period
(
KEDA_HTTP_SCALER_CONFIG_MAP_INFORMER_RSYNC_PERIOD, default 60m), the probeshould fail so the pod is pulled out of rotation.
Actual Behavior
Once the initial sync has passed,
/readyzreturns 200 forever, no matter howstale the routing table becomes. The user-facing symptom is 404s, or traffic sent
to deleted backends, on one interceptor pod out of many — exceptionally hard to
diagnose without this signal.
Steps to Reproduce the Problem
InterceptorRoute/HTTPScaledObject./readyzkeeps returning 200 while the served routing tablediverges from the actual route objects.
Note: identified by code review of
main@2ca9930c, not reproduced at runtime.Logs from KEDA HTTP Add-on
HTTP Add-on Version
main@2ca9930c(post-0.15.0)Kubernetes Version
Any
Platform
Any
Would you be open to contributing a fix?
Yes
Anything else?
Related closed issues for the opposite failure mode of the same check (failing at
startup): #1452, #1459.
A proposed fix exists: record
lastRefreshTimeand route count on each successfulrefreshMemory, re-sync the table on the resync-period cadence (an informerwatching zero objects delivers no resync events, so a periodic refresh is needed
to avoid false positives on empty tables), and fail
HealthCheckwhen the lastsuccessful refresh is older than a multiple of that period.