Skip to content

Interceptor readiness probe cannot detect a stalled routing table refresh #1741

Description

@Fedosin

Report

The interceptor's /readyz probe is backed by a single health check: the routing
table's HasSynced(). That is a latch that can never go false once it has passed —
the code acknowledges this itself:

// pkg/routing/table.go
func (t *table) HealthCheck(_ context.Context) error {
	// TODO: HasSynced never fails after passing once, it is not testing health over time
	if !t.HasSynced() {
		return errNotSyncedTable
	}

	return nil
}

If the refresh loop stops making progress — a wedged informer, a signaler that
stopped firing, an error path in refreshMemory — the interceptor keeps serving an
increasingly stale routing table while reporting ready.

Expected Behavior

/readyz reflects health over time: when the routing table has not been
successfully refreshed for longer than a multiple of the informer resync period
(KEDA_HTTP_SCALER_CONFIG_MAP_INFORMER_RSYNC_PERIOD, default 60m), the probe
should fail so the pod is pulled out of rotation.

Actual Behavior

Once the initial sync has passed, /readyz returns 200 forever, no matter how
stale the routing table becomes. The user-facing symptom is 404s, or traffic sent
to deleted backends, on one interceptor pod out of many — exceptionally hard to
diagnose without this signal.

Steps to Reproduce the Problem

  1. Deploy the interceptor with at least one InterceptorRoute/HTTPScaledObject.
  2. Disrupt the refresh path (e.g. block informer events) without killing the process.
  3. Observe that /readyz keeps returning 200 while the served routing table
    diverges from the actual route objects.

Note: identified by code review of main @ 2ca9930c, not reproduced at runtime.

Logs from KEDA HTTP Add-on

n/a — the defect is the absence of a failure signal

HTTP Add-on Version

main @ 2ca9930c (post-0.15.0)

Kubernetes Version

Any

Platform

Any

Would you be open to contributing a fix?

Yes

Anything else?

Related closed issues for the opposite failure mode of the same check (failing at
startup): #1452, #1459.

A proposed fix exists: record lastRefreshTime and route count on each successful
refreshMemory, re-sync the table on the resync-period cadence (an informer
watching zero objects delivers no resync events, so a periodic refresh is needed
to avoid false positives on empty tables), and fail HealthCheck when the last
successful refresh is older than a multiple of that period.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions