Skip to content

Regression in 2.2.8: operator restart may no longer trigger recovery for clusters already OFFLINE at startup #52

Description

@G-NamanGupta

Summary

After upgrading from 2.2.6 to 2.2.8, an existing InnoDBCluster that is already OFFLINE when the operator restarts may remain OFFLINE indefinitely if no further pod status change occurs after operator startup.

Versions

  • works as expected: 2.2.6 (9.5.0-2.2.6)
  • problematic: 2.2.8 (9.7.0-2.2.8)

Test setup

The reproduction I used was with an InnoDBCluster configured with 1 instance.

Steps to reproduce

  1. Deploy MySQL Operator and create an InnoDBCluster with 1 instance
  2. Put the cluster into a state where it is already OFFLINE when the operator comes back up
    • for example, after a node reboot or a simultaneous operator/pod restart scenario
  3. Let the operator restart
  4. Ensure no additional pod status change occurs after startup
  5. Observe the cluster status

Expected behavior

After operator restart, an existing cluster that is already OFFLINE should still be re-evaluated for recovery.

Actual behavior

In 2.2.8, the cluster may remain OFFLINE indefinitely unless a later pod status change happens.

Additional note

I compared the relevant source between 2.2.6 and 2.2.8, and this looks related to a change in the pod status watch / restart detection behavior after operator restart. I can provide the exact code comparison if that would be helpful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions