Failover & Switchover

15.3.1 How does replication-manager prevent false positive failovers?

Multi-layer protection:

1. Multiple checks - Master failure is verified N times before acting

2. False positive detection:

  • Network flap detection
  • Heartbeat verification
  • Multiple monitoring endpoints
  • Quorum requirements with external arbitrator

3. Time-based protection:

  • failover-time-limit: Prevents repeated failovers within specified timeframe
  • Protects against flip-flop failures from same root cause

4. Default manual mode:

  • failover-mode = "manual" (default)
  • Sends alerts and waits for human intervention
  • User confirms failover via console, API, or CLI

5. Replication checks:

  • Valid slave must be available
  • Cluster configuration must be valid
  • Replication state checks (unless check-replication-state = false)

Best practice configuration:

failover-mode = "automatic"
failover-limit = 3
failover-time-limit = 10
failover-at-sync = false
failover-max-slave-delay = 30

Reference: /pages/05.configuration/03.failover/01.false-positive-detetection/docs.md

15.3.2 What happens when the entire cluster goes down?

Default behavior: replication-manager waits for the old master to recover.

Reason: First node to restart after total cluster failure could be a delayed slave, leading to significant data loss if promoted.

Configuration parameter: failover-restart-unsafe

When failover-restart-unsafe = false (default):

  • Prevents failover to first restarted slave
  • Waits for old master to show up
  • Prioritizes data safety over availability

When failover-restart-unsafe = true:

  • Allows failover to first restarted node
  • Favors availability when master cannot recover
  • Assumes DC crash brought down all nodes simultaneously (minimizing data skew)

Recommendation:

  • Use default (false) unless availability is critical
  • If using true, ensure automated slave startup procedures
  • In DC crash scenarios, start old master first when possible

Reference: /pages/04.architecture/02.failover-workflow/docs.md:57

15.3.3 Why did my failover get rejected?

Failover can be rejected for multiple reasons:

No valid slave available:

  • All slaves exceed failover-max-slave-delay (default: 30 seconds)
  • Highest-positioned slave doesn't match preferred master constraints
  • No slaves meet db-servers-prefered-master requirements
  • Candidate is in db-servers-ignored-hosts list

Failover limits reached:

  • failover-limit counter exceeded (default: 5)
  • Reset counter via console, HTTP, or API to re-enable

Time limit not met:

  • Previous failover within failover-time-limit window
  • Prevents flip-flop failures from same issue
  • Set to 0 for unlimited (not recommended)

Sync status requirement:

  • failover-at-sync = true but no slaves in SYNC status
  • Protects old master recovery at cost of availability

Cluster state invalid:

  • Replication configuration errors
  • Network partition detected
  • Arbitrator unreachable in multi-DC setup

Check status: Review logs and cluster state for specific rejection reason.

Reference: /pages/04.architecture/02.failover-workflow/docs.md:20

15.3.4 How long should I wait between failover attempts?

Parameter: failover-time-limit

Purpose: Prevents repeated failovers from the same root cause (flip-flop protection).

Recommended value: 10 seconds

failover-time-limit = 10

Behavior: If previous failover occurred within this time window, new failover is canceled.

Why this matters:

  • Hardware failure may affect multiple nodes sequentially
  • Network issues can cause cascading failures
  • Prevents promoting slaves that will immediately fail

Set to 0: Unlimited failovers (not recommended for production)

Reset counter: Use console/API to manually reset if legitimate failover needed within time window.

Reference: /pages/05.configuration/03.failover/docs.md:33

15.3.5 What does "failover-at-sync" mean for data safety?

Parameter: failover-at-sync

Purpose: Controls whether failover requires semi-sync SYNC status.

When failover-at-sync = false (default):

  • Failover proceeds even if semi-sync is not synchronized
  • Prioritizes availability over zero data loss
  • Some transactions may be lost

When failover-at-sync = true:

  • Failover only when at least one slave is in SYNC status
  • Prioritizes data consistency over availability
  • May prevent failover when most needed
  • Protects old master recovery

Recommendation for automatic failover:

  • Balanced: failover-at-sync = false with failover-max-slave-delay = 30
  • Minimize data loss: failover-at-sync = true with failover-max-slave-delay = 0

Trade-off: Strict sync requirements reduce availability during network issues or high load.

Reference: /pages/04.architecture/02.failover-workflow/docs.md:39

15.3.6 Why is my switchover stuck or timing out?

Problem: Switchover hangs or takes longer than expected.

Common causes:

1. FLUSH TABLES WITH READ LOCK timeout:

  • Parameter: switchover-wait-kill (default timeout)
  • Long-running queries prevent FTWRL from acquiring lock
  • Switchover waits for queries to complete

Solution:

  • Kill long-running queries manually
  • Adjust switchover-wait-kill timeout
  • Pre-check for long queries before switchover:
SELECT * FROM information_schema.processlist
WHERE command != 'Sleep' AND time > 10;

2. Slave lag:

  • Switchover waits for slaves to catch up
  • Check SHOW SLAVE STATUS lag on all slaves
  • Large transactions can delay catchup

3. Semi-sync timeout:

  • If semi-sync is slow or timing out
  • Network issues between master and slaves

4. Proxy reconfiguration delays:

  • HAProxy, ProxySQL, or MaxScale taking time to update
  • Check proxy logs and status

Monitoring: Enable higher log verbosity to debug:

log-level = 4

Reference: /pages/05.configuration/04.switchover/docs.md

15.3.7 When will replication-manager NOT perform automatic failover?

Failover is prevented when:

Cluster state issues:

  • [ ] No valid slave available
  • [ ] All slaves exceed failover-max-slave-delay
  • [ ] Cluster configuration is invalid
  • [ ] Network partition without arbitrator quorum

Configuration limits:

  • [ ] failover-mode = "manual" (default)
  • [ ] failover-limit reached (default: 5 failovers)
  • [ ] Previous failover within failover-time-limit window
  • [ ] failover-at-sync = true but no slave in SYNC status

Replication constraints:

  • [ ] All slaves with highest GTID position are in db-servers-ignored-hosts
  • [ ] No slaves match db-servers-prefered-master at highest position
  • [ ] check-replication-state = true and replication is broken

All cluster down:

  • [ ] failover-restart-unsafe = false (default) and old master not recovered yet

Manual override available: User can force failover via console/API by temporarily disabling checks.

Reference: /pages/04.architecture/02.failover-workflow/docs.md

15.3.8 How do I keep the MySQL/MariaDB Event Scheduler enabled on the master after failover?

Problem: When read_only is set on slaves, MySQL/MariaDB automatically disables the Event Scheduler. After a failover or switchover, the new master needs the Event Scheduler re-enabled, but this doesn't happen automatically without configuration.

Solution: Set the following in your replication-manager configuration:

failover-event-scheduler = true

What this does:

When enabled, replication-manager automatically manages the Event Scheduler across topology changes:

  1. On failover/switchover: Enables SET GLOBAL event_scheduler=1 on the newly promoted master
  2. On demotion: Disables SET GLOBAL event_scheduler=0 on the old master when it becomes a slave and is set to read-only
  3. Continuous monitoring: If the master doesn't have the Event Scheduler running and the flag is set, replication-manager auto-enables it during the next monitoring tick — so even a manual restart of the master will get the scheduler re-enabled

No post-failover script is needed. The toggle is fully automated on both promotion and demotion.

GUI: The setting can also be toggled from the dashboard under Settings → Failover → Event Scheduler.

API: PUT /api/clusters/{name}/settings/actions/switch/failover-event-scheduler

15.3.9 Why was my switchover cancelled with "Long updates running on master"?

INFO   Starting master switchover
WARN   Long updates running on master db1:3306: 1 write query/transaction past switchover-wait-write-query=10s, waiting for them to complete, 9s left of switchover-wait-trx=10s
WARN   Long update on db1:3306: session 312619 user app host 10.0.0.5:35878 command Query running 15s, trx open 16s rows modified 1200 rows locked 1200: UPDATE orders SET ...
ERROR  Long updates running on master. Cannot switchover: 1 write query/transaction on db1:3306 still past switchover-wait-write-query=10s after switchover-wait-trx=10s, not killed (rollback time unknown)

A write query, or an InnoDB transaction left open, has been running on the master for at least switchover-wait-write-query seconds (default 10). Nothing is locked yet at that point, so the switchover waits for it, re-checking every 2 seconds for up to switchover-wait-trx seconds (default 10), and proceeds as soon as it is gone. The WARN lines name each session it is waiting for. If it is still running at the deadline the switchover is cancelled, and the transaction is never killed: its rollback would take an unknown time and run on the server being demoted.

What to do:

  1. Let the transaction finish and run the switchover again. The session id, its age and its rows modified are in the log lines above; the Top page lists it too.
  2. Make the switchover wait longer: raise switchover-wait-trx, since 3.1.42 under Settings → Replication Failover → Switchover Wait Transactions (no restart), before that in the cluster configuration file with a restart of replication-manager.
  3. Kill the session yourself only if its rows modified is small enough for the rollback to be immediate. A rollback of millions of rows cannot be stopped, not even by restarting the server.

There is no force option that kills the transaction, by design. A failover does not run this check.

The alert WARN0217 opens right after the switchover with the outcome of the wait, completed or cancelled, so the alert history keeps a trace of it.

See Switchover configuration.