Multiple Database Server Recovery

This topic aims to provide reference examples of commands that need to be executed on different multiple database scenarios.

As you enter commands, the system will automatically run them and may prompt for additional information.

Server List:

  • db1.mydomain.com - primary database server (DB) (DB role)

  • db2.mydomain.com - secondary database server (No role)

  • as.mydomain.com - application server (AS) (No role)

  • dc.mydomain.com - data collector (DC) (No role)

  • laptop.mydomain.com - Admin workstation

Scenario 1: Build initial ecosystem

Standing up a brand new environment

Answer any on-screen prompts. For example, "Do you want to continue? [y/N] or "Is this corrent? [y/N]

  1. DB1: Setup primary database in wait for new standby.

fmos ecosystem enable-multi-db

  1. DB2: Have standby attach to primary.

fmos ecosystem join db1.as.mydomain.com

  1. DB1: Backup CA on primary.

fmos ca backup ca_backup

  1. Local machine: Move backup to other host.

If you lose this file, you cannot connect the application server to the new database.

scp user@db1.mydomain.com:/home/user/ca_backup

Scenario 2: Add secondary database server to the initial ecosystem

Repeat the steps in Scenario 1.

Scenario 3: Primary server goes down, fail over to secondary database server

You have the multiple database up and running when the hard drive on DB1 goes kaput.

  1. AS: Put into maintenance mode.

fmos maintenance begin

  1. DB2: Promote standby database VM.

fmos ecosystem promote

  1. DB2: Clean up glusterfs.

busctl call com.firemon.fmos1 / com.firemon.fmos.gluster.GlusterService remove_brick ss firemon db1.mydomain.com:/var/lib/glusterfs/volumes/firemon/brick

busctl call com.firemon.fmos1 / com.firemon.fmos.gluster.GlusterService detach_peer s db1.mydomain.com

  1. Local machine: Move ca_backup from host.

scp ca_backup user@db2.mydomain.com:/home/user/ca_backup

  1. DB2: Restore CA on promoted standby database server.

fmos ca restore ca_backup

fmos ecosystem recover-ca

  1. AS: Switch over application server.

fmos ecosystem switchover db2.mydomain.com

Scenario 4: Reinstate DB1 as the primary after a rebuild (Rebuilt as NO_ROLE server)

You’ve done the steps in Scenario 3 and have the system up and running. After a week, the hardware vendor replaced the hard drive on DB1 and you now want DB1 to be the primary as it’s in the same data center as the application server.

  1. AS: Put into maintenance mode.

fmos maintenance begin

  1. DB1: Joins as standby to DB2.

ecosystem join db2.mydomain.com

  1. DB1: Promote to primary.

fmos ecosystem promote

  1. DB1: Clean up glusterfs.

busctl call com.firemon.fmos1 / com.firemon.fmos.gluster.GlusterService remove_brick ss firemon db2.mydomain.com:/var/lib/glusterfs/volumes/firemon/brick

busctl call com.firemon.fmos1 / com.firemon.fmos.gluster.GlusterService detach_peer s db2.mydomain.com

  1. DB2: Reset server.

fmos factory-reset

  1. Local machine: Move ca_backup from host.

scp ca_backup user@db1.mydomain.com:/home/user/ca_backup

  1. DB1: Restore recover-ca.

fmos ca restore ca_backup

fmos ecosystem recover-ca

  1. AS: Switch over application server.

fmos ecosystem switchover db1.mydomain.com

  1. DB2: Add DB2 back as standby.

fmos ecosystem join db1.mydomain.com