HA setup workflow

A proper Core HA setup configuration consists of the following steps:

•Step 1 – Set up network components

•Step 2 – Enable HA on Primary server

•Step 3 – Enable HA on Secondary server

•Step 4 – Exchange SSH keys between servers

•Step 5 – Configure and test HA sync on Secondary Core server

•Step 6 – Configure sync schedule and notifications

•Step 7 – Test and validate the Core HA environment

Step 1 – Set up network components

Set up the network components by completing all the necessary changes to the network to allow the Standby Core server to come up and function as the Primary Core. Setup and configure network traffic components such as load balancer or DNS as described in this section.

Table 1. Requirements for setting up the Core servers

Core Server	Collectivity	Software	Other
Primary	Bi-directional connection to Secondary Core over ports 8443, 443 & 22 in addition to standard Core FW requirements	Both Primary and Secondary servers must be running the same software release	Unique hostname for each Core
Secondary	Bi-directional connection to Primary Core over ports 8443, 443 & 22 in addition to standard Core FW requirements	Both Primary and Secondary servers must be running the same software release	Unique hostname for each Core

Step 2 – Enable HA on Primary server

Procedure 

1. Log into the Primary Core System Manager Portal.

2. Go to Maintenance > HA Configuration.

3. Go to the Service Configuration section.

4. Go to the High Availability option and select Enable.

5. Go to the Mode option and select Primary.

Step 3 – Enable HA on Secondary server

Procedure 

1. Log into the Secondary Core System Manager Portal.

2. Go to Maintenance > HA Configuration.

3. Go to the Service Configuration section.

4. Go to the High Availability option and select Enable.

5. Go to the Mode option and select Secondary.

6. Go to the Node Status option and select Passive.

Step 4 – Exchange SSH keys between servers

Procedure 

1. Log into the Primary Core System Manager Portal.

   1. Select Maintenance > HA Configuration.

   2. Go to the Service Configuration section and click Manage Trusted Hosts.

2. Log into the Secondary Core System Manager Portal.

   1. Select Maintenance > HA Configuration.

   2. Go to the Service Configuration section and click Manage Trusted Hosts.

3. Go to the Primary server and copy the Local Server SSH Key.

4. Go to Secondary server.

   1. Go to Add Trusted Host.

   2. Paste the key from the Primary server into the SSH Key field.

5. Go to the Secondary server and enter the hostname or IP address of the Primary server in Hostname/IP.

6. Then go to the Primary server and copy the Local server SSH key and paste it into the SSH key field on the Secondary server.

7. Click Add SSH Key to add the key to the configuration.

8. Go to the Primary server and enter the hostname or IP address of the Secondary server in Hostname/IP.

9. Go to the Secondary server and copy the Local server SSH key and paste it into the SSH key field on the Primary server.

10. Click Add SSH Key to add the key to the configuration.

11. Click Test Connection on both servers to ensure both systems can communicate with each other.

12. Click OK to close the HA - Test Connectivity window.

13. Click Save > Apply on both the Primary and the Secondary servers.

Important: Do not continue the setup/configuration if you do not see the Test was successful message on the HA - Test Connectivity window.

Step 5 – Configure and test HA sync on Secondary Core server

This step configures and tests the sync process testing the configuration of the Core HA components and if the replication is successful. This step assumes you are still logged into the System Manager Portal of the Secondary Core server. If you are not, log in now.

The initial sync will copies everything from one server to another while subsequent syncs only copy changes. Therefore, the initial sync will take more time than subsequent syncs.

Procedure 

1. Go to the Secondary server.

2. Go to the Secondary Configuration section.

3. Select the Primary server from the Hostname/IP drop-down menu.

4. Click Run Sync Now to test all the components in the HA environment.

5. Click Sync History to see the sync execution details.

6. Wait for the sync process to complete then click Refresh to update the Last Sync Status fields.

Step 6 – Configure sync schedule and notifications

The final step is configuring the HA sync schedule and process notifications. You should only start this step when you are satisfied that the two servers are communicating and able to execute a sync properly. Then you can configure the schedule any remaining parameters.

Procedure 

1. Go to the Secondary Configuration section.

2. Select a time in the StartTime (GMT) drop-down menu to set the starting time when the first replication is to occur. This time is in UTC time. Note that this value overwrites the scheduler every time you save this page.

3. Enter a value into the Run Every box to set how often the replication process executes. This value starts its reference point from the StartTime value. You can set the time in minutes, hours, or days.

4. Go to the Fail Over Controls section.

5. Enter the following values for:

   1. Heartbeat Interval: to control how often the Secondary server tries to connect to the Primary server to check it is alive.

   2. Consecutive Heartbeat failures before failover: to control how many consecutive connection attempts to the Primary can failed before attempting to promote the Standby server to Primary (active).

   3. Re-check interval: to define the amount of time between connection attempts when the previous attempt resulted in connection failure. This is the retry wait time between attempts.

6. Indicate what happens when the Primary is unavailable and a failover should occur by selecting one of the options for If Failure Detected:

   1. Keep in Standby: this option does not allow the Secondary to become Primary. The failover process gets initiated, notifications are completed, but the Secondary stays in standby mode.

   2. Promote to Primary: this option allows the Secondary to become Primary if the Primary becomes unavailable.

7. Go to the Notification Settings section and make the following selections:

   • Email: Enter a valid email address where the notifications should go to.

   • Notify On: Sync Completion (optional): When selected, the system generates a notification email every time the sync process executes and completes successfully.

   • Notify On: Failure / This server becomes primary (optional): When selected, the system generates an email notification every time the sync process executes and fails and/or when a Core HA failover event occurs.

8. Click Save and Apply.

Step 7 – Test and validate the Core HA environment

Use this test plan to help you test your Core HA systems.

Table 2. Test and validation plan for Core HA

Task	Description	Steps	Expected Results	Notes
1	Check System Connectivity	On Primary server, go to Manage Trusted Hosts > Test Connection (against the Secondary Core) On Secondary server, go to Manage Trusted Hosts > Test Connection (against the Primary Core)	Successful results from both tests.
2	Sync systems manually	On Secondary server, go to Select primary from the Primary Core Hostname menu. Click Run Sync Now.	In Sync History, all tasks show completed with no errors reported.
3	Verify Standby Core updated correctly	On Secondary server through the CLI Enable mode, start tomcat: >service tomcat start Log on through the Admin portal and confirm the device count matches that in the Primary Core.	The device count is exactly the same as the Primary.
4	Test Email Notification	On Secondary server, under Notification Settings, enter a valid email and click Test	A test email is sent by the Secondary server.
5	Confirm Heartbeat Process	On Standby server go to Troubleshooting > Logs select MICS > Apply. In View Module Logs select HighAvailability.	The heartbeat activity shows in the HighAvailability window.