High Availability To guarantee the highest level of data availability, the High Availability (HA) feature allows you to leverage an additional storage node to manage the underlying disk. Each storage node already comes built with all redundant hardware such as dual power supplies, multiple CPUs, two or more Host Bus Controllers (HBA), multiple network interfaces and so on. HA provides an additional layer of protection for other unforeseen system faults and zero-impact software upgrades. BrickStor HA nodes operate as active/active so additional performance can be gained depending on the application. High Availability Components A BrickStor High Availability Cluster consists of four main components: BrickStor Head Node – The Head Node is a hardware and software component responsible for managing underlying disk and presenting it as consumable data via SMB, NFS or iSCSI. BrickStor HA configuration consists of two Head Nodes communicating between each other with a shared configuration, system state and leverage a master election process. Both nodes always have identical hardware configurations and operate on the same software version. Some versions are backwards compatible but only during the upgrade process. Please reference the release notes to find an upgrade path. Heartbeat - Heartbeat is a method of Head Nodes communicating their health status. This is typically done over a dedicated network interface directly connecting both nodes. Additionally state is also communicated over the management interface "admin0". During complete loss of the node heartbeat the failover process will take place. RMM/iLO - RMM is Intel’s Remote Management Module and iLO is HPE’s Integrated Lights-Out management facilities for out-of-band server access. Both are proprietary dedicated hardware components embedded on the motherboard to provide hardware management during the lights-out scenarios. BrickStor HA relies on this interface during automated HA failover events to avoid split-brain situations. Split-brain is when heartbeat communications are compromised but both nodes are online and healthy. Witness – The witness is an essential component for leveraging automated failover events. It is used to act as the third party in the quorum to break a tie. A witness is a software component that can either run Windows Server or Linux as virtual machine or a bare metal system. It installs as a lightweight service and communicates with both HA Head Nodes via the management interface. The Witness does not take any part during manual failover initiated by a system administrator nor does it play any role in data presentation. Shared Storage – Shared Storage refers to the underlying physical or virtual disk accessible by both Head Nodes. Physical disk is presented with drive enclosures connected with redundant SAS connections to both nodes. It is highly advised to configure HA solutions with two or more enclosures and configure storage pool(s) with disks split across them. This ensures the solution can survive enclosure failure. Virtual disk refers to block storage volumes presented to BrickStor HA Head Nodes by one or more third-party SAN solution(s). In those cases BrickStor HA is acting as an NFS/SMB protocol server consuming SAN volumes via iSCSI/FibreChannel links. Storage Pool - A Storage Pool is an aggregation of physical or virtual devices describing physical characteristics of the storage system (capacity, performance and data redundancy). The pool is typically defined during system deployment and cannot be changed except to grow it by adding more devices. A given storage system can have one or more storage pools depending on the application. More on the storage pools can be found in Storage Pools section. In an HA configuration only a single Head Node can serve a given pool. The second node would simply wait to take over (failover). Be advised, one should not attempt to import or export pools using the CLI. This will result in data corruption. Always use RackTop supplied utilities such as BrickStor SP Manager. VNIC - A VNIC is a Virtual Network Interface which extends the functionality of a physical network port. VNICs are used by BrickStor HA to facilitate failover having data VNIC(s) float between the HA nodes. Use VNICs conservatively. Unusually large number of VNICs may affect failover times because each one must be reconstituted on failover. Resource Group – A Resource Group is a logical grouping of Storage Pools and one or more VNIC(s). An HA Cluster can have one or more Resource Group and are typically created during solution deployment time. Resource Groups can be modified, disabled, removed or moved between nodes. The following action can result in loss of data availability so use it with caution. Familiarize yourself with Managing Resource Groups before attempting to use them. Resource Group Pool States A pool within a Resource Group can be in one of five states when managing an HA cluster: Member of a Resource Group – Pool is part of an HA Resource Group and is Enabled. The enabled pool is imported on the specified node and the second node is ready for failover. Disabled Member of Resource Group – The pool is a member of a resource group but is administratively disabled. The disabled pool is exported from both nodes and data is not available. Once the Resource Group is enabled the pool will be imported on the specified HA node. Unmapped Pool – Pool is a member of the HA Cluster but is not assigned into any current Resource Groups. This typically results when the pool is protected from being imported on more than one node at a time or brought over from a foreign HA configuration. In this state the pool is not imported on either nodes and can either be assigned into a Resource Group (new or existing) or destroyed. Removed from Cluster – Pool is not a member of the HA configuration. In this state the pool is not imported on either node and can either be assigned into a Resource Group (new or existing) or destroyed. Missing – The pool devices are not accessible by both HA nodes. This can result from the drives being physically removed from the enclosures, loss of connectivity with a drive enclosures or SAN, or the drives are SED (Secure Encrypted Drive) and are currently locked. Standard Network Interfaces At a minimum an HA configuration requires each node to have at least three physical network interfaces. Management interface can also be referred to as "admin0". It is used for system management and HA communications. Heartbeat interface directly connects each node and is used for exchanging HA communications between the nodes. Data interface is client data access. This interface is typically composed of two or more physical interfaces aggregated together using LACP protocol (IEEE 802.3ad) It is highly advised to designate a secondary HA management interface over the data aggregate. This insures highly available network connectivity between the nodes and a witness server. The data aggregate interfaces should connect to two or more stacked high speed network switches. The HA witness server and RMM/iLO interfaces must reside on the same subnet as the HA management interface. HA Cluster Architecture HA Scenarios Loss of Management Network Connectivity Loss of the HA node’s management interface will prevent automated failover due to inability to communicate with the witness server. However, this will not directly impact the data availability. The system administrator would still be able to failover any Resource Groups using the second, healthy Head Node. To overcome this edge case an additional management interface can be established over the data aggregate and designated for HA communications. Loss of Data Network Connectivity Loss of data network entirely is highly unlikely when it is configured as an aggregate of two or more physical ports. In this unlikely event or when Head Node is configured with only a single data interface the HA can be configured to failover on data interface loss. Loss of RMM/iLO Connectivity RMM and iLO communication for HA functionality is only used as a third means of power state verification. Loss of lights-out interface has no impact on system functionality given the management and heartbeat interfaces are healthy. If all means of communication are unavailable for a given node, a failover event will take place. Manual Failover Manual failover is an action triggered by the system administrator to initiate a Resource Group(s) migration to the adjacent HA node. This action is typically performed as part of the solution maintenance or during upgrades. Automatic Failover Automatic failover takes place during HA node failure. In this event the surviving node will take over all the Resource Groups. After automatic failover takes place it is advised to disable the failed node to prevent further action. For example: Resource Groups can be configured to have a preferred node. Disabling a failed node will prevent resources from failing over back and forth in the event this HA node is exibiting an inconsistent behavior. Once the issue is cleared up or the failed node is repaired it can be enabled again to return it back to service. The HA Witness server must be online, healthy and accessible by the surviving node in order for the automatic failover to trigger. High Availability (HA) Best Practices Use a dedicated witness for each HA cluster. With HA witness being a VM be sure it is not running on the datastore using the same BrickStor HA shared storage. Use two or more resource groups to get more performance out of your BrickStor making it active/active. Use an LACP 802.1ad aggregate for the data network across two or more stacked network switches. This will boost network performance by load balancing traffic across multiple ports and improve availability. Avoid manually failing over Resource Groups with pools in the degraded state Degraded pools can take longer to import during failover and this can result in a self induced outage. Resolve pool issues first and only then fail over the Resource Group. Avoid using jumbo frames. Jumbo frames can boost performance for data transport. However, in NAS solutions it only fits in very specific environments and must be properly configured on all network devices. Improper use of jumbo frames can result in poor performance. Avoid using DNS hostnames for HA configuration. This eliminates dependency on DNS services. Configuring High Availability Prerequisites Before further instruction into the cluster setup wizard, the following prerequisites must be met in order to form a BrickStor SP HA cluster: All devices (2x HA Head Nodes and a witness server) must be properly connected and powered on. BrickStor SP Manager software installed and connected to both Head Nodes. Witness server: hiavd service must be installed and running. Must be able to ping both Head Nodes. Must be able to connect via TCP port 4746 to each Head Node telnet <node address> 4746 Head Nodes: Must be connected to disk enclosures with one or more disks present. Data pool must be created and accessible by both nodes. Heartbeat Ethernet port must be properly connected and configured. Data aggregate must be created and working. Once the following checks are completed you are ready to create the HA cluster using BrickStor SP Manager. Configuring Heartbeat BrickStor SP HA uses a heartbeat interface directly connecting both nodes over a dedicated network interface. In order to configure heartbeat interface do the following: Establishing Connection Choose one Ethernet port on each BrickStor SP node. Connect those ports together using a standard network patch cable. Modern servers do not require crossover cables. When the network connection is established, some hardware may not activate the port LED until the interface is configured in the OS. Creating Heartbeat Interface Using BrickStor SP CLI, list available physical interfaces by running: (Option -m displays MAC address which can be helpful to identify server’s network interfaces) dladm show-phys -m LINK SLOT ADDRESS INUSE CLIENT igb0 primary 0:1e:67:80:1d:8c yes -- igb1 primary 0:1e:67:80:1d:8d no -- igb2 primary 0:1e:67:80:1d:8f yes -- igb3 primary 0:1e:67:80:1d:8f no -- Configure heartbeat over the desired interface. For example, to use "igb2" interface run: dladm create-vnic -l igb2 hb0 Setting up Witness Server BrickStor HA Witness comes in the form of a single binary file shipped with each BrickStor system. It can be downloaded for either Windows Server or Linux by going to the web page of the BrickStor Appliance https://<BrickStor Admin0 IP>: The witness binary version must match the version of the HA Nodes. This process ensures one always has the correct binary for their deployment. Installing Witness (Windows) Retrieve a copy of the Windows hiavd executable by going to the web page of one of the BrickStor HA Head Nodes https://<BrickStor Admin0 IP>: Create a service home directory c:\racktop. Extract the downloaded hiavd.zip into the c:\racktop directory. Register as a Windows service Open a command prompt or Powershell as an Administrator Change directory to service home cd c:\racktop Install the service by typing hiavd.exe -install Configure the service to restart on failure by typing sc failure "hiavd" actions=restart/60000/restart/60000/restart/60000/60000 reset=0 Start service by typing sc start hiavd Configure Witness Firewall The Witness service communicates via TCP port 4746 as well as ICMP protocol with the HA Head Nodes. The traffic must be allowed for both inbound and outbound communication on the witness server. Open Windows Firewall configuration Using Control Panel open Firewall Control Panel\System and Security\Windows Defender Firewall. Select Advanced Setting. This will bring up a Windows Firewall Configuration window. Select Inbound Rules. Allow ICMP From the rules list find and edit File and Printer Sharing (Echo Request ICMPv4-In). Using the General tab be sure Action is set to Allow the connection. Using the Scope tab be sure Remote IP Address is set to Any IP Address. Click OK. Allow TCP port 4746 HA Head Nodes to communicate with the Witness service. Using the Action menu select New Rule… to create a new inbound firewall rule. In Rule Type select Port type For Protocol and Port use TCP and for Specific local ports enter 4746. For Action select Allow the connection. For Profile select all available profiles or choose ones that apply to your environment. For Name enter a meaningful name such as RackTop BrickStor HA Witness TCP 4746. Click Finish. When Antivirus software is installed on the witness server be sure to exclude hiavd service home directory c:\racktop from scans. Installing Witness (Linux) Dependencies The following prerequisite installations must be satisfied to install Witness on a Linux workstation: bzip ipmitool dmidecode To install them, enter the following command into your terminal: sudo yum install bzip2 ipmitool dmidecode -y Configure witness system’s firewall to allow inbound TCP port 4746 to communicate with BrickStor SP nodes. The below example uses firewalld. firewall-cmd --list-all firewall-cmd --permanent --zone=public --add-port=4746/tcp firewall-cmd --reload firewall-cmd --list-all In cases where outbound traffic is blocked, allow ports tcp/4746 and tcp+udp/623 from Witness to both BrickStor SP nodes. Install Witness (Linux) Ensure that installation prerequisites are satisfied (see above section). Download the Witness Open your BrickStor’s IP address using a web browser. url: https://<brickstor_ip>: Optionally, the Witness package can be downloaded using Secure Copy Protocol. scp root@xx.x.xx.xxx:/usr/racktop/bsrapid/static/witness/ha-witness-linux-xx.x.x.xxx.tar.bz2 . The authenticity of host 'xx.x.xx.xxx (xx.x.xx.xxx)' can't be established. RSA key fingerprint is SHA256:H3cUrhhTTTRSP1WBWSbh8gXRb8CrZmoIzIUFaNyaTgI. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'xx.x.xx.xxx' (RSA) to the list of known hosts. Password: ha-witness-linux-xx.x.xx.xxx.tar.bz2 Decompress the Witness tar -xf ha-witness-linux-xx.x.xx.xxx.tar.bz2 Copy the hiavd binary to /usr/sbin cp /root/ha-witness-xx.x.xx.xxx/hiavd /usr/sbin Make the binary executable. chmod 555 /usr/sbin/hiavd Configure hiavd as systemd service. Edit the hiavd.service file and add the contents below: vi /etc/systemd/system/hiavd.service [Unit] Description = RackTop Systems High Availability Daemon (hiavd) After = syslog.target network.target [Service] Type = simple ExecStart = /usr/sbin/hiavd -w /var/run -pid /var/run/hiavd.pid ProtectHome = true ProtectSystem = true Restart = on-failure RestartSec = 2 StandardOutput = journal StandardError = journal WorkingDirectory = /var/run [Install] WantedBy = multi-user.target Enable hiavd.service to run on boot. sudo systemctl enable hiavd.service Start hiavd for the first time. sudo systemctl start hiavd.service After creating your cluster using Brickstor SP Manager, the hiavd.service may exit. There will be an error in the gui stating the witness is faulted. If this happens, start the service again. Service Control The service can be stopped or restarted using standard systemd commands: sudo systemctl start hiavd.service sudo systemctl stop hiavd.service sudo systemctl restart hiavd.service sudo systemctl status hiavd.service High Availability requires time to be in-sync for all nodes as well as Witness. To enable Network Time Protocol (NTP): Install the NTP service. Modify the NTP configuration file, ‘/etc/ntp.conf’, with required options. Add reference clock peers to the configuration file. Add drift file location to the configuration file . Add optional statistics directory to the configuration file . Enable and start the NTP service. Check operation and synchronization status. Distributed Configuration Database (confd) Windows Install The following steps will guide the Distributed Configuration Database (confd) installation on a Windows system. The install handles all of the steps necessary for the confd instance to run as a standalone node (copying to proper directories, firewall, service install, etc). This does not actually join the cluster. Steps to join the cluster must be followed after installation is complete. From the target Windows machine, using a web browser, navigate to https://IP_OF_YOUR_BRICKSTOR. Login. Click the Download Witness for Windows. Unzip the file to the local Downloads folder. Right-click and choose Run as Admin. Select Install from the menu. Follow the prompts to install. View Existing Instances Launch the menu as administrator. Select option 3. To Remove Existing Instances Launch the menu as administrator. Select option 2. Select the instance intended for removal. Setup Confd To begin confd setup: confadm -instance <instance number> member show Ensure the member count is one. A new cluster cannot be joined if this HA cluster or node is already a member of one. The port being used for the PeerUrls. Setting peer address number: confadm -instance <instance number> member set-peer-address X.X.X.X:PORT_FROM_STEP_2 To join: confadm -instance <instance number> join There is a known bug where a display of setup failure may be shown. Validate true failure by running confadm member show. If the return shows more than one member, setup can be continued. Reset and Start Over Run confd.exe as admin and choose Remove option to remove the desired instance. Delete C:\Program Files\RackTop\Brickstor\confd\INSTANCE_NUMBER of the instance just removed. Perform the install steps over again. Forming HA Cluster Using BrickStor SP Manager select one of the Head Nodes and navigate to the System tab. Select Setup HA Cluster. This will bring up the HA setup wizard window. In the HA wizard window fill in the appropriate information Local Node - the node you are currently managing. Lets call it the first node. Remote Node - the second Head Node. Witness - HA witness server. Address - IP address or a hostname. Heartbeat - Heartbeat network interface directly connecting both Head Nodes. root pws - Root user password for each of the HA Head Nodes. Common Resource Group Physical Interface - sets data interface for the first Resource Group. It will also be used as a default data interface for additional Resource Groups. Common HA Comms Port (advanced) - HA communication port. This allows changing from the default TCP port 4746. To change the configuration of an existing HA cluster follow the same steps and enter new information. This can be handy should the IP addresses or another Default Resource Group interface needs to be established. Managing High Availability After the BrickStor HA cluster is formed it is managed from HA section in BrickStor SP Manager software. The HA Cluster section will present dynamic action buttons that will become visible depending on cluster status. Table 1. HA action buttons Button Action Adds new Resource Group Adds unmapped pool to HA configuration Configures advanced HA settings such as polling intervals, timeouts and failover on loss of data network Disables the Cluster Enables the Cluster. This action is only shown when HA Cluster is disabled. Rebalances the Cluster. Distributes Resource Groups according to their configured Preferred Node property. This action is only shown when at least one Resource Group is not on its preferred node. Along with action buttons find a round status ballon which changes in color depending on the HA cluster health state. Green is what you expect to see when everything is healthy otherwise the color will change followed by a message like the one below. You can also hover over the status balloon for status message. Green - all HA components are healthy Orange - one or more components are degraded and HA reliability is impaired. Red - one or more components are faulted and HA functionality is in critical state. Purple - commit change is in progress. HA Cluster Settings Clicking the gear icon next to HA Cluster allows the tuning of several advanced settings. Take extra care manipulating the following settings. It is highly advised to consult with RackTop support before changing the default values. Table 2. HA Settings Setting Default Value Description Auto move if network link changes unchecked enables/disables HA failover on loss of data network connectivity. Link change delay 3 seconds Waits n seconds after link state changes to down state before initiating failover. Power off in unresponsive checked When enabled, healthy node will forcefully power off unresponsive node using lights-out interface in order to safely facilitate failover. Power off if export pool times out checked When enabled, healthy node will forcefully power off peer node using lights-out interface in the event pool export timeout period is exceeded in order to safely facilitate failover. Export Pool Timeout 15 seconds On failover waits n seconds before forcefully powering off failed node. See Power off if export pool times out Sensor Poll Rate 5 minutes HA sensor polling interval in minutes Disabling and Enabling HA Head Nodes HA Cluster Head Nodes can be enabled or disabled. Disabling a node allows for taking one node out of the cluster for maintenance. When disabled the nodes do not assume any Resource Groups or participate in failover/move operations. Enabling HA Head Node returns in back into the HA cluster. Once enabled the node can assume Resource Groups and participate in failover/move operations. To Disable an HA Head Node: Using BrickStor SP Manager connect to one of the HA Head Nodes or select it from the list. In the Details pane, select the HA tab. Under HA Cluster, mouse over desired node and click the Stop button . The Disable HA Node dialog box will open with additional options. In the Disable HA Node dialog enter a Reason message. This message will be saved in the event log to provide a detailed explanation for this operation. (Optional) Check Prevent move resource groups to prevent automatically moving Resource Groups to the other HA Head Node. With this option selected all active Resource Groups on this node will become unavailable. Click the Disable button. Avoid using Prevent move resource groups option. This will result in loss of data availability for all active Resource Groups on this node. To Enable an HA Head Node: Using BrickStor SP Manager connect to one of the HA Head Nodes or select it from the list. In the Details pane, select the HA tab. Under HA Cluster mouse over desired node and click play button . Enable HA Node dialog box will open with additional options. In the Enable HA Node dialog check Acknowledge box to confirm delivery of the message provided during disabling operation. (optional) Check Prevent move resource groups to prevent automatically moving Resource Groups to this HA Head Node once enabled. (optional) Check Attempt rebalance cluster to attempt to perform rebalance operation now. This option can result in loss of data availability. First, familiarize with Moving Resource Groups before attempting to use this option. Click the Enable button. Enable button will only show when an HA Head Node is disabled. Managing Resource Groups The initial Resource Group is created when an HA cluster is formed, however, more can be created after the fact. Additional Resource Groups only apply to systems with two or more storage pools. When creating a Resource Group, the simple configuration contains a single pool and a single VNIC over the default interface defined during cluster creation. In other more complex configurations it is possible to create multiple VNICs, define VLAN tags, set MTU size, choose alternate data interfaces and/or define static routes to each VNIC. Existing Resource Groups can be managed by modifying the configured properties or by moving them manually between the HA Head Nodes. Resource Group Properties Description - text describing the purpose of this Resource Group (ex: "User Data") VNIC - Data sharing VNIC IP address in the form of CIDR notation (ex: 192.168.0.1/24) Route - Button for adding a static route for a given VNIC Pools - Storage Pool selection Select All - Checkbox for showing/hiding Unmapped Pools Node - Node where the Resource Group currently resides. When not specified, Resource Group will become Unmapped Resource Group. Preferred Node - Node where the Resource Group will reside after a Rebalance action. When set to None it will be ignored. Unmapped Resource Group Resource Groups that are not assigned to any HA Head Node are referred to as unmapped. Any resources allocated to this group are offline and unavailable for access. The Storage Pool(s) associated with this Resource Group will be in the exported state and VNIC configuration will not be present. Resource Group States There are two state messages that can be displayed next to the Resource Group name. The state messages will only show when a given Resource Group has Preferred Node property set moved either manually or automatically. Hover over the state message to display a detailed message showing an event timestamp and a reason. The state messages can be safely ignored or remediated as needed. "temp" - Indicates that this Resource Group resides not on its Preferred Node. This state would show when Resource Group was manually moved to another HA Head Node by a system administrator. To remediate, click the Rebalance icon to move Resource Groups to their preferred nodes. "auto-moved" - Indicates that this Resource Group has been moved to its Preferred Node by a Rebalance operation. Moving Resource Groups is a disruptive process and should be planned accordingly! Resource Group Health Resource Group health status is relayed via a balloon which will change colors accordingly. To see a detailed reason and timestamp of the last status change hover over the Resource Group or a status balloon. Green - all Resource Group components are healthy Orange - one or more Resource Group components are degraded and HA reliability is impaired Red - one or more Resource Group components are faulted and HA functionality is in critical state Purple - change commit is in progress Grey - this Resource Group is unmapped Creating Resource Groups Using BrickStor SP Manager connect to one of the HA Head Nodes or select it from the list. In the Details pane, select the HA section. Hover over the HA Cluster and then click the plus-R icon to add a Resource Group. This will bring up the HA Resource Group creation dialog. Another way to add a Resource Group is by hovering over one of the nodes and clicking the plus button . In the HA Resource Group dialog, enter the required information: Description - Enter meaningful text describing the purpose of this Resource Group (ex: "User Data") VNIC CIDR address - Enter the IP address using CIDR notation (ex: 192.168.0.1/24) Route - (optional) Add a static route for a given VNIC. Clicking this button will enter an Advanced Resource Group creation view. Pools Select at least one pool to be added to this Resource Group. Select All - (optional) When checked this will show Missing Pools. Node - Select the initial node where the Resource Group will reside once created. Preferred Node - (optional) Select the node where the Resource Group will reside after a Rebalance action. Click the Create button. Creating Advanced Resource Groups Creating advanced Resource Groups allows configuring additional properties for multiple VNICS, VLAN tags, and use interfaces other than the default cluster data interface. Using BrickStor SP Manager connect to one of the HA Head Nodes or select it from the list. In the Details pane, select the HA section. Hover over the HA Cluster and then click the plus-R icon to a Add Resource Group. This will bring up the HA Resource Group creation dialog. Another way to add a Resource Group is by hovering over one of the nodes and clicking the plus button . In the HA Resource Group dialog, enter the required and optional information. Click the Advanced button to show advanced property fields. Description - Enter meaningful text describing the purpose of this Resource Group (ex: "User Data") VNIC CIDR address - Enter IP address using CIDR notation (ex: 192.168.0.1/24) Over - Select a physical data interface for this VNIC to be created over. VID - Enter a VLAN ID. MTU - Enter a custom value for Maximum Transmission Unit (MTU). By default this will use "Auto" value to inherit MTU size of the physical interface. Description - Label describing this VNIC (ex: Replication). Route - (optional) Adds a static route for a given VNIC. Multiple entries are allowed. Destination - Route destination using CIDR notation (ex: 0.0.0.0/0) Gateway - Route gateway IP address. When VNIC IP address is already defined the value will default to the first host address of the subnet. (ex: 192.168.0.1) Add VNIC - (optional) Adds an additional VNIC. Pools Select at least one pool to be added to this Resource Group. Select All - (optional) When checked this will show Missing Pools. Node - Select the initial node where the Resource Group will reside once created. Preferred Node - (optional) Select the node where the Resource Group will reside after a Rebalance action. Click the Create button. Moving Resource Groups Resource Groups can move between HA cluster nodes automatically or can be manually triggered using BrickStor SP Manager. An automatic move can result by either a Rebalance operation or node failure. Manual moves typically take longer compared to a failover since the HA nodes are deconstructing and reconstructing resources, whereas in a failover event the failed node is dead and we are only reconstructing. Move times can also vary depending on the system’s configuration complexity. Having an unusually large amount of file systems, VNICs, static routes all contribute to extending the move/failover time. It is best to keep the configuration simple whenever possible and rather add more HA clusters to distribute complexity into multiple smaller configurations. This concept also reduces the outage impact or blast zone for the entire solution. The Moving Resource Groups action is disruptive and should only be used during system maintenance and upgrades. It does take only several seconds and most SMB/NFS clients are designed to recover from long IO waits. However, extra care should be taken to properly plan and execute this action according to own environment. To move a Resource Group Moving Resource Groups is a disruptive process and should be planned accordingly. Move requests do not trigger a change request and will execute upon clicking the Move button. Using BrickStor SP Manager select one of the Head Nodes and navigate to HA section. Click an arrow icon next to Resource Group to be moved. This will bring up the Resource Groups move dialog. Make your selections to continue or click the Cancel button to abort Selected - Select one or more Resource Group(s) with a single operation by using the checkboxes next to them. All on node - All Resource Groups on the specified node. An additional node selection drop down box will show. All unmapped - All unmapped Resource Groups All - All Resource Groups To - Destination HA Head Node where desired Resource Groups are to be moved to. Set preferred - Set/change Preferred Node to destination node used. Click Move to execute this move request