In the first two parts of this series, I have shown you how to deploy and configure Cluster Continuous Replication (CCR). Although those two articles walked you through the configuration process, I felt as though I had left my readers hanging a bit, because I did not talk about the Failover Cluster Management Console or about how server maintenance differs in a CCR environment from what you might be used to on a non clustered mailbox server. In this article, I want to take the opportunity to show you the basics of managing and maintaining your clustered mailbox server.

The Failover Cluster Management Console

The Failover Cluster Management Console is the GUI based interface for configuring and maintaining failover clustering. If you followed the steps outlined in the previous article then you have already had some exposure to this tool, but I want to show you a few of its features in a bit more depth.

Checking the Cluster’s Health

One of the most common uses for the Failover Cluster Management Console is checking the cluster’s health. When you select the Nodes container, the console will display the cluster nodes, as shown in Figure A. As you can see in the figure, you can use the Status column to verify that each node is up and running.

Figure A

The Nodes container displays the status of each cluster node.

As you look at the screen capture shown above, there is one important thing to keep in mind. CCR is based on a Majority Node Set (MNS) cluster, and an MNS cluster requires a minimum of three cluster nodes so that the cluster will be able to maintain quorum in a failure situation.

In the previous article I explained that CCR is limited to using two cluster nodes. As such, we created a MNS file share witness on the hub transport server. This file share takes the place of a third cluster node in a CCR environment. Since the hub transport server is not a cluster node though, the Failover Cluster Management Console does not report on its health.

If you need more detailed information about the individual cluster nodes, then you can simply expand the Nodes container and select the cluster node that you want to examine. As you can see in Figure B, the console will provide you with a brief summary of the selected node’s configuration, memory usage, and network connectivity.

While this information is handy to have, keep in mind that this portion of the Failover Cluster Management Console is incapable of providing you with anything more than a brief summary of the cluster’s health. Later on I will show you how to use the console to obtain more detailed information.

Figure B

Selecting a cluster node allows you to view more detailed information about the node.

The Clustered Mailbox Server

One thing that seems to confuse some Exchange administrators when they first begin clustering their mailbox servers is the way that clustered mailbox servers are portrayed within the Failover Cluster Management Console. Failover clustering is a Windows feature, not an Exchange feature. As such, the Failover Cluster Management Console knows nothing of CCR, and makes no references to the active or to the passive cluster node. Instead, the clustered mailbox server is treated as an application which can be moved between cluster nodes. The active node is defined as the Clustered Mailbox Server’s current owner.

If you look at Figure C, you can see that I have expanded the Services and Applications container to reveal the Clustered Mailbox Server (which is named Exch in the screen capture).  The upper portion of the console’s center pane lists the Current Owner, which is the active cluster node.  The lower portion of the center pane verifies that the Clustered Mailbox Server and its individual components (the System Attendant, Information Store, and mailbox database) are all online.

Figure C

The Failover Cluster Management Console displays the clustered mailbox server’s current status.

The Failover Cluster Management Console also provides you with an easy way of managing the clustered mailbox server. Right clicking on the clustered mailbox server reveals a shortcut menu that allows you to perform tasks such as taking the server offline, moving the server to another cluster node, or displaying critical events. You can what these options look like in Figure D.

Figure D

Right clicking on the Clustered Mailbox Server displays several different management options.

Cluster Events

Earlier I mentioned that the Failover Cluster Management Console makes it easy to quickly check on a cluster node’s status, but that there is a way to view more detailed information. You might have noticed in the previous screen captures that the console contains a Cluster Events container.

Contrary to what you might expect, clicking on this container does not cause the console to display all of the events related to the cluster. It does however allow you to query the event logs for cluster specific information.

Figure E displays the Cluster Events Filter dialog box. As you can see in the figure, you can filter your event log query by cluster node, event log, severity, or by time period. There is also an option that you can use to look for specific Event IDs.

Figure E

The Cluster Events Filter dialog box provides you with a way of querying the nodes for cluster specific events.

Managing the Cluster From the Command Line

As I mentioned earlier, failover clustering is a Windows feature that is completely separate from Exchange. Although both the clustered mailbox server and failover clustering can be managed from the command line, they must be managed in different ways.

Like any other Exchange Server related management tasks, command line based management of the clustered mailbox server is done through the Exchange Management Shell (EMS).

As I’m sure you already know, EMS commands are made up of verb-noun combinations. Most of the commands used in managing a clustered mailbox server use the ClusteredMailboxServer noun combined with various verbs. For example, it is possible to take a clustered mailbox server offline and bring it back online by using the Stop-ClusteredMailboxServer and the Start-ClusteredMailboxServer commands respectively. You will see some more concrete examples of these commands later on.

Although the cluster can be managed from the command line, Microsoft decided not to base cluster management on Windows PowerShell. Instead, command line based cluster management is based on the Cluster.exe command, which is typically  executed in a Command Prompt window.

I suspect that the reason why Microsoft chose to take this approach was to maintain backward compatibility with previous Windows operating systems. As you may recall, in the first article in this series, I showed you how to create a clustered mailbox server that was based on Windows Server 2003. In that article, I created a cluster by using the Cluster command and specifying the /Create and the /Wizard switches.

Although the Cluster command does exist in Windows Server 2008, it has been changed slightly since Windows Server 2003. For example, the /Wizard switch that I mentioned in the previous paragraph has been removed in Windows Server 2008. In Windows Server 2003, this switch told the Cluster command that you wanted to use a GUI based wizard to create the cluster. This wizard was replaced by the Failover Cluster Management Console in Windows Server 2008, so the /Wizard switch was removed. Most of the other command line switches still work though.

Server Maintenance

Although it is easy to think of failover clustering and CCR purely as a mechanism for improving server resilience, clustering is also useful for keeping the mailbox server online while performing maintenance on the cluster nodes.

One common maintenance task that you may occasionally have to perform is the application of an Exchange service pack to a clustered mailbox server. The task differs considerably from what is involved in applying an Exchange 2007 service pack to a non-clustered Exchange Server. The most important thing that you need to know about the process is that contrary to what you might expect, upgrading the Exchange Service pack on a clustered mailbox server does require a bit of down time. It is also important to note that the procedure that I am about to show you is only valid for a CCR environment. If you need to apply an Exchange service pack to a Single Copy Cluster (SCC) then you can find instructions for doing so at: http://technet.microsoft.com/en-us/library/bb691226(EXCHG.80).aspx

Before you attempt to apply a new service pack to a clustered mailbox server, I recommend that you perform a full, system state backup of your cluster nodes and of your schema master. The upgrade process modifies the Active Directory schema, so it is important to have a backup that you can revert to should something go wrong. Of course in larger organizations it is often the responsibility of the Active Directory team to perform any schema extensions.

When you finish backing up the necessary servers, determine which of the cluster nodes is currently the passive node.  In an effort to avoid any confusion, I will be referring to the node that is currently passive as Node 2, and I will refer to the node that is currently active as Node 1. Later on, the passive node will become the Active Node, but will retain its identity as Node 2.

For right now you can leave Exchange Server running, but you will need to stop any services that make use of the Performance Monitor counters on Node 2. Most of the time this requirement will only effect those who are running management tools such as Microsoft’s System Center Operations Manager (SCOM) Microsoft Operations Manager (MOM), Argent, or Tivoli. Once all of the necessary services have been shut down, you will need to restart the Remote Registry service.

The next step in the process is to install the Exchange service pack onto Node 2. As I mentioned earlier though, this is done using a different method from what you might be used to. Specifically, Exchange 2007 service packs can only be installed on clustered mailbox servers by using the command line. GUI based installations are not supported. To perform the installation, open a Command Prompt window, and navigate to the folder containing the service pack files. Now, enter the following command:

Setup will perform a quick prerequisite check, and will then install the service pack onto Node 2. When the installation process completes, reboot the server.

After the server finishes rebooting, there is still some work that needs to be done on Node 2, so go ahead and log in as an Administrator, and open the Exchange Management Shell.

At this point, you will have to shut down the clustered mailbox server. The easiest way of doing so is to use the Stop-ClusteredMailboxServer command in EMS. The syntax that you will use is fairly simple, but you will need to know the name that has been assigned to the clustered mailbox server. For example, if the clustered mailbox server is named EXCH1, then you would use a command similar to this one:

Although you have stopped the clustered mailbox server, you must now move it from Node 1 to Node 2. For this operation you will use the Move-ClusteredMailboxServer command. This command must be executed on Node 2. Following the previous example in which the clustered mailbox server was named EXCH1, the command would look something like this:

So far you have installed the service pack onto Node 2, shut down the clustered mailbox server, and then moved the clustered mailbox server to Node 2. Even though you have already installed the service pack onto Node 2, you will have to run the service pack’s setup file on Node 2 one more time so that the clustered mailbox server can be upgraded. To do so, open a Command Prompt window and then navigate to the folder containing your service pack files. Now, enter the following command:

When Setup completes, the Clustered Mailbox Server will have been upgraded and brought back online. Now, you must apply the service pack to Node 1. To do so, log onto Node 1 as an administrator. As was the case with Node 2, you will have to shut down any services that make use of the Performance Monitor counters, and you will have to restart the Remote Registry Service. Upon completing these tasks, open a Command Prompt window, and navigate to the folder containing your service pack installation files. You can now upgrade the cluster node by entering the following command:

When Setup completes, you must reboot Node 1. The upgrade process is technically complete at this point. You may optionally wish to move the clustered mailbox server back to Node 1 though. You can do so by using the following command:

It is also possible to move the clustered mailbox server by using the Failover Cluster Management Console or by using cluster.exe.

Backing Up the Clustered Mailbox Server

One last maintenance task that I want to briefly discuss is that of backing up the Clustered Mailbox Server. Microsoft usually recommends that organizations perform backups of the cluster’s passive node. That way, the backup process does not impact the active node’s performance.

While this technique probably sounds simple enough, it is important to remember that the passive node is not always bound to the same physical server. In a failure situation, the passive node becomes the active node. When the node that was previously active comes back online it becomes the passive node, unless you have configured the cluster to use automatic failback.

So how do you back up the passive node, when the passive node can change? The trick is to use a backup application that is failover clustering aware. To give you a better idea of what I am talking about, take a look at the screen capture shown in Figure F. This screen capture was taken from Microsoft’s System Center Data Protection Manager 2007.

Figure F

It is important to make sure that your backup application is cluster aware.

The dialog box in the figure above illustrates how Data Protection Manager 2007 (DPM 2007) allows you to select the resources that you want to protect. You will notice that both of my cluster nodes are listed individually (EXCHNODE1 and EXCHNODE2). However, there is a separate listing for the cluster group and for the clustered mailbox server (named E2K7CMS in the figure). This means that DPM 2007 only allows you to back up shares, volumes, and the system state at the individual node level. The cluster group and the clustered mailbox server are backed up separately.

Once you have chosen to back up the clustered mailbox server, DPM 2007 will display a dialog box which asks you how you want to back up the Exchange CCR Protection Node. As you can see in Figure G, you have the option of backing up the Active Node, the Passive Node, or a specific node.

Figure G

DPM 2007 allows you to choose which cluster node you want to back up.

Conclusion

As you can see, managing and maintaining a clustered mailbox server is a bit different from maintaining a non-clustered mailbox server.  Most of the management tasks related to the cluster can be performed through a GUI, but there will be times when you have to use EMS. Microsoft has published a TechNet article that is designed to introduce you to using PowerShell commands on a failover cluster. You can find the article at: http://technet.microsoft.com/en-us/library/ee619762(WS.10).aspx.