I have been working on a project recently where we had to migrate 25.000 mailboxes from a decentralized Exchange Server 2003 with 40 servers to a centralized Exchange Server 2007 environment. Designing and building an environment like this is fun and in this article I will explain the approach we took for designing this environment. I will not cover the actual migration and decommissioning the Exchange Server 2003 platform, just the design phase, especially for the mailbox servers.

Exchange Server 2003

The original platform that the customer was using was built on Windows Server 2003 and Exchange Server 2003 and was in use for only 3 years. There were 40 Exchange Server 2003 mailbox servers spread across the country hosting around 38.000 mailboxes of which 12.000 inactives. There was one access point to and from the Internet and at this location two Exchange Server 2003 front-end servers were located. These servers were providing POP3 (just a few users), Outlook Web Access and ActiveSync for Windows Mobile users.

Before designing anything for Exchange Server 2007 the existing environment needs to be investigated. The following figures were found, Type I, II and III are the type of users (or usage scenario):

	Type I	Type II	Type III
msg sent/day/mbx	10	20	30
msg rec/day/mbx	31	67	100
Total msg per day	41	87	130
msg /mbx per day (MB)	6	17	44
msg /mbx per week (MB)	42	119	308
average msg size (KB)	150	200	350
Total MBX	23384	2632	863

Table 1. Usage scenario on Exchange Server 2003

All users were using Outlook 2003. Desktop were running in online mode due to the roaming profile, laptops were running in cached mode. Outlook Web Access is used by only a small number of users and so is POP3. There were approximately 3,000 Blackberry device and approximately 1,000 Windows Mobile devices.

Exchange 2003 Storage Usage

Besides the usage profile of the current messaging environment it is also important to know how the current mailboxes are sized. And how many mailboxes are actually active. Using the Exchange Server 2003 current sizing, an appropriate storage design for Exchange Server 2007 can be made.

At the beginning of the project approximately 27,000 mailboxes could be separated into three categories:

Mailbox size	Number of users
Less then 500 MB	~23,000
Between 500 MB and 2 GB	~3,000
Between 2 GB and 8 GB	~1,000

Table 2. Mailbox limits and the number of mailboxes

The customer wanted to maintain the mailbox limits in the Exchange Server 2007 environment.

As it turned out the mailbox types did follow the mailbox profiles we found earlier, in general a large mailbox had a higher IO profile compared to a mailbox which was of a smaller type.

Exchange Server 2007

One of the most important design aspects in Exchange Server 2007 is a proper storage solution. Without a proper storage solution there’s a serious risk to end-up with an Exchange environment that does not perform well.

To help customers design a proper storage solution Microsoft has created a tool to assist, the Microsoft Mailbox Storage Requirement calculator which can be found here: ‘Exchange 2007 Mailbox Server Role Storage Requirements Calculator’

When you open the storage calculator you can enter variables like the number of users, the concurrency, the usage profile etc. and the storage calculator will calculate the amount of storage, the number of I/O (Input/output) operations per second etc. Please note that the storage calculator is updated regularly and that the output of the current version can differ from for example last years version. The last version per June 2009 is V.16.9.

Using the storage calculator and the mailbox profiles and sizes the following numbers were found for sizing the Exchange Server 2007 mailbox servers:

Mailbox limit	< 500 MB	< 2 GB	< 8 GB
Number of mailboxes	23,000	3,000	1,000
Number of mailboxes per server	4,000	2,000	800
Number of servers needed	5	2	2
Max mailbox database size	200 GB	200 GB	200 GB
Number of mailboxes per database	400	100	25
Number of databases per server	11	15	20
Total Database IOPS per server	2,100	3,000	2,000
Total log file IOPS per server	580	820	550

Table 3. Exchange Server 2007 mailbox server sizing

So now we know the total number of Exchange Mailbox servers: 9. The number of Hub Transport Servers and Client Access Servers can be derived from this number, or better, from the number of processors in the servers. For this customer no hard numbers were used though. The customer wanted to deploy Exchange Server 2007 into two separate rooms (in one building), so in each room 2 Hub Transport Servers and 3 Client Access Servers were used.

A Unix based anti-virus and anti-spam solution was already available so an Edge Transport Server is not used.

Hardware used for Exchange Server 2007

The customer was using a Standard Building Block (SBB) for their entire IT infrastructure. This SBB is based on a Dell PowerEdge 2950 with a dual Quad Core Intel Xenon processor, 2 local disks in a RAID-1 configuration, 16 GB internal memory and two Intel dual port network adapters. These servers are used for all Exchange Server 2007 servers. All servers are running Windows Server 2003 X64, Standard Edition or Enterprise Edition.

The storage being used for the Mailbox Servers is an iSCSI solution from Equallogic. The Equallogic PS-series should scale well and are very flexible, even in an enterprise environment like this. Well, so did the sales men tell us 😉

Before this deployment I’ve seen Equallogic PS-series storage in several scenarios, but never beyond three or four units. This is quite a different environment where one would expect a fiber channel storage solution from EMC or HP, but not an Equallogic iSCSI solution.

It took us three months of calculating, testing (JetStress), debugging, implementing a new firmware (which had to be written by Equallogic), calculating again, testing again and implementing before we got the Equallogic environment up-and-running the right way, sufficient for servicing the environment I outlined earlier.

Because EqualLogic virtualizes storage and handles volumes differently that mainstream storage the exchange storage calculator has to be used differently. The important values are the IOPS which are needed and because the storage calculator still uses disk spindels to calculate requirements the values differ. Diskspindels do not matter (or at least less) in an EqualLogic design as all data is spread over 14 disks per member in the pool anyway. Also the controller logic within the Equallogic and the cache memory used in an Equallogic mean you have to interpret the numbers differently.

This is how we designed the storage solution:

• Two Equallogic portals were designed. A portal is the target where the iSCSI initiators connect to;

• Each portal has three so called pools. A pool is a logical separation of units (that use the same RAIDSet) where data can be written;

• Two pools are configured each with three EqualLogic PS-5000X arrays; each array is equipped with 300 GB SAS disks (25TB usable storage). These will be used for storing the mailbox databases;

• One pool is configured with one EqualLogic PS-5000E array; each array is equipped with 1TB SATA disks (12TB usable storage). This will be used for storing the log files from the various Storage Groups.

Because EqualLogic virtualizes storage a volume is spread over multiple arrays in a pool to a maximum of three possibly utilizing 42 disks giving maximum performance and IOPS.

Figure 1. Schematical overview of the mailbox servers and the storage solution

All units are configured in a RAID-50 scenario, this will give the best performance on these particular devices. In total a net capacity of approximately 50 TB of storage is created this way and this should be sufficient to host the 25,000 mailboxes and provide enough for future growth.

Mailbox Server configuration

Performance

The performance of the Equallogic PS-series really surprised me, they were performing better than I initially expected. During testing we manged to get 35.000 IOPS from the storage which was a lot more than we needed. After migration 25,000 mailboxes to the new platform all nine mailbox servers are still running fine with a pretty low number of disk I/O’s and a fairly low processor utilization.

Are there challenges? Oh yes, think about the Online Maintenance. When using such an amount of databases and this sizing OLM really has a hard time to finish in time, so only two databases are configured each night to run the OLM. Each database is running OLM once every week; otherwise it cannot finish its run.

Reseeding a database is nothing else than a file copy over the network from the active node in the CCR cluster to the passive node. Copying a 150 GB or 200 GB mailbox database can take some time; imagine you have to reseed five databases.

You also need a good performing backup solution to have the backup finish within the available timeframes. Also monitoring becomes more important when dealing with these amounts of databases.

Conclusion

Designing a 25,000 mailbox solution is not a difficult task as long as you follow the Microsoft guidelines; use the Exchange Storage Calculator for designing a proper storage solution and understand exchange storage design principles. Failing to do so will always result in a poor performing Exchange server environment.

Using a storage solution from Dell/Equallogic is also possible, but special care has to be taken to design and implement the members the correct way, using the right number of portals, pools and members. If you do so, it will perform well. Please note that in this scenario the Equallogic solution was only used for storing the Exchange Server mailbox databases and logs. No other data, such as like SQL databases, is stored on the EqualLogic arrays.