Building an Exchange Server 2007 environment

Of course, changing a 32,000 mailbox system, based in 40 Exchange Servers, to a centralised 25,000 mailbox solution is not a difficult task as long as you follow the Microsoft guidelines; It just takes patience, and time to understand Exchange's storage design principles. Jaap describes the design process and shares his experiences with the chosen server hardware.

I have been working on a project recently where we had to migrate 25.000 mailboxes from a decentralized Exchange Server 2003 with 40 servers to a centralized Exchange Server 2007 environment. Designing and building an environment like this is fun and in this article I will explain the approach we took for designing this environment. I will not cover the actual migration and decommissioning the Exchange Server 2003 platform, just the design phase, especially for the mailbox servers.

Exchange Server 2003

The original platform that the customer was using was built on Windows Server 2003 and Exchange Server 2003 and was in use for only 3 years. There were 40 Exchange Server 2003 mailbox servers spread across the country hosting around 38.000 mailboxes of which 12.000 inactives. There was one access point to and from the Internet and at this location two Exchange Server 2003 front-end servers were located. These servers were providing POP3 (just a few users), Outlook Web Access and ActiveSync for Windows Mobile users.

Before designing anything for Exchange Server 2007 the existing environment needs to be investigated. The following figures were found, Type I, II and III are the type of users (or usage scenario):


Type I

Type II

Type III

msg sent/day/mbx




msg rec/day/mbx




Total msg per day




msg /mbx per day (MB)




msg /mbx per week (MB)




average msg size (KB)




Total MBX




Table 1. Usage scenario on Exchange Server 2003

All users were using Outlook 2003. Desktop were running in online mode due to the roaming profile, laptops were running in cached mode. Outlook Web Access is used by only a small number of users and so is POP3. There were approximately 3,000 Blackberry device and approximately 1,000 Windows Mobile devices.

Exchange 2003 Storage Usage

Besides the usage profile of the current messaging environment it is also important to know how the current mailboxes are sized. And how many mailboxes are actually active. Using the Exchange Server 2003 current sizing, an appropriate storage design for Exchange Server 2007 can be made.

At the beginning of the project approximately 27,000 mailboxes could be separated into three categories:

Mailbox size

Number of users

Less then 500 MB


Between 500 MB and 2 GB


Between 2 GB and 8 GB


Table 2. Mailbox limits and the number of mailboxes

The customer wanted to maintain the mailbox limits in the Exchange Server 2007 environment.

As it turned out the mailbox types did follow the mailbox profiles we found earlier, in general a large mailbox had a higher IO profile compared to a mailbox which was of a smaller type.

Exchange Server 2007

One of the most important design aspects in Exchange Server 2007 is a proper storage solution. Without a proper storage solution there’s a serious risk to end-up with an Exchange environment that does not perform well.

To help customers design a proper storage solution Microsoft has created a tool to assist, the Microsoft Mailbox Storage Requirement calculator which can be found here: ‘Exchange 2007 Mailbox Server Role Storage Requirements Calculator’

When you open the storage calculator you can enter variables like the number of users, the concurrency, the usage profile etc. and the storage calculator will calculate the amount of storage, the number of I/O (Input/output) operations per second etc. Please note that the storage calculator is updated regularly and that the output of the current version can differ from for example last years version. The last version per June 2009 is V.16.9.

Using the storage calculator and the mailbox profiles and sizes the following numbers were found for sizing the Exchange Server 2007 mailbox servers:

Mailbox limit

< 500 MB

< 2 GB

< 8 GB

Number of mailboxes




Number of mailboxes per server




Number of servers needed




Max mailbox database size

200 GB

200 GB

200 GB

Number of mailboxes per database




Number of databases per server




Total Database IOPS per server




Total log file IOPS per server




Table 3. Exchange Server 2007 mailbox server sizing

Note: The actual design was made in the summer of 2007 with an older version of the storage calculator, based on Exchange Server 2007 RTM. With SP1 and with a newer version of the storage calculator different values will be found.

So now we know the total number of Exchange Mailbox servers: 9. The number of Hub Transport Servers and Client Access Servers can be derived from this number, or better, from the number of processors in the servers. For this customer no hard numbers were used though. The customer wanted to deploy Exchange Server 2007 into two separate rooms (in one building), so in each room 2 Hub Transport Servers and 3 Client Access Servers were used.

A Unix based anti-virus and anti-spam solution was already available so an Edge Transport Server is not used.

Hardware used for Exchange Server 2007

The customer was using a Standard Building Block (SBB) for their entire IT infrastructure. This SBB is based on a Dell PowerEdge 2950 with a dual Quad Core Intel Xenon processor, 2 local disks in a RAID-1 configuration, 16 GB internal memory and two Intel dual port network adapters. These servers are used for all Exchange Server 2007 servers. All servers are running Windows Server 2003 X64, Standard Edition or Enterprise Edition.

The storage being used for the Mailbox Servers is an iSCSI solution from Equallogic. The Equallogic PS-series should scale well and are very flexible, even in an enterprise environment like this. Well, so did the sales men tell us 😉

Before this deployment I’ve seen Equallogic PS-series storage in several scenarios, but never beyond three or four units. This is quite a different environment where one would expect a fiber channel storage solution from EMC or HP, but not an Equallogic iSCSI solution.

It took us three months of calculating, testing (JetStress), debugging, implementing a new firmware (which had to be written by Equallogic), calculating again, testing again and implementing before we got the Equallogic environment up-and-running the right way, sufficient for servicing the environment I outlined earlier.

Because EqualLogic virtualizes storage and handles volumes differently that mainstream storage the exchange storage calculator has to be used differently. The important values are the IOPS which are needed and because the storage calculator still uses disk spindels to calculate requirements the values differ. Diskspindels do not matter (or at least less) in an EqualLogic design as all data is spread over 14 disks per member in the pool anyway. Also the controller logic within the Equallogic and the cache memory used in an Equallogic mean you have to interpret the numbers differently.

This is how we designed the storage solution:

  • Two Equallogic portals were designed. A portal is the target where the iSCSI initiators connect to;

  • Each portal has three so called pools. A pool is a logical separation of units (that use the same RAIDSet) where data can be written;

  • Two pools are configured each with three EqualLogic PS-5000X arrays; each array is equipped with 300 GB SAS disks (25TB usable storage). These will be used for storing the mailbox databases;

  • One pool is configured with one EqualLogic PS-5000E array; each array is equipped with 1TB SATA disks (12TB usable storage). This will be used for storing the log files from the various Storage Groups.

Because EqualLogic virtualizes storage a volume is spread over multiple arrays in a pool to a maximum of three possibly utilizing 42 disks giving maximum performance and IOPS.


Figure 1. Schematical overview of the mailbox servers and the storage solution

Note. In total 14 Equallogic units were used, but please not that since a CCR cluster is used each side of the cluster has to be equipped with identical storage. The passive side of the CCR cluster is equipped with 14 Equallogic units as well, so a total of 28 iSCSI units were used!

All units are configured in a RAID-50 scenario, this will give the best performance on these particular devices. In total a net capacity of approximately 50 TB of storage is created this way and this should be sufficient to host the 25,000 mailboxes and provide enough for future growth.

Mailbox Server configuration

So how is the mailbox server configured you might ask. All fourteen arrays are connected to a Cisco switch. The controllers are redundant; each controller has 3 active network interfaces and 3 passive interfaces. All servers are configured with two (Intel) GBit network adapters, each connected to the Cisco switch as well.

One problem we initially ran into was the amount of ISCSI connections we needed. Using two network cards in the server and three network cards in the Equallogic units resulted in six ISCSI connections per volume on the ISCSI portal. ISCSI connections are setup according the following rule; one connection per network interface on the server per array on which the data resides. When using 140 databases this means that 280 volumes are needed which resulted in 1680 ISCSI connections. The initial firmware (version 3.x) was not capable of handling this amount of connections (only 512 per portal), so we had to wait for the version 4.x firmware which was capable of handling this amount of connections (512 per pool). But still we had to use a separate pool for LOG volumes since one array only has 2 ISCSI connections per volume. This resulted in a total number of 560 ISCSI connections per portal instead of 860 ISCSI connections leaving enough space to grow.

A classic Windows limitation when using a large number of LUNs on a server is the drive letter limitation. There are only 26 characters in the alphabet, and 5 are already in use on the server, so only 21 are left. When configuring 20 Storage Groups 40 LUNs are used, so Volume Mount Points are configured on each mailbox server, resulting in only two drive letters being used. One for the databases and one for the LOG volumes.

When configuring an environment like this you really have to know how to script. Scripts were used to configure all the LUNs on the Equallogic environment, scripts were used to create all the partitions (be aware of the disk alignment!) and format the drivers. Best practice is a 64Kb disk offset which is not default in windows.

Finally a PowerShell (Exchange Management Shell) script was used to create and mount all the databases on the Exchange Server 2007 mailbox servers.

For backing up the Exchange environment a Microsoft System Center Data Protection Manager (DPM) 2007 solution was built. DPM is using a disk-to-disk-to-tape backup solution. A full backup is created every day and an incremental backup is created every two hours. Every week an entire full backup is written to tape for long term, off-site storage.


The performance of the Equallogic PS-series really surprised me, they were performing better than I initially expected. During testing we manged to get 35.000 IOPS from the storage which was a lot more than we needed. After migration 25,000 mailboxes to the new platform all nine mailbox servers are still running fine with a pretty low number of disk I/O’s and a fairly low processor utilization.


Are there challenges? Oh yes, think about the Online Maintenance. When using such an amount of databases and this sizing OLM really has a hard time to finish in time, so only two databases are configured each night to run the OLM. Each database is running OLM once every week; otherwise it cannot finish its run.

Reseeding a database is nothing else than a file copy over the network from the active node in the CCR cluster to the passive node. Copying a 150 GB or 200 GB mailbox database can take some time; imagine you have to reseed five databases.

You also need a good performing backup solution to have the backup finish within the available timeframes. Also monitoring becomes more important when dealing with these amounts of databases.


Designing a 25,000 mailbox solution is not a difficult task as long as you follow the Microsoft guidelines; use the Exchange Storage Calculator for designing a proper storage solution and understand exchange storage design principles. Failing to do so will always result in a poor performing Exchange server environment.

Using a storage solution from Dell/Equallogic is also possible, but special care has to be taken to design and implement the members the correct way, using the right number of portals, pools and members. If you do so, it will perform well. Please note that in this scenario the Equallogic solution was only used for storing the Exchange Server mailbox databases and logs. No other data, such as like SQL databases, is stored on the EqualLogic arrays.

Disclaimer: This article is a true scenario, but the entire project from the Exchange Server 2003 inventory up to decommissioning the Exchange Server 2003 servers took more than a year with three people, working full time on all aspects. The documentation that was created during the project covers more than 300 pages, so it is not possible to cover everything in great detail. My main objective was to explain a bit more about using the Equallogic hardware in combination with a 25,000 mailbox environment.