PST Horror Stories

Brien has been on the front line of Systems Administration for long enough to both respect and fear PST files. Though they are a handy tool when used wisely, they are too often misused, resulting in disaster. Brien relates a few tragedies to which he's had a front-row seat. Read on, and be warned.

I’ve been working with Exchange Server and Microsoft Outlook since the mid 90’s, so when I was asked to write about my experiences with PST files, I thought that it was going to be one of the easiest assignments that I’d had in a long time. As useful as PST files can be, I have encountered many, many situations over the years in which their use lead to disaster. As such, I decided to write about a few of those PST horror stories in the hope that it would help you to avoid suffering the same fate as some of the unlucky souls whose tales I will be retelling.

The Brand New Office

Out of all of the PST nightmares that I have ever had to deal with, this one sticks out in my mind the most (partly because it’s the one I was the most involved in, and partly because of the sheer scale of dataloss). I have a close friend who owns a technology consulting firm and, one day, he called me and told me that he had a big project, and that his staff wasn’t large enough to complete the project by the required deadline. Specifically, he was asking if I could lend a hand.

When he told me that the client was a local real estate firm which had outgrown their office, I assumed that all we’d be doing was wiring an office building with Ethernet cable. Although we did indeed have to pull lots of cable, the job was actually much bigger than that.

“The client had taken something
of a Wild West approach to their
computer systems; there was
absolutely no security in place,
and each employee was free to
configure their computers however
they wanted, and run any
applications that they wanted.”

Up until this office move, the client had taken something of a Wild West approach to their computer systems; there was absolutely no security in place, and each employee was free to configure their computers however they wanted, and run any applications that they wanted. In the name of improved manageability, one of the client’s primary goals was to now standardize the network, which meant installing a couple of domain controllers, a file server, and an Exchange Server. On top of that, we had to create user accounts and user mailboxes, and we had to upgrade each desktop to use the latest versions of Windows and Microsoft Office.

To add to the challenge, this was not only a big job, but it came with a tight deadline, too. The company was to spend all of Friday moving computers and furniture, and we were then scheduled to come in on Friday night to start wiring the building and setting up computers. The entire job had to be completed by 8:00 Monday morning.

Skip forward 3 days, and sometime around 6:00 AM on Monday morning we were finally finished after a grueling weekend. The office had been wired, the servers had been set up, everything was tested and we had dutifully cleaned up our mess when we were done. We had finished on time, but just barely. In fact, the only reason why we’d been able to meet the deadline was because we didn’t have to worry about any pre-existing data.

Prior to the move, the company simply didn’t have any servers, so there was nothing for us to upgrade or back up. Likewise, the owner of the company had told all of her employees to get anything of importance off of their computers, since we were going to be standardizing everything.

When the job was finally finished, my friend brought the woman who owned the company to the office so that he could show her all of the work that we had done, and explain how everything worked. Everything went well with the reveal, and the woman who owned the company seemed pleased with our work.

Exhausted, those of us who had just spent the weekend setting up the office decided to go get some breakfast. We had just gotten our food when my friend’s cell phone started ringing; it was the woman from the real estate firm, wondering where everyone’s email was. My friend explained to her that the mail server was brand new, so the mailboxes would start out empty until the employees started receiving mail. She went on to explain that everyone in the company had hundreds, if not thousands, of important messages that were nowhere to be found.

After breakfast we all went back to the real estate office to see if we could sort out the mystery of the missing email. By pure dumb luck, one of the employees happened to show up for work carrying a laptop which he had apparently kept with him all weekend, so it had not been upgraded. When the owner of the company told the laptop-hoarder that all of the company’s email was gone, he looked a bit panicked, but then came back to us a few minutes later and told us that he still had all of his messages.

What I discovered was that everyone had been using Outlook to connect to a POP3 mailbox, and that, unfortunately, all of the mail was being transferred to a local PST file and then removed from said mailbox.

Except for the guy with the laptop, everyone in the organization lost every bit of their email, although we could have easily salvaged the messages had we known that PST files were being used. In the end, the woman who owned the company was irate about the lost data, but there was nothing that she could do because she was the one who had told us that it was safe to reformat all of the workstation hard drives. The moral of the story? If you’re going to use PST files in any way, shape or form, make sure you have a regular (frequent) backup process in place. You may scoff at the apparent obviousness of this advice, but just having a backup process in place sometimes isn’t enough…

The Server Failure

A couple of years after The Big Bad Office Move, I ran into another PST nightmare that was almost as bad. This situation involved every bit as much data loss as the Real Estate firm suffered, but didn’t have the same personal impact for me because this time I wasn’t one of the people who had caused the data loss!

This story takes place at a medium-sized organization with a few hundred users. The network administrator for the organization had made every effort to run the network in a responsible manner, but had gotten some bad advice from a consultant. The consultant had told the network administrator that Exchange Server performs poorly if the mailbox database begins to accumulate messages, and that all of the messages should be offloaded to PST files at regular intervals.

Thankfully, the network administrator knew enough to realize that, if he kept the PST files on the user’s local hard drives, then he would have no way of backing them up and a user’s hard drive failure would take all of their mail with it. As such, the administrator had tried to be proactive about this by redirecting all of the PST files to a network share. Unfortunately, there were a couple of things that the administrator didn’t take into account.

For starters, he had created a major security hole, because all of the PST files resided in the same shared folder, and any user would have been able to open any other user’s PST file if they had known how (thank heavens for small mercies). More importantly though, Microsoft has always cautioned organizations not to host PST files on network shares, because if network problems occur, then the PST files can become corrupted.

“Having a regular backup
process is a good start,
but you need to check
and make sure it’s working
as intended. Preferably not
by suddenly relying on it…”

As you have probably already figured out, the network administrator called me for help because the storage array on his file server had failed – the same storage array that contained everyone’s PST files.

Painful as this was, the administrator was confident that all of the messaging data could be restored, because he backed the server up on a nightly basis. However, what he hadn’t counted on was that almost everyone at the company was in the habit of staying logged in 24/7 and, as such, the backup software always skipped the PST files because they were open. Unfortunately, there was nothing that I could do to help salvage the lost data because no usable backups existed. Having a regular backup process is a good start, but you need to check and make sure it’s working as intended. Preferably not by suddenly relying on it…

The CEO

The last story that I want to share involves the CEO of a company with about a thousand employees. The CEO had asked the IT department to configure his laptop so that his messages were moved out of his mailbox and into a local PST file, so that he could access his mailbox data even when he was not connected to the network (this was before Exchange Cached Mode existed).

At the time, PST files had a maximum file size of 2 GB, and one day (you guessed it), the CEO’s PST file exceeded the 2 GB threshold, and became corrupted.

Even though the CEO had made a backup the night before, he refused to use it because he had since accumulated a number of messages which he did not want to lose. The only solution was to use the Inbox Repair Tool (ScanPST) to fix the corrupt PST file. Unfortunately for the CEO, this tool works by cropping the PST file to bring it back down to an acceptable size.

To make a long story short, the CEO was furious when he wasn’t able to recover the messages from that morning. Furthermore, he had a lot of trouble accepting the idea that he was going to have to either move his data back to the Exchange Server, or create another PST file and move some of his data to it in order to prevent the problem from happening again. Clearly, even if you have a working backup, sometimes there’s just nothing you can do.

Conclusion

My ultimate goal in retelling these tales of woe should be fairly clear – even though PST files definitely have their place, administrators should avoid using them if at all possible. They often contain important, time-sensitive or hard-to-replace information, and while backing them up is obviously a good idea, we’ve seen that it doesn’t take much for one of these backups to be rendered useless. After recalling (on my part) and reading (on yours) these PST horror stories, it is easy to see why even Microsoft generally agrees that PSTs are difficult to manage and use well. While they don’t necessary recommend that organizations running Exchange Server avoid using PST files, many Exchange Admins might!

882-PSTImporterLogo.jpg
Prevent your own PST Horror Story
Find every PST file on your network and optionally transfer them straight to the appropriate mailbox on Exchange 2007 or 2010. No PowerShell scripts and cmdlets required, PST Importer 2010 does it automatically for you! Try it for free for 14 days and find where all your PSTs are hiding.