Wednesday, April 13, 2011

What if the data centers were to go away? In 1973, one did.

All of us silly Facebook users, playing with our farms (and yes, I've been revisiting Farm Town lately). Stupid stuff.

But when you have over 500 million people doing stupid stuff, you need to be really smart to manage it all. Stacey Higginbotham laid out the numbers on GigaOM:

Supporting 100 million photo uploads each day and as many as 18,000 comments ... per second requires the social network to perform 24 billion calculations a second at peak times to display everyone’s news feeds. That’s far less than a supercomputer but still enough to require some serious infrastructure.

Now I have been known to go around and argue that automated fingerprint identification systems are different, primarily because of the sizes of the images involved (a palmprint at 1000 pixels per inch is pretty substantial). But none of the AFIS with which I was ever associated in my product management days came near to approaching 100 million uploads per day or 18,000 comments per second.

This is why Facebook, and Google, and other companies managing a lot of data have been putting a lot of time into creating massive data centers. Think of what would happen if all this data were lost. (Well, I guess the world would survive the loss of my blogs.)

To illustrate what could happen, let's look at what could happen if a data center were to go away. Actually, this happened, long before Google or even Apple came into being. This story is from 1973:

On July 12, 1973, a disastrous fire at NPRC (MPR) destroyed approximately 16-18 million Official Military Personnel Files....

No duplicate copies of the records that were destroyed in the fire were maintained, nor was a microfilm copy ever produced. There were no indexes created prior to the fire. In addition, millions of documents had been lent to the Department of Veterans Affairs before the fire occurred. Therefore, a complete listing of the records that were lost is not available.

Now I have a relative who is a World War II veteran, and he needed to prove his eligibility for veteran benefits. However, his records were among those that were destroyed. This story has a happy ending, sort of - he proved his eligibility by providing newspaper clippings from World War II, in which a local paper described his status as a prisoner of war in Germany.

I know that everyone has been yammering about Epsilon lately, but data loss is a more critical issue than data theft. Now I don't use Facebook for business, but that is in some cases my sole way to contact certain people, including relatives. If Facebook's data were to disappear, how would I do that?
blog comments powered by Disqus