VITA's vitals vamaged - Virginia government computer outage

[FCC DISCLOSURE: My employer provides driver's licensing equipment and software, though not in this instance.]

I ran across this press release, as well as subsequent press releases that indicate that the situation has not changed.

FOR IMMEDIATE RELEASE
Thursday, August 26, 2010
Media Contact: Melanie Stokes
Department of Motor Vehicles
(804) 367-6623

Statewide Computer Issue Halts In-Person Driver's License Transactions
Vehicle Titles, Decals, Etc. Still Being Processed in DMV Offices

RICHMOND - Due to a statewide computer system problem, the Virginia Department of Motor Vehicles is currently unable to process driver's license transactions inside its 74 customer service centers in Virginia. Customers who can renew licenses through another service option are encouraged to go online at www.dmvNOW.com, or use the automated telephone service at 1-888-337-4782. The Virginia Information Technologies Agency (VITA) is working to resolve the server problem that is affecting 24 different state agencies. VITA has not reported a recovery time.

At DMV, all other types of transactions, including vehicle decals and titles, transcripts, etc. are being processed normally.
# # #

Richmond's WTVR cataloged the agencies affected:

The Virginia Information Technologies Agency (VITA) is working to restore a server problem that's affecting 24-different state agencies including: Virginia Department of Emergency Management, Virginia Department of Transportation and Virginia Employment Commission, to name a few.

As the systems started coming back up (although DMV is still anticipated to be down today), BusinessWeek identified the issue:

The failure occurred in one memory card in what is known as a "storage area network," or SAN, at VITA's large suburban Richmond computing center, one of several data storage systems in different parts of Virginia.

Even though the system had built-in redundancies, something else went wrong:

[W]hen the memory card failed Wednesday, a fallback that attempted to shoulder the load began reporting multiple errors.

VITA itself is providing updates on the repair work. Here's the Friday update:

Update on storage outage
11 a.m., Friday, Aug. 27, 2010

• Maintenance and repair work proceeded as expected overnight. Applications, servers and the damaged storage system were taken down as planned.
• The storage system has been repaired.
• More than 60 percent of the servers attached to that storage system have been brought back up and are operational. Agencies are testing those servers and related applications.
• Work continues to restore the remaining servers.
• The system is not yet 100 percent operational. We will be brining services up throughout the day. Some data must be restored.
• Unfortunately, DMV still cannot process driver’s licenses at its customer centers. Please check the DMV website for details on services available at the customer centers and online. Some other agencies continue to be impacted.