(empo-tymshft) (empo-plaaybizz) Again, gaming drives industry (a petabyte of Avatar data)

One of the people at my church is a Microsoft employee, and when he isn't rocking out on his Zune, he's sharing news related to his employer. But before I share the item that he shared, let's look at storage.

I've never seen "Remember The Titans" (I'm not a movie person), but I did attend summer school at T.C. Williams High School in Alexandria, Virginia a few years after the events depicted in the movie. And when we weren't listening to the Rolling Stones' "Some Girls," we were messing around with the class' Star Trek game, written in BASIC. The cassette containing the game could be run on two computers - one with 8 kilobytes of memory, and the other with a whopping 16 kilobytes of memory. I remember that I once edited the game on the 16K computer, and then discovered that it wouldn't run on the 8K computer - if I recall correctly, I was able to salvage the earlier version of the game. I hope.

Time progressed, and our demands for memory space, either to run programs or to store data, increased. When I got my Mac Plus computer a few years later, I had a hard drive with 20 megabytes of disk space. Yup, we had moved up a prefix, from "kilo" to "mega."

Fast forward to 2009, the last year that I attended Oracle OpenWorld. One of my co-workers was part of a panel about usage of large amounts of data in Oracle applications. Our fairly impressive usage, however, was dwarfed by one of the panelists that was managing telescope data. The telescope would regularly generate hundreds of terabytes of data.

Fast forward to the article that my friend shared. Remember that the Oracle OpenWorld example involved serious stuff with telescopes and scientists and things. But my friend's data storage example involves game stuff - namely, a movie.

Five years ago, Nadine Kano visited James Cameron at the offices of his film production company, Lightstorm Entertainment. Kano, a senior director in MSIT's Product Group Strategic Initiatives (PGSI) team, had flown to Los Angeles to explore whether Microsoft could help on the director's latest project.

Cameron then showed Kano a 3-D clip - one of his initial concepts for the movie "Avatar."

Kano was impressed, and everyone else was impressed. As a result, Microsoft provided technical assistance to Cameron.

If the record audiences flocking to "Avatar" sit through the credits, they'll see proof of Microsoft's involvement when the "Special Thanks" message rolls across the screen.

The acknowledgement is for what Cameron and Lightstorm dubbed Gaia, the Digital Asset Management (DAM) system that served as the hub for all the digital content and metadata generated during the production. Essentially, the entire movie was turned into data that was almost exclusively housed on Gaia. Microsoft built the data-collection application from the ground up for "Avatar,"

OK, so how much metadata was generated?

In "Avatar," every blade of grass, every cloud in the sky, every vine in the jungle existed digitally and had to be stored somewhere. According to [producer Jon] Landau, the production generated more than a petabyte–or one million gigabytes–of information.

However, Microsoft isn't the only one moving a bunch of data around. Compare the "Avatar" statistic to the September 2010 gaming statistics from Zynga (from TechCrunch):

Zynga’s properties move a whopping 1 petabyte of data daily, and the company operates its own data centers; using a hybrid private/public cloud infrastructure.

That's not metadata, that's petadata.

And of course this is going to go on beyond petabytes. I've already talked about zetta (10 to the 21st power) and yotta (10 to the 24th power), as well as proposals for even larger numbers (including 10 to the 42nd power, or sorta). What will things be like when he can hold a device in our hand that can manage 3 sortabytes of data?