Open Source

In the final session of OSCON 2004, the lead IT guy for the Lord Rings movies presents the technical issues that they have run in to in the production of the LOTR movies.

Over the course of the project, they started with 80 people, 200 for the first film, 400 for the second and 500 artists for the third movie. They also went from 400 processors in the render farm to 3300 processors for the third movie. They also had to open a second office with a long hall 10 gigabit to the other office, which is the first of its kind in the southern hemisphere. Now that the movies are over they now have a chance to go back and look back on what happened.

The did hit problems with all of the data for one shot was on one filesystem and when 1400 procs want to talk with it, you have a problem. They designed a lightweight filesystem that sits on top of the filesystem to distribute the load. Data management was also a massive issue. The production team used FileMaker for everything, which had to be sucked out and plugged into various relational databases. The cheap hardware is great for raw computing power, the problem is you have two big problems: 1) licenses costs per proc and 2) cooling.

Because of not being able to kill horses, the did motion capture of the horses running and had to simulate them being hit by a giant flying nazgul. Also, Shelob was completely CG, they moved the eyes to the top of the head. They showed the Sam being attacked by shelob and it was a foam head with fangs being forced onto him by a guy completely covered in blue.

The final scene in in Mt. Doom was really difficult because of the lava. They just ran the scene a bunch of different ways and picked the best spots. The shot of nazgul flying towards white city that was showing in the trail took nine month from start to finish.

They are clearly pissed at RedHat for changing the licensing, and they having been on Redhat 7.3 for a variety of reasons. The cost of qualifying high end software against Linux as gone up a lot and now with the cost of Enterprise Server being so damn much. They might be looking at other platforms and it won’t begin with “W”, which I’m assuming will mean MacOSX or Solaris on 64bit x86.

The final extended version of LOTR: RotK will be coming out and it will have almost as many additional CG shots as were in the entire theatrical release RotK!

As a biology major and a dba I made a point of attending this session on Structuring Structural Biology with PostgreSQLs. It was very interesting to hear that a researcher in the discovery phase of interesting genes was able to download Postgres, extract it, install it, put it to use and have better uptime then the production Oracle systems. Having going through a lot of Oracle training I will vouch for the complexity of Oracle databases. However Oracle is putting a lot of work into auto-configuring which I'm not sure if this will help or not.

The slides for this presentation are not online yet but I can summarize the presentation as taking the gene sequencing data, cut it up into manageable segments, and get a md5 hash against it to see if it matches existing sequences. One particularly cool data management method was being able to figure out most of the members of a family of a particular type of protein in minutes by running a SQL script. A similar project to come up with similar results took two months. When I find out the slides for this and other presentions I'll update this entry.

While I still like working on OSX, I have been moving almost entirely using Linux on the desktop (actually KDE and GNOME, two of the main GUIs for Linux). I have many reasons why i have been making the move, I can run GNOME on Solaris which is what I use at work, Linux can run on much, much wider range of hardware then OSX and thus allow me to have the same GUI (and the same troubleshooting steps) on almost any hardware.

However I have been becoming increasingly annoyed at Apple over their way or the highway mentality. I think Apple makes great hardware (except the damn one button mouse) and a great OS but the fact that Apple won't allow anyone to do anything interesting outside of their control (i.e. hack) is really starting to piss me off. Hacking is about the only way that new stuff is making it to the general public. Apple, please knock it off and let people and companies do what they are going to do and as long as it is label as being modified then what the hell is the problem?

Once again the Stonehenge sponsored OSCON party/open bar at ChampionsBar 71 was a major success. Major success in the sheer volume of alcohol consumed, and a major success in the number of people that are not going to make it to the 8:45 session tomorrow morning. I ended up running into two friends that I haven’t seen in a several years who were friends of people attending OSCON. I'm sitting in the lobby of the hotel waiting a few minutes for the booze to release their grip and I can drive home.

Thanks to a number of issues like Sarbanes-Oxley the quality of data is very important and now can result in put you in jail if you get it very wrong with financial data. Many of the problems associated with bad data are because the people designing the database don't know what they are doing and management won't send them to training. There are many examples of really bad encoding schemes that are shockingly short-sighted and examples of many organizations doing tests against live data.

Here are a few suggestions: