Compressor and Qmaster Cluster

In preparation to create the Harry Putter 2 DVD, I borrowed (with permission) a number of Macbooks from work and set up my own little render farm at home. I installed Apple’s Qmaster package on each machine, which would allow my laptop to use them as a distributed computing network. In principal, it’s supposed to be really easy. In practicality, it sort of was, but not really.

The Qadministrator application is incredibly picky. If you do things right, it works great, but if just one variable isn’t to its liking, it beach-balls and won’t respond. Worse, it destroys the cluster you were trying to save and you have to start all over. After doing this several times, it goes pretty quickly, but still, it’s frustrating.

Anyway, here are some tips and tricks I learned while battling with Qmaster and Compressor (version 3, so yes I know this is already out of date, but may be useful for anyone who hasn’t upgraded to Apple’s latest and greatest)

Software Update

Before you think about doing anything else, before you even open up Qadministrator to create a cluster, update all your machines to the latest OS (10.5.7 as of this writing). Also, after installing the Qmaster package on each render node machine, run Software Update again and install any Pro Application Support or Quicktime updates. Without these installed on every machine Qadministrator will not be able to set up your cluster.

To set up you Qmaster cluster, follow this excellent tutorial: http://www.macworld.com/article/49047/2006/01/marcchreateside.html

Also, note that if you’re setting up machines that have more than one processor (any dual core MacBook or MacBook Pro, for example), you can highlight the Compressor service in the Apple Qmaster System Preferences pane, click “Options for selected service”, and choose as many instances as you have processors (so, dual cores = 2 instances). I’ve found it convenient on my master node (my personal laptop) to leave this at 1 less than the maximum so that compressord leaves me a little bit of processor power to check email, etc, but it doesn’t matter too much. The important lesson is to take advantage of those dual cores where you can.

Lastly, whenever you make changes to your cluster in Qadminister, do it slowly. And by that I mean one change at a time. Then quit. Then open Qadministrator up again and make another change, and so on. The alternative is that you might get an error message saying it’s unable to save the changes to the cluster, and next time you open Qadministrator your cluster has disappeared. Now you must go to every single stinkin’ machine, open the Apple Qmaster System Preferences pane, stop sharing, hold down option and click “Reset Services”, then start sharing again. On every machine. And a reboot doesn’t hurt then, either.

Audio Sync Problems

When rendering some of my videos I ran into audio sync problems. Specifically, I found them when compressing MPEG-2 and Dolby 2.0 streams for DVD, using a video that was either 23.98 or 24 fps. Oh, right, by the way, in Compressor 3, in order to use Qmaster to render a video, you need to export a Self-Contained-Quicktime-Movie file (SCQTM as I’ve seen it commonly abbreviated) first, then send that file through Compressor. I understand Apple fixed this in Compressor 3.5 so that you can use QT reference movies, but in v.3 it needs to be a SCQTM.

The issue I saw was that, even for short, two minute clips, the audio would be off sync by almost 1 second toward the end. But only when I compressed through the cluster. If I encoded the job on my local machine, no issue, looked beautiful (and obviously, I checked, the SCQTM file played correctly). Sidenote, I used a program called MPEG Streamclip to preview the renders without having to bring them into DVD Studio Pro, or, worse, actually burn a real disc.

After literally hours scouring Google, I finally broke down and called Apple’s “expert support” service (I’m not saying that sarcastically, that’s what it’s actually called). The guy I spoke to (Jon), recommended I disconnect some of the machines from the cluster and see if that made a difference. Apparently “network latency” can cause audio sync issues.

I was skeptical, and I still can’t wrap my mind around why that would make any difference, but nevertheless I tried it, screwed up Qadministrator, reset services and rebooted, and then sent the batch job through again, this time with only 1 extra computer in the cluster. Everything came out correctly. Hmm. I added computers one at a time until I realized that it was the computers networked on my second switch that would cause problems. Well, not the computers themselves, but the fact that there was a second switch between those machines and the master node. Thus I’ve determined you can probably have as many computers as you want in a cluster, as long as they’re physically all on the same switch. At least, for certain types of encoding - oddly, one timeline that was 59.97fps encoded perfectly fine with the entire cluster, it’s just the 23.98fps sequences that didn’t.

Incomplete Rendered Movies

Another issue I had was that some movies came through incomplete. The last minute or two of footage just hadn’t been encoded. The SCQTM was all there, but the MPEG-2 stopped about 95% of the way through.

I’ve finally determined this was my own ingenuity working against me. Early on in the process, I saved my Compressor batch file with all my sources, destinations, and settings for my entire DVD contents. This way I made sure I wouldn’t forget anything when I needed to render everything after I finished editing. Unfortunately, Compressor apparently keeps track of movie length, so after I added some clips to my FCP timeline and exported a new SCQTM, Compressor didn’t recognize the movie as being any longer than it was before, so it only encoded the length of the original movie. Deleting the job from the Compressor file and re-adding it solved this problem.


That’s all for now. Hopefully this will help at least one desperate person someday.