Reconnecting Snow Leopard User Blogs after changing Open Directory Masters; Also: fixing the Internal Server Error (500) when viewing User Blogs

This is one of those technical posts that may only be useful to approximately 3 people in the entire world. My future self probably being one of them if I run into these problems again.

Last week at work our new Xserve arrived to replace our Open Directory Master. I first attempted to use the server setup migration assistant to transfer services, but, as I suspected would happen, the Open Directory didn’t quite look the way it was supposed to, so I blew away the new server and started from scratch. For several reasons, I decided not to migrate the OD at all, but rather re-create all users from scratch on the new server once it was up and running.

Now, because I automated the user creation process, several users' usernames changed.¹ This was a mistake.

My web server, an Open Directory Replica of the former master, had several user blogs in the Collaboration folder. Not a big deal, I should just be able to change a plist here or there and transfer those blogs to the new usernames.

Fail. No, that won't work - while the Collaboration folder hierarchy is relatively simple, unfortunately every file (seemingly) contains a reference to the author's shortname. This makes life very difficult if, all of a sudden, your users' shortnames have changed after you bind to a different OD Master. Specifically, it resulted in the yourdomain.net/users list of blogs returning an internal server error (500), and any specific user's blog (if you type in the full URL) returned a 404.

Here's how I fixed it.

Step 1: Delete and re-create any users whose shortname may have changed. Re-create them with their original shortname. You can add additional shortnames in Workgroup Manager later, but their first shortname is the one that must match with what's in the Collaboration folder.


Step 2: Open Terminal, type sudo serveradmin stop teams

This stops the teams service so you can do some file cleanup.


Step 2.5: Make a backup copy of your Collaboration folder (default path /Library/Collaboration). I ended up restoring and starting over several times in the course of my experimentation, and having that backup made life much easier.


Step 3: Delete these files: (if you use a custom path for the Collaboration folder like I do, adjust accordingly)

/Library/Collaboration/dataVersion.plist
/Library/Collaboration/globalIndex.db
/Library/Application Support/Apple/WikiServer/directoryIndex.db
/Library/Application Support/Apple/WikiServer/sessions.db


Step 4: Open /Applications/Utilities/Console.app and select the /Library/Logs/wikid error.log


Step 5: Switch back to Terminal and type sudo serveradmin start teams

This starts the teams daemon, which should automatically rebuild the blog and wiki indexes. Now watch the log closely. If it stops and hangs on a user or group name before you see the word "Running", then you'll have to fix things. Look at the last user or group listed in the log, then switch to Finder and poke around that folder (inside the Collaboration folder). (also stop teams using the command above)

If you're not able to navigate inside the Collaboration folder (little red permission denied circle), open Server Admin, click the server name, click File Sharing across the top, navigate to Collaboration by double clicking the hard drive in the list, then below, where it has permissions, set Others to Read & Write, click the drop down button below that, choose "Propagate Permissions", check "Others Permissions", then OK. And remember to change that back to Others None and propagate after you're done.)

Now, I didn't have any issues with my groups, but I'm guessing that procedure will be similar to this:

In Collaboration, look in Users, then the username that the index rebuild hung on. In there, compare the list of files in the "discussion" and "weblog" folders - for each .plist in discussion, there should be an identically named folder in weblog. If there isn't, then delete that .plist file. I don't know how or why these extra files come into existance, but when it happens, the server looks for the blog entry with the same number, and when it can't find one, it sticks out its tongue and stops indexing.

For safety, I also deleted the discussion.db file in the discussion folder. May not be necessary, but it gets rebuilt automatically, so no harm.

After your folders are in sync, start teams and lather, rinse, repeat until the log file indicates the teams daemon is "Running".



After all this, one of my user’s names still isn’t showing up in my master list of blogs, but typing his username into the URL manually loads his entire page of posts (if you get a “no posts” message, even after successfully reindexing, make sure that user's discussion and weblog folders are in sync - sometimes the teams service started up even though a blog wasn't actually indexed).


Lastly, on a related but different note, if you find that you can no longer authenticate to your blog even after typing your correct password, make sure your OD master has the WebDAV-Digest option turned on in Server Admin under Open Directory > Settings > Policies > Authentication. (then reset your password so the server saves the new hash)

That one took me several hours to figure out, I finally got a clue after seeing this line in the PasswordService log:

AUTH2: {0x4c3f90ef3b3b2bc20000003c0000003a, gustafson} WEBDAV-DIGEST authentication failed, SASL error -13 (password incorrect).

Previous to that I was only seeing this error in the wicked error.log:
Failure: twisted.cred.error.UnauthorizedLogin: Bad username or password: gustafson

If you found me from Google and solved your problem because of anything I wrote here, let me know!



¹ I wrote a nifty PHP script that queries our online directory for student and employee names, compares it against the Open Directory, then automatically creates new users and emails me a report. I highly recommend reading Apple’s User Management guide and the dscl [Directory Services Command Line utility] man page.

The 48 Hour Film Project 2010

After returning from Apple’s WWDC in San Francisco last week, I had a 9 hour layover at home before my parents picked me up for our road trip out east. We visited relatives in Indiana, Ohio, and Pennsylvania, then I flew home ahead of them on Friday to help with the 48 Hour Film Project kick-off event. While I was not planning to join a team (I was tired after two weeks away, preceded by an intense week prepping and shooting for the 50-Fifty Reel Challenge), I also knew that, if asked, I probably wouldn’t need much arm-twisting before signing on.

Minutes into the kick-off event, Crow and JD, the producers I’d worked with on the 50-Fifty, asked if I’d be interested in joining their 48 crew (technically not “theirs”, since they weren’t producing this time, but rather, “the team on which they were working”). Andy, one of the directors, echoed their request shortly thereafter, and I officially agreed. Who needs sleep, anyway, right?

Following the kick-off, with genre (Suspense/Thriller), character, prop, and line of dialogue in hand (figuratively), we journeyed to Andy’s house, eagerly awaiting news from the writing team upstairs. Knowing there would be an early call in the morning, I went home around 11:00 to get some sleep. When I woke up the script was patiently waiting in my inbox, and I took off for the set.

The shoot overall went well. Aside from Crow and JD I’d never worked with any of these folks before. And I didn’t have a specific job title, so I kind of did everything: AD, camera op, sound mixer, assistant to the DP, PA, etc. What was so awesome about this shoot, though, is that, despite the time pressure, stress, and many of the crew being sleep-deprived, no one stepped on anyone’s toes. I was throwing ideas at the DP, we were both throwing ideas at the directors and the AD, and the collaboration just worked.

I had an hilariously embarrassing moment in the middle of shooting: as we were prepping for a shot, I couldn’t remember one of our crew member’s names. Thinking he was in the other room, I whispered over to Crow, “hey, what’s that one script supervisor’s name? Is it Aiden?” Someone pointed and I realized Aiden was actually sitting right in front of me. Oops. But at least I got his name right. And provided the fodder for a running joke that continued the rest of the shoot.

There were some awesome crew members on this shoot. Smi, our second camera operator, exceeded all expectations. And Jason, our “Boom Goon”, was phenomenal - we literally never had boom shadow or the mic in frame. I don’t know how he did it. And Erin, the 1AD, who was so good with details all around. Definitely people with whom I want to work again.

We wrapped shooting about on schedule and sent the second set of tapes back to the editor to start importing while we cleaned up the apartment where we’d been shooting. It’s quite amazing how quickly that place returned to normal, given the extent of set-dressing/trashing that had been done to it.

Back at the director’s house we ate dinner and gathered the last few pickup shots with our primary actor (who, by the way, was absolutely phenomenal! He’s a theatre guy, never done film before, but he was by far one of the strongest actors I’ve ever worked with).

Crow, JD, and I all went home to shower and change clothes before coming back for the long-haul. When we returned, we set up our LAN party in the dining room: their PC for importing and clipping the sound recordings (did I mention we rolled sound separate from the cameras? It’s like we’re legitimate movie-makers or something), and then my MacBook Pro, second monitor, and some hard drives as a secondary editing machine.

After battling the wireless router (it had run out of IP addresses with all the new computers in the house), we waited. At some point in the middle of the night, we knew the editor would need to leave to go get sleep (what’s that?), so I’d be taking over at that point. Until then, there wasn’t much for us to do.

Then tragedy struck: my voice recorder stopped working, displaying a “memory error”. This has absolutely nothing to do with the 48 competition, but it’s part of the weekend’s story. I’ve had this voice recorder for 5 years, it’s become perhaps even more essential to my daily living than my phone, so I panicked. Crow and I took a middle-of-the-night run to Walmart, the only store still open at that hour, to buy a new one (which, unfortunately, turned out to be a cheap plastic wanna-be, so I returned it a few days later and ordered a quality one from Amazon).

Around 4:00 or 5:00 am, Katie, our editor, was dead tired and needed to go home. She gave me a copy of the Final Cut document (she’d already copied all the footage to one of my external drives earlier in the evening), and, thank you Apple, when I opened it on my computer everything “just worked”.

The directors and I continued doing tweaks for many hours. It got light outside. The composer showed up, we gave him a new rough cut to start working with. Finally around noon we had picture lock. I think. My memory is vague.

Leading up to that, I kept asking “who’s going to do sound syncing and mixing?” Uh-oh. There was no one lined up to sync sound from the recorder to the final picture. And the on-board camera mic recordings truly were not useable. Not a good situation. Worse: no one, including me, knew a good way of syncing all the video and audio clips. I had a couple ideas, but none were ideal due to their labor-intensive natures.

My first thought (which we ended up going with) was to find the marker/clap point in the video, find the same clap in the audio, then do some math to identify the correct audio in-point to set. For each and every clip. Granted, it’s only a 6 minute movie, but that’s still a lot of work.

My second thought was to strip the audio portion from the QuickTime source files and replace it with an identical length track of the audio recordings from the field recorder, spliced together to sync with the video. In an ideal world, that would have worked swell, but I realized (far too late) that in the time-limited world in which we were dealing, this was an inefficient idea. Oops.

On two friends’ suggestion I also tried a program called “Pluraleyes”, but I couldn’t get it to work in the time we had.

In any case, I knew I also had to do color correction, so my laptop would be unavailable to start working on sound for quite some time. We needed Katie’s laptop back if we had any hope of finishing on time. She graciously agreed to drop off her MBP and let us use it, even though she couldn’t be there to help. We owe her so much. With her computer, Crow was able to start audio syncing while I finished color. Now, Crow has never used Final Cut before, because he’s a PC guy. But the interface was similar enough to, and in many cases simpler than, Vegas, which he has used in the past. So with very few questions to me, he was off and running.

At this point I hadn’t slept for 30 hours, so I was getting a little loopy. Apparently I’m funnier, and also more laid-back when I’m tired.

Under time pressure, the color correction took about 30 minutes for every 1 minute of movie. Roughly. Some shots were much easier than others. At several points in the process I declared that really it’s no longer about making the scene look “good” as it it about making it look “less crappy than it was”. That got a good laugh.

Oh, have I mentioned the A camera footage yet? Somehow the XL-1 that was our A camera got switched to “frame mode”. As best I can tell, this is Canon’s way of screwing with indie filmmakers to make the footage look like crap. Truly. I am still completely baffled why they would add a button that changes your frame rate to 7.5fps and crapifies your footage. And somehow, mid-way through the shoot, this button got pushed, so some of our best camera angles for the latter part of the film were unusable. We snuck one or two shots in that weren’t too awful. I applied a motion blur and tween that, on a short clip, makes it almost decent looking. Hopefully the audience will think we added a pseudo-slow-motion filter to those clips on purpose, because that’s what it looks like.

Late Sunday afternoon (movie is due at 7:30), color was done, and then I started helping with sound. We were missing a lot of sound clips initially, so I kept JD busy pulling those up on her computer and flash-driving them over to me. Once I finished the first half of the movie with syncing, I grabbed Crow’s FCP document, copy/pasted those audio clips into my master (another thing that “just works”, thank goodness), and then walked away so he could start mixing things together.

Somewhere in there the soundtrack came down from upstairs, so I dumped that into the timeline. Didn’t have time to listen to it, of course, but at least it was there.

At 6:30, with an hour left before the deadline, we had to go. The car was waiting, along with an AC inverter, so we moved my computer and a hard drive out to our “mobile office”.

Arriving at the drop-off, we had a little more than half an hour. Crow finished audio balancing, the directors and he took one last watch, then I exported and burned the DVDs. We turned the finished product in with about a minute to spare.

Last year I helped with the 48 check-ins, and I was amused by the folks sitting in the lobby doing their final renders at the very last minute. Now I know how they must have felt. And we were lucky: we got ours in on time. A lot of those other teams didn’t. We felt even more lucky because hours earlier the directors were pretty much expecting we wouldn’t finish on time, given all the audio issues we ran into. At one point one of them literally said “we don’t have a movie, do we?”

But it turned out. Katie did a great rough cut, which got me to a place I could finish it. The sound sync issues got resolved, and now I know how to do that next time (I’m even writing a little helper program to do the math for me). And the story itself. It won’t be *the* winner, but it should at least be one that people talk about afterward. I think we have a product we can be proud of. I keep coming back to this mantra: I could have done better, but not in the time we had; I did my absolute best given the situation, I could not have given it any more.

By the time I got home I’d been awake for 40 hours straight. To my recollection, I don’t believe I’ve ever stayed awake that long in my life. It was exhausting, and I’m still recovering, but it was also totally worth it, and I’d do the whole thing again. Not next weekend, but eventually.

“Life should NOT be a journey to the grave with the intention of arriving safely in an attractive and well preserved body, but rather to skid in sideways at the last minute, champagne in one hand, strawberries in the other, body thoroughly used up, totally worn out and screaming "WOO HOO - That was Fun!”

Video Phones - The Future is Here

It's more than just marketing, I do believe this will change everything. I teared up when I saw the two people signing; a phone for deaf/hard-of-hearing folk, wow. Finally, after over 10 years since the Earth: Final Conflict props made me wish for video phones, the first generation has arrived. It won't be perfect, it will take time to go mainstream, but it's here. We're here. The future has arrived, and we get to live in it. How cool is that?

(Full video available on Apple's site: http://www.apple.com/iphone/features/facetime.html#facetime-video)

Paul Isaacs

Last night, my friend and coworker Paul Isaacs died from cancer, passing peacefully, surrounded by his wife and daughters.

Though I never had Paul as a classroom teacher, he became a mentor in my life in ways words can’t describe. The way he lived his life was a model I respected and admired. His wry sense of punny humor cracked me up almost every time. And his down-to-earth, welcoming voice calmed and gave a sense of peace.

As I’m discovering, befriending members of an older generation comes with a price. Though God alone may count our days, there exists an inherent likelihood that my older friends will die before me, perhaps long before me. I continue to struggle to wrap my mind around that reality. Simply put, I don’t want to.

Life rarely waits for anyone to “be ready”, though. And so, on the same day that I attended a wedding, heard God’s love spoken into my life, worshipped, prayed, cried, and even ran mundane errands, Paul had his last hours.

I found out this morning, the first email came from a mutual friend and coworker a little before 9:00. Official word from Paul’s principal came almost an hour later, after much of Facebook already knew. Finally, later in the day, we were given the go-ahead to post something on the school’s home page.

Days like today ignite in me a passion to work in the news industry. We live in a flat world, the word has to get out, people have to be told. To wait an hour, or worse, half a day, is unconscionable. News ought be instantaneous to the greatest degree possible.

Today God and I aren’t on good terms. If I’m honest, I’m pretty much pissed. Paul was younger than my parents, he has kids about my age, and he was a good man. If ever one needed proof that “only the good die young”, Paul epitomizes that. How God could allow this, why God offers healing to some but not others, these are the questions I’m battling. And I’m angry, and I’m sad, and I’m frustrated, and I’m torn.

And it’s okay, because God is big enough to handle my anger, my frustration, my questions. And it’s okay, because some day I will find peace. Maybe not answers, but peace. Paul did. His family did. I can too. Eventually.

Just not today.

Multithreading with PHP

It’s "not possible" according to Google. And maybe true multithreading isn't, but by modifying another user's code I found on PHP.net, this solution worked pretty well for me:

http://www.php.net/manual/en/function.proc-open.php#97713

Now a script that would have taken well over 10 minutes to run takes my server only 1 minute. Sweet.

A Loving Argument

Argumentative by nature, I’ve been in my share of heated debates. Recently I posted a controversial news item on my Facebook wall, and one of my friends sent me a message challenging what I had written. That’s nothing new - for me, Facebook is all about starting conversations.

What is new, though, is the attitude of his letter and our subsequent responses back and forth. I’ve had written debates with this friend before, and today I realized what distinguishes our disagreements from just any typical argument: we share a love and respect for one another that transcends our differences.

Truly, writing my responses today felt very different from any other confrontational message I’ve written. And it took me a while to pin it down. But once I did, it hit me deeply. And it also set me at ease. Yeah, I’m still out to prove a point, to “defend” my side of the issue, but I didn’t feel as intense/aggressive/antagonistic as I normally would. I also realized: I’m actually okay living with the disagreement, because I respect this friend, and even though we disagree on a lot, that just means we have more to talk about :) The brotherly love is stronger than the argument, and that’s really cool to experience.

Fjeldberg Lutheran Church

Last week I visited friends in and around Ames, IA, and Sunday morning, before heading home, I attended a service at Fjeldberg Lutheran Church in Huxley. It’s a traditional-style ELCA congregation, but they caught me off-guard by using a slightly altered liturgical setting that I hadn’t experienced before. Throughout the whole service, there were definitely elements of the familiar: the creeds and Borg-like recitations I’d grown up with. But there were also new things, musical responses where I wasn’t expecting a musical response, or even more scary: different music with a traditional text.

Okay, it wasn’t really scary. It was actually a nice change from normal. Previously I’d assumed all traditional Lutheran services are made more or less equal. Apparently I was wrong. This service was just different enough from both my parents’ church and the student congregation at St Olaf to make it unique in it’s own right, while still retaining enough elements of the traditional to remain familiar.

And so I determined that it’s more challenging to follow along in a service that uses only a slightly different liturgy, verses a service that’s completely non-liturgical, where you’re always on guard for what might be coming next. At Fjeldberg I was never sure when to let my guard down, when it was safe to sink back into rote memory, and when there was going to be a new melody thrown at me.

I got plenty of practice sight-reading last Sunday. And I loved the challenge.

Partly Cloudy with a chance of Sermon

In watching all the special features in my DVD collection, tonight I watched the Pixar short called Partly Cloudy (this short was played before Disney's Up in the theatres). Ever since I first saw it in the theatre, I knew there was a deeper theological meaning I should be getting (albeit perhaps eisegetically; I won't claim that Pixar intended this deeper meaning). And then tonight, watching it again, I once again felt that tug: "there's something deep here, figure it out".

After pondering for a few moments and failing to think of anything brilliant, I turned to Google. Who would have thought a search for "partly cloudy pixar theological message" would turn up anything useful? But it did.

http://www.firstthings.com/blogs/theanchoress/2009/07/07/caritas-in-veritate-by-b-16-pixar/

Some, and by "some" I mean "all", of the quotations from Pope Benedict's encyclical went over my head, but the author's explication of Partly Cloudy's theological implications down right made me tear up. Taken with the blog author's interpretation, Partly Cloudy becomes an incredibly poignant 5-minute movie illustrating God's unique love, and unique Call for each of us.

I know Your Call won't always be easy, yet here I am Lord. Send me.

(turns out the video link on the blog doesn't work; the video is available here, though: http://www.redbalcony.com/?vid=24992)

Red Condor / Visi Bouncing Emails with Error Message "554 Failed: Malformed MIME header (in reply to end of DATA command))"

For the last two weeks I've on-and-off been puzzling over this issue: every email sent to users at visi.com from a PHP script I wrote bounces back with this error message: "554 Failed: Malformed MIME header (in reply to end of DATA command))".

Today I finally found the solution.

In my early searches, Google had nothing useful to say, so I'm writing this post in the hopes of helping some future PHP programmer who's banging his or her head against a wall in angst.

Here's the setup: for our online back-to-school / class registration system at Minnehaha (that I wrote), there's a PHP script that allows users to reset their forgotten passwords. Part of this process involves sending them an email with a unique identifier. The problem is that email wasn't always being delivered: users with visi.com email addresses did not receive their message. After examining the web server's logs, I discovered the error message above, something about a malformed MIME header.

I emailed Red Condor, Visi's filtering service (their name showed up in the log next to the bounce notice), and they wrote back in about 10 minutes. Literally. I was floored at how quickly their tech support responded.

Our first place of investigation was my custom headers. I need these so that the message is sent as HTML instead of plan text, but since it's human-made, it's the most logical place for a mistake. The PHP code looks something like this:

$headers = 'MIME-Version: 1.0'."\n".
'Content-Type:text/html;charset=iso-8859-1'."\n".
'From: Name <name@domain.com>'."\n";

One possible culprit was the line endings. The mail message spec dictates that all lines in the header need to end with \r\n (carriage return and a new line. For those too young to know what a carriage return is, Google for "typewriter"; it's an old-school laptop our parents and grandparents grew up using). If you'll notice, in my code above, I only have \n, not \r\n. IMPORTANT NOTE: If you're running PHP on a UNIX or Mac OS X server, the mail() command will add \r for you! So if you manually put in \r\n, you'll end up getting \r\r\n, which is bad.

So as far as I could tell, my headers were all correct.

Next, I manually grabbed my entire inbox from the mail server and examined it in a hex editor (Mac OS X users I recommend a free program called Hex Fiend). All my line endings were as they should be (Hex values 0D 0A).

I emailed back and forth a few times with Red Condor, and my tech support person suggested I capture the email coming from the server before it gets to the mail server, and then send them those bytes. I spent a day trying to figure out how to get PHP to dump an email into a file before giving up. I thought I'd never find the solution, until...

RC tech support suggested I try a packet capture.

Well that sounds fun. I know the concept, and I found a free program (CocoaPacketAnalyzer) to let me do it easily, but could I really find what I was looking for?

Yes, turns out.

I started a capture on the webserver, told the script to send an email to one of my Visi users, and then examined the packets that were sent. After a little searching I found the packet with the actual mail message, and here's the gold: all the bytes were correct except for the X-PHP-Originating-Script header, which had the dreaded \r\r\n trailing it.

How the heck do you get rid of that? It's not an error in my custom code, it's an error in PHP's mail() code!

Our network consultant Dan suggested I disable that header to see if that would solve Red Condor's rejection issue.

Google was helpful on this front: you can disable the X-PHP-Originating-Script header by setting mail.add_x_header = OFF in your php.ini file (and then restarting web services, obviously).

Turns out, disabling that header ALSO removed the extra \r. In my next packet capture, the extra byte was GONE and the email sent successfully, no more bouncing.

It's always something so small, so innocent. I don't know if Red Condor was rejecting the X-PHP-Originating-Script header or was just unhappy about the extra carriage return byte, but either way, I'm happy it's solved!

Two Coins

Joanna and I have been hitting it hard this past week polishing up draft 2 of the FAR AWAY screenplay. We finished around 1:30am this morning, and I once again sent the script out to a lengthy list of friends and filmmakers for feedback.

Because of all the work we’ve been doing on the script, I’ve had no time to keep up with my RSS and podcast subscriptions until tonight. So this evening, as I was catching up with the posts from IMMD (ItMadeMyDay.com), I read this:

My 8 year old son wanted to donate money to Haiti after hearing a radio story about how difficult it is to get food to people right now. I expected $10, he gave $100 (half of his lifetime savings)! IMMD

Wow.

I mean, sure, even though January and February are both rather fiscally difficult months for me, I threw some money at the Red Cross after the earthquake. But not half my life savings. Not even half my paycheck for that week.

Now one can make an argument that, unlike the 8-year-old, I have other financial responsibilities: a house payment, food expenses, gas for the car, etc, that keep me from being able to donate at that percentage level. I don't think these are invalid, or even inappropriate, excuses. But they are excuses, nonetheless.

Last year I donated more to charities than any previous year in my life, and I'm hoping to best that again in 2010. Granted, having a full-time job sure helps :) But even so, I know I won't come anywhere close to what that 8-year-old did.

As he looked up, Jesus saw the rich putting their gifts into the temple treasury. He also saw a poor widow put in two very small copper coins. "I tell you the truth," he said, "this poor widow has put in more than all the others. All these people gave their gifts out of their wealth; but she out of her poverty put in all she had to live on." - Luke 21: 1-4