Reconnecting Snow Leopard User Blogs after changing Open Directory Masters; Also: fixing the Internal Server Error (500) when viewing User Blogs

This is one of those technical posts that may only be useful to approximately 3 people in the entire world. My future self probably being one of them if I run into these problems again.

Last week at work our new Xserve arrived to replace our Open Directory Master. I first attempted to use the server setup migration assistant to transfer services, but, as I suspected would happen, the Open Directory didn’t quite look the way it was supposed to, so I blew away the new server and started from scratch. For several reasons, I decided not to migrate the OD at all, but rather re-create all users from scratch on the new server once it was up and running.

Now, because I automated the user creation process, several users' usernames changed.¹ This was a mistake.

My web server, an Open Directory Replica of the former master, had several user blogs in the Collaboration folder. Not a big deal, I should just be able to change a plist here or there and transfer those blogs to the new usernames.

Fail. No, that won't work - while the Collaboration folder hierarchy is relatively simple, unfortunately every file (seemingly) contains a reference to the author's shortname. This makes life very difficult if, all of a sudden, your users' shortnames have changed after you bind to a different OD Master. Specifically, it resulted in the yourdomain.net/users list of blogs returning an internal server error (500), and any specific user's blog (if you type in the full URL) returned a 404.

Here's how I fixed it.

Step 1: Delete and re-create any users whose shortname may have changed. Re-create them with their original shortname. You can add additional shortnames in Workgroup Manager later, but their first shortname is the one that must match with what's in the Collaboration folder.


Step 2: Open Terminal, type sudo serveradmin stop teams

This stops the teams service so you can do some file cleanup.


Step 2.5: Make a backup copy of your Collaboration folder (default path /Library/Collaboration). I ended up restoring and starting over several times in the course of my experimentation, and having that backup made life much easier.


Step 3: Delete these files: (if you use a custom path for the Collaboration folder like I do, adjust accordingly)

/Library/Collaboration/dataVersion.plist
/Library/Collaboration/globalIndex.db
/Library/Application Support/Apple/WikiServer/directoryIndex.db
/Library/Application Support/Apple/WikiServer/sessions.db


Step 4: Open /Applications/Utilities/Console.app and select the /Library/Logs/wikid error.log


Step 5: Switch back to Terminal and type sudo serveradmin start teams

This starts the teams daemon, which should automatically rebuild the blog and wiki indexes. Now watch the log closely. If it stops and hangs on a user or group name before you see the word "Running", then you'll have to fix things. Look at the last user or group listed in the log, then switch to Finder and poke around that folder (inside the Collaboration folder). (also stop teams using the command above)

If you're not able to navigate inside the Collaboration folder (little red permission denied circle), open Server Admin, click the server name, click File Sharing across the top, navigate to Collaboration by double clicking the hard drive in the list, then below, where it has permissions, set Others to Read & Write, click the drop down button below that, choose "Propagate Permissions", check "Others Permissions", then OK. And remember to change that back to Others None and propagate after you're done.)

Now, I didn't have any issues with my groups, but I'm guessing that procedure will be similar to this:

In Collaboration, look in Users, then the username that the index rebuild hung on. In there, compare the list of files in the "discussion" and "weblog" folders - for each .plist in discussion, there should be an identically named folder in weblog. If there isn't, then delete that .plist file. I don't know how or why these extra files come into existance, but when it happens, the server looks for the blog entry with the same number, and when it can't find one, it sticks out its tongue and stops indexing.

For safety, I also deleted the discussion.db file in the discussion folder. May not be necessary, but it gets rebuilt automatically, so no harm.

After your folders are in sync, start teams and lather, rinse, repeat until the log file indicates the teams daemon is "Running".



After all this, one of my user’s names still isn’t showing up in my master list of blogs, but typing his username into the URL manually loads his entire page of posts (if you get a “no posts” message, even after successfully reindexing, make sure that user's discussion and weblog folders are in sync - sometimes the teams service started up even though a blog wasn't actually indexed).


Lastly, on a related but different note, if you find that you can no longer authenticate to your blog even after typing your correct password, make sure your OD master has the WebDAV-Digest option turned on in Server Admin under Open Directory > Settings > Policies > Authentication. (then reset your password so the server saves the new hash)

That one took me several hours to figure out, I finally got a clue after seeing this line in the PasswordService log:

AUTH2: {0x4c3f90ef3b3b2bc20000003c0000003a, gustafson} WEBDAV-DIGEST authentication failed, SASL error -13 (password incorrect).

Previous to that I was only seeing this error in the wicked error.log:
Failure: twisted.cred.error.UnauthorizedLogin: Bad username or password: gustafson

If you found me from Google and solved your problem because of anything I wrote here, let me know!



¹ I wrote a nifty PHP script that queries our online directory for student and employee names, compares it against the Open Directory, then automatically creates new users and emails me a report. I highly recommend reading Apple’s User Management guide and the dscl [Directory Services Command Line utility] man page.