Monday, September 24, 2012

New Glasses for under $60

Advertising isn't my strongsuit, but, please allow me to shamelessly plug a fantastic website, introduced to me by my coworker Carl:

Zenni Optical, www.zennioptical.com

What's so fantastic, you ask? Hundreds of dollars not spent, that's what's fantastic.

I haven't bought new glasses for a couple years, so I'm due. But like most of my friends, I don't have an extra $400 lying around burning a hole in my pocket. At Zenni, I didn't need $400. In fact, I didn't even need $60. That's right, for under $60, I got a pair of glasses, bendable titanium frame, special "expensive" oleophobic lenses (means they resist oil and fingerprints), and shipping. I ordered on September 12, and they arrived today, less than two weeks later.

At this price, I might get a few more pairs, just for fun. Maybe some bright pink frames...

How I didn't fix freeradius on Snow Leopard Server to work with RADIUS authentication

Preamble
Highly technical post ahead. If you are a friend/family member reading this post, skip it, not worth your time. If you’re an IT professional running Mac OS X Server 10.6, using OS X’s RADIUS service + Open Directory to provide authentication for your network, and are experiencing an issue where OD users who have the “Password must be reset on next login” checkbox checked cannot authenticate on wireless machines, this post will be very interesting to you.

Also, I should note up front: I was never able to get a working solution to this problem. However, if you're technically minded and have command line administration experience, perhaps you can take my work and bring it to fruition. If so, please add a comment with your solution!



Introduction
Minnehaha (high school where I work) recently implemented an enterprise-class secure wireless network, using Aerohive access points, tied to RADIUS running on Mac OS X Server 10.6, integrated with Open Directory. The idea being: rather than our wifi network being wide open, rather than requiring users to authenticate each time they open a web page, or rather than installing a buggy authentication daemon on school-owned laptops, users would instead log in to the wifi network itself, using their own personal username and password (as opposed to a single shared password approach, which would create an even larger maintenance nightmare than the wide-open network).



The problem
RADIUS + OD work as expected for normal users. However, users whose passwords must be changed at next login (aka, ALL of the students at the beginning of the school year, because we reset their passwords over summer), could not log in to the wireless network until they had changed their passwords.

It’s a catch-22: the laptop cannot authenticate to the wifi until the user’s password has been changed, and the user cannot change their password until the login window is allowed to access the Open Directory (which requires being on the wifi).

Logging in to a wired machine worked as expected: the OS X login window prompted the user to change their password on first login. Subsequent logins for that user on wireless machines then worked as expected, because the checkbox requiring the user to change their password was no longer checked. (you can Google for directions on setting up a login window profile for 802.1X authentication on Mac OS X; there are numerous resources on the web that walk through those steps, so that will not be discussed here).



Desired result
Rather than requiring users to sign in to a wired lab machine before using the wireless network, we wanted to grant them access to the wifi, even if while the “change password” box checked.

Theoretically this opens a situation in which the student could bring their own computer on campus and never change their password, and theoretically someone outside of the school could figure out the student’s default password, but this was deemed a low risk compared to the hassle of requiring all users to find a wired workstation at the beginning of each school year. If and when that situation arose, we could deal with it on a case-by-case basis.



The cause
RADIUS denies access because of two lines in its source code (freeradius, the RADIUS implementation packaged with and used by OS X Server, is open source and available for download from http://www.opensource.apple.com/release/mac-os-x-1068/)

/src/modules/rlm_opendirectory/rlm_opendirectory.c

 switch(odResult)
 {
 case eDSNoErr:
   ret = RLM_MODULE_OK;
   break;
   
  case eDSAuthUnknownUser:
  case eDSAuthInvalidUserName:
  case eDSAuthNewPasswordRequired:
  case eDSAuthPasswordExpired:
  case eDSAuthAccountDisabled:
  case eDSAuthAccountExpired:
  case eDSAuthAccountInactive:
  case eDSAuthInvalidLogonHours:
  case eDSAuthInvalidComputer:
   ret = RLM_MODULE_USERLOCK;
   break;
  
  default:
   ret = RLM_MODULE_REJECT;
   break;
 }


Courtesy of http://www.mentby.com/Group/freeradius-users/password-policy-expired-password-mschap.html, those two lines in bold need to move right under the eDSNoError case, so they return an authentication OK:

 switch(odResult)
 {
  case eDSNoErr:
  case eDSAuthNewPasswordRequired:
  case eDSAuthPasswordExpired:
   ret = RLM_MODULE_OK;
   break;
   
  case eDSAuthUnknownUser:
  case eDSAuthInvalidUserName:
  case eDSAuthAccountDisabled:
  case eDSAuthAccountExpired:
  case eDSAuthAccountInactive:
  case eDSAuthInvalidLogonHours:
  case eDSAuthInvalidComputer:
   ret = RLM_MODULE_USERLOCK;
   break;
  
  default:
   ret = RLM_MODULE_REJECT;
   break;
 }



The solution I attempted, but eventually could not get to work
First, on your Mac OS X 10.6 Server, enable the root user and change his password to something you'll know: http://support.apple.com/kb/HT1528. You will be using this password frequently.

Open a Terminal window. (if you're not sure what that means or where to find Terminal, then please stop reading now; if you go further you WILL end up breaking your server)

Type "su -" to switch users to root. Type your root password.

WARNING: Seriously, if you're not comfortable using the command line, stop now, because you will break something. Not my fault if you rm -rf / your server's hard disk.

First I tried downloading the freeradius source from Apple: http://www.opensource.apple.com/release/mac-os-x-1068/. Untar (fine to double-click in Finder if you'd prefer), and make the two line code change from above.

I tried configuring at this point (using the command from freeradius's /doc/MACOSX, but ran into dependency problems, needed something called libiodbc. Google led me here: http://www.iodbc.org/dataspace/iodbc/wiki/iODBC/ODBCMacOSX. Download, ./configure, and make install that. (I encountered no build errors. It is worth noting: you need the Xcode environment installed in order to compile).

The other issue I encountered was something related to libtool, but this site suggested a workaround: http://fedoraproject.org/wiki/Libtool2#old_libtool_files_may.2Fwill_cause_problems.

All told, my configure command for freeradius looked like this:

root# ./configure --disable-shared --with-iodbc-lib-dir=/usr/local/lib --enable-developer --with-system-libtool

And now, a couple weeks later, I don't remember if that actually worked or not, but after trying that I decided to go a different direction: MacPorts. Maybe I decided this because I was worried the make install would overwrite my existing /usr/sbin/radiusd and I didn't want to irreparably break the currently-mostly-working version.

Useful information:
The freeradius daemon is called radiusd and lives in /usr/sbin, along with its friends, radwatch, radmin, rc.radiusd, and a few others. The configuration files are in /etc/raddb. You need to be root anytime you edit these files.


Moving forward with MacPorts
Download MacPorts from http://www.macports.org and install.

As root ("su -"), enter "/opt/local/bin/port fetch freeradius" (or edit root's .profile to include /opt/local/bin in your $PATH, so you can type just "port fetch freeradius").

Once it's been fetched, create two files in
/opt/local/var/macports/sources/rsync.macports.org/release/ports/sysutils/freeradius/files



patch-configure.in.diff

--- configure.in.orig 2009-12-30 09:44:35.000000000 -0600
+++ configure.in 2010-04-05 01:23:46.000000000 -0500
@@ -92,7 +92,7 @@
 dnl use system-wide libtool, if it exists
 AC_ARG_WITH(system-libtool,
 [  --with-system-libtool   Use the libtool installed in your system (default=use our own)],
-[ AC_PATH_PROG(LIBTOOL, libtool,,$PATH:/usr/local/bin) ],
+[ AC_PATH_PROG(LIBTOOL, glibtool,,$PATH:/usr/local/bin) ],
 [
   LIBTOOL="`pwd`/libtool"
   AC_SUBST(LIBTOOL)



rlm_opendirectory.diff

--- src/modules/rlm_opendirectory/rlm_opendirectory.c.orig 2010-05-24 00:40:58.000000000 -0500
+++ src/modules/rlm_opendirectory/rlm_opendirectory.c 2012-08-24 20:36:53.000000000 -0500
@@ -308,13 +308,13 @@
  switch(odResult)
  {
   case eDSNoErr:
+  case eDSAuthNewPasswordRequired:
+  case eDSAuthPasswordExpired:
    ret = RLM_MODULE_OK;
    break;
    
   case eDSAuthUnknownUser:
   case eDSAuthInvalidUserName:
-  case eDSAuthNewPasswordRequired:
-  case eDSAuthPasswordExpired:
   case eDSAuthAccountDisabled:
   case eDSAuthAccountExpired:
   case eDSAuthAccountInactive:



These files are unified diffs, and can be created on your own by duplicating the file you want to modify, modifying it, then running this command:

root# diff -u modified-freeradius/src/modules/rlm_opendirectory/rlm_opendirectory.c original-freeradius/src/modules/rlm_opendirectory/rlm_opendirectory.c > rlm_opendirectory.diff

Note: if you run that command instead of copy/pasting the file contents I provided above, note that you will need to modify the file generated by diff, so the paths are relative, not absolute. Example:

--- /absolute/path/src/modules/rlm_opendirectory/rlm_opendirectory.c.orig 2010-05-24 00:40:58.000000000 -0500

becomes

--- src/modules/rlm_opendirectory/rlm_opendirectory.c.orig 2010-05-24 00:40:58.000000000 -0500

Now edit the patchfiles value in
/opt/local/var/macports/sources/rsync.macports.org/release/ports/sysutils/freeradius/Portfile

patchfiles              patch-configure.in.diff \
                        rlm_opendirectory.diff

Note: A less elegant and less-effectual way to accomplish this (aka, what I tried first), is to download the source, make the changes, re-tar and bz2, generate new hashes (shasum -a 256 and openssl dgst -ripemd160) and put those in the Portfile. But that didn't work, because MacPorts would figure out the new bz2 archive didn't match the repository, and would re-download and overwrite it. The unified diff and patchfiles are the correct way to go.

I ran into an issue after configuring, again a libtool issue. https://trac.macports.org/ticket/12961 suggested adding a "--tag=junk" into the LIBTOOL portion of the Make.inc file. At first I tried making a unified diff and applying it to the Make.inc file after I ran configure, but that didn't work. In the end, the solution is to add this to the Portfile:

post-configure {
    reinplace "|/opt/local/bin/glibtool|/opt/local/bin/glibtool --tag=junk|" ${worksrcpath}/Make.inc
}

This was a frustrating learning experience, due to a near-complete lack of documentation or useful Google results about this syntax. But, what I have there, works. It acts as a simple find/replace: find this normal glibtool path, and change it to have junk included. Refer to the link above for why this counter-intuitive change makes things work.



Installing freeradius
Finally! Let's try installing.

root# port install freeradius
Note: if you tried installing, then went back to correct a mistake in the portfile/diffs/etc, I found you must port clean freeradius before trying to compile/install again.
Copy some files into /opt/local/, because that's where the new version of radiusd will look for them:

root# cp -pr /etc/raddb /opt/local/etc
root# mkdir /opt/local/var/run/radiusd

Shut down the real radius, then try running the new radiusd in debug mode (the -X flag):

root# serveradmin stop radius
root# cd /opt/local/sbin
root# ./radiusd -X

When I tried that, mine failed with this error:

Could not link driver rlm_sql_sqlite: file not found
Make sure it (and all its dependent libraries!) are in the search path of your system's ld.
/opt/local/etc/raddb/sql.conf[22]: Instantiation failed for module "sql"

Googled it, found this link: http://wiki.freeradius.org/FAQ#It-says-%22Could-not-link-...-file-not-found%22%2C-what-do-I-do%3F. Also found this page, which may or may not be useful: http://support.apple.com/kb/TA25017



Installing MySQL
Download the source from Apple: http://www.opensource.apple.com/release/mac-os-x-1068/, which links to http://www.opensource.apple.com/source/MySQL/MySQL-55/.

Compile with this command from http://hivelogic.com/articles/compiling-mysql-on-snow-leopard/ (plus a few things I figured out on my own):

root# ./configure --prefix=/usr/local/mysql --with-extra-charsets=complex \
--enable-thread-safe-client --enable-local-infile --enable-shared --with-plugins=innobase \
--with-mysql-include-dir=/usr/local/mysql_new/include/mysql \
--with-mysql-lib-dir=/usr/local/mysql_new/lib/mysql



Re-configure and install freeradius
Use this suggestion to add configure arguments to the Portfile:
http://superuser.com/questions/306550/how-do-i-install-php5-via-macports-with-a-custom-configure-option

Finished version looks like this:

configure.args          --with-openssl-includes=${prefix}/include/openssl \
                        --with-openssl-libraries=${prefix}/lib \
                        --with-system-libtool \
                        --without-rlm_krb5 \
                        --with-mysql-include-dir=/usr/local/mysql_new/include/mysql \
                        --with-mysql-lib-dir=/usr/local/mysql_new/lib/mysql


This time it should launch successfully:

root# port install freeradius
root# cd /opt/local/sbin
root# ./radiusd -X


Yay! But now authentication wasn't working for any users. The output (and log file) showed many many errors like these:

Sat Aug 11 11:29:13 2012 : Error: Ignoring request to accounting address * port 1813 from unknown client 10.1.111.112 port 32774
Sat Aug 11 11:29:14 2012 : Error: Ignoring request to authentication address * port 1812 from unknown client 10.1.111.113 port 32778


After lots of Googling, my first effort was re-adding all the Aerohives into the sqlite database manually:

root# cd /etc/raddb
root# sqlite3 sqlite_radius_client_database
sqlite> DELETE * FROM nas;
sqlite> PRAGMA table_info(nas);

0|id|int(10)|1||0
1|nasname|varchar(128)|1||0
2|shortname|varchar(32)|0||0
3|type|varchar(30)|0||0
4|ports|int(5)|0||0
5|secret|varchar(60)|1||0
6|community|varchar(50)|0||0
7|description|varchar(1024)|0||0

...
sqlite> INSERT INTO nas VALUES ("12","10.1.111.112","AEROHIVE-NAME-12","Aerohive","","passwordvalue","","");
...


But that didn't work. Then my coworker found a solution: add the Aerohive base stations into the clients.conf file:

root# cp -p /etc/raddb/clients.conf /opt/local/etc/raddb/
root# vi /opt/local/etc/raddb/clients.conf

client NC-AHAP-12 {
        ipaddr  = 10.1.111.112
        type    = Aerohive
        nasname = 10.1.111.112
        secret  = passwordhere
        shortname       = AEROHIVE-NAME-12
        nastype         = other
}

# add one for each access point

Note: the MacPorts compiled radiusd wanted to read files from /opt/local/, because that's where it lives; honestly it got very confusing, though, because sometimes it would read from /etc/raddb, and sometimes from /opt/local/etc/raddb; my solution to this was hard linking the files, or symlinking the raddb directory.
Note: in order to make your custom radiusd the default as launched by Server Admin.app or the serveradmin CLI, it needs to live or be hard-linked in /usr/sbin/ (along with radwatch, radmin, rc.radiusd, etc):

root# ln /opt/local/sbin/radiusd /opt/local/sbin
or
root# cp -p /opt/local/sbin/radiusd /opt/local/sbin

When we next launched radiusd, the "authentication address" errors disappeared and radiusd said "Ready to process requests." We were able to authenticate on a wireless device as a normal user, but still not as a user with the reset-password box checked. I don't know why.

Also, we saw these errors in the log constantly, for all the Aerohives:

Error: rlm_radutmp: Logout entry for NAS AEROHIVE-NAME-20 port 0 has wrong ID

I tried editing /etc/raddb/users as suggested by http://hints.macworld.com/article.php?story=20071130134610850:

DEFAULT Auth-Type = opendirectory
        Fall-Through = 1

But this didn’t work, either.

The new radiusd was left running overnight, since it appeared to be working. But at some point it stopped working, so the next morning no users were able to log in at all. I reverted to my backups of the original radiusd and raddb, though oddly the "wrong ID" error messages have persisted despite everything being put back to normal (/usr/sbin/radiusd, /etc/raddb/, et al).

At that point, I am sad to report, I gave up. By then I had already spent 14+ hours on the project, and since:

A) we could not keep the new version of radiusd functional, and
B) we had a viable workaround (users sign in to a wired machine to reset passwords before using a wireless laptop), and
C) Apple has already started phasing out Open Directory in future versions of OS X Server, and
D) we have other options to explore for RADIUS server devices, such as our smart switches, or a Windows-based server (blech!)

it was deemed not worthwhile to pursue this particular solution any further.

Even though I was the one who suggested abandoning efforts, I am still very bothered that I couldn't solve this. The journey was educational, but frustrating, as I felt perpetually "this close" to cracking it. I hope what I have written here will be useful to some other server admin out there, and that you are able to take this the rest of the way into a working solution. Good luck.