Everybody Staze…

In late January, I was able to upgrade my server to a Mac Mini Server. This is a pretty sweet little box. 2 500GB HDs, 4GB of ram, Intel C2D 2.53ghz. No optical drive, but that’s fine with me, since you can use remote drive, or purchase one of the USB optical drives for the MacBook Air.

Initial setup was quite easy, once I found a copy of “Remote Install” to run on my PowerMac G5. This utility is basically “NetBoot” in an extremely simple interface. It allows the Mini Server to netboot the install DVD. I then changed the HDs to be in a RAID1, and installed 10.6. That took a bit, but other than that, it was quite painless.

bonnie++ reports look like:
/usr/local/sbin/bonnie++ -d / -s 8G -u root:wheel -qfb
server.example.com,8G,,,58807,12,27984,5,,,110101,8,212.3,0,16,12352,59,+++++,+++,
8137,47,543,5,+++++,+++,231,3

Which, isn’t bad at all for dual 5200rpm drives. The machine shipped with two Hitachi HTS545050B9SA02.

Upgrading from the previous server (a Powerbook G4, 1.67ghz, 2GB of RAM, and a single 80GB HD) to this new machine was, to put it simply, breathtaking. The average load on this new machine is consistently less than 0.05, yet on the old PB, it was often upwards of 0.50. So, by that regard, about 10x the performance (realistically, probably more like 6-8x as fast). =) Also, there is a noticeable improvement in performance of the Drobo. I’m pretty sure the FW800 port on the Powerbook is flakey. Added bonus is the 64-bit nature of the new machine… which is really quite nice (really the main time I’ve encountered it is playing with bigint’s in PHP). Also means I can upgrade the ram to 8GB when the prices come down on DDR3 SO-DIMMs.

Add to that the fact that I can now run 10.6, and am not suck with the PPC OS’s, it’s great.

The only thing Apple could have done better? Remove the Mini-DVI port and add a second gig-E port. If they did that, I could probably start using these things at work rather than Xserves. =) Especially if I had an iSCSI SAN.

The machine is very quite. I can only hear the HD’s access occasionally. The fan is near silent. The Mini really is one of the greatest computers Apple has ever designed/made. Other than the dumb single RAM slot the PPC mini’s had, they all have been great machines. I still have a 1st Generation Mac Mini running strong (1.25ghz G4, 1GB RAM, single 80GB HD, 10.5.8 client) at work (it runs Intermapper to monitor various devices around the building. Works great).

Verdict: 9.0/10. Only changes would be the second gig-E port, and it would be sweet if you could buy without the OS (if this was for work, I wouldn’t need even MORE copies of 10.6 (already have spares)). I’d give it a full 10, but I can’t think of anything that’s perfect. =)

If you have any questions, please feel free to leave a comment.

*Image lifted from Apple website.

Since January of this year, I’ve been actively seeing AppleFileServer crash regularly on a server at work. This server is our primary student account server, which at any given time has about 40-80 students logged in (network home directories).

Many days, AFP crashes several times. Every time, it’s the same error: kern_protection_failure. The thread that crashes is always talking about ByteRangeLockTreeKey. The only good thing about this problem, is seemingly AFP comes back up, and people’s computers reconnect (go autofs!). But this is a very poor consolation prize since for some people, this does cause a problem (anyone with Mail open usually gets an error about not being able to access their inbox, and do they want to rebuild, or quit, and some others occasionally get Final Cut project file corruption (this is rare, and only seems to impact those that have their autosave vault set to their home directory, and not the local HD)).

So, Apple was notified about this, officially, on Jan 22nd, 2009. Ticket number 6517425. After getting back to me and asking for some follow up info, they proceeded to roll the ticket into another one (6237420). This ticket, apparently, was not related, and after telling our Sales Engineer about this, he had them un-merge the tickets. Apple then rolled my bug into another ticket, 5859645. An even older ticket! From what I’ve gathered, this ticket may be related to some lower level issue than AFP… either filesystem level (perhaps ACLs?!?, or even general I/O level).

All the while, I am in contact with someone in Minnesota who is having my same issue, and has also opened tickets (and has the luxury of having AppleCare for 10.5 server (the high end AppleCare to boot). He had two open case numbers with them. He even had a regional service engineer come by and take a look at this system, which he said was set up correctly, and there’s nothing more they could do to help alleviate the problem until a patch was available.

So, also during this time, someone from London contacts me and says he’s having the same issue as well, and has a Developer account (pay for), so he tries a beta of 10.5.7. It does not fix the issue. Around this time, I downgrade to 10.5.4 hoping the issue will be lessened (long story short, it isn’t). But, a few weeks later, the gent from London says he’s fixed his problem by removing the “deny all” acl from all his share points and folders within share points. The “deny all” acl was added around 10.5.4 or so to mitigate something… no one’s sure what. Anyway, he then tells Apple about this “fix” and they reply that it’s an “unacceptable workaround” and that they’re working on a fix. This was April 9th he did this.

Well, so, 10.5.7 dropped last Tuesday (May 12th, 2009). I installed it on the server experiencing the issue Friday night, at about 2am. I didn’t have a single crash until Sunday, May 17th, 2009, at 5:52pm. Same exact error.

So, not only was Apple notified AT LEAST 110 days prior to 10.5.7 shipping, but they were notified of an actual “fix” about 33 days before hand. I really wish Apple’s bug database was public, so that I could post links to my bugs, but, alas it is not.

However, here are a few threads on the issue:

http://www.afp548.com/forum/viewtopic.php?showtopic=23311
http://discussions.apple.com/thread.jspa?threadID=1975848
http://discussions.apple.com/thread.jspa?messageID=8857952

At this point, I’m going to start actively poking buttons and prodding people until I get an answer. The last email I sent to devbugs@apple.com resulted in the “pat”, “There is no new information at this time”. What a load of horse crap. They know of at least one “option”… the least they could do would be to educate someone having this issue about that “fix” and it’s repercussions. Given the amount of time that 10.5.7 took to hit the street, and how far in advance I notified them about this bug, I have very little hope this will get fixed before 10.6. If we’re lucky, we’ll see the fix back ported, but I doubt it.

To cap this all off, the main reason I’m posting this is for posterity, as well as the hope that anyone else that has this bug can actually see they’re not alone! And that they can contact Apple and say “hey, I have some bug numbers here of others having this issue”. If you are having this issue, please, don’t hesitate to contact me and I’ll work to get you in contact with others having this issue, or with someone at Apple that will actually listen.

UPDATE 1: Today I got a call from the local Education SE, who has created an escalation of this issue. Assuming it gets signed off by his boss, I should be hearing from Apple Engineering in the next few days… which is good since AFP crashed 5 times today. I have decided, in the interim, to remove the “group:everyone deny delete” ACL from many of the home folders on the server. Hopefully this will ease the problem. We’ll have to see. And I’ll post more once I hear from Engineering.