Everybody Staze…

So, I’m a Sys Admin. I spend all day running down problems, and when I’m not doing that, I’m usually working on bigger problems that are multi-day/week/month issues. And when I’m not doing that, I’m playing with something cool that’ll give us a better user experience, show us data we haven’t seen, etc. Add to this, we basically have no money (we’re State Higher Ed). So, the answer for testing issues is, obviously, virtualization (since we can’t afford to have test machines sitting around).

Now, VMware and Parallels both support Virtualizing OS X Server on Apple Hardware. They do not, however, support Virtualizing OS X Client, since Apple’s EULA specifically says “You can’t do that”.

So, being a Sys Admin, and not taking anything at face value, you look around online, and find various ways around this. Yes, it’s breaking EULA. But, I am running this on Apple Hardware, and we have broken machines sitting in a closet not running any OS. So, far as I’m concerned, it evens out. Obviously this wouldn’t hold up in court, but seriously Apple. The EULA should be something like: “You can virtualize OS X Client on Apple Xserve hardware” or something like that. Make the end user pay for it, but don’t make us hack software to make it work.

Anyway, the big thing that made me create this post is… VMware Fusion 3.0 changed things a bit. This info is still online except one minor tidbit. You HAVE to use physical media when doing the install. Don’t ask me why, you just do.

As for the rest of the info… I leave it to the readers. As for fixing the EULA… Please, Apple… Please.

UPDATE: The issue with VMware and physical media seems to be that VMware is convinced it should be attached to the VMware tools disk image to install that. If there was some way to get around that, it should be easy enough to boot off an image. Because once I had it installed, and unmounted the VMWare Tools “disk”, the OS X installer image immediately mounted. *shrugs* Oh well… I got it installed.

UPDATE 2: So a big reason for this was testing. After I got a normal 10.5 client installed, I created a second VMware disk, and restored our lab image onto it. It’s actually pretty cool to have this. Means I can finally test some things that normally would require a separate machine. So again, please Apple, fix the EULA. At the very least, work with VMware to allow Mac OS X Server (and client ideally) to run on vSphere. Hell, sell some Dongle for $10k. Or hell, get them to port vSphere to the Mac, so you can put a couple Xserves in a vSphere cluster, and they’ll take care of running the Mac OS instances. I want to be able to virtualize OS X on vSphere without jumping through hoops.

Looking around online, I found several instances of people wanting to chroot sftp on 10.5 server. The purpose being, they want to give access to sftp for users they may not trust, and want to keep them where they belong over sftp.

Unfortunately, there were a couple pieces missing from the instructions. So, thought I would fix that.

First, make a backup of /etc/sshd_config. While it should be easy enough to back out these changes, it’s just good practice to make a backup.

Second, create a directory for the “jail”. In my case, this was in /Volumes/Data/Websites/username.

The key here is that all directories up to and including the username directory must be read only by everyone but root when it comes to POSIX directories. So / would need to be root:group, and something like rwxr-xr-x. That goes for /Volumes, /Volumes/Data and /Volumes/Data/Websites.

The rest is all in the /etc/sshd_config

Comment out (with a #):

Subsystem sftp /usr/libexec/sftp-server

And add:

Subsystem sftp internal-sftp

At the end of the sshd_config, add:

Match User username
ChrootDirectory /Volumes/Data/Websites/username/
ForceCommand internal-sftp
AllowTcpForwarding no
X11Forwarding no

Or, if you want to enforce on a group:

Match Group usergroup
ChrootDirectory /Volumes/Data/Websites/
ForceCommand internal-sftp
AllowTcpForwarding no
X11Forwarding no

You can add both, and ssh will read from first to last. So, if you want specific users to go to specific folders, you can add them first, then end with a group policy.

Lastly, while testing this, make sure to watch /var/log/secure.log. You’ll see errors there when it doesn’t work. My problem, when working on this, was the write ability for users other than root on the parent directories. I had to systematically remove group and other write before it would work.

Those errors looked like:

fatal: bad ownership or modes for chroot directory component "/"

In the case of the root directory.

Lastly, this will remove SSH capability for the user specified. They will only be able to SFTP, but they’ll be locked into the directory specified. Great for a random student groups, in my case, that need to have a website, but you don’t necessarily want running wild on your system.

Since January of this year, I’ve been actively seeing AppleFileServer crash regularly on a server at work. This server is our primary student account server, which at any given time has about 40-80 students logged in (network home directories).

Many days, AFP crashes several times. Every time, it’s the same error: kern_protection_failure. The thread that crashes is always talking about ByteRangeLockTreeKey. The only good thing about this problem, is seemingly AFP comes back up, and people’s computers reconnect (go autofs!). But this is a very poor consolation prize since for some people, this does cause a problem (anyone with Mail open usually gets an error about not being able to access their inbox, and do they want to rebuild, or quit, and some others occasionally get Final Cut project file corruption (this is rare, and only seems to impact those that have their autosave vault set to their home directory, and not the local HD)).

So, Apple was notified about this, officially, on Jan 22nd, 2009. Ticket number 6517425. After getting back to me and asking for some follow up info, they proceeded to roll the ticket into another one (6237420). This ticket, apparently, was not related, and after telling our Sales Engineer about this, he had them un-merge the tickets. Apple then rolled my bug into another ticket, 5859645. An even older ticket! From what I’ve gathered, this ticket may be related to some lower level issue than AFP… either filesystem level (perhaps ACLs?!?, or even general I/O level).

All the while, I am in contact with someone in Minnesota who is having my same issue, and has also opened tickets (and has the luxury of having AppleCare for 10.5 server (the high end AppleCare to boot). He had two open case numbers with them. He even had a regional service engineer come by and take a look at this system, which he said was set up correctly, and there’s nothing more they could do to help alleviate the problem until a patch was available.

So, also during this time, someone from London contacts me and says he’s having the same issue as well, and has a Developer account (pay for), so he tries a beta of 10.5.7. It does not fix the issue. Around this time, I downgrade to 10.5.4 hoping the issue will be lessened (long story short, it isn’t). But, a few weeks later, the gent from London says he’s fixed his problem by removing the “deny all” acl from all his share points and folders within share points. The “deny all” acl was added around 10.5.4 or so to mitigate something… no one’s sure what. Anyway, he then tells Apple about this “fix” and they reply that it’s an “unacceptable workaround” and that they’re working on a fix. This was April 9th he did this.

Well, so, 10.5.7 dropped last Tuesday (May 12th, 2009). I installed it on the server experiencing the issue Friday night, at about 2am. I didn’t have a single crash until Sunday, May 17th, 2009, at 5:52pm. Same exact error.

So, not only was Apple notified AT LEAST 110 days prior to 10.5.7 shipping, but they were notified of an actual “fix” about 33 days before hand. I really wish Apple’s bug database was public, so that I could post links to my bugs, but, alas it is not.

However, here are a few threads on the issue:

http://www.afp548.com/forum/viewtopic.php?showtopic=23311
http://discussions.apple.com/thread.jspa?threadID=1975848
http://discussions.apple.com/thread.jspa?messageID=8857952

At this point, I’m going to start actively poking buttons and prodding people until I get an answer. The last email I sent to devbugs@apple.com resulted in the “pat”, “There is no new information at this time”. What a load of horse crap. They know of at least one “option”… the least they could do would be to educate someone having this issue about that “fix” and it’s repercussions. Given the amount of time that 10.5.7 took to hit the street, and how far in advance I notified them about this bug, I have very little hope this will get fixed before 10.6. If we’re lucky, we’ll see the fix back ported, but I doubt it.

To cap this all off, the main reason I’m posting this is for posterity, as well as the hope that anyone else that has this bug can actually see they’re not alone! And that they can contact Apple and say “hey, I have some bug numbers here of others having this issue”. If you are having this issue, please, don’t hesitate to contact me and I’ll work to get you in contact with others having this issue, or with someone at Apple that will actually listen.

UPDATE 1: Today I got a call from the local Education SE, who has created an escalation of this issue. Assuming it gets signed off by his boss, I should be hearing from Apple Engineering in the next few days… which is good since AFP crashed 5 times today. I have decided, in the interim, to remove the “group:everyone deny delete” ACL from many of the home folders on the server. Hopefully this will ease the problem. We’ll have to see. And I’ll post more once I hear from Engineering.