Everybody Staze…

Looking around online, I found several instances of people wanting to chroot sftp on 10.5 server. The purpose being, they want to give access to sftp for users they may not trust, and want to keep them where they belong over sftp.

Unfortunately, there were a couple pieces missing from the instructions. So, thought I would fix that.

First, make a backup of /etc/sshd_config. While it should be easy enough to back out these changes, it’s just good practice to make a backup.

Second, create a directory for the “jail”. In my case, this was in /Volumes/Data/Websites/username.

The key here is that all directories up to and including the username directory must be read only by everyone but root when it comes to POSIX directories. So / would need to be root:group, and something like rwxr-xr-x. That goes for /Volumes, /Volumes/Data and /Volumes/Data/Websites.

The rest is all in the /etc/sshd_config

Comment out (with a #):

Subsystem sftp /usr/libexec/sftp-server

And add:

Subsystem sftp internal-sftp

At the end of the sshd_config, add:

Match User username
ChrootDirectory /Volumes/Data/Websites/username/
ForceCommand internal-sftp
AllowTcpForwarding no
X11Forwarding no

Or, if you want to enforce on a group:

Match Group usergroup
ChrootDirectory /Volumes/Data/Websites/
ForceCommand internal-sftp
AllowTcpForwarding no
X11Forwarding no

You can add both, and ssh will read from first to last. So, if you want specific users to go to specific folders, you can add them first, then end with a group policy.

Lastly, while testing this, make sure to watch /var/log/secure.log. You’ll see errors there when it doesn’t work. My problem, when working on this, was the write ability for users other than root on the parent directories. I had to systematically remove group and other write before it would work.

Those errors looked like:

fatal: bad ownership or modes for chroot directory component "/"

In the case of the root directory.

Lastly, this will remove SSH capability for the user specified. They will only be able to SFTP, but they’ll be locked into the directory specified. Great for a random student groups, in my case, that need to have a website, but you don’t necessarily want running wild on your system.

So, along the lines of my previous post, today I decided to look into Flash Player performance on 10.5 clients with a 10.5.8 server, using network home directories.

I had noticed a couple days ago that users playing youtube videos got occasional skips and stops in the video. This is on a brand new Intel iMac, with 2gb of ram. It should easily be able to handle a non-full screen, non-HD youtube video. Doing a `fs_usage AppleFileServer | grep <username>` (where <username> is the username of the user logged in) on our AFP server showed a LOT of traffic to ~/Library/Preferences/Macromedia/Flash Player/… First, seems like Adobe hasn’t touched that part of the Flash player code, if it’s still writing to a “Macromedia” folder. Second, why the hell is it caching information in the Preferences folder?! That’s what ~/Library/Caches is for (which I already redirect). But, I know there’s been a lot of complaining in the past about Flash caching data in non-standard places that aren’t affected by clearing the browser cache.

So, given that amount of traffic it was producing, I figured that was probably the cause (what a leap!), so, I open up the NHR scripts I use, and add a redirect for that folder to the local HD, and install those on the client. Guess what? No more Flash Player skips or stops on network home users.

At this point, I’ve copied those changes out to all our clients, and dropped them into our DeployStudio workflow (for machines that we image between now and the next image update). On the plus side, this should result in a sizable decrease in server IO and network traffic in the case of a whole class watching Flash videos.

So, once again, shoddy programming. The only caveat I can think with this would be any install bases that are using Macromedia Dreamweaver (already checked Dreamweaver CS3, and it doesn’t write to that folder), or maybe Macromedia Director. If you’re running those, I’d recommend just redirecting the “Flash Player” folder within the Macromedia folder. But, I’m not positive those other programs write to that folder, it’s just a guess.

So, good luck.

UPDATE: So, a student came up to me today and said Pandora stopped working. Looking into the issue, it looks like the newest full release of Flash doesn’t really like this “hack”. So, after beating my head against the wall for 2 hours with this, I decided “what the hell” and installed the beta of Flash player that adobe just released (which is Intel only, even though it claims to be a Universal Binary in the installer, and the Plugin itself), and what do you know, it fixed the issue. So, my guess at this point is Adobe is just slowly updating Macromedia’s code, and they found/fixed this issue while fixing something else (since they don’t support Network Homes).

While at the beach, I got an email from AppleCare basically saying they were closing my case because I had sent in an FYI saying things were working. They took that as a “issue resolved”. So, a few emails flew back and forth Monday, then I didn’t hear anything else. So, I sent a rather angry email Tuesday night and CC’d my Sales Engineer. That got some attention, and they reopened the case (there are other reasons too, but I NDA and all). The biggest pisser is the previous person I was working with was removed from the case, so now I’m dealing with someone else. Hopefully the miscommunication, and the rather guarded phone conversation on Friday are only due to his call being monitored… we’ll see. Anyway, they’ve said that they aren’t going to close the case until the issue is resolved in an official capacity. That was the crappy part of the week.

The better parts of the week were getting the new Xserve on Monday (I didn’t get to set it up until Wednesday). Once I got it in the rack, I can say honestly, it’s hella fast. I’ve had it running BOINC (setiathome) since Wednesday, and managed to get astropulse running on it, and it’s cranking through work units. It has 8 cores, with Hyperthreading. So, BOINC sees it as 16 cores. For some reason, it won’t use more than 8, so it’s running at 50% capacity, and still managing to blow the socks off of other machines I’ve used. It also uses less power than the older generation Xserves. It also chugs through encoding. While Handbrake doesn’t seem to take advantage of it, using MPEGStreamclip encoded an 8 minute video as H.264 in about a minute. Can’t wait until 10.6 rolls and we see GPU acceleration in Quicktime encoding.

Also got a new test server, in the form of a new Mac Mini. It’s pretty sweet for a Intel Core 2 Duo. It’s only 2.0ghz, but it seems to have Hyperthreading (VMWare sees 4 processors). I haven’t been able to get 10.5 Server Guest OS’s on it yet since we only got a Mini with 1gig of RAM. RAM should arrive tomorrow ($65 for 4gig of ram). I’m hoping to run 3 guest OS’s on there pretty regularly (one for Plone 3 development, one for Plone 2.5 testing (mirror of our current site), and one for 10.6 testing). What I did find pretty cool is that on at least Intel machines, or it may be 10.5, if you have 10.5 client installed, and put in a 10.5 server disk, you can click install, and go through the install without rebooting. After it’s installed, you run software update, it downloads the latest version of server, and then reboot. Boom, you have 10.5 server.

Last week I also finally implemented quotas on the SAN. I did this mainly because one of the LUNs on the SAN was down to about 10% free space (even though the SAN as a whole has about 40% available). This is due to the fact we only have 3 storage LUNs on the SAN. Ideally it should be even numbers. I’d love to buy new storage, but that’s about $15k we don’t really have. =/

Implementing quotas entitled a lot of steps. Since we didn’t have them enabled for the past 2 years, there were about 200 users that were over the quota I was setting. Rather than just setting them all to 4gig, and forcing them to delete work before they could function, I wrote up some scripting that would set those 200 users quotas to 1gig over what they were currently using, and then set a time limit to expire on Oct 15, 2009; at which time their quota would revert to 4gig. The script then sent out emails to those users stating what they were using, what their quota was, and that on 10/15/09, it would set down to 4gig. The more active users have already started clearing off data. I did receive an email from a user claiming they weren’t using that much… of course, I ran a `du` on their user directory, and showed that in fact, they were using that much data.

For all the other users that were under quota, I just set normal 4gig quotas. The only other thing I’m considering is to have the quotas for all those users who are over quota currently shrink down nightly so that it remains 1gig more than their current usage until it reaches 4gig. That way, they can’t delete stuff now, then fill back up the space again. But, it might be moot since they’ll be forced to be at 4gig come Oct 15th anyway.

Now, by Oct 15th, I need to have a script in place that grabs quota data from LDAP, and sets it on the SAN nightly. I also need to make sure all the LDAP quotas are set to 4gig. I’m not sure why Apple didn’t make Xsan look at LDAP for quota info (maybe a latency thing?), but it’s not that difficult to code up something that does that. Ideally, I want it to check the quotas already set, and only set those not already in place… then it won’t be writing 1600 quotas every night, but only at most, a few. I haven’t tested yet whether quotas can be set on the SAN for users that don’t have any data on the SAN… this would be nice so I could potentially set quotas before users have a chance to start writing data (then they won’t be able to go over quota). If I can implement this “diff quota”, I might be able to have the script run hourly rather than nightly. I did write a webpage where people can check their quota… I hope to improve it with a login, as well as a way to see what’s so big on their account.

That’s about all I have for the work week. It was a pretty productive week given that I was only there for 3 days, and one of the days was largely me playing with the new Xserve. Home related stuff here soon. Tara and I have been building something spiffy.

UPDATE
I am mistaken on the Hyperthreading ability of the 09 Mac Mini. It’s showing 4 processors in VMWare because I copied over the .app from a machine that did have 4 processors. I didn’t think this data would be stored in the .app, but I guess it is.

For the record, since it’s nearly impossible to find without other software, the ‘09 Mac Mini 2.0ghz, has a P7350 CPU. The details of which can be found here.

You’ll also note that the P7350 does NOT support VT. So, using the Mini as a VMware or Parallels host is probably not the best idea. It’s going to be slower than using something that does support VT. I can say it works, but it doesn’t work as well as machines that do have VT. =/

Since January of this year, I’ve been actively seeing AppleFileServer crash regularly on a server at work. This server is our primary student account server, which at any given time has about 40-80 students logged in (network home directories).

Many days, AFP crashes several times. Every time, it’s the same error: kern_protection_failure. The thread that crashes is always talking about ByteRangeLockTreeKey. The only good thing about this problem, is seemingly AFP comes back up, and people’s computers reconnect (go autofs!). But this is a very poor consolation prize since for some people, this does cause a problem (anyone with Mail open usually gets an error about not being able to access their inbox, and do they want to rebuild, or quit, and some others occasionally get Final Cut project file corruption (this is rare, and only seems to impact those that have their autosave vault set to their home directory, and not the local HD)).

So, Apple was notified about this, officially, on Jan 22nd, 2009. Ticket number 6517425. After getting back to me and asking for some follow up info, they proceeded to roll the ticket into another one (6237420). This ticket, apparently, was not related, and after telling our Sales Engineer about this, he had them un-merge the tickets. Apple then rolled my bug into another ticket, 5859645. An even older ticket! From what I’ve gathered, this ticket may be related to some lower level issue than AFP… either filesystem level (perhaps ACLs?!?, or even general I/O level).

All the while, I am in contact with someone in Minnesota who is having my same issue, and has also opened tickets (and has the luxury of having AppleCare for 10.5 server (the high end AppleCare to boot). He had two open case numbers with them. He even had a regional service engineer come by and take a look at this system, which he said was set up correctly, and there’s nothing more they could do to help alleviate the problem until a patch was available.

So, also during this time, someone from London contacts me and says he’s having the same issue as well, and has a Developer account (pay for), so he tries a beta of 10.5.7. It does not fix the issue. Around this time, I downgrade to 10.5.4 hoping the issue will be lessened (long story short, it isn’t). But, a few weeks later, the gent from London says he’s fixed his problem by removing the “deny all” acl from all his share points and folders within share points. The “deny all” acl was added around 10.5.4 or so to mitigate something… no one’s sure what. Anyway, he then tells Apple about this “fix” and they reply that it’s an “unacceptable workaround” and that they’re working on a fix. This was April 9th he did this.

Well, so, 10.5.7 dropped last Tuesday (May 12th, 2009). I installed it on the server experiencing the issue Friday night, at about 2am. I didn’t have a single crash until Sunday, May 17th, 2009, at 5:52pm. Same exact error.

So, not only was Apple notified AT LEAST 110 days prior to 10.5.7 shipping, but they were notified of an actual “fix” about 33 days before hand. I really wish Apple’s bug database was public, so that I could post links to my bugs, but, alas it is not.

However, here are a few threads on the issue:

http://www.afp548.com/forum/viewtopic.php?showtopic=23311
http://discussions.apple.com/thread.jspa?threadID=1975848
http://discussions.apple.com/thread.jspa?messageID=8857952

At this point, I’m going to start actively poking buttons and prodding people until I get an answer. The last email I sent to devbugs@apple.com resulted in the “pat”, “There is no new information at this time”. What a load of horse crap. They know of at least one “option”… the least they could do would be to educate someone having this issue about that “fix” and it’s repercussions. Given the amount of time that 10.5.7 took to hit the street, and how far in advance I notified them about this bug, I have very little hope this will get fixed before 10.6. If we’re lucky, we’ll see the fix back ported, but I doubt it.

To cap this all off, the main reason I’m posting this is for posterity, as well as the hope that anyone else that has this bug can actually see they’re not alone! And that they can contact Apple and say “hey, I have some bug numbers here of others having this issue”. If you are having this issue, please, don’t hesitate to contact me and I’ll work to get you in contact with others having this issue, or with someone at Apple that will actually listen.

UPDATE 1: Today I got a call from the local Education SE, who has created an escalation of this issue. Assuming it gets signed off by his boss, I should be hearing from Apple Engineering in the next few days… which is good since AFP crashed 5 times today. I have decided, in the interim, to remove the “group:everyone deny delete” ACL from many of the home folders on the server. Hopefully this will ease the problem. We’ll have to see. And I’ll post more once I hear from Engineering.