Everybody Staze…

While at the beach, I got an email from AppleCare basically saying they were closing my case because I had sent in an FYI saying things were working. They took that as a “issue resolved”. So, a few emails flew back and forth Monday, then I didn’t hear anything else. So, I sent a rather angry email Tuesday night and CC’d my Sales Engineer. That got some attention, and they reopened the case (there are other reasons too, but I NDA and all). The biggest pisser is the previous person I was working with was removed from the case, so now I’m dealing with someone else. Hopefully the miscommunication, and the rather guarded phone conversation on Friday are only due to his call being monitored… we’ll see. Anyway, they’ve said that they aren’t going to close the case until the issue is resolved in an official capacity. That was the crappy part of the week.

The better parts of the week were getting the new Xserve on Monday (I didn’t get to set it up until Wednesday). Once I got it in the rack, I can say honestly, it’s hella fast. I’ve had it running BOINC (setiathome) since Wednesday, and managed to get astropulse running on it, and it’s cranking through work units. It has 8 cores, with Hyperthreading. So, BOINC sees it as 16 cores. For some reason, it won’t use more than 8, so it’s running at 50% capacity, and still managing to blow the socks off of other machines I’ve used. It also uses less power than the older generation Xserves. It also chugs through encoding. While Handbrake doesn’t seem to take advantage of it, using MPEGStreamclip encoded an 8 minute video as H.264 in about a minute. Can’t wait until 10.6 rolls and we see GPU acceleration in Quicktime encoding.

Also got a new test server, in the form of a new Mac Mini. It’s pretty sweet for a Intel Core 2 Duo. It’s only 2.0ghz, but it seems to have Hyperthreading (VMWare sees 4 processors). I haven’t been able to get 10.5 Server Guest OS’s on it yet since we only got a Mini with 1gig of RAM. RAM should arrive tomorrow ($65 for 4gig of ram). I’m hoping to run 3 guest OS’s on there pretty regularly (one for Plone 3 development, one for Plone 2.5 testing (mirror of our current site), and one for 10.6 testing). What I did find pretty cool is that on at least Intel machines, or it may be 10.5, if you have 10.5 client installed, and put in a 10.5 server disk, you can click install, and go through the install without rebooting. After it’s installed, you run software update, it downloads the latest version of server, and then reboot. Boom, you have 10.5 server.

Last week I also finally implemented quotas on the SAN. I did this mainly because one of the LUNs on the SAN was down to about 10% free space (even though the SAN as a whole has about 40% available). This is due to the fact we only have 3 storage LUNs on the SAN. Ideally it should be even numbers. I’d love to buy new storage, but that’s about $15k we don’t really have. =/

Implementing quotas entitled a lot of steps. Since we didn’t have them enabled for the past 2 years, there were about 200 users that were over the quota I was setting. Rather than just setting them all to 4gig, and forcing them to delete work before they could function, I wrote up some scripting that would set those 200 users quotas to 1gig over what they were currently using, and then set a time limit to expire on Oct 15, 2009; at which time their quota would revert to 4gig. The script then sent out emails to those users stating what they were using, what their quota was, and that on 10/15/09, it would set down to 4gig. The more active users have already started clearing off data. I did receive an email from a user claiming they weren’t using that much… of course, I ran a `du` on their user directory, and showed that in fact, they were using that much data.

For all the other users that were under quota, I just set normal 4gig quotas. The only other thing I’m considering is to have the quotas for all those users who are over quota currently shrink down nightly so that it remains 1gig more than their current usage until it reaches 4gig. That way, they can’t delete stuff now, then fill back up the space again. But, it might be moot since they’ll be forced to be at 4gig come Oct 15th anyway.

Now, by Oct 15th, I need to have a script in place that grabs quota data from LDAP, and sets it on the SAN nightly. I also need to make sure all the LDAP quotas are set to 4gig. I’m not sure why Apple didn’t make Xsan look at LDAP for quota info (maybe a latency thing?), but it’s not that difficult to code up something that does that. Ideally, I want it to check the quotas already set, and only set those not already in place… then it won’t be writing 1600 quotas every night, but only at most, a few. I haven’t tested yet whether quotas can be set on the SAN for users that don’t have any data on the SAN… this would be nice so I could potentially set quotas before users have a chance to start writing data (then they won’t be able to go over quota). If I can implement this “diff quota”, I might be able to have the script run hourly rather than nightly. I did write a webpage where people can check their quota… I hope to improve it with a login, as well as a way to see what’s so big on their account.

That’s about all I have for the work week. It was a pretty productive week given that I was only there for 3 days, and one of the days was largely me playing with the new Xserve. Home related stuff here soon. Tara and I have been building something spiffy.

UPDATE
I am mistaken on the Hyperthreading ability of the 09 Mac Mini. It’s showing 4 processors in VMWare because I copied over the .app from a machine that did have 4 processors. I didn’t think this data would be stored in the .app, but I guess it is.

For the record, since it’s nearly impossible to find without other software, the ‘09 Mac Mini 2.0ghz, has a P7350 CPU. The details of which can be found here.

You’ll also note that the P7350 does NOT support VT. So, using the Mini as a VMware or Parallels host is probably not the best idea. It’s going to be slower than using something that does support VT. I can say it works, but it doesn’t work as well as machines that do have VT. =/

Today at work was fairly boring, save the fact that at 4:30am this morning, a drive died in one of the arrays. So it’s been rebuilding with the spare since then. Should be done by about 8pm tonight. It’s a 3.6TB array with 750gig PATA drives, so rebuilds take a while.

Thankfully, we have an extended service contract with the Apple on the array, so a replacement is free, and was shipped from either Portland or Seattle today. Should be able to replace it tomorrow. Turns out, these can be ordered from the campus repair shop, which is an Apple Authorized Service Center. Otherwise, Apple wanted to put a $1070 hold on my Credit Card until they got the old one back. =[

More work with ACE today. Again, can’t comment on much. But, progress is being made. =) Otherwise, no comment. =)

Today we also moved over a server from another department that we’re now hosting their hardware, which hosts their website(s). Pretty easy move. It’s a G5 tower, so the hope is to migrate it to something in the rack, or better, migrate it to our primary web server. Least we have the power and space in the room. Worst case, I might just put the tower on top of the rack.

Word on the street is the state legislature is going to approve our $7.5 million in bonds which will match the private donations we’ve gotten at work. This will give us a building expansion, remodel the remainder of the 2nd and 3rd floor, and replace all the windows with double glazed windows (our current windows are circa 1950, so the building leaks like a sieve, and dust builds up everywhere). Interesting part, the bonds would need to be used by the end of the 2009-2011 fiscal year. So, construction is probably going to need to start by next summer at the latest. Next summer and year is going to be… “fun”, but well worth it.

I’ve added a couple sections to the site. I hope to open up the Code section when I have something there, and to populate more of the “Plants” section. Basically I just excluded that category from the front page with a single line of php () in the post loop of the index_template.php (link). Here’s hoping. I have a couple other sections I need to start populating as well. Think I’ll do the same for the reviews section.

Also been playing with the iPhone Configuration Utility 2 for our faculty/staff that have iPhones. Overall, it’s extremely cool. We can create a template for each user, or just a generic one that sets up LDAP for an email directory, as well as set up a calendar server. Once 10.6 is out, we’re going to set up iCal Server 2, and get everyone onto that. Then we’ll finally be able to kill MeetingMaker. Once we can kill the Quark License Server (i.e. once we get rid of Quark), and we get rid of the APWire Satellite feed (which the software that parses the feed runs at 100% CPU, all the time), I might be able to basically decommission a server due to lack of services, rather than consolidation.

Alright, that’s it for now. Again, hope to post some about home in the next few days. But not a whole lot has gone on, so it’s kinda difficult to write about.

TTFN.

This weekend has been pretty good, and pleasantly (un)eventful. Got the yards mowed, pruned front tree (we call it “the ugly tree” because it leafs out late, drops leaves early, and has little leaflets that don’t rake up because they’re too small), a Gleditsia triacanthos inermis, or Thornless Honey Locust. The interesting thing is, this year is the first time the tree has flowered since we’ve lived here. So, it must be healthier than it has been. We’ll see if it sets any fruit. =/

I also planted some Glycine Max, or Edamame (variety Misono Green) in a raised bed. I planted 6 plants, hopefully I’ll get some nice Edamame out of it. =) I’d love to try making Tofu. Here’s some info on growing beans in the Valley: OSU Extension.

Work has wound down at this point. At the J-School, most projects are due dead week, and there are few finals. So, most students will be gone next week. Which means I should be able to start on my summer projects. I’ll probably start building a new image from scratch starting tomorrow.

Progress with AppleCare has kinda stalled. I ran some software they wanted, and got them more info. At this point, sounds like a 10.5 fix is unlikely. Hopefully we’ll see a fix in 10.6, but they can’t tell me one way or the other given Apple’s rather over zealous policies given unreleased products. We’ll see what happens. At this point, I’m looking at implementing NFS home directories as a short term “stop gap”. We’ll see how that goes… right now, I can’t get it to work at all.

Tomorrow is the WWDC keynote. My boss is betting we’ll see the new iPhone, the 3.0 iPhone OS, and possibly even 10.6 (or rather, a release date for 10.6). The rumor sites are saying a new beta build will be released to attendees… perhaps the last before GM. Who knows… I’d love to update to a new phone, since mine has been really laggy the past few weeks… but, I’m hoping 3.0 will help with that. I think the 2.x OS could use some help with memory reclamation after exiting an App. =/

The weather here is supposed to be fairly mild the next week. I’m hoping to get a bunch of yard work done before it warms up again. I’ve been trying to the last 2 years to remove the Morning Glory (Convolvulus sepium) from the yard. From reading online, it sounds like it’s nearly impossible. You just have to continually remove it. Unfortunately, it sounds like it’s pretty efficient at food storage, so just a few leaves can replenish food stores.

This is a pretty good PDF on Invasive Plants in the Willamette Valley: Remove Invasive Plants.

More later… maybe once I have some progress on NFS Home directories at work I’ll post again.

Since January of this year, I’ve been actively seeing AppleFileServer crash regularly on a server at work. This server is our primary student account server, which at any given time has about 40-80 students logged in (network home directories).

Many days, AFP crashes several times. Every time, it’s the same error: kern_protection_failure. The thread that crashes is always talking about ByteRangeLockTreeKey. The only good thing about this problem, is seemingly AFP comes back up, and people’s computers reconnect (go autofs!). But this is a very poor consolation prize since for some people, this does cause a problem (anyone with Mail open usually gets an error about not being able to access their inbox, and do they want to rebuild, or quit, and some others occasionally get Final Cut project file corruption (this is rare, and only seems to impact those that have their autosave vault set to their home directory, and not the local HD)).

So, Apple was notified about this, officially, on Jan 22nd, 2009. Ticket number 6517425. After getting back to me and asking for some follow up info, they proceeded to roll the ticket into another one (6237420). This ticket, apparently, was not related, and after telling our Sales Engineer about this, he had them un-merge the tickets. Apple then rolled my bug into another ticket, 5859645. An even older ticket! From what I’ve gathered, this ticket may be related to some lower level issue than AFP… either filesystem level (perhaps ACLs?!?, or even general I/O level).

All the while, I am in contact with someone in Minnesota who is having my same issue, and has also opened tickets (and has the luxury of having AppleCare for 10.5 server (the high end AppleCare to boot). He had two open case numbers with them. He even had a regional service engineer come by and take a look at this system, which he said was set up correctly, and there’s nothing more they could do to help alleviate the problem until a patch was available.

So, also during this time, someone from London contacts me and says he’s having the same issue as well, and has a Developer account (pay for), so he tries a beta of 10.5.7. It does not fix the issue. Around this time, I downgrade to 10.5.4 hoping the issue will be lessened (long story short, it isn’t). But, a few weeks later, the gent from London says he’s fixed his problem by removing the “deny all” acl from all his share points and folders within share points. The “deny all” acl was added around 10.5.4 or so to mitigate something… no one’s sure what. Anyway, he then tells Apple about this “fix” and they reply that it’s an “unacceptable workaround” and that they’re working on a fix. This was April 9th he did this.

Well, so, 10.5.7 dropped last Tuesday (May 12th, 2009). I installed it on the server experiencing the issue Friday night, at about 2am. I didn’t have a single crash until Sunday, May 17th, 2009, at 5:52pm. Same exact error.

So, not only was Apple notified AT LEAST 110 days prior to 10.5.7 shipping, but they were notified of an actual “fix” about 33 days before hand. I really wish Apple’s bug database was public, so that I could post links to my bugs, but, alas it is not.

However, here are a few threads on the issue:

http://www.afp548.com/forum/viewtopic.php?showtopic=23311
http://discussions.apple.com/thread.jspa?threadID=1975848
http://discussions.apple.com/thread.jspa?messageID=8857952

At this point, I’m going to start actively poking buttons and prodding people until I get an answer. The last email I sent to devbugs@apple.com resulted in the “pat”, “There is no new information at this time”. What a load of horse crap. They know of at least one “option”… the least they could do would be to educate someone having this issue about that “fix” and it’s repercussions. Given the amount of time that 10.5.7 took to hit the street, and how far in advance I notified them about this bug, I have very little hope this will get fixed before 10.6. If we’re lucky, we’ll see the fix back ported, but I doubt it.

To cap this all off, the main reason I’m posting this is for posterity, as well as the hope that anyone else that has this bug can actually see they’re not alone! And that they can contact Apple and say “hey, I have some bug numbers here of others having this issue”. If you are having this issue, please, don’t hesitate to contact me and I’ll work to get you in contact with others having this issue, or with someone at Apple that will actually listen.

UPDATE 1: Today I got a call from the local Education SE, who has created an escalation of this issue. Assuming it gets signed off by his boss, I should be hearing from Apple Engineering in the next few days… which is good since AFP crashed 5 times today. I have decided, in the interim, to remove the “group:everyone deny delete” ACL from many of the home folders on the server. Hopefully this will ease the problem. We’ll have to see. And I’ll post more once I hear from Engineering.