Everybody Staze…

While at the beach, I got an email from AppleCare basically saying they were closing my case because I had sent in an FYI saying things were working. They took that as a “issue resolved”. So, a few emails flew back and forth Monday, then I didn’t hear anything else. So, I sent a rather angry email Tuesday night and CC’d my Sales Engineer. That got some attention, and they reopened the case (there are other reasons too, but I NDA and all). The biggest pisser is the previous person I was working with was removed from the case, so now I’m dealing with someone else. Hopefully the miscommunication, and the rather guarded phone conversation on Friday are only due to his call being monitored… we’ll see. Anyway, they’ve said that they aren’t going to close the case until the issue is resolved in an official capacity. That was the crappy part of the week.

The better parts of the week were getting the new Xserve on Monday (I didn’t get to set it up until Wednesday). Once I got it in the rack, I can say honestly, it’s hella fast. I’ve had it running BOINC (setiathome) since Wednesday, and managed to get astropulse running on it, and it’s cranking through work units. It has 8 cores, with Hyperthreading. So, BOINC sees it as 16 cores. For some reason, it won’t use more than 8, so it’s running at 50% capacity, and still managing to blow the socks off of other machines I’ve used. It also uses less power than the older generation Xserves. It also chugs through encoding. While Handbrake doesn’t seem to take advantage of it, using MPEGStreamclip encoded an 8 minute video as H.264 in about a minute. Can’t wait until 10.6 rolls and we see GPU acceleration in Quicktime encoding.

Also got a new test server, in the form of a new Mac Mini. It’s pretty sweet for a Intel Core 2 Duo. It’s only 2.0ghz, but it seems to have Hyperthreading (VMWare sees 4 processors). I haven’t been able to get 10.5 Server Guest OS’s on it yet since we only got a Mini with 1gig of RAM. RAM should arrive tomorrow ($65 for 4gig of ram). I’m hoping to run 3 guest OS’s on there pretty regularly (one for Plone 3 development, one for Plone 2.5 testing (mirror of our current site), and one for 10.6 testing). What I did find pretty cool is that on at least Intel machines, or it may be 10.5, if you have 10.5 client installed, and put in a 10.5 server disk, you can click install, and go through the install without rebooting. After it’s installed, you run software update, it downloads the latest version of server, and then reboot. Boom, you have 10.5 server.

Last week I also finally implemented quotas on the SAN. I did this mainly because one of the LUNs on the SAN was down to about 10% free space (even though the SAN as a whole has about 40% available). This is due to the fact we only have 3 storage LUNs on the SAN. Ideally it should be even numbers. I’d love to buy new storage, but that’s about $15k we don’t really have. =/

Implementing quotas entitled a lot of steps. Since we didn’t have them enabled for the past 2 years, there were about 200 users that were over the quota I was setting. Rather than just setting them all to 4gig, and forcing them to delete work before they could function, I wrote up some scripting that would set those 200 users quotas to 1gig over what they were currently using, and then set a time limit to expire on Oct 15, 2009; at which time their quota would revert to 4gig. The script then sent out emails to those users stating what they were using, what their quota was, and that on 10/15/09, it would set down to 4gig. The more active users have already started clearing off data. I did receive an email from a user claiming they weren’t using that much… of course, I ran a `du` on their user directory, and showed that in fact, they were using that much data.

For all the other users that were under quota, I just set normal 4gig quotas. The only other thing I’m considering is to have the quotas for all those users who are over quota currently shrink down nightly so that it remains 1gig more than their current usage until it reaches 4gig. That way, they can’t delete stuff now, then fill back up the space again. But, it might be moot since they’ll be forced to be at 4gig come Oct 15th anyway.

Now, by Oct 15th, I need to have a script in place that grabs quota data from LDAP, and sets it on the SAN nightly. I also need to make sure all the LDAP quotas are set to 4gig. I’m not sure why Apple didn’t make Xsan look at LDAP for quota info (maybe a latency thing?), but it’s not that difficult to code up something that does that. Ideally, I want it to check the quotas already set, and only set those not already in place… then it won’t be writing 1600 quotas every night, but only at most, a few. I haven’t tested yet whether quotas can be set on the SAN for users that don’t have any data on the SAN… this would be nice so I could potentially set quotas before users have a chance to start writing data (then they won’t be able to go over quota). If I can implement this “diff quota”, I might be able to have the script run hourly rather than nightly. I did write a webpage where people can check their quota… I hope to improve it with a login, as well as a way to see what’s so big on their account.

That’s about all I have for the work week. It was a pretty productive week given that I was only there for 3 days, and one of the days was largely me playing with the new Xserve. Home related stuff here soon. Tara and I have been building something spiffy.

UPDATE
I am mistaken on the Hyperthreading ability of the 09 Mac Mini. It’s showing 4 processors in VMWare because I copied over the .app from a machine that did have 4 processors. I didn’t think this data would be stored in the .app, but I guess it is.

For the record, since it’s nearly impossible to find without other software, the ‘09 Mac Mini 2.0ghz, has a P7350 CPU. The details of which can be found here.

You’ll also note that the P7350 does NOT support VT. So, using the Mini as a VMware or Parallels host is probably not the best idea. It’s going to be slower than using something that does support VT. I can say it works, but it doesn’t work as well as machines that do have VT. =/

Well, today was “fun”.

So, first, some backstory.

Last month, my boss worked out a deal where we would take over management of a server for another group on campus, which hosts 2 websites, and a file share (shared over AFP). We took possession of that machine, a Power Mac G5, on the 30th. I hooked it into the KVM, network, and secured the firewall and services on it. And just yesterday, got backups working fully.

Okay, so, if you don’t see where this is going….

First thing in my inbox that isn’t junk is a message about one of the websites on this box being down. I take a look, and indeed, the system is hung. Reboot results in the system being responsive for about 30 seconds before hanging again. Booting to Single User Mode (SUM) and fdisk’ing results in errors along the lines of:

AppleATADiskQueueManager: setPowerState (blah blah) timed out after 10000 ms.

Great. So, I fiddle some more, and then pull the drive. Lo and Behold, it’s a Seagate ST31000340AS, with Firmware SD15. If you don’t know about this problem, see the interwebs. Basically, Seagate shipped a pretty large batch of drives with bad firmwares that resulted in drive failure. In some cases, the failure did not impact the data, and simply updating to the new firmware restored the drive. I was not so lucky. After updating the firmware on a PC I keep handy, the drive did the same thing.

So, I finally got the machine up and running somewhat without a network connection. Just long enough to do a mysql dump on the system. Once that was done, I pulled the drive, hooked it to an external firewire -> sata controller (WiebeTech UltraDock, and tried to grab data off the drive. This mainly required powering on the drive, copying some files until things started failing, then cycling off, back on, and doing some more. Yes, the backup had not happened yet, since there’s a weekly full, then nightly incrementals. *sigh* But, luckily the former owners had a backup (clone) from the machine on the 29th of last month. Right before we took possession.

So, between those two sources, and the mysql dump that did work, I managed to move the websites over to our main web server, which is much more capable, has a RAID1 drive (duh?), and the websites mainly exist on the SAN. Oh, and it’s backed up quite regularly.

So, about 5 hours of work and the sites were back up. And I moved their file share (which thankfully had not had any changes happen since the 29th) to another server (I don’t like running file services on the webserver, for obvious reasons).

That’s really about it. I have some basic (as in, same functionality as the badge had) for the Gas mileage. I’m hoping to add a lot more functionality in the next week or so.

And as a parting statement… I hate the new title of the SciFi channel (SyFy). It’s ridiculous. I guess they spell things “Syence Fyction?”

So, been a few days, lots to update on.

We got the replacement drive a day late, but luckily, there were no problems. Put the new drive in, and it got marked as a spare. Kinda wish the Xserve RAID behaved like the Promises, in that the new drive would be rebuilt and the original spare would go back to being a spare, but, no big deal.

We ordered some hardware late last week. Mainly a new power supply for one server, and a new Xserve. Should be a lot of fun to play with. The Xserve has a Geforce GT120 in it, so stuff that uses OpenCL or CUDA should be able to take advantage of it. I’m looking forward to running BOINC on it when it comes in. =)

Thursday was a short day, as I got off early and headed to Walla Walla, WA for the long weekend of the 4th. We were there until Sunday. Long, hot, weekend with Tara’s family. Was nice, though kinda missed a couple of her cousins not showing up. The drive is really frickin’ long (about 350 miles), but it’s pretty. The Columbia River Gorge is, I think, one of the most beautiful places on earth. And although a lot of people seem to dislike the Windmills, I love them. They’re really “other worldly”, and graceful. Plus, they’re not polluting. There’s a coal plant out on that end of the state, and I really would love to see it removed/shutdown once there are windmills there to replace it.

Overall, good weekend.

Back at work today, it’s been fairly boring. Summer is rather slow this year. But, it’s letting us get some stuff done. I really am needed to code up some more stuff, and I’m slowly working myself out of a coders block. I just imported my car gas log into MySQL so I can update the average mileage on my site, as well as do some more nifty google API graphing. But, that’s a bit down the road. I’m really hoping I can make a nifty page for adding fuel tanks via my iPhone while I’m on the road. We’ll have to see. It’s been a long time since I’ve done any real MySQL work, but I’m relearning, and picking up new stuff this time. Hopefully there’ll be something on the sidebar pretty soon (probably tomorrow).

For work coding, I need to get quotas going… but that means I need to get something setup to email all the over quota people telling them they need to get stuff off the system. All and all, about 210 people (out of 1600) use about 50% of the space on our system (enforcing quotas for those users would free up 3TB of storage. About 6TB total is used right now).

Once I get them under control, I might even be able to do some type of backups of user data *shock*. Really not sure about that though.

On the AFP front, I’ll say that the current attempt to address the issue seems to be working. AFP has been running successfully for over 5 days now, and today we had about 45 users logged in (which is, I think, the highest usage this summer). I’m going to see if we can get a few users using Mail.app tomorrow to increase the load on the system, and see if that tickles any bugs.

I hope to add more plants to my list in the next few days. So far I’ve only added the one that I remember specifically where I got, etc.

Alright, that’s it for tonight. More later this week.

Today at work was fairly boring, save the fact that at 4:30am this morning, a drive died in one of the arrays. So it’s been rebuilding with the spare since then. Should be done by about 8pm tonight. It’s a 3.6TB array with 750gig PATA drives, so rebuilds take a while.

Thankfully, we have an extended service contract with the Apple on the array, so a replacement is free, and was shipped from either Portland or Seattle today. Should be able to replace it tomorrow. Turns out, these can be ordered from the campus repair shop, which is an Apple Authorized Service Center. Otherwise, Apple wanted to put a $1070 hold on my Credit Card until they got the old one back. =[

More work with ACE today. Again, can’t comment on much. But, progress is being made. =) Otherwise, no comment. =)

Today we also moved over a server from another department that we’re now hosting their hardware, which hosts their website(s). Pretty easy move. It’s a G5 tower, so the hope is to migrate it to something in the rack, or better, migrate it to our primary web server. Least we have the power and space in the room. Worst case, I might just put the tower on top of the rack.

Word on the street is the state legislature is going to approve our $7.5 million in bonds which will match the private donations we’ve gotten at work. This will give us a building expansion, remodel the remainder of the 2nd and 3rd floor, and replace all the windows with double glazed windows (our current windows are circa 1950, so the building leaks like a sieve, and dust builds up everywhere). Interesting part, the bonds would need to be used by the end of the 2009-2011 fiscal year. So, construction is probably going to need to start by next summer at the latest. Next summer and year is going to be… “fun”, but well worth it.

I’ve added a couple sections to the site. I hope to open up the Code section when I have something there, and to populate more of the “Plants” section. Basically I just excluded that category from the front page with a single line of php () in the post loop of the index_template.php (link). Here’s hoping. I have a couple other sections I need to start populating as well. Think I’ll do the same for the reviews section.

Also been playing with the iPhone Configuration Utility 2 for our faculty/staff that have iPhones. Overall, it’s extremely cool. We can create a template for each user, or just a generic one that sets up LDAP for an email directory, as well as set up a calendar server. Once 10.6 is out, we’re going to set up iCal Server 2, and get everyone onto that. Then we’ll finally be able to kill MeetingMaker. Once we can kill the Quark License Server (i.e. once we get rid of Quark), and we get rid of the APWire Satellite feed (which the software that parses the feed runs at 100% CPU, all the time), I might be able to basically decommission a server due to lack of services, rather than consolidation.

Alright, that’s it for now. Again, hope to post some about home in the next few days. But not a whole lot has gone on, so it’s kinda difficult to write about.

TTFN.

Work has been going fairly smoothly since Spring term ended earlier this month (Jun 13th). Since that time, usage has been way down on the servers, so AFP has been crashing a lot less (once every few days). I’ve been having continued progress with Applecare Enterprise on the issue, but so far, looking at leaked Seed notes from 10.5.8, it doesn’t look like it’s addressed. I really can’t comment outside of that.

All and all, I’ve been spending a LOT of time documenting various things at work. From our Xsan topology, port layouts, service topologies, backups, etc. This has been something I’ve needed to do for a while, and going on vacation week before last really made me realize that if I’m out of cell range, and something breaks, work is kinda boned. Needless to say what would happen should I get hit by a bus.

To do the documentation, I’ve installed dokuwiki at work to handle it. Plone just doesn’t accomplish the collaborative aspect as well as a wiki, and Dokuwiki is just so damn simple, and yet extremely flexible. We used it briefly at work before Plone as an eportfolio experiment. So I knew it easily integrated into LDAP, which was necessary, and it stores all it’s data as flat files, making backups and moving stuff around a breeze. So, that’s pretty much taken up a lot of my time.

I would really like to get a code repository of some type up on this site in the near future to keep track of my various scripting/coding projects, and in a way, force myself to go back over that code, and comment it, as well as clean it up. I have so many scripts I’ve written at some point that I use once or twice, then forget about.

*sigh* svn is easy to set up, but I don’t have a ton of experience with it. But, I’m giving it a shot.

This week at work is short (holiday), and I’m going to leave early on Thursday. Because the Fiscal year starts Wednesday, we can start ordering hardware tomorrow. Basically, as long as we don’t receive the hardware until Wednesday or later, it’ll go onto the next fiscal year budget. So, here comes a new Mac Mini, and a Nehalem Xserve. And an extra PS for one of our other servers (we accidentally ordered it with only one). Once I have that server, I can update our ldap server, and secondary metadata controller (I’m merging the two together, since the secondary MDC never does anything anyway, and it’s actually how Apple suggests with smaller setups). The Mini is going to become a test server, which will run VMWare Fusion, and allow us to easily test different things (like a test web server, development web server, linux testing, windows web testing, and something to run OS seeds on).

Can’t wait to try out the Nehalem 8 core server (which has hyper-threading, so shit will see it as 16 cores). Handbrake, or better, Xgrid will be just awesome (Xgrid sees all 16 cores). 2.26ghz x 16 = 36ghz. From one server. crazy! That with the GeForce GT 120, should give us crazy performance. Hell, I’m just going to run SETI on it for a week. =)

So, other than waiting for all of that, this week will be a lot more documentation, and writing more of the quota reporting website. I need to implement quotas over the summer, which means just turning on quotas for those under, but some more special stuff for those that are over. Sending emails, setting soft and hard quotas, and timelimits. Hope to get that mainly done this week… at least the basic website functionality for students, and enabling quotas for those that are under quota. Once I get the secondary MDC upgraded, I’m hoping to turn on Spotlight on the SAN.

That’s all for now. More later when I get SVN up, or when I have more to talk about in some other way. I might post a “home” update in the next day or so.