So, I got the upgrade done, but it’s a mixed blessing.
We have 2 Xserve RAIDs, which each consist of 2 controllers, 7 drives each, except the one that’s for Metadata. So, that means we have 3 controllers for data, which means, basically 3 LUNs. Xsan really wants even numbers of LUNs. So, when I set up the “new” SAN, it stuck 2 LUNs in one “storage pool” and the other LUN in it’s own. Which blows. It means that Storage Pool 2 is filling faster than Storage Pool 1.
This was partially my fault, because I used the stupid wizard for “home folder storage” rather than custom. But, it doesn’t really tell you at any point what it’s going to do, it just bitches that “you really should have even numbers of LUNs”. Gar… Xsan 1.x didn’t do that crap… *sigh*. Basically, later in the summer I’ll have to do this again to fix the issue. Also will use this time to fix some other mistakes the wizard made… like, setting the stripe breadth too small at 64 blocks (should be, for a 2 LUN pool, equal to 128 blocks ( breadth = (transfer size (1M) / LUNs ) / block size (4k)). It set all of them to 64 blocks… the 1 LUN pool should be 256.
So, we’re on Xsan 2.1. Quotas work, but are off until the week after next (getting ready to send out the email to users that are using more than 4gig each). Yeah, interesting statistic… about 7% of users on our system (about 100 out of 1400 users), use nearly 40% of the storage. Cool huh! Small file storage does seem to perform better, so overall, things are better, but not as good as they could be.
Overall, the upgrade took all weekend. Copying stuff off took about 12 hours, then removing the SAN, and recreating it was quick, then copying stuff back took about 20 hours (which partially had to do with the split LUNs), and therefore decreased bandwidth. *double sigh*
On the positive side, the new fiscal year starts Tuesday. So, we can start spending money! We already have some stuff ordered… PCIe FC card for our FCS system, another copy of Xsan2, a really cool APC UPS serial multiplier (lets you hook the serial output of a UPS up to more devices (in this case, 8). That means I can finally hook the RAIDs up to the UPS, so they’ll be able to shut down gracefully incase of an outage.
On the super positive side, we are applying for a grant that would get us a Promise Vtrak array. That would be awesome because I could put some 15k SAS drives in it, and move MD to it…. as well as buy 7 750gig ATA drives, and fill up the current MD side of the Xserve RAID with drives, and have an even 4 LUNs)… which would give about 320MB/sec performance… would be pretty cool. The rest of the Promise would be for other stuff… not sure. Might be inclined to move user directories to it, and put video stuff on the Xserve RAIDs… but who knows.
