Everybody Staze…

UPDATE: Monday after a scheduled outage, I demoted my OD replicas to standalone (safer this way, I think), then ran `mkpassdb -kerberize` on the OD Master. About 5 minutes later, I had gone from 3500 Kerberos Principals to about 9500 (about 1500 of those are really old entries that I’ll clear out over the summer). I then added the replicas back. At that point, `kinit username` for previously failing users. We shall see.

UPDATE 2: Two days after the above, we have not seen any users having problems logging in. I will be talking to my AppleCare Enterprise friend tomorrow and seeing if he can shed some light on why AFP is trying to use Kerberos even though it’s supposed to only do “Standard” auth. More to come…

UPDATE 3: Well, that was nice while it lasted. Starting this week, on Monday, we started getting 2-3 users per day that couldn’t log in. Restarting AFP is the only way to get them logging in again. So, I’ve been doing that at about 6:45am each morning. So, I’ve got an open case with Apple at this point seeing what they can figure out. So far, it’s completely stumped them. We’ll have to see.

Starting with 10.5.7, I would occasionally see users (a small subset of users) that when they tried to login from a managed client (loginwindow, 10.5.8 client), they would get an error stating “You cannot login at this time because an error occurred”. If you then went to a computer that was unmanaged, and attempted to do a “Go-Connect to Server” and connect to the server over AFP, you would be presented with their home directory, only blank. Trying to connect over SMB would work, and everything was there.

The only way to make AFP work again would be to restart the AFP process. Obviously, this was really annoying, but I never could figure out the cause. Over the course of the summer break, we upgraded to 10.6 server, and didn’t see any instances of it.

Queue Fall term. We started seeing this problem the first week of the term, though slightly different. First, the clients are still 10.5.8 since we have about 36 PPC machines still in use (all G5 iMacs). Affected users would still get “Unable to login at this time because an error occurred”. So, I email a buddy that works for AppleCare Enterprise and forward on some log entries when it happens. Only things he sees are some IPv6 related messages (which is odd, since IPv6 is disabled), and maybe a Kerberos message… which, I don’t think much about at the time. Trying to connect to the server from “Go->Connect to Server” from an unmanaged client over AFP would result in a message saying “You do not have permission for any shares on this server”. Over SMB would result in seeing the shares, but trying to mount them would give you a permission denied error.

So, I go over my notes from the 10.5.x server days, and because it seemed to make things better with 10.5 server, I change AFP’s Authorization from “Any” to “Standard”. No change in results.

I bang my head against this for several days, trying many different options, but really don’t hit upon anything until an unrelated issue, where I am playing with some ACLs, and notice that if I “Deny” “Full Control” on a folder to a certain group, the folder disappears for that group. Not just “No access”, but it full on disappears. Huh. So many the issue is some kind of permissions thing. But, as my friend at AppleCare Enterprise mentions, the Effective Permissions Inspector (http://docs.info.apple.com/article.html?path=ServerAdmin/10.5/en/c2fs28.html) shows the permissions are fine for the user’s home folder. Okay…

So, I dig around some more, and randomly try “kinit” for an affected user. “kinit: Unable to acquire credentials for ‘user@REALM.EXAMPLE.COM’: Client not found in Kerberos database”. Hmmm. so I try for another affected user… same thing. I try it for all the users I’ve got records for having seen this issue. All are missing kerberos records. Well shit. So, I use kadmin to add a record for one of the users that’s seeing the problem (`kadmin -p admin -q addprinc user@REALM.EXAMPLE.COM` then type in the admin password, and their password twice). It adds, and after propagating, I can kinit. But, AFP still doesn’t work. Few hours later, I try AFP again, and I am allowed to mount their home, but it’s blank. Holy crap! Back to the 10.5.8 symptom. So obviously I’m getting somewhere. Later that night, I restart AFP, and suddenly the user account works perfectly. Ah ha!

K, so I get a list of all the kerberos principals on the server, ~3500. Hmm… given we have about 7600 users in the OD, that seems like a problem. But, after looking at most of the users that are seeing this, I find they’re all older user accounts. Meaning they were created when the OD Master was an older machine (G4 Xserve, or an old Quicksilver) running 10.3.9 or 10.4.x (depending on how old the accounts were). All the newer accounts seem to have Kerberos records. But, when we upgraded to 10.6 Server on the OD from 10.5, it seems ALL accounts got an attribute added that says “altSecurityIdentities: Kerberos:user@REALM.EXAMPLE.COM”. Hmm… I guess I could see this causing an issue.

So the question, other than “why do these users not have kerberos principals?” is “Why is AFP using Kerberos if it’s authorization is set to Standard?” This seems like a bug, or there’s something going on I’m not understanding. Obviously it seems the auth system in SMB changed a bit too between 10.5 and 10.6, since it used to behave differently.

Either way, I’ll be running “mkpassdb -kerberize” on the OD Master on Monday during our systems outage (there is a scheduled, 2 hour, power outage to test power resiliency on campus) (I already ran a test case on a test OD master, and it did add kerberos entries for all the users. So, that’s nice). This should hopefully resolve this issue permanently. I will update this post once I’ve kerberized all users, and things work, and I’ll update again later next week once I know whether or not it resolved the issue. I’m also expecting some info back from my friend about why this might be happening with AFP.

One thing I will say… this has really got me looking at Kerberos. Previous to this, I didn’t really use it at all on our systems. But since playing with it, it seems pretty damn cool. =)

Well, more in a few days.

So, been a few days, lots to update on.

We got the replacement drive a day late, but luckily, there were no problems. Put the new drive in, and it got marked as a spare. Kinda wish the Xserve RAID behaved like the Promises, in that the new drive would be rebuilt and the original spare would go back to being a spare, but, no big deal.

We ordered some hardware late last week. Mainly a new power supply for one server, and a new Xserve. Should be a lot of fun to play with. The Xserve has a Geforce GT120 in it, so stuff that uses OpenCL or CUDA should be able to take advantage of it. I’m looking forward to running BOINC on it when it comes in. =)

Thursday was a short day, as I got off early and headed to Walla Walla, WA for the long weekend of the 4th. We were there until Sunday. Long, hot, weekend with Tara’s family. Was nice, though kinda missed a couple of her cousins not showing up. The drive is really frickin’ long (about 350 miles), but it’s pretty. The Columbia River Gorge is, I think, one of the most beautiful places on earth. And although a lot of people seem to dislike the Windmills, I love them. They’re really “other worldly”, and graceful. Plus, they’re not polluting. There’s a coal plant out on that end of the state, and I really would love to see it removed/shutdown once there are windmills there to replace it.

Overall, good weekend.

Back at work today, it’s been fairly boring. Summer is rather slow this year. But, it’s letting us get some stuff done. I really am needed to code up some more stuff, and I’m slowly working myself out of a coders block. I just imported my car gas log into MySQL so I can update the average mileage on my site, as well as do some more nifty google API graphing. But, that’s a bit down the road. I’m really hoping I can make a nifty page for adding fuel tanks via my iPhone while I’m on the road. We’ll have to see. It’s been a long time since I’ve done any real MySQL work, but I’m relearning, and picking up new stuff this time. Hopefully there’ll be something on the sidebar pretty soon (probably tomorrow).

For work coding, I need to get quotas going… but that means I need to get something setup to email all the over quota people telling them they need to get stuff off the system. All and all, about 210 people (out of 1600) use about 50% of the space on our system (enforcing quotas for those users would free up 3TB of storage. About 6TB total is used right now).

Once I get them under control, I might even be able to do some type of backups of user data *shock*. Really not sure about that though.

On the AFP front, I’ll say that the current attempt to address the issue seems to be working. AFP has been running successfully for over 5 days now, and today we had about 45 users logged in (which is, I think, the highest usage this summer). I’m going to see if we can get a few users using Mail.app tomorrow to increase the load on the system, and see if that tickles any bugs.

I hope to add more plants to my list in the next few days. So far I’ve only added the one that I remember specifically where I got, etc.

Alright, that’s it for tonight. More later this week.

This weekend has been pretty good, and pleasantly (un)eventful. Got the yards mowed, pruned front tree (we call it “the ugly tree” because it leafs out late, drops leaves early, and has little leaflets that don’t rake up because they’re too small), a Gleditsia triacanthos inermis, or Thornless Honey Locust. The interesting thing is, this year is the first time the tree has flowered since we’ve lived here. So, it must be healthier than it has been. We’ll see if it sets any fruit. =/

I also planted some Glycine Max, or Edamame (variety Misono Green) in a raised bed. I planted 6 plants, hopefully I’ll get some nice Edamame out of it. =) I’d love to try making Tofu. Here’s some info on growing beans in the Valley: OSU Extension.

Work has wound down at this point. At the J-School, most projects are due dead week, and there are few finals. So, most students will be gone next week. Which means I should be able to start on my summer projects. I’ll probably start building a new image from scratch starting tomorrow.

Progress with AppleCare has kinda stalled. I ran some software they wanted, and got them more info. At this point, sounds like a 10.5 fix is unlikely. Hopefully we’ll see a fix in 10.6, but they can’t tell me one way or the other given Apple’s rather over zealous policies given unreleased products. We’ll see what happens. At this point, I’m looking at implementing NFS home directories as a short term “stop gap”. We’ll see how that goes… right now, I can’t get it to work at all.

Tomorrow is the WWDC keynote. My boss is betting we’ll see the new iPhone, the 3.0 iPhone OS, and possibly even 10.6 (or rather, a release date for 10.6). The rumor sites are saying a new beta build will be released to attendees… perhaps the last before GM. Who knows… I’d love to update to a new phone, since mine has been really laggy the past few weeks… but, I’m hoping 3.0 will help with that. I think the 2.x OS could use some help with memory reclamation after exiting an App. =/

The weather here is supposed to be fairly mild the next week. I’m hoping to get a bunch of yard work done before it warms up again. I’ve been trying to the last 2 years to remove the Morning Glory (Convolvulus sepium) from the yard. From reading online, it sounds like it’s nearly impossible. You just have to continually remove it. Unfortunately, it sounds like it’s pretty efficient at food storage, so just a few leaves can replenish food stores.

This is a pretty good PDF on Invasive Plants in the Willamette Valley: Remove Invasive Plants.

More later… maybe once I have some progress on NFS Home directories at work I’ll post again.

Friday I came back to work. It was fairly uneventful. The weekend proved to be hot, and resulted in a lot of indoor work on the laundry room, the garage, and the shed. Everything looks pretty darn good.

Today, I’ve been making more progress with Apple (AFP has been acting worse the last few days, but they’ve given me some things to try), and we’re starting to lay out plans for our summer work. It should be a fairly steady summer, I’m hoping.

This is a rather short post because I can’t really think of much to say. Oh! Comparing my TED to my Utility bill pretty much was dead on. Since I’m not tracking total KWH/day by the hour, and I don’t know when the utility meter reader was at the house, I can only assume that anything within about 1-2% between the bill and the TED is pretty darn close. Great to know! Now I just wish they had a “TED” for water usage.

Here’s hoping for some Thunderstorms this evening!

Have a good week. Hopefully I’ll post more.

So, after my last post, I figured I’d give an update on that issue as well as other projects for the coming weekend.

As to my last post, there’s cautiously good news. My Apple SE escalated the case to AppleCare Enterprise, who have been very good about gathering the needed data, and keeping me updated as to the status of the issue. So, this last Tuesday, after a weekend of running with 10.5.7 and having a really crappy day Tuesday of trying to make things work, I decided I would downgrade back to 10.5.4. Well, after 3 hours of wiping the two servers, and installing 10.5.4 and things looking good, the next day we still had issues. Better, but not by much. So, I heard from AppleCare Enterprise that day (Wednesday, which I took off), and I got him the info he needed that night, and Thursday. Today, seemingly, Engineering thinks they have a fix. Bad news is, it might not make it into 10.5 but rather 10.6. But, they’re going to try. So, goodish news on that front.

Also for the past week (since 10.5.7), we’ve been having issues with a program called KeyAccess. Basically, this program allows us to “key” an application, and install it on all the computers in the building, then a server piece basically keeps track of how many instances of the programs are running, and keeps that inline with how many licenses we own. It’s the best thing since sliced bread. Job would be extra impossible without it. So, 10.5.7 comes out, I install it in 4 labs, and things seem to work. Only, I only tested on Intel Macs. On the PPCs, KeyAccess doesn’t launch on the client, and therefore applications won’t run. So, I email the company that makes the software (Sassafras Software, Inc), and tell them what’s up, and we think we have a fix, until the next day, when it’s still happening. At that point, I email back, and they say they’ve also had word from another location having the same issue. So, at home I email back and forth with one of their people who is 3 hours ahead of me about the issue. I send some logs, and some ls output, and he basically says they’ll try to work on it in-house the next day. So, next day, I’m working with them, and they get some more info, and finally get the issue reproducible in house. Today, they send me a new build that seems to work. Say it’s a timing issue (not sure quite what they mean by that, whether it’s coming up before networking, or whether it’s not syncing with the server)… so, 2 days, bug fix. Gotta really love small companies (I’m guessing they have probably 12-15 people, at most).

On the home front, we planted a medium sized Daphne ordora ‘Marginata’, which is a “typical” winter daphne, in the space formerly occupied by the very unhealthy Rhododendron (which, I placed in a pot, and is doing much better now). Hopefully we’ll have some nice, knock you on your ass, daphne to smell come next February.

Blueberries are doing about the same, though they seem far less impacted by the warmer weather we’ve been having recently than they have in the past. Be that the mulch, or the older plants, I can’t say. But they seem VERY happy.

This weekend is probably really going to be a fair amount of cleaning the garage, yard work, and various household tasks. I’m going to weatherstrip the front door, maybe trim the door so it’ll clear a rug, which also means changing the threshold. I don’t think I’ll be doing the PRV, but I might try to at least dig the old one out so I can tackle it next week.

Oh, btw, I got some new pedals for my bike. They’re Nashbar (http://www.nashbar.com/) Highlander Pedals, which are rebranded Wellgo WAM-D10’s (Review: here). They’ve got a nice big platform for normal shoe riding (with great spikes to bite into your shoes) on one side, and MTB clipless on the other. So far, I’m very happy. Previously I was using the stock Shimano clipless pedals with plastic clipless platforms, which sucked. Now if I could just get my saddle to not kill me. Oh, and btw, the Nashbar version was $30. The Wellgo ones are $50.

BTW, my bike is a 2001 Bianchi Volpe. I’ve replaced the Saddle with a more comfortable one, the pedals (now), and the tires with some 28 x 700 Gatorskins (hard, but smooth and nearly puncture proof).

That’s all for now… maybe I’ll post more this weekend.