Everybody Staze…

UPDATE: Twitter tools 2.1 has been released, which fixes the issue described below. You may still need to run the code below to remove all the duplicates. Not sure if TT 2.1 does it automatically.

UPDATE 2: Looks like he did this by making the tw_id index “unique”. There may be other code changes elsewhere, but…

UPDATE 2b: Okay, so the “fix” is retarded. Basically, all they did was make the tw_id index “unique”. So, when Twitter Tools goes to add pre-existing tweets, the apache error log fills with errors about the index entry already existing. Rather than writing good code that just checks to see if the tw_id already exists, it just blindly tries to add it and fails when MySQL says “nope, already exists”. Heck, it COULD just only try to grab tweets posted since the last one downloaded. Either of these would be better than what it is doing. *sigh* If the code was better commented, I’d try to fix it myself… but my eyes cross when I look at it. =/

Quick post about an annoying bug with Twitter Tools.

There seems to be a bug with Twitter Tools 2.0 for Wordpress, and unfortunately, the creator seemed rather un-interested in fixing it (instead just directing me to his pay-for site for Wordpress support. =/).

Anyway, so, Twitter Tools, if you don’t know, keeps a database table of all the tweets you’ve written. It seems to check Twitter for posts each time it’s loaded, and if there’s a new tweet, it downloads it and adds it to the table.

Problem is, at some point, something changed (with Twitter I’m guessing), that made Twitter Tools re-grab ALL the tweets written by said user that are available on twitter’s site. In my case, that means it grabs 100+ tweets each time I tweet something and the page is loaded. So, the table in question quickly grows to a pretty large number of rows (I have about 475 tweets since I installed Twitter Tools, so the table should only be 475 rows. Instead, it had grown to 85,000 rows as of a few days ago).

Given the query used by Twitter Tools to show the latest tweets, looking at that many records can take a while. In my case, given my server, it took about 20 seconds (which slowed down my site by that much).

The query looks like this (this is assuming standard wordpress prefix, etc):

SELECT * FROM wp_ak_twitter WHERE tw_text NOT LIKE 'New blog post%' AND tw_text NOT LIKE '@%' GROUP BY tw_id ORDER BY tw_created_at DESC LIMIT 5;

Now, that’s a pretty slow query to run on 85k rows, but, not easily sped up. Easiest speed up, I guess, would be to add an index of the first 20 characters of “tw_text”, but that doesn’t really solve any problems, just speeds up the query. Adding an index for “tw_created_at” would help a bit too… but still, it’s 85k records, MANY (the vast majority) of which are duplicates.

First, are you impacted by this issue? Try this from the MySQL command line:

SELECT DISTINCT tw_id AS id,count(tw_id) AS idCount FROM wp_ak_twitter GROUP BY tw_id HAVING idCount > 1;

This should show you how many duplicates you have. If it comes back with nothing, then you’re good. But, I’d imagine you’ll be shocked by how many duplicates you have. Some of my tweets had around 500 duplicates.

So obviously, the first “fix” is to remove the duplicates. With some googling, I figured out the query looks like:

DELETE FROM wp_ak_twitter USING wp_ak_twitter,wp_ak_twitter AS vtable WHERE vtable.id > wp_ak_twitter.id AND vtable.tw_id = wp_ak_twitter.tw_id;

This basically creates a temporary table with the contents of wp_ak_twitter, then compares the row ID (which is unique, and a Primary Key), and then checks to see if the tweet ID is the same as any other rows. The first “where” basically keeps a row from matching itself. So, this will remove all duplicates and leave one original. After the above command, you’ll want to do:

OPTIMIZE TABLE wp_ak_twitter;

and

FLUSH TABLE wp_ak_twitter;

This will reduce the actual size of the table to what’s used by the remaining rows. Without this, the table will remain the same size as it was with all the duplicates.

But, this doesn’t really “fix” anything. It just removes the duplicates. The actual fix I haven’t quite finished yet. I’ll try to update this post in the next few days with a fix. But basically, it’s going to involve inserting a “SELECT” into the twitter tools code that checks for a tw_id before inserting the tweet.

Until then… good luck. Hopefully this will speed up your website if it’s been slow and you have twitter tools installed.

UPDATE: So there was one more issue going on with this. After re-reading all the Xsan 2.2 documentation, it indicates that the primary MDC should be either a replica, or the Master OD server. In my setup, the backup MDC is the Master, but the Primary MDC is only “Connected”. Apparently this doesn’t work right. So, I made the Primary a replica, and everything now works. So, while the below is true, I’d make sure the above is also true if you’re running Xsan.

So, as I talked talked about earlier, we recently updated to 10.6 server, and along with that, Xsan 2.2.1. Since then, we’d been seeing odd ACEs (Access Control Entries) on folders that are on the Xsan, on the 10.6 Servers (the 10.5 Server saw everything just fine). But, the 10.6 Servers would see many of the ACEs as FFFFEEEE-DDDD-CCCC-BBBB-AAAA82xxxxxx (where xxxxxx is a hex equivalent of something (seemingly not the user/group id).

Removing and reapplying the ACLs wouldn’t help. Some of the ACLs would show fine, but some no matter what would show up as the above. So obviously there is an issue with the client looking up the user/group associated with that ACL (yet 10.5 works).

The solution came to me a few days ago. As I said previously, our Open Directory server has been around for a while. It started life as a 10.1 or 10.2 server, and has been upgraded since that point to 10.6 now. Any several of the groups/users have stayed the same on this system since then. Which relates to some issues I had a while back with iCal server not working for our older users. Accounts/Groups back in the 10.2 days didn’t have a UUID created and assigned to them. I fixed this for the user accounts about 10 months ago with a script that generates UUIDs and adds them to the user record. But at the time, I didn’t think of it about the groups. Now I wish I had. Once I added GeneratedUIDs to the groups that didn’t have them, and then removed and re-added the ACEs, everything seems to have worked. We still have a couple that don’t resolve right visually, but access to the files seems to work fine, so no clue why that’s happening.

All and all, kind of an annoying issue. Apple really should have their upgrade from 10.x to 10.x check for users/groups that don’t have GeneratedUIDs add them to the record, since some people have thousands of users, and have been upgrading since the days before LDAP (NetInfo is what used to hold directory info).

Ah well. So, anyone having a similar issue, check the inspector in WGM for a GeneratedUID for the group/user in question. My script linked above should easily be able to be modified to add GUIDs for groups as well.

Just this last weekend, I upgraded our primary systems from 10.5.8 Server to 10.6.2 Server, and the Xsan to 2.2.1 from 2.1.1. All and all, everything went well, though there’s been an odd issue that arose.

Since the update, I’ve seen something like the following error every 2 hours on the 10.6 machines: “Dec 25 14:03:26 server DirectoryService[29]: Misconfiguration detected in hash ‘Global GID’ – see /Library/Logs/DirectoryService/DirectoryService.error.log for details”

You look in DirectoryService.error.log, and you find:
2009-12-25 14:03:26 PST - T[0x0000000104781000] - Group 'wheel' (/LDAPv3/od.example.com) - ID 0 - UUID 9E733C05-88DE-4F83-9E09-038A887F1327 - SID S-1-5-21-4096-2147483678-1391576524-1001
2009-12-25 14:03:26 PST - T[0x0000000104781000] - Group 'wheel' (/Local/Default) - ID 0 - UUID ABCDEFAB-CDEF-ABCD-EFAB-CDEF00000000 - SID S-1-5-21-4171259825-3059450906-1974363594-1001

This error is there for several system level groups: daemon, kmem, sys, wheel, etc. Basically, the OD clients are all complaining that there is a conflict between the local group “wheel”, and the “wheel” that exists in the directory. These accounts, seemingly, shouldn’t exist within the directory, as they’re local accounts that exist on all the OD clients.

So, at this point, I think I’m safe removing them from the directory at this point. Looking at an ldif dump of the directory, it shoes these groups were created in 2003, when I upgraded the directory server from 10.2 to 10.3 (Netinfo to LDAP).

All told, there are probably 15 of these groups. They all conflict with other groups on the local directories, or are antiquated and don’t need to exist on the directory.

UPDATE: I successfully removed all of these groups, and it seem to have resolved the error messages, and had no ill effects. So, if you’re getting a bunch of the above errors, check to make sure you don’t have some weird group sitting on your directory that’s conflicting with a local system group. In general, all the groups you create should start with UID 1000 or above. There are only a few that are supposed to exist on the directory (admin, staff, domainadmins, domaincomputers, domainusers… I think that’s it).

Good luck, and hopefully post again shortly after the new year and students return from vacation.

This post assumes the user is using a Macintosh running 10.5.x or greater. It will still work with Linux or other Unix OS’s, but you’ll need to do some digging to get the proper commands. This way is all GUI driven.

Assumptions:

• Have access to a computer running ≥10.5.x
• Have a self signed certificate your web server is using for SSL
• Have configured Wordpress to require SSL for XMLRPC and Login (using “define(‘FORCE_SSL_ADMIN’, true);”) (Admin over SSL).

Assuming all of this, the process is not too difficult.

The basic gist is that you need to create a Certificate Authority, and then sign your SSL web cert with this CA. Once you’ve done that, you can then take the CA certificate, and email it to your iphone, and install it there.

So basically, you need to create a CSR (Certificate Signing Request) from your SSL cert you’re using on your webserver. If you’re using Mac OS Server 10.4 or greater, you can do this in Server Admin very easily. If you’re not, then you need to do some command line magic.

Something like this here: http://sial.org/howto/openssl/csr/

Once you have that CSR, you then can go into Keychain Access on your mac, and under the “Keychain Access” menu, go to “Certificate Assistant” and say “Create a Certificate Authority…”. Walk through that wizard, and most of the defaults are fine. You’ll want to put in your name, email, etc.

Once that’s done, you can go back into Certificate Assistant, and say “Create a Certificate for Someone Else as a Certificate Authority…”. This basically will sign your certificate. It’s like a self-signed certificate, only you actually have the CA cert to “trust”. Walk through that, though you may have to change the defaults for what the certificate is for.

Once you’ve done that, you’ll need to use this new certificate, as your SSL certificate. You do that by exporting the SSL cert as a .pem, and then replacing your public SSL cert with this one. Restart your web server, and you should have a signed cert. Assuming you trusted your own CA, you also shouldn’t get any warning about it being untrusted.

Now, once that’s done, you can export the public CA cert, as a .crt, and email than to your iphone. Once you get that, you can click it, choose to install the profile, and you’re done. The wordpress app should now work.

Please contact me if none of this makes sense, or if you need additional info. I was going to put up screen shots, but that just delayed me posting this. Which I wanted to get done.

Have a great rest of the week, and weekend!