Mail.app does not like it when a mail server reports the NEXTUID to be greater than 2^31 (which is the limit for signed 32-bit integers). I could end this post here, but it’s worth explaining how I got here, and how I fixed it… just incase.
A few weeks back, a faculty member hit quota for his email. This isn’t unheard of, but for some reason, this faculty member hit quota in such a way as to cause Dovecot (the central mail server’s mail service) to rapidly increase the UIDs for his email. I’m guessing something like the following happened.
- User is near quota, and receives message
- Dovecot writes out message to Inbox, but lazily updates dovecot-uidlist
- Delivery of message causes quota full
- Dovecot goes to update uidlist, but can’t since the quota is full. Which causes a race condition where it tries repeatedly to write it out, and in the process, increasing the base UID each time
- After this repeats a few thousand times, the user gets below quota again, and we end up with a NEXTUID of 3829200751 (yes, 3.8 Billion).
Now, there are two things you can think about this. One, is “this is a dovecot problem”. Well, yes, but… they’ve warned users not to store the uidlist files on quota enforced volumes:
Dovecot can’t currently handle not being able to write the control files, so it will cause problems with filesystem quota. To avoid problems with this, you should place control files into a partition where quota isn’t checked. ((http://wiki2.dovecot.org/MailLocation/Maildir))
Also in defense of Dovecot, they had this in on their wiki (which I didn’t find until after I fixed this issue): http://wiki.dovecot.org/Clients/NegativeUIDs.
Then there’s, this is an Apple problem. Yes, it is. I’m sure some engineer honestly thought, probably rightly so, “Who the hell gets 2,147,483,648 emails. That’s 1 email per second, every second, for 68 years (((2^31)/60/60/24/365=68 years))”. And generally, they’d be right. The problem is, no other email client (Thunderbird, or even Outlook Express 6), had an issue with this. It could be they completely ignore what the server reports for NEXTUID… though I’m not sure how else they would know which message was which.
Mail.app however (on the Mac, iPad, iPhone, etc) all had the same issue. On the Mac, Mail.app was nice enough to throw this message in Console:
8/29/11 2:51:28 PM Mail[36105] An exception was thrown ‘*** -[NSMutableIndexSet addIndexesInRange:]: Range {3829200751, 1} exceeds maximum index value of NSNotFound – 1’ during invocation of -[LibraryIMAPStore openSynchronously].
Which was another clue that lead me to the UID issue. Notice the “Range” value is the same as what the first line of the dovecot-uidlist says: “3 V1189451504 N3829200751” ((the first number being the version format for the file, the second being the UIDValidity, which I’m not entirely sure what that is. And the last being the NEXTUID: UID to be given to the next message)). Also note that as unlikely as the above 2 billion emails is, to get 3.8 billion would be one email every second for ~120.5 years. Again, not really possible. But still, having this be limited to 31-bits seems silly. There are going to be cases where this number grows out of bounds (or rather, wraps to negative), as was the case with this user (looking at Info.plist for his Inbox showed UIDNEXT to be -465766559).
Anyway, enough with the forensics. The fix, as you can see from the link above, is to nuke the “dovecot-uidlist” files from the user’s account, and let Dovecot re-number everything. You also need to kill their imap index folder (in our system, this is ~/.imapidx). Do both, then load an email program to get the IMAP server to reindex. The user in question went from 3.8 billion, to just 11,700, which mail had no issue with. I hope to never see this issue again. It took far too long to fix, and I’m pissed that I couldn’t find that dovecot wiki article before I actually went through the trouble to “try” something that I wasn’t sure would work. But incase you run into it, this is the fix. I’ve filed a RADAR bug with Apple (#10042982). Here’s hoping they address the issue (Mail.app is 64-bit. Apparently this variable didn’t get the message. =P)
UPDATE (1/1/2012):I just saw this issue again with a user. Apparently Dovecot 2 does not fix this, though I’m not positive this is dovecot’s fault, so much as our setup is breaking one of the rules dovecot lays out, which is to not enforce quotas on the index files, or the uidlist file. =/
UPDATE (2/22/2016) 2:Just saw this issue again with a different user, but interestingly, Mac Mail.app works great. No problem at all with the uid. However, two iOS (9.2.1) devices do have an issue. They would download the headers of the email, but the message would just read “This message has no content”. Closing out of all Mail clients, then deleting the uidlist files, and .imapidx files, got the Dovecot server to regenerate them, and Mail started working again. For reference, the next uid before deletion was 3952913407 (well above a 32bit signed integer), and after was 1232. Odd. Still wish I could figure out the cause… but given Dovecot’s recommendation for these files to live on a non-quota enforced space, it seems unlikely they’d ever put much effort into tracking down the bug. Ah well.
