[besson3c]

We work alongside of a group that runs Active Directory Servers, and they've just come to discover that many many Macs on campus have been running 4 simultaneous LDAP queries starting with a wildcard, bringing the AD Servers completely to their knees and disrupting service across campus.

This is a horrendous bug on Apple's part. I believe there is a fix for this now, but how this could have passed their testing in the first place is beyond me.

Secondly, all of the new iPhones on campus are also causing a huge increase in loads in constant polling of the IMAP servers, including each individual folder in order to get a message count (rather than keeping a tally). I believe that this horrible design also affects OS X Mail on the Desktop. When I used to use OS X Mail my client would repeatedly take forever to crunch "updating unread message count" even on archived folders that haven't received new messages in years. It's a shame that the brain dead quirks of OS X Mail seem to have been inherited by the iPhone.

Apple's interfaces are great and all, but they really seem to have difficulty with getting their clients to operate in a sane way with the servers they communicate with.

Anybody else experience or hear about these sorts of problems?

[larkost]

The 4 queries are not not a bug. That is Apple Remote Desktop asking the servers if they have ARD groups setup to manage them. You can turn off this feature in ARD. I do think that the ARD group needs to better craft that one, but it is not something you can just refer to as a "bug". It works well until you get beyond a certain size of AD domain, and Apple is not known for having a large AD domain. And it sounds like you have ARD turned on in the first place... so you are already configuring it.

And I believe that if your IMAP servers are halfway decent (Exchange server is an awful IMAP server), then they would support IMAP IDLE, and the polling goes away. And I have no idea what you are talking about "keeping a tally". That does not fit my understanding of IMAP. If your servers don't support IDLE, then the clients are not going to have any way of telling when a folder has been updated other than asking for a count.

If you are using Exchange servers, then you should move the iPhones to use ActiveSync. That will reduce the load enormously because Exchange is good at that.

[besson3c]

Originally Posted by larkost 

The 4 queries are not not a bug. That is Apple Remote Desktop asking the servers if they have ARD groups setup to manage them. You can turn off this feature in ARD. I do think that the ARD group needs to better craft that one, but it is not something you can just refer to as a "bug". It works well until you get beyond a certain size of AD domain, and Apple is not known for having a large AD domain. And it sounds like you have ARD turned on in the first place... so you are already configuring it.

And I believe that if your IMAP servers are halfway decent (Exchange server is an awful IMAP server), then they would support IMAP IDLE, and the polling goes away. And I have no idea what you are talking about "keeping a tally". That does not fit my understanding of IMAP. If your servers don't support IDLE, then the clients are not going to have any way of telling when a folder has been updated other than asking for a count.

If you are using Exchange servers, then you should move the iPhones to use ActiveSync. That will reduce the load enormously because Exchange is good at that.

The 4 queries part of it isn't the bigger part of the bug, it's the recursive wild card search that rapes the entire AD structure looking for whatever it is it is looking for. I was not the one that configured the Mac workstations, so I don't know if they have ARD on or not.

We are running Cyrus, this is my group. There are a lot of misconceptions over IMAP IDLE. IMAP IDLE does not magically receive notifications of when new messages have arrived, it just holds connections open. It still polls every x number of minutes.

What I meant by keeping a tally is OS X Mail's constant amnesia of what folders have received new messages, and it's constant repolling when it isn't necessary. I don't own an iPhone, but I assume that the same problems exist there. We are still investigating this problem, but we have found that the iPhones are definitely increasing load far more than they should, and our logs are showing more raping going on that other clients do not exhibit, with the possible exception of OS X Mail. Every refresh causes the iPhone to scour through all folders. The fix for this is providing a setting for which folders should be checked in the case of shared folders (although not even OS X Mail offers this feature), only polling the Inbox every x minutes like OS X Mail does, and fix whatever bugs Mail has that makes it thinks it needs to update unread counts on random folders for no particular reason.

We host about 160,000 email accounts or so, but most students are using the web based email and not OS X Mail, which is why this wasn't a huge problem before, but now that everybody and their dog on campus has an iPhone, this is becoming an issue.

[goMac]

Either Exchange or 10.6's mail server would fix these problems.

Honestly, there is no magic bullet besides your students turning off automatic polling. Automatic polling is going to check folders to see if they have new messages. Either that or buy a few more servers.

[besson3c]

Isn't Apple just using Cyrus as their IMAP server in OS X Server? Regardless, you aren't solving this problem by throwing more resources at the problem in the way of software and hardware on the server end of things, the problem needs to be solved at the iPhone software level. The problem isn't with the number of users hitting the IMAP servers, it handles stateless Webmail fetch requests over an IMAP proxy and countless numbers of people using other standalone clients without these sorts of load increases even at peak hours. All you have to do to see the poor design of the iPhone client is to monitor the IMAP server logs to see what it is actually doing compared to other clients.

You're right though, there is nothing we can do, but this and the ADS thing sure does have a lot of people around here grumbling about Apple. I was really excited about the iPhone 3G working in the enterprise, and to their credit a lot of things work great - VPN, Exchange, etc. As always, it's the little details like developing a mail client that works in a sane way performance wise that always seem to elude Apple.


Edit: ahh, I guess you mentioned 10.6 Server because Apple is probably adding push support to whatever server implementation they are using. Yes, that will work, but I sure hope that Apple fixes these problems for IMAP servers that will not work with Apple's push extensions. I also hope that Apple's push extensions will be some sort of open source product that will work in other server implementations too.

[CatOne]

OS X's mail server is Cyrus for the IMAP store, and Postfix for the MTA.

That said, how often are each of the clients polling the IMAP server? Use a policy to set it to 15 minutes or less. Or do they have "push" turned on on the phone? In the latter case, I'd guess after say 2 PM it's not an issue as the iPhones run out of battery by then when push is turned on :-P

[besson3c]

Originally Posted by CatOne 

OS X's mail server is Cyrus for the IMAP store, and Postfix for the MTA.

That said, how often are each of the clients polling the IMAP server? Use a policy to set it to 15 minutes or less. Or do they have "push" turned on on the phone? In the latter case, I'd guess after say 2 PM it's not an issue as the iPhones run out of battery by then when push is turned on :-P

Unfortunately we do not have the luxury of being able to micromanage iPhone configs. We are not forcing the use of enterprise profiles or anything like that for the 160,000 students and staff who are using the IMAP servers (and I have no idea at this point how many of these users are iPhone users, we haven't crunched these numebrs yet) - it is just plain old RFC compliant IMAP. Besides, getting this many users to do anything is nearly impossible

[mduell]

Originally Posted by besson3c 

There are a lot of misconceptions over IMAP IDLE. IMAP IDLE does not magically receive notifications of when new messages have arrived, it just holds connections open. It still polls every x number of minutes.

What RFC are you reading? 2177 says:

The IDLE command may be used with any IMAP4 server implementation
that returns "IDLE" as one of the supported capabilities to the
CAPABILITY command. If the server does not advertise the IDLE
capability, the client MUST NOT use the IDLE command and must poll
for mailbox updates. In particular, the client MUST continue to be
able to accept unsolicited untagged responses to ANY command, as
specified in the base IMAP specification.

The IDLE command is sent from the client to the server when the
client is ready to accept unsolicited mailbox update messages. The
server requests a response to the IDLE command using the continuation
("+") response. The IDLE command remains active until the client
responds to the continuation, and as long as an IDLE command is
active, the server is now free to send untagged EXISTS, EXPUNGE, and
other messages at any time.

[besson3c]

Originally Posted by mduell 

What RFC are you reading? 2177 says:

The IDLE command may be used with any IMAP4 server implementation
that returns "IDLE" as one of the supported capabilities to the
CAPABILITY command. If the server does not advertise the IDLE
capability, the client MUST NOT use the IDLE command and must poll
for mailbox updates. In particular, the client MUST continue to be
able to accept unsolicited untagged responses to ANY command, as
specified in the base IMAP specification.

The IDLE command is sent from the client to the server when the
client is ready to accept unsolicited mailbox update messages. The
server requests a response to the IDLE command using the continuation
("+") response. The IDLE command remains active until the client
responds to the continuation, and as long as an IDLE command is
active, the server is now free to send untagged EXISTS, EXPUNGE, and
other messages at any time.

Pretty much every IMAP server supports IDLE now though, and has for a long time... Courier, Cyrus, you name it. Why is it then that push IMAP didn't work prior to MobileMe? Why is it that push IMAP doesn't work on any other servers that have IDLE enabled with the iPhone, or with any other client that also supports IDLE (OS X Mail, Thunderbird, etc.)?

[mduell]

Originally Posted by besson3c 

Pretty much every IMAP server supports IDLE now though, and has for a long time... Courier, Cyrus, you name it. Why is it then that push IMAP didn't work prior to MobileMe? Why is it that push IMAP doesn't work on any other servers that have IDLE enabled with the iPhone, or with any other client that also supports IDLE (OS X Mail, Thunderbird, etc.)?

Push IMAP doesn't work on the iPhone because it would kill the battery life to keep the network connection open; it will have to wait for Apple's PNS to go public and then it should work just like a Blackberry (Apple's server keeps the IMAP IDLE session open, notifies device as necessary).
Of the 5 devices we've tested at work (Blackberry 8700, Treo 680 running a 3rd party mail client, 3 Leopard Macs), the Blackberry, Treo, and one of the Macs get instant email with IMAP IDLE while the other 2 Macs do not.

[besson3c]

It's a mystery to me, which is why I'm thinking that not all IMAP IDLE supporting clients are equal, or else more is necessary to get this to work properly. I run a Cyrus server at work and a Courier server for my own company, and no client that I've ever used did push properly. Both Cyrus and Courier have IDLE enabled, as do all my clients. I'm fairly certain that the leaving connection open part is fine (although this is just empirical evidence) but it seems that something more is necessary for the server to signal to the clients when there are new messages.