1388461 - Invalid UID Fetch attribute XSENDER on iCloud Addresses

Status: RESOLVED INVALID

(Thunderbird :: Untriaged, defect)

Description

[Chad Garrett]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36

Steps to reproduce:

My client already had an iCloud account set up in Thunderbird.  On August 4th, the mail failed to open (though the list populates just fine) with an error of "Invalid UID Fetch attribute XSENDER."  All cached messages could still be opened, but any messages received on or after August 4th would not open.

I tried to delete and re-add the account.  This led to all of the messages re-populating, but none of the messages could be opened due to the error and the cache being deleted.

I realize this problem is almost certainly with changes to iCloud's implementation of IMAP, and possibly a small number of email clients that even try to use XSENDER.  I will do whatever I can to help diagnose this bug, but I am not a programmer.  I do not currently have my own iCloud account but will create one if necessary to continue troubleshooting.


Actual results:

Received error Invalid UID Fetch attribute XSENDER.


Expected results:

Message should have displayed.

Comment 1

[Chad Garrett]

There are already people discussing this bug online, but are comparing it to a completely unrelated bug. http://forums.mozillazine.org/viewtopic.php?f=39&t=3032257&p=14759100

Comment 2

[Chad Garrett]

When I said unrelated, I really only meant a different email provider and that provider has already solved the problem from their end.  Sorry I didn't include that in my earlier comment.

Comment 3

[Jorg K (CEST = GMT+2)]

Gene, any time to look into this one?

Comment 4

[Chad Garrett]

As of about 15 minutes ago, I personally have seen the problem correct itself.  I'm still waiting to see if anyone else can confirm the same at the forum thread.  I think Apple may have changed some server parameters, but I'll have to see if anyone else is still affected.

Comment 5

[gene smith]

My guess is iC reporting XSend cap but it's being rejected by iC when used. Waiting for more info from reporter.

Comment 6

[Jorg K (CEST = GMT+2)]

So that sounds like some logging is necessary here: https://wiki.mozilla.org/MailNews:Logging.
But then, the problem fixed itself. I'll close the bug, we can reopen of necessary.

Comment 7

[Jean-Francois Bourdeau]

(In reply to Chad Garrett from comment #0)
> User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
> 
> Steps to reproduce:
> 
> My client already had an iCloud account set up in Thunderbird.  On August
> 4th, the mail failed to open (though the list populates just fine) with an
> error of "Invalid UID Fetch attribute XSENDER."  All cached messages could
> still be opened, but any messages received on or after August 4th would not
> open.
> 
> I tried to delete and re-add the account.  This led to all of the messages
> re-populating, but none of the messages could be opened due to the error and
> the cache being deleted.
> 
> I realize this problem is almost certainly with changes to iCloud's
> implementation of IMAP, and possibly a small number of email clients that
> even try to use XSENDER.  I will do whatever I can to help diagnose this
> bug, but I am not a programmer.  I do not currently have my own iCloud
> account but will create one if necessary to continue troubleshooting.
> 
> 
> Actual results:
> 
> Received error Invalid UID Fetch attribute XSENDER.
> 
> 
> Expected results:
> 
> Message should have displayed.

SAME PROBLEM since somewhere in the last few days
Icloud @me.com address, 2 factor authentification (with password created for Thunderbird) , Linux MINT and Thunderbird 52.2.1 (64-bit)
JF
MTL CANADA

Comment 8

[gene smith]

I see it too after setting up "2-step" auth on linux (would not work with just my normal icloud password). 
Looking at the imap log, I see 2 out of 6 capability responses for server imap.mail.me.com supporting XSENDER. The other 4 don't list XSENDER. This seems similar to Bug 1383033 where some outlook imap capability responses supported MOVE and others didn't.

Also, when I connect with with openssl,
openssl s_client -connect imap.mail.me.com:993 -crlf

I don't see XSENDER when I request capability (just an observation):

3 capability
* CAPABILITY IMAP4 IMAP4rev1 ACL RIGHTS=tekx QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES CONDSTORE ENABLE QRESYNC CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES METADATA ID LIST-STATUS SPECIAL-USE CREATE-SPECIAL-USE MULTISEARCH SORT=DISPLAY LOGIN-REFERRALS X-SUN-SORT ANNOTATE-EXPERIMENT-1 X-SUN-IMAP XUM1 X-ORCL-AS IDLE XSNIPPET=FUZZY

Finally, when I restarted tb, I didn't see the XSENDER error and I could see newly received emails. So apparently sometimes tb catches and uses the "correct" capability list and avoids the bad server response by not including the XSENDER flag. Looking at the help forum discussion mentioned in Comment 1, it also sounds like people are seeing somewhat random behavior.

Just for info, here are the types of imap.map.me.com responses from the tb imap.log I recorded. These are not responses to a capability request but "un-tagged" responses to "authenticate PLAIN" and other imap requests.

No XSENDER (short response):

[Unnamed thread 0x7f52e02f4bc0]: I/IMAP 0x7f52dda6e800:imap.mail.me.com:NA:CreateNewLineFromSocket: * OK [CAPABILITY st11p00mm-iscream011 17E71 XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 SASL-IR AUTH=ATOKEN AUTH=PLAIN] iSCREAM ready to rumble (17E71-26414-16A-2268-c90b5397169a:2479) st11p00mm-iscream011 [10:14837:05:45:08:05]

No XSENDER (long response):

[Unnamed thread 0x7f52e02f4bc0]: I/IMAP 0x7f52dda6e800:imap.mail.me.com:NA:CreateNewLineFromSocket: 1 OK [CAPABILITY XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 ACL RIGHTS=tekx QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES CONDSTORE ENABLE QRESYNC CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES METADATA ID LIST-STATUS SPECIAL-USE CREATE-SPECIAL-USE MULTISEARCH SORT=DISPLAY LOGIN-REFERRALS X-SUN-SORT ANNOTATE-EXPERIMENT-1 X-SUN-IMAP XUM1 X-ORCL-AS IDLE XSNIPPET=FUZZY] User gd.smth logged in

Contains XSENDER (long response):

[Unnamed thread 0x7f52cfac5ce0]: I/IMAP 0x7f52cf220800:imap.mail.me.com:NA:CreateNewLineFromSocket: 1 OK [CAPABILITY XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 ACL RIGHTS=tekx QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES CONDSTORE ENABLE QRESYNC CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES METADATA ID XSENDER X-SUN-SORT ANNOTATE-EXPERIMENT-1 X-UNAUTHENTICATE X-SUN-IMAP XUM1 IDLE] User gd.smth logged

Comment 9

[gene smith]

Looking closer at the log for imap.mail.me.com it appears that tb is doing the right thing. When a mailbox authenticate response contains XSENDER capability, tb includes the XSENDER in the "UID FETCH" request. For mailboxes that do not contain XSENDER capability in the authenticate response there is *no* added XSENDER flag in the "UID FETCH" request to the server.

The problem occurs when the server reports XSENDER capability for the connected mailbox but then rejects the "UID FETCH X:Y (XSENDER ...)" request. For example, the connection for "Sent" mailbox reports the following capability:

[Unnamed thread 0x7f74f97771a0]: I/IMAP 0x7f74dfc11800:imap.mail.me.com:NA:CreateNewLineFromSocket: 1 OK [CAPABILITY XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 ACL RIGHTS=tekx QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES CONDSTORE ENABLE QRESYNC CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES METADATA ID XSENDER X-SUN-SORT ANNOTATE-EXPERIMENT-1 X-UNAUTHENTICATE X-SUN-IMAP XUM1 IDLE] User gd.smth logged

Then on the same mailbox the following UID FETCH occurs with an error response from the server:

[Unnamed thread 0x7f74f97771a0]: I/IMAP 0x7f74dfc11800:imap.mail.me.com:S-Sent:SendData: 8 UID fetch 1,2:3 (XSENDER UID RFC822.SIZE BODY.PEEK[])
[Unnamed thread 0x7f74f97771a0]: D/IMAP ReadNextLine [stream=0x7f74e03952c0 nb=43 needmore=0]
[Unnamed thread 0x7f74f97771a0]: I/IMAP 0x7f74dfc11800:imap.mail.me.com:S-Sent:CreateNewLineFromSocket: 8 BAD Invalid UID Fetch attribute XSENDER

This information should be reported to Apple so they can fix their server. The only way I see to "fix" this in tb is to add a new pref so that XSENDER capability is ignored so the UID FETCH request never includes the XSENDER flag.

Note: Bug 1262054 reports a similar problem for another imap server. That server also reported capability XSENDER but failed in the same way with UID FETCH. However, the bug reporter indicated that the problem fixed itself when the server stopped falsely reporting XSENDER capability.

Comment 10

[dnewman]

I believe this is now resolved.  Issue related to a rolling upgrade of internal message stores within iCloud whose IMAP capabilities changed from those advertised by their front end IMAP proxies.  (I.e., the front end IMAP proxies were out of sync with changes to the backends; success upon connecting then fell upon which backend you got connected to by the proxies.  Rolling nature of the upgrades caused some people to start seeing it early on -- 4 Aug? -- and others not until days later.)