513309 - MS Exchange message disappears, reappears.

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Mahesh Asolkar]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2a2pre) Gecko/20090828 Namoroka/3.6a2pre
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4pre) Gecko/20090828 Lightning/1.0pre Shredder/3.0b4pre

I use Shredder to access MS Exchange email via IMAP.

Almost always, when a new message arrives, and I am reading it (in the classic layout) the message suddenly disappears from the message list. The message pane goes empty - no header, no content. Within a second or two, the message reappears in the message list - but is not selected, so message pane is still empty. I need to select the message again in order for it to appear in the message pane and read it.

Sometimes, this happens multiple times with a message. Either ways, it is slightly annoying.

Reproducible: Always

Steps to Reproduce:
1. In an IMAP (Exchange?) account, wait for a new message to arrive
2. Select it and read it in the classic layout

Actual Results:  
Message disappears from the message pane and message list only to reappear in the message list in a second or two.

Expected Results:  
Message pane should not go empty when I am reading a message

I do use other clients (alpine/outlook web) simultaneously to access this account.

Most of my messages are filed in folders with Outlook rules on the server. So I am not quite sure if the same happens with messages in the INBOX. But I'll keep my eyes open and report what I find.

Comment 1

[Wayne Mery (:wsmwk)]

do you still see this in version 3.1.7?

Comment 2

[Mahesh Asolkar]

I am not using 3.1.7 at the moment. I do see this in the comm-central nightly builds (3.3a2pre). The behavior is still exactly the same.

When I reported the problem, I was using Thunderbird on Linux. Now I see this both in Linux and Windows.

I use the Thunderbird Conversations add-on. The behavior occurs in Conversations view too.

Comment 3

[Ludovic Hirlimann [:Usul]]

Can we get an imap log , as described at https://wiki.mozilla.org/MailNews:Logging ?

Comment 4

[Mahesh Asolkar]

I found the following thread discussing, what looks like, exactly this situation:

  http://www.blackberryforums.com/bes-admin-corner/134823-bes-causes-exchange-emails-via-imap-disappear-reappear.html

I do notice that messages appear/disappear right around the time I hear the blackberry notification. But I didn't think of the connection.

To reply to comment#3, I tried to get the generate IMAP log, but it has too much sensitive information.

Comment 5

[David :Bienvenu]

this looks like a bug with Exchange, or BES+Exchange+IMAP.

Comment 6

[WADA:World Anti-bad-Duping Agency]

It may be next;
(1) A mail arrives, is presented to IMAP client as UID=N by Exchange server.
(2) After a short while, filtering program runs at server, adds some headers and re-orders mail headers, and, removes old version(UID=N) and generates new version(UID=N+1).
Above was reported to a bug, but I couldn't reach again. In this case, content of existent header was not changed, so difference of headers can be noticed by view message source only.
Above can explain next in forum post pointed by comment #4.
> * 150 EXPUNGE
> * 151 EXISTS
As "IDLE command use" is enabled by default if Tb, and as many IMAP servers already support IDLE, and other mailers usually don't use IDLE command if default setting, only Tb user usually notices first version of UID=N.

Mahesh Asolkar, please check with next;
(1) Show "Order Received" column. If IMAP folder, value is UID of mail.
(2) Server Settings, choose IMAP delete model of "Just mark it as deleted".
    Mail flagged as \Deleted is shown with strike line at thread pane.
(3) View message source. If multiple versions are seen, compare headers.

Comment 7

[WADA:World Anti-bad-Duping Agency]

I couldn't reach bug for two versions of original mail and mail of "added some headers and re-ordered headers" again, but I could reach bug for "two versions of same mail"(two different UIDs, different RFC822.SIZE, same standard headers) with Exchange/BES again.
> Bug 482472 [BlackBerry] BES causes Exchange emails via IMAP to disappear and reappear
I don't know which version(which UID) is kept after multiple versions of a mail is created at server in that bug, because no log is provided for it and because of no answer about UID of remained mail.

Comment 8

[Mahesh Asolkar]

I can confirm observation in comment #7. After making the changes suggested in comment #6, I do see multiple UID (order received) attached to the same message. For example, when the message first arrives, it is 109304. After a while it changes to 109305, then 109306. However, I have not seen all these versions in the thread pane - all at once. The latest one always stays, the older ones disappear.

I wonder why I see up to 3 different UIDs associated with a message. Is it because I use Thunderbird, BB and Re-Alpine?

Having said that, as a user, it doesn't bother me that a message has multiple UIDs. I did not even notice it until I saw the 'order received' column. What annoys me is that when I am reading a new message in the message pane, it suddenly disappears and interrupts my reading. Why can't the changing UIDs only affect the thread pane and leave the message pane alone?

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #8)
> I wonder why I see up to 3 different UIDs associated with a message. Is it
> because I use Thunderbird, BB and Re-Alpine?

It's simply because;
- Three versions of a mail are created at server(call UID=X, UID=Y, UID=Z
  where X < Y < Z), and non-last versions(UID=X, UID=Y) are deleted
  at server.
- As Tb uses IDLE, Tb can easily track all of creation of UID=X/UID=Y/UID=Z
  and deletion of UID=X/UID=Y.

> Why can't the changing UIDs only affect the thread pane and leave the message pane alone?

Different UID == Different mail.
It's true even for mail copy of absolutely same mail data. It's similar to multiple uploads of a mail from local mail folder to an IMAP folder.
In your case, filtering software generated multiple versions, and finally kept newest/last version only.
"Delete of a mail in IMAP" == "Store \Deleted flag to a mail".
If IMAP delete model of "move to trash" or "remove it immediately" is used, Tb removes locally held data of mail(data in .msf) when Tb detects \Deleted flag of a mail, and removes it from thread pane(==mail doesn't exist any more for Tb).
If IMAP delete model of "just mark it as deleted", Tb keeps data for mail flagged as \Deleted until the mail of \Deleted flag is permanently removed from mbox by EXPUNGE command by IMAP client or expunging operation at server.

Have you checked with "just mark it as deleted" and compared mail source of multiple versions?

Comment 10

[Mahesh Asolkar]

There's got to be something I am missing. There are two things you expect, that I don't see happening:

1. Even if I set my server setting for 'When I delete a message:' to 'Just mark it as deleted', I don't see multiple copies of a message (with different UIDs). Can you confirm if you indeed expect that to happen?

2. When I view message source (Ctrl-U), I don't see the UID of the message in the source. That is, if a messages has 101066 in the 'Order Received' column, I don't see the number 101066 anywhere in the message source. Do you expect the number to appear in the message source?

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #10)

"Whether multiple versions is seen at thread pane or not" depends on;
(a) Multiple versions are generated at server or not.
(b) When generated new version(s) or falgged-as-\Deleted mail(s) are notified from IMAP server to Tb via IDLE.
For example;
- If server notified existence of last UID=Z only via IDLE after creation of
  UID=X/Y/Z and deletion of UID=X/Y and execution of expunge operation,
  Tb can know about mail of UID=Z only.
- If filtering software doesn't generate multiple versions, multiple versions
  can't occur at thread pane of Tb, even with "just mark it as deleted".

UID of IMAP is never data in mail source. It's assigned number to a mail in Mbox at IMAP server by the IMAP server. It's shown to user by Tb as "Order Received" column value only. (If POP3, Tb adds X-UIDL: header to mail data written in local mail folder file, and puts UIDL data passed from POP3 server in the X-UIDL: header. This is not applicable to IMAP folder.)  
When you saw multiple versions at thread pane with "just mark it as deleted", save mails to .eml files(even mail shown with strike line==flagged \Deleted), and compare .eml file size, header order, added or removed or altered headers, please.

Comment 12

[WADA:World Anti-bad-Duping Agency]

If you can see multiple versions at thread pane consistently with "Just mark it as deleted" model, get IMAP log(with timestamp) requested by Ludovic Hirlimann in Comment #3, please.
> https://wiki.mozilla.org/MailNews:Logging
> MS Win example : SET NSPR_LOG_MODULES=timestamp,imap:5
Please attach log file to this bug via "Add an attachment" link(never paste long log data), please. Because mail data itself is not needed for log analysis, remove or replace mail data except Message-Id: header line from log file before open log data to public. Replace or remove other sensitive data too, please.

Comment 13

[Will Roberts [:bws42]]

I see this issue in Thunderbird 17.0.6 on Linux connecting to exchange via IMAP.

I've added the Order Received column, and the number there is not changing for the message that is disappearing/reappearing.

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to Will Roberts [:bws42] from comment #13)
> I've added the Order Received column, and the number there is
> not changing for the message that is disappearing/reappearing.

It indicates that Excange server (1) adds new mail with UID=AA, (2) deletes the mail of UID=AA(perhaps by store \Deleted flag), then (3) Tb knows \Deleted status of the mail and removes data for the mail, then Exchange server (4) replaces header data of mail of UID=AA, and (5) removes \Deleted flag of mail of UID=AA.

> Status: UNCONFIRMED → NEW

"New" means "Tb's flaw in code actually exists", unless this bug is Severity=Enhancement.
How can above phenomenon be Tb's bug?

Comment 15

[Will Roberts [:bws42]]

(In reply to WADA from comment #14)
> (In reply to Will Roberts [:bws42] from comment #13)
> > I've added the Order Received column, and the number there is
> > not changing for the message that is disappearing/reappearing.
> 
> It indicates that Excange server (1) adds new mail with UID=AA, (2) deletes
> the mail of UID=AA(perhaps by store \Deleted flag), then (3) Tb knows
> \Deleted status of the mail and removes data for the mail, then Exchange
> server (4) replaces header data of mail of UID=AA, and (5) removes \Deleted
> flag of mail of UID=AA.

I have an imap log (following the instructions above) that should contain this issue happening. It's quite large and has quite a bit of sensitive content in it. If someone can tell me what to grep for I'll post snippets here.

> > Status: UNCONFIRMED → NEW
> 
> "New" means "Tb's flaw in code actually exists", unless this bug is
> Severity=Enhancement.
> How can above phenomenon be Tb's bug?

To me "UNCONFIRMED" means no one can reproduce this issue or has triaged it. I've seen this issue all day today, so it definitely occurs. If this ends up being not an issue with Thunderbird (or something that can be worked around), then this should be resolved INVALID and not left UNCONFIRMED.

Comment 16

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 17

[Wayne Mery (:wsmwk)]

Mahesh, Will, Do you still see this problem?

Comment 18

[Will Roberts [:bws42]]

I no longer see this issue. I can't say for sure when it stopped, but my best guess is that it happened when my company moved from Exchange 2007 to Exchange 2013 last year.

I'm currently using 52.6.0 (64-bit) on Debian Linux 8.x

Comment 19

[Wayne Mery (:wsmwk)]

Thanks for the update