433806 - Mail is lost upon Work Online when deletions/moves of IMAP mails are done while offline mode (move between folder of same IMAP acount)

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[Michael Baffoni]

I have a lot of users that work on airplanes: so before they get on the plane, they sync the inbox (etc.) from the IMAP server so they have local copies of the messages, then go offline.  Right now they are able to read/reply/forward while offline, however any deletions of emails, moves from inbox to subfolders, etc. are not captured and replayed (true syncing) when they later go online.  Since there should be a unique relationship between the offline copies and the messages on the IMAP server, it should be relatively straightforward to capture deletions/moves of messages based on those IDs and give the option to resync those changes upon reconnect to the IMAP server.  This is a capability that Outlook has that a lot of users require when working outside of the network for any length of time.

Comment 1

[Magnus Melin [:mkmelin]]

Haven't tested moves, but offline deletes show up fine for me.

Improving the offline experience is one of the main goals for tb3, xref bug 435153 and others.

Comment 2

[Wayne Mery (:wsmwk)]

Michael, can you confirm problem is gone using current nightlies and beta?

Comment 3

[Jens Müller (:tessarakt)]

I have a similar problem: I _read_ emails offline (on the plane), and they were marked read while offline.

After I was online again, it synced the folders, and they were unread again.

I am using the latest nightly of 3.0 ...

Comment 4

[Jens Müller (:tessarakt)]

Ah, it _might_ be that I closed TB after it got online but before it had replayed - where is that "redo log" persisted?

Comment 5

[Ludovic Hirlimann [:Usul]]

(In reply to comment #2)
> Michael, can you confirm problem is gone using current nightlies and beta?

It's not - I saw it very recently.

(In reply to comment #4)
> Ah, it _might_ be that I closed TB after it got online but before it had
> replayed - where is that "redo log" persisted?

I think it should be persistent - but I don't know. David what do we do in that case ?

Comment 6

[Jens Müller (:tessarakt)]

This should be marked with the dataloss keyword ...

Comment 7

[Wayne Mery (:wsmwk)]

(odd that this was filed ENH)
not seen this myself.
however, I would expect this to be a duplicate of another bug.
perhaps a protocol log will be needed.

Comment 8

[David :Bienvenu]

It's persistent - the offline operations are stored in the .msf files.

I have NSPR logging for some of the offline stuff as well, so if you set NSPR_LOG_MODULES to IMAPOFFLINE:5;IMAP:5, you'll get both. If the issue does involve shutting down and restarting, you'll need to have the logs of both sessions, i.e., save off the first log before restarting.

When you go back online, we try to play back offline operations right away.

Comment 9

[Michael Baffoni]

It was ENH at the time I filed the bug because they hadn't added the feature at
that time (at least to my knowledge).  I should have modified at Magnus's
reply.  

This has been a very very difficult bug to reproduce.  The only time that I can
see that it happens consistently is when an error occurs in the replay, it
seems to clear out the entire replay log at that point, rather than replaying
what it can, or stopping replay until the issue can be fixed.  I'm not familiar
enough with the guts of IMAP to know what would be the desired result (e.g. we would need to be smart enough to cancel/hold the delete if the initial copy failed in an email move).  Still trying to find a consistent test case.  This does however prevent it from being "persistent".

Also possible related issue: I have at various times (although not recently, so take with a grain of salt)seen sync errors with say "Account B" preventing going offline even if "Account A" successfully synced folders (e.g. if A is inside our network and B is for an outside account, neither will be reachable at the same time).  Again, trying to find a repeatable test case and grab logs.

Of course right now I'm fighting with the removal of the "Sync new folders"
option.  I have shared folders with a number of "prolific" users so every time
they create a new folder and my client does a rescan of folders, it adds it to
the list.  I don't see a clear way of only having my inbox, sent, and drafts
set to sync; so my test cases get cluttered up with all sorts of other folder sync activity.

Comment 10

[Jens Müller (:tessarakt)]

(In reply to comment #9)

> This has been a very very difficult bug to reproduce.  The only time that I can
> see that it happens consistently is when an error occurs in the replay, it
> seems to clear out the entire replay log at that point, rather than replaying
> what it can, or stopping replay until the issue can be fixed.  

Does that include connection-related errors?!

Comment 11

[David :Bienvenu]

I can't tell if you're using a TB 3 b3 build, or TB 2. several bugs have been fixed in this area in TB 3 b3, but not it TB 2.0x

Comment 12

[Jens Müller (:tessarakt)]

See comment #3: "I am using the latest nightly of 3.0 ..."

Comment 13

[David :Bienvenu]

sorry, I was directing my comment to Michael.

Comment 14

[Michael Baffoni]

RE: Comment #10:  Yes, connection errors have in the past, I haven't seen much of them lately.  There was a time I couldn't go offline if I couldn't reach all the accounts (I ended up turning off sync on one of the accounts so I could sync another).  I haven't revisited this in a while.

RE: Comment #11/13: All comments are directed from latest Tbird 3 prebeta4: Currently, Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090831 Shredder/3.0b4pre  but I was using prior builds of TB3 in earlier comments....

Comment 15

[Jens Müller (:tessarakt)]

This is quite reproducible for me when I have an instable network connection when I go back online (which is usually the case when I have a mobile connection).

Should I try procucing a log?

As said before: My guess would be that the replay log is simply forgotten on network errors.

Any decision yet if this will be a goal for TB3?

Comment 16

[Ludovic Hirlimann [:Usul]]

Jens , please yes try to produce the logs.

Comment 17

[Jens Müller (:tessarakt)]

OK, I think I have managed to reproduce it, but I still have to check the 489k log for personal information - it shows actual text lines of messages ...

What I did:

1. Disconnect all network connections.
2. Start TB
3. Mark some/many messages as read.
4. Enable connection sharing on my HTC Hero.
5. Wait a few seconds.
6. Disable connection sharing.
7. Enable connection sharing.
8. Leave the folder in question (Inbox), enter it again.

After TB had synced, the messages were unread again.

The log contains entries like this:

127404[3ec3100]: ReadNextLine [stream=4d047e8 nb=40 needmore=0]
127404[3ec3100]: 36c7000:mail.beckendorf.de:S-INBOX:CreateNewLineFromSocket:  sondern die Mitte und die Armen. <br>

I don't really know how much I may edit the log file so that it stays useful ...

Just logging imap:5 was ok, yeah?

Comment 18

[Jens Müller (:tessarakt)]

Attachment: IMAP log file from when the bug appeared

Comment 19

[Jens Müller (:tessarakt)]

Sorry, you'll probably also need imapoffline:5? I can just do it again ...

Comment 20

[Jens Müller (:tessarakt)]

I created another log with imap:5,imapoffline:5,timestamps. I can't see if there is more in it, but I'll attach it anyway.

Comment 21

[Jens Müller (:tessarakt)]

Attachment: Log with imap:5,imapoffline:5,timestamps

Comment 22

[Jens Müller (:tessarakt)]

Bug 397910 looks similar.

Comment 23

[Jens Müller (:tessarakt)]

Any work going on on this? Were the logs helpful?

Comment 24

[Jens Müller (:tessarakt)]

Can you clarify on the relation to bug 518768 and bug 517256?

Comment 25

[David :Bienvenu]

afaik, no relation.

I don't see any of the IMAPOFFLINE logging in your log file, for some reason.

Comment 26

[Jens Müller (:tessarakt)]

I know, but I'm pretty sure I had the variable set correctly ...

Will retry, but all this is fairly time-consuming.

Comment 27

[Michael Baffoni]

FWIW, I'm still seeing email deletions/moves not completed after going back online.  This is while flipping back and forth from online to offline while connected over a very stable wired connection at my desk (sorry for not responding to comment #10).  It is intermittent, some moves/deletions work fine, others are whole swathes of moves/deletes that I'm sure I completed, but after going online, I see the emails there (all in a "read" state, even emails that I deleted while unread).  I'm using the TB3 RC (2 I think): Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091117 Thunderbird/3.0.  Is there a newer RC I should be using/testing with?  If this is the correct version, I'll do my best to test with logging if desired (maybe you already have sufficient logs ...?).

Comment 28

[Jens Müller (:tessarakt)]

Michael, I think you can just try with current TB3 release.

Sorry, I don't have time for another attempt on logging ...

Comment 29

[Jens Müller (:tessarakt)]

(In reply to comment #25)
> afaik, no relation.
> 
> I don't see any of the IMAPOFFLINE logging in your log file, for some reason.

How do I recognise it?

I have a new log ...

2010-05-19 21:57:24.938659 UTC - -279024784[f2ee0450]: e7140800:mail.beckendorf.de:NA:ProcessCurrentURL: aborting queued urls

-> is that ok?

Can I send the log to some developer? I do not want to search everything for personal data ...

Comment 30

[Jens Müller (:tessarakt)]

Attachment: log

Comment 31

[Jens Müller (:tessarakt)]

Build was Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.9.2.4) Gecko/20100517 Lightning/1.0b2pre Lanikai/3.1pre

Comment 32

[Jens Müller (:tessarakt)]

(In reply to comment #8)
> If the issue does
> involve shutting down and restarting, you'll need to have the logs of both
> sessions, i.e., save off the first log before restarting.

No, just pulling the network plug semi-randomly and changing the folder is enough ...

Comment 33

[Jens Müller (:tessarakt)]

Has a developer been able to reproduce this? For me, it was really really easy to reproduce it.

* Open a folder A with many (and only) unread messages (in my log: the Spam folder ...)
* Switch to offline mode.
* Read some dozens of messages, marking them read.
* Have your hand at the network plug.
* Switch to online mode.
* After a second, plug network out and in several times.
* Switch to another folder
* Network on, switch to A
-> All messages unread again ...

Comment 34

[Jens Müller (:tessarakt)]

 if 397910 depends on 433806 and is blocking-thunderbird3.1+, shouldn't 433806 also be blocking-thunderbird3.1+?

Comment 35

[Mark Banner (:standard8)]

(In reply to comment #34)
>  if 397910 depends on 433806 and is blocking-thunderbird3.1+, shouldn't 433806
> also be blocking-thunderbird3.1+?

Not necessarily. Bugs can be related by the blocking/depends links but it doesn't necessarily mean that 397910 has to be fixed by fixing 433806.

Comment 36

[Ludovic Hirlimann [:Usul]]

Can someone build a debug build so we get imapoffline ?

Comment 37

[WADA:World Anti-bad-Duping Agency]

(Phenomenon-A) Quick check result with Tb 3.0.4 and next build, with Gmail IMAP.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a5pre) Gecko/20100513 Shredder/3.2a1pre
(0) IMAP delete model : remove immediately
    Max cached connection count : 2
    auto-compact is enabled, so dialog before auto-compact is enabled to test.
    As dialog before auto-compact appeared once, I replied "No"(or "Cancel").
    Folders: A1,A2,A3 (no mail)  A4 (10 small text/plain mails)
             offline-use=off, for quick check
(1) Work Offline.
(2) Move mails while offline mode
(2-1) Move 10 mails(all mail) from A4 to A3
(2-2) Move 10 mails(all mail) from A3 to A2
(2-3) Move 10 mails(all mail) from A2 to A1
(3) Work Online
    => 9 mails in A1, 1 mails in A2
    I tried two times with trunk, and same result for two checking.
    With Tb 3.0.4, result was slightly different.
      8 mails in A1, 2 mails in A2 in many same tests
      9 mails in A1, 1 mails in A2 in some same tests
Contention of connection between "A2:uid 10-th-mail copy A1" and "A2: uid 9-th-mail store flag \Deleted" after copy of 9-th mail?  

(Phenomenon-B) I happened to see by delete model change while offline mode.
(1) With IMAP delete model of "Mark it as deleted",
    move mails in A4 to A3
    => mails in A4 are marked as deleted with strike line
(2) Change IMAP delete model to "Remove immediately"
    => strike line of mails in A4 is removed, and shown as active mail
(3) As they look active mail, I moved mails in A4 to A2,
    and did some "mail move while offline mode".
(4) Work Online => all mails disappeared.
This is different problem.
  Marked as deleted mail should be hidden, if delete model is changed while
  offline mode, or delete model change while offline mode should be prohibited.
  Latter seems better solution.

(Phenomenon-C) I happened to see, by copy from local to IMAP followed by delete at local while offline mode.
(1) Work Offline
(2) Copy mails from local mail folder(say L1) to IMAP folder.
    AFAIR, I checked it with folder of offline-use=off.
(3) Delete mails in L1 (compact was not needed)
(4) Work Online => mails in IMAP folder disappeared.
This is also different problem.
  If mail copy from local to IMAP while offline mode is executed for a mail,
  deletion of the mail should be prohibited,
  as move from local to IMAP while offline mode is prohibited.
  Or mail copy from local to IMAP should be prohibited, 
  as move from local to IMAP while offline mode is prohibited.

I wrote above phenomena as examples of "moved mail loss upon Work Online" by different causes.

Changing bug summary, to focus on original problem of comment #0(moved mail loss upon Work Online), and to avoid confusion by comment posters.

David :Bienvenu, do I need to open separate bugs for above phenomena I observed?
(I have NSPR log for phenomenon-A. Gmail IMAP particular issue was involved, although main cause is invalid order of move operation for a mail upon Work Online.)

Comment 38

[Jens Müller (:tessarakt)]

(In reply to comment #37)
> although main cause is invalid order of move operation for a mail upon Work
> Online.)

So I think this is something different than what I observed. In my case, changing flags of messages is simply forgotten after network error.

Comment 39

[Jens Müller (:tessarakt)]

I wonder if it would be best to disable offline operations until this is fixed. The dataloss can be quite nasty because if often goes unnoticed.

Comment 40

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #38)
> > although main cause is invalid order of move operation for a mail upon Work
> > Online.)

Correction about my case. Cause was "multiple moves is not tracked".
  Work offline, move a mail in a1 to a2, move the mail in a2 to a3, Work online.
  => Only "move from a1 to a2" is executed upon changing back to online,
     so mail in a3 disappears.

> So I think this is something different than what I observed.
> In my case, changing flags of messages is simply forgotten after network error.

You are right. Your case and my case are different problems, although same cause may be involved in both cases.
 Your case: Network error upon Work Online, flag change loss, no mail loss
 My case  : No network error upon Work Online, mail loss at final destination,
            mail is move multiple times while offline mode
It looks for me that both cases are involved in bug operer's case.

I believe we are better to open separate bugs for specific and reprodusable case, one bug for your case, three bugs for my cases. Because bug opener's case is problems while daily use and it's very hard to isolate specific phenomenon from other phenomena. So analysis of each specific case in separate bug is required.

Comment 41

[WADA:World Anti-bad-Duping Agency]

I've opened separate Bug 567716 (Phenomenon-A) of Comment #37.

Comment 42

[David :Bienvenu]

Attachment: potential fix for what Jens is seeing

This might fix a very recent regression in our handling of connection errors while playing back operations, from the fix for bug 403603. Jens, I'll try to generate a try server build, but it may take a bit since my try server build expired. To repeat, *a very recent regression*, so it would only explain issues seen with builds from the past 10 days or so.

Comment 43

[Jens Müller (:tessarakt)]

(In reply to comment #42)
> To repeat, *a very recent regression*, so it would only explain issues
> seen with builds from the past 10 days or so.

Then I doubt this is what I am seeing, but I will happily test the tryserver build.

Comment 44

[David :Bienvenu]

Standard8, we might want to respin for the regression fix, if we can determine for sure that what Jens is seeing is a recent regression, and that this fixes it. It's a bit hard to track since there's a lot of piling on and completely different issues described in this bug and others, but this fix is pretty straightforward.

Jens, the one build with an interesting log is from 5/19, which, if the build is a nightly, would have the regression that might make it a lot easier to recreate the issue.

Comment 45

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #42)
> potential fix for what Jens is seeing

I believe fixing of Jens's specific/clear/re-producable case is better to be done in separate/clean bug. David, what do you think?

Comment 46

[David :Bienvenu]

yes, if we can determine that it is specific/clear and a new regression.

Comment 47

[WADA:World Anti-bad-Duping Agency]

Jens Müller, can you open separate bug for your case, for ease of problem analysis and fixing?
(no need to attach log, which you attached to this bug, to new bug again, because B.M.O linkifies string of "atachment NNNN".)

Comment 48

[Jens Müller (:tessarakt)]

Bug 567773

Comment 49

[David :Bienvenu]

If someone with current try server cert wants to generate a try server build with the patch in this bug, that would be helpful. I'm waiting for my  new cert...

Comment 50

[WADA:World Anti-bad-Duping Agency]

I've opened separate Bug 568167 for (Phenomenon-C) of Comment #37.

Comment 51

[ecser99]

When using Thunderbird mail client 3.1 with IMAP server the user encounters several error messages if the server is not constantly available:
- Draft messages can not be saved (the Draft folder is set as an IMAP folder, not a local folder
- Folders can’t be created (IMAP folders,as subfolders of your IMAP account)
- Messages can’t be moved to folders (to IMAP folders)
As a result: basic functions of the client can’t be used in case of connection outage or offline work. 
Possible solution:
Separation of the above functions from the connection to the server by creating an intermediate storage layer on the PC.
This layer communicates to the TB client like an IMAP server (online and ofline) and also communicates to the IMAP server (when online) like an IMAP client.

Offline working and working when connections is broken shoud function in the same way. If connection is broken, a floating notification should come up for the user and disappear again telling that there is no connection, and the client should automatically fuction in offline mode. 
My version:Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.4) Gecko/20100526 Thunderbird/3.1

All this works fine using gmail with the offline mail extention. So, we are a bit behind with TB... This is really crutical to get this right especially for corporate users.

Comment 52

[ecser99]

Attachment: High level cocncept for offline functions in TB

Comment 53

[ecser99]

Please also check the attached file:  High level cocncept for offline functions in TB. Does this make sense to you?

Comment 54

[David :Bienvenu]

Yeah, so Thunderbird does try to detect the network connection state and operate in offline mode if there's no network connection.

Comment 55

[ecser99]

Yes, there should be no functional difference between the ofline mode which is switched on by the user and the mode when the network connection is broken.

Comment 56

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #51)
> When using Thunderbird mail client 3.1 with IMAP server the user encounters
> several error messages if the server is not constantly available:
> My version:Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.4)
> Gecko/20100526 Thunderbird/3.1

Your build may have issue of Bug 567773 pointed by comment #48. ecser99, check in that bug first, please. Check with build on which patch is surely landed, please.

Comment 57

[Jens Müller (:tessarakt)]

(In reply to comment #51)
> Possible solution:
> Separation of the above functions from the connection to the server by creating
> an intermediate storage layer on the PC.
> This layer communicates to the TB client like an IMAP server (online and
> ofline) and also communicates to the IMAP server (when online) like an IMAP
> client.

Have you read, let alone understood, what this bug is about? TB already allows (some?) offline operations which it replays at re-connect. This bug is about a very specific defect in this functionality. So please open a new bug for your proposal, but don't flood this bug with comments.

Comment 58

[Wayne Mery (:wsmwk)]

(In reply to comment #42)
> Created attachment 447086 [details] [diff] [review]
> potential fix for what Jens is seeing
> 
> This might fix a very recent regression in our handling of connection errors
> while playing back operations, from the fix for bug 403603. Jens, I'll try to
> generate a try server build, but it may take a bit since my try server build
> expired. To repeat, *a very recent regression*, so it would only explain issues
> seen with builds from the past 10 days or so.

bienvenu, is your patch still good?

or, is this bug only and entirely dependent on cases in bug 567716 and bug 568167 getting fixed?

Comment 59

[David :Bienvenu]

the patch landed as part of bug 567773

Comment 60

[Wayne Mery (:wsmwk)]

Michael Baffoni, can you still reproduce your issue?

Comment 61

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #60)
> Michael Baffoni, can you still reproduce your issue?

Comment 62

[Michael Baffoni]

I haven't seen this issue in a while.  I've been running 18.a2, recently switched to 17.b2 for spellcheck testing.  WFM?

Comment 63

[Ludovic Hirlimann [:Usul]]

(In reply to Michael Baffoni from comment #62)
> I haven't seen this issue in a while.  I've been running 18.a2, recently
> switched to 17.b2 for spellcheck testing.  WFM?

Yes