Closed Bug 475257 Opened 16 years ago Closed 10 years ago

Thunderbird discards offline messages (including changes not yet synced) and re-downloads (imap)

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: cbook, Unassigned)

Details

(4 keywords, Whiteboard: [need steps to reproduce])

Attachments

(1 file)

autosync info
6.43 KB, text/plain
Details
I have 2 different Maschines (XP and Mac 10.5) with my work profile. Now with Shredder i have the Problem that large mail folders, like my bugzilla emails, are resynchronized over and over again. This means all imap mails in this folder disappear and are reloaded, this takes a long time (because a big mailbox). This was fine with Thunderbird 2. Maybe its caused by the offline synchronization feature.
Flags: blocking-thunderbird3?
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
Keywords: regression
Does reindexing your bugzilla mailbox help?
Tomcat, what data do you have to indicate that it's pulling in the same emails again? Denying blocking for now.
Flags: blocking-thunderbird3? → blocking-thunderbird3-
Hardware: x86 → All
Sorry, I should have chimed in earlier. I've seen this in the past and encouraged Tomcat to file the bug, because when it happens it's bad: you lose work like moving messages that you've done while offline. I've been seeing this occasionally for years, including in Thunderbird 2, so I suspect there's no regression here and would be curious to know the motivation for why that keyword is currently set. I agree that it can't block until we get steps to reproduce, but I think it's worth spending some QA effort on trying to nail down those steps because the failure mode is very painful if you've done hours of mail triage on a plane flight. It would also be interesting to know if there are other actions that get lost when this happens (eg tagging). Assigning to Gary in the hopes that he can figure out more details.
Assignee: nobody → nth10sd
Flags: wanted-thunderbird3+
Keywords: qawanted
Whiteboard: [need steps to reproduce]
(In reply to comment #0) > I have 2 different Maschines (XP and Mac 10.5) with my work profile. Now with > Shredder i have the Problem that large mail folders, like my bugzilla emails, > are resynchronized over and over again. > > This means all imap mails in this folder disappear and are reloaded, this takes > a long time (because a big mailbox). This was fine with Thunderbird 2. Maybe > its caused by the offline synchronization feature. Something I'd like to clarify first with Tomcat here to ensure we're on the same channel: - Are you using the same work profile on your two machines, same path, same folder, <same random name>.default? (I suspect not) - Are you using profiles with identical settings, but on their different locations in XP / Mac 10.5? - Are you continuously online or as dmose mentions, occasionally offline, losing some of the offline work after you go online, and the resynchronization occurs? - Does the re-synchronization occur only after you go online after some offline work, or always occur whenever you go online after being offline for a period of time (no offline work)? I'm already running debug builds, I did notice something weird recently -- iirc, I moved emails from an Exchange IMAP folder to another Exchange IMAP folder, though I couldn't remember if I was on/offline. Later, when I checked the latter folder, those emails were missing. Lucky thing was, I went online in the web-based login and found my emails in the latter folder, and I'm quite sure 99% that I did a rebuild index to make them appear in TB. I thought this was a local issue to me but this might be related to what Tomcat is experiencing, though I'm not sure if the on/offline issue plays a part.. I'll do more twiddling over the next few... Emre, is there a way to turn on debug foo for offline synchronization? (I only get: Begin mail message delivery. Abort mail message delivery. with debug builds, and I don't think IMAP logs completely give you what is needed)
Attached file autosync info —
Bienvenu says to set autosyncActivities.logging.dump to "All" - and here's what I found, after I resumed debug TB from sleep and switched wifi connection (while running around school and lecture theatres) I don't know why TB autosync keeps re-syncing all my All Mail mails. Bienvenu / Emre, does this help?
There's nothing strange about these logs, although the logging format could be improved.. Autosync didn't finish downloading all of the mail, it still has 5602 to go. Autosync _will_ check the all mail folder periodically, and autosync again, downloading emails that are needed.
(In reply to comment #6) > Autosync didn't finish downloading all of the mail, it still has > 5602 to go. Autosync _will_ check the all mail folder periodically, and > autosync again, downloading emails that are needed. I am very sure that All Mail _was_ already fully downloaded prior to going offline (i.e. letting my mac go to sleep), that log shows what happened _after_ i went online again (i.e. opening my mac at another wireless access point and going online again). No new mail of any sort came in.
There are a few possibilities (and it's possible different people are experiencing different things): 1. The server is rolling UID validity - this means we have to throw away all our cached information per the IMAP RFC. So we throw away the .msf file, and redownload all the headers over again, and then autosync will eventually redownload the messages bodies. If that happens, you would see us redownloading all the message headers in the log, followed by the bodies. There's some hope that Kent James' work to use backup .msf files might be extended to help in this situation. 2. Folder discovery gets interrupted in such a way that we think all imap folders have been removed from the server, and we remove the .msf files and offline stores for everything but the inbox. We had this problem before if the first url we ran was interrupted (the first url does folder discovery), but it was fixed. In fact, I think the problem was limited to the url we run when the user collapses and expands the server to rediscover folders. 3. We somehow decide that the threading information in the folder is horked and we throw away the .msf file. We try hard not to do this and it doesn't sound like what's going on. Do you often work with your All Mail folder, or do you mainly stay in the INBOX? Given the intermittent nature of Gary's connection, it sounds like something like #2 is happening, but I've gone over the code and I don't see anything obviously wrong. As a user, you would actually see the folders disappear and then re-appear - have you seen that?
(In reply to comment #8) > As a user, you would actually see the folders disappear and > then re-appear - have you seen that? When I've lost offline work in the past, this is exactly what I've seen.
Keywords: dataloss
Summary: Shredder is trying to synchronize already downloaded emails → Thunderbird discards offline messages (including changes not yet synced) and re-downloads
Changed the bug title under the assumption that we really are all talking about the same bug, because the behavior described in that case is far and away the worst user experience and worth caring about the most. If we do come to the conclusion that there really are multiple bugs here, we should probably spin off the others.
If this happens, you should see folderDeleted notifications in the activity manager, I think.
(In reply to comment #8) > 2. Folder discovery gets interrupted in such a way that we think all imap > folders have been removed from the server, and we remove the .msf files and > offline stores for everything but the inbox. We had this problem before if the > first url we ran was interrupted (the first url does folder discovery), but it > was fixed. In fact, I think the problem was limited to the url we run when the > user collapses and expands the server to rediscover folders. This happens really occasionally, i.e. when I resume going online after changing wifi connections, but I'd have to look out for more folderDeleted notifications to really be sure.. (See the OnFolderRemovedFromQ line below) > 3. We somehow decide that the threading information in the folder is horked and > we throw away the .msf file. We try hard not to do this and it doesn't sound > like what's going on. Do you often work with your All Mail folder, or do you > mainly stay in the INBOX? Yes, I use some form of threading, and I hardly ever work with my All Mail folder, so yes, I mainly stay in the inbox. (Probably) due to this re-synchronization, I keep my inbox free of clutter by archiving mails, and to keep it responsive, but only now I realize it might be resync of All Mail causing the problem. Here's a log _after_ resuming my Mac from sleep at home -- I never changed wifi connection prior to going to bed, and waking up: 2009-02-19 07:28:33 autosyncActivities INFO OnStatusChanged: running 2009-02-19 07:28:33 autosyncActivities INFO OnDownloadCompleted: Inbox of nth10sd@gmail.com 2009-02-19 07:28:33 autosyncActivities INFO OnDownloadCompleted: Auto-Sync Manager is paused, pausing the process 2009-02-19 07:28:33 autosyncActivities INFO OnFolderRemovedFromQ [0] Inbox of nth10sd@gmail.com 2009-02-19 07:28:33 autosyncActivities INFO Auto_Sync OnFolderRemovedFromQ Last folder of the server: true 2009-02-19 07:28:34 autosyncActivities INFO onDiscoveryQProcessed: Processed 250/12852 of All Mail 2009-02-19 07:28:34 autosyncActivities INFO onDiscoveryQProcessed: Processed 250/12602 of All Mail 2009-02-19 07:28:35 autosyncActivities INFO onDiscoveryQProcessed: Processed 250/12352 of All Mail
OnFolderRemovedFromQ just means autosync has finished processing the folder. That's normal, and good.
(In reply to comment #13) > OnFolderRemovedFromQ just means autosync has finished processing the folder. > That's normal, and good. The next hundred or so messages from that autosync log is about how All Mail tries to get itself re-synchronized (I have 2k + messages amounting to 3Gb plus in All Mail of that account) That said, my Thunderbird is always running pre-MacSleeping and post-MacWaking. Notes: - I never really noticed if my folders disappear and re-appear in the folder pane. At least, not yet. - Forgot to turn on IMAP protocol logging. - Will update my debug build later -- my build was from just before Activity Manager landed.
I can't reproduce this in any way till the questions in comment #4 are answered by Tomcat and whether this is still seen with Thunderbird 3 Beta 2 (which is about to be released real soon), deassigning myself since it's about as much QA I can do..
Assignee: nth10sd → nobody
Tomcat? Still see this with nightly? And see comment 15.
(In reply to comment #4) yes i still have seen the problem in the last days > Something I'd like to clarify first with Tomcat here to ensure we're on the > same channel: > > - Are you using the same work profile on your two machines, same path, same > folder, <same random name>.default? (I suspect not) no, different profile but some identical settings on mac and xp > - Are you continuously online or as dmose mentions, occasionally offline, > losing some of the offline work after you go online, and the resynchronization > occurs? I work "only" online, no offline :) > - Does the re-synchronization occur only after you go online after some offline > work, or always occur whenever you go online after being offline for a period > of time (no offline work)? happen when i work with the profile at the same time on mac and windows. As example when both computer are online and shredder is started
(In reply to comment #18) > Carsten, do you still see this issue? > fixed+open bugs: =short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=component > (but two of those mention pop) nope also i don't used a email programm for awhile
Also experienced this twice in the last few months, this morning, Thunderbird kept downloading the [Gmail]/* folders all over again and again, until I deleted all the msf files and restarted Thunderbird. No STR...
bienvenu, if there is nothing here that you want to do? ... then we can close this incomplete. (bienvenu in comment #8) > There are a few possibilities (and it's possible different people are > experiencing different things): > > 1. The server is rolling UID validity - this means we have to throw away all > our cached information per the IMAP RFC. So we throw away the .msf file, and > redownload all the headers over again, and then autosync will eventually > redownload the messages bodies. If that happens, you would see us > redownloading all the message headers in the log, followed by the bodies. > There's some hope that Kent James' work to use backup .msf files might be > extended to help in this situation. > > 2. Folder discovery gets interrupted in such a way that we think all imap > folders have been removed from the server, and we remove the .msf files and > offline stores for everything but the inbox. We had this problem before if > the first url we ran was interrupted (the first url does folder discovery), > but it was fixed. In fact, I think the problem was limited to the url we run > when the user collapses and expands the server to rediscover folders. > > 3. We somehow decide that the threading information in the folder is horked > and we throw away the .msf file. We try hard not to do this and it doesn't > sound like what's going on. Do you often work with your All Mail folder, or > do you mainly stay in the INBOX? > > Given the intermittent nature of Gary's connection, it sounds like something > like #2 is happening, but I've gone over the code and I don't see anything > obviously wrong. As a user, you would actually see the folders disappear and > then re-appear - have you seen that?
I saw it when I applied the update to the latest nightly of Thunderbird (and restarted it because of this reason).
(In reply to :Ehsan Akhgari (Away 2/7-2/15) from comment #24) > I saw it when I applied the update to the latest nightly of Thunderbird (and > restarted it because of this reason). Ehsan, do you still see this? Gary, do you still see this? Everyone - was it only with gmail account? xref Bug 816327 Constantly "Downloading M of N in All Mail" or Inbox messages from Gmail IMAP account, slows system
Flags: needinfo?(ehsan)
I don't remember anything from when I posted comment 24.
Flags: needinfo?(ehsan)
Severity: normal → critical
Keywords: perf
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: