1444553 - with imap - not all emails in folders enabled for offline store are downloaded when Sync Now is run

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: IMAP, defect)

Description

[Karanbir Singh]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Build ID: 20180206200532

Steps to reproduce:

Using IMAP ( with gmail and with cyrus-imapd ), I have folders setup that I filter inbound email into. I've then selected these folders to in the 'Manage Folders Subscription' list. The accounts have mbox storage, and the profile has been forward ported since the very early thunderbird days. The dir size across the 2 accounts is 32 GB.

In account prefs, "Keep Messages for this account on this computer" is enabled.
"Sync all messages regardless of age" is enabled
"Dont delete any messages " is enabled
"Always keep Stared messages" is enabled

In prefs: 
mailnews.offline_sync_mail is set to True
mailnews.offline_sync_work_offline is True
mail.imap.use_status_for_biff is False
mail.imap.mime_parts_on_demand_threshold is set to 400000
mail.imap.mime_parts_on_demand is False

From the file menu -> offline -> Download/Sync Now is run; with the option to go offline once done. eye balling the status bar at the bottom, thunderbird seems to go through all the folders ( unsure since many fly past - there are about 40 folders on the mailbox )


Actual results:

Only some emails are downloaded, lots of them are skipped. From observation this looks to mostly be previously unread emails in the folders.  Navigating to these emails results in the msg :

"The body of this message has not been downloaded from the server for reading offline. To read this message, you must reconnect to the network, choose Offline from the File menu and then untick Work Offline. In the future, you can select which messages or folders to read offline. To do this, choose Offline from the file menu and then select Download/Sync Now. You can adjust the Disc Space preference to prevent the downloading of large messages."




Expected results:

I would expect all emails in the folders selected to be available to read once thunderbird goes offline.

Comment 1

[Karanbir Singh]

Quit from thunderbird 52

downloaded 59.0b2/64bit/English-British/Linux

starter 59b2, made no changes in profiles etc, let it stabalise after checking all imap folders, did the Download/Sync and work offline. I can reproduce the behaviour. There is no change in status, still unable to access some of the emails after thunderbird says sync is complete. These are all still mostly all previously unread emails, with the odd previously read email also complaining about being missing. Note that this is an arbitary sample. I've not actally sat down and been through each and every email in my sub folders.

Comment 2

[Wayne Mery (:wsmwk)]

Thanks for teswting the beta

Comment 3

[Wayne Mery (:wsmwk)]

Michael, is this something you could test?

Comment 4

[gene smith]

Wayne, I am CC'd on this bug. I think maybe if the reporter tried the latest 60.x beta that the problem he sees may be fixed. The patch with Bug 1430480, now in beta, also fixes missing offline storage when a message is moved/copied via filters.
A question I have for the reporter would be what happens if you remain online after the "sync now"? Are the emails and any attachments properly accessible in the destination folders then?

Comment 5

[Karanbir Singh]

(In reply to gene smith from comment #4)
> A question I have for the reporter would be what happens if you remain
> online after the "sync now"? Are the emails and any attachments properly
> accessible in the destination folders then?

yes, inspite of being-offline, thunderbird does connect to the imap server and download the emails on demand.

Comment 6

[Karanbir Singh]

using thunderbird-60.0b5.tar.bz2 situation remains unchanged.

Comment 7

[Karanbir Singh]

(In reply to Karanbir Singh from comment #5)
> (In reply to gene smith from comment #4)
> > A question I have for the reporter would be what happens if you remain
> > online after the "sync now"? Are the emails and any attachments properly
> > accessible in the destination folders then?
> 
> yes, inspite of being-offline, thunderbird does connect to the imap server
> and download the emails on demand.

slight correction here - while thunderbird does change status bar message to be 'loading...' and the progress bar on the bottom right does indicate movement, there is no network activity and the email is NOT downloaded.

Comment 8

[gene smith]

Probably doesn't matter but I assume the filters you are using to do the copy are tb filters and not server-side filters?

It did work for me with just a simple one-message example with 52.7. However, I think there is still a problem based on other things I have seen.

For example, I have a gmail folder f3 that has offline store (a file f3 and f3.msf). I then set the folder property to not have offline store, shutdown tb and remove file f3. Then restart tb. As expected, the emails in f3 are accessible.

Then I return the folder property of f3 to again have offline store and press the "download now" button. Nothing downloads and the file f3 does not return. The emails are still readable in f3 (of course I am still online).

The only way I can cause a download and re-creation of f3 is to shutdown tb and remove f3.msf and restart tb. Then download occurs and f3 and f3.msf are re-created. 

You don't mention anything about changing the offline/online status of folders so this may not be exactly related.

You might also make sure the the preference "autosync" is enabled (it is by default) and the property to check for new emails for the problem folders is selected.

Did this work OK for you at one time and if now failing? 
 
> 
> slight correction here - while thunderbird does change status bar message to
> be 'loading...' and the progress bar on the bottom right does indicate
> movement, there is no network activity and the email is NOT downloaded.

I assume you are connected to the network at this point (i.e., online)? So are you saying you don't see the new email(s) in the destination folder? For the destination folder X please check if you have both files X and X.msf in your imapmail profile directory.

Comment 9

[Karanbir Singh]

Thanks for working with me on this,

(In reply to gene smith from comment #8)
> Probably doesn't matter but I assume the filters you are using to do the
> copy are tb filters and not server-side filters?

I have a mix of both, i see the same problems with folders that have sieve managed and local tb managed filters. This problem exists even with INBOX where no filtering has taken place.

> For example, I have a gmail folder f3 that has offline store (a file f3 and
> f3.msf). I then set the folder property to not have offline store, shutdown
> tb and remove file f3. Then restart tb. As expected, the emails in f3 are
> accessible.
>
> Then I return the folder property of f3 to again have offline store and
> press the "download now" button. Nothing downloads and the file f3 does not
> return. The emails are still readable in f3 (of course I am still online).
> 
> The only way I can cause a download and re-creation of f3 is to shutdown tb
> and remove f3.msf and restart tb. Then download occurs and f3 and f3.msf are
> re-created. 
> 
> You don't mention anything about changing the offline/online status of
> folders so this may not be exactly related.

I have all the folders subscribed, with account set to make content available offline, with max size set to disabled and no content expiry based on date etc. Unsure if this is worth mentioning, but the mail storage locally has been around for a while, and I've been upgrading pretty much on the clock when updates are available.


> You might also make sure the the preference "autosync" is enabled (it is by
> default) and the property to check for new emails for the problem folders is
> selected.

mail.server.default.autosync_max_age_days = -1
mail.server.default.autosync_offline_stores = true
mailnews.offline_sync_mail = true
mailnews.offline_sync_news = false
mailnews.offline_sync_send_unsent = false
mailnews.offline_sync_work_offline = true

> Did this work OK for you at one time and if now failing? 

Its been a problem for a while, Maybe last time this worked was in 2005 / 2006 era (?)

> I assume you are connected to the network at this point (i.e., online)? So
> are you saying you don't see the new email(s) in the destination folder? For
> the destination folder X please check if you have both files X and X.msf in
> your imapmail profile directory.

correct: thunderbird in offline mode, while i am on the network , clicking on an email with its body missing results in the bottom right status bar moving, status saying 'loading...' ; no email is displayed.

in :
/home/kbsingh/.thunderbird/GmocjwyR2.default/ImapMail/imap.gmail.com
-rw-------. 1 kbsingh kbsingh  824273777 May  3 17:17 0_inbox
-rw-rw-r--. 1 kbsingh kbsingh   18507056 May  3 17:58 0_inbox.msf

and for root folder:
-rw-------. 1 kbsingh kbsingh 1087985991 May  3 17:58 INBOX
-rw-rw-r--. 1 kbsingh kbsingh    8939553 May  3 17:58 INBOX.msf

Comment 10

[gene smith]

> mailnews.offline_sync_work_offline = true

Maybe this is set via UI somewhere. However, it defaults to false and I see it set false even when I set tb offline.

>mail.imap.use_status_for_biff is False

This defaults to true. When false it uses imap select instead of imap status when checking for new mail in folders. Seems like false should be OK but maybe not. Select puts more load on the server from what I understand.

> mail.imap.mime_parts_on_demand_threshold is set to 400000
> mail.imap.mime_parts_on_demand is False

These default to 30000 and true. However, since you have offline store and I assume you display attachments inline these probably don't matter.

(In reply to Karanbir Singh from comment #7)
> 
> slight correction here - while thunderbird does change status bar message to
> be 'loading...' and the progress bar on the bottom right does indicate
> movement, there is no network activity and the email is NOT downloaded.

Do you sometimes see the "progress bar" constantly moving? I've notice sometimes this is a symptom of a possibly inconsistent folder. You might try optionally backing up the X and X.msf files and then delete them and then let tb rebuild them while online (probably shutdown tb before you delete them). Another way is to do the repair folder property action but there is no option to backup the files with this and they just get re-written, so you still might want to back them up (of course, deleting/repairing these files has no effect on the imap server).

I think you are saying in the quoted comment that you do not see the copied email in the destination folder even if you stay online? I.e., NOT downloaded == don't see the email?

Also, how are you determining that there is "no network activity"? Tcpdump, wireshark, tb imap.log? I can suggest that you create an imap.log file that shows what happens, with as simple a failing example as possible. Google "thunderbird logging" for details (set two env vars and then run tb from cmd line). Then attach the imap.log file with some explanations as to what was happening.

Comment 11

[Karanbir Singh]

(In reply to gene smith from comment #10)
> (In reply to Karanbir Singh from comment #7)
> > 
> > slight correction here - while thunderbird does change status bar message to
> > be 'loading...' and the progress bar on the bottom right does indicate
> > movement, there is no network activity and the email is NOT downloaded.
> 
> Do you sometimes see the "progress bar" constantly moving? I've notice
> sometimes this is a symptom of a possibly inconsistent folder. You might try
> optionally backing up the X and X.msf files and then delete them and then
> let tb rebuild them while online (probably shutdown tb before you delete
> them). Another way is to do the repair folder property action but there is
> no option to backup the files with this and they just get re-written, so you
> still might want to back them up (of course, deleting/repairing these files
> has no effect on the imap server).

Stopped thunderbird, backed up files, removed them, started up thunderbird, it found the folder, downloaded only a few emails, but all headers, I asked to 'sync mail, work offline', it did download every email. Thunderbird went offline, I was able to browse to unread emails and see content just fine. Random test across 2 dozen emails and they all seem to be there. shutdown thunderbird, starter it again, email is still there, with the right 'READ' flag. Go online, and imap server is updated correctly to indicate read status for these emails.

The above process was run against a gmail imap account; i now reran the same operation on a cyrus-imapd hosted mailbox, and achieved similar results.

the new msf file is larger, but the new mailbox itself is smaller ( likely needed compaction ? ) ; note also, no change in actual emails added to this mailbox between the two listings below.

before:
-rw-------.  1 kbsingh kbsingh 824404429 May  3 21:56 0_inbox
-rw-rw-r--.  1 kbsingh kbsingh  18581477 May  3 21:57 0_inbox.msf

newly rebuilt:
-rw-------. 1 kbsingh kbsingh 815296457 May  3 22:47 0_inbox
-rw-rw-r--. 1 kbsingh kbsingh  19027101 May  3 23:14 0_inbox.msf

Additional data point, i removed a .msf file from a folder. Thunderbird is able to reproduce that, overall mail counts dont change - however, this does NOT result in the fix as above. Unread emails, not cached, remain uncached locally.

> I think you are saying in the quoted comment that you do not see the copied
> email in the destination folder even if you stay online? I.e., NOT
> downloaded == don't see the email?

correct

> Also, how are you determining that there is "no network activity"? Tcpdump,
> wireshark, tb imap.log? I can suggest that you create an imap.log file that
> shows what happens, with as simple a failing example as possible. Google
> "thunderbird logging" for details (set two env vars and then run tb from cmd
> line). Then attach the imap.log file with some explanations as to what was
> happening.

yeah, tcpdump on the remote imap server + lsof -i, to confirm no connection is (re)established.

will try and do the imap.log but it might take me a day or two.

Comment 12

[Wayne Mery (:wsmwk)]

Karanbir  any results?

Comment 13

[Karanbir Singh]

rebuilding the mailbox's resulted in the problem going away for a few days, unfortunately I was back on the road for nearly 3 weeks flat out, got back yesterday; during this time the online/offline pattern got quite hectic and as a result I'm back to the same problem as before.

Is it possible that if a download-session does not complete ( say, if I were to suspend the laptop midway through an operation ), that part of the process is lost entirely and not re-attempted ?

Comment 14

[gene smith]

If I understand right, you normally run with offline storage enabled on *all* your folders. If that's the case then all emails should already be stored in file X for folder X so, theoretically, there should be no need to download emails when you set tb to offline mode. If you were to just take tb offline *without* trying to download anything, do see the error when you access some messages?

Assuming you do see the error, try this: Set tb back online and access the "bad" message plus any attachments it might have, then restart tb (just to clear any memory caches).  Now go back offline (again, don't download anything). Is the "bad" message now readable in tb offline mode?

Comment 15

[gene smith]

(In reply to Karanbir Singh from comment #13)
> 
> Is it possible that if a download-session does not complete ( say, if I were
> to suspend the laptop midway through an operation ), that part of the
> process is lost entirely and not re-attempted ?
 
Good question. I'll have to try it sometime when I'm feeling lucky. :)

Comment 16

[Wayne Mery (:wsmwk)]

Karanbir?

(In reply to gene smith from comment #14)

If I understand right, you normally run with offline storage enabled on
all your folders. If that's the case then all emails should already be
stored in file X for folder X so, theoretically, there should be no need to
download emails when you set tb to offline mode. If you were to just take tb
offline without trying to download anything, do see the error when you
access some messages?

Assuming you do see the error, try this: Set tb back online and access the
"bad" message plus any attachments it might have, then restart tb (just to
clear any memory caches). Now go back offline (again, don't download
anything). Is the "bad" message now readable in tb offline mode?

Comment 17

[Wayne Mery (:wsmwk)]

Think we need a protocol log.
https://wiki.mozilla.org/MailNews:Logging
IMAPOffline:5,IMAP:5,timestamp

Comment 18

[Karanbir Singh]

Ver: 78.4.0 (64-bit)

Once again, I deleted my local mail ( 28 GB ), let it resync over a week. It takes that long due to service side rate limiting. I'd like to confirm RESOLVED FIXED state here, the last 48 hrs I've tried to reproduce the issue with bad network and premature thunderbird termination etc and not had the problem on this mailbox.

Happy to leave this as resolved, and will re-open if problem comes back.