Open Bug 1776823 Opened 1 month ago Updated 3 days ago

Thunderbird offline autosynchronization not happening

Categories

(MailNews Core :: Backend, defect, P2)

Thunderbird 102

Tracking

(thunderbird_esr102+ affected)

Tracking Status
thunderbird_esr102 + affected

People

(Reporter: KaiE, Assigned: gds)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 1 obsolete file)

Offline IMAP folder synchronization isn't working for me on 102.

I had a folder with missing messages in offline mode. We suspected some folder corruption. I quit Thunderbird, identified the file in my profile folder below ImapMail by its folder name, and delete both primary and .msf files.

I started Thunderbird, and it re-synched the headers of the folder.

However, after having kept it running for 8 hours, overnight with no other activity, the folder still was NOT downloaded. (To confirm, I quit Thundebird again, and I only had a .msf file, no mail contents.)

The folder is included in my message sync advanced list of folders. Right click folder, properties, sync, also shows "select for offline use" is checked.

Sounds like the feature is broken, or randomly nonworking for some folders.

This is an old Thunderbird profile, but I think it should work anyway. We should find out what's causing this failure to sync.

Who understands how this is supposed to work?

Do we have a background timer, which routinely checks if all configured folders have been fully downloaded already?

Why do we even have a button "download now" in folder properties? Isn't that supposed to be done automatically?

My impression is, the syncing happens for new arriving emails, and also when copying to a folder.

However, there's apparently no automatism to re-sync old folders with incomplete offline cache.

Maybe this state can be reached, if settings in the past didn't ask for offline sync (I might have had this disabled in the past).

Also, the default workflow for going offline involves the question "want to download messages before going offline?". It seems this was done, because otherwise offline availability of messages isn't guaranteed.

I'm concluding: Simply enabling the prefs for offline sync is insufficient to have full local offline data. You'll have to use the UI to go offline and allow to download messages to ensure it.

If this is the intended behavior, we can close this bug.

I think this is handled by something called "autosync". When you visit a folder and you have offline store enabled for the folder, any new messages are downloaded and stored in a/the file (mbox or maildir). There is no downloads and store of new messages for a folder until you visit (open) it except when new messages are checked for Inbox or any other folder that is set to be checked for new mail or if the folder has recently been opened and imap IDLE is in effect for the folder to also detect new mail. If autosync is disable (via a pref) messages are not downloaded and stored until they are explicitly opened.

However, I also see a somewhat intermittent problem like what you describe. If I delete the mbox and .msf file for a folder with offline store and then visit that folder, I see the messages appear in the list. But I also sometime see no .msf or mbox file appear and nothing is downloaded except when a message is opened, and this doesn't create even the .msf file for the folder. Only by repairing the folder do I see the .msf and mbox files appear and download of messages occur. But sometimes this works fine and a repair is not need and both files appear and download occurs upon opening (visiting) the folder.

I just now saw this on 91.9 and on 102rc-build-2 so it doesn't seem to be a new bug.

I'll look closer at this tomorrow.

Blocks: tb102found

(In reply to gene smith from comment #3)

I'll look closer at this tomorrow.

Don't suppose you found something?

Summary: Thunderbird offline synchronization not happening → Thunderbird offline autosynchronization not happening

(In reply to Magnus Melin [:mkmelin] from comment #4)

(In reply to gene smith from comment #3)

I'll look closer at this tomorrow.

Don't suppose you found something?

No not yet, other things came up but I see it when testing stuff. For example, I copy new messages to a folder and they sometimes don't appear in the mbox file until I repair the folder. But usually they do appear, so it's definitely not consistent.

Flags: needinfo?(gds)
OS: Unspecified → All
Priority: -- → P2
Hardware: Unspecified → All

(In reply to Kai Engert (:KaiE:) from comment #0)

Offline IMAP folder synchronization isn't working for me on 102.

I had a folder with missing messages in offline mode. We suspected some folder corruption. I quit Thunderbird, identified the file in my profile folder below ImapMail by its folder name, and delete both primary and .msf files.

First of all, just to be clear, deleting .msf and mbox files to fix a busted folder is not typically suggested as the 1st step for end users. Instead, right-click->Properties->Repair Folder is the documented method. However, removing the .msf and mbox files is something I do a lot while debugging problems.

I started Thunderbird, and it re-synched the headers of the folder.

However, after having kept it running for 8 hours, overnight with no other activity, the folder still was NOT downloaded. (To confirm, I quit Thundebird again, and I only had a .msf file, no mail contents.)

Probably the first thing to check is make sure autosync is enabled. Possibly at some point mail.server.default.autosync_offline_stores got set false. Without this true, individual messages are only downloaded and stored in the mbox file when they are opened.
Also, make sure there were no attempts to create account specific autosync settings with mail.server.serverX.autosync_offline_stores where X is the server/account index/key number, 1, 2, 3 ....
Another thing is check to make sure mail.imap.use_status_for_biff is true (the default). I think I saw somewhere that this is needed for autosync.

I did this several times on two accounts with successful results by following these steps:

  • Shutdown TB
  • Delete the <folder-name>.msf file for folder having offline store, e.g., INBOX.msf
  • Delete the <folder-name> mbox file, e.g., INBOX
  • Repeat the above 2 step for any other folder in the account you want to re-download that has offline store configured
  • Start TB and select any folder in the same account
  • Autosync requires TB be "idle" for 10 seconds before it starts, so don't do anything in TB for at least 10 seconds.
  • First, all the headers for the removed files are downloaded and stored in re-created <folder-name>.msf files
  • After header downloads finish each full message in <folder-name> having offline store is downloaded and stored in <folder-name> mbox files

If there are enough messages in the folders so that the download is not almost instantaneous, you should see the in the status bar at the bottom the activity for downloading the headers for all folders followed by slower progressing activity for downloading the full messages for all folders which are added to the <folder-name> mbox files.

However, I tried this again with a folder that is a subfolder of Inbox that has offline store. But on restart that folder didn't automatically download and create a new mbox file containing all the messages. I had to click the subfolder of Inbox before it downloaded and created the mbox file. I don't have to do this, it seems, with Inbox or other folders at the same level as Inbox.
Possibly the folder that didn't download for you was a subfolder? Also, did clicking on/opening the folder finally trigger the download and mbox creation?

Other that this, it appears that autosync is downloading and re-creating and populating the user-deleted .msf and mbox files properly.

I will look closer at the code for autosync and try to determine why lower level folders are not automatically autosync-ing and require that the folder be clicked (opened).

Flags: needinfo?(gds)

However, I tried this again with a folder that is a subfolder of Inbox that has offline store. But on restart that folder didn't automatically download and create a new mbox file containing all the messages. I had to click the subfolder of Inbox before it downloaded and created the mbox file. I don't have to do this, it seems, with Inbox or other folders at the same level as Inbox.

After more testing and code examination, my observation of a lack of automatic autosync for subfolders is a red-herring. It had nothing to do with the level of the folder in the hierarchy.

After a lot more testing and debugging I now see that there is a real problem with autosync that will cause a folder with offline store to never update even when new messages are added to it either by new mail arriving naturally into Inbox, messages being filtered into the folder or messages copied to to the folder by another client. So what is described in comment 0 is confirmed.

Unless the folder is opened by the user, new messages will stop being detected if the imap folder status, triggered on biff interval after TB enters the idle state (10s after the last keyboard or mouse activity in a TB window), detects a change in any of several counts (e.g., number of messages in folder), a folder update (an imap select url) is triggered by autosync. But if the folder update response doesn't detect the need to download any missing message bodies, the autosync state gets stuck in "stUpdateIssued" and never returns to the initial state, "stCompletedIdle", locking out any further autosync downloads for the folder. Also, the autosync manager state never returns to "completed" which also stop any further autosync new message detection and download.

The last change to autosync functionality was bug 428266. Before that, TB always did just a folder update (imap select url) to detect new messages needing to be downloaded. As part of the fix for that bug, instead of always doing an update, first a status (imap folderstatus url) is done to see if anything has really changed. This is a lot less network activity since typically nothing really changes and a imap STATUS command produces a lot less network traffic than the imap SELECT and flag re-fetches used for update. So only if something appears to be changed, indicated by the status response, is the full folder update (imap select and full flag fetch) done.

The problem occurs because imap STATUS returns 4 values such as MESSAGES 21 RECENT 0 UNSEEN 2 UIDNEXT 1173.
If autosync sees a change in any of these, it triggers an update. In my testing, there is often a change seen in the UNSEEN value even though there was no change in the number of unseen (a.k.a, unread) messages in the folder. This causes the update to be triggered and update detects no real changes, causing the autosync state for the folder to get hung in "stUpdateIssued".

So far I've haven't found out exactly why the unread (unseen) count appears changed. (I think also the recent count may trigger a false update but I haven't verified it.)

I've made some experimental changes (see next attachment) to the autosync and related code to tolerate the false change indications from status. However, I still need to find out why the the false indications occur. If it can't be fixed with reasonable effort then I might propose just removing the change check for UNSEEN and maybe RECENT from the autosync code since the number of MESSAGES changing is the primary reason to download the new messages and that is detected accurately.

See Also: → 428266
Attached patch autosync-debug-changes.diff (obsolete) — Splinter Review

Here's my changes to autosync that I've been working on. Most of the changes are just printfs or MOZ_LOGs for better debugging so I can try to see what is going on better. It also temporarily changes the update and discovery timers so that I don't have to wait 1 hour before something is added to the queues.

The actual fix to handle updates that find no new messages to download is marked with "fixes it".

Assignee: nobody → gds
See Also: → 1373940

The reason false changes are detected, which triggers a folder update that finds no need to download anything, is caused by a couple of factors.

The first one is a parsing error that appears to have been present since day one. It occurs when trying to determine the number of unread (UNSEEN in imap parlance) messages. The UNSEEN response from imap STATUS returns the correct value. However, the UNSEEN response from imap SELECT is not really the number of unseen (unread) messages but is instead the message sequence number of the 1st unseen message in the mailbox. So the UNSEEN response from SELECT should be ignored when determining the number of unread messages.

But just ignoring the UNSEEN response code from SELECT doesn't completely fix the problem.

The other problem is that the folderCache.json item serverUnseen gets set or set back to 0 from the correct value that is returned in the imap STATUS response. The reason is because the value written to serverUnseen is initialized to zero and imap STATUS does not always occur during a session. Also, during a folder update (select URL) the value to be written to serverUnseen is first zero'd but, as mentioned above, the UNSEEN value from SELECT is not really the number of unseen messages so it can't be used so the value remains, often incorrectly, zero.

Autosync reads the value from folderCache serverUnseen and compares it to the value from the STATUS response. Since zero is usually stored in cache, this can appear to be a false change and triggers the update (SELECT) and finds nothing needing downloading.

The vars for handling the UNSEEN value from STATUS are all signed int32_t but negative UNSEEN count is never valid. So my solution is change the var initialization to -1 instead of as they currently are now, initialized to 0 and avoid saving 0 back into folderCache serverUnseen when imap STATUS has yet to occur.

This WIP diff keeps the fix from the previous diff (mark obsolete) and just adds the initialization to -1 fix described above.

Attachment #9287313 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.