757328 - Rapid archiving of messages results in message loss (archive from local folder to local Archives/yymm. Except first selected message, mail data is not appended to target. At source, msDBHeader is deleted but EXPUGE bit is not written in X-Mozilla-Status)

Status: UNCONFIRMED

(Thunderbird :: Folder and Message Lists, defect)

Description

[Samir]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0
Build ID: 20120420145725

Steps to reproduce:

Rapidly archive messages in succession (e.g., by pressing the "a" key several times in quick succession when focus is on the message list).


Actual results:

The first message was successfully archived, but the next one disappeared. An empty message was created in the archive folder instead. 

Deleting the .msf file associated with the folder caused the second message to reappear there. It appears that the msf file is being updated, but actually moving the message to the archive mailbox fails when you hit "a" before the previous archive operation has been completed.


Expected results:

All the messages should have been properly archived.

Comment 1

[WADA:World Anti-bad-Duping Agency]

Which case?  (local mail folder : POP3 or "Local Folders")
  Source mail folder     Archives folder
    local mail folder      local mail folder
    local mail folder      IMAP  mail folder
    IMAP  mail folder      local mail folder
    IMAP  mail folder      IMAP  mail folder(same account)
    IMAP  mail folder      IMAP  mail folder(different account)

If IMAP folder is relevant, Offline-use=On? Off?
(folder's Properties/Synchronization)

If IMAP folder && Offline-use=On, when did you do "Rapidly archive messages in succession"?
Mails just arrived are archived rapidly? Or mails already fetched(saved in offline-store file by auto-sync) are archived rapidly?

If arcives folder is IMAP folder, what is "Archive options..." setting of Copies&Folders?
Is subfolder of Archives created upon your each archive operation?
Or mails are moved to already existent folder by archive operation?

> Deleting the .msf file associated with the folder caused the second message to reappear there.

At which mail folder? Source mail folder? Or Archives folder? Or both folders?

> Build ID: 20120420145725

Do you still use Tb 12.0.0? (Tb 12.0.1 is dated 20120428)

Comment 2

[Samir]

(In reply to WADA from comment #1)
> Which case? 

Everything is offline. No IMAP.

local mail folder -> local mail folder

> Mails just arrived are archived rapidly? Or mails already fetched(saved in
> offline-store file by auto-sync) are archived rapidly?

Mail already fetched (fully downloaded and saved in mailbox file).

> Is subfolder of Archives created upon your each archive operation?
> Or mails are moved to already existent folder by archive operation?

Mail is being moved to an already existing folder ("Archives -> 2012" in this case).

> > Deleting the .msf file associated with the folder caused the second message to reappear there.
> 
> At which mail folder? Source mail folder? Or Archives folder? Or both
> folders?

Deleting the msf file of the source folder causes the lost message to reappear in the source folder. The archive folder is fine (except for an empty message being placed there, which you have to delete).

> Do you still use Tb 12.0.0? (Tb 12.0.1 is dated 20120428)

The build ID is automatically reported from my web browser (FF 12.0) and not my email client. I am using Thunderbird 12.0.1.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Samir from comment #2)
> local mail folder -> local mail folder
> Mail is being moved to an already existing folder ("Archives -> 2012" in this case).

It looks. Local Archives folder, Year, no folder structure.

To avoid misunderstanding your setting from words only, can you show us your actual archive setting in prefs.js?
=> find serverN for the POP3 account
> mail.server.serverN.type == pop3
> mail.server.serverN.hostname or mail.server.serverN.realhostname == your server
> mail.server.serverN.userName or mail.server.serverN.realuserName == your ID
=> find accountP for serverN
mail.account.accountP.server == serverN
=> get main identity number for accountP 
mail.account.accountP.identities == idX,idy,idz,...
=> archive setting for the main identity of accountP
> mail.identity.idX.archive_enabled
> mail.identity.idX.archive_folder
> mail.identity.idX.archive_granularity
> mail.identity.idX.archive_keep_folder_structure
> mail.identity.idX.archives_folder_picker_mode

(In reply to Samir from comment #0)
> Steps to reproduce:
> Rapidly archive messages in succession (e.g., by pressing the "a" key
> several times in quick succession when focus is on the message list).

At which Folder? Ordinal local mail folder at Folder View=All? Or Unified Folder at Folder View=Unified Folder?

When "pressing the a key several times in quick succession", is "archived mail by previus a key press" removed from thread pane or removed from selected mails(highlighted at thread pane)?

An "archive mails" roughly consists of;
(1) Selected messages are grouped by,
    SourceFolder+TargetFolder(ArchiveFolder,Year,Month,Structure).
    and msgDBHeader of archived mails is held for each group
(2) For each group, if target subfolder does't exist, create the subfolder
(3) For each group, move mails held fr the group to archive target folder.

> Deleting the msf file of the source folder causes the lost message to reappear in the source folder.
> The archive folder is fine (except for an empty message being placed there, which you have to delete).

Because (2) is asynchronous process at least when IMAP, I assume that all archive requests by "a" key press is queued by step (1) before actual "move mail from source folder to Archives[/Year[/Month[/Structure/...]]] by step (2)/(3) for each a key press" is executed.

In this case, at target folder, mail data of msgDBHeader_N(MsgKey==OffsetN because local mail folder) is appended to target msgFolder for first "a" key press, and msgDBHeader_N is removed from msgDatabase of source folder. Mail data itself is not removed, and EXPUNGED flag will be written to X-Mozilla-Status: of the mail.  

When archive for second or later "a" key press is executed, requested msgDBHeader_N doesn't exist in source folder. This perhaps produces phantom msgDBHeader in target folder.

Because archived mail re-appears by "delete .msf of source folder", it looks that "writing EXPUNGED flag to X-Mozilla-Status: of the archived mail" was fortunately not invoked.

Because only first selected message looks copied to target folder, it looks for me;
- "archive of msgDBHeader_1 by first a key press" is executed normally
- "archive of msgDBHeader_2 to N by first a key press" is interfered 
   by "archive of msgDBHeader_1 to N by second or later a key press"
- "archive of msgDBHeader_1 to N by second or later a key press"
  interferes each other, or internal failure occurs in archive execution
  because requested mail is already deleted at source folder
  by "archive of msgDBHeader_1 to N by former a key press"

Comment 4

[Samir]

(In reply to WADA from comment #3)
> To avoid misunderstanding your setting from words only, can you show us your
> actual archive setting in prefs.js?

Yes:
user_pref("mail.server.server1.type", "pop3");
user_pref("mail.server.server1.realhostname", "pop.googlemail.com");
user_pref("mail.account.account1.server", "server1");
user_pref("mail.account.account1.identities", "id1");
user_pref("mail.identity.id1.archive_folder", "mailbox://[e-mail]/Archives");

All the other archive settings you mention are not present in prefs.js (i.e., they take the default values).

> At which Folder? Ordinal local mail folder at Folder View=All? Or Unified
> Folder at Folder View=Unified Folder?

It doesn't matter which folder - it happens everywhere.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Samir from comment #4)
> All the other archive settings you mention are not present in prefs.js
> (i.e., they take the default values).

Can you paste from Config Editor(Tools/Options/Advanced/General, Config Editor)?
(should we do additional search works to know what is default of Tb?)

> > At which Folder? Ordinal local mail folder at Folder View=All? Or Unified
> > Folder at Folder View=Unified Folder?
> It doesn't matter which folder - it happens everywhere.

It's understandable, because data of selected messages was obtained by Archive code from gFolderView.SelectedMessages which is Array of msgDBHeader of selected messages at thread pane. It doesn't depend on currently used View itself.
Thanks for the info. View related problem is surely ruled out.

Comment 6

[Samir]

(In reply to WADA from comment #5)
> Can you paste from Config Editor(Tools/Options/Advanced/General, Config
> Editor)?

Sure:
mail.identity.default.archive_enabled; true
mail.identity.default.archive_granularity; 1
mail.identity.default.archive_keep_folder_structure; false
mail.identity.default.archives_folder_picker_mode; 0

Comment 7

[po]

I have suffered several times (i.e. lost some data!) due to this bug over a period of a few years, a few times on my own installation and once on that of a family member who I persuaded to try Thunderbird and whom I therefore provide support for. Once I realized it was a bug, we stopped using the archive function.
The bug has occurred for us when archiving from online IMAP folders to local storage dedicated archive folder. (We have not tried the archive command in local-to-local configurations).
In one case I managed to restore most of the lost e-mails by importing from folders extracted from my local system backup media. But it was not a perfect process (the status of restored e-mails was not correctly restored) and it took a lot of time.

The existence of the bug means that we do not dare to use the Archive command, since it is inherently unreliable in a catastrophic way, nor do I dare to recommend the use of Thunderbird to others. Instead we have to follow a more careful and error-prone manual path for archiving (something along the lines of "sync, go offline, copy all e-mails to be archived into the appropriate local folder, check, then go online again and delete all the source e-mails from the original folder"). Considering that the bug is 10 years old the unresolved status is devastating for any promotion of Thunderbird to others, particularly non-technical users. In the typical setup of users here in Denmark, a reliable archive function would be one of the most desirable features of Thunderbird. Since IMAP storage is expensive, users will eventually run out of space, and the convenient archive function appears to "normal" users as a safe and reliable way of reducing server storage use while moving e-mails to local (backed-up) storage. That is, until they are hit by this bug.

I have now created an extra e-mail account which I can use from time to time in order to see if I can trigger the bug.
If somebody from the Thunderbird programming and/or test team could be assigned to this bug, I would be willing to enter into a dialogue and run some more experiments in order to hopefully find a systematic way of triggering the bug. I currently run Thunderbird 91.11.0 (64-bit) on Linux (Ubuntu LTS 18.04).
An alternative to actually solve the bug in the current "archive" command could be to develop a better archiving function which does not immediately delete the source e-mails but instead offers a function allowing the user to request bulk-deletion of those original e-mails which have been archived from IMAP folders. This second stage of the archiving would be requested when one has assured that the copies in local storage folders have been copied to backup media.
I don't know of any e-mail client which provides such a function, but I would love to have it available.