Open Bug 495666 Opened 16 years ago Updated 4 months ago

If mark Read/Unread, Add/Remove Tag etc. is executed while "Compact Folder" is running, marking/tagging is lost after end of compaction

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Platform:
x86
All
Type:
defect

Tracking

(Not tracked)

People

(Reporter: World, Unassigned)

References

(Blocks 3 open bugs)

Details

[Build ID] > Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1pre) Gecko/20090527 Shredder/3.0b3pre If "Mark Read/Unread"(update of X-Mozilla-Status:) is executed while "Compact Folder" is running, marking is lost after end of compaction. I think it's same for tag, because "Add/Remove Tag" is update of X-Mozilla-Keys: header. (Note-1:) I didn't check "Mark As Junk/Non-Junk" case. (Note-2:) I don't know Tb 2's behaviour. [Steps to produce] 1. Create big mail box which takes long to compact. Mark all mails as "Read". Shift+Delete for a mail. 2. Open 2 Tb windows. Open the mail folder at both Tb windows. 3. Window-1 : "Compact Folder" of the mail folder. 4. Window-2 : Select All, Mark As Unread => Unread mail count is shown. 5. After end of compaction, Unread status is lost. (Note-3:) If Tb trunk, error message of "other process is using folder" was displayed when "Shift+Delete" was executed while "compact folder" is running. So phenomenon of Bug 370409 couldn't be observed. I don't know Tb 2's behaviour.
This clearly falls into my war on dataloss area, so I am assigning it to myself to keep it visible to me. But if someone feels inspired to fix it, you won't hurt my feelings.
Assignee: nobody → kent
Bug 368768 seems "Mark As Junk/Non-Junk" case.
"Add Tag" case was better to observe phenomenon. Tag of mails already copied to nstmp is lost.
If "Add Tag" case, loss of tag data in X-Mozilla-Keys is also observed, even when tag is kept at thread pane. (1) mail_1 to mail_M: (1-A) Tag at thread pane is lost. (1-B) No added tag data in X-Mozilla-Keys: header (2) mail_(M+1) to mail_N: (2-A) Tag at thread pane is kept. (2-B) No added tag data in X-Mozilla-Keys: header (1-B) & (2-B) is possibly cause of some status losses after ordinal compaction.
Can confirm the same happens on Linux. I share my profile between Windows and Ubuntu and this happens pretty often. It is pretty irritating, because the Unread messages tally gets horribly messed up (e.g I actually have 10 unread messages in Inbox, but the sidebar shows Inbox(3), etc) and I have to remember which messages were unread, and mark each of them read then remark them as unread. If any other information is required, I will be glad to provide. Thanks. Umang
Also, if I move a file from Inbox to another folder in Local folders, the Inbox copy stays, and a copy is made to the other folder. If I don't realize this, then I move the mails again after the compacting is done and find two copies in the other folder, both unread.
(In reply to comment #5 and comment #6) Umang, what is your auto-compact setting? > mail.prompt_purge_threshhold = true/false > mail.purge_threshhold = NNNN (threshold value in KB, default=100KB) > mail.purge.ask = true/false If you set prompt_purge_threshhold=true, set mail.purge.ask=true and restart Tb. You can cancel auto-compact by "Cancel" reply. (I recommend you to put it in user.js, because it can be set to false at dialog.) If small purge_threshhold value, dialog is displayed frequently and it's annoying. Increase the value. (In reply to comment #6) > if I move a file from Inbox to another folder in Local folders, > the Inbox copy stays, and a copy is made to the other folder. If compaction of move source folder is executed while bulk move between local mail folders is running, "delete from move source folder step of bulk move" fails. It's known phenomenon in some other bugs. But I don't know bug for this specific issue. "bulk move" case is different from this bug (this bug is for status update case. bulk move contains "delete of mail".) Umang, open separate bug for it, please. Note: If you set mail.purge.ask=true, you can always test with "Shift+Delete of some mails" and "manual compaction". If you create dummy POP3 account(no Global Inbox use), you can do "bulk move" easily by message filter of "Match all messages, Move to a local folder".
OS: Windows XP → All
Summary: If mark Read/Unread is executed while "Compact Folder" is running, marking is lost after end of compaction → If mark Read/Unread, Add/Remove Tag etc. is executed while "Compact Folder" is running, marking/tagging is lost after end of compaction
OK. Just a question before I post a new bug, so I'm sorry that I'm off-topic. What is user.js? Where can I find it (not there in my profile folder) How do I edit it? Why can't I change from config editor? (haven't exactly fooled around with Tb before) Anyways, so what you want me to do, is to try to move, copy, delete mails when compacting is going on and also check how often compacting happens by making Tb ask me before compacting? Then report a bug on errors while moving and deleting files when a folder is getting compacted. If yes, should I also explain about how read/unread doesn't change while compacting? Also, here're my settings: mail.prompt_purge_threshhold = true mail.purge_threshhold = 200 mail.purge.ask = false Thanks WADA, Umang
(Off-Topic) > What is user.js? Where can I find it (snip) How do I edit it? Google search for "thunderbird user.js" may help you. > Why can't I change from config editor? You can do it via Config Editor. However, if you check "Do this automatically ..." in dialog before auto-compact, mail.purge.ask is set to false. If mail.purge.ask=true is set by user.js, you can recover from mail.purge.ask=false by "Restart of Tb" only. > so what you want me to do, (snip) Simply (1) to reproduce problem, with { auto-compact=off } or { auto-compact=on + mail.purge.ask=true + "Cancel reply to dialog" }, with manual "Compact Folder", (2) to report separate bug.
Blocks: 498274
Blocks: 498278
Assignee: kent → nobody
Blocks: 286888
Severity: normal → S3

Should this issue be gone with the compact rewrite?

Flags: needinfo?(benc)

(In reply to Wayne Mery (:wsmwk) from comment #10)

Should this issue be gone with the compact rewrite?

No, I think it'll be the same. The compaction locks the folder, but there are loads of places which fiddle with things, ignoring the locking.
A bit part of the issue is that there isn't really any good definition of what the locking is suppose to protect - it's kind of a catchall covering the folder, database and local store, but it's not a good model.

For this issue, I think quick/easy thing would be to check the lock in the folder nsIMsgFolder functions, and fail the operation (NS_MSG_FOLDER_BUSY) if it is.
These functions would need the check (there's likely a few others I've missed):

.markMessagesRead()
.markAllMessagesRead()
.markMessagesFlagged()
.markThreadRead()
.markMessagesRead()
.addKeywordsToMessages()
.removeKeywordsFromMessages()

The lock checking shouldn't be done down at the database level (things like compaction, "repair folder" and IMAP checking messages hold a lock and call the DB-level functions, so they'd fail).
It also shouldn't be done up at the GUI level - way too much noise to add to code that shouldn't have to deal with it.
So, folder level it is.

It's possible that some of those folder-level functions are called in contexts where a lock is already held, which would complicate things for any server-linked protocols (IMAP, EWS etc).

In any case, I'd say the first step here would be to implement failing unit tests to simulate mark/tag operations being lost during compaction, to reliably demonstrate the issue...
Then write fixes to the code to make the tests pass.

Flags: needinfo?(benc)
You need to log in before you can comment on or make changes to this bug.