195737 - Junk mail flag should be stored in mbox message flags, in addition to the msf. file

Status: NEW

(MailNews Core :: Database, defect)

Description

[u69748]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3) Gecko/20030301
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3) Gecko/20030301

Currently junk mail flag is stored in .msf file.
So, if we remove broken .msf file (sometimes we need to remove it 
in mail trouble), junk mail information will be lost. 
The flag should be stored in mbox.


Reproducible: Always

Steps to Reproduce:

Comment 1

[Boris Zbarsky [:bzbarsky]]

Junk mail stuff is under filters, I think....; we could indeed toss in a header
with the spam status into the mail.

Comment 2

[Blu3]

This would be incredibly useful, particularly in the case of multiple clients
accessing the same imap mailbox.  I.e. my desktop and my laptop, or a shared
imap folder.

Comment 3

[(not reading, please use [e-mail] instead)]

mass re-assign.

Comment 4

[David Favor]

Fixing this bug will probably also fix 193229 and 207389.

Comment 5

[hjp]

*** Bug 274933 has been marked as a duplicate of this bug. ***

Comment 6

[hjp]

Re comment #2:
> This would be incredibly useful, particularly in the case of multiple clients
> accessing the same imap mailbox.

Yes, but only if the training.dat file is also shared between clients (stored in
an IMAP folder?). Otherwise the displayed junk status won't correspond to the
"real" junk status (computed from the local training.dat) which may mess up the
training badly.

Comment 7

[hjp]

[Sorry, hit the Commit button too soon]

If the training.dat cannot be shared between clients it is probably better to
reevaluate the junk status for each message not in the .msf file. (Yes, that may
take some time with large folders, but hopefully you don't trash your .msf files
that often and at least then you have up to date information)

Comment 8

[:aceman]

I think the component should be MailNews: database. Try it.

Comment 9

[:aceman]

I don't think this is an enhancement but a dataloss bug. Msf files are often
rebuilt (system crash, daylight saving change) and the junk flags are then lost.
User has to mark messages again, which causes them to attribute to the
training.dat file twice (or again).
But I leave it to the assignee to set the flags as he thinks.

Comment 10

[Jef Driesen]

(In reply to comment #7)
> Yes, but only if the training.dat file is also shared between clients (stored in
> an IMAP folder?). Otherwise the displayed junk status won't correspond to the
> "real" junk status (computed from the local training.dat) which may mess up the
> training badly.

(In reply to comment #7)
The training.dat file is dynamic in nature (e.g changes by means of training). This means the "displayed" junk status (calculated at some point in the past) doesn't have to correspond with the "real" junk status (calculated now).

> If the training.dat cannot be shared between clients it is probably better to
> reevaluate the junk status for each message not in the .msf file. (Yes, that may
> take some time with large folders, but hopefully you don't trash your .msf files
> that often and at least then you have up to date information)

I think re-classifying is a very bad idea. Once a message is classified as junk/notjunk that information should be considered correct. If the automatic classification made a mistake, you can correct manually, but after that the junk status is correct. If you are going to classifying again (because of the removed msf file), you'll also have to check for mistakes again. Once a message is junk/nonjunk, it should stay that way, unless you change the status manually of course. This means after classification it is not relevant anymore which training.dat file was used to perform the classification.

Comment 11

[Richard Nott]

Win XP/ Thunderbird version 2 beta 2 (20070116)

My symptom:  On using "Rebuild Index" on local folders, junk flags are lost.  Always reproducible.

I assume that this is another result of this same issue and not a new bug.

Comment 12

[Daniel Beardsmore]

Bug 274933 is marked as a duplicate of this one, but I am a little unsure of the connection. Bug 274933 also looks similar to mine, bug 367010 – is it likely to be a related issue?

Comment 13

[Kent James (:rkent)]

The approach that I have been taking to metadata such as junk status is to clean up the cases where that data is needlessly blown away. For example, bug 449768 deals with the specific issue from comment 11. Other bugs, including for example bug 459680, will solve other specific issues. The goal is to have databases files that are viewed as valuable data to be preserved, and not as an index that can be thrown away and easily recreated. I also don't think mailnews should be expected to work correctly if the user manually deletes system files such as these .msf files.

Concerning junk, I think that the prevailing attitude is that these messages are worthless, so the proper approach is to delete them, not to waste valuable prime real estate, like the metadata space in the mbox file headers, to preserve information about them.

For the IMAP case, this bug is irrelevant BTW, as the mailnews code already uses the available server capability (user-defined flags) to store status. There is no capability to modify the message in IMAP to add additional headers with status.

Comment 14

[David Fraser]

(In reply to comment #13)
> Concerning junk, I think that the prevailing attitude is that these messages
> are worthless, so the proper approach is to delete them, not to waste valuable
> prime real estate, like the metadata space in the mbox file headers, to
> preserve information about them.
I don't think 1 bit's worth of space in the metadata space is a waste. Agreed that the messages are junk, but the fact that they are junk is valuable information that can be used while they are temporarily around.

Comment 16

[Wayne Mery (:wsmwk)]

bienvenu, wontfix, since bug 449768 resolved the .msf issue? Of not, should this depend/wait on one of the storage format bugs?

Comment 17

[guanxi]

FWIW, msf files can be lost for reasons besides reindexing.

Comment 18

[:aceman]

What is the current state of the problem? And is this IMAP only?