194020 - timestamp differences between OS causes mail summary files (.msf) to be rebuilt when used across platforms (Windows 2000 - Mac OS X)

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Wolfgang Eschner]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; de-AT; rv:1.2.1) Gecko/20021130
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; de-AT; rv:1.2.1) Gecko/20021130

When I copy or sync mail folders and mail summary files across platforms
(Windows 2000 - Mac OS X 10.2.3), the mail status is not preserved.
Example: 
In Windows 2000: 10 mails in folder "test", 2 of them unread. After copying
files "test" and "test.msf" to OS X, the same mails are shown, but all of them
unread.


Reproducible: Always

Steps to Reproduce:
1. Start Mozilla Mail on PC (Windows 2000), open any mail folder underneath
"Local Files", example here: folder "test". Mark some mails as "read", close Mozilla
2. Copy "test" and "test.msf" from PC to Mac 
3. Start Mozilla Mail on Mac (OS X 10.2.3), open folder "test": all mails have
"unread" status.

Actual Results:  
mail status not preserved across platforms

Expected Results:  
preserve mail status across platforms

Note 1:
This is true for folders underneath "Local Files" including the mandatory folder
like "Sent", "Drafts". It is not true (i.e., copying or syncing Works) for
folders underneath individual mail accounts.
Note 2:
Upon startup, Mozilla Mail/News under Mac OS X deletes all .msf files underneath
"Local Files" and only creates new ones when the folder is opened or compacted.
Note 3:
The affected folders contain mail which was imported from Outlook Express v6.0

Comment 1

[David :Bienvenu]

the .msf files can be read fine, but what's wrong is the timestamp used in the
.msf file to make sure the corresponding mailbox file is in sync is no longer
valid when you copy to the new file system.

the read state is stored in the mailbox file itself, in the x-mozilla-status
line header of the mail message. I'm not sure if we have trouble reading the
x-mozilla-status line across platforms - that would be surprising.

Comment 2

[Wolfgang Eschner]

The mail messages which lose their status across platforms do NOT have an
"X-Mozilla-Status" line in the respective mailbox file. Maybe the import from
Outlook Express sometimes "forgets" to assign that status? Because it seems that
the older messages tend to have it, the newer ones don't.

Then, those messages without X-Mozilla-Status are treated differently depending
on which .msf file is used. 
Sample sequence, again folder "test", 1 message with X-Mozilla-Status , 1 without:
Message without X-Mozilla-Status ist marked "read" on the PC,
mailbox file and summary file are copied to the Mac,
on Mozilla startup, Mac deletes msf, shows the message without X-Mozilla-Status
as "unread", writes a new .msf when exiting Mozilla;
mailbox file and summary file are copied to the PC,
on Mozilla startup, PC shows the message without X-Mozilla-Status as "unread".

Maybe there is a way of assigning an X-Mozilla-Status to those messages which
have none?

Comment 3

[David :Bienvenu]

there was a bug in the outlook express import tool which caused it not to put
the x-mozilla status lines in (which was fixed, I believe - how long ago did you
import your mail folders?)

It's difficult to insert x-mozilla-status lines into the middle of messages,
because then we'd need to rewrite the whole mailbox to the end. We should fix
these messages at compact time, but we're not doing that.

Comment 4

[Wolfgang Eschner]

seems that import bug has not been fixed (see also #43297, which i just came
across; the thread has been taken up again recently). I imported from OE6 3 days
ago.

It makes sense to me to insert an X-Mozilla-Status line (in case there is none)
upon compacting the mailbox, but i understand that that's not trivial.

Comment 5

[Wolfgang Eschner]

btw, after inserting missing X-Mozilla-Status values into the mailbox file, it
works nicely across platforms (back and forth)

Comment 6

[Joe Infla]

*** Bug 217838 has been marked as a duplicate of this bug. ***

Comment 7

[Max Alekseyev]

I see the bug across Linux and OS/2 Mozilla builds. Each one re-builds all
summary files previously built by another one.

It's worth to say that in my case they share the same profile (thanks to fixed
bug 137006), and nothing is copied or synced.

Confirming...

Comment 8

[David :Bienvenu]

different file systems report different time stamps. That's not a mozilla bug
and there's little we can do about it. You can, however, set a hidden pref with
about:config to allow leeway in the timestamp differences to account for the
file systems' different times - add the pref "mail.db_timestamp_leeway" and set
it to a value in seconds that stops OS/2 and Linux from regenerating summary files.

Comment 9

[Max Alekseyev]

Could timestamps be turned into some universal OS-independent values?

BTW, what's reasonable value for mail.db_timestamp_leeway ? Should it be as
large as possible or what?

Comment 10

[Max Alekseyev]

David, you say ``different file systems report different time stamps''. 

That's not a case for me. Both OS/2 and Linux Mozillas use the same profile at
the same physical location on HPFS file system.

Comment 11

[David :Bienvenu]

the main reason we regenerate the summary files is if the time stamp or the file
size of the mailbox is different from what we stored in the summary file. That's
why I said the file systems must be reporting different time stamps, since I
thought that was more likely than the file system reporting different file
sizes). One possibility is different handling of daylight savings time.

You can make the leeway whatever you want. The reason we check the time stamp is
because we're worried that either some other program has changed the mailbox,
and thus our summary file information is potentially invalid, or we've updated
the mailbox w/o updating the summary file (the latter would be a bug). If you're
not worried about some other program changing the mailbox, then you can set the
leeway to as large a value as you want...  

Comment 12

[:Gavin Sharp [email: [e-mail]]]

*** Bug 298265 has been marked as a duplicate of this bug. ***

Comment 13

[Jens Bannmann]

Note that this works fine when Linux Tb just accesses the mail files on the
Windows FAT partition.

Comment 14

[Thomas K. (:tom)]

*** Bug 290415 has been marked as a duplicate of this bug. ***

Comment 15

[Tomasz Poradowski]

(In reply to comment #13)
> Note that this works fine when Linux Tb just accesses the mail files on the
> Windows FAT partition.

I've been looking for this bug to file my own report, because this doesn't work for me. On Windows XP I use Thunderbird 1.5.0.4 and the same version on Linux, where I access the whole Tb's profile folder through a mounted VFAT partition. Some of the folders are ok, but about half of the most used/checked by me (main and trash ;) need to be entered and the summary files need to be rebuilt. This causes another problem of not showing new mail in those folders (in folder tree view) if I don't check them first.

Comment 16

[David :Bienvenu]

changed summary - in 2.0, I'm planning on ignoring small differences between the timestamps, as long as the file sizes are what we expect.

Comment 17

[David :Bienvenu]

We now ignore small timestamp differences, so this should be wfm.