Closed Bug 136049 Opened 22 years ago Closed 17 years ago

Daylight savings change invalidates summary files

Categories

(MailNews Core :: Database, defect)

x86
Windows 2000
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: lohphat, Assigned: Bienvenu)

References

Details

(Keywords: fixed1.8.1)

Attachments

(1 file)

2002040608 win32

This has also been a long standing bug in Netscape (since 2.0 I think)!

After the client TZ changes or Daylight Savings changes, as mail is retrieved
(POP3) messages are filtered into folders but number of unread messages is not
set.  You have to access every folder to have the summary file rebuilt to
reflect the time change for the messages to show up.
WorksForMe... I didnt notice any difference after Daylight Savings Time change,
reporter can you try a new build or something ?
QA Contact: gayatri → laurel
Same here... had this bug with Netscape Communicator 4.x
and now with Mozilla 1.2b... yesterday when Europe switched back from DST
Experienced on Windows 2000

Basically all .msf files get ignored (as if the client was considering them
corrupted). Side effects are:
 - they are rebuilt the first time the folder is accessed
 - sorting preferences are lost (they are probably stored into the discarded msf?)
 - until the folder is accessed, unread message counts are not available
*** Bug 176984 has been marked as a duplicate of this bug. ***
Status: UNCONFIRMED → NEW
Ever confirmed: true
This bug is still alive in Mozilla 1.3.

A few of the eastern states of Australia switched back from DST to normal time
yesterday and hence all my Mozilla mail summary files are now being rebuilt as
each folder is accessed...

Worse, it appears that the junk message status is held in the summary file - and
so junk mail filtering breaks (well you have to reclassify junk messages etc -
tedious).

Why does this happen?
Confirming. Happened for me too in 1.3
Well, it happened to me too yesterday (Mozilla 1.3 Win2000),
but then I've seen this happening for the last 5 years or so,
starting with Netscape Navigator...
argh - this bug just got worse!
I have a rather large mail archive, and ever since the daylight change, mail
won't tell me how many new mails I've gotten in a folder - it just marks it bold.
Whenever I the view a folder, the content is no longer displayed in threaded
mode - and many of the read mails have been marked as unread.
Try deleting the .msf file while Mozilla isn't running
and let it rebuild it -- that's how I used to solve such corruption
with Netscape Mail.
I'm expeiencing this also on 1.3

I'm getting really erratic results too. The first time I went to my sent folder
since Dalight Savings, night before last, mozilla duplicated a little over 800
of my sent emails, twice giving me 1600 or so unread new entries in my sent
folder which I had to remove. This is an ugly bug!
OK, this bug has been plagueing people since at least 1996 and NSCP 4.x if not
earlier. It only affects people around time changes then goes back into the cave
and hides.

Becasue it invalidates the files filters continue to pour mail into folders and
not update the unread counter becasue the summary file's horked.  You have to
touch each folder to rebuild it to see if you've got mail.

This is just plain silly that it's not been fixed in 6+ years.
Not only that...  Sending a message will eventually fail because
the message cannot be copied to the sent folder until the state
file is rebuilt.
Just for the record, this bug is alive and well in Thunderbird 0.3.  DST started
this Sunday for me, and until I found this bug report I thought I had done
something very wrong to my profile.
just got hit by this one, again!
using 
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5) Gecko/20031007 Firebird/0.7
So I commented back in Comment #9 on this one on 2003-04-07 10:14 when it hit me
the first time. And low and behold, how many revisions later and this bug STILL
plagues mail/news users?!

So in case anyone is wondering, this bug is manifesting in the 1.5 codebase as well.

I wish I fix the damn thing myself. How do we get this thing addressed, it's a
pain the a*s?

The owner of this bug, David Bienvenu has 361 bugs that he's working on right
now. Is there any way we can get this bug moved to someone who might address it.
It has been one and half YEARS that this bug has been with us now and no progress.
the problem is, I've never seen this happen. It didn't happen today, either,
even though daylights savings time started today. Did you have mozilla running
when  daylight savings time went into effect?
I didn't have it running
It manifested for me, and Mozilla was running during the changeover.
*** Bug 223822 has been marked as a duplicate of this bug. ***
Had this problem yesterday again, with Mozilla/1.5 RTM on WinXP

not running during the DST change (at 3am central european time)

David -> to reproduce, I suggest you switch your clock to
Timezone: Central European Time (Paris, Madrid, Berlin)
at 02:45 (am) on Saturday, 25 Oct 2003

and access Mozilla folders to ensure the .MSF are correct
then quit Mozilla and wait until after 03:00 am when the clock will switch back
to 02:00 am, then open Mozilla again.

This bug might not be reproductible in PST (California/Washington State)
timezone, after all MSDOS didn't have any timezone support and defaulted
everyone in Seattle, so maybe PST is immune to this for some histerical reason :)
Scratch this -- the OS keeps track of more than just the DST adjustment,
according to bug 223822

so in order to reproduce, you should instead
- change your timezone to CET (w/ DST)
- move the clock FORWARD to just before the next winter -> summer time change

Does anyone know when we're supposed to switch back to summer time in Europe ?

Worst case you can use last march's date for the switch, which according to
http://www.timeanddate.com/time/dst2003a.html should be:
30 March 2003, 01:45 am (switch is at 02:00 am, which becomes 03:00 am)
I had 2 users in the german Newsgroup with the same problem.
I think many people just think that this is only a temporary glitch and don't
file a bug. And both users imported outlook mails and this is worse together
with the bug in the oulook import routine (missing status headers) because many
mails are now marked as unread.

I know that one user started Mozilla AFTER the timezone changed and Mozilla
didn't run during the change.
didn't have mozilla 1.5 download anything twice (i don't leave messages on
server) but it does kill the file view settings for everything in my mail
folders when change occurred for eastern standard time, GMT-0500. when new
messages filter in mail defaults to sort by date, oldest first; this means in
many folders new messages are hidden - folder doesn't get bold, doesn't display
# of new messages, just the little green arrow above the folder. not as easy to
see as the bold, especially with lots of subfolders in my inbox.

(side note - anybody know why the new message display is inconsistent like this
in the first place, some folders show bold + # of new messages, others the green
arrow but not bold? i imagine someone has reported this bug...)
oops forgot to add mozilla was not running during change, and this happened on
both of my computers (XP pro and XP home, in sleep mode and hibernating during
changeover, respectively)
Mozilla Mail/News was not running during DST change.
I had this problem on both, my Home PC (Windows 2000) and work PC (Windows XP).
Another Co-Worker using Mozilla 1.5 (because I encouraged him) has the same
problem on his XP PC.

On both, home and work PC I originally imported my mails from Outlook Express.
Regarding Comment #19 PST is not immune to this in that I'm in PST and I've seen
it twice this year, once each for each DST change.

David, if you have any questions, since you haven't seen it manifest, or any
actions you'd like me to take, I'd be more than happy.
-Gene Wood
I have the same problem.
Messages imported from Outlook are marked as unread.
Furthermore, somehow messages that were deleted showed up again in some mail
folders.
Probably I was hit by this bug last year too. However, I did not search for a
bug then, and I marked the messages manually as read again.
Another (maybe unrelated) problem I have since the messages were marked unread
is that I cannot sort the messages by newest message first anymore when they are
viewed as threads. Formerly this was possible.
again, the solution to the reappearing/marked unread messages problems is to
compact your inbox or the folder in question with a 1.5 build. This will prevent
the imported messages from reappearing after being deleted, or being marked unread.
if anyone wants to perform some experiments and report back to me, that might be
helpful.  The experiments I have in mind are moving the system clock back and
forth while mozilla is shutdown, and seeing if it invalidates the .msf files for
any of your mail folders.

The reason Mozilla regenerates the .msf files is that the time stamp on disk of
the mail folder does not match the time stamp it has stored in the .msf file.
Changing the system time should not affect the time stamp of a file on disk, but
it seems like it sometimes does. This is a bit out of my control, since it's
happening while Mozilla is not running. 

Given that the time change is causing the time stamp on disk to change, the next
question is by how much.  There is a hidden pref in mozilla/tbird that tells the
mail db to be tolerant of small differences in the time stamp -
"mail.db_timestamp_leeway", which is in seconds. So, if for example, the
daylights savings time change somehow causes the timestamp on disk to drift by a
second or two, setting the leeway to 5 would fix the problem. But if it changes
the timestamp by, say, an hour, that's no good.
I believe the offset will be one hour. It is a "feature" (not bug, thank
heavens!) of Win9x. See
http://support.microsoft.com/default.aspx?scid=kb;EN-US;129574

"In the case of WINDOWS95 - The time returned by the file system is not adjusted
for Daylight Savings Time. This feature is by design, and was implemented to
work with servers such as NetWare that do not support the Daylight Savings time
APIs and time functions in Windows 95. In this case the files will display off
by one hour. this is BY DESIGN for WINDOWS95 clients."

I know this well. It affects also Win98 and WinME. This "non-bug" makes the
timestamps of any installed update files (unpacked from CAB files) vary by 1h
depending whether they are installed in summer or winter. No kidding.
On the second thought:

Seeing that this bug affects Win2K and WinXP users and _not_ Win9x (I use WinME
and do not see the problem), maybe Mozilla expects the Win9x behavior?
I have the check box "automatically adjust for daylights savings time" checked.
For the people that this is happening to, do you have a FAT file system, or an
NTFS file system? I have a FAT file system. The article seems to state that this
only happens on NTFS file systems.
I'm running WinXP Pro, with an NTFS file system.

mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.5) gecko/20031007
A very good article on the different FAT/NTFS timestamp behavior difference:
http://techsupt.winbatch.com/TS/T000002001F17.html
OK, it's all clear to me now. And I tried it on my new laptop that has ntfs, and
the summary file was regenerated.

However, fixing this could be problematic for a couple reasons. For one thing,
this code is cross-platform code so I can't just make a win32 call to try to
correct this problem. Our underlying file system library could try to make the
correct call to get UTC time instead of the stat time - maybe, in fact, it
already does. I'll look into it.
Status: NEW → ASSIGNED
Just some observations. I went round and round with this in WinCVS. WinCVS (at
least one version) suffers much the same problem. One issue is that when you get
local time for a file, it uses the current DST setting. So while the file may
have been created using the DST adjustment, you're now in standard time and it
no longer offsets, so the time appears to have changed. At least that's what I
saw when I was messing around in WinCVS and what I've gathered from reading. So
if the code can use UTC instead of stat or the local time, that may take care of it.
Attached patch proposed fix — — Splinter Review
using the nsILocalFile time stamp seems to fix the problem.
Attachment #135488 - Flags: superreview?(mscott)
Attachment #135488 - Flags: superreview?(mscott) → superreview+
fix checked in - we'll see in a few months :-)
Status: ASSIGNED → RESOLVED
Closed: 21 years ago
Resolution: --- → FIXED
Thank you very much for your efforts David.
-Gene
I hope it works - it could cause one more round of summary file invalidations
Hehe :D
Status:   	RESOLVED
Resolution: 	FIXED

Nope!
Mozilla 1.6 - this bug again!
Marry Christmas bug #136049  ;))
This happened for me again with the automatic time change (Windows) on 04-04-04.
(Mozilla build 2004031616)  All Mail Summary Files had to be rebuilt.

Did anyone else see it?  If so, file a new bug or reopen this one.
what OS are you using, and what kind of file system? FAT? NTFS?
Problem is fixed for me, currently using WinXP NTFS and Moz/1.7b

(had this problem for years since Netscape 4, maybe even Netscape 2)
*** Bug 244358 has been marked as a duplicate of this bug. ***
(In reply to comment #44)
> *** Bug 244358 has been marked as a duplicate of this bug. ***

Nope, not a complete duplicate although clearly linked. My issue is about time
zone changes, not daylight saving. On my Win2K system with FAT and Mozilla 1.6,
I did NOT have this problem when DST started in March, but I DID get it when I
changed time zones (4 hour change each way) because I was travelling with my
laptop. I expect to be travelling quite frequently, and it is very annoying that
all my mail archives are messed up (many imported from Outlook), as well as my
junk mail setup getting confused.
This happened to me again with change to standard time on 10/31/04.  Using build
2004102505 on Windows ME with FAT file system.  I always install new nightly
builds to a new directory.  I'm not sure whether Mozilla was running at 2AM when
the clock was changed, but that could be a factor.

I can also reproduce the problem by changing the Time Zone setting as in Bug
244358. 

Fortunately, now when summary files are rebuilt, the message labels seem to be
retained, so there doesn't seem to be any data loss.
Correction, message JUNK STATUS was still lost in the summary rebuild.
Same O/S, same experience as comment 46 and comment 47.

The difference is I know my computer was off at the time of the time change.
all my summary files were invalidated too, on win2k. Maybe the new method we're
using to get timestamps breaks on non-XP systems, and works on XP...
David, do you use a FAT filesystem?

I believe FAT and NTFS have different ideas about timestamps. So fixing it on
NTFS screwed it up on FAT. At least it looks like it... 
I have a FAT system on win2k and an NTFS file system on xp...
See http://support.microsoft.com/kb/129574
"Time Stamp Changes with Daylight Savings"
Not very informative but at least we know it's a Microsoft feature ;-)

BTW, Xchange Agent seems to have the same NTFS vs. FAT daylight time problem:
http://www.laplink.com/support/kb/print.asp?id=48
See also bug 127246. Do any of you who see this bug also see that one? I had
that problem on my old Windows 2000 system which was FAT32, but it has gone away
on my new Windows XP system which is NTFS.
Glad to hear I'm not going crazy.  REOPENING.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to comment #53)
> I had that problem on my old Windows 2000 system which was FAT32, but
> it has gone away on my new Windows XP system which is NTFS.

Happened to me yesterday on a WinXP machine with only NTFS - junk status lost.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a4) Gecko/20040927,
i.e. 1.8a4 milestone
(In reply to comment #55)
> 
> Happened to me yesterday on a WinXP machine with only NTFS - junk status lost.

Interesting. Do you see bug 127246?
Happened to me too. All msf files invalidated. New messages filtered to such
folders were invisible (no new mail flag on the folder) until I opened the
folders. Win98SE, Fat32, Mozilla 1.7.3 not running at the time change (at night).
And I can also confirm the loss of junk mail flags on messages. David Bienvenu,
are those also stored in the mbox file (x-mozilla-status1/2) or only in the msf
file?
aceman, see: Bug 195737 - Junk mail flag should be stored in mbox, instead of msf.

also related: Bug 231802 - Junk mail marking lost on upgrade to 1.6 and or
rebuild of summary file
Actually, bug 195737 seems invalid, because David says in bug 231802 that Moz
should write the junk status into the mail headers (just accidentally didn't do so).
(In reply to comment #60)
> Actually, bug 195737 seems invalid, because David says in bug 231802 that Moz
> should write the junk status into the mail headers (just accidentally didn't
do so).

Surely what you mean is that David has confirmed in bug 231802 that bug 195737
is valid, and not intended behaviour. Also bug 231802 seems to be mostly a
duplicate of bug 195737. I will report this there as well.

Fixing bug 195737 would reduce the impact of this bug, but not fix it.
Yeah, we are both right :) I meant that the wording of bug 195737 ('Junk mail
flag should be stored in mbox, instead of msf.') implies that junk flag is not
stored in mbox BY DESIGN. That probably isn't true, as said by David. So in this
sense, it would be invalid. But yes, CURRENTLY, the flag is not stored there
because of some temporary bug. And, the junk flag should be stored in the mbox
AND the msf, not instead of. The msf is just a cache of the flags in mbox.
That's how I understand it.
Product: MailNews → Core
The problem happened again on the current daylight savings change. Win98, FAT32.
That is on 1.7.3.
It happened to me again on 1.8b2 nightly build 2005040105.  Win ME w/FAT32.  

Computer was powered off at the time of the time change.  On next time running
Mozilla Suite, all summary files are being rebuilt when I first access a folder.
Nightly build is installed in a new folder.
Confirming this is still a bug in Seamonkey 1.1a Build 20051013.  Win ME/FAT 32.

Windows and Seamonkey browser (but not mail) were running at 2AM
*** Bug 314389 has been marked as a duplicate of this bug. ***
My GOD -- this has got to be the longenst running bug EVAR.

I found this bug TEN YEARS AGO with Navigator 2.0 while I was working @ NSCP and it's STILL NOT FIXED.

For the love of Jeebus FIX this already...
Actually I've had this bug for 10 years because I've always used NTFS,
so now it's fixed for me at last :-)

I believe the bug only appears on FAT partitions now.
mscott, any chance of a review of this?
bug 290273 might have the same cause (timestamps on FAT32 aren't stored in GMT).
Status: REOPENED → NEW
QA Contact: laurel
Possibly, I get also got hit by this bug on Windows XP (see bug 314556).
(In reply to comment #72)
> Possibly, I get also got hit by this bug on Windows XP (see bug 314556).
> 
The issue relates not to the OS but to the filing system. XP can use both NTFS and FAT32, although NTFS is the default for the main disk drive - which is not necessarily where the folders are stored.

As far as I can see, the only way to fix this bug completely is to check for each set of folders whether they are stored on FAT32 or NTFS, and adjust or not adjust the msf files accordingly. The alternative is to wait until every Windows user is on NTFS so that the problem goes away, which will probably take less than another ten years!
I have a report from the VOKD (http://www.vokd.cz/) system administrator that because of this bug their users with TB 1.0.x on win98/FAT32 machines weren't able to work with the INBOX folder. Although the deletion of msf files "saved" their emails, it didn't gave too much credit to this new email client they had switched to.
Pavel, if you don't have any information regarding helping fix the bug, please don't comment.
*** Bug 319288 has been marked as a duplicate of this bug. ***
I got this problem again this weekend (DST). I'm using Thunderbird 1.5 on Windows 2000 with NTFS - but my profile is on a file share on a Mac (OSX).
*** Bug 319288 has been marked as a duplicate of this bug. ***
David is there any chance of getting this committed?
Ray, that fix was checked in, which should fix the problem for people using NTFS, which should be most users.

Going forward, we're probably just going to optionally ignore the time stamp and just pay attention to the file size.
Hello!!
I keep getting mysterious and meaningless emails regarding my problem. Can someone tell me in plain English if this bug can be fixed, and what do I do - or do I have to wait for the next time change and back up the night before and restore the next day. Come on MOZILLA give me a fix in plain language PLEASE!!

I use W98SE on an unpartitioned drive FAT32. You've had the last year to sort it - or do I go back to IE6 or Opera or Eudora??? 
I have experienced this bug on about 20 clients after Daylight Savings changes.
The problem happen on XP and W98 with profile stored on samba file system, or XP with shared profile. Did someone explain what is the logic that force thunderbird to recostruct msf files? what date it look, because is a date of a file at start of problem, to rebuild a index?
Maybe remounting samba filesystem after Daylight Savings changes can eliminate the problem?
I wonder if this problem could be fixed by noting that if the time stamp on a .msf file has changed by exactly one hour, in either direction, that file should be assumed to be still valid - and perhaps its timestamp should be corrected. The chance of a change by exactly one hour, to the second, when the file has really been corrupted is extremely small.

The similar problem with moving a computer across time zones could be dealt with by making a similar exception for changes of time in exact multiples of half an hour up to 24. This should allow for all possible time zone changes.
*** Bug 287913 has been marked as a duplicate of this bug. ***
The one hour delta is all that's needed to check for because: 

1) Most people don't change their timezone when they travel as there are too many things which break when they do so.
2) Some timezones are 00:15 off from their neighbors. (It's good to be the king!)

Since the problem manifests itself by placing unread mail into folders but not notifying the user that there's new mail, it would simply suffice to detect the timestamp is off, warn the use of the anomoly, and prompt with an OK/Cancel to rebuild the index files so new unread mail is apparent.

Since I reported this in NSCP Communicator 2.0 in 1995, I hope we can put this to bed finally.  Having a bug open for a decade has redefined the development term "target fix" to new territory.
(In reply to comment #85)
> 1) Most people don't change their timezone when they travel as there are too
> many things which break when they do so.

I've changed timezones ten times in the last three years while travelling (so much for people not doing that), and the only thing which has ever "broken" for me is this.

(With apologies for the spam. Anyway, David proposed a fix in comment 80 which should work perfectly, so let's just cut the chatter and wait for someone to implement that (or any other fix), okay? The problem is very well understood by all concerned, so further comments will only have a detrimental effect now.)
fix checked into 2.0 branch builds
Keywords: fixed1.8.1
That works for me. (Mac file share configuration as per comment <a href="#c77">#77</a>).

However, why, if I delete the .msf files, do exactly the same emails always get marked unread when the new .msf files are built? (It's always the same emails and no others). Anyone else see this? I can't find any other bug that mentions this - but maybe that's just me.  
those messages were probably imported from an other mail client, with a version of TB that didn't do import quite right, so that x-mozilla-status headers weren't added to those messages, which is where we store read state outside of the .msf file. If you delete a single message in the folder in question, and then do a "compact this folder" command, the problem should stop happening, because compacting inserts x-mozilla-status headers if there aren't any.
I believe there might be a more global approach to this bug:

As I mentioned in issue 358369, there exists a network time protocol. Could we make use of this accurate time to correct this problem.

Network Time Protocol (RFC-1305)
> The NIST Internet Time Service (ITS) allows users
> to synchronize computer clocks via the Internet.
> ( see http://tf.nist.gov/service/its.htm )
The NIST servers listen for a NTP request on port 123, ... (see Comment 10).

When Mozilla saves/ modifies the corresponding file, could it:
 - save internally (in that msf file) an accurate GMT time
 - whenever a new e-mail is read, Moz can get the accurate GMT time and save it as well
 - IF msf file seems broken due to time, get accurate GMT time; IF this time matches the saved time, msf file is OK, no need to rebuild;
(In reply to comment #90)
...
> As I mentioned in issue 358369, there exists a network time protocol. Could we
> make use of this accurate time to correct this problem.
> 
...
> The NIST servers listen for a NTP request on port 123, ... (see Comment 10).
> 
In principle this is a good idea, but there are some possible pitfalls. What if no NIST server is available, or no response is received? We can't assume that all users are connected to the Internet, rather than through a private network, and firewalls may block NTP; or responses may be very slow on dial-up lines in remote areas. What if the computer's own time is significantly different from the NIST time? There are a number of potentially tricky issues here.

(In reply to comment #91)

> What if no NIST server is available, or no response is received?
...
> firewalls may block NTP

I believe we can find additional Servers that report time:
 - using the Time Protocol
 - using http: you can get the accurate GMT time from the internet
   -- e.g. http://wwp.greenwichmeantime.com/ (or google for "GMT time")
   -- the URL to the GMT clock is: 
http://ntp1.greenwichmeantime.com/time/scripts/clock-6-1/clock.php?tzone_choice=main_large
   -- the GMT time is sent via the variable stime inside a java script,
   -- e.g.: var stime = new Date(1.16205331419E+12);
 - there might be other sites with accurate GMT time

Because *Time* is so improtant in the modern world, maybe big corporations will start their own time servers, and other institutons will run time servers, too. Maybe even using e-mail protocols (that would remove any firewall constraints).

Computers NOT connected to the internet may still pose a problem though. Here an inadequate solution may be implemented. But I have NO ideea how to implement something more smart in this case, ... on a M$ machine.
(In reply to comment #92)

> Because *Time* is so improtant in the modern world, maybe big corporations will
> start their own time servers, and other institutons will run time servers, too.

They already do and have done for at least 10 or 15 years.
(In reply to comment #93)
> (In reply to comment #92)
> 
> > Because *Time* is so improtant in the modern world, maybe big corporations will
> > start their own time servers, and other institutons will run time servers, too.
> 
> They already do and have done for at least 10 or 15 years.
> 
I was thinking more of small business etc situations, where individual users are not connected directly to the Internet (maybe to stop unauthorised surfing) but are connected to a local mail server, or to an external one through a firewall configured to allow only mail access.
*** Bug 358724 has been marked as a duplicate of this bug. ***
fixed on trunk
Status: NEW → RESOLVED
Closed: 21 years ago17 years ago
Resolution: --- → FIXED
Fixed 11 years after I first reported it (the original bug was an internal NSCP bug)? I'll believe it when October rolls around as it's been "fixed" before.

;-)
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: