131983 - some email/news messages display incorrectly with the date of 1/1/1970

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Message Display, defect)

Description

[Shoshannah Forbes]

Mozilla 2002031708 / Mac OS 9.2.2:
Some email messages recived disaplay in the mail pane as being from 
1/1/1970. However, the headers seem to have the correct date in them.
I am attaching the headers of 3 sample emails.
I should note that other emails do display proper dates

Comment 1

[Shoshannah Forbes]

Attachment: headers from email1

Comment 2

[Shoshannah Forbes]

Attachment: headers from 2nd email

Comment 3

[Shoshannah Forbes]

Attachment: headers from 3rd email

Comment 4

[Ian Neal]

I get this on build id 2002032008 on linux.

Comment 5

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 132514 has been marked as a duplicate of this bug. ***

Comment 6

[Frederic Bezies]

I'd posted a duplicate of this bug, so I write here !

This bug seems to happen with long titles. More than 20 or 25 characters in it.

Is this a clue ?!

I hope so :-)

Comment 7

[marina]

Shoshannah, could you please attach to this bug report a mail that has a correct
date.The messages that you've attached do show 12/31/69 as date. We have several
cases like that with malparsed headers. My guess for the emails in this bug
report being incorrectly sorted is a missing Date attribute or an incorrect one;
i see missing commas and Day in some server stamps. Thanks.

Comment 8

[marina]

after some experimenting with the attached message , here is what i found:
1. if i edit the message source and swap the header with date (placing header
after the date) then the message shows in my Inbox with the correct date
(03/19/2002;
- if i edit the page source and  substitute the 1255 encoded header for another
one (like iso-2022 ja) without changing the place of a Date tag (still before
the date) the message shows up in the thread as 03-19-2002;
- changing the mime 1255 header to "abc" will show the message in the thread as
03/19/2002;
i am wondering whether this particular header is not missing any byte promting
the mail server not to parse it correctly before the Date stamp. Cc'ing Naoki
and Simon. Simon, could you please look if the header is properly encoded? Thanks.

Comment 9

[Simon Montagu :smontagu]

The encoding of the headers of the attachments in here (and attachment 75394 [details] in
the dupe) all seem correct to me.

Comment 10

[Frederic Bezies]

Still having it, kind of random ! :-(

It seems to be related with accent in the title (é,è and others).

Maybe a clue ?!

Comment 11

[Dimitrios]

I 'm seeing this bug for a couple of weeks (currently on Win2K, 2002041103).
Needless to say, it's quite annoying because, in a long message list, it's
difficult to find those messages. Imo, 1.0 should not be released with this
regression. As far as I can remember, I see this bug on some messages with
non-ascii (ISO-8859-7 encoded) subjects. Date formatting in the message source
seems to be correct. Attaching such a message (slightly modified to remove
personal info) that might help you as another testcase. Platform->All, OS->All.

Comment 12

[Dimitrios]

Attachment: msg source from another email

Comment 13

[Christos Cheretakis]

Attachment: headers of another message with same problem

Comment 14

[Dimitrios]

Nominating for mozilla 1.0. This bug is highly visible for intl users.

Comment 15

[bobj]

From comment #8, it seems like the MIME encoded subject header may be
confusing the parsing of the next line which is the date header.

Possibly the subject line is malformed and we are not gracefully recovering
for the next line, or we have a problem with our MIME decoding of the
subject header which has a side affect on the next line...

Someone should get in a debugger and see what is going on as we parse these
2 headers.

Comment 16

[nhottanscp]

This is caused by bug 21267. 
It removes 'RE' from MIME encoded subject, decode it first then encode again. It
assumes that the re-encoded length is smaller than the original length. But like
the examples attached here, it may be encoded differently and the result could
be longer than the original. It should not copy the re-encoded string if the new
length is larger than the original length.

Reassign to nhotta.

Comment 17

[nhottanscp]

Nominating for nsbeta1. It always happen when the subject is encoded differently
from the Mozilla's encode type.

Comment 18

[nhottanscp]

Attachment: Do not stip 'RE' for MIME encoded header if re-enocded string is larger than the original.

This prevents the data corruption.
In future, we need a capability to specify encoding method to MIME encoder
instead of always use our pre-defined method.

Comment 19

[Frank Tang]

when will this bug happen
1. the subject in message header is MIME encoded, and
2. the MIME "encoding" is B for single bytes charset or Q for Multibyte charset. 
3. there are a "Re" inside the Subject field

The same buggy code could also possible also will cause crash (altought we have
not see an incident of that but that is also possilble) or damage other record. 

Impact users- mainly Hebrew (1.9M), Greek (1.6M)
The patch potential fix other mail crash or data damage issues. 

let's do some research to see the buggy code really damange the data or cause
some crash. 

Comment 20

[nhottanscp]

I looked at nsParseMailMessageState which is one of the callers of
NS_MsgStripRE, it maintains pointers of original headers instead of copies. "RE"
stripping is done by changing the pointer but not altering the original header.
But the patch of bug 21267 modifies the header if MIME encoded which is wrong
even when the result is smaller than the original. So far, only known problem is
this bug.
cc to putterman, ducarroz, bienvenu, taka to share the info.

Comment 21

[nhottanscp]

Attachment: Changed to use 'B' encoding for arabic, greek, hebrew. Added an assertion, so we can catch more cases.

Comment 22

[David :Bienvenu]

Comment on attachment 80051 [details] [diff] [review]
Changed to use 'B' encoding for arabic, greek, hebrew. Added an assertion, so we can catch more cases.

sr=bienvenu

Comment 23

[Jan Schnackenberg]

I have observed something similar, with much worse effects.

Something similar happens in newsgroups in my college. It _might_ be related to
Netscape 6.2 being installed in the computer-rooms.

Anyway. The effect is there for everybody to be seen:

news-server: news.fh-wedel.de
newsgroup  : fhw.talk

In said newsgroup look for a thread "ab der TU müßt man sein...." and another
thread "Re: ab der TU müßt man sein....". The "Re"-thread is aktually nothing
more than lots of anwsers to the main-thread, which mozilla seems to be unable
to sort into the correct location (in threaded-view). The awnsers in that thread
are readable by mozilla, but I cannot view the source of them (<ctrl>-u) and the
sender and date in the thread-pane are completely mangled.

Comment 24

[Jean-Francois Ducarroz]

Comment on attachment 80051 [details] [diff] [review]
Changed to use 'B' encoding for arabic, greek, hebrew. Added an assertion, so we can catch more cases.

R=ducarroz

Comment 25

[nhottanscp]

*** Bug 138222 has been marked as a duplicate of this bug. ***

Comment 26

[nhottanscp]

Bug 138222 contains examples of ISO-8859-1 and ISO-8859-2. 
My guess is that MIME decoder now changed to encode entire subject which may
produce a larger result. Taka, is that correct?

Comment 27

[Takayuki Tei]

You don't have to compromise with sorting issue.
NS_MsgStripRE() is used at 3 places only.

So, why don't you add an argument (PRBool *) to NS_MsgStripRE() to let caller
know whether it has to free returned string rather than sticking to original
length and using strcpy()?

Comment 28

[nhottanscp]

adt2 -> adt1, because it also impact Latin1 and other users

Comment 29

[Adam Macznik]

Please change the summary to something that contains 'news' and 'broken
threads', for example 'Some email/news messages display incorrectly with the
date of 1/1/1970 and in broken threads'. It will make it easier to find for news
users since bug 138222 is now resolved.

Comment 30

[nhottanscp]

I was chatting with bobj and come up with an idea.
It is possible to skip "RE:" without encoding the string again. It needs to
decode MIME encode but without charset conversion then we can search for "RE:",
remember the positions and copy the rest of the string over. This way, the
result is always shorter than the original and no modification to the subject
except for removing "RE:". Probably, we want that functionality in the MIME
decoder/encoder interface. I think this would be not a small change, so not for
moz1.0.
I will try taka's suggestion for now.

Changed summary to include 'news'.

Comment 31

[Boris 'pi' Piwinger]

Taking severity and dataloss keyword from duped bug 138222. Not sure about mail3
(I leave the mail[0-9] keyword for someone else).

The summary could need a cleanup, maybe:
mail/news: broken threads, lost articles, date of 1/1/1970

pi

Comment 32

[Takayuki Tei]

[e-mail] wrote:
>It is possible to skip "RE:" without encoding the string again.

NS_MsgStripRE() must return "Hello, world." for the all of following input (it's
in raw ISO-2022-JP for viewing purpose):

    Re: Hello, world.
    \x1b(BRe: Hello, world.
    \x1b(JRe: Hello, world.

Although I don't know actual implementation works this way, they are all
comformance to RFC 1468.

Comment 33

[nhottanscp]

Filed bug 139317 for adding "Re:" strip support to nsIMimeConverter.

Comment 34

[nhottanscp]

Attachment: Added out argument for NS_MsgStripRE to avoid altering the const argument.

Comment 35

[Jean-Francois Ducarroz]

Comment on attachment 80478 [details] [diff] [review]
Added out argument for NS_MsgStripRE to avoid altering the const argument.

R=ducarroz

Comment 36

[nhottanscp]

*** Bug 136461 has been marked as a duplicate of this bug. ***

Comment 37

[David :Bienvenu]

Comment on attachment 80478 [details] [diff] [review]
Added out argument for NS_MsgStripRE to avoid altering the const argument.

sr=bienvenu

Comment 38

[scottputterman]

adding adt1.0.0+.  Please check this into the branch as soon as possible and add
the fixed1.0.0 keyword.

Comment 39

[nhottanscp]

checked in to the trunk, asked branch check in approval 

Comment 40

[ji]

Marina, could you take care of this? Thanks.

Comment 41

[marina]

in the 2002-04-23 trunk build the original mail from Shoshannah is displayed
correctly (3/19/2002) but bug# 136461 is still present:the headers in newsgroups
are still diplsaying as 12/31/69 or 01/01/70 

Comment 42

[nhottanscp]

It's showing fine using my debug build.
Please make sure to remove old .msf file and try again.
Please try data in bug 138222 also.

Comment 43

[marina]

after removing old msf files for all intl newsgroups i don't see this bug any
more, gone! fixed!  Will verify other dups as well. ready to get checked into
the branch

Comment 44

[Frank Widmaier]

@marina

i tried with deleting the msf -files too.. several times. the phenomen returned
and returned.
Now I have the prob in about 10 groups ... before trying to find out, whats the
problem, only 2 groups showed me the 1970s.
(098: few postings, 099: more postings, RC1: you can say: 1970s with each
request for new postings)

I hope, you're right and it's fixed... let's see ;-)

Comment 45

[nhottanscp]

fixed in the trunk

Comment 46

[marina]

[e-mail], after you delete all msf in your profile, restart the
application and you shouldn't see this bug. One of the newsgroups i tried was
de.test on news.mcom.com. I don't see it there any more. If your problem will
persist please point me to the newsgroup you are having problem with. Thanks.

Comment 47

[nhottanscp]

I think the safe way to verify is testing with a new profile.

Comment 48

[Frederic Bezies]

Following comment #46, I deleted .msf file. Can see it again, my build should be
a little older :-)

"Only" using 2002042306... So, let's wait for a new and younger one :-)

Comment 49

[nhottanscp]

2002042306 should contain the fix.
Please test with a new profile.

Comment 50

[Frank Widmaier]

I tried to download the "latest"  - both, exe and zip are corrupt.

- deinstalled rc1
- reinstalled RC1.. closed all instances and fast-start-icon
- deleted all msf
- startet mozilla
- the "old 1970" are still there, when download up the indicated "here are 
  waiting xxx postings for you" in the tree.. there will be downloaded new 1970s

I will watch, what will happen in the other groups with the older ones. If there
will be new ones, I'll tell you.

new profile: won't help. others in a german newsgroup tried this, if I would
correctly remember.. I'll test it tomorrow

I noted two new bugs today.. I will look up, if I can find numbers
(the "new mail"-symbol in the tray will not disappear, when reading in the
folders .. when closing the last tab on the right side via the "X", the new last
one  won't have a right (shadow-)borderline...  ) If I dont find sth I'll open
new bugs

Comment 51

[nhottanscp]

Please note this was checked in to the "trunk". 
The fix is not included in the branch (including RC1).

Comment 52

[Frederic Bezies]

To avoid spamming :-)

In answer to comment #49 : I donwloaded 2002042306 from latest-1.0.0 folder

In answer to comment #51 : for now, I am only download branch build, not trunk ones.

I had this bug for one month or so. I could live with it some more days. Just
hopping this patch would be included soon in branch builds.

Anyway, thanks for fixing it.

Comment 53

[Frank Widmaier]

I downloaded rv:0.9.9+ Gecko/20020423 from /2002-04-23-06-trunk/ you told me

It works properly with my old profile and with a new profile. thx

the file (zip, exe that will be shown in /latest/ , /latest-trunk/,
/latest-1.0.0/ is corrupt

Comment 54

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 80478 [details] [diff] [review]
Added out argument for NS_MsgStripRE to avoid altering the const argument.

a=[e-mail] for branch checkin

Comment 55

[nhottanscp]

fixed1.0.0, checked in to the branch

Comment 56

[Boris 'pi' Piwinger]

*** Bug 142501 has been marked as a duplicate of this bug. ***

Comment 57

[Jean-Francois Ducarroz]

*** Bug 133847 has been marked as a duplicate of this bug. ***

Comment 58

[R.K.Aa.]

*** Bug 142501 has been marked as a duplicate of this bug. ***

Comment 59

[marina]

verified with 2002-05-29-1.0.0 build