402426 - Gmail IMAP: chats' headers are not recognized in the thread pane but are in the message pane (No Date:/Subject: in response to FETCH BODY.PEEK[HEADER.FIELDS)

Status: REOPENED

(Thunderbird :: Mail Window Front End, defect)

Description

[Dmytro Pavlov]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9
Build Identifier: 2.0.0.6 (20070728)

Gmail chat messages are downloaded by IMAP among other messages in [Gmail]/All Mail. 

But their Subject column is empty and their Date header is not picked up that causes incorrect 

sorting by the Date header. Such messages are given date of IMAPing although the correct date is displayed in the message pane (as well as "Chat with <CONTACT>" subject). 

Here is the quote of the source of the Gmail chat message (ids are cut out and replaced with 

contact-name and my-gmail-id):

=============== begin of source =============
Message-ID: <[e-mail]>
Date: Fri, 2 Nov 2007 10:04:56 -0700 (PDT)
From: conact-name <[e-mail]>
To: [e-mail]
Subject: Chat with rrom
MIME-Version: 1.0
Content-Type: multipart/mixed; 
	boundary="----=_Part_3341_29872833.1194176982401"

------=_Part_3341_29872833.1194176982401
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<SPAN><SPAN style="font-weight:bold">contact-name</SPAN>: http://www.<URL_CUT>.htm</SPAN><BR>
------=_Part_3341_29872833.1194176982401--
=============== end of source =============

We see that the message's date is 2 Nov 2007 10:04:56, but it is given the date of IMAP-downloading in the message list: November 3 14:42.

The worst thing is that it's impossible to sort Gmail char messages by date.

The screenshot can be found at http://keep4u.ru/imgs/b/071104/25f630841ee5b27709.jpg

This is not Gmail's problem but Thunderbird's because I also tested this with The Bat! mailer but the problem exists with Thunderbird only.


Reproducible: Always

Steps to Reproduce:
1. Create a new IMAP account in the Thunderbird for a Gmail account that has previous days' chat messages saved.

2. Go to the [Gmail]/All Mail directory and wait until it is updated from the Gmail.

3. Choose descending sorting by the Date.

4. Find out that all the past chat messages are at the top of the list and that their Date is properly displayed in the pane but is displayed as the IMAPing date in the message list.

Actual Results:  
Find out that all the past chat messages are at the top of the list and that their Date is properly displayed in the pane but is displayed as the IMAPing date in the message list.

This makes it impossible to properly sort by the Date.

Expected Results:  
The date of IMAPed Gmail chat messages should be the date of actual chating, not of the IMAPing.

See screenshot at http://keep4u.ru/imgs/b/071104/25f630841ee5b27709.jpg

Comment 1

[Dmytro Pavlov]

Attachment: The screenshot that illustrates the problem

This screenshot is also addresses as http://keep4u.ru/imgs/b/071104/25f630841ee5b27709.jpg in the bug description.

Comment 2

[Magnus Melin [:mkmelin]]

Confirmed on trunk.

Comment 3

[Brian Kennelly]

This is another Gimap bug.  The server does not return the 'Date:' field when downloading the headers. Oddly, the header does include the Date: field when downloading the full message.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(For date part)
> Date: Fri, 2 Nov 2007 10:04:56 -0700 (PDT)

What is your Timezone?
Do you set mailnews.display.original_date=true?

With Seamonkey 1.1.5 on MS Win-XP(Timezone=+0900,Japan), manually generated mail in local mail folder(dummy POP3 account):
 (1) Date column in thread pane (similar result as yours)
     2007/11/03 2:04
 (2) Date: header data display in header box in mail pain  
    (2-1) mailnews.display.original_date=false
          2007/11/03 2:04
    (2-2) mailnews.display.original_date=true
          Fri, 2 Nov 2007 10:04:56 -0700 (PDT)

(A) "Fri, 2 Nov 2007 10:04:56 -0700 (PDT)"
    == "2007/11/02 17:04:56" in GMT
    == "2007/11/02 26:04:56" in JST (+0900 from GMT)
    == "2007/11/03 02:04:56" 
(B) Therefore, "2007/11/03 2:04" in thread pane is properly displayed.

(For blank subject part)

Different Message-ID are used for the mails?
Or same Message-Id is set for the mails? 

Comment 5

[WADA:World Anti-bad-Duping Agency]

Above (A) and (B) is incorrect, never be proper.
It looks to be a result of scenario of above (A).
i.e.
 - Inappropriate Date header handling when malformed Date: header.
 - When your case, Date: header may be "obsolete" format, instead of malformed.
Other possibility in your Date: display in message pane:
As seen in Bug 387387, different Date data handling logic is applied when malformed  date value or unexpected date value. Difference of date display in your case may be a result of this.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Oh, you said "14:42" in Date column for "10:04" in Date: header.
It indicates "Current Time is used because of no Date: header", and reason why "no Date: header" was already explained clearly by Brian Kennelly in Comment #3.
Sorry for my confusion and spam.

Comment 7

[WADA:World Anti-bad-Duping Agency]

To Dmytro Pavlov(bug opener), Magnus Melin, Brian Kennelly:
Can you attach IMAP protocol log? (See Bug 402793 Comment #1 for NSPR logging)    

Comment 8

[Brian Kennelly]

I actually verified with direct login to the server, but I can create a log.

Comment 9

[Brian Kennelly]

Attachment: IMAP trace of Gimap response to header download.

I trimmed the trace to show just the UID fetch command for the header fields:

10 UID fetch 4:16 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)])

and the response for a chat log message.

Comment 10

[WADA:World Anti-bad-Duping Agency]

No Date: header, no Subject: header. Sigh...

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)
> Oddly, the header does include the Date: field when downloading the full message.

Date: & Subject: headers are probably generated on later full message FETCH time only. And another minor issue exists - format of generated Date: header looks to be obsolete format.

Comment 13

[Brian Kennelly]

I found a workaround using a hidden preference in Thunderbird 2.0.0.6

Set 'mail.imap.use_envelope_cmd' to true and restart Thunderbird.  
To get existing chat messages to appear correctly, you will need to rebuid the index on the folder.

Comment 14

[WADA:World Anti-bad-Duping Agency]

FYI.
Value of mail.imap.use_envelope_cmd is saved in gUseEnvelopeCmd, and is used by following logic. 
> http://lxr.mozilla.org/seamonkey/source/mailnews/imap/src/nsImapProtocol.cpp#3031
> 3031 if (gUseEnvelopeCmd)
> 3032  what = PR_smprintf(" ENVELOPE BODY.PEEK[HEADER.FIELDS (%s)])", headersToDL);
> 3033 else
> 3034  what = PR_smprintf(" BODY.PEEK[HEADER.FIELDS (%s)])",headersToDL);

RFC 3501 says as follows.
ENVELOPE forced Gmail IMAP to generate lacked Date:/Subject: header data?

> ENVELOPE
>  A parenthesized list that describes the envelope structure of a
>  message.  This is computed by the server by parsing the
>  [RFC-2822] header into the component parts, defaulting various
>  fields as necessary.
>
>  The fields of the envelope structure are in the following
>  order: date, subject, from, sender, reply-to, to, cc, bcc,
>  in-reply-to, and message-id.  The date, subject, in-reply-to,
>  and message-id fields are strings.  The from, sender, reply-to,
>  to, cc, and bcc fields are parenthesized lists of address
>  structures.

Comment 15

[Brian Kennelly]

(In reply to comment #14)
> ENVELOPE forced Gmail IMAP to generate lacked Date:/Subject: header data?
> 

Gimap returns the data correctly in the ENVELOPE structure:

. fetch 5 (envelope body[header])^M
* 5 FETCH (ENVELOPE ("Fri, 9 Nov 2007 13:24:08 -0800 (PST)" "Chat with [e-mail]" (("[e-mail]" NIL "xxx" "gmail.com")) (("[e-mail]" NIL "xxx" "gmail.com")) (("[e-mail]" NIL "xxx" "gmail.com")) ((NIL NIL "xxx2" "gmail.com")) NIL NIL NIL "<[e-mail]>") BODY[HEADER] {493}
Delivered-To: [e-mail]
Received: by 10.142.143.17 with SMTP id q17cs479565wfd; Fri, 9 Nov 2007
 13:24:08 -0800 (PST)
Received: by 10.65.213.4 with SMTP id p4mr8063442qbq.1194643448265; Fri, 09
 Nov 2007 13:24:08 -0800 (PST)
Message-ID: <[e-mail]>
From: "[e-mail]" <[e-mail]>
To: [e-mail]
MIME-Version: 1.0
Content-Type: multipart/alternative; 
 boundary="----=_Part_7757958_24586100.1194643448249"

)
. OK Success

Comment 16

[David :Bienvenu]

we stopped using the envelope command because some servers didn't implement it correctly, if I remember correctly. Of course, now we have a server that doesn't implement the fetching of rfc822 headers correctly in some situations, which I would have guessed is a less likely scenario.

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #16)
> we stopped using the envelope command because some servers didn't implement it correctly

It was Bug 42863 fixed on 2001-03-27, which introduced mail.imap.use_envelope_cmd (default=false).
David, does we still need to withhold use of ENVELOPE as default?  

Comment 18

[WADA:World Anti-bad-Duping Agency]

If Mozilla family has to support user who uses both of following IMAP servers,
 (a) Gmai IMAP (problem occurs if mail.imap.use_envelope_cmd=false)
 (b) IMAP server with incorrectly implemented ENVELOPE
     (problem occurs if mail.imap.use_envelope_cmd=true)
per account mail.imap.use_envelope_cmd will be required.
I hope all Gmail IMAP users don't use IMAP server of (b).   

Comment 19

[Dmytro Pavlov]

Well, mail.imap.use_envelope_cmd=true did help.
Thank you for the tip.

But this preference did not exist even with the default value = false. And there was no idea about the new preference.

> WADA   2007-11-10 00:39:24 PST
> David, do we still need to withhold use of ENVELOPE as default?  

I think this preference needs to be added to the IMAP account settings.

The new entry linked to this "bug" needs to be open about the new check box in the IMAP account settings.

Thank you for your help again!

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> Gimap returns the data correctly in the ENVELOPE structure:
> . fetch 5 (envelope body[header])^M
> * 5 FETCH (ENVELOPE ("Fri, 9 Nov 2007 13:24:08 -0800 (PST)" "Chat with [e-mail]" (snip)
>(snip)
> BODY[HEADER] {493}
>(snip)
> )

Date: data & Subject: data are returned as ENVELOPE structure by Gmail IMAP when ENVELOPE is specified in FETCH request by Tb, but header data as response to BODY[HEADER] part doesn't contain Date: header nor Subject: header.
To Brian Kennelly: It means next, isn't it?
 (1) Date&Subject column of thread pane is proper (saved in ".msf").
 (2) But, mail header data in "Mail Header Box" of "Mail Pane"(preview pane)
     can not display Subject: header data and Date: header data.  

Comment 21

[Brian Kennelly]

When Thunderbird fetches the message body, it uses FETCH BODY[], which returns the header and body.  In that header, which is used in the preview pane and message source, the Date and Subject fields are included, so the header data is available. 

Comment 22

[Magnus Melin [:mkmelin]]

Reported to gmail as "technical issue".

Comment 23

[Brian Kennelly]

Gmail completely blocks chat transcripts now; should this bug be closed as INVALID?

Comment 24

[Magnus Melin [:mkmelin]]

Yes, lets.

->INVALID

Comment 25

[Jimmy]

Gmail has officially supported accessing chats via IMAP since 2011:
http://dataliberation.blogspot.com/2011/09/gmail-liberates-recorded-chat-logs-via.html

With that being the case, it seems time to re-open this bug.

Comment 26

[Magnus Melin [:mkmelin]]

Well, it seems to be working fine for most chats, and if when it's not, there's probably not that much to do about it, except complaining to gmail and have them fix whatever is broken.

Comment 27

[Jimmy]

There was talk of setting mail.imap.use_envelope_cmd to true by default, as that resolves the problem.

Also, what do you mean it "seems to be working fine for most chats"? I'm not aware of any instance in which this bug does not apply.

Comment 28

[Magnus Melin [:mkmelin]]

Well some chats do show up correctly for me, no idea why not all are.