517837 - X-Mozilla-Status: and X-Mozilla-Status2: header is written to offline-store for some mails by auto-sync (Gmail IMAP)

Status: RESOLVED DUPLICATE of bug 697635

(MailNews Core :: Networking: IMAP, defect)

Description

[WADA:World Anti-bad-Duping Agency]

X-Mozilla-Status: and X-Mozilla-Status2: header is not written to offline-store by auto-sync, if the mail was uploaded from local mail folder.

[ Build Id ]
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090915 Shredder/3.0b4pre

If mail in local mail folder has next header,
(imported mail, "compact" is done twice, one for X-Mozilla-Status/Status2, one for X-Mozilla-Keys)  
> From - Sun Sep 13 13:27:43 2009
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> X-Mozilla-Keys: tag-odd                                                                         
> Message-ID: <1KB-MAIL.000001.000001>
mail data uploaded to IMAP server becomes next. (This is Bug 426651. X-Account-Key, X-Mozilla-Keys, X-UIDL is not removed upon upload.)
> X-Mozilla-Keys: tag-odd                                                                         
> Message-ID: <1KB-MAIL.000001.000001>

When this mail is downloaded to offline-store by auto-sync, mail header in offline-store becomes as follows.
("- ...[CRLF]X-Mozilla-Status: ...[CRLF]X-Mozilla-Status2: ..." is lost)
> From 
> X-Mozilla-Keys: tag-odd                                                                         
> Message-ID: <1KB-MAIL.000003.000001>

Although X-Mozilla-Status: and X-Mozilla-Status2: header is missing in offline-store, there is no problem in mail viewing.

Note:
If mail data in IMAP folder(arrived mail at IMAP server), mail data starts with next header,
> Delivered-To: [e-mail]
and mail data in offline-store becomes next. (no problem, as designed.)  
> From - Mon Sep 21 15:30:20 2009
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> Delivered-To: [e-mail]

Comment 1

[David :Bienvenu]

there's no need for the x-mozilla-status headers in the offline store.

Comment 2

[WADA:World Anti-bad-Duping Agency]

No problem with corrupted "From " line? (no '-', no date value)
Problem is "needless x-mozilla-status is written"? (code for "copy to local folder" is partially used for writing to offline-store?)

Comment 3

[WADA:World Anti-bad-Duping Agency]

After "Compact", X-Mozilla-Status/X-Mozilla-Status2 was removed.
(1) mail data in IMAP folder(arrived mail at IMAP server), mail data starts with
next header.
> Delivered-To: [e-mail]
(2) mail data in offline-store (downloaded by auto-sync)
> From - Mon Sep 21 15:30:20 2009
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> Delivered-To: [e-mail]
(3) After "Compact". Minimum "From " line only.
> From 
> Delivered-To: [e-mail]

Phenomenon looks "needless X-Mozilla-Status etc. is written for mail of Gmail".
The mail was sent by Tb(via Gmail's SMTP) to a Gmail account, and downloaded by Tb via Gmail IMAP. It's possibly relevant to "[LF] instead of [CRLF] by Gmail IMAP".
I'll get IMAP log and check new line character sent by Gmail IMAP.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log for Gmail IMAP : [CR] is returned

Comment 5

[WADA:World Anti-bad-Duping Agency]

It was [CR] instead of [LF].

Some log data lines. (get on MS Win-XP, [CR]=0x0D, [CRLF]=0x0D0A)
> CreateNewLineFromSocket: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY[CR][CRLF]
> STREAM:OPEN Size: 938: Begin Message Download Stream[CRLF]
> CreateNewLineFromSocket: Message-ID: <1KB-MAIL.000001.000001>[CR][CRLF]
Line for response from Gmail IMAP always ends with [CR][CRLF].

Gmail IMAP seems to return [CR][CRLF]. Problem like next?
  Mail data is saved(with [CRLF]) => ...[CRLF]
  Data is read with line mode on Unix => ...[CR]
  Gmail IMAP sends it with [CRLF] as RFC requests => ...[CR][CRLF]

Comment 6

[WADA:World Anti-bad-Duping Agency]

Log line for command by Tb. 
> :SendData: 1 capability[CR][CRLF]
The [CRLF] may be written by Tb's logger as newline, and Gmail IMAP may be using standalone [CR] as newline character.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Attachment: Local mail folder file : crafted mail. stand alone [CR], [LF] exists in text mail

Comment 8

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log with Gmail IMAP. Upload the crafted mail, and FTECH it.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Attachment: Offline-store file after download.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(A) Log data lines for APPEND.
({CR}=0x0D, {LF}=0x0A, {CRLF}=0x0D0A, because [CRLF] etc. is used in mail text.)
> Subject: bug.5092780-test-1{CR}{CRLF}
> Content-Type: text/plain{CR}{CRLF}
> {CR}{CRLF}
> Line-1[CRLF]{CR}{CRLF}
> Line-2[CR]{CR}{CRLF}  
> Line-3[LF]{CR}{CRLF}
> Line-4[CRLF]{CR}{CRLF}   (No log for Line-5 after is seen in log) 
> SendData: {CR}{CRLF} 

(1) {CRLF} in log looks newline of log written by logger. (not sent to server)
(2) Tb(9/15 build) treated stand-alone {CR}/{LF} in local mail folder as {CRLF}.
    (Pass thru of {LF} doesn't occur if server uses {CR} instead of {CRLF}?)
     
(B) X-Mozilla-Status etc. was created in offline-store.
> From - Wed Sep 23 15:29:47 2009
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> X-Mozilla-Keys:
Phenomenon is next? {CR} by Gmail IMAP is relevant?
(3) If X-Mozilla-Keys: exists, Tb doesn't write "- Wed Sep ..." and X-Mozilla-xx.
(4) If X-Mozilla-xx: doesn't exists, Tb writes headers in offline-store as done
    for local mail folder.
(5) "Compact" removes such needless data in offline-store.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log for ease of view. 0x0A -> {LF}, 0x0D -> {CR}, 0x0D0A -> {CRLF}+0x0D0A

Comment 12

[WADA:World Anti-bad-Duping Agency]

nb=N looks to be "real data length" + 1. It's design?
> 1196[4129180]: ReadNextLine [stream=4a1a108 nb=2 needmore=0]{CRLF}
> 1196[4129180]: 4ba7400:imap.gmail.com:S-Test-Offline-1/X-02:CreateNewLineFromSocket: {CR}{CRLF}
> 1196[4129180]: ReadNextLine [stream=4a1a108 nb=3 needmore=0]{CRLF}
> 1196[4129180]: 4ba7400:imap.gmail.com:S-Test-Offline-1/X-02:CreateNewLineFromSocket: ){CR}{CRLF}

Comment 13

[WADA:World Anti-bad-Duping Agency]

FYI. "IMAP log for ease of view" was generated by next PHP script(CLI, on MS Win).
> <?php
> $CRLF="\r\n"; $LF="\n"; $CR="\r";
> $fn='C:\\wada\\NSPR\\LOG\\Gmail-IMAP-FETCH-by-auto-sync-with-LF.LOG';
> $fnx=realpath($fn); $log=file($fnx);
> $fn_new=$fnx.'-NEW'; $fh=fopen($fn_new,'wb');
> for($xx=0;$xx<count($log);$xx++)
> {
>   $rr=str_replace($CRLF,'{CRLF}',$log[$xx]);
>   $rr=str_replace($CR,'{CR}',$rr); $rr=str_replace($LF,'{LF}',$rr);
>   $putRC=fwrite($fh,$rr.$CRLF);
> }
> $ClsRC=fclose($fh);
> exit;
> ?>

Comment 14

[Brian Kennelly]

I captured Gimap sessions using openssl, and again using netcat (through stunnel), and the lines returned from Gmail only show CRLF line ends, with no extra or missing CR or LF characters. I also verified that the number of characters returned for the message matched the RFC822.SIZE, which matched length announced in brackets.

In both cases the logging was done by piping through tee.  I do not think tee cleans up the line ends, but if it does, that would invalidate my tests.

Comment 15

[David :Bienvenu]

it doesn't happen on every message by any means, afaik.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #14)
> In both cases the logging was done by piping through tee.

If tee uses "line mode" in file reading instead of "binary mode", original newline character is possibly lost when used with pipe.
> http://en.wikipedia.org/wiki/Tee_(command)
> Note: When tee is used with a pipe, the output of the previous command is written to a temporary file.
> When that command finishes, tee reads the temporary file, displays the output, and writes it to the file(s) given as command-line argument.
PHP's file() is similar to "read in line mode"(CRLF,CR,LF is considered 'newline'), but it keeps the 'newline' at end of each line data, unless user changes PHP's settings or requests removal of it explicitly. It's the reason why I used PHP's file() to check CR/LF/CRLF in Tb's log file.

Comment 17

[Brian Kennelly]

That note does not apply to Linux systems.  I thought about the possibility, but the file was recorded with CRLF after every line, but the "natural" EOL on my system is a bare LF.  If tee was stripping and re-writing, I would not expect to see CRLF.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Oldest IMAP log I have is dated 2008/3/01(for Gmail IMAP, on MS Win). Next is seen in the oldest log too.
> :SendData: 1 capability{CR}{CRLF}
{LF} of {CRLF} was somehow removed or lost upon IMAP logging by Tb?
If so, next may be explained.
> nb=N looks to be "real data length" + 1.

Comment 19

[WADA:World Anti-bad-Duping Agency]

Following is IMAP log data attached to Bug 516914. (After run of my PHP script) 
It seems that {LF} of {CRLF} was somehow removed or lost upon IMAP logging by Tb.

> 2009-08-16 10:33:50.780000 UTC - 2860[3a8d880]: 433ec00:imap.mail.ru:NA:CreateNewLineFromSocket: * OK Mail.Ru IMAP server ready.{CR}{CRLF}
> 2009-08-16 10:33:50.780000 UTC - 2860[3a8d880]: 433ec00:imap.mail.ru:NA:SendData: 1 capability{CR}{CRLF}
> 2009-08-16 10:33:50.790000 UTC - 2860[3a8d880]: ReadNextLine [stream=37eee28 nb=124 needmore=0]{CRLF}
> 2009-08-16 10:33:50.790000 UTC - 2860[3a8d880]: > 433ec00:imap.mail.ru:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4rev1 SORT THREAD=REFERENCES UNSELECT LITERAL+ IDLE CHILDREN LISTEXT LIST-SUBSCRIBED NAMESPACE AUTH=PLAIN{CR}{CRLF}
> 2009-08-16 10:33:50.790000 UTC - 2860[3a8d880]: ReadNextLine [stream=37eee28 nb=28 needmore=0]{CRLF}
> 2009-08-16 10:33:50.790000 UTC - 2860[3a8d880]: 433ec00:imap.mail.ru:NA:CreateNewLineFromSocket: 1 OK Capability completed.{CR}{CRLF}

Comment 20

[WADA:World Anti-bad-Duping Agency]

Problem was bug 697635, excess/useless X-Mozilla-Status:/X-Mozilla-Status2: was generated upon first mail download even though "From " line only is sufficcient as "place holder of a mail".
Duping to that bug.