639702 - Negative offset value for replied mail is written in .msf of Outbox(Unsent Messages.msf) and Drafts(Drafts.msf), if offset of replied mail in local mail folder is greater than 2GB, then "replied mark" is not set after send of reply mail

Status: RESOLVED FIXED

(MailNews Core :: Composition, defect)

Description

[WADA:World Anti-bad-Duping Agency]

Negative offset value for replied mail is written in .msf of Outbox(Unsent Messages.msf) and Drafts(Drafts.msf), if offset of replied mail in local mail folder is grater than 2GB, then "replied mark" is not set after send of the reply mail.
This phenomenon was found during analysis of next report to a forum in Japan.
  Replied mark is never set by any of Send Now, Draft Save/Edit Draft/Send Now,
  Send Later/Send Unsent Messages. 

Replied mail : in Big-Mbox2/2.2GB-Mbox (local mail folder)
               offset("Order Received" column value) = 2361393152
Data in Unsent Messages.msf, after Reply/Send Later.
> (8F=mailbox-message://x@x.x.x/Big-Mbox2/2.2GB-Mbox#-1933574144)
Data in Drafts.msf, after Reply/Save As Draft.
> (9C=mailbox-message://x@x.x.x/Big-Mbox2/2.2GB-Mbox#-1933574144)

2GB = 2**31 = 2147483648
2361393152 = 2**31 + 213909504
4GB = 2**32 = 4294967296
1933574144 = 2**32-2361393152

It looks that 32bits signed integer is still used for offset of replied mail.
I guess same problem can occur when "Send Now". 

Note-1:
On Win, "Order Received" column value is already shown as positive value even when offset is greater than 2GB. 

Note-2:
As offset value of replied mail is written in .msf upon mail composition and is used to search replied mail after mail send, "replied mark" can not be set if offset of the replied mail in local mail folder is changed by auto-compact or Compact Folder/Compact Folders before send of reply mail.
To resolve this kind of issue, "Unique mail identifier of local mails in whole Tb system" instead of "offset in local file" will be needed, unless "offset change notification" like feature will be implemented.

Comment 1

[:aceman]

WADA, can you please retest this in a recent version of TB. You know David Bienvenu did the rewrite for supporting pluggable mailstores and some of these 2GB/4GB issues have been affected by that change.

Comment 2

[:aceman]

We have enabled support for folders larger than 4GB in current nightlies, so file offsets should be 64bit long. Can you still reproduce the problem in some case?

Comment 3

[WADA:World Anti-bad-Duping Agency]

Following is Unsent Messages.msf content for following reply mail.
  Name Thunderbird
  Version  50.0a1
  Application Build ID  20160801030226
  Reply to mail of messageKey=2, messageOffset=785 in Inbox of a POP3 account
  Send Later 
> @$${8{@
> < <(a=c)> // (f=iso-8859-1)
>   (C6=pseudoHdr)(C7=origURIs)(C8=queuedDisposition)(C9=X-Identity-Key)>
> <(99=mailbox-message://a1%40a.a.a@a.a.a/Inbox#2)(9A=replied)(9B=id1)>

It looks that messegeKey value is correctly used as "messageKey instead of Offset" after support for folders larger than 4GB.
Confusion like "messageKey value = Offset of the mail" was perhaps removed completely by fixing Bug 793865.