1911916 - After "Repair Folder" I see 81 old messages appearing at the date/time of the repair ! (Date: header missing)

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Bert Cuzeau]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36

Steps to reproduce:

After a messed up list of Inbox messages, I tried "Repair Folder".
It seemed to have solved the main issue (thousands of messages with no boday accessible).

Actual results:

But now I have 81 older messages (dated between Mon, 27 Mar 2023 and July 27 2024) showing up dated today same time (date time of the Repair Folder execution).
Example of header of this fake "new" messages :
Received: from smtp-3-9003.mail.infomaniak.ch ([10.4.36.133])
by mda-4-0141.mail.infomaniak.ch with LMTP
id M/+qEgxjpGa2wycAZhuTfA
(envelope-from <[e-mail]>)
for <[e-mail]>; Sat, 27 Jul 2024 05:01:32 +0200

Curiously, all these messages come from Ariba SAP accounting system).
<<<<<
Also : they all show up as having attachment(s), but when selected, this tag disappears...

Expected results:

I expected these messages to appear with their correct date time.

Comment 1

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello,

I have tried to reproduce your issue using Windows 10, macOS 14 and Ubuntu 22&24 with 115.14(20240801155430) , 128.0esr(20240710185639) and 128.0.1esr(20240717233102) and 131.0a1(20240806103955) but did not succeed.

If you can still reproduce this issue with the same repro steps, it would very helpful if you could indicate the affected versions on which you are able to reproduce it(besides the affected build from yesterday on which you have encountered it) and if time allows, help us with a regression range for this issue.
I will provide the steps necessary.

You have to determine a build that reproduces the issue.

Then you should find one that does NOT reproduce it. Detailed steps:

a. Open Mozregression app;
b. Click "File" -> "Run a single build";
c. On the "Single Run Wizard" pop-up, "Basic configuration", select "Thunderbird" and click "Next".
d. On the "Profile selection" page, just click the "Next" button.
e. On the "Build selection" select a date (dates before yesterday to have a better chance finding one that does not reproduce the issue) from the drop-down on the left and click "Finish".
f. Now the mozregression app will open a thunderbird build of the selected date and you can use it, close it and open another. (make a note of the version that does not reproduce the issue)

You will use mozregression app to "bisect" builds that reproduce the issue by builds that do not reproduce it in search of the one build/changeset that introduced the issue, in the first place:
a. Open mozregression-gui.exe
b. Click "File" -> "Run a new bisection"
c. On "Basic configuration" screen, select "Thunderbird" and click "Next" button.
d. Skip "Profile selection" screen by the "Next" button.
e. On the Bisection wizard screen, you will need to select a build that reproduces the issue and one that does not:
e1. In the "Last known good build:" section, select "date" on the right drop-down and the date of the build you found NOT to reproduce the issue.
e2. In the "First known bad build:" section, select "date" on the right drop-down and the date of the build you found to reproduce the issue.
f. Click "Finish" to start the bisection process.
g. Builds will open one-by-one, you will need to test each one of them and see whether the issue reproduces. If it reproduces, then you need to select the "bad" button in the mozregression window and if not, you need to select the "good" button.
h. When bisection is done, you will have the information in the "Log View" section of the mozregression window; bisection may also fail due to not enough builds, but the logs can always be useful.
Copy the logs in a text file and attach it to this bug.

If there is still information you need regarding the regression process, please request information from me.
Thank you for your contribution!

Comment 2

[Bert Cuzeau]

I can send you the source code of one of the 81 messages that TB can't anymore parse correctly, not being able to extract the timestamp !
Attached below as failed_email.txt
Please consider that this email is confidential information !

Comment 3

[Bert Cuzeau]

Attachment: failed_email.txt

one of the messages that TB 128 don't parse correctly any more

Comment 4

[Bert Cuzeau]

In terms of regression, it's very simple : all versions I used prior to 128.1.0esr (32-bit) do not have the problem.

Comment 5

[Bert Cuzeau]

Here is what happens in the Error console if I just click on such a message in the list :

...
13:51:04.146 IETNG: mboximportExport.js -v7 mboxImportExport-7.js:39:9
13:51:18.415 Use of nsIFile in content process is deprecated. 3 NetUtil.sys.mjs:249:50
13:51:18.428 Unknown descriptor ‘font-named-instance’ in @font-face rule. Skipped to next declaration. start-style.css:260:23
13:51:18.428 Unknown descriptor ‘font-named-instance’ in @font-face rule. Skipped to next declaration. start-style.css:268:23
13:51:18.646
Unknown property ‘-moz-border-radius’. Declaration dropped. codemirror.css:244:22
13:51:29.108 Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. 2 msgHdrView.js:4214:7
13:51:32.477
Unknown property ‘-moz-border-radius’. Declaration dropped. 2 Ariba>79:69:28
13:51:32.477
Unknown property ‘-moz-border-radius’. Declaration dropped. 2 Ariba>79:75:28
13:51:32.477
Unknown property ‘mso-hide’. Declaration dropped. 2 Ariba>79:86:18
13:51:32.477
Unknown property ‘-moz-border-radius’. Declaration dropped. 2 Ariba>79:105:28
13:51:32.477
Unknown property ‘-moz-border-radius’. Declaration dropped. 2 Ariba>79:117:28
13:51:32.477
Expected declaration but found ‘;’. Skipped to next declaration. 2 Ariba>79:126:67
13:51:32.477
Unknown property ‘-moz-border-radius’. Declaration dropped. 2 Ariba>79:13:24
13:51:32.478 Unknown property ‘-moz-border-radius’. Declaration dropped. 2 Ariba>79:1:47
13:51:32.478 Unknown property ‘-moz-border-radius’. Declaration dropped. 2 Ariba>79:1:172
13:51:32.479 Error in parsing value for ‘vertical-align’. Declaration dropped. 2 service.ariba.com:1:17
13:51:32.484 Unknown property ‘-moz-border-radius’. Declaration dropped. 2 service.ariba.com:1:47
13:51:32.484 Unknown property ‘-moz-border-radius’. Declaration dropped. 2 service.ariba.com:1:172
13:52:16.393 tb.account.size_on_disk - Attempted to set the scalar to an incompatible value. 4
13:52:16.393 tb.account.size_on_disk - Truncating float/double number. 2

​
Note : there also seems to be an old problem of too short integers (32 instead of 64?) to capture the space on HDDs !

Comment 6

[Magnus Melin [:mkmelin]]

The sample message doesn't have a Date: header. Since it doesn't, Thunderbird doesn't know what date it has (Received headers are not normally accessible at the time, in code). If no date is there, the "current date" of receiving will normally be used, and usually that's close enough that you wouldn't care. With the repair, such data was thrown away, now getting the current date.
So while not ideal, it's a bit of garbage in garbage out.

Comment 7

[Bert Cuzeau]

Hi,
This strategy doesn't seem right to me at all, most of all in the "Repair Folder" code !
I suppose that when parsing the remote IMAP "folder", the messages are collected based on the receive date order ?
In that case, the Date could at least be set to the date of the previous message. No ?
I don't know if this also explains why these messages all appear with an attachment tag when there is no attachment ?
Thanks for your help,
Bert

Comment 8

[Bert Cuzeau]

Latest news :
The trouble was that : "You are currently on the esr update channel."
I found on Reddit and other forums how to downgrade (to 115.14).
I had to do it twice because of the auto-update which triggered before I can do anything !
So I disabled auto-update (distribution/policies.json + windows registry key)
But now, I can modify the update setting in the configuration panel ! It says : "You are currently on the esr update channel."
It's a catch-22 :-( If I remove the admin disabling, then TB will auto-update like crazy.

Can I change manually in a text file the auto-update AND GET COMPLETELY OUT OF ESR !!! ?

Thanks in advance,
Bert

Comment 9

[Bert Cuzeau]

I found update-settings.ini
It contains :
[Settings]
ACCEPTED_MAR_CHANNEL_IDS=thunderbird-comm-esr

What should it contain to be removed from ESR ?

Comment 10

[Nicolas Mandil (:NicolasWeb)]

Considering the summary / Title and comment 4 , couldn't it be linked to an auto Folder compaction ? So blocking Bug 1890230

Comment 11

[Magnus Melin [:mkmelin]]

I think I understand now why there's such an uptick in complaints about "current date" being shown on old mails.
We used to output the date into the "From - " part of the mailbox (before bug 1719121), and use that date as a last resort when no other date is available.
Ben, maybe we should consider adding it back?

To reproduce, try putting an mbox like

From - Tue Dec 09 15:30:45 2014
Content-Type: text/plain; charset="iso-8859-1"
Subject: no date.
MIME-Version: 1.0

Test

... into thunderbird. For 115 it displays the 2014 date, for 128 it displays current date.

Comment 12

[Bert Cuzeau]

It was certainly a very bad decision to break a well working code (retrieving the date information reliably instead of not even try).
Like upgrading automatically 64 bits 115 into 32 bits 128 "Nebula" and messing the messages databases :-(
(which was the root cause for having to use the "repair function").

Comment 14

[Francesco]

See bug 1898635 comment #11 for a discussion around the date in the From mbox separator.

Comment 15

[Ben Campbell]

Given that "Date:" and "From:" are the only /required/ fields for a valid email, I'm quite impressed that there are systems out there that are missing one of them :-)

However, it seems reasonable to restore the old "From " line work-around in some form, given that there's affected data out there in the wild...

It's a little tricky: the date was being pulled out of the "From " line by the header parser, but the whole point of the mbox refactoring I did was to hide the mbox details away from the rest of the system! So the header parser never even sees the "From " line any more.
Any "From " parsing would have to be handled further down, in the mbox-specific code, and the date made available by some special out-of-band mechanism, which could get a tad ugly...

However, I think it'd be a little simpler in the specific localfolder "Repair Folder" case, which I think should address this particular issue...

The steps would be:

1. change the mbox code to write out a timestamp in the "From " line. This timestamp would just be whenever the message was written, but that should be close enough to time-of-reception.
2. change the nsMsgLocalMailFolder::ParseFolder() and all the systems below it (StoreIndexer, nsMsgBrkMBoxStore::AsyncScan(), MboxMsgInputStream and MboxMsgOutputStream to allow passing about the out-of-band "From " date.

Caveats would be:

• obviously doesn't work on maildir (although maybe could use the timestamp of the file perhaps?)
• only useful for local folders (I think IMAP just ignores the mbox file entirely and goes by what's on the server... so if there's no Date field there, you're out of luck. I'm pretty sure you can't make any assumptions about order-of-reception from IMAP)

Comment 16

[Alfred Peters [:infofrommozilla]]

(In reply to Ben Campbell from comment #15)

• only useful for local folders (I think IMAP just ignores the mbox file entirely and goes by what's on the server... so if there's no Date field there, you're out of luck. I'm pretty sure you can't make any assumptions about order-of-reception from IMAP)

IMAP will be fixed by Bug 402594 or Bug 570355.

---

(1(1¢)

Comment 17

[Yury]

However, it seems reasonable to restore the old "From " line work-around in some form, given that there's affected data out there in the wild...

Are you planning to restore this soon, since in a month or so, all "envelop dates" will have been lost due to compaction, repairing or simply moving messages?

Point 1 above would be to emit a date here:
https://searchfox.org/comm-central/rev/d2a0beaf18a63423e396d3270eaa673ecf9bbc5d/mailnews/base/src/MboxMsgOutputStream.cpp#127
either a new date, or the date that comes from the message being processed in compaction/repairing/moving/etc.
Since it can't be passed to Write(), it would have to be initialised in the CTOR of MboxMsgOutputStream which isn't called that often at all:
https://searchfox.org/comm-central/search?q=new+MboxMsgOutputStream&path=&case=false&regexp=false
The question is: how it will get there? Maybe point 2 was addressing that. Can you give more details?

The next issue is that the "envelop date" was picked up in code that has since been removed:
https://hg.mozilla.org/comm-central/rev/5dd17b198727#l1.210
BTW, m_envelope_from and m_envelope_date have been made redundant by this changeset without actually being removed:
https://searchfox.org/comm-central/rev/d2a0beaf18a63423e396d3270eaa673ecf9bbc5d/mailnews/local/src/nsParseMailbox.cpp#984-985
https://searchfox.org/comm-central/rev/d2a0beaf18a63423e396d3270eaa673ecf9bbc5d/mailnews/local/src/nsParseMailbox.h#75-76

Comment 18

[Ben Campbell]

I've got some patches lined up which should do the trick.
I still need to write some test cases.
So one thing that would be useful are some examples of real-world "From <SENDER> <DATE>" lines to make sure. I'm not 100% happy about the date parsing (although it's the same data parser the old code used), so it'd be good to have some examples which exercise it properly.

Bert: any chance you could load one of your mboxes in to a text editor and post a few of your "From " lines here? Redacted/changed sender addresses are fine. Just the "From <SENDER> <DATE>" - I'll make up some test messages myself.

Comment 19

[Ben Campbell]

Attachment: Bug 1911916 - Remove dead function MsgGenerateNowStr() in nsMsgUtils. r=#thunderbird-reviewers

Comment 20

[Ben Campbell]

Attachment: Bug 1911916 - Part 1 of 6. Write envelope sender and timestamp values into mbox "From " separator lines. r=#thunderbird-reviewers

If no sender or timestamp are provided, then defaults will be used:
"-" for sender, in line with previous versions of TB.
Current time for received timestamp.

Comment 21

[Ben Campbell]

Attachment: Bug 1911916 - Part 2 of 6. Make MboxMsgInputStream attempt to parse sender and date in "From " lines. r=#thunderbird-reviewers

Comment 22

[Ben Campbell]

Attachment: Bug 1911916 - Part 3 of 6. Change nsIStoreScanListener.onStartMessage() to pass out envAddr and envDate data if available. r=#thunderbird-reviewers

Comment 23

[Ben Campbell]

Attachment: Bug 1911916 - Part 4 of 6. Provide a fallback date during folder repair. For malformed messages missing "Date:" header. r=#thunderbird-reviewers

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/50068ae473f0
Remove dead function MsgGenerateNowStr() in nsMsgUtils. r=aleca

Comment 25

[Ben Campbell]

Attachment: Bug 1911916 - Part 5 of 6. Preserve mbox "From "-line sender/timestamp during compaction. r=#thunderbird-reviewers

Comment 26

[Ben Campbell]

Attachment: Bug 1911916 - Part 6 of 6. Add xpcshell test to check that msgStore is providing timestamps. r=#thunderbird-reviewers

Comment 27

[Ben Campbell]

Attachment: Bug 1911916 - Fix browser_accountTelemetry.js to be less brittle and more forgiving of mbox overhead. r=#thunderbird-reviewers

Comment 28

[Ben Campbell]

A try run with all those patches which I think looks OK:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&fromchange=66e6d678ae02c2f88bb16aa78fa20e0544c76dae

(some orange warnings, but I'm pretty sure they're all due to other issues)

It'd be nice if someone who was experiencing this issue (Bert?) could give the try build a test to see if it fixes the issue for them...
you can install a build from the treeherder link above:

From the list on the right pick the build you want (eg "Windows 2012 x64 opt") and click the "B" to it's right.
Then click the "Artifacts and Debugging Tools" tab in the bottom pane.
Download the thing that looks most like an installer ("target.installer.exe"? "setup.exe"? Could have sworn there was an .msi file generated last time I looked at this... sorry, not so familiar with the details here, especially on windows)

Comment 29

[Bert Cuzeau]

All I could do was providing a sample mailbox with messages affected by the problem. I hope it was useful.
I have 46926 messages in my main Inbox, all critical to my business, so I can't afford the risk to run a beta version.
Getting back to 115 was also a scary process and I have disabled all the automatic updates to stay at 115.14.0.
I hope the TB update process will be fixed and back under control to avoid the mess I and many other users experienced with 128.
Note : the messages without a Date: header were all coming from SAP/Ariba, and they seem to have fixed this.

Comment 30

[Ben Campbell]

Sure thing - thanks for reporting it!
Thanks also to Yuri for doing the digging he did!

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/dcdd28ba3484
Part 1 of 6. Write envelope sender and timestamp values into mbox "From " separator lines. r=tobyp,mkmelin
https://hg.mozilla.org/comm-central/rev/92810f1a9183
Part 2 of 6. Make MboxMsgInputStream attempt to parse sender and date in "From " lines. r=tobyp
https://hg.mozilla.org/comm-central/rev/149a6a71b20a
Part 3 of 6. Change nsIStoreScanListener.onStartMessage() to pass out envAddr and envDate data if available. r=tobyp
https://hg.mozilla.org/comm-central/rev/4fe9c4fc0e3c
Part 4 of 6. Provide a fallback date during folder repair. For malformed messages missing "Date:" header. r=tobyp
https://hg.mozilla.org/comm-central/rev/7ed6bc2f25ea
Part 5 of 6. Preserve mbox "From "-line sender/timestamp during compaction. r=tobyp
https://hg.mozilla.org/comm-central/rev/c5466c2e3d69
Part 6 of 6. Add xpcshell test to check that msgStore is providing timestamps. r=tobyp,mkmelin
https://hg.mozilla.org/comm-central/rev/4a8df87d0585
Fix browser_accountTelemetry.js to be less brittle and more forgiving of mbox overhead. r=tobyp

Comment 33

[Yury]

If tested four cases:

1. Mbox file from comment 11: That now picks up the envelop date.
2. Compact: That now retains the original envelop date.
3. Repair: I'm confused, that doesn't modify the mbox file at all, so the envelop dates are preserved. My understanding is that Repair only rebuilds the MSF file, so how could it cause the bug in the first place?
4. Move/copy a local message from one folder to another. There the envelop date is not preserved like it used to be. Is that by design or is the implementation missing for this? If the implementation is missing, could you please give some pointers?

Comment 34

[Ben Campbell]

(In reply to Yury from comment #33)

3. Repair: I'm confused, that doesn't modify the mbox file at all, so the envelop dates are preserved. My understanding is that Repair only rebuilds the MSF file, so how could it cause the bug in the first place?

Previously, the header parser assumed that it was always dealing with a whole mbox file, and did the "From " parsing itself (along with dealing with multiple messages and escaping "From " lines in the message bodies - it got really complicated and messy).
But that is no longer the case - the parser only ever sees a single message at a time, never sees the "From " separator lines and has no idea that if it's getting data from mbox or somewhere else.
So we needed some out-of-band way to feed a timestamp (read from the "From " line for mbox, and I think I set it up to use the file timestamp for maildir) into the parser, so it has it available as a last resort if there's a malformed message with no "Date:" header.
That's what this patch does:
https://hg.mozilla.org/comm-central/rev/4fe9c4fc0e3c
(it builds on the earlier patches which add functionality to the lower-level mbox code to parse the "From " lines and pass that metadata up to interested parties. For example, the StoreIndexer (repair) code in this patch is an interested party!)

4. Move/copy a local message from one folder to another. There the envelop date is not preserved like it used to be. Is that by design or is the implementation missing for this? If the implementation is missing, could you please give some pointers?

Partly it's a quirk of the design, partly it's the implementation that's missing.
What should happen when messages are copied between folders is that the source nsMsgHdr should be duplicated in the destination folder's database. That would carry the Date entry across, even if the message has no "Date: " header and was taken from an mbox "From " line.
However, most of the message-copy pathways seem to pass the message through the header parser to build up a new nsMsgHdr during the copy, so if the "Date: " header is missing, it just uses the current time (I think). There's currently nothing feeding any out-of-band timestamps into that destination header parser.

You could probably figure out ways to obtain the out-of-band timestamp from the source and pass it along as part of the copy (if it's even available - not all copies are from local messages, you could be copying an online-only IMAP message direct from the server, say)... but it'd be fiddly and there are lots of different pathways to fix.
But really, all the message copy code needs to be ripped out and simplified, with the common aspects extracted and the protocol-specific parts made more robust. That's a really big job.
Bug 1731177 covers some of the issues involved, but just for local folders. IMAP is much more complex.

Comment 37

[Geoff Lankow (:darktrojan)]

Time for uplift?

Comment 38

[Ben Campbell]

Not sure - is it worth it? It's a little niche (i.e. only for messages without a "Date:" field)...
But if it applies cleanly, I'd be happy with it being uplifted.

Comment 39

[Magnus Melin [:mkmelin]]

Yeah I'd uplift this, there's a fair amount of complaints about such messages.

Comment 40

[Wayne Mery (:wsmwk)]

@benc, from #tbdrivers, "Should we uplift" ? (128 building soon)

Comment 41

[Ben Campbell]

OK then, sounds like uplift is worthwhile. But I think someone else should request it - I'm pretty hazy on version numbers and releases.

Comment 43

[Geoff Lankow (:darktrojan)]

Comment on attachment 9424037 [details]
Bug 1911916 - Remove dead function MsgGenerateNowStr() in nsMsgUtils. r=#thunderbird-reviewers

[Approval Request Comment]
Regression caused by (bug #): bug 1719121
User impact if declined: messages have the wrong dates - the time they arrived in the folder instead of when they were actually sent
Testing completed (on c-c, etc.): has been right through the 132 beta cycle
Risk to taking this patch (and alternatives if risky): hard to say, probably no riskier than not taking it

Request is for all 8 patches on this bug plus the backout of https://hg.mozilla.org/releases/comm-esr128/rev/84f49037eb35f442355c7bceb74cda9982bc44e6. I've started a try run to check there's no test failures introduced.

Edit: There's an ESR version of the test patch, in comment 45.

Comment 44

[Geoff Lankow (:darktrojan)]

[Duplicate approval request removed.]

Comment 45

[Geoff Lankow (:darktrojan)]

Attachment: WIP: Bug 1911916 - [ESR128] Fix browser_accountTelemetry.js to be less brittle and more forgiving of mbox overhead.

Comment 46

[Corey Bryant]

Geoff, I wonder if we still want to hold off on uploading this to 128 in order to not muddy the waters with message corruption fixes that went into 128.3.3esr. What do you think?

Comment 47

[Geoff Lankow (:darktrojan)]

I think it's a separate-enough issue. And even if it muddies the waters all we need to know is whether the problems stop or not.

Comment 48

[Corey Bryant]

Comment on attachment 9424037 [details]
Bug 1911916 - Remove dead function MsgGenerateNowStr() in nsMsgUtils. r=#thunderbird-reviewers

[Triage Comment]
Approved for esr128
Note: There are 8 patches to uplift, one of them being the 128-specific patch titled "Fix browser_accountTelemetry.js to be less brittle and more forgiving of mbox overhead. "

Comment 49

[Daniel Darnell [:dandarnell]]

Thunderbird 128.4.2esr:
https://hg.mozilla.org/releases/comm-esr128/rev/43a3c8a8d8e3
https://hg.mozilla.org/releases/comm-esr128/rev/2da8ec998a01
https://hg.mozilla.org/releases/comm-esr128/rev/85d795e2e8e1
https://hg.mozilla.org/releases/comm-esr128/rev/82e9d8759d7d
https://hg.mozilla.org/releases/comm-esr128/rev/ea7181efba63
https://hg.mozilla.org/releases/comm-esr128/rev/725d438f74c0
https://hg.mozilla.org/releases/comm-esr128/rev/945d8ab25c83
https://hg.mozilla.org/releases/comm-esr128/rev/18c64db8003e