1911951 - after upgrade to v128 mail tree view is gone - for hostname "invalid.192.168.1.2"

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[John]

Steps to reproduce:

Update from v115 tot v128.

Actual results:

The left window normally showing a tree of accounts is not displayed at all. The space is blank. A single message which was already opened is displayed at the right in a tab.

Comment 1

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello,

I have tried to reproduce your issue using Windows 10, macOS 14 and Ubuntu 22&24 updating from 115.14(20240801155430) to 128.0esr(20240710185639) and 128.0.1esr(20240717233102) but did not succeed.

If you can still reproduce this issue with the same repro steps, it would very helpful if you could indicate the affected versions on which you are able to reproduce it(besides the affected build from today on which you have encountered it) and if time allows, help us with a regression range for this issue.
I will provide the steps necessary.

You have to determine a build that reproduces the issue.

Then you should find one that does NOT reproduce it. Detailed steps:

a. Open Mozregression app;
b. Click "File" -> "Run a single build";
c. On the "Single Run Wizard" pop-up, "Basic configuration", select "Thunderbird" and click "Next".
d. On the "Profile selection" page, just click the "Next" button.
e. On the "Build selection" select a date (dates before today to have a better chance finding one that does not reproduce the issue) from the drop-down on the left and click "Finish".
f. Now the mozregression app will open a thunderbird build of the selected date and you can use it, close it and open another. (make a note of the version that does not reproduce the issue)

You will use mozregression app to "bisect" builds that reproduce the issue by builds that do not reproduce it in search of the one build/changeset that introduced the issue, in the first place:
a. Open mozregression-gui.exe
b. Click "File" -> "Run a new bisection"
c. On "Basic configuration" screen, select "Thunderbird" and click "Next" button.
d. Skip "Profile selection" screen by the "Next" button.
e. On the Bisection wizard screen, you will need to select a build that reproduces the issue and one that does not:
e1. In the "Last known good build:" section, select "date" on the right drop-down and the date of the build you found NOT to reproduce the issue.
e2. In the "First known bad build:" section, select "date" on the right drop-down and the date of the build you found to reproduce the issue.
f. Click "Finish" to start the bisection process.
g. Builds will open one-by-one, you will need to test each one of them and see whether the issue reproduces. If it reproduces, then you need to select the "bad" button in the mozregression window and if not, you need to select the "good" button.
h. When bisection is done, you will have the information in the "Log View" section of the mozregression window; bisection may also fail due to not enough builds, but the logs can always be useful.
Copy the logs in a text file and attach it to this bug.

If there is still information you need regarding the regression process, please request information from me.
Thank you for your contribution!

Comment 2

[Wayne Mery (:wsmwk)]

Also...

What OS do you run?
And how (and when) did you get to version 128, did you do a manual Help > About to update?

Comment 3

[John]

Windows 11 23H2 up to date (a personal computer I manage, but do not own or use)

The first update was manual, v128 was not being served yet through the update function. Using --allow-downgrade a downgrade was performed. After some time I retried installing V128 through the update function. It failed in the same way.

The user likes to clean up emails regularly. Compressing the mailboxes did not help (third attempt). The user has some old defunct mailboxes and one remote account at an ISP that is the current working account. Thunderbird has been used for about 15 years.

Due to privacy considerations, I cannot install a networked data collection tool. If you are looking for something specific and offline, I may be able to help.

Comment 4

[Magnus Melin [:mkmelin]]

Check the Error Console (Ctrl+Shift+J). Do you see any errors of relevance?

Comment 5

[John]

Nothing out of the ordinary in v115.3 (current version). I'm currently backing up the profile, parts of the registry and user environment.

Comment 6

[John]

Installed 128.1 over 115.3, then started Thunderbird, results in these error log messages:

* Uncaught (in promise) NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.server] folder-tree-row.mjs:399
    setFolderPropertiesFromFolder chrome://messenger/content/folder-tree-row.mjs:399
    setServer chrome://messenger/content/folder-tree-row.mjs:262
    _createServerRow chrome://messenger/content/about3Pane.js:2135
    initServer chrome://messenger/content/about3Pane.js:962
    _initMode chrome://messenger/content/about3Pane.js:2121
    _toggleMode chrome://messenger/content/about3Pane.js:2069
    set activeModes chrome://messenger/content/about3Pane.js:1972
    init chrome://messenger/content/about3Pane.js:1611

* Uncaught (in promise) TypeError: win.messageBrowser is undefined mailTabs.js:102:13
    openTab chrome://messenger/content/mailTabs.js:102

folder-tree-row.mjs:399 is this line:
if (folder.server.type != "rss") {

More errors appear minutes after this.

Comment 7

[John]

I ran Sysinternals Process Monitor during start of Thunderbird.
The error log happens as Thunderbird is reading C:\Program Files\Mozilla Thunderbird\omni.ja. That may be unrelated. Tenths of a second before Thunderbird is reading folderTree.json and other folder or tree related files. Also reading prefs.js and sqlite files.

Comment 8

[John]

Tried creating a new profile and importing emails only from a backup. It does not display any message in the import folder. The number of messages is shown as 0.

I guess there is something in the mail files that v128.1 does not like.

While testing sending and receiving messages, at some point messages in the inbox were no longer shown. They were all related (replies), sending a completely new message from another account, using a different subject did show newly received messages, but not the old messages. The number of messages was the total of all messages, inclusing those no longer visible. Manually inspecting the mailbox file did not reveal anything interesting, messages have From as the first line. After the msf file for the Inbox file was deleted, the unseen messages appeared again.

The previous mbox style first line looks like this: From - Sun Jul 26 10:01:16 2009. The current style seems to be just From. That's From and a ASCII 32 decimal (0x20) space character.

Unexpected content or format may be an issue on the original mail folder. Perhaps there are messages that contain forbidden characters or a date which is considered invalid? Dates may have extra hours (24 or more) or leap seconds (60 or 61). Or perhaps messages not containing a required SMTP or Thunderbird header?

Comment 9

[John]

Date formats found using regular expression ^From\

From - Tue Sep 29 16:56:13 2020
From - Wed Oct  2 16:24:44 2019
From - Wed Apr 01 11:06:40 2015
From - Wed, 01 Dec 2021 13:58:01 GMT

Comment 10

[John]

I have not found any From mbox message header that looked wrong and checked some files against control character. I just found a few non-encoded UTF8 characters (e.g. E2 80 99).

I tried importing various top level folders on another computer with an existing install (v128.1). These imports failed. Messages did not become available. This issue may be related to #1907984 .

Reproduce using a sbd folder containing :
#method 1
Import from another Thunderbird installation
Choose a profile folder
Select Mail Messages
Start Import
Finish (restarts)

#method 2
Import from a file
Import backed-up Profile
Choose a profile folder
Select Mail Messages
Start Import
Finish (restarts)

This creates empty Thunderbird Import and Thunderbird Import1 numbered folders that are empty. I tried to import copied local folders too with the same result.

Manually copying mbox files did not work either.

Comment 11

[Magnus Melin [:mkmelin]]

John, please file another bug for the import issue (and reference it in this bug).

For this bug (blank 3pane), can you send me your prefs.js file from the profile? Or if you can, debug it yourself: check if one of the hostname prefs is set to something unexpected. Remove the mail.server.server###.hostname prefs one by one and try starting up, checking which one is causing the issue.

Comment 12

[John]

Attachment: screenshot.png

Screenshot of the 'bad' server folder.

Comment 13

[John]

(In reply to Magnus Melin [:mkmelin] from comment #11)

check if one of the hostname prefs is set to something unexpected. Remove the mail.server.server###.hostname prefs one by one and try starting up, checking which one is causing the issue.

I check one by one. Display came back after this one was removed:
user_pref("mail.server.server#.hostname", "invalid.192.168.1.2");

The host was renamed to invalid.192.168.1.2 at some point so it wouldn't work any more.

The mail files in the directory are zero sized. See the uploaded image screenshot.png.

Junk is redirected to the current working account.
user_pref("mail.server.server#.spamActionTargetFolder", "mailbox://[e-mail]/Junk");

Comment 14

[Magnus Melin [:mkmelin]]

Thanks for checking!

Comment 15

[kompre]

Hey, I've got the same problem after upgrading to 128.1.0esr.

I've tried to remove mail.server.server###.hostname and narrowed it down to the local accounts (created with the extension localfolders) which name have the following naming schema:

[server] [e-mail]

the culprit seems to be the @ in the string.

To reproduce, just create a new local folder:

1. insert a name with @ inside and choose whatever directory path you like
2. click on All to initiate the message folder
3. click on OK
4. an alert window will pop up and thunderbird will be blank at the next startup

(markdown preview is not showing the screenshot, so here the link in case https://imgur.com/2lIc5V4)

Comment 16

[Toby Pilling]

Attachment: Bug 1911951 - Handle folder tree rendering for invalid servers. r=mkmelin,#thunderbird-reviewers

== How to test ==

• Use a profile with a few accounts
• Change the pref "hostname" value of one of your IMAP accounts (e.g. mail.server.server2.hostname) to be "invalid.192.168.1.2"
• Close TB
• Restart
• [server/folder tree should be visible rather than blank]

Comment 17

[Toby Pilling]

Thanks for reporting this kompre.

Your problem looks a lot like this issue reported to the extension owner so it might be worth following up there.

We will work on gracefully handling problems so the tree still renders in the case of an invalid server url but it may not take care of cases involving this extension.

Comment 18

[Magnus Melin [:mkmelin]]

Did some regression hunting. This is from bug 1723456.

var spec = "imap://test@invalid.192.168.1.2";
Cc['@mozilla.org/network/standard-url-mutator;1'].createInstance(Ci.nsIURIMutator).setSpec(spec).finalize();

-> failure

But.... I notice

Services.io.isValidHostname("invalid.192.168.1.2");

-> true, which is lying then. Hmm :-/

Comment 19

[Magnus Melin [:mkmelin]]

Note that bug 1908033 (due to bug 1906992) made changes in the area and those are only on trunk.

Comment 20

[Magnus Melin [:mkmelin]]

Attachment: Bug 1911951 - Fix hostname validity check - don't create servers with hostnames that could not be used. r=babolivier,tobyp

Comment 21

[Toby Pilling]

Attachment: Bug 1911951 - Write test coverage for adding contacts via sidebar using keyboard. r=freaktechnik,#thunderbird-front-end-reviewers

Comment 22

[Phabricator Automation]

Comment on attachment 9420922 [details]
Bug 1911951 - Write test coverage for adding contacts via sidebar using keyboard. r=freaktechnik,#thunderbird-front-end-reviewers

Revision D220206 was moved to bug 1915022. Setting attachment 9420922 [details] to obsolete.

Comment 23

[betterbird.project+18]

This

      hostname.Truncate();
      hostname.Append(key);
      hostname.Append(".invalid");

can be replaced by hostname = key + ".invalid"_ns or even hostname = key + ".invalid" ? Or at least hostname.Assign(key).

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/d2de072c0cfc
Fix hostname validity check - don't create servers with hostnames that could not be used. r=babolivier,tobyp

Comment 25

[Magnus Melin [:mkmelin]]

Comment on attachment 9420121 [details]
Bug 1911951 - Fix hostname validity check - don't create servers with hostnames that could not be used. r=babolivier,tobyp

[Approval Request Comment]
Regression caused by (bug #): bug 1723456 (and probably other bugs as well)
User impact if declined: blank 3pane
Testing completed (on c-c, etc.): c-c
Risk to taking this patch (and alternatives if risky): should bake on beta for a while. 128 will need a slightly modified patch due to post-128 changes

Comment 26

[Corey Bryant]

Comment on attachment 9420121 [details]
Bug 1911951 - Fix hostname validity check - don't create servers with hostnames that could not be used. r=babolivier,tobyp

[Triage Comment]
Approved for beta

Comment 27

[Daniel Darnell [:dandarnell]]

Thunderbird 131.0b1:
https://hg.mozilla.org/releases/comm-beta/rev/675bf3ba3b12

Comment 28

[Corey Bryant]

Magnus, when ready, can you link the modified 128 patch?

Comment 29

[Magnus Melin [:mkmelin]]

Getting a 128 version was more complicated than I had imagined. Trying to uplift all the dependencies was in the end not a good alternative, so I modified the trunk version to just do the bare minimal which is handling invalid servers coming from the prefs.
Try run here: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=a1209914ac5e658f026ba2913b85ef633609a199&selectedTaskRun=H8eNTl75Q8i8pTx_PI_Yxw.0
Can grab the patch for uplift from https://hg.mozilla.org/try-comm-central/rev/8e8e375acc8a275600db771cf909322adba9dac2?

Comment 30

[Corey Bryant]

Comment on attachment 9420121 [details]
Bug 1911951 - Fix hostname validity check - don't create servers with hostnames that could not be used. r=babolivier,tobyp

[Triage Comment]
Approved for esr128

As mentioned by Magnus, the esr128 patch handles invalid hostnames coming from prefs only, which the bug was filed for. It does not include the validation of new accounts with invalid hostnames.

Comment 31

[Daniel Darnell [:dandarnell]]

(In reply to Magnus Melin [:mkmelin] from comment #25)

Comment on attachment 9420121 [details]
Bug 1911951 - Fix hostname validity check - don't create servers with hostnames that could not be used. r=babolivier,tobyp

[Approval Request Comment]
Regression caused by (bug #): bug 1723456 (and probably other bugs as well)
User impact if declined: blank 3pane
Testing completed (on c-c, etc.): c-c
Risk to taking this patch (and alternatives if risky): should bake on beta for a while. 128 will need a slightly modified patch due to post-128 changes

This patch does not uplift properly to 128, and will need a 128-specific patch.

Comment 32

[Magnus Melin [:mkmelin]]

128 patch in comment 29

Comment 33

[Daniel Darnell [:dandarnell]]

Thunderbird 128.2.2esr:
https://hg.mozilla.org/releases/comm-esr128/rev/7016fbe576c5

Comment 34

[Daniel Darnell [:dandarnell]]

(In reply to Magnus Melin [:mkmelin] from comment #32)

128 patch in comment 29

My bad, I just glossed over it. Thank you!