Massive MozMill failure on 2019-01-10: 300+ failing tests, 3pane not showing, Daily completely busted

RESOLVED FIXED in Thunderbird 66.0

Status

defect
--
blocker
RESOLVED FIXED
4 months ago
4 months ago

People

(Reporter: jorgk, Assigned: mkmelin)

Tracking

({regression})

unspecified
Thunderbird 66.0
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

Reporter

Description

4 months ago

Looks like almost all tests failed, in many cases teardownModule:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=b7fe10c2833b71275de4a8583b193f1a30f3e199

Check the Nightly run.

M-C last good: 9d40c98b9efbf728f7fa542150ca8023df
M-C first bad: 74bb778f78793e82cfcae11446387795cb
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9d40c98b9efbf728f7fa542150ca8023df&tochange=74bb778f78793e82cfcae11446387795cb

Only 22 changeset. To me it looks like bug 1441935, but it could be bug 1454357.

About the former, Magnus said: With luck, it shouldn't affect us. Well, no luck :-(

Flags: needinfo?(mkmelin+mozilla)
Reporter

Comment 1

4 months ago

Looking at the log
https://taskcluster-artifacts.net/ORBrFNo5QNeGYmwojamIPA/0/public/logs/live_backing.log
I see things like:

INFO - EXCEPTION: this._treeElement is undefined
INFO - at: folderPane.js line 412
INFO - get mode folderPane.js:412 7
INFO - teardownModule test-folder-display-helpers.js:364 5

Also heaps of JS errors:

JavaScript error: chrome://messenger/content/msgViewPickerOverlay.js, line 95: TypeError: gFolderDisplay is null
JavaScript error: file:///Users/cltbld/tasks/task_1547118557/build/tests/mozmill/shared-modules/test-window-helpers.js, line 1563: TypeError: target.ownerDocument is null

Not sure what hit us there.

Reporter

Comment 2

4 months ago

Running it locally, the UI doesn't come up. No folders, and I see:
JavaScript error: chrome://messenger/content/specialTabs.js, line 623: NS_ERROR_XPC_BAD_CONVERT_JS: Could not convert JavaScript argument arg 0 [nsIObserverService.addObserver]

openSpecialTabsOnStartup() {
let browser = document.getElementById("dummycontentbrowser");

// Manually hook up session and global history for the first browser
// so that we don't have to load global history before bringing up a
// window.
// Wire up session and global history before any possible
// progress notifications for back/forward button updating
browser.docShell.initSessionHistory();

623 Services.obs.addObserver(browser, "browser:purge-session-history");

There's your browser issue. No luck at all. Over to Magnus.

Reporter

Updated

4 months ago
Summary: Massive MozMill failure on 2019-01-10: 300+ failing tests → Massive MozMill failure on 2019-01-10: 300+ failing tests, 3pane not showing, Daily completely busted
Assignee

Comment 3

4 months ago

Ah yes. Custom elements don't like to be QId.

Assignee: nobody → mkmelin+mozilla
Flags: needinfo?(mkmelin+mozilla)
Reporter

Comment 4

4 months ago

If it's easy to fix, I can re-run the nightly.

Assignee

Comment 5

4 months ago

It should be. My build should finish in a short while.
Assuming this is causing all the other failures as a consequence of course.

Comment 6

4 months ago

Please disable automatic updates until we have a solution as this totally breaks Daily.

Reporter

Comment 7

4 months ago

Not the first time it got broken like this. We try to avoid it, but we're not always lucky :-(

Comment 8

4 months ago

Not a problem. I expect to hit issues on Daily and accept the risk. I'm just hoping not to freak out or drive away valuable Daily users.

Assignee

Comment 9

4 months ago

At least part of the fix. Didn't test more than briefly yet

Attachment #9035608 - Flags: review?(jorgk)
Assignee

Updated

4 months ago
Severity: normal → blocker
Keywords: regression
Assignee

Updated

4 months ago
Status: NEW → ASSIGNED
Reporter

Comment 10

4 months ago
Comment on attachment 9035608 [details] [diff] [review]
bug1519078_mozmill_fail_due_to_tabbrowser_ce.patch

Review of attachment 9035608 [details] [diff] [review]:
-----------------------------------------------------------------

Works OK. Let's see how it fares:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=373e34f82fd686c89f19a7d5b9d37da3fa196293

If it's OK, I'll run a new Daily with it.
Attachment #9035608 - Flags: review?(jorgk) → review+

Comment 11

4 months ago

Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/cb009c395002
use the observer helper object of the <browser> which is now a CE. r=jorgk DONTBUILD

Status: ASSIGNED → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → FIXED
Reporter

Comment 12

4 months ago

I ran a few random tests and they all looked OK. Let's take the risk an retrigger the Daily.

Target Milestone: --- → Thunderbird 66.0
Assignee

Comment 13

4 months ago

Chat seems to be busted though
JavaScript error: chrome://messenger/content/chat/imconversation.xml, line 1638: TypeError: this.browser.init is not a function

Reporter

Comment 14

4 months ago

Please keep an eye on the tree for other failures. I'm mostly afk for the rest of the day.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee

Comment 15

4 months ago

Filed the im change as bug 1519091.

Status: REOPENED → RESOLVED
Last Resolved: 4 months ago4 months ago
Resolution: --- → FIXED
Reporter

Comment 16

4 months ago

Magnus, there are still a bunch of Calendar test failures left. Do you want to fix them here or in another bug?

I've run
mozmake -C comm/calendar/test/mozmill SOLO_TEST=testTodayPane.js mozmill-one

I see

JavaScript error: chrome://calendar/content/calendar-multiday-view.xml, line 2532: TypeError: document.getAnonymousElementByAttribute(...) is null

And that code is:

      let scrollbox = document.getAnonymousElementByAttribute(
                     this, "anonid", "scrollbox");
      document.getAnonymousElementByAttribute(
          scrollbox, "class", "box-inherit scrollbox-innerbox").flex = "1";

That seems to come from bug 1454357.

Assignee

Comment 17

4 months ago

For calendar, bug 1519114

Assignee

Updated

4 months ago
Blocks: 1441935
Reporter

Comment 19

4 months ago

Are you in the wrong bug here?

Flags: needinfo?(mgaudet)

Sure typo'd that one. Thanks Jorg!

Flags: needinfo?(mgaudet)
Attachment #9041472 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.