1519078 - Massive MozMill failure on 2019-01-10: 300+ failing tests, 3pane not showing, Daily completely busted

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Jorg K (CEST = GMT+2)]

Looks like almost all tests failed, in many cases teardownModule:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=b7fe10c2833b71275de4a8583b193f1a30f3e199

Check the Nightly run.

M-C last good: 9d40c98b9efbf728f7fa542150ca8023df
M-C first bad: 74bb778f78793e82cfcae11446387795cb
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9d40c98b9efbf728f7fa542150ca8023df&tochange=74bb778f78793e82cfcae11446387795cb

Only 22 changeset. To me it looks like bug 1441935, but it could be bug 1454357.

About the former, Magnus said: With luck, it shouldn't affect us. Well, no luck :-(

Comment 1

[Jorg K (CEST = GMT+2)]

Looking at the log
https://taskcluster-artifacts.net/ORBrFNo5QNeGYmwojamIPA/0/public/logs/live_backing.log
I see things like:

INFO - EXCEPTION: this._treeElement is undefined
INFO - at: folderPane.js line 412
INFO - get mode folderPane.js:412 7
INFO - teardownModule test-folder-display-helpers.js:364 5

Also heaps of JS errors:

JavaScript error: chrome://messenger/content/msgViewPickerOverlay.js, line 95: TypeError: gFolderDisplay is null
JavaScript error: file:///Users/cltbld/tasks/task_1547118557/build/tests/mozmill/shared-modules/test-window-helpers.js, line 1563: TypeError: target.ownerDocument is null

Not sure what hit us there.

Comment 2

[Jorg K (CEST = GMT+2)]

Running it locally, the UI doesn't come up. No folders, and I see:
JavaScript error: chrome://messenger/content/specialTabs.js, line 623: NS_ERROR_XPC_BAD_CONVERT_JS: Could not convert JavaScript argument arg 0 [nsIObserverService.addObserver]

openSpecialTabsOnStartup() {
let browser = document.getElementById("dummycontentbrowser");

// Manually hook up session and global history for the first browser
// so that we don't have to load global history before bringing up a
// window.
// Wire up session and global history before any possible
// progress notifications for back/forward button updating
browser.docShell.initSessionHistory();

623 Services.obs.addObserver(browser, "browser:purge-session-history");

There's your browser issue. No luck at all. Over to Magnus.

Comment 3

[Magnus Melin [:mkmelin]]

Ah yes. Custom elements don't like to be QId.

Comment 4

[Jorg K (CEST = GMT+2)]

If it's easy to fix, I can re-run the nightly.

Comment 5

[Magnus Melin [:mkmelin]]

It should be. My build should finish in a short while.
Assuming this is causing all the other failures as a consequence of course.

Comment 6

[Bob Clary [:bc] (inactive)]

Please disable automatic updates until we have a solution as this totally breaks Daily.

Comment 7

[Jorg K (CEST = GMT+2)]

Not the first time it got broken like this. We try to avoid it, but we're not always lucky :-(

Comment 8

[Bob Clary [:bc] (inactive)]

Not a problem. I expect to hit issues on Daily and accept the risk. I'm just hoping not to freak out or drive away valuable Daily users.

Comment 9

[Magnus Melin [:mkmelin]]

Attachment: bug1519078_mozmill_fail_due_to_tabbrowser_ce.patch

At least part of the fix. Didn't test more than briefly yet

Comment 10

[Jorg K (CEST = GMT+2)]

Comment on attachment 9035608 [details] [diff] [review]
bug1519078_mozmill_fail_due_to_tabbrowser_ce.patch

Review of attachment 9035608 [details] [diff] [review]:
-----------------------------------------------------------------

Works OK. Let's see how it fares:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=373e34f82fd686c89f19a7d5b9d37da3fa196293

If it's OK, I'll run a new Daily with it.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/cb009c395002
use the observer helper object of the <browser> which is now a CE. r=jorgk DONTBUILD

Comment 12

[Jorg K (CEST = GMT+2)]

I ran a few random tests and they all looked OK. Let's take the risk an retrigger the Daily.

Comment 13

[Magnus Melin [:mkmelin]]

Chat seems to be busted though
JavaScript error: chrome://messenger/content/chat/imconversation.xml, line 1638: TypeError: this.browser.init is not a function

Comment 14

[Jorg K (CEST = GMT+2)]

Please keep an eye on the tree for other failures. I'm mostly afk for the rest of the day.

Comment 15

[Magnus Melin [:mkmelin]]

Filed the im change as bug 1519091.

Comment 16

[Jorg K (CEST = GMT+2)]

Magnus, there are still a bunch of Calendar test failures left. Do you want to fix them here or in another bug?

I've run
mozmake -C comm/calendar/test/mozmill SOLO_TEST=testTodayPane.js mozmill-one

I see

JavaScript error: chrome://calendar/content/calendar-multiday-view.xml, line 2532: TypeError: document.getAnonymousElementByAttribute(...) is null

And that code is:

      let scrollbox = document.getAnonymousElementByAttribute(
                     this, "anonid", "scrollbox");
      document.getAnonymousElementByAttribute(
          scrollbox, "class", "box-inherit scrollbox-innerbox").flex = "1";

That seems to come from bug 1454357.

Comment 17

[Magnus Melin [:mkmelin]]

For calendar, bug 1519114

Comment 18

[Matthew Gaudet (he/him) [:mgaudet]]

Attachment: Bug 1519078 - Don't mutate inputs in CacheIR operations r?jandem

Comment 19

[Jorg K (CEST = GMT+2)]

Are you in the wrong bug here?

Comment 20

[Matthew Gaudet (he/him) [:mgaudet]]

Sure typo'd that one. Thanks Jorg!