1697196 - Frequent toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js | Test timed out -

Status: RESOLVED FIXED

(WebExtensions :: General, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=332541968&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Qx2fAxvHS9qOsw3wuFrhtA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-03-09T15:17:56.173Z] 15:17:56     INFO - TEST-PASS | toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js | About:welcome page background should be reset. - "rgb(237, 237, 240)" == "rgb(237, 237, 240)" - 
[task 2021-03-09T15:17:56.173Z] 15:17:56     INFO - Console message: [JavaScript Error: "TypeError: Window.getComputedStyle: Argument 1 is not an object." {file: "chrome://mochitests/content/browser/toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js" line: 86}]
[task 2021-03-09T15:17:56.173Z] 15:17:56     INFO - Console message: [JavaScript Error: "Failed to get the latest add-on version for Return to AMO" {file: "resource:///actors/AboutWelcomeChild.jsm" line: 239}]
[task 2021-03-09T15:17:56.173Z] 15:17:56     INFO - getAddonInfo@resource:///actors/AboutWelcomeChild.jsm:239:10
[task 2021-03-09T15:17:56.173Z] 15:17:56     INFO - 
[task 2021-03-09T15:17:56.174Z] 15:17:56     INFO - Buffered messages finished
[task 2021-03-09T15:17:56.174Z] 15:17:56     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js | Test timed out - 
[task 2021-03-09T15:17:56.174Z] 15:17:56     INFO - Not taking screenshot here: see the one that was previously logged
[task 2021-03-09T15:17:56.174Z] 15:17:56     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js | Extension left running at test shutdown - 
[task 2021-03-09T15:17:56.174Z] 15:17:56     INFO - Stack trace:
[task 2021-03-09T15:17:56.174Z] 15:17:56     INFO - chrome://mochikit/content/browser-test.js:test_ok:1331
[task 2021-03-09T15:17:56.175Z] 15:17:56     INFO - chrome://mochikit/content/tests/SimpleTest/ExtensionTestUtils.js:ExtensionTestUtils.loadExtension/<:117
[task 2021-03-09T15:17:56.175Z] 15:17:56     INFO - chrome://mochikit/content/browser-test.js:nextTest:571
[task 2021-03-09T15:17:56.175Z] 15:17:56     INFO - GECKO(2050) | [Child 2051, Main Thread] WARNING: '!CanSend() || !mManager || !mManager->CanSend()', file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorChild.cpp:40
[task 2021-03-09T15:17:56.175Z] 15:17:56     INFO - Not taking screenshot here: see the one that was previously logged```

Comment 1

[Noemi Erli[:noemi_erli]]

Tim, could this be a regression from https://hg.mozilla.org/integration/autoland/rev/f2cbc95f6826c0a92c2fca423eef096d08881b03?

Comment 2

[Tim Nguyen :ntim]

I'm not sure, but this is just re-using the approach from https://bugzilla.mozilla.org/show_bug.cgi?id=1695345. Also, this is proton only.

I would say this is more likely to be related to a recent about:newtab / Activity Stream change.

Maybe mixedpuppy or ewright (for new tab) knows more?

Comment 3

[Tim Nguyen :ntim]

This is the bit that seems to be failing (according to the log above): https://searchfox.org/mozilla-central/rev/62ddcc03195036b355cb23b6f2e2b88d3f8099ad/toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js#85-86

So I suspect about:newtab isn't properly initialized in that test?

Comment 4

[Erica Wright [:ewright]]

Hmmm, I'm not sure here, I would try asking @sdowne if we think it is a new tab initialization failure.

Comment 5

[Alexandru Michis [:malexandru]]

prathiksha, could you please take a look at these failures?

Comment 6

[Alexandru Michis [:malexandru]]

Sorry for the ni, meant to write on another bug!
Also, I did some backfills for this and it doesn't seem to be caused by Bug 1697062.

Comment 7

[Alexandru Michis [:malexandru]]

Continued backfills until the 5th of March.

Comment 8

[Intermittent Failures Robot]

17 failures in 908 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 1

Platform and build breakdown:

• macosx1014-64-qr: 16
  • debug: 16
• windows10-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-03-09&endday=2021-03-09&tree=all

Comment 9

[Shane Caraveo (:mixedpuppy)]

This could be hard to chase down. qr + debug intermittent. I suspect some timing issue between loading the tab using BrowserTestUtils.withNewTab and the test relying on the page load. I'm unable to force the failure. Will wait to see what other responses happen.

Comment 10

[Intermittent Failures Robot]

46 failures in 4622 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #25 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 36
• mozilla-central: 5
• try: 5

Platform and build breakdown:

• macosx1014-64-qr: 24
  • debug: 24
• windows10-64-qr: 11
  • debug: 11
• windows7-32: 7
  • debug: 7
• windows10-64: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-03-08&endday=2021-03-14&tree=all

Comment 11

[Intermittent Failures Robot]

26 failures in 761 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 5

Platform and build breakdown:

• windows10-64-qr: 12
  • debug: 12
• windows7-32: 7
  • debug: 7
• windows10-64: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-03-15&endday=2021-03-15&tree=all

Comment 12

[Alexandru Michis [:malexandru]]

There have been 69 total failures in the last 7 days, recent failure log.
Affected platforms are:

• macosx1014-64-qr debug
• windows10-64 debug / windows10-64-qr debug / windows7-32 debug

Failure rate looks to have come down considerably since the start of the week.

Comment 13

[Intermittent Failures Robot]

53 failures in 4257 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 38
• mozilla-central: 13
• try: 2

Platform and build breakdown:

• windows10-64-qr: 27
  • debug: 27
• windows7-32: 9
  • debug: 9
• windows10-64: 15
  • debug: 15
• macosx1014-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 14

[Florian Quèze [:florian]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #9)

This could be hard to chase down. qr + debug intermittent. I suspect some timing issue between loading the tab using BrowserTestUtils.withNewTab and the test relying on the page load. I'm unable to force the failure.

In case this helps, BrowserTestUtils.openNewForegroundTab awaits BrowserTestUtils.switchTab which waits for a 300ms timer at https://searchfox.org/mozilla-central/rev/4e87b5392eafe1f1d49017e76f7317b06ec0b1d8/browser/modules/AsyncTabSwitcher.jsm#88

It's very likely that the test depends implicitly on something being completed during these 300ms, which might not be enough on slow runs (eg. debug) of the browser.

If I replace the 300ms timer there with a 10ms, when I run ./mach test toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors.js --headless --run-until-failure --profiler half of the time I can reproduce a failure with this message:

toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors.js
  FAIL Uncaught exception - at chrome://mochitests/content/browser/toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors.js:36 - TypeError: Window.getComputedStyle: Argument 1 is not an object.
Stack trace:
@chrome://mochitests/content/browser/toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors.js:36:32
execute@resource://specialpowers/SpecialPowersSandbox.jsm:141:12
_spawnTask@resource://specialpowers/SpecialPowersChild.jsm:1611:15
receiveMessage@resource://specialpowers/SpecialPowersChild.jsm:310:21
  FAIL Found an unexpected tab at the end of test run: about:newtab -

The test log also contains: JavaScript error: resource:///actors/ContentSearchParent.jsm, line 624: TypeError: can't access property "addEventListener", browser is null

Comment 15

[Intermittent Failures Robot]

4 failures in 3967 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 4

Platform and build breakdown:

• windows10-64-qr: 2
  • debug: 2
• macosx1014-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-03-22&endday=2021-03-28&tree=all

Comment 16

[Shane Caraveo (:mixedpuppy)]

This was very intermittent, less so in the last report, but we should try and look at it soon.

Comment 17

[Intermittent Failures Robot]

50 failures in 511 pushes (0.098 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 50

Platform and build breakdown:

• linux1804-64: 19
  • debug: 19
• linux1804-64-tsan: 13
  • opt: 13
• windows10-64: 2
  • asan: 2
• linux1804-64-asan: 10
  • opt: 10
• windows10-64-qr: 3
  • debug: 3
• windows10-32: 2
  • debug: 2
• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-05-28&endday=2021-05-28&tree=all

Comment 18

[Intermittent Failures Robot]

51 failures in 103 pushes (0.495 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 32
• mozilla-central: 17
• try: 2

Platform and build breakdown:

• linux1804-64-tsan: 7
  • opt: 7
• macosx1015-64-qr: 7
  • debug: 7
• linux1804-64: 16
  • debug: 16
• windows10-64-qr: 7
  • debug: 7
• windows10-32: 4
  • debug: 4
• linux1804-64-ccov: 1
  • opt: 1
• windows10-64: 4
  • debug: 2
  • asan: 2
• linux1804-64-asan: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-05-29&endday=2021-05-29&tree=all

Comment 19

[Natalia Csoregi [:nataliaCs]]

Update:

There have been 148 failures within the last 7 days as it follows:

• 18 failures on Windows 10 x64 WebRender debug
• 11 failures on Windows 10 x64 asan/debug
• 9 failures on windows10-32 debug
• 12 failures on OS X 10.15 WebRender debug
• 28 failures on Linux 18.04 x64 tsan opt
• 2 failures on Linux 18.04 x64 CCov opt
• 18 failures on Linux 18.04 x64 asan opt
• 50 failures on Linux 18.04 x64 debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=341232055&repo=autoland&lineNumber=29747

[task 2021-05-30T23:35:39.112Z] 23:35:39 INFO - TEST-PASS | toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js | New tab page background should be set. - "rgb(173, 216, 230)" == "rgb(173, 216, 230)" -
[task 2021-05-30T23:35:39.113Z] 23:35:39 INFO - Console message: [JavaScript Error: "TypeError: Window.getComputedStyle: Argument 1 is not an object." {file: "chrome://mochitests/content/browser/toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js" line: 44}]
[task 2021-05-30T23:35:39.114Z] 23:35:39 INFO - Buffered messages finished
[task 2021-05-30T23:35:39.115Z] 23:35:39 INFO - TEST-UNEXPECTED-FAIL | toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js | Test timed out -
[task 2021-05-30T23:35:39.115Z] 23:35:39 INFO - Not taking screenshot here: see the one that was previously logged
[task 2021-05-30T23:35:39.117Z] 23:35:39 INFO - TEST-UNEXPECTED-FAIL | toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js | Extension left running at test shutdown -
[task 2021-05-30T23:35:39.117Z] 23:35:39 INFO - Stack trace:
[task 2021-05-30T23:35:39.117Z] 23:35:39 INFO - chrome://mochikit/content/browser-test.js:test_ok:1334
[task 2021-05-30T23:35:39.118Z] 23:35:39 INFO - chrome://mochikit/content/tests/SimpleTest/ExtensionTestUtils.js:ExtensionTestUtils.loadExtension/<:117
[task 2021-05-30T23:35:39.118Z] 23:35:39 INFO - chrome://mochikit/content/browser-test.js:nextTest:564
[task 2021-05-30T23:35:39.119Z] 23:35:39 INFO - GECKO(2172) | [Child 2256, Main Thread] WARNING: '!CanSend() || !mManager || !mManager->CanSend()', file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorChild.cpp:40
[task 2021-05-30T23:35:39.120Z] 23:35:39 INFO - Not taking screenshot here: see the one that was previously logged

Comment 20

[Intermittent Failures Robot]

156 failures in 2908 pushes (0.054 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 124
• mozilla-central: 29
• try: 3

Platform and build breakdown:

• linux1804-64: 53
  • debug: 53
• linux1804-64-tsan: 29
  • opt: 29
• windows10-64: 11
  • asan: 7
  • debug: 4
• linux1804-64-asan: 19
  • opt: 19
• windows10-64-qr: 20
  • debug: 20
• windows10-32: 9
  • debug: 9
• macosx1015-64-qr: 13
  • debug: 13
• linux1804-64-ccov: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-05-24&endday=2021-05-30&tree=all

Comment 21

[Intermittent Failures Robot]

135 failures in 517 pushes (0.261 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 111
• mozilla-central: 16
• mozilla-beta: 7
• try: 1

Platform and build breakdown:

• linux1804-64: 44
  • debug: 42
  • opt: 2
• linux1804-64-tsan: 29
  • opt: 29
• windows10-64: 15
  • asan: 6
  • debug: 9
• windows10-32: 7
  • debug: 7
• windows10-64-qr: 12
  • debug: 12
• macosx1015-64-qr: 9
  • debug: 9
• linux1804-64-asan: 19
  • opt: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-05-31&endday=2021-05-31&tree=all

Comment 22

[Narcis Beleuzu [:NarcisB]]

Attachment: Bug 1697196 - Disable browser_ext_themes_ntp_colors_perwindow.js on Linux64 and Win10-64 for frequent failures. r=#intermittent-reviewers

Comment 23

[Narcis Beleuzu [:NarcisB]]

Update:

There have been 311 failures within the last 7 days as it follows:

- 205 failures on linux1804
- 65 failures on windows10-64
- 23 failures on macosx1015
- 18 failures on windows10-32

I added a patch to disable this test on Linux1804-64 and Win10-64 until there will be a fix

Comment 24

[Rob Wu [:robwu]]

The failures associated with this bug should have been attributed to bug 1697430.

The common error in these logs is TypeError: Window.getComputedStyle: Argument 1 is not an object.

That happens when the outer-wrapper element hasn't been initialized at the time of the invocation, at https://searchfox.org/mozilla-central/rev/e9eb869e90a8d717678c3f38bf75843e345729ab/toolkit/components/extensions/test/browser/browser_ext_themes_ntp_colors_perwindow.js#85

While it's possible to fix the failures, I've noticed that the tests aren't testing anything realistic any more. E.g. in https://hg.mozilla.org/mozilla-central/rev/d36693bbaf559ed4d3e67d1951aed6b58e564a9f some CSS properties were set to get the test to pass, but those properties do not match what I can see with my eyes.

The tests are supposed to verify that the new tab page's background/font color can be changed via extension themes.
When I open about:welcome on Nightly, I don't see any color change. I suppose that it's fine since it's an onboarding page.
But about:newtab also offers limited ways to customize the color; the background color is changed, but the font color is not (because of changes from bug 1686384) - filed as bug 1713778.

Comment 25

[Rob Wu [:robwu]]

Attachment: Bug 1697196 - Wait for newtab to have been initialized before proceeding

Fix based on the observed error, see https://bugzilla.mozilla.org/show_bug.cgi?id=1697196#c24

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8fc65986c64e
Wait for newtab to have been initialized before proceeding r=rpl

Comment 27

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/8fc65986c64e

Comment 28

[Intermittent Failures Robot]

91 failures in 545 pushes (0.167 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 58
• mozilla-central: 14
• try: 15
• mozilla-beta: 4

Platform and build breakdown:

• linux1804-64-tsan: 16
  • opt: 16
• linux1804-64-asan: 14
  • opt: 14
• linux1804-64: 37
  • debug: 37
• windows10-64: 8
  • debug: 5
  • asan: 2
  • opt: 1
• windows10-32: 4
  • debug: 4
• windows10-64-qr: 9
  • debug: 7
  • opt: 2
• macosx1015-64-qr: 2
  • debug: 2
• linux1804-64-ccov: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-06-01&endday=2021-06-01&tree=all

Comment 29

[Intermittent Failures Robot]

226 failures in 3065 pushes (0.074 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 169
• mozilla-central: 30
• mozilla-beta: 11
• try: 16

Platform and build breakdown:

• linux1804-64: 81
  • debug: 79
  • opt: 2
• linux1804-64-tsan: 45
  • opt: 45
• windows10-64: 23
  • asan: 8
  • debug: 14
  • opt: 1
• windows10-32: 11
  • debug: 11
• windows10-64-qr: 21
  • debug: 19
  • opt: 2
• macosx1015-64-qr: 11
  • debug: 11
• linux1804-64-asan: 33
  • opt: 33
• linux1804-64-ccov: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1697196&startday=2021-05-31&endday=2021-06-06&tree=all

Comment 30

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/f77433509861