1673786 - Intermittent testing/firefox-ui/tests/functional/safebrowsing/test_notification.py TestSafeBrowsingNotificationBar.test_notification_bar | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect, P5)

Description

[Natalia Csoregi [:nataliaCs]]

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=319962774&repo=autoland&lineNumber=1681

[task 2020-10-28T02:43:51.413Z] 02:43:51 INFO - TEST-UNEXPECTED-ERROR | testing/firefox-ui/tests/functional/safebrowsing/test_notification.py TestSafeBrowsingNotificationBar.test_notification_bar | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Connection timed out after 360.0s)
[task 2020-10-28T02:43:51.413Z] 02:43:51 INFO - Traceback (most recent call last):
[task 2020-10-28T02:43:51.414Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_harness/marionette_test/testcases.py", line 214, in run
[task 2020-10-28T02:43:51.414Z] 02:43:51 INFO - testMethod()
[task 2020-10-28T02:43:51.414Z] 02:43:51 INFO - File "/builds/worker/workspace/build/tests/firefox-ui/tests/testing/firefox-ui/tests/functional/safebrowsing/test_notification.py", line 60, in test_notification_bar
[task 2020-10-28T02:43:51.414Z] 02:43:51 INFO - self.check_get_me_out_of_here_button()
[task 2020-10-28T02:43:51.414Z] 02:43:51 INFO - File "/builds/worker/workspace/build/tests/firefox-ui/tests/testing/firefox-ui/tests/functional/safebrowsing/test_notification.py", line 111, in check_get_me_out_of_here_button
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - message="The default home page has not been loaded",
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/wait.py", line 134, in until
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - rv = condition(self.marionette)
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - File "/builds/worker/workspace/build/tests/firefox-ui/tests/testing/firefox-ui/tests/functional/safebrowsing/test_notification.py", line 110, in <lambda>
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - lambda mn: self.default_homepage in mn.get_url(),
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1464, in get_url
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - return self._send_message("WebDriver:GetCurrentURL", key="value")
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 37, in _
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - m._handle_socket_failure()
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 712, in _handle_socket_failure
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - IOError, IOError(message.format(returncode=returncode, reason=exc)), tb
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 27, in _
[task 2020-10-28T02:43:51.415Z] 02:43:51 INFO - return func(*args, **kwargs)
[task 2020-10-28T02:43:51.416Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 617, in _send_message
[task 2020-10-28T02:43:51.416Z] 02:43:51 INFO - msg = self.client.request(name, params)
[task 2020-10-28T02:43:51.416Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/transport.py", line 288, in request
[task 2020-10-28T02:43:51.417Z] 02:43:51 INFO - return self.receive()
[task 2020-10-28T02:43:51.417Z] 02:43:51 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/transport.py", line 192, in receive
[task 2020-10-28T02:43:51.417Z] 02:43:51 INFO - "Connection timed out after {}s".format(self.socket_timeout)
[task 2020-10-28T02:43:51.417Z] 02:43:51 INFO - TEST-INFO took 445423ms

Comment 1

[Intermittent Failures Robot]

41 failures in 863 pushes (0.048 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 32
• mozilla-central: 7
• try: 2

Platform breakdown:

• linux1804-64: 16
• windows10-64: 3
• windows7-32: 7
• linux1804-64-asan: 13
• macosx1014-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1673786&startday=2020-10-28&endday=2020-10-28&tree=all

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It's failing kinda lot over the last days, and it looks to be a regression. Can we please run a back-fill so we can see which changeset was causing it?

To fix it we will need bug 1669169 landed, which will at least fix it for the actor usage.

Comment 3

[Dorel Luca [:dluca]]

Hi Henrik,

Retriggers show that it most likely started here:
https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=Rj3YYfswQHuuzFgc2MzLuw.0&searchStr=linux%2C18.04%2Cx64%2Cdebug%2Cfirefox%2Cfunctional%2Ctests%2C%28remote%29%2Ctest-linux1804-64%2Fdebug-firefox-ui-functional-remote-e10s%2Cen-us&tochange=4b2288d37d5627f6a7c989e16ff98364e10c095d&fromchange=37be148ce68375e17c141fd8d034cc54e1c1754b

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so bug 1665404 made about:home a privileged site. The test that is failing here does the following:

It navigates to https://www.itisatrap.org/firefox/its-a-trap.html, and then clicks the get me out of here button of the navigation bar. That forces the page behind the pref browser.startup.homepage to be loaded. And that is about:home.

There is not that much we can do right now, except waiting for bug 1669169 to be landed. Hopefully this should happen in the next couple of days.

Dorel, how often does it fail? Would it be ok to wait those couple days, or should we mark the test as expected fail for now?

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

~50% of the Linux asan, Linux x64 debug and Windows 7 debug runs fail. Comment 4 sounds like there won't be a fix landed before the weekend. Please disable the tests for now.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1673786 - [firefox-ui] Temporarily skip test_notification.py for debug and asan builds on Linux and Windows.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8050b4336680
[firefox-ui] Temporarily skip test_notification.py for debug and asan builds on Linux and Windows. r=marionette-reviewers,maja_zf

Comment 8

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/8050b4336680

Comment 9

[Intermittent Failures Robot]

115 failures in 4275 pushes (0.027 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 72
• mozilla-central: 34
• try: 9

Platform breakdown:

• linux1804-64: 33
• windows10-64: 7
• windows7-32: 27
• linux1804-64-asan: 20
• macosx1014-64: 7
• windows10-64-mingwclang: 1
• windows7-32-mingwclang: 3
• linux1804-64-ccov: 7
• windows10-64-ccov-qr: 4
• windows7-32-shippable: 3
• linux1804-64-shippable: 1
• macosx1014-64-qr: 1
• macosx1014-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1673786&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 10

[Intermittent Failures Robot]

16 failures in 666 pushes (0.024 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 10
• mozilla-central: 4
• try: 2

Platform breakdown:

• windows7-32: 8
• linux1804-64-ccov: 1
• windows7-32-shippable: 1
• linux1804-64: 3
• macosx1014-64: 2
• windows10-64-ccov-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1673786&startday=2020-11-02&endday=2020-11-02&tree=all

Comment 11

[Intermittent Failures Robot]

15 failures in 747 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 3
• try: 1

Platform breakdown:

• linux1804-64: 2
• windows7-32: 5
• linux1804-64-ccov: 2
• macosx1014-64-qr: 4
• windows7-32-shippable: 1
• linux1804-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1673786&startday=2020-11-04&endday=2020-11-04&tree=all

Comment 12

[Noemi Erli[:noemi_erli]]

As an update, we had 58 occurrences of this failure in the past week. It is now failing on Win7 and Win10 opt, MacOS opt and debug, Linux64 opt.
Should we move forward and disable this on all platforms until Bug 1669169 will be landed?

Comment 13

[Intermittent Failures Robot]

63 failures in 4128 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 34
• mozilla-central: 22
• try: 7

Platform breakdown:

• windows7-32: 21
• linux1804-64-ccov: 9
• windows7-32-shippable: 5
• linux1804-64: 8
• macosx1014-64: 2
• windows10-64-ccov-qr: 6
• macosx1014-64-qr: 7
• linux1804-64-devedition: 2
• linux1804-64-shippable: 2
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1673786&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are most likely going to enable actors by default for all builds tomorrow. So I would say - if it's ok with you - to not specifically disable it.

Comment 15

[Intermittent Failures Robot]

15 failures in 904 pushes (0.017 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• try: 2
• mozilla-central: 1

Platform breakdown:

• macosx1014-64-qr: 2
• windows7-32: 9
• windows10-64: 1
• linux1804-64: 1
• linux1804-64-shippable: 1
• linux1804-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673786&startday=2020-11-10&endday=2020-11-10&tree=all

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given that we no longer have failures since Marionette switched to JSWindowActor by default, I pushed a try to get the skip patch backed out. Lets see what the results are:

https://treeherder.mozilla.org/jobs?repo=try&revision=5ad4f8bd933a204408d5303ae928e73de5343fcb

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The try build is all fine. Sheriffs please backout the following changeset:
https://hg.mozilla.org/mozilla-central/rev/8050b4336680

Comment 18

[Natalia Csoregi [:nataliaCs]]

Backout: https://hg.mozilla.org/integration/autoland/rev/b7242d2a7c50dd452df14863b0ddff2dcf1135b3

Comment 19

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout merged: https://hg.mozilla.org/mozilla-central/rev/b7242d2a7c50

Comment 20

[Intermittent Failures Robot]

29 failures in 4040 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #47 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 19
• mozilla-central: 7
• try: 3

Platform breakdown:

• windows7-32: 14
• linux1804-64-ccov: 3
• macosx1014-64-qr: 3
• windows7-32-shippable: 3
• windows10-64-ccov-qr: 1
• linux1804-64: 2
• linux1804-64-shippable: 2
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673786&startday=2020-11-09&endday=2020-11-15&tree=all