Closed Bug 1495002 Opened 5 years ago Closed 2 years ago

Permafailure ccov tier2 Wd3 TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/[+/- folder]/test_name| expected OK

Categories

(Remote Protocol :: Marionette, defect, P5)

Product:
Remote Protocol
Component:
Marionette
Version:
Version 3
Platform:
All
Windows
Type:
defect

Tracking

(firefox66 disabled, firefox70 disabled, firefox72 disabled, firefox73 disabled, firefox74 disabled, firefox94 disabled, firefox95 disabled, firefox96 fixed)

Status:
RESOLVED FIXED
Milestone:
96 Branch
Tracking Flags:
Tracking Status
firefox66 --- disabled
firefox70 --- disabled
firefox72 --- disabled
firefox73 --- disabled
firefox74 --- disabled
firefox94 --- disabled
firefox95 --- disabled
firefox96 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: whimboo)

References

Details

(Keywords: intermittent-failure)

Attachments

(9 files)

Disable timeouts.py on win ccov for frequent failures
591 bytes, patch
whimboo
: review+
Details | Diff | Splinter Review
Bug 1495002 - disable permissions/set.py on win ccov debug for frequent failures r=whimboo
47 bytes, text/x-phabricator-request
Details | Review
Bug 1495002 - disable new_session/create_alwaysMatch.py and new_session/timeouts.py on linux ccov debug for frequent failures r=whimboo
47 bytes, text/x-phabricator-request
Details | Review
Bug 1495002 - Don't disable create_alwaysMatch.py on Linux ccov debug, since this build doesn't exist anymore. r=whimboo
47 bytes, text/x-phabricator-request
Details | Review
Bug 1495002 - Don't disable timeouts.py on Linux ccov debug, since this build doesn't exist anymore. r=whimboo
47 bytes, text/x-phabricator-request
Details | Review
Bug 1495002 - Disable webdriver/tests/permissions/set.py on Windows coverage build (either opt or debug). r=whimboo
47 bytes, text/x-phabricator-request
Details | Review
Bug 1495002 - [wdspec] Disable minimize_window/user_prompts.py for win ccov builds.
47 bytes, text/x-phabricator-request
Details | Review
Bug 1495002 - [wdspec] Re-enable all tests for ccov except permissions/set.py.ini which still perma-fails.
47 bytes, text/x-phabricator-request
Details | Review
Bug 1495002 - disable create_alwaysMatch.py and timeouts.py in win ccov r=whimboo
47 bytes, text/x-phabricator-request
Details | Review 
Filed by: aciure [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=202218864&repo=autoland

https://queue.taskcluster.net/v1/task/fmsgI2FIRq-tuJQrX40qeg/runs/0/artifacts/public/logs/live_backing.log

[task 2018-09-28T13:32:41.958Z] 13:32:41     INFO - PID 26971 | [Parent 6361, Gecko_IOThread] WARNING: pipe error (92): Connection reset by peer: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 356
[task 2018-09-28T13:32:41.965Z] 13:32:41     INFO - PID 26971 | [Parent 6361, Gecko_IOThread] WARNING: pipe error (91): Connection reset by peer: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 356
[task 2018-09-28T13:32:42.062Z] 13:32:42     INFO - PID 26971 | 1538141562052	Marionette	DEBUG	Received observer notification xpcom-will-shutdown
[task 2018-09-28T13:32:42.063Z] 13:32:42     INFO - PID 26971 | 1538141562052	Marionette	DEBUG	Resetting recommended pref security.turn_off_all_security_so_that_viruses_can_take_over_this_computer
[task 2018-09-28T13:32:42.064Z] 13:32:42     INFO - PID 26971 | 1538141562052	Marionette	DEBUG	Resetting recommended pref apz.content_response_timeout
[task 2018-09-28T13:32:42.065Z] 13:32:42     INFO - PID 26971 | 1538141562053	Marionette	DEBUG	Resetting recommended pref browser.download.panel.shown
[task 2018-09-28T13:32:42.065Z] 13:32:42     INFO - PID 26971 | 1538141562053	Marionette	DEBUG	Resetting recommended pref browser.pagethumbnails.capturing_disabled
[task 2018-09-28T13:32:42.066Z] 13:32:42     INFO - PID 26971 | 1538141562053	Marionette	DEBUG	Resetting recommended pref browser.search.update
[task 2018-09-28T13:32:42.068Z] 13:32:42     INFO - PID 26971 | 1538141562053	Marionette	DEBUG	Resetting recommended pref toolkit.cosmeticAnimations.enabled
[task 2018-09-28T13:32:42.069Z] 13:32:42     INFO - PID 26971 | 1538141562053	Marionette	DEBUG	Resetting recommended pref browser.tabs.disableBackgroundZombification
[task 2018-09-28T13:32:42.071Z] 13:32:42     INFO - PID 26971 | 1538141562053	Marionette	DEBUG	Resetting recommended pref browser.tabs.warnOnCloseOtherTabs
[task 2018-09-28T13:32:42.075Z] 13:32:42     INFO - PID 26971 | 1538141562054	Marionette	DEBUG	Resetting recommended pref browser.tabs.warnOnOpen
[task 2018-09-28T13:32:42.076Z] 13:32:42     INFO - PID 26971 | 1538141562054	Marionette	DEBUG	Resetting recommended pref browser.usedOnWindows10.introURL
[task 2018-09-28T13:32:42.076Z] 13:32:42     INFO - PID 26971 | 1538141562054	Marionette	DEBUG	Resetting recommended pref browser.urlbar.suggest.searches
[task 2018-09-28T13:32:42.077Z] 13:32:42     INFO - PID 26971 | 1538141562054	Marionette	DEBUG	Resetting recommended pref datareporting.policy.dataSubmissionPolicyAccepted
[task 2018-09-28T13:32:42.079Z] 13:32:42     INFO - PID 26971 | 1538141562055	Marionette	DEBUG	Resetting recommended pref dom.disable_beforeunload
[task 2018-09-28T13:32:42.080Z] 13:32:42     INFO - PID 26971 | 1538141562055	Marionette	DEBUG	Resetting recommended pref dom.disable_open_during_load
[task 2018-09-28T13:32:42.081Z] 13:32:42     INFO - PID 26971 | 1538141562055	Marionette	DEBUG	Resetting recommended pref dom.file.createInChild
[task 2018-09-28T13:32:42.082Z] 13:32:42     INFO - PID 26971 | 1538141562055	Marionette	DEBUG	Resetting recommended pref dom.max_chrome_script_run_time
[task 2018-09-28T13:32:42.088Z] 13:32:42     INFO - PID 26971 | 1538141562055	Marionette	DEBUG	Resetting recommended pref dom.max_script_run_time
[task 2018-09-28T13:32:42.091Z] 13:32:42     INFO - PID 26971 | 1538141562055	Marionette	DEBUG	Resetting recommended pref extensions.getAddons.cache.enabled
[task 2018-09-28T13:32:42.092Z] 13:32:42     INFO - PID 26971 | 1538141562055	Marionette	DEBUG	Resetting recommended pref extensions.webservice.discoverURL
[task 2018-09-28T13:32:42.094Z] 13:32:42     INFO - PID 26971 | 1538141562055	Marionette	DEBUG	Resetting recommended pref network.http.prompt-temp-redirect
[task 2018-09-28T13:32:42.096Z] 13:32:42     INFO - PID 26971 | 1538141562056	Marionette	DEBUG	Resetting recommended pref network.http.speculative-parallel-limit
[task 2018-09-28T13:32:42.097Z] 13:32:42     INFO - PID 26971 | 1538141562056	Marionette	DEBUG	Resetting recommended pref security.fileuri.strict_origin_policy
[task 2018-09-28T13:32:42.097Z] 13:32:42     INFO - PID 26971 | 1538141562056	Marionette	DEBUG	Resetting recommended pref security.notification_enable_delay
[task 2018-09-28T13:32:42.098Z] 13:32:42     INFO - PID 26971 | 1538141562056	Marionette	DEBUG	Resetting recommended pref signon.autofillForms
[task 2018-09-28T13:32:42.099Z] 13:32:42     INFO - PID 26971 | 1538141562056	Marionette	DEBUG	Resetting recommended pref signon.rememberSignons
[task 2018-09-28T13:32:42.100Z] 13:32:42     INFO - PID 26971 | 1538141562056	Marionette	DEBUG	Remote service is inactive
[task 2018-09-28T13:33:11.712Z] 13:33:11     INFO - TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK
[task 2018-09-28T13:33:11.712Z] 13:33:11     INFO - TEST-INFO took 30002ms
[task 2018-09-28T13:33:11.776Z] 13:33:11  WARNING - u'runner_teardown': ()
[task 2018-09-28T13:33:11.777Z] 13:33:11     INFO - Starting runner
[task 2018-09-28T13:33:11.806Z] 13:33:11     INFO - PID 6578 | 1538141591800	webdriver::httpapi	DEBUG	Creating routes
[task 2018-09-28T13:33:11.807Z] 13:33:11     INFO - WebDriver HTTP server listening at http://127.0.0.1:4445/
Hang during shutdown of Firefox.
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → INCOMPLETE

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Status: REOPENED → RESOLVED
Closed: 5 years ago → 5 years ago
Resolution: --- → INCOMPLETE

New occurrences: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=220710845&repo=mozilla-central&lineNumber=73713

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

This started to perma fail with this merge: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=windows%2C10%2Cx64%2Cccov%2Cdebug%2Cweb%2Cplatform%2Ctests%2Cwith%2Ce10s%2Ctest-windows10-64-ccov%2Fdebug-web-platform-tests-wdspec-e10s-4%2Cw-e10s%28wd4%29&revision=1f7a8905fa7cf88143161cb9c65ca69cb1daec65&selectedJob=220646216

Summary: Intermittent TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK → Perma [tier2] TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK

Looks like this is like a rat tail. Once we skip a wdspec test for ccov the next in the row starts to fail. This has been most likely caused by bug 1507121.

Maybe Christina can have a look again, and we might have to skip them all?

Flags: needinfo?(ccoroiu)

Created a patch to disable timeouts.py

Assignee: nobody → ccoroiu
Flags: needinfo?(ccoroiu)
Attachment #9035765 - Flags: review?(hskupin)

Please trigger a try build for ccov tests to verify that we don't trigger another follow-up failure with disabling this test. As best retrigger the Wd jobs multiple times.

Flags: needinfo?(ccoroiu)
Attachment #9035765 - Flags: review?(hskupin) → review+
Keywords: checkin-needed, leave-open
Whiteboard: [stockwell disabled]

Pushed by dvarga@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/39346b9630de
disable timeouts.py on win ccov for frequent failures. r=whimboo

Keywords: checkin-needed
Assignee: ccoroiu → nobody
Status: REOPENED → NEW
status-firefox66: --- → disabled

The test got accidentally unskipped for ccov builds, and has been marked as skipped again via https://hg.mozilla.org/mozilla-central/rev/ab359d0369df.

Tier 1 occurrence: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=260691369&repo=mozilla-central&lineNumber=101153

Summary: Perma [tier2] TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK → Perma TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK
Summary: Perma TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK → Intermittent TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK

Why the change from perma to intermittent? This test might still perma fail if it would be enabled.

Flags: needinfo?(ccoroiu)
Summary: Intermittent TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK → Permafailing TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK

Seeing as this bug was intermittent when it was first created https://bugzilla.mozilla.org/show_bug.cgi?id=1495002#c9, and then the summary was changed to reflect that it became a permafail, I'd say it isn't worth it to create a new bug for the intermittent variant.

Also disable has been done 6 months ago, so my guess would be that it's only going to be re-enabled after fixing the issue.

Flags: needinfo?(ccoroiu)

(In reply to Alexandru Michis [:malexandru] from comment #25)

Seeing as this bug was intermittent when it was first created https://bugzilla.mozilla.org/show_bug.cgi?id=1495002#c9, and then the summary was changed to reflect that it became a permafail, I'd say it isn't worth it to create a new bug for the intermittent variant.

Also disable has been done 6 months ago, so my guess would be that it's only going to be re-enabled after fixing the issue.

Please note that this is a new regression and started happening for this test on Aug 2nd, and that only on the Linux platform. Everyone filed those failures against this bug without noticing that the platform is totally different. A check for the disable patch would have been enough. So please file a new bug because it is totally different, and would need to be investigated.

status-firefox70: --- → disabled
Flags: needinfo?(malexandru)
Keywords: test-disabled
OS: Unspecified → Windows
Hardware: Unspecified → All
Summary: Permafailing TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK → Permafailure ccov win TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK

Understood, I've made a new bug for the failures unrelated to win ccov: https://bugzilla.mozilla.org/show_bug.cgi?id=1574837 and reclassified these failures against it.

Flags: needinfo?(malexandru)
See Also: → 1574837
Depends on: 1574837
No longer depends on: 1574837
Summary: Permafailure ccov win TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK → Permafailure ccov win tier2 Wd3 TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/[+/- folder]/test_name| expected OK
Attachment #9103539 - Attachment description: Bug 1495002 - disable set.py on win ccov debug for frequent failures r=whimboo → Bug 1495002 - disable permissions/set.py on win ccov debug for frequent failures r=whimboo
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3538810d52da
disable permissions/set.py on win ccov debug for frequent failures r=whimboo
Summary: Permafailure ccov win tier2 Wd3 TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/[+/- folder]/test_name| expected OK → Permafailure ccov tier2 Wd3 TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/[+/- folder]/test_name| expected OK

Henrik, I closed the above bugs as duplicates based on https://bugzilla.mozilla.org/show_bug.cgi?id=1526678#c27

Carrying over dependency on bug 1577713.

Depends on: 1577713
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5aa41cf3904b
disable new_session/create_alwaysMatch.py and new_session/timeouts.py on linux ccov debug for frequent failures r=whimboo
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2e0027ec4127
Don't disable create_alwaysMatch.py on Linux ccov debug, since this build doesn't exist anymore. r=whimboo
https://hg.mozilla.org/integration/autoland/rev/0c2f333260eb
Don't disable timeouts.py on Linux ccov debug, since this build doesn't exist anymore. r=whimboo

It hasn't been happening for a while since we switched to an opt build.

Assignee: mcastelluccio → nobody
Status: ASSIGNED → NEW
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Assignee: mcastelluccio → nobody
Status: ASSIGNED → NEW
Flags: needinfo?(mcastelluccio)
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Assignee: mcastelluccio → nobody
Status: ASSIGNED → NEW
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3f799b7f6517
Disable webdriver/tests/permissions/set.py on Windows coverage build (either opt or debug). r=whimboo
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Assignee: mcastelluccio → nobody
Status: ASSIGNED → NEW
status-firefox72: --- → disabled
status-firefox73: --- → disabled
status-firefox74: --- → disabled
See Also: → 1608082
See Also: → 1590740

webdriver/tests/minimize_window/user_prompts.py is also failing permanently on Windows and as such need to be disabled.

Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Assignee: hskupin → nobody
Status: ASSIGNED → NEW
Depends on: 1610260

Marco, should we be ok now to unskip all of those tests now?

Flags: needinfo?(mcastelluccio)

Yeah, we could try and if they still fail so often we'll disable them again.

Flags: needinfo?(mcastelluccio)

Hi Henrik, want us to unskip the tests or will you do it?

Flags: needinfo?(hskupin)

Andreea, if one of you has the time to do that I would kinda appreciate. Thanks!

Flags: needinfo?(hskupin) → needinfo?(apavel)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #76)

Andreea, if one of you has the time to do that I would kinda appreciate. Thanks!

Sure, we'll do it. Thank you.

Flags: needinfo?(apavel)
Assignee: nobody → apavel
Status: NEW → ASSIGNED
Attachment #9124462 - Attachment description: Bug 1495002 - re-enable tests r=whimboo → Bug 1495002 - re-enable create_alwaysMatch.py and timeouts.py on win ccov r=whimboo
Attachment #9124462 - Attachment description: Bug 1495002 - re-enable create_alwaysMatch.py and timeouts.py on win ccov r=whimboo → Bug 1495002 - re-enable create_alwaysMatch.py, timeouts.py, close.py, delete.py, get_window_handle/user_prompts.py,minimize_window/user_prompts.py.ini, response.py.ini on win ccov r=whimboo
Attachment #9124462 - Attachment description: Bug 1495002 - re-enable create_alwaysMatch.py, timeouts.py, close.py, delete.py, get_window_handle/user_prompts.py,minimize_window/user_prompts.py.ini, response.py.ini on win ccov r=whimboo → Bug 1495002 - [wdspec] Re-enable all tests for ccov except permissions/set.py.ini which still perma-fails.
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6436566bc9a1
[wdspec] Re-enable all tests for ccov except permissions/set.py.ini which still perma-fails. r=whimboo

We still have perma-failures for alwaysMatch and timeout on Windows 10:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&tier=1%2C2%2C3&searchStr=wd%2Cccov

I will have a look why we have such hangs during startup now, so that we can hopefully fix this problem (bug 1382162). As such I will push some try builds for investigating the issue.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8bf3deae3915870cee59704cfdd2db070467389f

Andreea, if you want to skip them again feel free to create a patch.

Flags: needinfo?(apavel)

The job failed again:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288672033&repo=try

[task 2020-02-13T09:54:53.196Z] 09:54:53     INFO - TEST-START | /webdriver/tests/new_session/timeouts.py
[..]
[task 2020-02-13T09:55:48.203Z] 09:55:48     INFO - TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/timeouts.py | expected OK

This test doesn't use the meta tag for a longer timeout, so it should be added given that we start multiple instances of Firefox.

[task 2020-02-13T09:44:36.974Z] 09:44:36     INFO - TEST-START | /webdriver/tests/new_session/create_alwaysMatch.py
[..]
[task 2020-02-13T09:50:41.985Z] 09:50:41     INFO - TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/create_alwaysMatch.py | expected OK

This test runs about 6 minutes because it starts/stops even more Firefox instances. Seeing that it takes about 11s for a safe shutdown of Firefox, and 8s for startup we most likely should split this test into multiple files.

Marco, I assume those startup/shutdown times are normal?

Flags: needinfo?(mcastelluccio)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] (away 02/17 - 02/21) from comment #83)

We still have perma-failures for alwaysMatch and timeout on Windows 10:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&tier=1%2C2%2C3&searchStr=wd%2Cccov

I will have a look why we have such hangs during startup now, so that we can hopefully fix this problem (bug 1382162). As such I will push some try builds for investigating the issue.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8bf3deae3915870cee59704cfdd2db070467389f

Andreea, if you want to skip them again feel free to create a patch.

Sure, will do it in a couple of minutes.

Flags: needinfo?(apavel)

Yep, they sound reasonable given that it is a coverage build (I would have expected even slower than that!).

Flags: needinfo?(mcastelluccio)
Pushed by apavel@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/df278a39cb41
disable create_alwaysMatch.py and timeouts.py in win ccov r=whimboo

Recent failures here are bug 1507121.

Same for the last Intermittent Failures Robot comment, those failures are not related to this bug.

Marco,
I've reclassified each failure against it's specific Bug.

We should no use this to classify failures anymore, since ccov builds don't run anymore.

Just clarifying here. We are still running Wd jobs for ccov builds:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=ccov%2Cwd

But a lot of them are still disabled.

Blocks: 1608086
Assignee: apavel → nobody
Status: ASSIGNED → NEW
Depends on: 1418778

Bug 1418778 re-enabled all the tests.

Assignee: nobody → hskupin
Status: NEW → RESOLVED
Closed: 5 years ago → 2 years ago
status-firefox94: --- → disabled
status-firefox95: --- → disabled
status-firefox96: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 96 Branch
Whiteboard: [stockwell disabled]
Moving bug to Remote Protocol::Marionette component per bug 1815831.
Component: geckodriver → Marionette
Product: Testing → Remote Protocol
You need to log in before you can comment on or make changes to this bug.