Closed Bug 1418778 Opened 2 years ago Closed 2 months ago

Intermittent TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/create_alwaysMatch.py | expected OK

Categories

(Testing :: geckodriver, defect, P5)

Version 3
defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1495002

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Depends on 4 open bugs, Blocks 1 open bug)

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disabled])

Attachments

(2 files, 1 obsolete file)

Component: web-platform-tests → geckodriver
There seem to be delays during startup of Firefox as caused by bug 1451621.
Depends on: 1451621
Lately those failures are caused by a slow startup of Firefox, which is investigated on bug 1460193.
Depends on: 1460193
Summary: Intermittent TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/sessions/new_session/create_alwaysMatch.py | expected OK → Intermittent TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/create_alwaysMatch.py | expected OK
Duplicate of this bug: 1462815
For QuantumRender builds this is blocked on bug 1464756.
Depends on: 1464756
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → INCOMPLETE
There are 17 occurrences in the last 7 days all on windows10-64-ccov debug

Failure rate seems to be increasing from here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=d3f4d556a89d3e5e72749314a0cf81d942fafc13&selectedJob=211621872
There are 39 occurrences in the last 7 days. 

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=213697083&repo=mozilla-central&lineNumber=72545

00:41:14     INFO - PID 5780 |  => Process ID: 4824, Thread ID: 5808
00:41:14     INFO - PID 5780 | [Parent 8740, Main Thread] WARNING: '!gThread', file z:/build/build/src/xpcom/threads/nsTimerImpl.cpp, line 382
00:41:14     INFO - PID 5780 | [Parent 8740, Main Thread] WARNING: '!gThread', file z:/build/build/src/xpcom/threads/nsTimerImpl.cpp, line 382
00:41:14     INFO - PID 5780 | [Parent 8740, Main Thread] WARNING: '!gThread', file z:/build/build/src/xpcom/threads/nsTimerImpl.cpp, line 382
00:41:14     INFO - TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/new_session/create_alwaysMatch.py | expected OK
00:41:14     INFO - TEST-INFO took 725016ms
00:41:15     INFO - Closing logging queue
00:41:15     INFO - queue closed
00:41:15     INFO - Starting runner
00:41:15     INFO - PID 5060 | 1543106475343	webdriver::httpapi	DEBUG	Creating routes
00:41:15     INFO - PID 5060 | 1543106475350	geckodriver	DEBUG	Listening on 127.0.0.1:4444
00:41:15     INFO - WebDriver HTTP server listening at http://127.0.0.1:4444/

Henrik can you take a look at this? or is there someone who can?
Flags: needinfo?(hskupin)
This is due to ccov builds being incredibly slow. Tempted to get this platform removed until ccov builds get faster
Flags: needinfo?(hskupin)
Thanks for the update David.
There are 33 failures asociated to this bug in the last 7 days. These are occurring on windows10-64-ccov, debug builds.

Any updates on this?
Flags: needinfo?(hskupin)
There have been 38 failures within the last 7 days
- 5 failures on linux x64 qr opt
- 33 failures on windows 10 x64 ccov debug
Marco, could you disable the test for ccov builds?
Flags: needinfo?(hskupin) → needinfo?(mcastelluccio)
Joel, I couldn't find an example of how to skip this, is it this simple: https://irccloud.mozilla.com/file/58CDEthG/image.png? 
I doubt it :D
Flags: needinfo?(jmaher)
Created a new file create_alwaysMatch.py.ini and disabled for windows ccov
Assignee: nobody → ccoroiu
Attachment #9033634 - Flags: review?(jmaher)
Comment on attachment 9033634 [details] [diff] [review]
Bug 1418778 - disable create_alwaysMatch.py on windows ccov for frequent failures

Review of attachment 9033634 [details] [diff] [review]:
-----------------------------------------------------------------

I am not sure if this is right, but it looks ok based on some source file queries.
Attachment #9033634 - Flags: review?(jmaher) → review+
Flags: needinfo?(jmaher)
Whiteboard: [stockwell disabled]
Pushed by apavel@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/dc37a2bc9cda
disable create_alwaysMatch.py on windows ccov for frequent failures. r=jmaher
Keywords: checkin-needed
Backed out for failing wd tests on linux, osx and windows

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=f534d2edbd062efe0537cd7966e87bcf5ef73a84

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219251039&repo=mozilla-inbound&lineNumber=1007

Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/22465ecf3808b71eae9e73d0472a271bb505e673

Joel, the failures mention:

[task 2018-12-29T22:52:55.279Z] 22:52:55 CRITICAL - Test harness output was not a valid structured log message: 
[task 2018-12-29T22:52:55.279Z] 22:52:55 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/wptmanifest/parser.py", line 395, in expr_state
[task 2018-12-29T22:52:55.280Z] 22:52:55 CRITICAL - Test harness output was not a valid structured log message: 
[task 2018-12-29T22:52:55.280Z] 22:52:55 CRITICAL -     raise ParseError(self.filename, self.line_number, "Comment before end of expression")
[task 2018-12-29T22:52:55.281Z] 22:52:55 CRITICAL - Test harness output was not a valid structured log message: 
[task 2018-12-29T22:52:55.281Z] 22:52:55 CRITICAL - wptrunner.wptmanifest.parser.ParseError: Comment before end of expression: /builds/worker/workspace/build/tests/web-platform/meta/webdriver/tests/new_session/create_alwaysMatch.py.ini line 3
[task 2018-12-29T22:52:55.634Z] 22:52:55    ERROR - Return code: 1
Flags: needinfo?(jmaher)
I am not sure why this is failing.  It appears that we cannot disable the entire file, only a subtest, the problem is we never get to a subtest, we fail to get the file loaded.

:ato, can you help us figure out how to disable this test on windows10 code coverage?  Or maybe fix it so it works would be fine as well.  I would needinfo :whimboo, but he doesn't accept them and I see that you reviewed this test originally.
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(jmaher)
Flags: needinfo?(ato)
A comment like that is not allowed at the end of the line. See https://searchfox.org/mozilla-central/source/testing/web-platform/meta/css/css-backgrounds/background-repeat/background-repeat-round-roundup.xht.ini in how to actually do it.

Christina, can you please fix that? It would be great if you could also push a try build to verify that it works.
Flags: needinfo?(ato) → needinfo?(ccoroiu)
Modified the patch, please take a look.
Attachment #9033634 - Attachment is obsolete: true
Flags: needinfo?(hskupin)
Attachment #9034156 - Flags: review?(jmaher)
Attachment #9034156 - Flags: review?(jmaher) → review+
Pushed by ccoroiu@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/63027ff03eff
disable create_alwaysMatch.py on windows ccov for frequent failures. r=jmaher
Keywords: checkin-needed
Flags: needinfo?(hskupin)
Assignee: ccoroiu → nobody
Status: REOPENED → NEW

The landed skip patch does no longer exist:
https://hg.mozilla.org/mozilla-central/file/tip/testing/web-platform/meta/webdriver/tests/new_session/create_alwaysMatch.py.ini

James, any reason why this was removed by an I assume downstream sync?

Flags: needinfo?(james)

No that was removed in Bug 1515432, persumably because they didn't correctly distinguish between Disabled and Expected.

Flags: needinfo?(james)

Now I see. Here the changeset:
https://hg.mozilla.org/mozilla-central/rev/463619034aae

The disable flag was bound to Windows 10 version == "10.0.15063". Joel, which version are we running those days?

Maybe we should re-evaluate each of the removals and if they are still failing make them disable without a specific Windows version? Or is that version added specifically to re-evaluate the state of the test each time we bump the version?

Blocks: 1515432
Flags: needinfo?(jmaher)

Everything should be running on Windows 10.0.17134 now, as far as I know. If the tests are failing there then we should add back the disabled clause with the updated Windows version.

Also I wonder why those entries haven't been updated to version "10.0.17134" but were left untouched, and afterward falsely got removed as obsolete.

Yes, they are all failing. Can you please add it back? Affected intermittent bugs are this one, bug 1507121, and bug 1495002.

Flags: needinfo?(kats)

thanks :kats. :whimboo, we tested things many times on win10 17134 and didn't see this failure, so assumed it was ok.

I will add it back tomorrow.

Flags: needinfo?(kats)

Thanks Joel!

disable tests on windows 10 for intermittent failures in webdriver new_session tests.

Flags: needinfo?(jmaher)
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ab359d0369df
disable tests on windows 10 for intermittent failures in webdriver new_session tests. r=whimboo

There is a new kind of intermittent which came up over the last month on Linux, which all seem to be related to a delayed marionette-startup-requested notification as covered by bug 1577713.

Depends on: 1577713
Status: NEW → RESOLVED
Closed: Last year2 months ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1495002
You need to log in before you can comment on or make changes to this bug.