Closed Bug 1416747 Opened 2 years ago Closed Last year

Intermittent test_cli_arguments.py TestCommandLineArguments.test_startup_timeout | AssertionError: IOError not raised

Categories

(Testing :: Marionette, defect, P2)

Version 3
Unspecified
Windows
defect

Tracking

(firefox65 fixed)

RESOLVED FIXED
mozilla65
Tracking Status
firefox65 --- fixed

People

(Reporter: aryx, Assigned: whimboo)

References

(Blocks 1 open bug)

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

+++ This bug was initially created as a clone of Bug #1400594 +++

https://treeherder.mozilla.org/logviewer.html#?job_id=144154567&repo=autoland

01:06:51     INFO -  Application command: Z:\task_1510529682\build\application\firefox\firefox.exe -no-remote -marionette -profile c:\users\genericworker\appdata\local\temp\tmpnt6nwe.mozrunner
01:06:51     INFO -  *** You are running in headless mode.
01:06:51     INFO -  1510535211536	Marionette	DEBUG	Received observer notification "profile-after-change"
01:06:51     INFO -  1510535211571	Marionette	DEBUG	Received observer notification "command-line-startup"
01:06:51     INFO -  1510535211571	Marionette	INFO	Enabled via --marionette
01:06:52     INFO -  1510535212108	Marionette	DEBUG	Received observer notification "sessionstore-windows-restored"
01:06:52     INFO -  1510535212246	Marionette	DEBUG	New connections are accepted
01:06:52     INFO -  1510535212246	Marionette	INFO	Listening on port 2828
01:06:52     INFO -  1510535212300	Marionette	DEBUG	Accepted connection 0 from 127.0.0.1:49426
01:06:52     INFO -  1510535212311	Marionette	DEBUG	Accepted connection 1 from 127.0.0.1:49434
01:06:52     INFO -  1510535212312	Marionette	DEBUG	Closed connection 0
01:06:52     INFO -  1510535212352	Marionette	TRACE	1 -> [0,1,"newSession",{}]
01:06:52     INFO -  1510535212414	Marionette	DEBUG	Register listener.js for window 2147483649
01:06:52     INFO -  1510535212433	Marionette	TRACE	1 <- [1,1,null,{"sessionId":"4cc97a8d-db25-410e-98a3-013606aa43d1","capabilities":{}}]
01:06:52     INFO -  1510535212436	Marionette	DEBUG	Closed connection 1
01:06:52     INFO -  1510535212437	Marionette	DEBUG	Accepted connection 2 from 127.0.0.1:49437
01:06:52     INFO -  1510535212449	Marionette	TRACE	2 -> [0,1,"newSession",{}]
01:06:52     INFO -  1510535212451	Marionette	DEBUG	Register listener.js for window 2147483649
01:06:52     INFO -  1510535212462	Marionette	TRACE	2 <- [1,1,null,{"sessionId":"577d3718-9574-4413-a14b-cd1f77434bf7","capabilities":{}}]
01:06:52     INFO -  1510535212490	Marionette	TRACE	2 -> [0,2,"getContext",{}]
01:06:52     INFO -  1510535212490	Marionette	TRACE	2 <- [1,2,null,{"value":"content"}]
01:06:52     INFO -  1510535212503	Marionette	TRACE	2 -> [0,3,"setContext",{"value":"chrome"}]
01:06:52     INFO -  1510535212504	Marionette	TRACE	2 <- [1,3,null,{}]
01:06:52     INFO -  1510535212508	Marionette	TRACE	2 -> [0,4,"takeScreenshot",{"highlights":null,"full":true,"hash":false,"id":null,"scroll":true}]
01:06:52     INFO -  1510535212548	Marionette	TRACE	2 <- [1,4,null,{"value":"iVBORw0KGgoAAAANSUhEUgAABVYAAAMACAYAAADPPjzCAAAgAElEQVR4nOzdb3DcB33n8X3YmXtAH/hxcvW1nUKrdOagtNyU67UeMi05Jr1RTxzqFULTiDNDgdStr ... YBQAAAACYxCoAAAAAwCRWAQAAAAAmsQoAAAAAMIlVAAAAAIBJrAIAAAAATGIVAAAAAGASqwAAAAAAk1gFAAAAAJjEKgAAAADAFI2g8JK5nUHBAAAAAElFTkSuQmCC"}]
01:06:52     INFO -  1510535212551	Marionette	TRACE	2 -> [0,5,"setContext",{"value":"content"}]
01:06:52     INFO -  1510535212551	Marionette	TRACE	2 <- [1,5,null,{}]
01:06:52     INFO -  1510535212553	Marionette	TRACE	2 -> [0,6,"getContext",{}]
01:06:52     INFO -  1510535212554	Marionette	TRACE	2 <- [1,6,null,{"value":"content"}]
01:06:52     INFO -  1510535212557	Marionette	TRACE	2 -> [0,7,"setContext",{"value":"content"}]
01:06:52     INFO -  1510535212557	Marionette	TRACE	2 <- [1,7,null,{}]
01:06:52     INFO -  1510535212561	Marionette	TRACE	2 -> [0,8,"getPageSource",{}]
01:06:52     INFO -  1510535212562	Marionette	TRACE	2 <- [1,8,null,{"value":"<html><head></head><body></body></html>"}]
01:06:52     INFO -  1510535212565	Marionette	TRACE	2 -> [0,9,"setContext",{"value":"content"}]
01:06:52     INFO -  1510535212565	Marionette	TRACE	2 <- [1,9,null,{}]
01:06:52    ERROR -  TEST-UNEXPECTED-FAIL | test_cli_arguments.py TestCommandLineArguments.test_startup_timeout | AssertionError: IOError not raised
01:06:52     INFO -  Traceback (most recent call last):
01:06:52     INFO -    File "Z:\task_1510529682\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 156, in run
01:06:52     INFO -      testMethod()
01:06:52     INFO -    File "Z:\task_1510529682\build\tests\marionette\tests\testing\marionette\harness\marionette_harness\tests\unit\test_cli_arguments.py", line 48, in test_startup_timeout
01:06:52     INFO -      self.marionette.start_session()
01:06:52     INFO -  TEST-INFO took 1401ms
[Mass Change 2018-01-15] Moving bugs to backlog
Priority: -- → P3
Priority: P3 → P5
Fixing this might also help for bug 1391545. While this specific failure doesn't happen nowadays I want to improve this test to may cause lesser race conditions. Right now I can see specifically for fast pgo/nightly builds a couple of failures on Windows 7 32bit. As such using a startup_timeout of 0s (and not 0.1s) should be best, so that we fail immediately. Note that those fast builds come up within 100ms, even with Marionette available!
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Priority: P5 → P2
Using a startup timeout of 0s should always cause an IOError,
even for builds which startup within 100ms (pgo, Nightly opt).

Further the test should not modify the internal startup timeout
property, but pass the timeout as parameter to "start_session()".
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bb0fb973da23
[marionette] Hardening test_cli_arguments.py TestCommandLineArguments.test_startup_timeout. r=ato
https://hg.mozilla.org/mozilla-central/rev/bb0fb973da23
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
You need to log in before you can comment on or make changes to this bug.