Open Bug 1459118 Opened 2 years ago Updated 6 months ago

Intermittent /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_twice - NoSuchAlertException: no such alert (404): No modal dialog is currently open

Categories

(Testing :: geckodriver, defect, P5)

Version 3
defect

Tracking

(firefox-esr52 unaffected, firefox-esr60 unaffected, firefox60 unaffected, firefox61 disabled, firefox62 disabled, firefox63 disabled, firefox64 disabled, firefox65 disabled)

Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- unaffected
firefox60 --- unaffected
firefox61 --- disabled
firefox62 --- disabled
firefox63 --- disabled
firefox64 --- disabled
firefox65 --- disabled

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Depends on 2 open bugs, )

Details

(Keywords: intermittent-failure, leave-open, regression, Whiteboard: [retriggered] )

Attachments

(1 file, 1 obsolete file)

Filed by: shindli [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=176894538&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/dgteqjIGTN6U7oXUC_XOkA/runs/0/artifacts/public/logs/live_backing.log

[task 2018-05-04T05:05:53.963Z] 05:05:53     INFO - tests/web-platform/tests/tools/webdriver/webdriver/client.py:438: InvalidArgumentException
[task 2018-05-04T05:05:53.963Z] 05:05:53     INFO - TEST-PASS | /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_default 
[task 2018-05-04T05:05:53.964Z] 05:05:53     INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_twice - NoSuchAlertException: no such alert (404): No modal dialog is currently open
[task 2018-05-04T05:05:53.964Z] 05:05:53     INFO - new_session = <function create_session at 0x7f9c02b03848>
[task 2018-05-04T05:05:53.964Z] 05:05:53     INFO - add_browser_capabilites = <function update_capabilities at 0x7f9c02b03938>
I did some retriggers and backfills to find where this started and worked on this range:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=linux%20x64%20wd%20pgo&tochange=f579a28444855e4ccbf45c8c82f9bb78ebbe822b&fromchange=afbec9a6802cb45f5032301b997c15f7d1c25884 

The next shift will continue with the retriggers.
Whiteboard: [retriggered]
Continued the retriggers and the failure rate is 2 out of 20 in general, on the push where this started:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=linux%20x64%20wd%20pgo&tochange=355349b199c457d45e420d775df07f7b6e9df859&fromchange=5bde59ad4082e94f3deea06808152b3ad0077876&selectedJob=176170088
we have 20 failures out of 21.
Flags: needinfo?(jmaher)
Thanks Eliza- this is great work and shows the exact problem.

I see that :aryx fixed the manifest in the next commit that landed:
https://hg.mozilla.org/integration/mozilla-inbound/rev/355349b199c457d45e420d775df07f7b6e9df859
Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(jmaher)
Resolution: --- → FIXED
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)
> Thanks Eliza- this is great work and shows the exact problem.
> 
> I see that :aryx fixed the manifest in the next commit that landed:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/
> 355349b199c457d45e420d775df07f7b6e9df859

Yes, he did, however the issue still occurs: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=41d06ef57fe6894c4ebd5a2e89b8788b1fd5fd8f&selectedJob=177152800


[task 2018-05-06T03:46:43.085Z] 03:46:43     INFO -         if response.status != 200:
[task 2018-05-06T03:46:43.085Z] 03:46:43     INFO - >           raise error.from_response(response)
[task 2018-05-06T03:46:43.085Z] 03:46:43     INFO - E           InvalidArgumentException: invalid argument (400): Invalid unhandledPromptBehavior value: ignore
[task 2018-05-06T03:46:43.086Z] 03:46:43     INFO - 
[task 2018-05-06T03:46:43.086Z] 03:46:43     INFO - body       = {'capabilities': {'alwaysMatch': {'moz:firefoxOptions': {'binary': '/builds/worker/workspace/build/application/firefox...xn--lve-6lad.web-platform.test,www2.web-platform.test,www1.web-platform.test'}}, 'unhandledPromptBehavior': 'ignore'}}}
[task 2018-05-06T03:46:43.086Z] 03:46:43     INFO - method     = 'POST'
[task 2018-05-06T03:46:43.086Z] 03:46:43     INFO - response   = <Response status=400 error=<InvalidArgumentException http_status=400>>
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - self       = <Session (disconnected)>
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - url        = 'session'
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - 
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - tests/web-platform/tests/tools/webdriver/webdriver/client.py:438: InvalidArgumentException
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - TEST-PASS | /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_default 
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/execute_script/user_prompts.py | test_handle_prompt_twice - NoSuchAlertException: no such alert (404): No modal dialog is currently open
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - new_session = <function create_session at 0x7f7ba0038aa0>
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - add_browser_capabilites = <function update_capabilities at 0x7f7ba0038b90>
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO - 
[task 2018-05-06T03:46:43.089Z] 03:46:43     INFO -     def test_handle_prompt_twice(new_session, add_browser_capabilites):
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO -         _, session = new_session({"capabilities": {"alwaysMatch": add_browser_capabilites({"unhandledPromptBehavior": "accept"})}})
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO -         value = session.execute_script("window.alert('Hello');window.alert('Bye');")
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO -         assert value is None
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO -         session.alert.dismiss()
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO -         # The first alert has been accepted by the user prompt handler, the second one remains.
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO -         # FIXME: this is how browsers currently work, but the spec should clarify if this is the
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO -         #        expected behavior, see https://github.com/w3c/webdriver/issues/1153.
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO - >       assert session.alert.text == "Bye"
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO - 
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO - _          = {'capabilities': {'acceptInsecureCerts': False, 'browserName': 'firefox', 'browserVersion': '61.0a1', 'moz:accessibilityChecks': False, ...}, 'sessionId': 'b581a7cb-8b56-42fb-a9be-a7e4d4f7f92d'}
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO - add_browser_capabilites = <function update_capabilities at 0x7f7ba0038b90>
[task 2018-05-06T03:46:43.090Z] 03:46:43     INFO - new_session = <function create_session at 0x7f7ba0038aa0>
[task 2018-05-06T03:46:43.091Z] 03:46:43     INFO - session    = <Session b581a7cb-8b56-42fb-a9be-a7e4d4f7f92d>
[task 2018-05-06T03:46:43.091Z] 03:46:43     INFO - value      = None
[task 2018-05-06T03:46:43.092Z] 03:46:43     INFO - 
[task 2018-05-06T03:46:43.092Z] 03:46:43     INFO - tests/web-platform/tests/webdriver/tests/execute_script/user_prompts.py:73:
Flags: needinfo?(jmaher)
Not sure why this bug has been marked as fixed. As we can see it still occurs. So reopening the bug. I can have a look later today.
Status: RESOLVED → REOPENED
Flags: needinfo?(jmaher) → needinfo?(hskupin)
Resolution: FIXED → ---
A commit as listed in these changesets caused the regression:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=6b980ffeab0c61f0eae10fb6cdac204c34f23c63

Given that this happens for `execute_script` and `execute_async_script` I would point out to bug 1453105 which originally comes from https://github.com/w3c/web-platform-tests/pull/10401.
Blocks: 1453105
Flags: needinfo?(hskupin)
Keywords: regression
Ok, so the change as landed via the following PR now closes the second alert as opened by the execute_script call:

https://github.com/w3c/web-platform-tests/pull/10401/files#diff-3c9886c5a3df541e02044c66f125020fR65

Given that in Firefox we do not yet support multiple alerts even not within the same browsing context (see bug 1264259) we should completely disable this specific test. Andreas, given that you reviewed this PR and spec change, do you agree?
Depends on: 1264259
Flags: needinfo?(ato)
thanks :whimboo!
What's strange is that this doesn't fail that constantly anymore now that some platforms / builds have been disabled. I wonder if there is some kind of race which causes that to appear.
(In reply to Henrik Skupin (:whimboo) from comment #8)

> Given that in Firefox we do not yet support multiple alerts even
> not within the same browsing context (see bug 1264259) we should
> completely disable this specific test. Andreas, given that you
> reviewed this PR and spec change, do you agree?

Yes, I agree with that.

(In reply to Henrik Skupin (:whimboo) from comment #10)

> What's strange is that this doesn't fail that constantly anymore
> now that some platforms / builds have been disabled. I wonder if
> there is some kind of race which causes that to appear.

I think it is likely there might be a collation of problems.  Even
though Marionette doesn’t support the user prompt handler it does
have _some_ support for interacting with user prompts.  The test
harness tries to clean up any user prompts between tests and this
might “conceal” some of the test failures/hangs.
Flags: needinfo?(ato)
Hm, I wonder how the disabled flag works for subtests as what wpt-sync adds like in this case. When I check the logs I can clearly see that tests are still run. Also it would be in conflict of what I have seen on bug 1447708. James, can you please explain?
Flags: needinfo?(james)
We run the test, but ignore the result of the subtest.
Flags: needinfo?(james)
I see. Thanks. So let me ask you for review.
Comment on attachment 8974285 [details]
Bug 1459118 - [wdspec] Disable test_handle_prompt_twice for execute (async) script.

https://reviewboard.mozilla.org/r/242596/#review248578
Attachment #8974285 - Flags: review?(james) → review+
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8359164a5945
[wdspec] Disable test_handle_prompt_twice for execute (async) script. r=jgraham
https://hg.mozilla.org/mozilla-central/rev/8359164a5945
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Assignee: nobody → hskupin
This is not fixed but the test has been disabled.
Assignee: hskupin → nobody
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
Target Milestone: mozilla62 → ---
Attachment #8974285 - Attachment is obsolete: true
Assignee: nobody → hskupin
Status: REOPENED → ASSIGNED
Priority: P5 → P2
This is still failing for us, and as far as I can see it is somewhat blocked on https://github.com/w3c/webdriver/issues/1153. Andreas or David, do you think we can get some clarification for that spec issue?
Assignee: hskupin → nobody
Status: ASSIGNED → NEW
Flags: needinfo?(dburns)
Flags: needinfo?(ato)
Priority: P2 → P5
Actually this might be dependent on bug 1270585, which covers the real problem and not only the failing test. Appropriate feedback is still welcome.
https://github.com/w3c/webdriver/issues/1153 should be addressed
by https://github.com/w3c/webdriver/pull/1248.

When a user prompt appears during script evaluation, the script is
immediately aborted and WebDriver returns to the user with data
value of null.

I confirmed a few days ago that this is the current behaviour of
Marionette.
Flags: needinfo?(dburns)
Flags: needinfo?(ato)
So we discussed this problem in our WebDriver Meeting (https://wiki.mozilla.org/WebDriver/Meetings/2018-07-30) today. The implementation of the test is actually still the old behavior before the last change to the spec was done for the two execute script commands. 

As such the test has to be updated in that it will assert that only a single prompt is open and the second one never has been created.
Well, the outcome of our discussion wasn't that perfect. Issue https://github.com/w3c/webdriver/issues/1153 has not been addressed yet, and any change as mentioned in my last comment would break the IEDriver. As such the issue has been reopened and we still need a clarification for the spec.

This bug has to wait until then.
We are trying to build a tool to automatically classify intermittent failures, which would provide a starting point for fixing the bug, reducing the manual work for the developers.
We are collecting some feedback on the results, to see if they’re good enough and where we need to improve.
    
For this bug, the tool says that the intermittent failure is most likely a:
Concurrency Issue: This includes tests in which failures occur due to thread management issues (different threads or their outcomes depending on an implicit ordering), race conditions and/or deadlocks, and issues related to an asynchronous waits (e.g. a process trying to access an external resource or continuing before the external resource is available).


Once you’re done investigating and/or fixing the bug, could you tell me:
- Did the tool correctly recognize the type of intermittent failure?
- Did the information from the tool help your analysis, the bug fixing process, or anything in the process? (please also let us know how the tool was useful and/or what would improve the tool's usefulness for you)
This is actually blocked by bug 1487358 if opening two alerts is the expected behavior. The discussion is still ongoing on bug https://github.com/w3c/webdriver/issues/1153.
No longer depends on: 1487358
No longer depends on: 1560010
You need to log in before you can comment on or make changes to this bug.