Closed Bug 1984331 Opened 1 year ago Closed 10 months ago

Perma [tier 2] Headless TEST-UNEXPECTED-OK | /webdriver/tests/interop/beforeunload_prompt.py | expected TIMEOUT

Categories

(Remote Protocol :: Marionette, defect, P5)

defect

Tracking

(firefox-esr128 unaffected, firefox-esr140 unaffected, firefox142 unaffected, firefox143 unaffected, firefox144 disabled, firefox147 disabled, firefox148 disabled, firefox149 fixed)

RESOLVED FIXED
149 Branch
Tracking Status
firefox-esr128 --- unaffected
firefox-esr140 --- unaffected
firefox142 --- unaffected
firefox143 --- unaffected
firefox144 --- disabled
firefox147 --- disabled
firefox148 --- disabled
firefox149 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: Sasha)

References

(Regression)

Details

(Keywords: intermittent-failure, regression, test-disabled)

Attachments

(1 external file)

Filed by: agoloman [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=523381241&repo=mozilla-central&task=J3YywzWiThGd0_XrAgUqVQ.0
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/J3YywzWiThGd0_XrAgUqVQ/runs/0/artifacts/public/logs/live_backing.log


[task 2025-08-21T11:36:07.840+00:00] 11:36:07     INFO - STDOUT: =================================== FAILURES ===================================
[task 2025-08-21T11:36:07.840+00:00] 11:36:07     INFO - STDOUT: _________________________________ test_accept __________________________________
[task 2025-08-21T11:36:07.841+00:00] 11:36:07     INFO - STDOUT: fut = <Future cancelled>, timeout = 8.0
[task 2025-08-21T11:36:07.842+00:00] 11:36:07     INFO - STDOUT:     async def wait_for(fut, timeout):
[task 2025-08-21T11:36:07.843+00:00] 11:36:07     INFO - STDOUT:         """Wait for the single Future or coroutine to complete, with timeout.
[task 2025-08-21T11:36:07.843+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.844+00:00] 11:36:07     INFO - STDOUT:         Coroutine will be wrapped in Task.
[task 2025-08-21T11:36:07.844+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.845+00:00] 11:36:07     INFO - STDOUT:         Returns result of the Future or coroutine.  When a timeout occurs,
[task 2025-08-21T11:36:07.846+00:00] 11:36:07     INFO - STDOUT:         it cancels the task and raises TimeoutError.  To avoid the task
[task 2025-08-21T11:36:07.846+00:00] 11:36:07     INFO - STDOUT:         cancellation, wrap it in shield().
[task 2025-08-21T11:36:07.847+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.848+00:00] 11:36:07     INFO - STDOUT:         If the wait is cancelled, the task is also cancelled.
[task 2025-08-21T11:36:07.849+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.849+00:00] 11:36:07     INFO - STDOUT:         If the task suppresses the cancellation and returns a value instead,
[task 2025-08-21T11:36:07.850+00:00] 11:36:07     INFO - STDOUT:         that value is returned.
[task 2025-08-21T11:36:07.851+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.851+00:00] 11:36:07     INFO - STDOUT:         This function is a coroutine.
[task 2025-08-21T11:36:07.852+00:00] 11:36:07     INFO - STDOUT:         """
[task 2025-08-21T11:36:07.852+00:00] 11:36:07     INFO - STDOUT:         # The special case for timeout <= 0 is for the following case:
[task 2025-08-21T11:36:07.853+00:00] 11:36:07     INFO - STDOUT:         #
[task 2025-08-21T11:36:07.854+00:00] 11:36:07     INFO - STDOUT:         # async def test_waitfor():
[task 2025-08-21T11:36:07.857+00:00] 11:36:07     INFO - STDOUT:         #     func_started = False
[task 2025-08-21T11:36:07.857+00:00] 11:36:07     INFO - STDOUT:         #
[task 2025-08-21T11:36:07.857+00:00] 11:36:07     INFO - STDOUT:         #     async def func():
[task 2025-08-21T11:36:07.857+00:00] 11:36:07     INFO - STDOUT:         #         nonlocal func_started
[task 2025-08-21T11:36:07.857+00:00] 11:36:07     INFO - STDOUT:         #         func_started = True
[task 2025-08-21T11:36:07.857+00:00] 11:36:07     INFO - STDOUT:         #
[task 2025-08-21T11:36:07.858+00:00] 11:36:07     INFO - STDOUT:         #     try:
[task 2025-08-21T11:36:07.858+00:00] 11:36:07     INFO - STDOUT:         #         await asyncio.wait_for(func(), 0)
[task 2025-08-21T11:36:07.859+00:00] 11:36:07     INFO - STDOUT:         #     except asyncio.TimeoutError:
[task 2025-08-21T11:36:07.859+00:00] 11:36:07     INFO - STDOUT:         #         assert not func_started
[task 2025-08-21T11:36:07.860+00:00] 11:36:07     INFO - STDOUT:         #     else:
[task 2025-08-21T11:36:07.861+00:00] 11:36:07     INFO - STDOUT:         #         assert False
[task 2025-08-21T11:36:07.861+00:00] 11:36:07     INFO - STDOUT:         #
[task 2025-08-21T11:36:07.862+00:00] 11:36:07     INFO - STDOUT:         # asyncio.run(test_waitfor())
[task 2025-08-21T11:36:07.862+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.863+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.864+00:00] 11:36:07     INFO - STDOUT:         if timeout is not None and timeout <= 0:
[task 2025-08-21T11:36:07.864+00:00] 11:36:07     INFO - STDOUT:             fut = ensure_future(fut)
[task 2025-08-21T11:36:07.865+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.865+00:00] 11:36:07     INFO - STDOUT:             if fut.done():
[task 2025-08-21T11:36:07.866+00:00] 11:36:07     INFO - STDOUT:                 return fut.result()
[task 2025-08-21T11:36:07.866+00:00] 11:36:07     INFO - STDOUT:     
[task 2025-08-21T11:36:07.867+00:00] 11:36:07     INFO - STDOUT:             await _cancel_and_wait(fut)
[task 2025-08-21T11:36:07.868+00:00] 11:36:07     INFO - STDOUT:             try:
[task 2025-08-21T11:36:08.025+00:00] 11:36:08     INFO - TEST-UNEXPECTED-OK | /webdriver/tests/interop/beforeunload_prompt.py | expected TIMEOUT
[task 2025-08-21T11:36:08.025+00:00] 11:36:08     INFO - TEST-INFO expected TIMEOUT | took 97474ms
[task 2025-08-21T11:36:17.994+00:00] 11:36:17  WARNING - Forcibly terminating runner process
Flags: needinfo?(hskupin)

Interesting fact here is that this test only passes (and not timeout) on mozilla-central but not on autoland and also only in headless mode. This is maybe due to some other test running before which in this constellation hasn't been run on autoland yet and some headless mode related behavior. I'll dig deeper.

Flags: needinfo?(hskupin)
Summary: Perma [tier 2] TEST-UNEXPECTED-OK | /webdriver/tests/interop/beforeunload_prompt.py | expected TIMEOUT → Perma [tier 2] Headless TEST-UNEXPECTED-OK | /webdriver/tests/interop/beforeunload_prompt.py | expected TIMEOUT

In case of headless we have the following tests run, which is all what I would expect:
https://treeherder.mozilla.org/logviewer?job_id=523549797&repo=mozilla-central&task=Ip5EsfEqTC25UT47a0YUVQ.0

[task 2025-08-22T05:21:56.993+00:00] 05:21:56     INFO - TEST-START | /webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:21:57.232+00:00] 05:21:57     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_accept 
[task 2025-08-22T05:22:10.356+00:00] 05:22:10     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_accept_and_notify 
[task 2025-08-22T05:22:30.538+00:00] 05:22:30     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_dismiss 
[task 2025-08-22T05:22:50.662+00:00] 05:22:50     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_dismiss_and_notify 
[task 2025-08-22T05:23:10.560+00:00] 05:23:10     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_ignore 
[task 2025-08-22T05:23:36.877+00:00] 05:23:36     INFO - TEST-UNEXPECTED-OK | /webdriver/tests/interop/beforeunload_prompt.py | expected TIMEOUT

But in case of the timeout I see some suspicious logging:
https://treeherder.mozilla.org/logviewer?job_id=523549710&repo=mozilla-central&task=BV8Cvf8pSr6vVzX7k__2BA.0

[task 2025-08-22T05:21:32.398+00:00] 05:21:32     INFO - TEST-START | /webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:21:32.615+00:00] 05:21:32     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_accept 
[task 2025-08-22T05:21:46.771+00:00] 05:21:46     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_accept_and_notify 
[task 2025-08-22T05:22:06.852+00:00] 05:22:06     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_dismiss 
[task 2025-08-22T05:22:26.669+00:00] 05:22:26     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_dismiss_and_notify 
[task 2025-08-22T05:22:46.469+00:00] 05:22:46     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_ignore 
[task 2025-08-22T05:23:12.136+00:00] 05:23:12     INFO - TEST-OK | /webdriver/tests/interop/beforeunload_prompt.py | took 99685ms
[task 2025-08-22T05:23:12.208+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.210+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.340+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.343+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.492+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.494+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.641+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.643+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.796+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.798+00:00] 05:23:12     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py
[task 2025-08-22T05:23:12.849+00:00] 05:23:12     INFO - STDOUT: FAILED tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_accept - webdriver.error.TimeoutException: timeout (Future did not resolve within the given timeout)
[task 2025-08-22T05:23:12.850+00:00] 05:23:12     INFO - STDOUT: FAILED tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_accept_and_notify - webdriver.error.TimeoutException: timeout (Future did not resolve within the given timeout)
[task 2025-08-22T05:23:12.850+00:00] 05:23:12     INFO - STDOUT: FAILED tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_dismiss - webdriver.error.TimeoutException: timeout (Future did not resolve within the given timeout)
[task 2025-08-22T05:23:12.851+00:00] 05:23:12     INFO - STDOUT: FAILED tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_dismiss_and_notify - webdriver.error.TimeoutException: timeout (Future did not resolve within the given timeout)
[task 2025-08-22T05:23:12.852+00:00] 05:23:12     INFO - STDOUT: FAILED tests/web-platform/tests/webdriver/tests/interop/beforeunload_prompt.py::test_ignore - webdriver.error.TimeoutException: timeout (Future did not resolve within the given timeout)

The timeout that we are seeing is coming from:

        # Using WebDriver classic's navigation command to navigate away from
        # the page can hang if the prompt is not accepted. As such, wrap the
        # command to fail on timeout gracefully.
        def sync_navigate():
            current_session.url = page_target

        await wait_for_future_safe(
            asyncio.create_task(execute_as_async(sync_navigate)))

By default wait_for_future_safe() has a timeout of just 2s, which in case of a navigation is clearly too short. We should change that to at least use the default navigation timeout which is 3s, which gives us 12s for ASAN builds instead of just 8s due to the timeout multiplier of 4. Maybe these extra 4s will help - but still not sure why it is passing in headless mode. Maybe some widget related behavior with the beforeunload prompt.

Here a try build:
https://treeherder.mozilla.org/jobs?repo=try&revision=84d67379c9303b8f127356998260b39a1321c29f

When extending the timeout of the navigation we surprisingly now as well timeout for headless. Because we restart the browser a couple of times I would suggest that we extend the timeout similar to what we do for other user prompt tests. That should then make the test file to pass.

Lets see if that works:
https://treeherder.mozilla.org/jobs?repo=try&revision=8804f5a01f4f506a6584dd038a5c7aceefb426d5

Ok, so the problem here is actually that we timeout for the navigation that opens the beforeunload prompt. But instead of a shorter page load timeout of 3s * multiplier we run the full 300s now:

[task 2025-08-22T20:01:31.756+00:00] 20:01:31     INFO - PID 61880 | 1755892891754	RemoteAgent	TRACE	[15] No navigation found to commit for url: chrome://global/content/commonDialog.xhtml
[task 2025-08-22T20:01:32.149+00:00] 20:01:32     INFO - PID 61880 | 1755892892142	RemoteAgent	TRACE	Received observer notification common-dialog-loaded
[task 2025-08-22T20:01:32.149+00:00] 20:01:32     INFO - PID 61880 | 1755892892144	RemoteAgent	TRACE	Received observer notification common-dialog-loaded
[task 2025-08-22T20:01:32.149+00:00] 20:01:32     INFO - PID 61880 | 1755892892144	RemoteAgent	TRACE	[98ad9977-126e-4947-815d-36f0c7dccd5a] Navigation started for url:  (56854542-6dd6-417e-abb2-b98fb2986be5)
[task 2025-08-22T20:01:32.149+00:00] 20:01:32     INFO - PID 61880 | 1755892892144	RemoteAgent	TRACE	Received observer notification common-dialog-loaded
[task 2025-08-22T20:01:32.153+00:00] 20:01:32     INFO - PID 61880 | 1755892892148	RemoteAgent	DEBUG	WebDriverBiDiConnection d0f49a3e-4cbc-463f-996a-ed0e9e78d089 <- {"type":"event","method":"browsingContext.userPromptOpened","params":{"context":"98ad9977-126e-4947-815d-36f0c7dccd5a","handler":"accept","message":"This page is asking you to confirm that you want to leave — information you’ve entered may not be saved.","type":"beforeunload"}}
[task 2025-08-22T20:01:32.946+00:00] 20:01:32     INFO - PID 61880 | -----------------------------------------------------
[task 2025-08-22T20:01:32.947+00:00] 20:01:32     INFO - PID 61880 | Suppressions used:
[task 2025-08-22T20:01:32.948+00:00] 20:01:32     INFO - PID 61880 |   count      bytes template
[task 2025-08-22T20:01:32.949+00:00] 20:01:32     INFO - PID 61880 |      31      16288 nsComponentManagerImpl
[task 2025-08-22T20:01:32.950+00:00] 20:01:32     INFO - PID 61880 |       3        320 libfontconfig.so
[task 2025-08-22T20:01:32.951+00:00] 20:01:32     INFO - PID 61880 |       1       9496 style::sharing::SHARING_CACHE_KEY
[task 2025-08-22T20:01:32.951+00:00] 20:01:32     INFO - PID 61880 |       1       4104 style::bloom::BLOOM_KEY
[task 2025-08-22T20:01:32.951+00:00] 20:01:32     INFO - PID 61880 | -----------------------------------------------------
[taskcluster 2025-08-22T20:04:31.978Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/SiB3Nj2ySKGfBbJNpGPfkA/0/on/us-central1-c/4977546067492281329/until/1755894271.94
[task 2025-08-22T20:06:33.135+00:00] 20:06:33     INFO - PID 61880 | 1755893193134	Marionette	DEBUG	0 <- [1,6,{"error":"timeout","message":"Navigation timed out after 300000 ms","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\nWebDriverError@chrome://remote/content/shared/webdriver/Er ... tError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:847:5\nbail@chrome://remote/content/shared/Sync.sys.mjs:440:19\n"},null]

Because my patch uses current_session.timeouts.page_load we do not seem to setup correctly the custom page load timeout.

Because all the tests in this file wont work anyway until we have bug 1905086 implemented we potentially should disable the full test file.

Depends on: 1905086
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Pushed by hskupin@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/acd724973002 https://hg.mozilla.org/integration/autoland/rev/1cf434d08fec [wdspec] Temporary disable /webdriver/tests/interop/beforeunload_prompt.py. r=webdriver-reviewers,jdescottes
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/54523 for changes under testing/web-platform/tests
Duplicate of this bug: 1984737
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → 144 Branch
Assignee: hskupin → nobody
Status: RESOLVED → REOPENED
Keywords: test-disabled
Resolution: FIXED → ---
Target Milestone: 144 Branch → ---

Set release status flags based on info from the regressing bug 1965465

Upstream PR merged by moz-wptsync-bot

This is now fixed by bug 1905086 and the test was re-enabled for headless mode.

Status: REOPENED → RESOLVED
Closed: 1 year ago → 10 months ago
Resolution: --- → FIXED
Target Milestone: --- → 149 Branch
Assignee: nobody → aborovova
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: