<a class="header-button" href="https://bugzilla.mozilla.org/home" title="Go to home page"> Bugzilla

Comment 39

•

11 months ago

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #38)

Well, for recent failures at least I can see the following log entries:
https://treeherder.mozilla.org/logviewer?job_id=509217630&repo=autoland&lineNumber=15451-15453

task 2025-05-20T11:55:51.579Z] 11:55:51     INFO - GECKO(2633) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /var/folders/8h/42m81v7x53xc8ycgyw2nl0vw0000jb/T/tmphcvisqvp.mozrunner/runtests_leaks_tab_pid2641.log
[task 2025-05-20T11:55:51.787Z] 11:55:51     INFO - GECKO(2633) | Couldn't convert chrome URL: chrome://branding/locale/brand.properties
[task 2025-05-20T11:55:51.788Z] 11:55:51     INFO - GECKO(2633) | [Child 2641, Main Thread] WARNING: Could not get the program name for a cubeb stream.: 'NS_SUCCEEDED(rv)', file /builds/worker/checkouts/gecko/dom/media/CubebUtils.cpp:476
[task 2025-05-20T12:07:51.164Z] 12:07:51     INFO - runtests.py | Waiting for browser...

Gijs, do you have an idea/

The screenshot from this is super weird. Joel, are we expecting raritan hardware connected to our thing? Can we dismiss this prompt? Is this issue specific to particular (virtual) machines?

If there is a problem with chrome://branding/locale/brand.properties it most likely is a broken build? I have seen things like that only when updating a local checkout and failing to build before starting Firefox. Here we clearly hang and also cannot get a connection to Marionette to install the mochijar and specialpowers addons.

Note that this line is only printed in debug builds and as per list only those builds are failing here. IMHO this is a different issue than bug 1960740.

I don't understand what "as per list" means. Do other, working debug build tests report this same error? Have you tried downloading the artifacts that are being run and do they run locally? Any evidence that things are getting lost when moving content out of dmgs and such?

Flags: needinfo?(jmaher)

Flags: needinfo?(hskupin)

Flags: needinfo?(gijskruitbosch+bugs)

Joel Maher ( :jmaher ) (UTC -8)

Comment 40

•

11 months ago

These lines in the log about not killing things are also weird: https://treeherder.mozilla.org/logviewer?job_id=509217630&repo=autoland&lineNumber=15385-15390

Comment 41

•

11 months ago

thanks for pointing that out :gijs, the raritan is what we use for pdu/kvm, but we need to dismiss this dialog properly and it shouldn't be popping up in general, maybe setup had some issues.

:rcurran, can you look at the kvm setup and ensure all "devices" are properly setup on the 14.7.5 machines (per screenshot)?

Flags: needinfo?(jmaher) → needinfo?(rcurran)

Comment 42

•

11 months ago

(In reply to :Gijs (he/him) from comment #39)

I don't understand what "as per list" means. Do other, working debug build tests report this same error? Have you tried downloading the artifacts that are being run and do they run locally? Any evidence that things are getting lost when moving content out of dmgs and such?

I meant the intermittent failures list. Sorry when it was not that clear.

These kind of hang also happens when this Raritan dialog is not open. It looks like Firefox doesn't open any window, and the last notification that we receive in Marionette is final-ui-startup.

I now downloaded such a DMG and run it locally after installing with mozinstall. I didn't observe an issue.

Flags: needinfo?(hskupin)

Comment 43

•

11 months ago

Note that those failures started on May 13th. Joel and Ryan, was it the time when we started to upgrade from 14.7.1 to 14.7.5?

Comment hidden (Intermittent Failures Robot)

Comment 46

•

11 months ago

•

Edited

We started that rollout on May 6th, completed it on May 8th, re-puppetized the workers on May 9th (so that the gw config would update reflecting the correct OS version), and updated Safari on May 12th.

Flags: needinfo?(rcurran)

Comment hidden (Intermittent Failures Robot)

Comment 50

•

11 months ago

Recent failures still have the same dialog popping up: https://firefoxci.taskcluster-artifacts.net/AzpAHKtFTfutFuIRkXmsGw/0/public/test_info/mozilla-test-fail-screenshot_d9ufdv9p.png

Renewing this needinfo for that bit in case it is interfering with Firefox in some way:

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #41)

thanks for pointing that out :gijs, the raritan is what we use for pdu/kvm, but we need to dismiss this dialog properly and it shouldn't be popping up in general, maybe setup had some issues.

:rcurran, can you look at the kvm setup and ensure all "devices" are properly setup on the 14.7.5 machines (per screenshot)?

Flags: needinfo?(rcurran)

Comment 51

•

11 months ago

(In reply to :Gijs (he/him) from comment #50)

Recent failures still have the same dialog popping up: https://firefoxci.taskcluster-artifacts.net/AzpAHKtFTfutFuIRkXmsGw/0/public/test_info/mozilla-test-fail-screenshot_d9ufdv9p.png

Note that not all of these failures actually show this dialog, but still a lot of notifications for firefox and xyz: https://firefoxci.taskcluster-artifacts.net/DHVZwPDCTK6TOaxzY87T2Q/0/public/test_info/mozilla-test-fail-screenshot_r3ig06b1.png.

But there is probably some other factor which is causing this issue.

Comment hidden (Intermittent Failures Robot)

Comment 53

•

11 months ago

:Gijs Is the needinfo a request to suppress the kvm keyboard prompt? Or something else? Thank you

Flags: needinfo?(rcurran)

Comment hidden (Intermittent Failures Robot)

Comment 55

•

11 months ago

(In reply to Ryan Curran from comment #53)

:Gijs Is the needinfo a request to suppress the kvm keyboard prompt? Or something else? Thank you

Yes - I quoted comment #41 where my read of Joel's comment was that hopefully we could get rid of the dialog. Even if it isn't itself interfering with the test, as long as it's there other people with intermittent failures will probably wonder about whether it is a cause (and/or may occlude what the actual app is doing), so it seems better to make sure it's not present.

Flags: needinfo?(rcurran)

Comment hidden (Intermittent Failures Robot)

Cristina Horotan [:chorotan]

Comment 59

•

10 months ago

(In reply to :Gijs (he/him) from comment #55)

(In reply to Ryan Curran from comment #53)

:Gijs Is the needinfo a request to suppress the kvm keyboard prompt? Or something else? Thank you

Yes - I quoted comment #41 where my read of Joel's comment was that hopefully we could get rid of the dialog. Even if it isn't itself interfering with the test, as long as it's there other people with intermittent failures will probably wonder about whether it is a cause (and/or may occlude what the actual app is doing), so it seems better to make sure it's not present.

Thank you. I have created a work item for this issue https://mozilla-hub.atlassian.net/browse/RELOPS-1601

Flags: needinfo?(rcurran)

Comment hidden (Intermittent Failures Robot)

Updated

•

7 months ago

Whiteboard: [stockwell disable-recommended]

Comment hidden (Intermittent Failures Robot)

Comment 77

•

6 months ago

Note that I was able to see the exact problem locally on my MBP. I filed bug 1997841 with a Gecko profile attached. Lets hope it's useful and we can figure out why it's happening. this is a high frequent failure - whereby the underlying reason in the mochitest harness is the same as bug 1414495.

Depends on: 1997841

Updated

•

6 months ago

Depends on: 1812622

Comment hidden (Intermittent Failures Robot)

Comment 84

•

4 months ago

As it looks like the infra changes as pushed to macOS workers on bug 2004631 has fixed the timeouts completely for that platform. Lets revisit later this week if we can close this bug now given that only macOS was affected by it over the last 30 days.

Depends on: 2004631
No longer depends on: 1812622

Comment 85

•

4 months ago

(In reply to Henrik Skupin [:whimboo][⌚️UTC+1] from comment #84)

As it looks like the infra changes as pushed to macOS workers on bug 2004631 has fixed the timeouts completely for that platform. Lets revisit later this week if we can close this bug now given that only macOS was affected by it over the last 30 days.

Actually macOS machines need to be provisioned again. The whole process will take a bit longer. So no improvements will be seen immediately.

Comment hidden (Intermittent Failures Robot)

Andrew Halberstadt [:ahal]

Comment 90

•

3 months ago

With the fix on bug 2004631 most of the failures are gone now.

But what I wonder why we are waiting twice that long (740s) on macOS instead of the 360s on other platforms. I'm asking because this bug is basically all the same as bug 1414495, which is about Waiting for browser....

From the logs it looks like that only verify mode is actually affected and I can see those lines:

[task 2026-01-06T01:21:12.194+00:00] 01:21:12     INFO - Increasing default timeout to 90 seconds (asan or debug)
[task 2026-01-06T01:21:12.194+00:00] 01:21:12     INFO - Increasing default timeout to 180 seconds (MOZ_CHAOSMODE)

But still there is a factor of 4 still missing which might get applied at some other point and that is not listed anywhere in the logs.

Andrew, do you have an idea where this could be done and especially why we need that? I don't see an actual reason for it.

Flags: needinfo?(ahal)

Comment 91

•

3 months ago

Sorry, I'm not sure. There could be some logic in one of the transforms under taskcluster/gecko_taskgraph/transforms/test (try grepping for max-run-time), or sometimes test manifests can increase the timeoutfactor as well, e.g:
https://searchfox.org/firefox-main/source/browser/components/backup/tests/xpcshell/xpcshell.toml#35

Flags: needinfo?(ahal)

https://treeherder.mozilla.org/logviewer?job_id=542468879&repo=autoland&lineNumber=14584

Comment 92

•

3 months ago

(In reply to Andrew Halberstadt [:ahal] from comment #91)

Sorry, I'm not sure. There could be some logic in one of the transforms under taskcluster/gecko_taskgraph/transforms/test (try grepping for max-run-time), or sometimes test manifests can increase the timeoutfactor as well, e.g:
https://searchfox.org/firefox-main/source/browser/components/backup/tests/xpcshell/xpcshell.toml#35

But that should as well be reflected in the log output as shown above. When I query the log viewer for Increasing default timeout nothing is listed which would explain an increase of the default timeout. The maximum is still set to 180s like for this job:

If we silently applying other multiplicators we should as well log that and not doing it silently.

Comment hidden (Intermittent Failures Robot)

Updated

•

3 months ago

Depends on: 2013816

Comment hidden (Intermittent Failures Robot)

amarc

Updated

•

1 month ago

Summary: Intermittent TV automation.py | application timed out after 740 seconds with no output → Intermittent TV automation.py | application timed out after 740.0 seconds with no output | single tracking bug

Comment hidden (Intermittent Failures Robot)

Comment 102

•

1 month ago

It looks like that the increase of this failure on macOS might be related to the Keyboard Setup Assistent dialog which pops-up for all the affected tests.

But not sure if that is actually the real underlying reason why the Firefox window doesn't open at all.

@Ryan are you aware of this dialog? Is that something new or do you already have a bug for it?

Flags: needinfo?(rcurran)

Comment hidden (Intermittent Failures Robot)

Comment 104

•

1 month ago

hi :whimboo

we landed a change with a potential fix for this about a month ago. are the keyboard prompts still showing up since then?

thank you

Flags: needinfo?(rcurran) → needinfo?(hskupin)

https://treeherder.mozilla.org/logviewer?job_id=554874271&repo=autoland&lineNumber=6556
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/HmFo3kbaSUeJrnyMAIY0BQ/runs/0/artifacts/public/test_info/mozilla-test-fail-screenshot_0wck_6ic.png

Comment 105

•

1 month ago

Yes it does. Here a failing job from Saturday with the associated screenshot:

Flags: needinfo?(hskupin) → needinfo?(rcurran)

Comment 106

•

1 month ago

thank you. i have a new approach im going to test https://github.com/mozilla-platform-ops/ronin_puppet/pull/1141

Updated

•

1 month ago

See Also: → https://github.com/mozilla-platform-ops/ronin_puppet/pull/1141

Comment 107

•

1 month ago

merged the pr. the change should land on all the mac14/15 workers in an hour or so. keep me posted :whimboo if you still see the issue after that time. thanks

Flags: needinfo?(rcurran)

Comment 108

•

1 month ago

Great! Let it bake some days given that sometimes new failures appear days after. I'll check back on Monday with the regular triage work if the situation has been improved, and if that keyboard dialog was the reason for the failures at all.

Comment hidden (Intermittent Failures Robot)