Closed Bug 1569419 Opened 11 months ago Closed 11 months ago

Frequent OSX 10.14 browser-chrome failures due to OS notification, e.g. browser/components/customizableui/test/browser_984455_bookmarks_items_reparenting.js | Test timed out

Categories

(Firefox :: Toolbars and Customization, defect, P3)

defect

Tracking

()

RESOLVED WORKSFORME
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- unaffected
firefox68 --- unaffected
firefox69 --- unaffected
firefox70 --- affected

People

(Reporter: dvarga, Unassigned)

References

(Regression)

Details

(Keywords: intermittent-failure, regression, Whiteboard: [stockwell unknown])

Attachments

(2 files)

[Tracking Requested - why for this release]:

Central as Beta: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=258644109&revision=afb90a351f555e2fe80f843b1a86fda495cc543a&searchStr=browser%2Cchrome

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=258644109&repo=try

[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - Closing bookmarks menu
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - Leaving test bound testOverflowingBookmarksButtonContextMenu
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - Entering test bound testOverflowingBookmarksItemsContextMenu
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - Ensuring panel is ready.
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - Buffered messages logged at 13:05:20
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - Waiting for context menu on personal-bookmarks
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - Buffered messages finished
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - TEST-UNEXPECTED-FAIL | browser/components/customizableui/test/browser_984455_bookmarks_items_reparenting.js | Test timed out - 
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - GECKO(1826) | MEMORY STAT | vsize 7783MB | residentFast 484MB | heapAllocated 130MB
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - TEST-OK | browser/components/customizableui/test/browser_984455_bookmarks_items_reparenting.js | took 45035ms
[task 2019-07-27T13:06:03.812Z] 13:06:03     INFO - checking window state
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - GECKO(1826) | JavaScript error: resource://testing-common/PromiseTestUtils.jsm, line 112: uncaught exception: Object
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - TEST-START | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - Not taking screenshot here: see the one that was previously logged
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - Buffered messages logged at 13:06:03
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - Entering test bound 
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - Buffered messages finished
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - TEST-UNEXPECTED-FAIL | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js | Should start in default state. - 
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - Stack trace:
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - chrome://mochikit/content/browser-test.js:test_ok:1576
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - chrome://mochitests/content/browser/browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js:null:8
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1346
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1381
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1209
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:803
[task 2019-07-27T13:06:03.813Z] 13:06:03     INFO - Console message: [JavaScript Error: "uncaught exception: Object" {file: "resource://testing-common/PromiseTestUtils.jsm" line: 112}]
[task 2019-07-27T13:06:03.867Z] 13:06:03     INFO - Console message: OpenGL compositor Initialized Succesfully.
[task 2019-07-27T13:06:03.867Z] 13:06:03     INFO - Version: 2.1 INTEL-12.9.22
[task 2019-07-27T13:06:03.867Z] 13:06:03     INFO - Vendor: Intel Inc.
[task 2019-07-27T13:06:03.867Z] 13:06:03     INFO - Renderer: Intel Iris OpenGL Engine
[task 2019-07-27T13:06:03.867Z] 13:06:03     INFO - FBO Texture Target: TEXTURE_2D
[task 2019-07-27T13:06:04.199Z] 13:06:04     INFO - Not taking screenshot here: see the one that was previously logged
[task 2019-07-27T13:06:04.200Z] 13:06:04     INFO - TEST-UNEXPECTED-FAIL | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js | Reset button should be disabled - 
[task 2019-07-27T13:06:04.200Z] 13:06:04     INFO - Stack trace:
[task 2019-07-27T13:06:04.200Z] 13:06:04     INFO - chrome://mochikit/content/browser-test.js:test_ok:1576
[task 2019-07-27T13:06:04.201Z] 13:06:04     INFO - chrome://mochitests/content/browser/browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js:null:14
[task 2019-07-27T13:06:04.217Z] 13:06:04     INFO - TEST-PASS | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js | Toolbar should be uncollapsed in private window - 
[task 2019-07-27T13:06:04.217Z] 13:06:04     INFO - TEST-PASS | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js | Toolbar should be uncollapsed in normal window - 
[task 2019-07-27T13:06:04.217Z] 13:06:04     INFO - TEST-PASS | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js | Reset button should be enabled - 
[task 2019-07-27T13:06:04.349Z] 13:06:04     INFO - TEST-PASS | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js | Toolbar should be collapsed in private window - 
[task 2019-07-27T13:06:04.350Z] 13:06:04     INFO - TEST-PASS | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js | Toolbar should be collapsed in normal window - 
[task 2019-07-27T13:06:04.350Z] 13:06:04     INFO - TEST-PASS | browser/components/customizableui/test/browser_985815_propagate_setToolbarVisibility.js | Reset button should be disabled - 
[task 2019-07-27T13:06:04.559Z] 13:06:04     INFO - Leaving test bound 

OS X browser-chrome from 10.10 to 10.14 directly before the first fail (bug 1555454 comment 71).

Will create a beta simuldation with the changes from 1555454 reverted (to be able to test it on OS X 10.10).

Flags: needinfo?(bzbarsky)
No longer regressed by: 1568278

Indeed a regression from the 10.10=>10.14 move, not occurring in the reversion: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d4693716eeca6d82db8f9713414d95ba529147a6

Jared, what should be done with this one?

Regressed by: 1555454
Summary: Perma TEST-UNEXPECTED-FAIL | browser/components/customizableui/test/browser_984455_bookmarks_items_reparenting.js | Test timed out when when Gecko 70 merges to Beta on 2019-08-26 → Perma OSX 10.14 DevEdition browser/components/customizableui/test/browser_984455_bookmarks_items_reparenting.js | Test timed out

Here's a screenshot from the test failure from comment 0.

Is it possible that the "Welcome to macOS Mojave" notification is somehow interfering with us here? Is there any way to suppress that?

Flags: needinfo?(aryx.bugmail)

David, can you turn those macOS notifications off? That should also fix bug 1569368.

Flags: needinfo?(aryx.bugmail) → needinfo?(dhouse)

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #5)

David, can you turn those macOS notifications off? That should also fix bug 1569368.

This softwareupdater setting may be what we need to disable the notification: https://www.jamf.com/jamf-nation/discussions/32550/macos-tour-notifications
softwareupdate --ignore macOSInstallerNotification_GM

I verified that we didn't have this set (no updates were set as ignored) and I ran this across all of the mojave workers (so the notification *should stop on the workers from now), and I'll put in a patch to apply it through puppet also.

Flags: needinfo?(dhouse)

(In reply to Dave House [:dhouse] from comment #6)

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #5)

David, can you turn those macOS notifications off? That should also fix bug 1569368.

This softwareupdater setting may be what we need to disable the notification: https://www.jamf.com/jamf-nation/discussions/32550/macos-tour-notifications
softwareupdate --ignore macOSInstallerNotification_GM

I verified that we didn't have this set (no updates were set as ignored) and I ran this across all of the mojave workers (so the notification *should stop on the workers from now), and I'll put in a patch to apply it through puppet also.

This didn't remove the notification. I took screenshots across the pool and found the notification still appearing on machines that had rebooted (and not) and kept the ignored update setting.

There are some other notes that the upgrade prompt is disabled by setting noticeboard plist values. I'll test if that stops the welcome message, or keep looking.

Summary: Perma OSX 10.14 DevEdition browser/components/customizableui/test/browser_984455_bookmarks_items_reparenting.js | Test timed out → Frequent OSX 10.14 browser-chrome failures due to OS notification, e.g. browser/components/customizableui/test/browser_984455_bookmarks_items_reparenting.js | Test timed out

(In reply to Dave House [:dhouse] from comment #8)

There are some other notes that the upgrade prompt is disabled by setting noticeboard plist values. I'll test if that stops the welcome message, or keep looking.

Still no luck. I tried the noticeboard settings, but get the welcome message:

sudo /usr/bin/defaults write /Library/Preferences/com.apple.noticeboard.plist LastNoticeboardCatalogCheck "$(date -u "+%F %T %z")"
sudo /usr/bin/defaults write /Library/Preferences/com.apple.noticeboard.plist "com.apple.noticeboard.notification.mojave.2.0" -dict dismissalCount 4 lastDismissedDate "$(date -u "+%F %T %z")"
sudo /usr/bin/defaults write /Library/Preferences/com.apple.noticeboard.plist identifiers -array "com.apple.noticeboard.notification.mojave.2.0"
sudo rm -rf /Library/Bundles/OSXNotification.bundle #not found on any of the workers

I checked the tour plists, and the values set match what I have on my mojave laptop.

I'll next try checking/setting these values for the worker user specifically instead of the system. It may need to be set (less likely for softwareupdate --ignore, but maybe for the noticeboard) per user.

(In reply to Cosmin Sabou [:CosminS] from comment #9)

The actual failure rate here is actual pretty high and it's spread across a few other bugs that fail because of this and are suggested by treeherder.
Example of failure rate: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=os%2Cx%2C10.14%2Cdebug%2Cmochitests%2Ctest-macosx1014-64%2Fdebug-mochitest-browser-chrome-e10s-4%2Cm%28bc4%29&tochange=927abd2c418b308a474ba96e58df4bfdbc6a3ca5&fromchange=18a2e793ef0a0c815158214b86f2c942bda33095&selectedJob=258854373
and this is only for bc failures but it fails also on dt for eg: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258866492&repo=autoland&lineNumber=972
Update: in fact those dt-wr were permafail from bug 1559659 https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=os%2Cx%2C10.14%2Cshippable%2Copt%2Cmochitests%2Ctest-macosx1014-64-shippable%2Fopt-mochitest-devtools-webreplay-e10s%2Cm%28dt-wr%29&tochange=55be6c59850ad8c8497eefa1799826a633f60798&fromchange=b33efbfec65ca65c509806ebf4225b4e4e365cad&selectedJob=258866492

The "welcome to mojave" notification is appearing on all of the mojave workers, but it does not appear always (I think there is a timeout/delay for re-displaying it).

I'll keep looking for a solution (we may just need to hack a click/accept of the welcome notification).

(In reply to Dave House [:dhouse] from comment #12)

I'll keep looking for a solution (we may just need to hack a click/accept of the welcome notification).

I tested with setting the com.apple.touristd entry for the tour as the cltbld user. This changes when I manually view the tour, but there must be another flag being set because when I manually set this without actually viewing the tour, the welcome notification continues appearing.

sudo -u cltbld defaults write com.apple.touristd "seed-viewed-lQlm1yrMS0GPVyAL44id+A" "$(date -v -10d -u "+%F %T %z")"

I also tested with setting the count and due date fields, but the notification still appears with various settings. ex:

sudo -u cltbld defaults write com.apple.touristd "seed-numNotifications-lQlm1yrMS0GPVyAL44id+A" 1
sudo -u cltbld defaults write com.apple.touristd "seed-notificationDueDate-lQlm1yrMS0GPVyAL44id+A" -date "2019-07-26 20:25:46 +0000"

(In reply to Dave House [:dhouse] from comment #13)

(In reply to Dave House [:dhouse] from comment #12)

I'll keep looking for a solution (we may just need to hack a click/accept of the welcome notification).

I tested with setting the com.apple.touristd entry for the tour as the cltbld user. This changes when I manually view the tour, but there must be another flag being set because when I manually set this without actually viewing the tour, the welcome notification continues appearing.

sudo -u cltbld defaults write com.apple.touristd "seed-viewed-lQlm1yrMS0GPVyAL44id+A" "$(date -v -10d -u "+%F %T %z")"

I also tested with setting the count and due date fields, but the notification still appears with various settings. ex:

sudo -u cltbld defaults write com.apple.touristd "seed-numNotifications-lQlm1yrMS0GPVyAL44id+A" 1
sudo -u cltbld defaults write com.apple.touristd "seed-notificationDueDate-lQlm1yrMS0GPVyAL44id+A" -date "2019-07-26 20:25:46 +0000"

When these are manually set, the notification still appears and if I manually accept the tour (clicking the "show" option), the tour is not started (safari does not start). So that appears to be checking these fields for actually starting the tour or not (if we have to script a click of the "show", then that may prevent needing the kill safari afterward).

:jmaher, can we turn off the notification service on macos mojave? Or are some tests checking for firefox notifications?

I checked over build-puppet, and it doesn't look like it was disabled/turned-off for yosemite. On yosemite, we were able to prevent the "welcome to macos" notification, but I have not found how to disable it on mojave yet.
If we can just turn off notifications, then that will be simpler and prevent other unknown ones from interfering with tests.

Flags: needinfo?(jmaher)

(In reply to Mike Conley (:mconley) (:⚙️) from comment #4)

Is it possible that the "Welcome to macOS Mojave" notification is somehow interfering with us here? Is there any way to suppress that?

(I want to point out, btw, that this is speculation, and I'm not 100% certain that suppressing this notification will actually fix the test failures)

I believe we have tests that do notifications and expect them. I believe we have always had some notification on the screen on 10.10.

Flags: needinfo?(jmaher)
Priority: -- → P3

egao :jmaher Here is comparing a 10.10 tour/upgrade notification to the 10.14 one. The new ones are wider I think. So maybe it is overlapping something or similarly causing a problem?

Flags: needinfo?(jmaher)
Flags: needinfo?(egao)

interesting that there were 2 before and now there is 1. your point on the width is interesting and might be valid. In the screenshot from comment 4, we have accessed the menu yet the mouse is in the position where it is hovering over the leftmost side of the notification which is overlapped with the browser.

Maybe we need a script to run as part of a job to click show and then close down any resulting processes?

Flags: needinfo?(jmaher)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #19)

interesting that there were 2 before and now there is 1. your point on the width is interesting and might be valid. In the screenshot from comment 4, we have accessed the menu yet the mouse is in the position where it is hovering over the leftmost side of the notification which is overlapped with the browser.

Maybe we need a script to run as part of a job to click show and then close down any resulting processes?

That would work; the "show" opens Safari with the tour files. So the job could kill the Safari processes and it would be done.

But like :mconley wrote, maybe it is not necessary to stop/end the notification?

(In reply to Mike Conley (:mconley) (:⚙️) (PTO - August 5 - 9) from comment #16)

(In reply to Mike Conley (:mconley) (:⚙️) from comment #4)

Is it possible that the "Welcome to macOS Mojave" notification is somehow interfering with us here? Is there any way to suppress that?

(I want to point out, btw, that this is speculation, and I'm not 100% certain that suppressing this notification will actually fix the test failures)

Do you need anything from me at the moment? I do agree that the wider notification likely overlaps with chrome.

Flags: needinfo?(egao)

(In reply to Edwin Takahashi (:egao, :etakahashi) from comment #22)

Do you need anything from me at the moment? I do agree that the wider notification likely overlaps with chrome.

I guess I have the same question. I wasn't able to get the notification removed automatically (and further discouraged to find we see the same notification on Yosemite/10.10 macos). So, I am expecting that the notification is not the cause of the failures.

:CosminS do you see anything in common between the tests that are failing? I wonder if the notification width is overlapping the Firefox window and interfering with screenshot comparisons(?).

Flags: needinfo?(csabou)

browser-chrome failures for 10.14 might still have a higher failure rate than for 10.10, but it's manageable and the screenshots I checked didn't show the notification anymore.

Status: NEW → RESOLVED
Closed: 11 months ago
Flags: needinfo?(csabou)
Resolution: --- → WORKSFORME

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #24)

browser-chrome failures for 10.14 might still have a higher failure rate than for 10.10, but it's manageable and the screenshots I checked didn't show the notification anymore.

Thanks! The notification comes and goes (timeouts and repeats after a period; same on yosemite). So if it appears in some failures again and is causing problems, we can look at doing something to stop it (scripted click if it is found or maybe we can find what plist/etc controls it).

See Also: → 1572955
You need to log in before you can comment on or make changes to this bug.