1312923 - Mac OS X 10.10 test machines receive 2 OS popups ("What's new in OS X Yosemite" followed by "Upgrade to macOS Sierra") which might interfere with tests (but probably don't)

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Daniel Holbert [:dholbert]]

Per bug 1261507 comment 7, we've got a test failure (in a rewritten hover/focus-related test) that seems potentially to be caused by some Mac OS X 10.10 "what's new" / "upgrade" popups.

Screenshot: http://mozilla-releng-blobs.s3.amazonaws.com/blobs/try/sha512/ba75bf7ccbd34ba7d3cfba043ba61253c78a844b742c88659f32cf829307e5f766979105bf38146ff2dd8cf35b20e0de94b2ff24f3e375becc586fe14157cc44

For consistency/predicability, it'd be great if we could configure these test boxes so that such popups don't appear. (If nothing else: start the systems from a state in which these popups have already previously appeared & been dismissed in a previous boot of the OS.)

I suspect this will help with bug 1261507 (or at least rule out one possible cause of the test failure there).

Comment 1

[Daniel Holbert [:dholbert]]

catlee, do you know who might be the right person to investigate dismissing/preventing these popups?

Comment 2

[Daniel Holbert [:dholbert]]

(RyanVM is skeptical that these popups have anything to do with the test failure that I'm concerned with, actually; so take comment 0 with a grain of salt.  Still: it'd be great if we could get rid of them, to reduce the sources of confusion & possible-targets-of-blame when we get mysterious mac-specific mochitest failures.)

Comment 3

[Chris AtLee [:catlee]]

Do we know if we need the notification center enabled at all for any of our tests?

Comment 4

[Amy Rich [:arr] [:arich]]

We don't have cycles to work on this, but buildduty has been working on package updates and might.

Comment 5

[Chris Cooper [:coop] (he/him)]

This gets into the kind of open-ended, speculative work that I'd prefer to keep out of buildduty's hands. 

Apple is notoriously poor about documenting this kind of thing, so there may be *weeks* of trial-and-error to find something that works in automation with no guarantees. Buildduty is meant to be handling interrupt work, and this kind of research would take them out of that mode.

We can give this to buildduty to investigate, but I'll put a strict timebox of 2 days on it. If there's no progress after that, we punt.

Alin: let me know when you pick this up so I can start the clock.

Comment 6

[Phil Ringnalda (:philor)]

And then in the first minute of the timer ask Kim, since she already spent pretty close to that long on it while we were standing up some version, possibly 10.8, without any success at getting the .plist entries that are supposed to make it think you've already clicked the "Piss Off" button to stick (and without finding any reason to believe they are causing any harm, since they absolutely do not kill focus, though screenshots make it look like they could hypothetically get in the way of some hypothetical future test that clicks on the hamburger menu).

Comment 7

[Alin Selagea [:aselagea]]

(In reply to Chris Cooper [:coop] from comment #5)

> Alin: let me know when you pick this up so I can start the clock.

Started working on this now.

Comment 8

[Kim Moir [:kmoir] ET]

I've worked on similar things before

See
http://hg.mozilla.org/build/puppet/file/tip/modules/disableservices

I'm not sure where this alert is coming from, perhaps it is in the notification centre
System Preferences -> Notifications 

One thing I have done before is run this from the command line 
defaults read > redirect to a file

change a preference in the system preferences

then defaults read > redirect to a file

and compare the two files to see if there is a difference.  If so, you can use defaults write to change the default in puppet for your mac machine

Or perhaps it is in a plist

look in 
/Library/Preferences for the machine as a whole

or /Users/cltbld/Library/Preferences
for the cltbld users that the tests run as

It's a huge pain.  Especially since these preferences aren't api and Apple changes them every release.  If you get lucky you can find a blog post on how someone else disabled this preference via the command line and then test it out.


Hmm, just found, this seems useful

https://www.jamf.com/jamf-nation/discussions/12694/suppress-yosemite-app-store-notification-via-command

It looks like 

Kims-MacBook-Pro:procedures kmoir$ ssh -l cltbld t-yosemite-r7-0006
Last login: Mon Nov 21 08:02:53 2016
This host is set to follow security level "low"
Unauthorized access prohibited
[[e-mail] ~]$ defaults read com.apple.notificationcenterui 
{
    TodayView =     {
        NoContent =         (
            "com.apple.RemindersNC",
            "com.apple.nc.today.summary"
        );
        order =         (
            "com.apple.iCal.CalendarNC",
            "com.apple.RemindersNC",
            "com.apple.ncplugin.stocks",
            "com.apple.ncplugin.weather",
            "com.apple.ncplugin.calculator",
            "com.apple.ncplugin.WorldClock",
            "com.apple.nc.social",
            "com.apple.iTunes.today.TodayExtension"
        );
        preferences =         {
            "com.apple.nc.disclosures" =             {
                enabled = 1;
            };
            "com.apple.nc.social" =             {
                enabled = 0;
            };
            "com.apple.nc.today.date" =             {
                enabled = 1;
            };
            "com.apple.nc.today.dnd" =             {
                enabled = 1;
            };
            "com.apple.nc.today.summary" =             {
                enabled = 1;
            };
            "com.apple.nc.tomorrow.summary" =             {
                enabled = 1;
            };
        };
    };
    "last-messagetrace-stamp" = "500833499.664254";
}

may have some preferences that need to be disabled so notifications don't appear

Alin: good luck and let me know if you need help

Comment 9

[Alin Selagea [:aselagea]]

Many thanks for the hints Kim, those were really useful.

To sum this up: I spent some time investigating this, tried various solutions but unfortunately I couldn't get rid of those notifications.

For the several t-yosemite-r7 machines that I checked, the only notification that was shown up was the first one: "What's new in OS X Yosemite", while the second one ("Upgrade to macOS Sierra") was missing. 

I followed the suggestions in the link above and looked for "/Library/Preferences/com.apple.noticeboard.plist" which contains only one entry: LastNoticeboardCatalogCheck. Changed the date to be one year in the future and rebooted the machine.
[[e-mail] ~]$ defaults read /Library/Preferences/com.apple.noticeboard.plist
{
    LastNoticeboardCatalogCheck = "2017-11-23 12:08:39 +0000";
}
That didn't had any effect though.

At a closure look at the system preferences, I noticed a preference like:
[[e-mail] ~]$ defaults read /Library/Preferences/com.apple.commerce.plist
{
    AutoUpdateRestartRequired = 0;
    AutoUpdateRestartRequiredMajorOSVersion = "10.10";
    WhatsNewNotificationDate = "2015-10-06 14:55:57 +0000";
    WhatsNewNotificationLatestOS = "10.10.5";
}
For the builder user:
[[e-mail] ~]$ defaults read /Users/cltbld/Library/Preferences/com.apple.commerce.plist
{
    AllowLegacyConversion = 1;
    LastAutoUpdateInvocation = "2016-11-23 12:44:50 +0000";
    NextClientIDPingDate = "2016-11-11 16:29:04 +0000";
    PrimaryAccountMigrated = 1;
    WhatsNewNotificationDate = "2015-10-06 14:55:57 +0000";
    WhatsNewNotificationLatestOS = "10.10.5";
}

Tried updating/removing the above keys in various combinations, yet the notification didn't clear up (even afte reboot).

Next, I looked at the services that start at boot up, investigated which of them could be related to those notifications and disabled them one by one. However, that wasn't useful either. I was even thinking that we may be using some caches for these kinds of notifications, yet they we're still triggered after re-image + cleaning up preferences.

Comment 10

[Alin Selagea [:aselagea]]

@Daniel: how does debugging go? I'm not sure if it's worth spending more time on fixing this, so I'm wondering if you still see these notifications as being related to the test failures mentioned in the bug description.

Comment 11

[Daniel Holbert [:dholbert]]

I haven't been working on debugging this, myself (not sure anyone is at the moment).

Once thing I'm curious about, though -- from your experience, does this notification *always* show up on these test boxes? (during a test run of ~30 minutes)

(The main reason I suspected a connection between the popups and the failures was the fact that 100% of the failure screenshots show the popup.  But I'm not sure if that's *truly* suspicious, or if these popups are just present in every single run.  If they are, then their presence in these failure screenshots wouldn't be so suspicious after all.  RyanVM noted earlier that they're frequent, but I'm wondering if you have more up-to-date and/or definitive information on the omnipresence of these popups.)

Comment 12

[Alin Selagea [:aselagea]]

(In reply to Daniel Holbert [:dholbert] from comment #11)
> I haven't been working on debugging this, myself (not sure anyone is at the
> moment).
> 
> Once thing I'm curious about, though -- from your experience, does this
> notification *always* show up on these test boxes? (during a test run of ~30
> minutes)

By default, these boxes don't have VNC access enabled so I didn't connect to such a machine to see if the notification is present while running a job. 

However, I used two such loaners on which I enabled VNC - what I can say here is that the first notification ("What's new in OS X Yosemite") showed up *every single time*, no matter what changes I did. As mentioned above, the second notification was missing though.

Comment 13

[Alin Selagea [:aselagea]]

Is there any reason to keep this bug opened? Based on the investigation I did here, I don't think these notifications are responsible for the test failure mentioned in description.

Comment 14

[Daniel Holbert [:dholbert]]

I guess not. Clarifying bug summary to indicate that this probably doesn't impact tests. Feel free to WONTFIX.