1848400 - Upgrade OSX M2s from 13 to 14 to resolve safari permafails

Status: RESOLVED FIXED

(Testing :: Performance, enhancement, P2)

Description

[Andrej (:aglavic)]

Safari does not have WebDriver support for osx 13 this results in permafails when running speedometer and spedometer3

This bug is for re-enabling safari tests on osx1300 when WebDriver is enabled

Comment 1

[Andrej (:aglavic)]

Waiting to get webdriver support here: https://mozilla-hub.atlassian.net/browse/RELOPS-644

Comment 3

[Andrej (:aglavic)]

Relops came back with a solution to get safari working but a try push was unable to start safari, so we're blocked while they take a look and re-analyze

Comment 4

[Andrej (:aglavic)]

Update that I am still working with relops to get a fix for this

Comment 5

[Andrej (:aglavic)]

I am working on this now, I am applying the changes made in bug 1858236 to see if using the intel version of node was causing these perma fails

Comment 6

[Andrej (:aglavic)]

Using the intel versions did not resolve the issues on try at least
I ran the test using moz-unified and could not reproduce the issue, however upon downloading the CI way I was able to reproduce which is good news!

Comment 7

[Andrej (:aglavic)]

Safari perma fails appear to be an issue with the safaridriver permissions.
We deduced this by running safaridriver --diagnose in the code and kept running into an error saying that safaridriver was not enabled
We have since passed this on to releng to resolve the safaridriver setup issue

Comment 8

[Greg Mierzwinski [:sparky]]

:aglavic, have there been any updates on this issue?

Comment 9

[Andrej (:aglavic)]

Bug has been passed to back to relops as it appears to be a permission issue with safaridriver not being enabled(see my update in the jira bug)
We are blocked on this still, no updates as of yet from them

Comment 10

[Andrej (:aglavic)]

Relops has been looking into the reason they believe that the safaridriver has not been enabled
They have come back with a solution that appears to be working.
What we discovered was that when rebooting the machine it was discovered that the safaridriver was not enabled, they have put in a fix for the m2-1 that they believe will resolve the issue(it looks promising). I am testing it today and will have an update later tonight/early tomorrow morning.
They have closed the issue on jira and I will re-open the issue if the solution does not work

Comment 11

[Andrej (:aglavic)]

Relops has come back with a newer solution and has verified that from their end that safaridriver should now be working
I am meeting with Sparky today and we're going to dig into the issue and figure out if the solution is working for us, and what I need to do as next steps

Comment 12

[Andrej (:aglavic)]

Attachment: Bug 1848400 - Upgrade M2s to use OSX 14 instead of OSX 13. r?#perftest

During the enabling of safaridriver we ran into user permission issues and as a result were unable to enable run safari tests on M2s
It was decided to upgrade the machines from OSX 13 to OSX 14 to resolve the issue and give us a more modern OS to test speedometer against in general

Comment 13

[Andrej (:aglavic)]

FYI: we upgraded the M2s from 13 to 14 and got successful runs for both speedometer and speedometer3 for Firefox, Chrome, and Safari
Our next steps are going to be upgrading the pool of M2s running OS 13 to OS 14.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a700ab961a22
Upgrade M2s to use OSX 14 instead of OSX 13. r=perftest-reviewers,sparky

Comment 15

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/a700ab961a22

Comment 16

[Andrej (:aglavic)]

[Tracking Requested - why for this release]:
We want to get speedometer numbers for OSX 14 with M2 chips so I am requesting a beta uplift
Adding a need info to :dmeehan, as I am not sure if I did this correctly and we'd like to have this uplifted sooner rather than later

Comment 17

[Donal Meehan [:dmeehan]]

:aglavic uplift requests need to be added to the patch itself. Either as a comment on the attachment here OR through Phabricator,
For more information please see instructions https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift.

This landed in central during the 125 nightly cycle, so not sure why you set status-firefox125 to affected. I'll set it back now to fixed.
Next week is RC week for 124, the final 124 beta builds tomorrow.
125 rides the train to beta on 2024-03-18, this change will hit beta then eitherway.

Comment 18

[Andrej (:aglavic)]

Okay, I didn't realize that the uplift was only 2 weeks away we can wait until then
Sorry for all the headaches and yes I mis-counted and added the 125 incorrecrlt thank you for fixing that

Comment 19

[Andrej (:aglavic)]

Attachment: Bug 1848400 - Upgrade M2s to use OSX 14 instead of OSX 13. r?#perftest

During the enabling of safaridriver we ran into user permission issues and as a result were unable to enable run safari tests on M2s
The issue we were encountering was kernel: (Sandbox) Sandbox: com.apple.WebDriver.HTTPService(30714) deny(1) network-bind local:*:7055.
This issue was precenting from webdriver from starting and the permission issue was caused by running a taskcluster-worker as system daemon vs. a per-user launch agent.

The issue was specific to OSX 13, and so it was decided to upgrade the machines from OSX 13 to OSX 14 to resolve the issue and give us a more modern OS to test speedometer against in general

Original Revision: https://phabricator.services.mozilla.com/D202503

Comment 20

[Phabricator Automation]

Uplift Approval Request

• Needs manual QE test: no
• Explanation of risk level: This fix would give us data into the M2 apple chips, if something were to go wrong it would affect a teir 2 test that is not giving any value
• Code covered by automated testing: yes
• Risk associated with taking this patch: low
• Steps to reproduce for manual QE testing: N/A
• User impact if declined: None
• Is Android affected?: no
• String changes made/needed: None
• Fix verified in Nightly: yes

Comment 21

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/6b7825536578