1840660 - Run performance tests against safari technology preview

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Kash Shampur [:kshampur] ⌚EST]

With apple silicon coming soon Bug 1808636, we will be running on macOS 13.x so we would be able to run Safari technology preview

Unlike safari, I believe this is updated more regularly whereas safari is usually tied to system updates, and also features future incoming performance improvements

Comment 1

[Jeff Muizelaar [:jrmuizel]]

wpt.fyi runs Safari TP in automation so if we run into issues we can perhaps look there for solutions.

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

One approach is installing it on the M2's and have auto update enabled some how... but I am not sure how automating the auto updates would work as it seems to be suggested in the link You can update Safari Technology Preview right from the Mac App Store. Is there a CLI workaround? Also historically we've had hiccups when it comes to auto updates in general so maybe this isn't reliable...

Another approach is to have this as a cached toolchain or fetch task with a cron running once every 1-2 weeks so that we get the updated version. Through the catalog browser script I think we can easily find the latest TP version. May need an additional script/method to unpack/package it, similar to the mac sdk stuff.

:glandium wondering if you could weigh in on this. Would a toolchain or fetch task be more appropriate for something like this? (or does it even matter if it's a toolchain vs fetch?)

:rcurran would you have thoughts on the auto-update if Safari Technology Preview were to be installed directly on the M2s? And, in either the case of a toolchain/fetch route or auto update route, would the webdriver automation you recently setup with regular Safari on the m2 work with Safari-TP every version update?

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

:jgraham pointed out that the strategy used in wpt.fyi for safari TP (and other browsers) is here
https://searchfox.org/mozilla-central/rev/098f910d0593b12a42089dd8f40dcd19d1121430/testing/web-platform/tests/tools/wpt/browser.py#1859-2092

We can probably leverage this somehow, so probably no need for an additional toolchain/fetch (sorry for the ni? glandium I will remove it now)

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #3)

:jgraham pointed out that the strategy used in wpt.fyi for safari TP (and other browsers) is here
https://searchfox.org/mozilla-central/rev/098f910d0593b12a42089dd8f40dcd19d1121430/testing/web-platform/tests/tools/wpt/browser.py#1859-2092

We can probably leverage this somehow, so probably no need for an additional toolchain/fetch (sorry for the ni? glandium I will remove it now)

Sorry my mistake, :glandium

after re-reviewing the code, it actually might make sense to extract some part of it into a separate script... So back to my previous question, would a toolchain or fetch make more sense?

Comment 5

[Ryan Curran]

We could possibly land the initial pkg with puppet and set the Mac App Store to auto update

Comment 6

[Mike Hommey [:glandium]]

So back to my previous question, would a toolchain or fetch make more sense?

toolchain would be more versatile.

Comment 7

[Kash Shampur [:kshampur] ⌚EST]

Great, thank you both for your answers.
It seems app store update may not be timely, so I will look into a toolchain task

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

Hit a blocker in that browsertime (actually selenium) doesn't support custom binary paths the same way they do for every other browser. It's basically hardcoded to point to /Applications/<...> for Safari and Safari-tp. It may take a while to get that upstream (if at all). I've filed an issue in browsertime, and may have to file a follow up in selenium

And it may not be wise to directly write into the /Applications folder with the toolchain fetch?
In mean time I will look into the alternative of auto update in the appstore as :rcurran suggested.

Comment 9

[Ryan Curran]

Just to add a small data point. It looks like when you install Safari Technology Preview the updates show up in the system software updates vs the App Store. Not sure if that changes anything but I just wanted you to be aware of the location.

Comment 10

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Ryan Curran from comment #9)

Just to add a small data point. It looks like when you install Safari Technology Preview the updates show up in the system software updates vs the App Store. Not sure if that changes anything but I just wanted you to be aware of the location.

ah yes thanks, I noticed this too. If it shows up in the system updates rather than app store, could the update still be automated?

Comment 11

[Ryan Curran]

I am not certain. I think there is a way via the CLI to install ALL system updates however installing a particular update via the CLI would require knowing what label Apple has given the update each time, which could be tricky.

Comment 12

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Ryan Curran from comment #11)

I am not certain. I think there is a way via the CLI to install ALL system updates however installing a particular update via the CLI would require knowing what label Apple has given the update each time, which could be tricky.

not sure if this is the same as the label, but for the hacky toolchain I have currently, I was using the catalog browser link 1 and link 2 (modifying the url to include os 14) to get the product and package id and download url of STP. Not sure of something like that could be used for the label.

Another alternative if that doesnt work is: is it safe to copy the STP folders from the toolchain fetch directly to the /Applications folder, every test run? (tbd if that even works for me locally)

Comment 13

[Kash Shampur [:kshampur] ⌚EST]

making note I've found an additional blocker. currently browsertime isn't able to run safari-tp, it could be an issue on the selenium side with NodeJS, tbd.
I've filed an issue upstream in browsertime to investigate

Comment 14

[Kash Shampur [:kshampur] ⌚EST]

example passing test
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=ac5af6ce4b4e945f5e77f02f37a5ba9bc643951d&selectedTaskRun=Cvuy5tbmTJGqmXhugFIN8Q.0

Ryan had created a script deployed to the machines last week, and we can call this install script every time before the test portion to keep STP up to date.
This circumvents need for dealing with toolchain and permissions for installing said toolchains. And i believe we don't have to worry about 'auto update' since we can call this script directly and keep things up to date? (well, we'll find out next time update roles around!)

Comment 15

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1840660 - Run performance tests against safari technology preview. r?#perftest

Comment 16

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1840660 - Add documentation on running Safari-tp locally. r?#perftest

This patch updates our documents to include a section with instructions
on how a user can run a browsertime or raptor-browsertime test locally
with safari tech preview

Comment 17

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1840660 - Add safari technology preview to try perf selector. r?#perftest

This patch enables a user to select safari technology preview with the
try perf selector by specifying --safari-tp flag.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6d449f6a7644
Run performance tests against safari technology preview. r=perftest-reviewers,sparky
https://hg.mozilla.org/integration/autoland/rev/c2627bc2061e
Add documentation on running Safari-tp locally. r=perftest-reviewers,aglavic
https://hg.mozilla.org/integration/autoland/rev/9a87a991201a
Add safari technology preview to try perf selector. r=perftest-reviewers,sparky

Comment 19

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/6d449f6a7644
https://hg.mozilla.org/mozilla-central/rev/c2627bc2061e
https://hg.mozilla.org/mozilla-central/rev/9a87a991201a