1922537 - vpl-av1, vpl-h264, and vpl-vp9 tasks fail to run on intel macosx 14.70

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=3855435465136341f1d5c6d31e19d2a903fe4038&searchStr=browsertime%2Cvpl

I am not familiar with these jobs, but in looking at upgrading to this new OS version from OSX 10.15, I find [only] 3 perf related tasks that seem to fail consistently.

To run tests on here (at least the month of October) you can use:
./mach try fuzzy --worker-override="t-osx-1015-r8=releng-hardware/gecko-t-osx-1400-r8-staging" -q 'test-macosx1015 browsertime vpl'

Comment 1

[Greg Mierzwinski [:sparky]]

:aosmond, do you think you could look into this? These are the video-playback-latency tests failing.

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Necessary permissions update

From my local testing, I believe we need to reconfigure the OSX machines to allow screen recording from the terminal, see my attached screenshot. I have the same failures on my local machine when I remove that permission.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Ryan- is this something you can adjust to make it work like 10.15 ? not sure if something changed permissions/toolwise on 14.70

Comment 4

[Ryan Curran]

Attachment: Screenshot 2024-10-09 at 12.38.23 PM.png

Comment 5

[Ryan Curran]

Joel- it looks like Terminal/bash have the same screen recording permissions on macOS 14 set as they do on 10.15 (see attached screen shot)

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

thanks Ryan and :aosmond. I am not sure what else to look at here- Do we need screen recording for Firefox?

Comment 7

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #6)

thanks Ryan and :aosmond. I am not sure what else to look at here- Do we need screen recording for Firefox?

No, it is ffmpeg running in the terminal (I assume it is the same in CI), not Firefox doing the recording. I'm going to try running with the CI version of ffmpeg locally and see what I get. I have a much newer version.

Comment 8

[Greg Mierzwinski [:sparky]]

Would the test work without using ffmpeg to do the screenrecording? We use the Firefox Window Recorder in our browsertime macosx tests, and they all pass. You could check by removing this line: https://searchfox.org/mozilla-central/source/taskcluster/kinds/test/browsertime-desktop.yml#888

Locally, if you don't have the --browsertime-no-ffwindowrecorder option, then it'll default to using the Firefox Window Recorder.

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1922537 - Match video playback latency recording configuration to match other browsertime tests.

Switch to only using ffmpeg to record on Windows like browsertime-tp6,
browsertime-responsiveness, etc. This will unblock testing for Firefox
on new OSX as well as being more consistent with how we have configured
other browsertest test suites.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cb7b0e609825
Match video playback latency recording configuration to match other browsertime tests. r=sparky

Comment 11

[amarc]

https://hg.mozilla.org/mozilla-central/rev/cb7b0e609825