1746207 - OSX browsertime visual metrics dependencies not installing correctly

Status: RESOLVED FIXED

(Testing :: Performance, defect, P1)

Description

[Kimberly Sereduck :kimberlythegeek]

When running a browsertime test with --browsertime-visualmetrics, the dependencies should be installed but appear to be failing often.

Outcome on osx (unresolved):

• convert and compare FAIL

ran the following but made no impact, convert and compare still fail:

• ./mach browsertime --clobber --setup --install-vismet-reqs
• Tried running mach clobber and mach build and repeating above steps, outcome is the same

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:kimberlythegeek, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I had this issue after updating the dependencies for Apple Silicon support. It was resolved by installing ImageMagick via brew:

brew install imagemagick

I also found this comment, which suggests users should install ImageMagick manually. We should update the documentation to reflect this, and provide this hint when we fail to run Raptor for this reason.

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

mini update on this

ffmpeg:  
OK
convert: 
OK
compare: 
OK
Pillow:  
OK
SSIM:    
OK

managed to get this after running ./mach browsertime --setup --clobber --install-vismet-reqs on an intel mac

Only got this locally by changing pillow version 6 -> 9.0.0, since my local python, 3.9, is not supported by 6.0 LINK

Would it be wise to just have a python version check during the install time and install pillow accordingly?

will update this comment after a test local run with --browsertime-visualmetrics

edit:

also should add, brew installed imagemagick as mentioned above by :davehunt

also m1 seems to be working with --browsertime-visualmetrics but not the intel.
for m1 - (using this patch https://bugzilla.mozilla.org/show_bug.cgi?id=1748821)
for intel - I think it has to do with the numpy/scipy version as well. But i dont think we have x86 latest versions of scipy/numpy right? https://pypi.pub.build.mozilla.org/pub/

alternatively i'll try to use a virtual env with a lower python version and see if even that works as expected

edit 2:
intel mac worked when I used a lower python version (3.7.12)

Comment 4

[Kimberly Sereduck :kimberlythegeek]

thanks for this, Kash!

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

couple of thoughts/suggestions/propositions for some potentially hacky solutions

1

for imagemagick LINK

how would people feel about adding a shutil routine (macos specifically, for now) to basically just call brew install imagemagick since presumably, based on the macos install instructions LINK mac users would likely be using homebrew anyway?

(if yes, i think something similar may need to be done for ffmpeg, will have to double check with a fresh environment)

2

for visual metric packages, LINK 1 , LINK 2

I was thinking we could add some logic here that, IF it is macos, and python version <= 3.7, use version XX package ( for 3.6 and 3.7 should work as is with what is currently listed: ["numpy==1.16.1", "Pillow==6.1.0", "scipy==1.2.3", "pyssim==0.4"])

BUT if python >=3.8, use version's >XX (and this will need a bit of investigating on my part, and also seeing what is, or needs to be added, to https://pypi.pub.build.mozilla.org/pub/, i think?)

3

everything I said above was for intel consideration, but could also be modified a bit to handle m1 like in 1748821, assuming this is even a path we want to take

Comment 6

[Kimberly Sereduck :kimberlythegeek]

Sounds good to me

Comment 7

[Greg Mierzwinski [:sparky]]

(In reply to Kash [:kshampur] ⌚EST from comment #5)

couple of thoughts/suggestions/propositions for some potentially hacky solutions

1

for imagemagick LINK

how would people feel about adding a shutil routine (macos specifically, for now) to basically just call brew install imagemagick since presumably, based on the macos install instructions LINK mac users would likely be using homebrew anyway?

(if yes, i think something similar may need to be done for ffmpeg, will have to double check with a fresh environment)

That sounds good, but we should make sure that the version is pinned here. Outside of this, you'll also need to consider how this will need to be done/improved for Linux and Windows (next steps).

2

for visual metric packages, LINK 1 , LINK 2

I was thinking we could add some logic here that, IF it is macos, and python version <= 3.7, use version XX package ( for 3.6 and 3.7 should work as is with what is currently listed: ["numpy==1.16.1", "Pillow==6.1.0", "scipy==1.2.3", "pyssim==0.4"])

BUT if python >=3.8, use version's >XX (and this will need a bit of investigating on my part, and also seeing what is, or needs to be added, to https://pypi.pub.build.mozilla.org/pub/, i think?)

Sounds good, and yes, we'll need to add the pre-built wheels to the pypi mirror you linked to.

3

everything I said above was for intel consideration, but could also be modified a bit to handle m1 like in 1748821, assuming this is even a path we want to take

:kshampur++

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1746207 - Fix OSX browsertime visual metrics installation r=#perftest

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/670e01af4473
Fix OSX browsertime visual metrics installation r=perftest-reviewers,sparky

Comment 10

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/670e01af4473