Closed Bug 1746207 Opened 1 year ago Closed 11 months ago

OSX browsertime visual metrics dependencies not installing correctly

Categories

(Testing :: Performance, defect, P1)

Default
defect

Tracking

(firefox100 fixed)

RESOLVED FIXED
100 Branch
Tracking Status
firefox100 --- fixed

People

(Reporter: kimberlythegeek, Assigned: kshampur)

References

Details

Attachments

(1 file)

When running a browsertime test with --browsertime-visualmetrics, the dependencies should be installed but appear to be failing often.

Outcome on osx (unresolved):

  • convert and compare FAIL

ran the following but made no impact, convert and compare still fail:

  • ./mach browsertime --clobber --setup --install-vismet-reqs
  • Tried running mach clobber and mach build and repeating above steps, outcome is the same
Priority: -- → P2

The severity field is not set for this bug.
:kimberlythegeek, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(ksereduck)
Severity: -- → S3
Flags: needinfo?(ksereduck)

I had this issue after updating the dependencies for Apple Silicon support. It was resolved by installing ImageMagick via brew:

brew install imagemagick

I also found this comment, which suggests users should install ImageMagick manually. We should update the documentation to reflect this, and provide this hint when we fail to run Raptor for this reason.

Assignee: nobody → kshampur

mini update on this

ffmpeg:  
OK
convert: 
OK
compare: 
OK
Pillow:  
OK
SSIM:    
OK

managed to get this after running ./mach browsertime --setup --clobber --install-vismet-reqs on an intel mac

Only got this locally by changing pillow version 6 -> 9.0.0, since my local python, 3.9, is not supported by 6.0 LINK

Would it be wise to just have a python version check during the install time and install pillow accordingly?

will update this comment after a test local run with --browsertime-visualmetrics

edit:

also should add, brew installed imagemagick as mentioned above by :davehunt

also m1 seems to be working with --browsertime-visualmetrics but not the intel.
for m1 - (using this patch https://bugzilla.mozilla.org/show_bug.cgi?id=1748821)
for intel - I think it has to do with the numpy/scipy version as well. But i dont think we have x86 latest versions of scipy/numpy right? https://pypi.pub.build.mozilla.org/pub/

alternatively i'll try to use a virtual env with a lower python version and see if even that works as expected

edit 2:
intel mac worked when I used a lower python version (3.7.12)

thanks for this, Kash!

couple of thoughts/suggestions/propositions for some potentially hacky solutions

1

for imagemagick LINK

how would people feel about adding a shutil routine (macos specifically, for now) to basically just call brew install imagemagick since presumably, based on the macos install instructions LINK mac users would likely be using homebrew anyway?

(if yes, i think something similar may need to be done for ffmpeg, will have to double check with a fresh environment)

2

for visual metric packages, LINK 1 , LINK 2

I was thinking we could add some logic here that, IF it is macos, and python version <= 3.7, use version XX package ( for 3.6 and 3.7 should work as is with what is currently listed: ["numpy==1.16.1", "Pillow==6.1.0", "scipy==1.2.3", "pyssim==0.4"])

BUT if python >=3.8, use version's >XX (and this will need a bit of investigating on my part, and also seeing what is, or needs to be added, to https://pypi.pub.build.mozilla.org/pub/, i think?)

3

everything I said above was for intel consideration, but could also be modified a bit to handle m1 like in 1748821, assuming this is even a path we want to take

Flags: needinfo?(ksereduck)
Flags: needinfo?(gmierz2)

Sounds good to me

Flags: needinfo?(ksereduck)

(In reply to Kash [:kshampur] ⌚EST from comment #5)

couple of thoughts/suggestions/propositions for some potentially hacky solutions

1

for imagemagick LINK

how would people feel about adding a shutil routine (macos specifically, for now) to basically just call brew install imagemagick since presumably, based on the macos install instructions LINK mac users would likely be using homebrew anyway?

(if yes, i think something similar may need to be done for ffmpeg, will have to double check with a fresh environment)

That sounds good, but we should make sure that the version is pinned here. Outside of this, you'll also need to consider how this will need to be done/improved for Linux and Windows (next steps).

2

for visual metric packages, LINK 1 , LINK 2

I was thinking we could add some logic here that, IF it is macos, and python version <= 3.7, use version XX package ( for 3.6 and 3.7 should work as is with what is currently listed: ["numpy==1.16.1", "Pillow==6.1.0", "scipy==1.2.3", "pyssim==0.4"])

BUT if python >=3.8, use version's >XX (and this will need a bit of investigating on my part, and also seeing what is, or needs to be added, to https://pypi.pub.build.mozilla.org/pub/, i think?)

Sounds good, and yes, we'll need to add the pre-built wheels to the pypi mirror you linked to.

3

everything I said above was for intel consideration, but could also be modified a bit to handle m1 like in 1748821, assuming this is even a path we want to take

:kshampur++

Flags: needinfo?(gmierz2)
Blocks: 1748821
Depends on: 1757792
Attachment #9265797 - Attachment description: WIP: Bug 1746207 - Fix OSX browsertime visual metrics installation → WIP: Bug 1746207 - Fix OSX browsertime visual metrics installation r=#perftest
Attachment #9265797 - Attachment description: WIP: Bug 1746207 - Fix OSX browsertime visual metrics installation r=#perftest → Bug 1746207 - Fix OSX browsertime visual metrics installation r=#perftest
Attachment #9265797 - Attachment description: Bug 1746207 - Fix OSX browsertime visual metrics installation r=#perftest → WIP: Bug 1746207 - Fix OSX browsertime visual metrics installation r=#perftest
Attachment #9265797 - Attachment description: WIP: Bug 1746207 - Fix OSX browsertime visual metrics installation r=#perftest → Bug 1746207 - Fix OSX browsertime visual metrics installation r=#perftest
Pushed by kshampur@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/670e01af4473
Fix OSX browsertime visual metrics installation r=perftest-reviewers,sparky
Priority: P2 → P1
Status: NEW → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → 100 Branch
You need to log in before you can comment on or make changes to this bug.