1656963 - macos ronin: install ffmpeg and imagemagick on macos testers

Status: RESOLVED INACTIVE

(Infrastructure & Operations :: RelOps: Posix OS, enhancement)

Comment 1

[:dhouse]

ffmpeg macos static binaries:
https://evermeet.cx/ffmpeg/
Notes "On macOS 10.15 (Catalina) the binary has to be removed from quarantine: xattr -dr com.apple.quarantine <path_to_binary>"

$ curl -JOL https://evermeet.cx/ffmpeg/getrelease/zip
100 22.0M  100 22.0M    0     0  5205k      0  0:00:04  0:00:04 --:--:-- 6499k
curl: Saved to filename 'ffmpeg-4.3.1.zip'
$ unzip ffmpeg-4.3.1.zip
Archive:  ffmpeg-4.3.1.zip
  inflating: ffmpeg                  
$ ./ffmpeg -version
ffmpeg version 4.3.1-tessus  https://evermeet.cx/ffmpeg/  Copyright (c) 2000-2020 the FFmpeg developers
built with Apple clang version 11.0.0 (clang-1100.0.33.17)
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

Comment 2

[:dhouse]

imagemagick binary and setup notes:
https://imagemagick.org/script/download.php#macosx

However, it depends on dylib's for freetype, and possibly X11(for display).
So I may need to compile it and then install freetype libs as part of the pkg.

Comment 3

[Tarek Ziadé (:tarek)]

Thanks. Is there a pool where I can try it?

Comment 4

[:dhouse]

(In reply to Tarek Ziadé (:tarek) from comment #3)

Thanks. Is there a pool where I can try it?

No yet; I set up a static build for it last week and I'll get a test pool set up this week.

Comment 5

[:dhouse]

(In reply to Dave House [:dhouse] from comment #4)

(In reply to Tarek Ziadé (:tarek) from comment #3)

Thanks. Is there a pool where I can try it?

No yet; I set up a static build for it last week and I'll get a test pool set up this week.

In testing the build, I found that I missed some dependencies.
I'm fixing the build and getting a new pkg+pool set up.

Comment 6

[:dhouse]

:tarek I got this worked out. So there is a test pool here: https://firefox-ci-tc.services.mozilla.com/provisioners/releng-hardware/worker-types/gecko-t-osx-1014-bug1656963

The workers there (2 right now) have imagemagick 6.9.11-33 and ffmpeg 4.3.1. I ended up using macports to build a dmg with all of the imagemagick dependencies bundled, and install everything into /usr/local/.

[[e-mail] ~]$ identify --version
Version: ImageMagick 6.9.11-33 Q16 x86_64 2020-10-13 https://imagemagick.org
Copyright: © 1999-2020 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC Modules 
[[e-mail] ~]$ convert logo: logo.png
[[e-mail] ~]$ mogrify -format heic logo.png
[[e-mail] ~]$ identify logo.heic
logo.heic HEIC 640x480 640x480+0+0 8-bit YCbCr 0.000u 0:00.000
[[e-mail] ~]$ ffmpeg -version
ffmpeg version 4.3.1-tessus  https://evermeet.cx/ffmpeg/  Copyright (c) 2000-2020 the FFmpeg developers
built with Apple clang version 11.0.0 (clang-1100.0.33.17)
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

Comment 7

[:dhouse]

:sparky, are the worker-base installs of imagemagick and/or ffmpeg needed?

detail:
I would like to close this bug as wontfix and not install imagemagick specifically and also ffmpeg (if it can be in tooltool/other instead).
ffmpeg: has a static build available and so it is not a problem, but could be in tooltool?
ImageMagick: We removed Homebrew from these workers last year, but Imagemagick's recommended install on MacOS is with Homebrew, and so I had captured the MacPorts install of it as a workaround. But it is large (~2.2gb) and bundles 61 dependencies. I am concerned it will be difficult to maintain because of potential dependency issues, and a non-standard install requiring the 3rd-party macports install script to be up-to-date and for us to manually capture the install and place it into a non-standard location to prevent conflicts.

Comment 8

[Greg Mierzwinski [:sparky]]

Hmm, it's unfortunate that ImageMagick is that complex. But yes, they are still needed since macos is the only platform we can't run visual metrics on in mozperftest CI tests. We also need this for moving raptor-browsertime.

:dhouse, one thing I am wondering is if we would be able to get a container of some sort that has imagemagick and ffmpeg installed in it and use that container within the tests? Maybe this would be more maintainable. FFMPEG would still need to be installed on the worker-base for video recording but ImageMagick wouldn't (it's only needed when we run the visual metric calculations at the end).

Comment 9

[:dhouse]

Thanks for getting back to me so quickly!

I hadn't thought of a container, and the container image itself would be easier to maintain. But that would be running under a vm and require other tools like docker. So I think it would create more complexity until we could have native darwin containers.

My simple-path ideal is to buy-in to using homebrew again and just use the standard expected macos install for imagemagick, or use a static build of a different image utility to provide the same function as imagemagick (I assume it is too slow or adds unnecessary complexity to do the operations in python/etc when imagemagick does it and is available on all platforms).

For now, I'll set up a pool with the MacPorts imagemagick again for testing. I'll NI you for testing when I have that ready (probably next week, but there will be no rush on my side). Maybe when this comes up again, or breaks, we can re-evaluate and consider an alternative install.

Comment 10

[:dhouse]

I've set up a pool releng-hardware/gecko-t-osx-1014-bug1656963 with 10 workers. I'll keep these available for you to test with for about the next month (let me know if you need them for a longer time).

they include ffmpeg 4.3.1 and imagemagick 6.9.11-33:

t-mojave-r7-079.test.releng.mdc2.mozilla.com
ffmpeg version 4.3.1-tessus  https://evermeet.cx/ffmpeg/  Copyright (c) 2000-2020 the FFmpeg developers
built with Apple clang version 11.0.0 (clang-1100.0.33.17)
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100
Version: ImageMagick 6.9.11-33 Q16 x86_64 2020-10-13 https://imagemagick.org
Copyright: © 1999-2020 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC Modules 
Delegates (built-in): bzlib djvu fftw fontconfig freetype gslib heic jbig jng jp2 jpeg lcms ltdl lzma openexr png ps raw tiff webp x xml zlib

Comment 11

[Greg Mierzwinski [:sparky]]

Thanks for getting this setup :dhouse! We might need to keep them up for a little longer but I'll keep you updated. (Leaving the ni? open for myself).

Comment 12

[:dhouse]

I'm leaving the test pool up still, but I'm reducing the worker count to 2 (I can increase it later for testing).

Comment 13

[Andrew Erickson [:aerickson]]

I don't see this pool (releng-hardware/gecko-t-osx-1014-bug1656963) any longer. Closing the bug as it's so old.