Closed Bug 1087010 Opened 10 years ago Closed 10 years ago

[Air Mozilla] Install ffmpeg

Categories

(Infrastructure & Operations :: IT-Managed Tools, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: peterbe, Assigned: gozer)

References

Details

(Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1807] )

Attachments

(1 file)

As of https://bugzilla.mozilla.org/show_bug.cgi?id=1085679 we have code that, on a cron job, periodically queries our videos to figure out how long they are using ffmpeg. The command is something like::

    ffmpeg -i "https://vid.ly/0b3w5j?content=video&format=hd_mp4"

That works here on my OSX where I have ffmpeg installed with these flags: https://gist.github.com/peterbe/d75785d696b88ee60e8d

The code is ready to use it. The code assumes it can simply call out to `ffmpeg` on the command line but if we need to change it to an absolute path you can add something like this to the settings/local.py::

    FFMPEG_LOCATION = '/opt/ffmpeg-1.0/bin/ffmpeg'
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1710]
to clarify, will this cron be running on the admin node, or webheads?
Flags: needinfo?(peterbe)
(In reply to Chris Turra [:cturra] from comment #1)
> to clarify, will this cron be running on the admin node, or webheads?

Where all the other cron jobs are run. Is that the admin node?
It won't need to call out to ffmpeg in runtime.
Flags: needinfo?(peterbe)
Assignee: server-ops-webops → gozer
r95208 | cturra@mozilla.com | 2014-10-21 18:40:05 -0400 (Tue, 21 Oct 2014) | 1 line

[root@genericadm.private.phx1 ]# which ffmpeg
/usr/bin/ffmpeg

[root@genericadm.private.phx1 ]# ffmpeg -version
FFmpeg version 0.6.5, Copyright (c) 2000-2010 the FFmpeg developers
  built on Apr  5 2012 10:23:20 with gcc 4.4.5 20110214 (Red Hat 4.4.5-6)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --incdir=/usr/include --disable-avisynth --extra-cflags='-O2 -g -fPIC' --enable-avfilter --enable-avfilter-lavf --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libx264 --enable-gpl --enable-nonfree --enable-postproc --enable-pthreads --enable-shared --enable-swscale --enable-vdpau --enable-version3 --enable-x11grab
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
FFmpeg 0.6.5
libavutil     50.15. 1 / 50.15. 1
libavcodec    52.72. 2 / 52.72. 2
libavformat   52.64. 2 / 52.64. 2
libavdevice   52. 2. 0 / 52. 2. 0
libavfilter    1.19. 0 /  1.19. 0
libswscale     0.11. 0 /  0.11. 0
libpostproc   51. 2. 0 / 51. 2. 0
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Maybe I just don't know what it's called when installed on linux, but can you check that it supports HTTPS?
E.g. 

   ffmpeg -i "https://vid.ly/0b3w5j?content=video&format=hd_mp4"
Nope, that version isn't built with OpenSSL, but http works

$> ffmpeg -i 'http://vid.ly/0b3w5j?content=video&format=hd_mp4'
FFmpeg version 0.6.5, Copyright (c) 2000-2010 the FFmpeg developers
  built on Apr  5 2012 10:23:20 with gcc 4.4.5 20110214 (Red Hat 4.4.5-6)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --incdir=/usr/include --disable-avisynth --extra-cflags='-O2 -g -fPIC' --enable-avfilter --enable-avfilter-lavf --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libx264 --enable-gpl --enable-nonfree --enable-postproc --enable-pthreads --enable-shared --enable-swscale --enable-vdpau --enable-version3 --enable-x11grab
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1a89670]multiple edit list entries, a/v desync might occur, patch welcome
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'http://vid.ly/0b3w5j?content=video&format=hd_mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf54.29.104
  Duration: 00:38:24.39, start: 0.000000, bitrate: 3625 kb/s
    Stream #0.0(eng): Video: h264, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 3507 kb/s, 30 fps, 30 tbr, 30 tbn, 60 tbc
    Stream #0.1(eng): Audio: aac, 48000 Hz, stereo, s16, 109 kb/s
At least one output file must be specified
Private events on vid.ly (e.g MoCo meetings) have to be accessed with HTTPS because we put a short term token into the URL. 

How hard would it be?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Peter Bengtsson [:peterbe] from comment #6)
> Private events on vid.ly (e.g MoCo meetings) have to be accessed with HTTPS
> because we put a short term token into the URL. 
> 
> How hard would it be?

Well, it would require rebuilding a custom version of ffmpeg, and I'd rather avoid that.

Couldn't you just use curl or similar to download the mp4 file and then do the ffmpeg invocation locally?
(In reply to Philippe M. Chiasson (:gozer) from comment #7)
> (In reply to Peter Bengtsson [:peterbe] from comment #6)
> > Private events on vid.ly (e.g MoCo meetings) have to be accessed with HTTPS
> > because we put a short term token into the URL. 
> > 
> > How hard would it be?
> 
> Well, it would require rebuilding a custom version of ffmpeg, and I'd rather
> avoid that.
> 
> Couldn't you just use curl or similar to download the mp4 file and then do
> the ffmpeg invocation locally?

Ok. I can try that.
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1710]
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1761]
Great, thanks!
Status: REOPENED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
It works! On dev only so far.
After a long discussion on IRC we concluded that this is not working very well on prod. 
The cron job is really struggling and causes a lot of IO downloading so much. We suspect it gets stuck sometimes causing the cron job to bunch up and ultimately not executing as often as we'd like. 

The reason we switched to first downloading; then doing `ffmpeg -i /local/file.mp4` was because ffmpeg doesn't have SSL support unless custom built. 

Hacking it to only download a partial file could work but not a sustainable solution because we want to expand the current functionality to download screencaptures into PNGs (and possibly animated gifs) and then we're going to need the whole file. If our ffmpeg supports SSL we'll be able to capture screencaptures by URL too. 

In the meantime I'm going to change it so that it does `ffmpeg -i http://...` for non private events (which can be reached over HTTP) and only does download for private events.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
So last night I optimized (aka. "hacked") the code a bunch. Because ffmpeg can't do SSL we have to download it. However, Vidly allows us to drop the S in https and still can access the public videos. So, what it does now is that it downloads only those videos that are privacy!=public. Two thirds of all videos are public. So it means two thirds *less downloading*. 

Overnight it appears to have done better. There are now only 638 events left to process of the backlog. 

However, some day we'll do this all over again with not just fetching the video duration but also screenshots from the video. 

In other words, I guess, don't kill yourself over making a non-standard package of ffmpeg if it's too much pain, blood and tears.
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1761]
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1807]
Built and deployed a new version of ffmpeg (0.11) on genericadm that has https support, enjoy!

It can be used/found here:

/opt/air.mozilla.org/bin/ffmpeg

Let me know if you encounter problems with it. Just make sure to either fix your PATH or to call *that* ffmpeg directly.
Status: REOPENED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
That's great! I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1096570 as a followup. Thank you.
Blocks: 1096576
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: