[Air Mozilla] Install ffmpeg

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: IT-Managed Tools
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: peterbe, Assigned: gozer)

Tracking

Details

(Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1807] )

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
As of https://bugzilla.mozilla.org/show_bug.cgi?id=1085679 we have code that, on a cron job, periodically queries our videos to figure out how long they are using ffmpeg. The command is something like::

    ffmpeg -i "https://vid.ly/0b3w5j?content=video&format=hd_mp4"

That works here on my OSX where I have ffmpeg installed with these flags: https://gist.github.com/peterbe/d75785d696b88ee60e8d

The code is ready to use it. The code assumes it can simply call out to `ffmpeg` on the command line but if we need to change it to an absolute path you can add something like this to the settings/local.py::

    FFMPEG_LOCATION = '/opt/ffmpeg-1.0/bin/ffmpeg'

Updated

3 years ago
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1710]
to clarify, will this cron be running on the admin node, or webheads?
Flags: needinfo?(peterbe)
(Reporter)

Comment 2

3 years ago
(In reply to Chris Turra [:cturra] from comment #1)
> to clarify, will this cron be running on the admin node, or webheads?

Where all the other cron jobs are run. Is that the admin node?
It won't need to call out to ffmpeg in runtime.
Flags: needinfo?(peterbe)
(Assignee)

Updated

3 years ago
Assignee: server-ops-webops → gozer
(Assignee)

Comment 3

3 years ago
r95208 | cturra@mozilla.com | 2014-10-21 18:40:05 -0400 (Tue, 21 Oct 2014) | 1 line

[root@genericadm.private.phx1 ]# which ffmpeg
/usr/bin/ffmpeg

[root@genericadm.private.phx1 ]# ffmpeg -version
FFmpeg version 0.6.5, Copyright (c) 2000-2010 the FFmpeg developers
  built on Apr  5 2012 10:23:20 with gcc 4.4.5 20110214 (Red Hat 4.4.5-6)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --incdir=/usr/include --disable-avisynth --extra-cflags='-O2 -g -fPIC' --enable-avfilter --enable-avfilter-lavf --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libx264 --enable-gpl --enable-nonfree --enable-postproc --enable-pthreads --enable-shared --enable-swscale --enable-vdpau --enable-version3 --enable-x11grab
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
FFmpeg 0.6.5
libavutil     50.15. 1 / 50.15. 1
libavcodec    52.72. 2 / 52.72. 2
libavformat   52.64. 2 / 52.64. 2
libavdevice   52. 2. 0 / 52. 2. 0
libavfilter    1.19. 0 /  1.19. 0
libswscale     0.11. 0 /  0.11. 0
libpostproc   51. 2. 0 / 51. 2. 0
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 4

3 years ago
Maybe I just don't know what it's called when installed on linux, but can you check that it supports HTTPS?
E.g. 

   ffmpeg -i "https://vid.ly/0b3w5j?content=video&format=hd_mp4"
(Assignee)

Comment 5

3 years ago
Nope, that version isn't built with OpenSSL, but http works

$> ffmpeg -i 'http://vid.ly/0b3w5j?content=video&format=hd_mp4'
FFmpeg version 0.6.5, Copyright (c) 2000-2010 the FFmpeg developers
  built on Apr  5 2012 10:23:20 with gcc 4.4.5 20110214 (Red Hat 4.4.5-6)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --incdir=/usr/include --disable-avisynth --extra-cflags='-O2 -g -fPIC' --enable-avfilter --enable-avfilter-lavf --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libx264 --enable-gpl --enable-nonfree --enable-postproc --enable-pthreads --enable-shared --enable-swscale --enable-vdpau --enable-version3 --enable-x11grab
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1a89670]multiple edit list entries, a/v desync might occur, patch welcome
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'http://vid.ly/0b3w5j?content=video&format=hd_mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf54.29.104
  Duration: 00:38:24.39, start: 0.000000, bitrate: 3625 kb/s
    Stream #0.0(eng): Video: h264, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 3507 kb/s, 30 fps, 30 tbr, 30 tbn, 60 tbc
    Stream #0.1(eng): Audio: aac, 48000 Hz, stereo, s16, 109 kb/s
At least one output file must be specified
(Reporter)

Comment 6

3 years ago
Private events on vid.ly (e.g MoCo meetings) have to be accessed with HTTPS because we put a short term token into the URL. 

How hard would it be?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 7

3 years ago
(In reply to Peter Bengtsson [:peterbe] from comment #6)
> Private events on vid.ly (e.g MoCo meetings) have to be accessed with HTTPS
> because we put a short term token into the URL. 
> 
> How hard would it be?

Well, it would require rebuilding a custom version of ffmpeg, and I'd rather avoid that.

Couldn't you just use curl or similar to download the mp4 file and then do the ffmpeg invocation locally?
(Reporter)

Comment 8

3 years ago
(In reply to Philippe M. Chiasson (:gozer) from comment #7)
> (In reply to Peter Bengtsson [:peterbe] from comment #6)
> > Private events on vid.ly (e.g MoCo meetings) have to be accessed with HTTPS
> > because we put a short term token into the URL. 
> > 
> > How hard would it be?
> 
> Well, it would require rebuilding a custom version of ffmpeg, and I'd rather
> avoid that.
> 
> Couldn't you just use curl or similar to download the mp4 file and then do
> the ffmpeg invocation locally?

Ok. I can try that.

Updated

3 years ago
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1710]

Updated

3 years ago
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1761]

Comment 9

3 years ago
Commit pushed to master at https://github.com/mozilla/airmozilla

https://github.com/mozilla/airmozilla/commit/9f5778e118f1269074ee5b3170d6127170dd1a9e
bug 1087010 - download then run ffmpeg on local disk file
(Assignee)

Comment 10

3 years ago
Great, thanks!
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 11

3 years ago
Created attachment 8513635 [details]
Screenshot 2014-10-29 10.26.27.png

It works! On dev only so far.
(Reporter)

Comment 12

3 years ago
After a long discussion on IRC we concluded that this is not working very well on prod. 
The cron job is really struggling and causes a lot of IO downloading so much. We suspect it gets stuck sometimes causing the cron job to bunch up and ultimately not executing as often as we'd like. 

The reason we switched to first downloading; then doing `ffmpeg -i /local/file.mp4` was because ffmpeg doesn't have SSL support unless custom built. 

Hacking it to only download a partial file could work but not a sustainable solution because we want to expand the current functionality to download screencaptures into PNGs (and possibly animated gifs) and then we're going to need the whole file. If our ffmpeg supports SSL we'll be able to capture screencaptures by URL too. 

In the meantime I'm going to change it so that it does `ffmpeg -i http://...` for non private events (which can be reached over HTTP) and only does download for private events.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 13

3 years ago
Commit pushed to master at https://github.com/mozilla/airmozilla

https://github.com/mozilla/airmozilla/commit/91c5e75b3493a410626295d39d4f75aff86533bc
bug 1087010 - only save private events locally for ffmpeg command
(Reporter)

Comment 14

3 years ago
So last night I optimized (aka. "hacked") the code a bunch. Because ffmpeg can't do SSL we have to download it. However, Vidly allows us to drop the S in https and still can access the public videos. So, what it does now is that it downloads only those videos that are privacy!=public. Two thirds of all videos are public. So it means two thirds *less downloading*. 

Overnight it appears to have done better. There are now only 638 events left to process of the backlog. 

However, some day we'll do this all over again with not just fetching the video duration but also screenshots from the video. 

In other words, I guess, don't kill yourself over making a non-standard package of ffmpeg if it's too much pain, blood and tears.

Updated

3 years ago
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1761]

Updated

3 years ago
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1807]
(Assignee)

Comment 15

3 years ago
Built and deployed a new version of ffmpeg (0.11) on genericadm that has https support, enjoy!

It can be used/found here:

/opt/air.mozilla.org/bin/ffmpeg

Let me know if you encounter problems with it. Just make sure to either fix your PATH or to call *that* ffmpeg directly.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 16

3 years ago
That's great! I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1096570 as a followup. Thank you.
(Reporter)

Updated

3 years ago
Blocks: 1096576
You need to log in before you can comment on or make changes to this bug.