1162292 - [Air Mozilla] ffmpeg on airmozilla dev always fail

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Peter Bengtsson [:peterbe]]

We're currently writing a new uploader for Air Mozilla. As soon as the user has uploaded the file to S3 we execute 2 ffmpeg commands.

1. Extract duration of the video using ffmpeg
2. Extract 15 screencaptures of the video

This works fine on our server when the video is a transcoded MP4 (transcoded later with Vid.ly) but it doesn't work on .mov files. 

.mov files are very common because that's what people get when they use their iOS devices. 

A command to test is::

  ffmpeg -i https://air-mozilla-uploads-dev.s3.amazonaws.com/2015/05/05/2-abe98-173118.mov 2>&1 | grep Duration

This works on my OSX and spits out::

    Duration: 00:00:29.69, start: 0.000000, bitrate: 17086 kb/s

Comment 1

[Shyam Mani [:fox2mike]]

(In reply to Peter Bengtsson [:peterbe] from comment #0)
> A command to test is::
> 
>   ffmpeg -i
> https://air-mozilla-uploads-dev.s3.amazonaws.com/2015/05/05/2-abe98-173118.
> mov 2>&1 | grep Duration
> 
> This works on my OSX and spits out::
> 
>     Duration: 00:00:29.69, start: 0.000000, bitrate: 17086 kb/s

Which version of ffmpeg are you running?

Seems like we have a really old one on genericadm and that can't grab files (say like over http) unless they're local...

[[e-mail]1 ~]$ ffmpeg -i https://air-mozilla-uploads-dev.s3.amazonaws.com/2015/05/05/2-abe98-173118.mov
FFmpeg version 0.6.5, Copyright (c) 2000-2010 the FFmpeg developers
  built on Apr  5 2012 10:23:20 with gcc 4.4.5 20110214 (Red Hat 4.4.5-6)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --incdir=/usr/include --disable-avisynth --extra-cflags='-O2 -g -fPIC' --enable-avfilter --enable-avfilter-lavf --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libx264 --enable-gpl --enable-nonfree --enable-postproc --enable-pthreads --enable-shared --enable-swscale --enable-vdpau --enable-version3 --enable-x11grab
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
https://air-mozilla-uploads-dev.s3.amazonaws.com/2015/05/05/2-abe98-173118.mov: No such file or directory

[[e-mail]1 ~]$ wget https://air-mozilla-uploads-dev.s3.amazonaws.com/2015/05/05/2-abe98-173118.mov
--2015-05-06 21:00:26--  https://air-mozilla-uploads-dev.s3.amazonaws.com/2015/05/05/2-abe98-173118.mov
Resolving air-mozilla-uploads-dev.s3.amazonaws.com... 54.231.65.89
Connecting to air-mozilla-uploads-dev.s3.amazonaws.com|54.231.65.89|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 63402085 (60M) [video/quicktime]
Saving to: “2-abe98-173118.mov”

100%[=============================================================================================>] 63,402,085  24.9M/s   in 2.4s

2015-05-06 21:00:29 (24.9 MB/s) - “2-abe98-173118.mov” saved [63402085/63402085]

[[e-mail]1 ~]$ ffmpeg -i 2-abe98-173118.mov 2>&1 | grep Duration
  Duration: 00:00:29.74, start: 0.000000, bitrate: 17052 kb/s

Comment 2

[Peter Bengtsson [:peterbe]]

That's not the one gozer compiled. 
A couple of months ago he compiled a very recent version of ffmpeg dedicated for airmozilla. 
I can't tell you where it is but if you look in the local.py for airmozilla/settings/ it'll be under FFMPEG_LOCATION.

Comment 3

[Peter Bengtsson [:peterbe]]

ffmpeg is a beast but one way we can make progress on this is possibly to do: ``path/to/compiled/ffmpeg -codecs > /tmp/allcodes.txt`` and upload that file here. I'll then compare that with what I have on my homebrew installed version and perhaps that can give us some clues. 

Without this fixed, we will have a very stunted user experience for people who upload videos taken with their iOS devices.

Comment 4

[Peter Bengtsson [:peterbe]]

I have expanded my python calls to the subprocess so I now get the OSError *and* the command it attempted. This is what I just got:

OSError: ("[Errno 2] No such file or directory {command: u'/opt/air.mozilla.org/bin/ffmpeg -i https://air-mozilla-uploads-dev.s3.amazonaws.com/2014/12/17/2-f0ea0-225317.mov'}",)


It's strange that it would say that error. As if `/opt/air.mozilla.org/bin/ffmpeg` doesn't exist as a file?!

Any idea Shyam?

Comment 5

[Peter Bengtsson [:peterbe]]

Hmm... Maybe it's not related to .mov files. I got this too:

OSError: ("[Errno 2] No such file or directory {command: u'/opt/air.mozilla.org/bin/ffmpeg -i https://air-mozilla-uploads-dev.s3.amazonaws.com/2015/05/13/2-edaba-203332.mp4'}",)

Comment 6

[Peter Bengtsson [:peterbe]]

Equally here, it's a file from Vid.ly:


OSError: ("[Errno 2] No such file or directory {command: '/opt/air.mozilla.org/bin/ffmpeg -i https://d3fenhwk93s16g.cloudfront.net/f1y1r2/hd_mp4.mp4?t=14315489565553b41cb2ef2'}",)


I suspect that the executable /opt/air.mozilla.org/bin/ffmpeg simply doesn't exist on the webhead of air-dev.allizom.org and this has nothing to do with the file being a .mov.

Comment 7

[Peter Bengtsson [:peterbe]]

Hmm... Here's another clue. It turns out that when the **cron job** executes the relevant code that processes the video URL it DOES work. E.g. this line actually worked and produced something:

/opt/air.mozilla.org/bin/ffmpeg -ss 00:00:00 -i http://d3fenhwk93s16g.cloudfront.net/f1y1r2/hd_mp4.mp4?t=14315529645553c3c4873ae

My guess is that the cron job as run as a different user compared to the user that's running through mod_wsgi.

Comment 8

[Shyam Mani [:fox2mike]]

So, under /opt we have :

[[e-mail]1 ~]$ /opt/air.mozilla.org/bin/ffmpeg -v
ffmpeg version 2.5 Copyright (c) 2000-2014 the FFmpeg developers
  built on Dec 11 2014 18:53:32 with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-4)

And :

[[e-mail]1 ~]$ /opt/air.mozilla.org/bin/ffmpeg -i https://air-mozilla-uploads-dev.s3.amazonaws.com/2015/05/05/2-abe98-173118.mov 2>&1 | grep Duration

  Duration: 00:00:29.69, start: 0.000000, bitrate: 17086 kb/s

(In reply to Peter Bengtsson [:peterbe] from comment #6)
> OSError: ("[Errno 2] No such file or directory {command:
> '/opt/air.mozilla.org/bin/ffmpeg -i
> https://d3fenhwk93s16g.cloudfront.net/f1y1r2/hd_mp4.
> mp4?t=14315489565553b41cb2ef2'}",)
 
> I suspect that the executable /opt/air.mozilla.org/bin/ffmpeg simply doesn't
> exist on the webhead of air-dev.allizom.org and this has nothing to do with
> the file being a .mov.

And yes, this makes sense. This is only on the admin host, not on the webheads.

Comment 9

[Peter Bengtsson [:peterbe]]

(In reply to Shyam Mani [:fox2mike] from comment #8)
> 
> And yes, this makes sense. This is only on the admin host, not on the
> webheads.

Ah. Different filesystems. So, what would it take to get the webheads access to ffmpeg too?

Comment 10

[Shyam Mani [:fox2mike]]

(In reply to Peter Bengtsson [:peterbe] from comment #9)

> Ah. Different filesystems. So, what would it take to get the webheads access
> to ffmpeg too?

sysadmins r104470 has the commit in puppet to make that happen. This should work once it's pushed out (next 30-60 mins).

Closing, please re-open if it fails after that.