766092 - High failure rate for videos at http://videos-cdn.mozilla.net

Status: RESOLVED INVALID

(mozilla.org Graveyard :: Server Operations, task)

Description

[David Humphrey (:humph)]

For the Popcorn Maker launch happening this week, we're using a half-dozen videos hosted on http://videos-cdn.mozilla.net, specifically:

http://videos-cdn.mozilla.net/serv/webmademovies/makecorn.mp4
http://videos-cdn.mozilla.net/serv/webmademovies/makecorn.webm
http://videos-cdn.mozilla.net/serv/webmademovies/robots.mp4
http://videos-cdn.mozilla.net/serv/webmademovies/robots.webm
http://videos-cdn.mozilla.net/serv/webmademovies/studentresponse.mp4
http://videos-cdn.mozilla.net/serv/webmademovies/studentresponse.webm
http://videos-origin.mozilla.org/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4
http://videos-origin.mozilla.org/serv/webmademovies/Moz_Doc_0329_GetInvolved.ogv
http://videos-origin.mozilla.org/serv/webmademovies/Moz_Doc_0329_GetInvolved.webm

We're seeing a high rate of failure trying to load these, whether in our app (which loads them via <video>) or linking directly.  The problem has been reported more highly by users of Chrome, though I'm not convinced that this indicates anything other than the browser they are using.

One that often seems to fail is:

http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4

I'm not sure how to give better data, and perhaps you have thoughts.  We'd love to be able to leave these videos on the CDN due to traffic, but it's currently pretty unreliable.

Comment 1

[David Humphrey (:humph)]

Here's a reduced test case of our app, where I load the video and wait for the first setup event (loadedmetadata).

http://rome.proximity.on.ca/~dave/video-cdn.html

Comment 2

[David Humphrey (:humph)]

NOTE: I copied some videos-origin URLs above, but those are not being used (we only link to -cdn).

Comment 3

[David Humphrey (:humph)]

Here's what I see in Chrome when it fails (in the Network tab of Dev Tools), with the following URL: http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4

Request:
--------
GET /serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4 HTTP/1.1
Host: videos-cdn.mozilla.net
Connection: keep-alive
Cache-Control: no-cache
Pragma: no-cache
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.53 Safari/536.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3


Response:
---------
Access-Control-Allow-Origin: *
Content-Type: video/mp4
Date: Tue, 19 Jun 2012 14:40:15 GMT
ETag: "16b9695"
Last-Modified: Thu, 31 Mar 2011 17:47:03 GMT
Server: ECD (jfk/4DA3)
X-Backend-Server: videos-origin.webapp.scl3.mozilla.com
X-Cache: HIT
Content-Length: 23828117

I'll post a screenshot too.

Comment 4

[David Humphrey (:humph)]

Attachment: Failed load of mp4 from videos-cdn in Chrome

Screenshot of what I'm seeing in Chrome on fail

Comment 5

[David Humphrey (:humph)]

Not sure that this is useful, but more network info from Chrome on fail (via chrome://net-internals):

271996: URL_REQUEST
http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4
Start Time: 2012-06-19 11:25:17.749

t=1340119517749 [st= 0] +REQUEST_ALIVE  [dt=66]
t=1340119517749 [st= 0]    URL_REQUEST_START_JOB  [dt=0]
                           --> load_flags = 68223104 (ENABLE_LOAD_TIMING | MAIN_FRAME | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                           --> method = "GET"
                           --> priority = 0
                           --> url = "http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4"
t=1340119517749 [st= 0]   +URL_REQUEST_START_JOB  [dt=33]
                           --> load_flags = 68223104 (ENABLE_LOAD_TIMING | MAIN_FRAME | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                           --> method = "GET"
                           --> priority = 0
                           --> url = "http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4"
t=1340119517749 [st= 0]      HTTP_CACHE_GET_BACKEND  [dt=0]
t=1340119517749 [st= 0]      HTTP_CACHE_OPEN_ENTRY  [dt=1]
t=1340119517750 [st= 1]      HTTP_CACHE_ADD_TO_ENTRY  [dt=0]
t=1340119517750 [st= 1]      HTTP_CACHE_READ_INFO  [dt=0]
t=1340119517750 [st= 1]     +HTTP_STREAM_REQUEST  [dt=1]
t=1340119517751 [st= 2]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                               --> source_dependency = 271998 (HTTP_STREAM_JOB)
t=1340119517751 [st= 2]     -HTTP_STREAM_REQUEST
t=1340119517751 [st= 2]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]
t=1340119517751 [st= 2]        HTTP_TRANSACTION_SEND_REQUEST_HEADERS
                               --> GET /serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4 HTTP/1.1
                                   Host: videos-cdn.mozilla.net
                                   Connection: keep-alive
                                   User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.53 Safari/536.5
                                   Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                                   Accept-Encoding: gzip,deflate,sdch
                                   Accept-Language: en-US,en;q=0.8
                                   Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
                                   Range: bytes=5441-5441
                                   If-Range: "16b9695"
t=1340119517751 [st= 2]     -HTTP_TRANSACTION_SEND_REQUEST
t=1340119517751 [st= 2]     +HTTP_TRANSACTION_READ_HEADERS  [dt=30]
t=1340119517751 [st= 2]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=30]
t=1340119517781 [st=32]        HTTP_TRANSACTION_READ_RESPONSE_HEADERS
                               --> HTTP/1.1 206 Partial Content
                                   Accept-Ranges: bytes
                                   Access-Control-Allow-Origin: *
                                   Content-Range: bytes 5441-5441/23828117
                                   Content-Type: video/mp4
                                   Date: Tue, 19 Jun 2012 15:25:17 GMT
                                   ETag: "16b9695"
                                   Last-Modified: Thu, 31 Mar 2011 17:47:03 GMT
                                   Server: ECD (jfk/4DA3)
                                   X-Backend-Server: videos-origin.webapp.scl3.mozilla.com
                                   X-Cache: HIT
                                   Content-Length: 1
t=1340119517781 [st=32]     -HTTP_TRANSACTION_READ_HEADERS
t=1340119517781 [st=32]      HTTP_CACHE_WRITE_INFO  [dt=1]
t=1340119517782 [st=33]   -URL_REQUEST_START_JOB
t=1340119517786 [st=37]    HTTP_CACHE_READ_DATA  [dt=1]
t=1340119517787 [st=38]    HTTP_CACHE_READ_DATA  [dt=0]
t=1340119517787 [st=38]   +HTTP_STREAM_REQUEST  [dt=2]
t=1340119517789 [st=40]      HTTP_STREAM_REQUEST_BOUND_TO_JOB
                             --> source_dependency = 272000 (HTTP_STREAM_JOB)
t=1340119517789 [st=40]   -HTTP_STREAM_REQUEST
t=1340119517789 [st=40]   +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]
t=1340119517789 [st=40]      HTTP_TRANSACTION_SEND_REQUEST_HEADERS
                             --> GET /serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4 HTTP/1.1
                                 Host: videos-cdn.mozilla.net
                                 Connection: keep-alive
                                 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.53 Safari/536.5
                                 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
                                 Accept-Encoding: gzip,deflate,sdch
                                 Accept-Language: en-US,en;q=0.8
                                 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
                                 Range: bytes=5441-23828116
                                 If-Range: "16b9695"
t=1340119517789 [st=40]   -HTTP_TRANSACTION_SEND_REQUEST
t=1340119517789 [st=40]   +HTTP_TRANSACTION_READ_HEADERS  [dt=26]
t=1340119517789 [st=40]     +HTTP_STREAM_PARSER_READ_HEADERS  [dt=26]
t=1340119517815 [st=66]        HTTP_CACHE_WRITE_INFO  [dt=0]
t=1340119517815 [st=66] -REQUEST_ALIVE
271997: DISK_CACHE_ENTRY
http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4
Start Time: 2012-06-19 11:25:17.750

t=1340119517750 [st= 0] +DISK_CACHE_ENTRY_IMPL  [dt=65]
                         --> created = false
                         --> key = "http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4"
t=1340119517750 [st= 0]   +ENTRY_READ_DATA  [dt=0]
                           --> buf_len = 354
                           --> index = 0
                           --> offset = 0
t=1340119517750 [st= 0]   -ENTRY_READ_DATA
                           --> bytes_copied = 354
t=1340119517781 [st=31]   +ENTRY_WRITE_DATA  [dt=1]
                           --> buf_len = 354
                           --> index = 0
                           --> offset = 0
                           --> truncate = true
t=1340119517782 [st=32]   -ENTRY_WRITE_DATA
                           --> bytes_copied = 354
t=1340119517786 [st=36]   +ENTRY_READ_DATA  [dt=1]
                           --> buf_len = 5441
                           --> index = 1
                           --> offset = 0
t=1340119517787 [st=37]   -ENTRY_READ_DATA
                           --> bytes_copied = 5441
t=1340119517815 [st=65]   +ENTRY_WRITE_DATA  [dt=0]
                           --> buf_len = 354
                           --> index = 0
                           --> offset = 0
                           --> truncate = true
t=1340119517815 [st=65]   -ENTRY_WRITE_DATA
                           --> bytes_copied = 354
t=1340119517815 [st=65]    ENTRY_CLOSE
t=1340119517815 [st=65] -DISK_CACHE_ENTRY_IMPL
272002: URL_REQUEST
http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4
Start Time: 2012-06-19 11:25:17.817

t=1340119517817 [st=0] +REQUEST_ALIVE  [dt=?]
t=1340119517817 [st=0]   +URL_REQUEST_START_JOB  [dt=?]
                          --> load_flags = 67174528 (ENABLE_LOAD_TIMING | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                          --> method = "GET"
                          --> priority = 2
                          --> url = "http://videos-cdn.mozilla.net/serv/webmademovies/Moz_Doc_0329_GetInvolved.mp4"
t=1340119517817 [st=0]      HTTP_CACHE_GET_BACKEND  [dt=0]
t=1340119517817 [st=0]      HTTP_CACHE_OPEN_ENTRY  [dt=0]
t=1340119517817 [st=0]     +HTTP_CACHE_ADD_TO_ENTRY  [dt=?]

Comment 6

[David Humphrey (:humph)]

After more debugging, this turns out to be a Chrome bug, http://code.google.com/p/chromium/issues/detail?id=31014.  The browser will only let you have a single video (keyed on url) load at once, and a subsequent request to load that same resource will get stopped.  If I load foo.com/video.mp4 and foo.com/video.mp4?random it will work, since URLs are different.

Closing.