Closed
Bug 1214971
Opened 9 years ago
Closed 9 years ago
delivery: download errors for firefox files on http://ftp.m.o
Categories
(Cloud Services :: Operations: Miscellaneous, task)
Cloud Services
Operations: Miscellaneous
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: nthomas, Assigned: oremj)
References
Details
Attachments
(1 file)
1.28 KB,
text/plain
|
Details |
We've hit three download errors while working on Firefox 41.0.2, so filing in case it's a canary in the traffic mines.
eg:
Downloading http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/39.0-candidates/build6/update/linux-i686/pa-IN/firefox-39.0.complete.mar to firefox-39.0.pa-IN.complete.mar
retry: Calling <function downloadUpdateIgnore404 at 0x2bb19b0> with args: ('ftp.mozilla.org', 'firefox', '39.0', 6, 'linux', 'pa-IN'), kwargs: {}, attempt #2
Downloading http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/39.0-candidates/build6/update/linux-i686/pa-IN/firefox-39.0.complete.mar to firefox-39.0.pa-IN.complete.mar
The retry indicates an error occurred, although we don't have debug logging enabled so don't have any specifics. The second request succeeded, but returned 17235968 bytes instead of 48374718. The file has a modification time of 17:45 Pacific today; the request came from an AWS instance in us-east-1.
The other two were
* http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/update/linux-x86_64/bs/firefox-41.0.1.complete.mar, returning 9617408 of 49031170 at 17:53.
* http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/update/linux-x86_64/en-ZA/firefox-40.0.3.complete.mar, returning 17530880 of 48016923 at 17:32.
Do we have any logs at the nginx or CDN layers to determine what happened ?
Reporter | ||
Comment 1•9 years ago
|
||
We recently swapped from http://stage.mozilla.org to http://ftp.mozilla.org, but I think that ends up at the same place and probably isn't relevant.
Assignee | ||
Comment 2•9 years ago
|
||
Cloudfront log entries for these requests:
> 2015-10-15 00:37:21 SFO9 61252 63.245.214.82 GET d34chcsvb7ug62.cloudfront.net /pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/update/linux-x86_64/en-ZA/firefox-40.0.3.complete.mar 200 - Python-urllib/2.6 - - Error 0JNjzXcGtXH28COzzkYDvR9k5_zWV-34QYBjaFxcW4g4_W7bPKZxzw==
>ftp.mozilla.orghttp 231 330.082 - - - Miss
2015-10-15 00:32:53 SFO9 17531494 63.245.214.82 GET d34chcsvb7ug62.cloudfront.net /pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/update/linux-x86_64/en-ZA/firefox-40.0.3.complete.mar 200 - Python-urllib/2.6 - - Hit JrTtHSkzXrkiIZjxUbrD5B-KUIVP56XpFflTFiQLUd6pv4HnF90CfA== ftp.mozilla.org http 231 1.735 - - - Hit
> 2015-10-15 00:57:27 SFO9 576731 63.245.214.82 GET d34chcsvb7ug62.cloudfront.net /pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/update/linux-x86_64/bs/firefox-41.0.1.complete.mar 200 - Python-urllib/2.6 - - Error 2XGMP2IKsY23HYyyuiwi1fEtiK24yra8-IMN_ZGVwJ5EFeDEId0Ubg== ftp.mozilla.org http 228 330.056 - - - Miss
2015-10-15 00:53:00 SFO9 9618022 63.245.214.82 GET
> d34chcsvb7ug62.cloudfront.net /pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/update/linux-x86_64/bs/firefox-41.0.1.complete.mar 200 - Python-urllib/2.6 - - Hit DBaKQ2wPUQWmmgbUQivwnXUcO6b0XPK0RhNmZALdnkryDyc81D6Thw== ftp.mozilla.org http 228 1.963 - - - Hit
Assignee | ||
Comment 3•9 years ago
|
||
Yikes, ugly formatting. Attaching.
Assignee | ||
Comment 4•9 years ago
|
||
It's hard to tell exactly what happened, but it looks like there was some Error between the origin (zeus lb) and cloudfront. Probably not worth debugging since we are moving to S3 in a few days.
Assignee | ||
Comment 5•9 years ago
|
||
Log format is described here: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html#BasicDistributionFileFormat
Reporter | ||
Comment 6•9 years ago
|
||
We did another release without hitting this, and we're migrating tomorrow, so that sounds fair.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
Comment 7•9 years ago
|
||
We see an identical behavior a couple of days now for our firefox-ui-tests. We fail to run the tests at all because the downloaded installer file is invalid. See bug 1219934 for more details.
Shall we reopen this bug or file a new one for netops given that it seems to be an issue only for scl3 -> cloudfront?
Flags: needinfo?(nthomas)
Reporter | ||
Comment 8•9 years ago
|
||
We shouldn't reopen this for netops, new clean bugs work best there.
Flags: needinfo?(nthomas)
Comment 9•9 years ago
|
||
Jeremy suggest to use bug 1170832 for that underlying problem.
You need to log in
before you can comment on or make changes to this bug.
Description
•