Blobber upload files not served with correct content type

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
3 years ago
21 days ago

People

(Reporter: zac, Assigned: ahal)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

3 years ago
The html output report for the Gu job is not showing correctly.

STR:
1. Open a b2g-i job:
(this is not the first failing job, just an example)
https://tbpl.mozilla.org/?tree=B2g-Inbound&rev=fba986dba3ac
2. Copy to a new tab the url where it says "Uploaded output.html to ....." 

direct link to the result of step 2:
http://mozilla-releng-blobs.s3.amazonaws.com/blobs/b2g-inbound/sha512/dac9c8a3886813b0d2119e0810db1309fd2b23770387b965f16324a03a6f02e833bae868ea2984b52235b8ddd2ec70bdd5c3a2fd04956aa072d113c52d39a9db

The report should be a nicely formatted HTML report.
I can see the same issue on treeherder (http://treeherder-dev.allizom.org). Does the report work okay locally Zac?
(Reporter)

Comment 2

3 years ago
With pinned Marionette/gaiatest/etc it's fine. I haven't had time to test with in-tree Marionette yet.
(Reporter)

Comment 3

3 years ago
Still busted, this is making it hard to debug TBPL failures.
For some reason, all of the blobber uploads are now tar.gz files.  So, you can still view the file by downloading it, appending a .tar.gz extension to it, and then extracting it.

I'm not sure why this is happening; :rail, can we serve these with the correct content-type so that people can view them more easily?
Flags: needinfo?(rail)
Summary: Gu output report on TBPL is broken → Blobber upload files not served with correct content type
IIRC ted added the compression feature.

302 ted
Flags: needinfo?(rail) → needinfo?(ted)
This was originally implemented in bug 962356, but it had a bug which I attempted to fix in bug 981654, but apparently it's broken in some other way now. :-(
Flags: needinfo?(ted)
Not a fire drill, but would be nice to have fixed.
Assignee: nobody → ted
This is an annoyance for Android 4.0 Debug mochitests. They have intermittent failures of the form "Assertion count X is greater than the expected range"; the actual assertions are often only found in the logcat uploaded to blobber. Unfortunately, the logcat is served as compressed content, so sheriffs need to click-save-gunzip-edit-search to star.
(Assignee)

Comment 9

3 years ago
Created attachment 8420326 [details] [review]
Fix missing quotes in dict key

I think I found the culprit.
Assignee: ted → ahalberstadt
Status: NEW → ASSIGNED
Attachment #8420326 - Flags: review?(ted)
Comment on attachment 8420326 [details] [review]
Fix missing quotes in dict key

Wow, that's terrible. Good catch.
Attachment #8420326 - Flags: review?(ted) → review+
ahal: nice one, gg!

I've merged the code. 

Guys, would you mind if we wait 1-2 more days 'till we deploy the changes in production? There are also some changes regarding the sync up for filetypes whitelisting awaiting review here ( https://bugzilla.mozilla.org/show_bug.cgi?id=998796#c7 ). I'd like to deploy them both in a single version bump so Rail doesn't have to do the puppet-beanstalk-workaround twice. Thanks!
Nevermind that, a new PR on bug 1011518 enforced a new blouploader version release - 1.1.4
As soon as Rail deploys that, it should be fine here too.
(Assignee)

Comment 13

3 years ago
Hmm, so this still appears to be broken :(
(Assignee)

Comment 14

3 years ago
My theory is that we aren't uploading the files properly to amazon. This says something about needing to upload both an uncompressed and compressed (with .gz extension) version of the same file:
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html

If that's the case, seems like compression isn't worth it? Can we just turn it off? This bug is very annoying.
Assignee: ahalberstadt → nobody
Status: ASSIGNED → NEW
That's CloudFront, we're not using that AFAIK. We're just serving blobs right from S3.
Okay, I looked into this again, and it turns out that bug 998796 just happened to break this in a *different* way for uploading from a directory. Pull request up:
https://github.com/MihaiTabara/blobuploader/pull/13

We seriously need some unit tests for blobberc/blobuploader.
Created attachment 8428209 [details] [diff] [review]
Bump blobuploader version to 1.1.7

Thanks for the patch, apologize for the mistake.

Merged. New release here (1.1.7) here: https://pypi.python.org/pypi/blobuploader

> Ted: We seriously need some unit tests for blobberc/blobuploader.
Will handle this as soon as my exam session is over in two weeks.
Attachment #8428209 - Flags: review?(rail)
Comment on attachment 8428209 [details] [diff] [review]
Bump blobuploader version to 1.1.7

Waiting for the merge to production (probably today)
https://hg.mozilla.org/build/mozharness/rev/2ec56c226500

Let me know if the change is urgent so I just transplant it.
Attachment #8428209 - Flags: review?(rail)
Attachment #8428209 - Flags: review+
Attachment #8428209 - Flags: checked-in+
In production
I verified that all of the Android logcats are now served correctly. Great to see this working -- thanks!
Good news then :-)
I suggest we leave the bug open for a few more days just to make sure nothing else arises.
Haven't heard any complaints.
Assignee: nobody → ahalberstadt
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.