986032 - Blobber upload files not served with correct content type

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Zac C (:zac)]

The html output report for the Gu job is not showing correctly.

STR:
1. Open a b2g-i job:
(this is not the first failing job, just an example)
https://tbpl.mozilla.org/?tree=B2g-Inbound&rev=fba986dba3ac
2. Copy to a new tab the url where it says "Uploaded output.html to ....." 

direct link to the result of step 2:
http://mozilla-releng-blobs.s3.amazonaws.com/blobs/b2g-inbound/sha512/dac9c8a3886813b0d2119e0810db1309fd2b23770387b965f16324a03a6f02e833bae868ea2984b52235b8ddd2ec70bdd5c3a2fd04956aa072d113c52d39a9db

The report should be a nicely formatted HTML report.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I can see the same issue on treeherder (http://treeherder-dev.allizom.org). Does the report work okay locally Zac?

Comment 2

[Zac C (:zac)]

With pinned Marionette/gaiatest/etc it's fine. I haven't had time to test with in-tree Marionette yet.

Comment 3

[Zac C (:zac)]

Still busted, this is making it hard to debug TBPL failures.

Comment 4

[Jonathan Griffin (:jgriffin)]

For some reason, all of the blobber uploads are now tar.gz files.  So, you can still view the file by downloading it, appending a .tar.gz extension to it, and then extracting it.

I'm not sure why this is happening; :rail, can we serve these with the correct content-type so that people can view them more easily?

Comment 5

[Rail Aliiev [:rail]]

IIRC ted added the compression feature.

302 ted

Comment 6

[(not currently active) Ted Mielczarek]

This was originally implemented in bug 962356, but it had a bug which I attempted to fix in bug 981654, but apparently it's broken in some other way now. :-(

Comment 7

[Jonathan Griffin (:jgriffin)]

Not a fire drill, but would be nice to have fixed.

Comment 8

[Geoff Brown [:gbrown]]

This is an annoyance for Android 4.0 Debug mochitests. They have intermittent failures of the form "Assertion count X is greater than the expected range"; the actual assertions are often only found in the logcat uploaded to blobber. Unfortunately, the logcat is served as compressed content, so sheriffs need to click-save-gunzip-edit-search to star.

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Fix missing quotes in dict key

I think I found the culprit.

Comment 10

[(not currently active) Ted Mielczarek]

Comment on attachment 8420326 [details] [review]
Fix missing quotes in dict key

Wow, that's terrible. Good catch.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

ahal: nice one, gg!

I've merged the code. 

Guys, would you mind if we wait 1-2 more days 'till we deploy the changes in production? There are also some changes regarding the sync up for filetypes whitelisting awaiting review here ( https://bugzilla.mozilla.org/show_bug.cgi?id=998796#c7 ). I'd like to deploy them both in a single version bump so Rail doesn't have to do the puppet-beanstalk-workaround twice. Thanks!

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Nevermind that, a new PR on bug 1011518 enforced a new blouploader version release - 1.1.4
As soon as Rail deploys that, it should be fine here too.

Comment 13

[Andrew Halberstadt [:ahal]]

Hmm, so this still appears to be broken :(

Comment 14

[Andrew Halberstadt [:ahal]]

My theory is that we aren't uploading the files properly to amazon. This says something about needing to upload both an uncompressed and compressed (with .gz extension) version of the same file:
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html

If that's the case, seems like compression isn't worth it? Can we just turn it off? This bug is very annoying.

Comment 15

[(not currently active) Ted Mielczarek]

That's CloudFront, we're not using that AFAIK. We're just serving blobs right from S3.

Comment 16

[(not currently active) Ted Mielczarek]

Okay, I looked into this again, and it turns out that bug 998796 just happened to break this in a *different* way for uploading from a directory. Pull request up:
https://github.com/MihaiTabara/blobuploader/pull/13

We seriously need some unit tests for blobberc/blobuploader.

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bump blobuploader version to 1.1.7

Thanks for the patch, apologize for the mistake.

Merged. New release here (1.1.7) here: https://pypi.python.org/pypi/blobuploader

> Ted: We seriously need some unit tests for blobberc/blobuploader.
Will handle this as soon as my exam session is over in two weeks.

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 8428209 [details] [diff] [review]
Bump blobuploader version to 1.1.7

Waiting for the merge to production (probably today)
https://hg.mozilla.org/build/mozharness/rev/2ec56c226500

Let me know if the change is urgent so I just transplant it.

Comment 19

[Chris AtLee [:catlee]]

In production

Comment 20

[Geoff Brown [:gbrown]]

I verified that all of the Android logcats are now served correctly. Great to see this working -- thanks!

Comment 21

[Mihai Tabara [:mtabara]⌚️GMT]

Good news then :-)
I suggest we leave the bug open for a few more days just to make sure nothing else arises.

Comment 22

[(not currently active) Ted Mielczarek]

Haven't heard any complaints.