Closed
Bug 1277499
Opened 8 years ago
Closed 8 years ago
New Relic stage "exceptions:IOError: Not a gzipped file" during webapp.api.logslice:LogSliceView.list
Categories
(Tree Management :: Treeherder: API, defect, P1)
Tree Management
Treeherder: API
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: emorley, Assigned: wlach)
References
(Blocks 1 open bug)
Details
Attachments
(1 file)
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/553eb5-24168bec-2856-11e6-b947-b82a72d22a14
Triggered by:
https://treeherder.allizom.org/api/project/mozilla-inbound/logslice/?job_id=26525415&start_line=4000&end_line=4300
Traceback (most recent call last):
...
File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/api/logslice.py", line 75, in list
File "/usr/lib64/python2.7/gzip.py", line 464, in readline
File "/usr/lib64/python2.7/gzip.py", line 268, in read
File "/usr/lib64/python2.7/gzip.py", line 303, in _read
File "/usr/lib64/python2.7/gzip.py", line 197, in _read_gzip_header
exceptions:IOError: Not a gzipped file
Not sure if this is a regression from bug 1275903 or not?
Reporter | ||
Comment 1•8 years ago
|
||
The job in question:
https://treeherder.allizom.org/logviewer.html#?job_id=26525415&repo=mozilla-inbound#L4151
Raw log:
https://queue.taskcluster.net/v1/task/ExYcsXIVTlO9lv8iaqan7Q/runs/0/artifacts/public%2Flogs%2Flive_backing.log
However this redirects:
$ curl -I https://queue.taskcluster.net/v1/task/ExYcsXIVTlO9lv8iaqan7Q/runs/0/artifacts/pub lic%2Flogs%2Flive_backing.log
HTTP/1.1 303 See Other
...
Location: https://public-artifacts.taskcluster.net/ExYcsXIVTlO9lv8iaqan7Q/0/public/logs/live_backing.log
Reporter | ||
Comment 2•8 years ago
|
||
There's also this:
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/553eb7-1566a282-284c-11e6-b947-b82a72d22a14
exceptions:OSError: [Errno 13] Permission denied: '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/log_cache/tmp0Q46ih'
Reporter | ||
Comment 3•8 years ago
|
||
We need to hold off on the next prod deploy until this is resolved - loading log content doesn't work for any log at the moment on stage, eg:
buildbot gzipped log:
https://treeherder.allizom.org/logviewer.html#?job_id=26544334&repo=mozilla-inbound#L29211
http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1464859494/mozilla-inbound-linux-bm72-build1-build1838.txt.gz
TC log:
https://treeherder.allizom.org/logviewer.html#?job_id=26550075&repo=mozilla-inbound#L4070
https://queue.taskcluster.net/v1/task/arRoybJ2TpS0Qfjx-APc5w/runs/0/artifacts/public%2Flogs%2Flive_backing.log
Assignee | ||
Comment 4•8 years ago
|
||
Hmm, not sure how this could be happening, I thought we fixed this up pretty good. Regardless, I'll take this one.
Assignee: nobody → wlachance
Comment 5•8 years ago
|
||
Assignee | ||
Comment 6•8 years ago
|
||
Comment on attachment 8759261 [details] [review]
[treeherder] wlach:1277499 > mozilla:master
The implementation could be more elegant, but I think this is o.k. Bonus: unit tests!
Attachment #8759261 -
Flags: review?(emorley)
Reporter | ||
Updated•8 years ago
|
Attachment #8759261 -
Flags: review?(emorley) → review+
Assignee | ||
Comment 7•8 years ago
|
||
(In reply to Ed Morley [:emorley] from comment #2)
> There's also this:
>
> https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/
> 553eb7-1566a282-284c-11e6-b947-b82a72d22a14
>
> exceptions:OSError: [Errno 13] Permission denied:
> '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/
> log_cache/tmp0Q46ih'
This turned out to because the '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/log_cache/' directory was owned by newrelic, instead of treeherder. I chown'ed it back. Do you have any idea how this could have happened Ed?
Flags: needinfo?(emorley)
Assignee | ||
Comment 8•8 years ago
|
||
(In reply to William Lachance (:wlach) from comment #7)
> This turned out to because the
> '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/
> log_cache/' directory was owned by newrelic, instead of treeherder. I
> chown'ed it back. Do you have any idea how this could have happened Ed?
Should add this was only on the *second* webhead (th-stage-webapp-2). The other webheads had the correct permissions set (which was this API worked intermittently)
Reporter | ||
Comment 9•8 years ago
|
||
Ah the 2nd stage webhead was the replaced one (bug 1260707).
Flags: needinfo?(emorley)
Comment 10•8 years ago
|
||
Pull request has landed in master: https://github.com/mozilla/treeherder/commit/a48c9ea1e8a4f6d1b0199d6403f8533b7aa933a4
Comment 11•8 years ago
|
||
Commits pushed to master at https://github.com/mozilla/treeherder
https://github.com/mozilla/treeherder/commit/0c1ccce689b84a2f550387b6f37a5d6ff8c25371
Bug 1277499 - Don't test same functionality multiple times in logslice tests
https://github.com/mozilla/treeherder/commit/89d6e6e84c0b5fe83c259f8c1ccce3a03350a414
Bug 1277499 - Handle non-gzipped files in logslice API
You need to log in
before you can comment on or make changes to this bug.
Description
•