New Relic stage "exceptions:IOError: Not a gzipped file" during webapp.api.logslice:LogSliceView.list

RESOLVED FIXED

Status

P1
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: emorley, Assigned: wlach)

Tracking

(Blocks: 1 bug)

Details

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/553eb5-24168bec-2856-11e6-b947-b82a72d22a14

Triggered by:
https://treeherder.allizom.org/api/project/mozilla-inbound/logslice/?job_id=26525415&start_line=4000&end_line=4300

Traceback (most recent call last):
...
File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch
File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/api/logslice.py", line 75, in list
File "/usr/lib64/python2.7/gzip.py", line 464, in readline
File "/usr/lib64/python2.7/gzip.py", line 268, in read
File "/usr/lib64/python2.7/gzip.py", line 303, in _read
File "/usr/lib64/python2.7/gzip.py", line 197, in _read_gzip_header
exceptions:IOError: Not a gzipped file 

Not sure if this is a regression from bug 1275903 or not?
(Reporter)

Comment 2

3 years ago
There's also this:

https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/553eb7-1566a282-284c-11e6-b947-b82a72d22a14

 exceptions:OSError: [Errno 13] Permission denied: '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/log_cache/tmp0Q46ih'
Hmm, not sure how this could be happening, I thought we fixed this up pretty good. Regardless, I'll take this one.
Assignee: nobody → wlachance
Created attachment 8759261 [details] [review]
[treeherder] wlach:1277499 > mozilla:master
Comment on attachment 8759261 [details] [review]
[treeherder] wlach:1277499 > mozilla:master

The implementation could be more elegant, but I think this is o.k. Bonus: unit tests!
Attachment #8759261 - Flags: review?(emorley)
(Reporter)

Updated

3 years ago
Attachment #8759261 - Flags: review?(emorley) → review+
(In reply to Ed Morley [:emorley] from comment #2)
> There's also this:
> 
> https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/
> 553eb7-1566a282-284c-11e6-b947-b82a72d22a14
> 
>  exceptions:OSError: [Errno 13] Permission denied:
> '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/
> log_cache/tmp0Q46ih'

This turned out to because the '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/log_cache/' directory was owned by newrelic, instead of treeherder. I chown'ed it back. Do you have any idea how this could have happened Ed?
Flags: needinfo?(emorley)
(In reply to William Lachance (:wlach) from comment #7)
> This turned out to because the
> '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/
> log_cache/' directory was owned by newrelic, instead of treeherder. I
> chown'ed it back. Do you have any idea how this could have happened Ed?

Should add this was only on the *second* webhead (th-stage-webapp-2). The other webheads had the correct permissions set (which was this API worked intermittently)
(Reporter)

Comment 9

3 years ago
Ah the 2nd stage webhead was the replaced one (bug 1260707).
Flags: needinfo?(emorley)
Keywords: autoland
Pull request has landed in master: https://github.com/mozilla/treeherder/commit/a48c9ea1e8a4f6d1b0199d6403f8533b7aa933a4
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Keywords: autoland
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.