https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/553eb5-24168bec-2856-11e6-b947-b82a72d22a14 Triggered by: https://treeherder.allizom.org/api/project/mozilla-inbound/logslice/?job_id=26525415&start_line=4000&end_line=4300 Traceback (most recent call last): ... File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/rest_framework/views.py", line 463, in dispatch File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/api/logslice.py", line 75, in list File "/usr/lib64/python2.7/gzip.py", line 464, in readline File "/usr/lib64/python2.7/gzip.py", line 268, in read File "/usr/lib64/python2.7/gzip.py", line 303, in _read File "/usr/lib64/python2.7/gzip.py", line 197, in _read_gzip_header exceptions:IOError: Not a gzipped file Not sure if this is a regression from bug 1275903 or not?
The job in question: https://treeherder.allizom.org/logviewer.html#?job_id=26525415&repo=mozilla-inbound#L4151 Raw log: https://queue.taskcluster.net/v1/task/ExYcsXIVTlO9lv8iaqan7Q/runs/0/artifacts/public%2Flogs%2Flive_backing.log However this redirects: $ curl -I https://queue.taskcluster.net/v1/task/ExYcsXIVTlO9lv8iaqan7Q/runs/0/artifacts/pub lic%2Flogs%2Flive_backing.log HTTP/1.1 303 See Other ... Location: https://public-artifacts.taskcluster.net/ExYcsXIVTlO9lv8iaqan7Q/0/public/logs/live_backing.log
There's also this: https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/553eb7-1566a282-284c-11e6-b947-b82a72d22a14 exceptions:OSError: [Errno 13] Permission denied: '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/log_cache/tmp0Q46ih'
We need to hold off on the next prod deploy until this is resolved - loading log content doesn't work for any log at the moment on stage, eg: buildbot gzipped log: https://treeherder.allizom.org/logviewer.html#?job_id=26544334&repo=mozilla-inbound#L29211 http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1464859494/mozilla-inbound-linux-bm72-build1-build1838.txt.gz TC log: https://treeherder.allizom.org/logviewer.html#?job_id=26550075&repo=mozilla-inbound#L4070 https://queue.taskcluster.net/v1/task/arRoybJ2TpS0Qfjx-APc5w/runs/0/artifacts/public%2Flogs%2Flive_backing.log
Hmm, not sure how this could be happening, I thought we fixed this up pretty good. Regardless, I'll take this one.
Assignee: nobody → wlachance
Comment on attachment 8759261 [details] [review] [treeherder] wlach:1277499 > mozilla:master The implementation could be more elegant, but I think this is o.k. Bonus: unit tests!
Attachment #8759261 - Flags: review?(emorley)
Attachment #8759261 - Flags: review?(emorley) → review+
(In reply to Ed Morley [:emorley] from comment #2) > There's also this: > > https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/ > 553eb7-1566a282-284c-11e6-b947-b82a72d22a14 > > exceptions:OSError: [Errno 13] Permission denied: > '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/ > log_cache/tmp0Q46ih' This turned out to because the '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/log_cache/' directory was owned by newrelic, instead of treeherder. I chown'ed it back. Do you have any idea how this could have happened Ed?
(In reply to William Lachance (:wlach) from comment #7) > This turned out to because the > '/data/www/treeherder.allizom.org/treeherder-service/treeherder/webapp/ > log_cache/' directory was owned by newrelic, instead of treeherder. I > chown'ed it back. Do you have any idea how this could have happened Ed? Should add this was only on the *second* webhead (th-stage-webapp-2). The other webheads had the correct permissions set (which was this API worked intermittently)
Ah the 2nd stage webhead was the replaced one (bug 1260707).
2 years ago
Pull request has landed in master: https://github.com/mozilla/treeherder/commit/a48c9ea1e8a4f6d1b0199d6403f8533b7aa933a4
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/0c1ccce689b84a2f550387b6f37a5d6ff8c25371 Bug 1277499 - Don't test same functionality multiple times in logslice tests https://github.com/mozilla/treeherder/commit/89d6e6e84c0b5fe83c259f8c1ccce3a03350a414 Bug 1277499 - Handle non-gzipped files in logslice API
You need to log in before you can comment on or make changes to this bug.