Closed Bug 1287113 Opened 6 years ago Closed 4 years ago

build-pending.js and builds-running.js should always be json decodable

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: emorley, Unassigned)

References

(Blocks 1 open bug)

Details

builds-running.js (and also builds-pending) should be json decodable, however due to bug 1286942 currently contain:

"""
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
 webmaster@mozilla.com and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
"""
(From https://secure.pub.build.mozilla.org/builddata/buildjson/builds-running.js)

This causes Treeherder to get:
 simplejson.scanner:JSONDecodeError: Expecting value: line 1 column 1 (char 0) 
...since we do a `raise_for_status()` however the file is returned with an HTTP 200, so it looks fine to try to decode it. eg:
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/0e9daf90-4a92-11e6-9e2c-c81f66b8ceca_31398_36910

If the script generating the file hits an error, I would expect either of:
1) The old file is left as-is, but is just stale
2) build-{pending,running}.js is created, but contains only `{}` or say `{"error": "Script failed with HTTP 500"}`
3) The file is removed, so HTTP 404s when we try tom request it

My preference would be to fix this via on the releng side (rather than us adding handling), with option #1, since:
* it would mean the stale-ness nagios checks work as expected (I'm presuming we still have them yeah?)
* Treeherder and other consumers could at least still fetch the data that was in the file the last time it successfully ran, rather than that being overwritten too

Fwiw I've seen errors ingesting these files before too, bug 1286942 isn't the first time this has happened.

Chris, what are your thoughts?
Thanks :-)

[1] https://secure.pub.build.mozilla.org/builddata/buildjson/builds-running.js
Flags: needinfo?(catlee)
Just to clarify, you were getting a 200 OK response with the 500 ISE error message in the body above?
Flags: needinfo?(catlee)
Yeah that's correct :-)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.