1807861 - In TreeHerder view of single push, the details panel is disabled/grayed-out with a permanent throbber, with "502 Bad Gateway" for bug_suggestions, & Uncaught (in promise) SyntaxError: JSON.parse: unexpected character at line 2 column 1 of the JSON data

Status: NEW

(Tree Management :: Treeherder: API, defect)

Description

[Daniel Holbert [:dholbert]]

STR:

1. Load this TreeHerder link (pointing at the Details for a Linux 18.04 x64 WebRender tsan opt / wpt23 orange task on a Dec 14th push to autoland):
   https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=SkmR62joQhGZjg8AgPempg.0&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=34a0b97102d44ff359c2758e651dba878d7ddf55

2. Wait for the details pane (at the bottom) to finish loading.

EXPECTED RESULTS:
It should finish loading. Throbber should disappear, and I should be able to interact with the details pane (e.g. clicking to open log or open pinboard). Or: if it can't because e.g. the job is too old, TreeHerder should tell me that. (I don't think this one is too old, though; it's from 2 weeks ago, so we definitely still have the log & other artifacts.)

ACTUAL RESULTS:
It never finishes loading. The throbber just stays there forever, and the log links etc. are grayed out and unclickable. Also: after a while, I eventually see this in my Web Console for this tab (not sure if it's related to the infinite throbber or not):

Uncaught (in promise) SyntaxError: JSON.parse: unexpected character at line 2 column 1 of the JSON data

Comment 1

[Daniel Holbert [:dholbert]]

Hmm, it seems like some tasks from that same revision are actually fine, e.g. this one (the first orange, for Linux 18.04 x64 WebRender opt/wpt5) loads the details pane in a second or two:
https://treeherder.mozilla.org/jobs?repo=autoland&revision=34a0b97102d44ff359c2758e651dba878d7ddf55&selectedTaskRun=PAqni0JBTHSReKu9X7fjKg.0

The orange X1 task on that same platform also loads its details just fine.

But the wpt8 task in between them seems to hit this issue and never finishes loading:
https://treeherder.mozilla.org/jobs?repo=autoland&revision=34a0b97102d44ff359c2758e651dba878d7ddf55&selectedTaskRun=ZQYgawG8QLGMJ3CMYFPo1Q.0

When this fails, I also see a 502 Bad Gateway response for an XHR to a URL like the following one:
https://treeherder.mozilla.org/api/project/autoland/jobs/399666660/bug_suggestions/
https://treeherder.mozilla.org/api/project/autoland/jobs/399667009/bug_suggestions/

And indeed, if I try to load those^ URLs directly, I eventually hit this after about 60 seconds:

Error: Server Error
The server encountered a temporary error and could not complete your request.

Please try again in 30 seconds.