748720 - TBPL getLogExcerpt.php?type=annotated timing out

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ed Morley [:emorley]]

1) Go to https://tbpl.mozilla.org/?onlyunstarred=1
2) Press 'k' to scroll through the failures

Expected:
For summaries that hadn't been loaded prior to the failure: "Retrieving summary..." shown, then say <5-10 secs later the summary appears complete with bug # suggestions.

Actual:
"Retrieving summary..." -> times out with "Fetching summary failed."

Or every now and again, the summary does load, but with no bug suggestions (the suggestions are found with a bzapi search for whiteboard orange and summary containing test filename).

The summary is generated using:
https://tbpl.mozilla.org/php/getLogExcerpt.php?id=11189710&tree=Mozilla-Inbound&type=annotated

File does:
http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/1d987babf0d7/php/getLogExcerpt.php

Reported to #build at 12:17 UTC+1, ashish started looking into it straight away :-)

**Will mean closing the trees soon**

Many thanks :-)

Comment 1

[Graeme McCutcheon [:graememcc]]

This is more to do with Bugzilla being slow and unwell today rather than anything TBPL-infra, right?

I can't load https://api-dev.bugzilla.mozilla.org/latest/bug?id=748720 in the browser, so I'd assume TBPL is choking there.

That said, in the future, we should definitely look at wrapping the file_get_contents call in AnnotatedSummaryGenerator.php in a timeout, so that we can read the log snippet regardless of bugzilla being down. Those snippets should still be useful for frequent starrers.

Comment 2

[Bill Gianopoulos [:WG9s]]

(In reply to Ed Morley [:edmorley] from comment #0)
> **Will mean closing the trees soon**

Rather than closing trees for this, updating the tree status to indicate tbpl is not properly giving bug suggestions and a manual bugzilla search is required, would probably be sufficient.,  Just so that people starring builds know there is an issue.  That makes this more of a it is more work than builds are not being starred.

I can;t really speak for others who star builds, but I am willing to do a little extra work to keep the tree open.

Comment 3

[Ed Morley [:emorley]]

(In reply to Bill Gianopoulos [:WG9s] from comment #2)
> Rather than closing trees for this, updating the tree status to indicate
> tbpl is not properly giving bug suggestions and a manual bugzilla search is
> required, would probably be sufficient

That's why I haven't closed yet - however it's not giving the summaries most of the time, not just suggestions - and given how busy the tree was last night (since everything's been blocked on approval-required for 10 days), manually loading every single log isn't going to be viable once the US devs wake up.

Comment 4

[Bill Gianopoulos [:WG9s]]

OK. Current state seems to be we are now getting the summary but it does not match anything.  This makes me wonder if there is some kind of periodic job that creates a database of ornage-factor bugs that tbpl uses instead of querying bugzilla directly that failed and needs to be rerun.

Comment 5

[Ed Morley [:emorley]]

(In reply to Bill Gianopoulos [:WG9s] from comment #4)
> OK. Current state seems to be we are now getting the summary

It's still not working for many of the yet to be loaded summaries. The ones that eventually load are just being cached, making it look like all is well.

Comment 6

[Ashish Vijayaram [:ashish]]

Fixed after gerv restarted apache on api-dev