707553 - tbpl has stopped generating logs and summaries of failures

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Phil Ringnalda (:philor)]

I'm sorry, I know this is a terrible bug to file on a Sunday, but it is what it is.

When you click on an orange letter on https://tbpl.mozilla.org/ it requests two URLs, like https://tbpl.mozilla.org/php/getLogExcerpt.php?id=7738573&type=tinderbox_print and https://tbpl.mozilla.org/php/getLogExcerpt.php?id=7738573&tree=Firefox&type=annotated and presents links to the annotated log like https://tbpl.mozilla.org/php/getParsedLog.php?id=7738573&tree=Firefox

The last test failure I can see that was starred finished around 7am, so things must have been working then, but now production tbpl just returns blanks for all three of those URLs. tbpl-dev mostly but not always manages to return something for type=annotated and for getParsedLog, but doesn't return anything for type=tinderbox_print, which is particularly bizarre.

An incomplete list of wild guesses that might result in that:

* a change in firewall rules
* no access to ftp.m.o from tbpl.m.o
* MySQL errors
* a change in max script execution time
* extremely slow access from tbpl.m.o to bugzilla (tbpl searches for bugs on the failures, which (handwaving) might maybe run it over max execution time if it slowed down)
* a whole lot of things I can't think of from the outside

Comment 1

[Benoit Jacob [:bjacob] (mostly away)]

> * no access to ftp.m.o from tbpl.m.o

Can't be that: the files are not even present at all on the FTP servers:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/
doesn't have any test logs for recent pushes.

Comment 2

[Phil Ringnalda (:philor)]

Not sure where bjacob went wrong looking for logs (or whether it's something like a delay in them showing up on ftp.m.o), but http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1323026301/mozilla-inbound_fedora_test-mochitests-1-build477.txt.gz does exist.

Comment 3

[Jason Thomas [:jason]]

Apache was not able to write to the cache directory. I am now able to reach the mentioned URLs above.

Comment 4

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Phil Ringnalda (:philor) from comment #2)
> Not sure where bjacob went wrong looking for logs (or whether it's something
> like a delay in them showing up on ftp.m.o), but
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> inbound-linux/1323026301/mozilla-inbound_fedora_test-mochitests-1-build477.
> txt.gz does exist.

Yes, sorry, I'm stupid.

Comment 5

[Phil Ringnalda (:philor)]

Thanks, Jason :)