tbpl has stopped generating logs and summaries of failures

VERIFIED FIXED

Status

mozilla.org Graveyard
Server Operations
--
blocker
VERIFIED FIXED
6 years ago
3 years ago

People

(Reporter: philor, Assigned: jason)

Tracking

Details

(Reporter)

Description

6 years ago
I'm sorry, I know this is a terrible bug to file on a Sunday, but it is what it is.

When you click on an orange letter on https://tbpl.mozilla.org/ it requests two URLs, like https://tbpl.mozilla.org/php/getLogExcerpt.php?id=7738573&type=tinderbox_print and https://tbpl.mozilla.org/php/getLogExcerpt.php?id=7738573&tree=Firefox&type=annotated and presents links to the annotated log like https://tbpl.mozilla.org/php/getParsedLog.php?id=7738573&tree=Firefox

The last test failure I can see that was starred finished around 7am, so things must have been working then, but now production tbpl just returns blanks for all three of those URLs. tbpl-dev mostly but not always manages to return something for type=annotated and for getParsedLog, but doesn't return anything for type=tinderbox_print, which is particularly bizarre.

An incomplete list of wild guesses that might result in that:

* a change in firewall rules
* no access to ftp.m.o from tbpl.m.o
* MySQL errors
* a change in max script execution time
* extremely slow access from tbpl.m.o to bugzilla (tbpl searches for bugs on the failures, which (handwaving) might maybe run it over max execution time if it slowed down)
* a whole lot of things I can't think of from the outside
> * no access to ftp.m.o from tbpl.m.o

Can't be that: the files are not even present at all on the FTP servers:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/
doesn't have any test logs for recent pushes.

Updated

6 years ago
Assignee: server-ops → rbryce
(Reporter)

Comment 2

6 years ago
Not sure where bjacob went wrong looking for logs (or whether it's something like a delay in them showing up on ftp.m.o), but http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1323026301/mozilla-inbound_fedora_test-mochitests-1-build477.txt.gz does exist.
(Assignee)

Comment 3

6 years ago
Apache was not able to write to the cache directory. I am now able to reach the mentioned URLs above.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(In reply to Phil Ringnalda (:philor) from comment #2)
> Not sure where bjacob went wrong looking for logs (or whether it's something
> like a delay in them showing up on ftp.m.o), but
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> inbound-linux/1323026301/mozilla-inbound_fedora_test-mochitests-1-build477.
> txt.gz does exist.

Yes, sorry, I'm stupid.
(Reporter)

Comment 5

6 years ago
Thanks, Jason :)
Status: RESOLVED → VERIFIED
(Assignee)

Updated

6 years ago
Assignee: rbryce → jthomas
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.