725654 - TBPL is missing some jobs

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

TBPL is not display all running, pending, nor finished jobs for some changesets. For example: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=c4c7eaaeec91

The above link shows linux opt tests being run, but not linux opt build being created. This job shows up in self serve: https://build.mozilla.org/buildapi/self-serve/mozilla-inbound/build/8688434

AFAICT self serve pulls from the database directly, which means the update_from_files.py jobs are OK. Given that, I strongly suspect that the JSON file created on cruncher are missing the information, but I haven't confirmed that yet.

There was a TBPL update overnight in https://bugzilla.mozilla.org/show_bug.cgi?id=725220, but the changes don't look suspicious (http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/61b49334bc31 and http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/f86f3e6360af). Last week, http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/27c901950707 was deployed in https://bugzilla.mozilla.org/show_bug.cgi?id=722887, but we should've seen issues prior to now if that was at fault.

I haven't dug through the json files to see if they are indeed missing the jobs.

Comment 1

[[e-mail] (:bhearsum)]

This seems to have resolved itself for now....maybe cruncher was just really slow for awhile?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

IIRC we're generating some/all of that data on buildapi01.b.m.o now, but still publish to build.m.o (which builddata.pub.b.m.o is a cname for). There nagios checks on dm-wwwbuild01 which check the freshness.