905554 - buildapi/scripts/reporter.py hangs generating builds-4hr.js.gz, resulting in tbpl.m.o not showing completed builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This may be similar to bug 810049, bug 821232, bug 827443.  Or maybe not.

Currently completed builds are not showing up on https://tbpl.mozilla.org/?tree=Mozilla-Inbound .  Builds that are running just disappear when they finish rather than showing up as completed, which means people aren't seeing results.

Trees are closed (shortly) as a result.

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

A clear example is that:

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=31c08ca022b3 shows no completed builds (only gray)

https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/31c08ca022b3 shows many completed jobs (in the "builds" section, below "pending" and "running")

If the first link shows plenty of green builds, then the problem is fixed (or at least that part of the queued-up problem is fixed).

Comment 2

[Ed Morley [:emorley]]

Looks like the script that generates builds-4hr.js.gz is stuck, last modified is:
15-Aug-2013 05:38              

http://builddata.pub.build.mozilla.org/buildjson/builds-4hr.js.gz

Note builds-running.js has modified of 15-Aug-2013 07:59, so the server is UTC+0.

Comment 3

[Ed Morley [:emorley]]

If it helps, this is the puppet script that generates that file:
http://hg.mozilla.org/build/puppet-manifests/file/tip/modules/buildapi/templates/report-4hr.sh.erb

Comment 4

[Ed Morley [:emorley]]

(In reply to David Baron [:dbaron] (don't cc:, use needinfo? instead) from comment #0)
> This may be similar to bug 810049, bug 821232, bug 827443.  Or maybe not.

Those failure modes hopefully will not occur now that bug 801461 is fixed :-)

Comment 5

[Ed Morley [:emorley]]

(Note to self: we have until 09:37 UTC+0 before we'll need to manually import TBPL data, since the 4 hr windows won't overlap)

Comment 6

[Ashish Vijayaram [:ashish]]

Indeed report-4hr.sh cron is stuck running since 00:05.

buildapi 11898  0.0  0.0   8704   996 ?        Ss   00:05   0:00  |   \_ /bin/sh /home/buildapi/bin/report-4hr.sh
buildapi 11901  0.0  1.1 164788 48384 ?        S    00:05   0:00  |       \_ /home/buildapi/bin/python /home/buildapi/src/buildapi/scripts/reporter.py -z -o /var/www/buildapi/buildjson/builds-4hr.js.gz --starttime 1376535901

Comment 7

[Ashish Vijayaram [:ashish]]

Killed the process and removed the lock file. Subsequent run from cron has kicked off.

Comment 8

[Ed Morley [:emorley]]

Not sure how long the script takes to run, don't think it's long - think it may be hung again, since builds-4hr.js.gz.tmp has been created at 15-Aug-2013 08:27 UTC+0 with filesize of 0 bytes, and hasn't changed since.

Comment 9

[Ed Morley [:emorley]]

Ah, there is a log file, missed that oops.

Could you see what ~/reporter-4hr.log contains?

Comment 10

[Ed Morley [:emorley]]

(From IRC)

The log file hasn;t been updated since 00:04.

Running reporter.py manually gave:
{
Thu Aug 15 01:51:13 2013
0.65 get builds
}

Last line of stdout was at:
https://hg.mozilla.org/build/buildapi/file/ce6ca3a6c23e/buildapi/scripts/reporter.py#l108

Comment 11

[Ed Morley [:emorley]]

I've just seen bug 898688, don't know if it's related. That issue was supposed to be fixed by http://hg.mozilla.org/build/buildapi/rev/f988446ed820

Comment 12

[Ed Morley [:emorley]]

It also looks like the daily report generation script has hung too (builds-2013-08-14.js.gz.tmp and builds-2013-08-15.js.gz.tmp present at http://builddata.pub.build.mozilla.org/buildjson/), but we can regenerate them later (less important than builds-4hr.js.gz).

Comment 13

[Ashish Vijayaram [:ashish]]

:nthomas from Releng/Build has been engaged and is looking into the issue.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

redis was hung in the same way as bug 898739, so I've restarted the service. The builds-4hr.js.gz is generated, working on others.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

builds-2013-08-14.js.gz & builds-2013-08-15.js.gz are up to date (not actually used by tbpl), and edmorley says tbpl is working again.

Comment 16

[Ed Morley [:emorley]]

Trees reopened - thank you David, Ashish & Nick! :-)

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Bug 905587 for the followup investigate/fix.