657631 - code coverage builds timeout in "make check" and xpcshell-tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[(not currently active) Ted Mielczarek]

From the log:
======== BuildStep started ========
'bash -c ...' failed
=== Output ===
bash -c 'make -k check 2>&1 | bzip2 > ../logs/check.log.bz2'
 in dir /builds/slave/cen-lnx-codecoverage/build/objdir (timeout 600 secs)
<...>
command timed out: 600 seconds without output, attempting to kill
process killed by signal 9

======== BuildStep started ========
'bash -c ...' failed
=== Output ===
bash -c 'make xpcshell-tests 2>&1 | bzip2 > ../logs/xpcshell.log.bz2'
<...>
command timed out: 3600 seconds without output, attempting to kill
process killed by signal 9

The codecoverage steps pipe their output to a log file, so these are probably being killed for producing no output:
http://hg.mozilla.org/build/buildbotcustom/file/4eaf32eb9bfd/process/factory.py#l5792

We could swap that to be:
| tee >(bzip2 -c > log.bz2) 
so that output would go to stdout as well, and things might work again.

Longer-term, it might be nice to get these builds using the standard build steps, since we now upload compressed logs for build steps anyway.

Comment 1

[(not currently active) Ted Mielczarek]

Attachment: make code coverage build steps tee their output to log files so they don't get killed for producing no output.

I haven't tested this, but it seems pretty straightforward.

Comment 2

[(not currently active) Ted Mielczarek]

I filed bug 657703 on simplifying the codecoverage setup.

Comment 3

[Mike Hommey [:glandium]]

A downside of using tee is that the tinderbox log is going to be a single massive log.

Comment 4

[(not currently active) Ted Mielczarek]

http://hg.mozilla.org/build/buildbotcustom/rev/ce5d68fbb992