516726 - Running tests on debug builds shouldn't be so noisy

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[Chris AtLee [:catlee]]

Running debug tests can generate output well over 600 MB before the test is killed.  Our buildbot master process usually crashes when trying to process a log file that large.

I'm looking at ways we can set a limit on the size of the log from within buildbot, but we could get debug tests up and running faster if we didn't have this problem to deal with right now.

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

If there's a hang that's producing output in an infinite loop, it's going to keep producing output until the process is killed.  Buildbot *has* to deal with that.

One of the things that's going to help a lot is that once we have unit tests on debug builds running, we'll actually be able to create some pressure to get the assertion counts down.  But we can't actually make useful progress on that until assertions turn the tree orange, and we can't do that until unit tests on debug builds are in production.

Comment 2

[Chris AtLee [:catlee]]

(In reply to comment #1)
> If there's a hang that's producing output in an infinite loop, it's going to
> keep producing output until the process is killed.  Buildbot *has* to deal with
> that.

And if the way of dealing with that is by truncating the log?  That could take some time to implement.

> One of the things that's going to help a lot is that once we have unit tests on
> debug builds running, we'll actually be able to create some pressure to get the
> assertion counts down.  But we can't actually make useful progress on that
> until assertions turn the tree orange, and we can't do that until unit tests on
> debug builds are in production.

...and we can't start running them in production until we have this fixed somehow.

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #2)
> (In reply to comment #1)
> > If there's a hang that's producing output in an infinite loop, it's going to
> > keep producing output until the process is killed.  Buildbot *has* to deal with
> > that.
> 
> And if the way of dealing with that is by truncating the log?  That could take
> some time to implement.

It seems like we should kill the process if the log gets significantly over its normal size, on the assumption that that means it's generating an infinite amount of output.

Do you have any better alternatives?  The reality is that a bunch of our random orange is hangs.  One of the useful things about debug builds is that they print output, so in some of the hangs there's going to be output.

What are the normal log sizes?

Comment 4

[Chris AtLee [:catlee]]

(In reply to comment #3)
> It seems like we should kill the process if the log gets significantly over its
> normal size, on the assumption that that means it's generating an infinite
> amount of output.
> 
> Do you have any better alternatives?  The reality is that a bunch of our random
> orange is hangs.  One of the useful things about debug builds is that they
> print output, so in some of the hangs there's going to be output.

Is it possible to make assertions remember that they've been hit, and to be quiet after printing out a warning the first time (or first N times)?

> What are the normal log sizes?

The largest logs at this point are the win32 mochitest logs, which weigh in at around 13 MB uncompressed.

Comment 5

[(not currently active) Ted Mielczarek]

Assertions aside (since they're useful info), debug builds print a lot of other crap that I don't think people are going to find useful in tinderbox logs. I think we should be able to turn that off (stuff like the dom window counts etc) for our tinderbox builds.

Comment 6

[Chris AtLee [:catlee]]

We've worked around this by setting a maximum log size of 50 MB on the buildbot master.