485809 - Run reftest/crashtest using make+python rather than calling seamonkey directly

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, enhancement)

Description

[Serge Gautherie (:sgautherie)]

The main feature I miss now on the tinderboxes is the leak reports.
Other (new) harness features, like crashdump/stack report, will be good to get too.

Comment 1

[Serge Gautherie (:sgautherie)]

Attachment: (Av1) Call |make| [Superseded by comment 3]

Like this !?

Comment 2

[Robert Kaiser]

Comment on attachment 370147 [details] [diff] [review]
(Av1) Call |make| [Superseded by comment 3]

AFAIK, no, but I haven't even investigated how to use the new system of buildbotcustom steps, so I can't tell. Leaving the review stalled until I actuall come around to look into rewriting our buildbot configs.

Comment 3

[Robert Kaiser]

In a first step towards bug 485820, I today have made the SeaMonkey test buildbot configs be very similar to the steps in the not-yet-generic-enough buildbotcustom UnittestBuildFactory, which switched us to those make-driven steps.

Comment 4

[Serge Gautherie (:sgautherie)]

Attachment: (Bv1) Reduce timeouts to low (but safe) values

About timeouts and durations:

Reftest:
timeout: 5 mn
Linux  : slowest=5s total=14mn
MacOSX : slowest=5s total=07mn
Windows: slowest=5s total=06mn

Crashtest:
timeout: default (whatever that is)
Linux  : slowest=2s total=1mn
MacOSX : slowest=2s total=1mn
Windows: slowest=2s total=1mn

Then it looks like we could set the timeouts to as low as 30s and 15s respectively, let's say ?

Comment 5

[Robert Kaiser]

Comment on attachment 370690 [details] [diff] [review]
(Bv1) Reduce timeouts to low (but safe) values

We're trying to stay as closely as possible to UnittestBuildFactory in buildbotcustom.process.factory, I won't change timeouts to other values than there unless there's very compelling reasons (basically meaning I only change this if we fail without the change).

Comment 6

[Serge Gautherie (:sgautherie)]

Attachment: (Cv1-FF) Reduce timeouts to low (but safe) values

About timeouts and durations: (1.9.2 / 1.9.1)

Reftest:
timeout: 5 mn
Linux  : slowest=3s total=13mn / slowest=5s total=11mn
MacOSX : slowest=2s total=05mn / slowest=5s total=04mn
Windows: slowest=2s total=08mn / slowest=5s total=08mn

Crashtest:
timeout: default (whatever that is)
Linux  : slowest=2s total=2mn / slowest=2s total=1mn
MacOSX : slowest=3s total=1mn / slowest=2s total=1mn
Windows: slowest=3s total=2mn / slowest=2s total=1mn

Fwiw, the harness timeout is set at 10s:
|const LOAD_FAILURE_TIMEOUT = 10000; // ms|

Then it looks like we could set the timeouts to as low as 30s and 15s
respectively, let's say ?
(or anywhere between 10 and 30, as you prefer.)

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 370707 [details] [diff] [review]
(Cv1-FF) Reduce timeouts to low (but safe) values

I appreciate what you're trying to do here, Serge, but I don't think we should make this change. During times of increased load, or times when we have hardware problems the tests take longer, and the timeouts need to be higher. I'd rather keep things how they are are sacrifice a bit of time when something hangs than burn builds when load jumps up.

Comment 8

[[e-mail] (:bhearsum)]

By the way...this bug talks about seamonkey, but the patch appears to be against the main buildbotcustom repository...am I missing something?

Comment 9

[Robert Kaiser]

Ben, see comment #5, now that I've tried to come as close as possible (right now) to using the same config for SeaMonkey buildbots as for Firefox ones, I don't want to diverge with things like those timeouts.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to comment #9)
> Ben, see comment #5, now that I've tried to come as close as possible (right
> now) to using the same config for SeaMonkey buildbots as for Firefox ones, I
> don't want to diverge with things like those timeouts.

That makes sense, yeah. Why do the Seamonkey timeouts need to be changed, though?

Comment 11

[Robert Kaiser]

(In reply to comment #10)
> (In reply to comment #9)
> > Ben, see comment #5, now that I've tried to come as close as possible (right
> > now) to using the same config for SeaMonkey buildbots as for Firefox ones, I
> > don't want to diverge with things like those timeouts.
> 
> That makes sense, yeah. Why do the Seamonkey timeouts need to be changed,
> though?

Because Serge thinks they're too long. I personally don't care as long as we don't hang forever if we hang at all ;-)

Comment 12

[[e-mail] (:bhearsum)]

In that case, comment #7 stands.

Comment 13

[Serge Gautherie (:sgautherie)]

Ben, why does reftest has 5 mn and crashtest doesn't ? Is there (and what is) the default value ?

Comment 14

[(not currently active) Ted Mielczarek]

Comment on attachment 370707 [details] [diff] [review]
(Cv1-FF) Reduce timeouts to low (but safe) values

bhearsum's review is all you need here anyway.