Closed Bug 485809 Opened 12 years ago Closed 12 years ago

Run reftest/crashtest using make+python rather than calling seamonkey directly

Categories

(SeaMonkey :: Build Config, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
seamonkey2.0b1

People

(Reporter: sgautherie, Assigned: sgautherie)

References

()

Details

(Keywords: autotest-issue)

Attachments

(2 files, 1 obsolete file)

The main feature I miss now on the tinderboxes is the leak reports.
Other (new) harness features, like crashdump/stack report, will be good to get too.
Depends on: 485820
Like this !?
Attachment #370147 - Flags: review?(kairo)
Comment on attachment 370147 [details] [diff] [review]
(Av1) Call |make| [Superseded by comment 3]

AFAIK, no, but I haven't even investigated how to use the new system of buildbotcustom steps, so I can't tell. Leaving the review stalled until I actuall come around to look into rewriting our buildbot configs.
In a first step towards bug 485820, I today have made the SeaMonkey test buildbot configs be very similar to the steps in the not-yet-generic-enough buildbotcustom UnittestBuildFactory, which switched us to those make-driven steps.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Attachment #370147 - Attachment description: (Av1) Call |make| → (Av1) Call |make| [Superseded by comment 3]
Attachment #370147 - Attachment is obsolete: true
Attachment #370147 - Flags: review?(kairo)
About timeouts and durations:

Reftest:
timeout: 5 mn
Linux  : slowest=5s total=14mn
MacOSX : slowest=5s total=07mn
Windows: slowest=5s total=06mn

Crashtest:
timeout: default (whatever that is)
Linux  : slowest=2s total=1mn
MacOSX : slowest=2s total=1mn
Windows: slowest=2s total=1mn

Then it looks like we could set the timeouts to as low as 30s and 15s respectively, let's say ?
Assignee: build-config → sgautherie.bz
Attachment #370690 - Flags: review?(kairo)
Attachment #370690 - Flags: review?(kairo) → review-
Comment on attachment 370690 [details] [diff] [review]
(Bv1) Reduce timeouts to low (but safe) values

We're trying to stay as closely as possible to UnittestBuildFactory in buildbotcustom.process.factory, I won't change timeouts to other values than there unless there's very compelling reasons (basically meaning I only change this if we fail without the change).
About timeouts and durations: (1.9.2 / 1.9.1)

Reftest:
timeout: 5 mn
Linux  : slowest=3s total=13mn / slowest=5s total=11mn
MacOSX : slowest=2s total=05mn / slowest=5s total=04mn
Windows: slowest=2s total=08mn / slowest=5s total=08mn

Crashtest:
timeout: default (whatever that is)
Linux  : slowest=2s total=2mn / slowest=2s total=1mn
MacOSX : slowest=3s total=1mn / slowest=2s total=1mn
Windows: slowest=3s total=2mn / slowest=2s total=1mn

Fwiw, the harness timeout is set at 10s:
|const LOAD_FAILURE_TIMEOUT = 10000; // ms|

Then it looks like we could set the timeouts to as low as 30s and 15s
respectively, let's say ?
(or anywhere between 10 and 30, as you prefer.)
Attachment #370707 - Flags: review?(ted.mielczarek)
Attachment #370707 - Flags: review?(bhearsum)
Comment on attachment 370707 [details] [diff] [review]
(Cv1-FF) Reduce timeouts to low (but safe) values

I appreciate what you're trying to do here, Serge, but I don't think we should make this change. During times of increased load, or times when we have hardware problems the tests take longer, and the timeouts need to be higher. I'd rather keep things how they are are sacrifice a bit of time when something hangs than burn builds when load jumps up.
Attachment #370707 - Flags: review?(bhearsum) → review-
By the way...this bug talks about seamonkey, but the patch appears to be against the main buildbotcustom repository...am I missing something?
Ben, see comment #5, now that I've tried to come as close as possible (right now) to using the same config for SeaMonkey buildbots as for Firefox ones, I don't want to diverge with things like those timeouts.
(In reply to comment #9)
> Ben, see comment #5, now that I've tried to come as close as possible (right
> now) to using the same config for SeaMonkey buildbots as for Firefox ones, I
> don't want to diverge with things like those timeouts.

That makes sense, yeah. Why do the Seamonkey timeouts need to be changed, though?
(In reply to comment #10)
> (In reply to comment #9)
> > Ben, see comment #5, now that I've tried to come as close as possible (right
> > now) to using the same config for SeaMonkey buildbots as for Firefox ones, I
> > don't want to diverge with things like those timeouts.
> 
> That makes sense, yeah. Why do the Seamonkey timeouts need to be changed,
> though?

Because Serge thinks they're too long. I personally don't care as long as we don't hang forever if we hang at all ;-)
In that case, comment #7 stands.
Ben, why does reftest has 5 mn and crashtest doesn't ? Is there (and what is) the default value ?
Attachment #370707 - Flags: review?(ted.mielczarek)
Comment on attachment 370707 [details] [diff] [review]
(Cv1-FF) Reduce timeouts to low (but safe) values

bhearsum's review is all you need here anyway.
You need to log in before you can comment on or make changes to this bug.