Sisyphus/JavaScript Tests - use restart mode for browser tests

RESOLVED FIXED

Status

Testing
Sisyphus
RESOLVED FIXED
11 years ago
10 years ago

People

(Reporter: bc, Assigned: bc)

Tracking

Trunk
x86
All
Points:
---
Bug Flags:
in-testsuite -

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Assignee)

Description

11 years ago
Too many times since we have switched to non-restart mode, I have had the browser tests fail to complete due to regressions on one or the other branch. This means the entire browser test run will fail if a single test crashes or fails to complete. 

Since we are running these once per day, the browser tests therefore sometimes will only find the one regression whereas the restart mode browser tests would have run to completion and reported on an unlimited number of regressions. In addition, a timeout regression which prevents a single test from completing can result in the entire test run waiting for the full five hours before it times out.

I recommend adding a -R parameter to all invocations of runtests.sh for the browser tests on each branch. This will use restart mode and allow the tests to complete regardless of regressions.
Flags: in-testsuite-
Flags: in-litmus-
(Assignee)

Comment 1

11 years ago
Created attachment 303734 [details] [diff] [review]
patch

rob, I just copied the method from class SisyphusJSTest(ShellCommand) into class SisyphusBrowserTest(SisyphusJSTest) thinking that it will inherit everything else and only override the start method.
Attachment #303734 - Flags: review?(rcampbell)
Attachment #303734 - Flags: review?(rcampbell) → review+
(Assignee)

Comment 2

11 years ago
Created attachment 303738 [details] [diff] [review]
patch v2

sorry, I should have realized that since we are doing -R now, we can include the tests which crash. I've also added the crash tests to the shell since they don't care.
Attachment #303734 - Attachment is obsolete: true
Attachment #303738 - Flags: review?(rcampbell)
Attachment #303738 - Flags: review?(rcampbell) → review+
(Assignee)

Comment 3

11 years ago
Checking in sisyphus.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/sisyphus/sisyphus.py,v  <--  sisyphus.py
new revision: 1.6; previous revision: 1.5
done

and copied to qm-jstest01.

Thanks!
Status: ASSIGNED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
(Assignee)

Comment 4

11 years ago
I missed the windows parts of this... :-(
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 5

11 years ago
Created attachment 303868 [details] [diff] [review]
patch for windows

this adds crashes to the windows shell, and crashes and restart to the windows browser tests.
Attachment #303868 - Flags: review?(rcampbell)
Attachment #303868 - Attachment is patch: true
Attachment #303868 - Attachment mime type: application/octet-stream → text/plain
Comment on attachment 303868 [details] [diff] [review]
patch for windows

yep, that should be fine. I'll find a spot today to move that onto the master.
Attachment #303868 - Flags: review?(rcampbell) → review+
Whiteboard: [install on master]
(Assignee)

Comment 7

11 years ago
/cvsroot/mozilla/tools/buildbot-configs/testing/sisyphus/sisyphus.py,v  <--  sisyphus.py
new revision: 1.7; previous revision: 1.6

I actually copied this to jstest01 and restarted everyone already. No need to do more.
Status: REOPENED → RESOLVED
Last Resolved: 11 years ago11 years ago
Resolution: --- → FIXED
(Assignee)

Updated

11 years ago
Whiteboard: [install on master]
Mass move of Sisyphus bugs to Testing : Sisyphus. Filter on SisyphusMassMove to ignore.
Component: Testing → Sisyphus
Flags: in-litmus-
Product: Core → Testing
QA Contact: testing → sisyphus
You need to log in before you can comment on or make changes to this bug.