Too many times since we have switched to non-restart mode, I have had the browser tests fail to complete due to regressions on one or the other branch. This means the entire browser test run will fail if a single test crashes or fails to complete. Since we are running these once per day, the browser tests therefore sometimes will only find the one regression whereas the restart mode browser tests would have run to completion and reported on an unlimited number of regressions. In addition, a timeout regression which prevents a single test from completing can result in the entire test run waiting for the full five hours before it times out. I recommend adding a -R parameter to all invocations of runtests.sh for the browser tests on each branch. This will use restart mode and allow the tests to complete regardless of regressions.
Created attachment 303734 [details] [diff] [review] patch rob, I just copied the method from class SisyphusJSTest(ShellCommand) into class SisyphusBrowserTest(SisyphusJSTest) thinking that it will inherit everything else and only override the start method.
Attachment #303734 - Flags: review?(rcampbell)
Attachment #303734 - Flags: review?(rcampbell) → review+
Created attachment 303738 [details] [diff] [review] patch v2 sorry, I should have realized that since we are doing -R now, we can include the tests which crash. I've also added the crash tests to the shell since they don't care.
Attachment #303738 - Flags: review?(rcampbell) → review+
Checking in sisyphus.py; /cvsroot/mozilla/tools/buildbot-configs/testing/sisyphus/sisyphus.py,v <-- sisyphus.py new revision: 1.6; previous revision: 1.5 done and copied to qm-jstest01. Thanks!
Status: ASSIGNED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
I missed the windows parts of this... :-(
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Created attachment 303868 [details] [diff] [review] patch for windows this adds crashes to the windows shell, and crashes and restart to the windows browser tests.
Attachment #303868 - Flags: review?(rcampbell)
Comment on attachment 303868 [details] [diff] [review] patch for windows yep, that should be fine. I'll find a spot today to move that onto the master.
Attachment #303868 - Flags: review?(rcampbell) → review+
/cvsroot/mozilla/tools/buildbot-configs/testing/sisyphus/sisyphus.py,v <-- sisyphus.py new revision: 1.7; previous revision: 1.6 I actually copied this to jstest01 and restarted everyone already. No need to do more.
Status: REOPENED → RESOLVED
Last Resolved: 11 years ago → 11 years ago
Resolution: --- → FIXED
Mass move of Sisyphus bugs to Testing : Sisyphus. Filter on SisyphusMassMove to ignore.
Component: Testing → Sisyphus
Product: Core → Testing
QA Contact: testing → sisyphus
You need to log in before you can comment on or make changes to this bug.