Closed Bug 589003 Opened 12 years ago Closed 12 years ago

Mozmill calls sys.exit() if test failures occur - restart-test run doesn't get executed

Categories

(Mozilla QA Graveyard :: Mozmill Automation, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: u279076, Assigned: whimboo)

References

Details

(Keywords: regression)

Attachments

(1 file)

I'm not sure why but I noticed today, during a 4.0b4 test run on all platforms, that restart tests no longer run.  I see no errors in the logs or on brasstacks.  The tests just do not execute.  It is almost as if they don't exist.

Looking back at the brasstacks logs I noticed the restart tests have not run for at least a couple of days.  The following is date the most recent report of a restart-test testrun appears:

default: -> 2010-08-12
mozilla1.9.2 -> 2010-08-18
mozilla1.9.1 -> 2010-08-19

As you can see, this primarily affects the default branch with no results appearing in about a week.
Summary: Restart Tests not running as part of test-general.py testrun → Restart Tests not running as part of BFT testrun
According to the logs on brasstacks the restart tests seem to work fine for 3.6 and 3.5 builds. Only for 4.0 builds we do not have any result. This is strange. I will investigate.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
This is breaking on 3.5 right now doing 3.5.12 testing. I've seen failures on Windows 7 and Ubuntu 9.10 there.
It looks like to be a regression from bug 586336 which I had to land to solve the Mozmill refactoring failure in our scripts introduced by bug 579791.
Blocks: 586336
Keywords: regression
When I only run a single sub folder below the firefox folder for normal tests, the restart tests get started. Running the complete suite of tests raise this problem. I will have to trim down the list of sub folders and check which specific tests are responsible for it.
It's somehow related to the testSecurity sub folder. One of those tests is causing this failure.
For the security tests it happens when the tests testIdentityPopupOpenClose.js and testSecurityNotification.js get run. Now that I can reproduce it easily, I can check what's wrong with our automation scripts or if it's a Mozmill issue.
Ok, the reason why it doesn't happen everywhere is that not all systems are running Python 2.6 yet. With this version the exception handling has been changed. Only when a test failure occurs in the normal test run we fail and don't run the restart tests. Bug 568726 should fix this problem.

        try:
            self.restart_tests = False
            self.test_path = os.path.join('firefox')
            TestRun.run_tests(self)
>       except Exception, e:
            print e

        try:
            self.restart_tests = True
            self.test_path = os.path.join('firefox','restartTests')
            TestRun.run_tests(self)
        except Exception, e:
            print e


Al, have you updated the machines in the 2nd week of August? Looks like that a system update has installed a new Python version, i.e. for Ubuntu.
Depends on: 568726
Whiteboard: [mozmill-automation]
Bug 579791 is the real bug which caused this failure in our scripts.
Blocks: 579791
No longer blocks: 586336
No need to rework all the exception handling for now. I simply have to fetch the SystemExit exception and scratch it.
No longer depends on: 568726
Summary: Restart Tests not running as part of BFT testrun → Mozmill calls sys.exit() if test failures occur - restart-test run doesn't get executed
Attached patch Patch v1Splinter Review
This patch catches the SystemExit exception directly in the MozmillWrapper now and throws it away. Other general exceptions are still catched and printed to the console/log. That means we always run following test-runs whether the current one passes or fails.
Attachment #469101 - Flags: review?(jhammel)
Comment on attachment 469101 [details] [diff] [review]
Patch v1

looks good
Attachment #469101 - Flags: review?(jhammel) → review+
Landed as:
http://hg.mozilla.org/qa/mozmill-automation/rev/f7d1f75bf28d

I have updated all of our machines in the QA lab to have the latest version of the automation scripts installed.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Had to push a follow-up to fix the breakage with Python 2.5:
http://hg.mozilla.org/qa/mozmill-automation/rev/8710add372e4
Move of Mozmill related project bugs to newly created components. You can
filter out those emails by using "Mozmill-Tests-to-MozillaQA" as criteria.
Component: Mozmill Tests → Mozmill Automation
Product: Testing → Mozilla QA
QA Contact: mozmill-tests → mozmill-automation
Whiteboard: [mozmill-automation]
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.