589003 - Mozmill calls sys.exit() if test failures occur - restart-test run doesn't get executed

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Automation, defect)

Description

[u279076]

I'm not sure why but I noticed today, during a 4.0b4 test run on all platforms, that restart tests no longer run.  I see no errors in the logs or on brasstacks.  The tests just do not execute.  It is almost as if they don't exist.

Looking back at the brasstacks logs I noticed the restart tests have not run for at least a couple of days.  The following is date the most recent report of a restart-test testrun appears:

default: -> 2010-08-12
mozilla1.9.2 -> 2010-08-18
mozilla1.9.1 -> 2010-08-19

As you can see, this primarily affects the default branch with no results appearing in about a week.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

According to the logs on brasstacks the restart tests seem to work fine for 3.6 and 3.5 builds. Only for 4.0 builds we do not have any result. This is strange. I will investigate.

Comment 2

[Al Billings [:abillings - ex-MoCo]]

This is breaking on 3.5 right now doing 3.5.12 testing. I've seen failures on Windows 7 and Ubuntu 9.10 there.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It looks like to be a regression from bug 586336 which I had to land to solve the Mozmill refactoring failure in our scripts introduced by bug 579791.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

When I only run a single sub folder below the firefox folder for normal tests, the restart tests get started. Running the complete suite of tests raise this problem. I will have to trim down the list of sub folders and check which specific tests are responsible for it.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's somehow related to the testSecurity sub folder. One of those tests is causing this failure.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For the security tests it happens when the tests testIdentityPopupOpenClose.js and testSecurityNotification.js get run. Now that I can reproduce it easily, I can check what's wrong with our automation scripts or if it's a Mozmill issue.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, the reason why it doesn't happen everywhere is that not all systems are running Python 2.6 yet. With this version the exception handling has been changed. Only when a test failure occurs in the normal test run we fail and don't run the restart tests. Bug 568726 should fix this problem.

        try:
            self.restart_tests = False
            self.test_path = os.path.join('firefox')
            TestRun.run_tests(self)
>       except Exception, e:
            print e

        try:
            self.restart_tests = True
            self.test_path = os.path.join('firefox','restartTests')
            TestRun.run_tests(self)
        except Exception, e:
            print e


Al, have you updated the machines in the 2nd week of August? Looks like that a system update has installed a new Python version, i.e. for Ubuntu.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Bug 579791 is the real bug which caused this failure in our scripts.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

No need to rework all the exception handling for now. I simply have to fetch the SystemExit exception and scratch it.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1

This patch catches the SystemExit exception directly in the MozmillWrapper now and throws it away. Other general exceptions are still catched and printed to the console/log. That means we always run following test-runs whether the current one passes or fails.

Comment 11

[Jeff Hammel]

Comment on attachment 469101 [details] [diff] [review]
Patch v1

looks good

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed as:
http://hg.mozilla.org/qa/mozmill-automation/rev/f7d1f75bf28d

I have updated all of our machines in the QA lab to have the latest version of the automation scripts installed.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Had to push a follow-up to fix the breakage with Python 2.5:
http://hg.mozilla.org/qa/mozmill-automation/rev/8710add372e4

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Move of Mozmill related project bugs to newly created components. You can
filter out those emails by using "Mozmill-Tests-to-MozillaQA" as criteria.