704859 - Refactoring of automation scripts revealed underlying issues in testrun.py

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Automation, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

With the change on bug 693306 we now no longer execute the individual testruns via subprocess.call but directly create an instance of the testrun class. Looks like we carry over settings from previous testruns.

1. If you are running the daily tests for two builds, the second build will get the update channel from the first build. That's why our Aurora builds on qa-horus have been updated to Nightly builds:

http://mozmill-release.brasstacks.mozilla.com/#/update/report/d41f7e0eb42da3cd5270e990a21ad631

2. Also it looks like we cache results and send a combined report of multiple testruns in a single report. That behavior only happened with yesterdays daily testruns across platforms but not with the todays ones:

http://mozmill-release.brasstacks.mozilla.com/#/endurance/report/d41f7e0eb42da3cd5270e990a2494e08


I will have to investigate that further and check the remaining results from todays testrun later.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yeah, somehow we are setting the 'nightly' update channel for successive builds. In this case 5.0.1 should still be on the release channel.

* 11.0a1 => 11.0a1, minor, en-US, complete, nightly, 2011-11-23, '''FAIL'''
* 5.0.1 => 5.0.1, minor, en-US, complete, nightly, 2011-11-23, '''FAIL'''

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I found the cause for the update issue. It's simply that we modify self.options.channel and set it to a new value for the first build. Successive builds in this testrun will get the same channel assigned to.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1 (update channel) [checked-in]

If anyone else comes first and want to review, I would kinda appreciate it. This has to be checked-in ASAP.

Comment 4

[David Burns :automatedtester]

Comment on attachment 576546 [details] [diff] [review]
Patch v1 (update channel) [checked-in]

looks good to me

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 576546 [details] [diff] [review]
Patch v1 (update channel) [checked-in]

Landed as:
http://hg.mozilla.org/qa/mozmill-automation/rev/3f1221d87fa6

I will now investigate the issue we have with Endurance tests, after I have updated all boxes with the correct builds again. :S I kinda would like to have the CI system.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1 (Endurance tests) [checked-in]

This fixes the endurance testrun and does not blow up the result array for each additional binary we are testing.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 576782 [details] [diff] [review]
Patch v1 (Endurance tests) [checked-in]

Landed as:
http://hg.mozilla.org/qa/mozmill-automation/rev/d6b87eec7e03

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, looks like that any regression has been fixed now. If not feel free to reopen. For now I will close this bug.