Closed Bug 625995 Opened 13 years ago Closed 13 years ago

Mozmill 2.0 completely mixes up the test execution order

Categories

(Testing Graveyard :: Mozmill, defect)

Product:
Testing Graveyard
Component:
Mozmill
Platform:
All
macOS
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: whimboo, Unassigned)

Details

(Keywords: regression, Whiteboard: [CLI]) 

Running our tests with Mozmill 2.0 completely mixes up the execution order.

Steps:
1. Run: mozmill -b %APP% -t mozmill-tests/firefox

It looks like that:

TEST-START | /Volumes/data/testing/mozmill/default/firefox/testFormManager/testAutoCompleteOff.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testFormManager/testAutoCompleteOff.js | setupModule
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testFormManager/testAutoCompleteOff.js | testAutoCompleteOff
TEST-UNEXPECTED-FAIL | /Volumes/data/testing/mozmill/default/firefox/testFormManager/testAutoCompleteOff.js | testAutoCompleteOff
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testGeneral/testGoogleSuggestions.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testGeneral/testGoogleSuggestions.js | setupModule
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testGeneral/testGoogleSuggestions.js | testGoogleSuggestedTerms
TEST-UNEXPECTED-FAIL | /Volumes/data/testing/mozmill/default/firefox/testGeneral/testGoogleSuggestions.js | testGoogleSuggestedTerms
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testInstallation/testBreakpadInstalled.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testInstallation/testBreakpadInstalled.js | setupModule
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testInstallation/testBreakpadInstalled.js | testBreakpadInstalled
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testInstallation/testBreakpadInstalled.js | testBreakpadInstalled
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testLayout/testNavigateFTP.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testLayout/testNavigateFTP.js | setupModule
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testLayout/testNavigateFTP.js | testNavigateFTP
TEST-UNEXPECTED-FAIL | /Volumes/data/testing/mozmill/default/firefox/testLayout/testNavigateFTP.js | testNavigateFTP
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testSecurity/testSecurityNotification.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testSecurity/testSecurityNotification.js | setupModule

It's a regression from Mozmill 1.5.x which needs to be fixed.
Dupe or related to bug 623992 since you say this is a Mozmill 1.5.x regression?
This is not related to bug 623992. It's really when you execute the tests with Mozmill itself on the local system.
I assume all OS's same testing order as presented?
I'm not sure I understand what is being said here.  The tests above seem to be in alphabetical order.  Is this not what is desired?  Additionally, I am unaware that the test order for non-restart tests was ever specced.
Not sure what happend end of last week while I was testing this. There was jump from testLayout to testSecurity as what can be seen above. I'm absolutely not able to reproduce it anymore. Closing for now.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → WORKSFORME
Whiteboard: [mozmill-2.0?][CLI] → [CLI]
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.