Mozmill 2.0 completely mixes up the test execution order

RESOLVED WORKSFORME

Status

Testing Graveyard
Mozmill
--
major
RESOLVED WORKSFORME
7 years ago
a year ago

People

(Reporter: whimboo, Unassigned)

Tracking

({regression})

Trunk
All
Mac OS X
regression

Details

(Whiteboard: [CLI])

(Reporter)

Description

7 years ago
Running our tests with Mozmill 2.0 completely mixes up the execution order.

Steps:
1. Run: mozmill -b %APP% -t mozmill-tests/firefox

It looks like that:

TEST-START | /Volumes/data/testing/mozmill/default/firefox/testFormManager/testAutoCompleteOff.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testFormManager/testAutoCompleteOff.js | setupModule
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testFormManager/testAutoCompleteOff.js | testAutoCompleteOff
TEST-UNEXPECTED-FAIL | /Volumes/data/testing/mozmill/default/firefox/testFormManager/testAutoCompleteOff.js | testAutoCompleteOff
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testGeneral/testGoogleSuggestions.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testGeneral/testGoogleSuggestions.js | setupModule
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testGeneral/testGoogleSuggestions.js | testGoogleSuggestedTerms
TEST-UNEXPECTED-FAIL | /Volumes/data/testing/mozmill/default/firefox/testGeneral/testGoogleSuggestions.js | testGoogleSuggestedTerms
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testInstallation/testBreakpadInstalled.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testInstallation/testBreakpadInstalled.js | setupModule
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testInstallation/testBreakpadInstalled.js | testBreakpadInstalled
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testInstallation/testBreakpadInstalled.js | testBreakpadInstalled
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testLayout/testNavigateFTP.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testLayout/testNavigateFTP.js | setupModule
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testLayout/testNavigateFTP.js | testNavigateFTP
TEST-UNEXPECTED-FAIL | /Volumes/data/testing/mozmill/default/firefox/testLayout/testNavigateFTP.js | testNavigateFTP
TEST-START | /Volumes/data/testing/mozmill/default/firefox/testSecurity/testSecurityNotification.js | setupModule
TEST-PASS | /Volumes/data/testing/mozmill/default/firefox/testSecurity/testSecurityNotification.js | setupModule

It's a regression from Mozmill 1.5.x which needs to be fixed.
Dupe or related to bug 623992 since you say this is a Mozmill 1.5.x regression?
(Reporter)

Comment 2

7 years ago
This is not related to bug 623992. It's really when you execute the tests with Mozmill itself on the local system.
I assume all OS's same testing order as presented?

Comment 4

7 years ago
I'm not sure I understand what is being said here.  The tests above seem to be in alphabetical order.  Is this not what is desired?  Additionally, I am unaware that the test order for non-restart tests was ever specced.
(Reporter)

Comment 5

7 years ago
Not sure what happend end of last week while I was testing this. There was jump from testLayout to testSecurity as what can be seen above. I'm absolutely not able to reproduce it anymore. Closing for now.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WORKSFORME
Whiteboard: [mozmill-2.0?][CLI] → [CLI]
(Assignee)

Updated

a year ago
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.