Closed Bug 533246 Opened 12 years ago Closed 12 years ago

Support latest Mozmill version in Thunderbird tests.

Categories

(Thunderbird :: Testing Infrastructure, defect)

defect
Not set
normal

Tracking

(thunderbird3.1 beta1-fixed, thunderbird3.0 .2-fixed)

RESOLVED FIXED
Thunderbird 3.1b1
Tracking Status
thunderbird3.1 --- beta1-fixed
thunderbird3.0 --- .2-fixed

People

(Reporter: standard8, Assigned: asuth)

References

Details

Attachments

(1 file, 2 obsolete files)

Mozmill 1.3 was released a while ago, and last I heard we didn't support it. This bug will do the work necessary to upgrade our scripts to it.
A lot of stuff changed.  This is not remotely backwards compatible.

Should we try and conditionalize / use a different script based on the installed mozmill version?

The bad news is that the newer version of mozmill is not really helping my focus problems out.
Assignee: nobody → bugmail
Status: NEW → ASSIGNED
Attachment #417699 - Flags: review?(bugzilla)
Component: Build Config → Testing Infrastructure
QA Contact: build-config → testing-infrastructure
Blocks: 538452
I've fixed the slight bitrot.
Attachment #417699 - Attachment is obsolete: true
Attachment #421210 - Flags: review?(bugzilla)
Attachment #417699 - Flags: review?(bugzilla)
We need to get the maxversion in the mozmill extension's chrome.manifest bumped up.
I've submitted a pull request on github.
Blocks: 540259
In an ideal world, all of our existing tests would work nicely either mozmill 1.2 and mozmill 1.3, and we could allow both trunk and 1.9.2 to support both.  Sid, have you tried (or can you try) a full test-run with mozmill 1.3 and see if that's anywhere close to possible?
This is needed for a full |make mozmill| run.
Attachment #421210 - Attachment is obsolete: true
Attachment #422483 - Flags: review?(bugzilla)
Attachment #421210 - Flags: review?(bugzilla)
(In reply to comment #5)
> In an ideal world, all of our existing tests would work nicely either mozmill
> 1.2 and mozmill 1.3, and we could allow both trunk and 1.9.2 to support both. 
> Sid, have you tried (or can you try) a full test-run with mozmill 1.3 and see
> if that's anywhere close to possible?

A full test run with Mozmill 1.3 + the above patch looks like it works fine on Windows. There don't seem to be any API changes that affect tests themselves. This is borne out by ctalbert's post in the newsgroup -- I don't think we use any of the APIs mentioned there.

I wouldn't be in favour of supporting both Mozmill 1.2 and Mozmill 1.3, because:
- Mozmill 1.3 seems more reliable. Mozmill 1.2 used to crash quite often at shutdown -- I haven't seen it with 1.3 yet.
- I haven't confirmed it yet, but it seemed like an issue where anything printed to stdout got eaten up on Linux (thus making dump() from chrome code useless) looks fixed.
- As asuth mentioned, the surrounding python harness code has changed a lot, and trying to support both Mozmill 1.2 and Mozmill 1.3 will require maintaining two copies of runtest.py, and testing on both each time a change is made to the harness. I think it's a good idea to keep our "test testing matrix" simple, even if our testing matrix isn't. :)
Agreed that it seems hard to justify the extra complexity that supporting both would imply.  Supposing we do switch to entirely to 1.3, what does that look like?  Steps that I can imagine include:

* announce our plan
* get this patch reviewed 
* simultaneously:
** update the wiki page
** update all the test tinderboxen
** land this patch

Is there anything I've left out?

Presumably this is less of a hassle to do before comm-1.9.2 is cut, and but not a particularly big hassle to do afterwards either.  Is that actually true?
Adding gozer since he'll be interested in the tinderbox bit, I suspect.
Comment on attachment 422483 [details] [diff] [review]
fix runtestlist.py too

r=Standard8 but I'll arrange check in time with gozer & others.
Attachment #422483 - Flags: review?(bugzilla) → review+
Notes: I tested the patch with:

easy_install http://pypi.python.org/packages/source/m/mozrunner/mozrunner-2.4.tar.gz#md5=73f19c06bb1986e8038fb8088e6babe9
easy_install http://pypi.python.org/packages/source/j/jsbridge/jsbridge-2.3.4.tar.gz#md5=f880ab9330caf00a92146188372f3687
easy_install http://pypi.python.org/packages/source/m/mozmill/mozmill-1.4.tar.gz#d649c0634a04a9433c4792ca0a503068

I have set up instructions for deploying this on our builders at:

https://wiki.mozilla.org/Thunderbird/Infrastructure/Builder_Installations#MozMill

(they can also be used for dev installs).

Still todo:

- Post to mdat & co about upgrade; I'll do this once I've spoken to gozer about a deployment time.
- Deploy (i.e. land, update buildbots) & update devmo page.
Blocks: 506929
Blocks: 545085
Comment on attachment 422483 [details] [diff] [review]
fix runtestlist.py too

Due to needing to upgrade all the builders at the same type, we're going to be landing this patch in sync with the trunk patch. a=Standard8 as its effectively NPOTB anyway.
Attachment #422483 - Flags: approval-thunderbird3.0.2+
Checked in:
http://hg.mozilla.org/comm-central/rev/2ce6e173b23c
http://hg.mozilla.org/releases/comm-1.9.1/rev/53f64fe1c09d
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 3.1b1
Verifying MozMill test harness fix for 3.0.2 on the basis that MozMill tests are still running fine and no-one has complained.
You need to log in before you can comment on or make changes to this bug.