487380 - Mozmill features and bugs needed for Thunderbird [meta]

Status: RESOLVED WORKSFORME

(Thunderbird :: Testing Infrastructure, defect)

Description

[Ludovic Hirlimann [:Usul]]

A meta bug to track issue needed to automate testing of thunderbird with mozmill.

Comment 1

[Wayne Mery (:wsmwk)]

bug 516997 would appear to be the last blocker, whose last comment is 1.5 years ago. 

Is bug 516997 that difficult?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We simply haven't had the time yet. If it's necessary for one of you, someone might getting started on it. Keep in mind that we will flip to Marionette (webdriver protocol) most likely for Mozmill 2.1.

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #1)
> bug 516997 would appear to be the last blocker, whose last comment is 1.5
> years ago. 
> 
> Is bug 516997 that difficult?

usure why 516997 wasn't added to dependency. although it no longer matters from a practical POV

Comment 4

[Wayne Mery (:wsmwk)]

Should we skip this and just move to Marionette?  How hard would it be to leave mozmill?

"MozMill is a deprecated test tool and framework that has been superceded by Marionette. The tests are being migrated over to Marionette." https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mozmill

And a look back ... Siddharth's (aka sid0) http://www.less-broken.com/blog/2011/07/mozmill-in-thunderbird-looking-back-and.html

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Short answer: yes, move to Marionette, but you'll probably want to keep much of the mozmill in-Thunderbird test logic which means migrating it into Thunderbird.

Disclaimer: I'm very likely restating things you know, but this could be informative for others.  Also, I've very obviously been out of the Thunderbird Mozmill game for quite some time, but I was in the JSMarionette game for Firefox OS, so I have some good Marionette context.

== An Overview of Mozmill Testing ==

Mozmill proper has/had the following moving parts:
- The mozrunner logic to create a Firefox/Thunderbird profile with particular preferences and extensions and then launch it.  I think this largely still exists but might be rolled into the in-tree mozbase super-package now.
- The jsbridge extension that the mozrunner logic would install into that profile and which exposed a remoting API.  The management process (that did the mozrunner stuff) could now talk to the Thunderbird process.
- The mozmill extension logic that mozrunner installed that had most of the actual test support code, like knowing how to synthesize clicks, wait for clicks by spinning nested event loops, etc.

What's notable about the design is that it allowed the testing to either be remotely driven by python code in the "management" process, or by JS code running inside the process.  Thunderbird mozmill test logic relied almost entirely on in-Thunderbird test logic.  The management process would usually only become involved if a test explicitly wanted to restart Thunderbird.  Well, and gathering the test results and logs as they occurred.

== An Overview of Marionette Testing ==
- The mozrunner logic is still there, albeit as mozbase.
- The wire protocol for communicating with Firefox/Thunderbird is the built-in marionette protocol.

Since JSMarionette is dead/abandoned with the demise of Firefox OS (and which ran the "management" process stuff in JS using node.js), it's my understanding that the common Marionette testing idiom is to have the test logic in Python code.  However, it's still absolutely possible to use Marionette to cause JS code to be run in Thunderbird.

== Proposal ==
I'd suggest:
- Move to use the current Marionette and mozbase infrastructure to launch Thunderbird.
- Keep Thunderbird's tests as they are, largely.  The nested event loop spinning logic and the various pieces of event injection logic should continue to largely work.  As things break, simple fixes can probably be found in the fixes to the marionette in-browser logic.
- Most of the work is probably:
  - Adapting the test reporting logic, but that was already specialized for Thunderbird, so shouldn't be horrible.
  - Figuring out the easiest and least-breakage-prone way to have the mozmill JS logic accessible and bootstrapped.  As the Gecko extension infrastructure continues to change, it's not clear that's a good way.  It might be easiest to create a bootstrapping .jsm that lives in-tree that Marionette just triggers a top-level Chrome import of.  If there are security concerns, have it bail out immediately if an environment variable isn't set (and have the mozrunner logic set it).  This potentially makes debugging the whole transition easier since you can test much of the migration from inside Thunderbird via the browser console with a single import, etc.

It's still not a trivial undertaking, but I think there's a benefit to minimizing the Thunderbird-specific pieces and avoiding responsibility for maintaining old versions of mozmill, etc.

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Uh, and I'd absolutely advise against trying to port the Thunderbird tests to Python.

Comment 7

[aleth [:aleth]]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #4)
> Should we skip this and just move to Marionette?  How hard would it be to
> leave mozmill?

Moving away from mozmill would make more sense. I don't know how much work the transition would be though.

Comment 8

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #7)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #4)
> > Should we skip this and just move to Marionette?  How hard would it be to
> > leave mozmill?
> 
> Moving away from mozmill would make more sense. I don't know how much work
> the transition would be though.

Comment superseded by asuth's much more informed one above ;)

Comment 9

[Wayne Mery (:wsmwk)]

Magnus, do we have a task or bug to address Comment 5?

All the blocking bugs are closed, so closing this bug

Comment 10

[Magnus Melin [:mkmelin]]

Not really. I think most of comment 5 isn't relevant now, but maybe some of it could be useful. Geoff?

Comment 11

[Geoff Lankow (:darktrojan)]

We've basically reduced Mozmill to just the pieces we're still using, so not much useful there, no.