487041 - Automate Thunderbird Litmus Tests using Mozmill

Status: RESOLVED WONTFIX

(Thunderbird :: General, defect)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

We should start automating litmus tests by using Mozmill.

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: not-yet-a-full-port of Litmus test 5576

For backup.

https://litmus.mozilla.org/show_test.cgi?id=5576 is the litmus test.

Comment 2

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: Mozmill test to open up the Activity Manager

From a not-yet-written Litmus test.

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Notes to self:

- Mozmill tests relying on modal windows (which TB has a lot of) are heavily dependent on bug 479311.
- Another test at https://bugzilla.mozilla.org/show_bug.cgi?id=471510#c5

Comment 4

[Mark Banner (:standard8)]

Adding bug 458352 to the deps as that includes potential scripts and discussions we've had on what to do about actually scripting getting tests running automatically.

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: Litmus test 5576 - Now supports simulated triggering of links inside panes

In the case of 5576, has working support of clicking "Manage Message Filters" in Account Central.

Notes:
a) Templatized DOM tree support not yet implemented in Mozmill - Bug 477079
b) Modal dialogs are supported in Mozmill, but I couldn't yet make it work, so code is commented out.
c) A bunch of steps are still coded to be manually clicked, some email account names and emails are assumed to be of a particular name till at least bug 458352 is fixed.

Comment 6

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Now left blocking significantly on bug 477079 for clicking on the left folder selection tree.

Comment 7

[Ludovic Hirlimann [:Usul]]

I'd like to use the following to store the test cases into the tree :

 /mail/test/mozmill/
                    litmus-litmustestid/test.js

and I was thinking also to store some profiles at the same location so they could be shared for our tests.

Mark comments ?

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

As a developer, a litmus test id has even less meaning to me than a bugzilla bug number.  I would suggest that we try and name things to provide as much possible info about what the test does in the file name and directory name.

For example, we have a number of xpcshell tests that are based on bug numbers in mailnews/base/test/unit.  I have to look inside them or visit the bug to figure out what they do.

I think bug numbers and litmus test numbers are potentially useful information, but if we are including them, I propose we append them onto a more useful description of what they do, and categorize them based on the directory they go in.  For example "account-dialog/deletion-settings-litmus12345.js" is useful to me.  If you need each litmus test to have its own directory, there is nothing in the mozmill idiom that prevents us from having a more nested directory structure.  (So "litmus/account-dialog/deletion-settings-litmus12345.js").

If the mozmill test is permanently replacing a litmus test so that we never need to manually run it again, I'm not sure we even need to put the litmus test number in the name.  In that case I think it would be sufficient to just put a comment in the test, like "// litmus test #12345".  Otherwise organizing tests along a strict litmus basis might create a test burden that isn't really required.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Er, in the own directory example, I glitched, I meant to say:
litmus/account-dialog/deletion-settings-litmus12345/test.js

And of course, in the first example, I meant
account-dialog/test-deletion-settings-litmus12345.js

Comment 10

[Ludovic Hirlimann [:Usul]]

(In reply to comment #8)
> For example, we have a number of xpcshell tests that are based on bug numbers
> in mailnews/base/test/unit.  I have to look inside them or visit the bug to
> figure out what they do.
> 
> I think bug numbers and litmus test numbers are potentially useful information,
> but if we are including them, I propose we append them onto a more useful
> description of what they do, and categorize them based on the directory they go
> in.  For example "account-dialog/deletion-settings-litmus12345.js" is useful to
> me.  If you need each litmus test to have its own directory, there is nothing
> in the mozmill idiom that prevents us from having a more nested directory
> structure.  (So "litmus/account-dialog/deletion-settings-litmus12345.js").

So this needs to be taken into account and a description is way better than any number scheme we can come up with. Issues will end up probably like in litmus with non very descriptive Test Case (TC) - but better than numbers. Let's just drop them - as the only function I had in mind for them was to easily know which TC have been moved from litmus to mozmill.
 
> If the mozmill test is permanently replacing a litmus test so that we never
> need to manually run it again, I'm not sure we even need to put the litmus test
> number in the name.  In that case I think it would be sufficient to just put a
> comment in the test, like "// litmus test #12345".  Otherwise organizing tests
> along a strict litmus basis might create a test burden that isn't really
> required.

Yeah that would do. Thanks for the input andrew.

Comment 11

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: Litmus test 5576 that runs perfectly in the Mozmill harness - steps 1 & 2

Here's the mozmill test for steps 1 & 2 - step 3 is not yet done because I haven't yet have time to figure out the added complexity needed for that step.

https://litmus.mozilla.org/show_test.cgi?id=5576

It runs perfectly in the mozmill harness.

Comment 12

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 393771 [details] [diff] [review]
Litmus test 5576 that runs perfectly in the Mozmill harness - steps 1 & 2

I was mis-using the review flag to keep this on my radar to fix up since my understanding was gary was not available to perform further iterations on the test.

I'm r-'ing the patch because the dependency on short sleeps will break on the tinderboxen.  Also, I'd personally prefer that we avoid the giant automated XPath blob.  We have sufficiently stable element id's that they should be directly usable and it's more clear why a test might have failed.

Comment 13

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

This bug goes back to the pool - haven't been thinking about this and don't think I'll have time to work on this.

Comment 14

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #13)
> This bug goes back to the pool - haven't been thinking about this and don't
> think I'll have time to work on this.

Also seems like a perfect student-project, btw.

Comment 15

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Simon might be taking this on as a student project - assigning to him.

Comment 16

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Litmus will be replaced by MozTrap ( https://moztrap.mozilla.org ), and it will be decommissioned soon. See bug 802674.

Comment 17

[Wayne Mery (:wsmwk)]

isn't the premise of automating our manual tests still valid?
And preferable to moving all manual tests to moztrap?

Comment 18

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Wayne Mery (:wsmwk) from comment #17)
> isn't the premise of automating our manual tests still valid?
> And preferable to moving all manual tests to moztrap?

Yes, perhaps the premise would be, but probably in another bug since there's a lot of litmus stuff here.

Comment 19

[Wayne Mery (:wsmwk)]

filed bug 802990