Note: There are a few cases of duplicates in user autocompletion which are being worked on.

Automate Thunderbird Litmus Tests using Mozmill

RESOLVED WONTFIX

Status

Thunderbird
General
RESOLVED WONTFIX
8 years ago
5 years ago

People

(Reporter: gkw, Unassigned)

Tracking

(Depends on: 1 bug, {student-project})

Trunk
student-project
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 2 obsolete attachments)

(Reporter)

Description

8 years ago
We should start automating litmus tests by using Mozmill.
(Reporter)

Updated

8 years ago
Assignee: nobody → nth10sd
Depends on: 479311, 487055
(Reporter)

Updated

8 years ago
Summary: Automate Litmus Tests using Mozmill → Automate Thunderbird Litmus Tests using Mozmill
(Reporter)

Updated

8 years ago
Depends on: 487062
(Reporter)

Comment 1

8 years ago
Created attachment 371273 [details]
not-yet-a-full-port of Litmus test 5576

For backup.

https://litmus.mozilla.org/show_test.cgi?id=5576 is the litmus test.
(Reporter)

Comment 2

8 years ago
Created attachment 371442 [details]
Mozmill test to open up the Activity Manager

From a not-yet-written Litmus test.
(Reporter)

Comment 3

8 years ago
Notes to self:

- Mozmill tests relying on modal windows (which TB has a lot of) are heavily dependent on bug 479311.
- Another test at https://bugzilla.mozilla.org/show_bug.cgi?id=471510#c5
(Reporter)

Updated

8 years ago
Depends on: 477079
(Reporter)

Updated

8 years ago
Depends on: 487380
Adding bug 458352 to the deps as that includes potential scripts and discussions we've had on what to do about actually scripting getting tests running automatically.
Depends on: 458352
(Reporter)

Updated

8 years ago
Depends on: 488315

Updated

8 years ago
OS: Mac OS X → All
Hardware: x86 → All
(Reporter)

Comment 5

8 years ago
Created attachment 376628 [details]
Litmus test 5576 - Now supports simulated triggering of links inside panes

In the case of 5576, has working support of clicking "Manage Message Filters" in Account Central.

Notes:
a) Templatized DOM tree support not yet implemented in Mozmill - Bug 477079
b) Modal dialogs are supported in Mozmill, but I couldn't yet make it work, so code is commented out.
c) A bunch of steps are still coded to be manually clicked, some email account names and emails are assumed to be of a particular name till at least bug 458352 is fixed.
Attachment #371273 - Attachment is obsolete: true
(Reporter)

Updated

8 years ago
Attachment #376628 - Attachment description: Now supports simulated triggering of links inside panes → Litmus test 5576 - Now supports simulated triggering of links inside panes
(Reporter)

Comment 6

8 years ago
Now left blocking significantly on bug 477079 for clicking on the left folder selection tree.
(Reporter)

Updated

8 years ago
Depends on: 486407
I'd like to use the following to store the test cases into the tree :

 /mail/test/mozmill/
                    litmus-litmustestid/test.js

and I was thinking also to store some profiles at the same location so they could be shared for our tests.

Mark comments ?
As a developer, a litmus test id has even less meaning to me than a bugzilla bug number.  I would suggest that we try and name things to provide as much possible info about what the test does in the file name and directory name.

For example, we have a number of xpcshell tests that are based on bug numbers in mailnews/base/test/unit.  I have to look inside them or visit the bug to figure out what they do.

I think bug numbers and litmus test numbers are potentially useful information, but if we are including them, I propose we append them onto a more useful description of what they do, and categorize them based on the directory they go in.  For example "account-dialog/deletion-settings-litmus12345.js" is useful to me.  If you need each litmus test to have its own directory, there is nothing in the mozmill idiom that prevents us from having a more nested directory structure.  (So "litmus/account-dialog/deletion-settings-litmus12345.js").

If the mozmill test is permanently replacing a litmus test so that we never need to manually run it again, I'm not sure we even need to put the litmus test number in the name.  In that case I think it would be sufficient to just put a comment in the test, like "// litmus test #12345".  Otherwise organizing tests along a strict litmus basis might create a test burden that isn't really required.
Er, in the own directory example, I glitched, I meant to say:
litmus/account-dialog/deletion-settings-litmus12345/test.js

And of course, in the first example, I meant
account-dialog/test-deletion-settings-litmus12345.js
(In reply to comment #8)
> For example, we have a number of xpcshell tests that are based on bug numbers
> in mailnews/base/test/unit.  I have to look inside them or visit the bug to
> figure out what they do.
> 
> I think bug numbers and litmus test numbers are potentially useful information,
> but if we are including them, I propose we append them onto a more useful
> description of what they do, and categorize them based on the directory they go
> in.  For example "account-dialog/deletion-settings-litmus12345.js" is useful to
> me.  If you need each litmus test to have its own directory, there is nothing
> in the mozmill idiom that prevents us from having a more nested directory
> structure.  (So "litmus/account-dialog/deletion-settings-litmus12345.js").

So this needs to be taken into account and a description is way better than any number scheme we can come up with. Issues will end up probably like in litmus with non very descriptive Test Case (TC) - but better than numbers. Let's just drop them - as the only function I had in mind for them was to easily know which TC have been moved from litmus to mozmill.
 
> If the mozmill test is permanently replacing a litmus test so that we never
> need to manually run it again, I'm not sure we even need to put the litmus test
> number in the name.  In that case I think it would be sufficient to just put a
> comment in the test, like "// litmus test #12345".  Otherwise organizing tests
> along a strict litmus basis might create a test burden that isn't really
> required.

Yeah that would do. Thanks for the input andrew.
(Reporter)

Comment 11

8 years ago
Created attachment 393771 [details] [diff] [review]
Litmus test 5576 that runs perfectly in the Mozmill harness - steps 1 & 2

Here's the mozmill test for steps 1 & 2 - step 3 is not yet done because I haven't yet have time to figure out the added complexity needed for that step.

https://litmus.mozilla.org/show_test.cgi?id=5576

It runs perfectly in the mozmill harness.
Attachment #376628 - Attachment is obsolete: true
Attachment #393771 - Flags: review?(bugmail)
Comment on attachment 393771 [details] [diff] [review]
Litmus test 5576 that runs perfectly in the Mozmill harness - steps 1 & 2

I was mis-using the review flag to keep this on my radar to fix up since my understanding was gary was not available to perform further iterations on the test.

I'm r-'ing the patch because the dependency on short sleeps will break on the tinderboxen.  Also, I'd personally prefer that we avoid the giant automated XPath blob.  We have sufficiently stable element id's that they should be directly usable and it's more clear why a test might have failed.
Attachment #393771 - Flags: review?(bugmail) → review-
(Reporter)

Comment 13

8 years ago
This bug goes back to the pool - haven't been thinking about this and don't think I'll have time to work on this.
Assignee: gary → nobody
(Reporter)

Comment 14

8 years ago
(In reply to comment #13)
> This bug goes back to the pool - haven't been thinking about this and don't
> think I'll have time to work on this.

Also seems like a perfect student-project, btw.
Keywords: student-project
(Reporter)

Comment 15

8 years ago
Simon might be taking this on as a student project - assigning to him.
Assignee: nobody → teochangchiew
Status: NEW → ASSIGNED
(Reporter)

Updated

7 years ago
Assignee: teochangchiew → nobody
Status: ASSIGNED → NEW
(Reporter)

Comment 16

5 years ago
Litmus will be replaced by MozTrap ( https://moztrap.mozilla.org ), and it will be decommissioned soon. See bug 802674.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
isn't the premise of automating our manual tests still valid?
And preferable to moving all manual tests to moztrap?
(Reporter)

Comment 18

5 years ago
(In reply to Wayne Mery (:wsmwk) from comment #17)
> isn't the premise of automating our manual tests still valid?
> And preferable to moving all manual tests to moztrap?

Yes, perhaps the premise would be, but probably in another bug since there's a lot of litmus stuff here.
filed bug 802990
You need to log in before you can comment on or make changes to this bug.