500469 - Mozmill tests for Calendar

Status: RESOLVED FIXED

(Calendar :: General, defect)

Description

[Merike Sell (:merike)]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; et; rv:1.9.1) Gecko/20090616 Firefox/3.5
Build Identifier: 

This bug is needed to track writing testscripts specified here: https://wiki.mozilla.org/Calendar:QA/Automation_Wish_List

Reproducible: Always

Comment 1

[Merike Sell (:merike)]

Attachment: Smoke test v1

Comment 2

[cmtalbert]

Comment on attachment 385167 [details]
Smoke test v1

r=ctalbert, This looks great.  I would change its name from testSmokeTest to something more descriptive that tells us what it is testing.  Perhaps testCalendarUI since this test just goes through the UI and ensures that everything is visible and working properly.  Great first effort!

Comment 3

[cmtalbert]

Also, going to confirm this bug.

Comment 4

[Merike Sell (:merike)]

Attachment: Day view test

Comment 5

[Merike Sell (:merike)]

Attachment: Week view test

Comment 6

[Merike Sell (:merike)]

Attachment: Test multiweek view

Comment 7

[Merike Sell (:merike)]

Attachment: Test month view

For some reason view won't switch if multiweek view is the last view used before running the test.

Comment 8

[Merike Sell (:merike)]

Smoke test will need the following changed
-controller.rightClick(new elementslib.Lookup(controller.window.document, path 
    + '/id("ltnSidebar")/id("calendar-panel")/id("calendar-list-pane")/id("calendar-listtree-pane")/'
    + 'id("calendar-list-tree-widget")/anon({"anonid":"tree"})/anon({"anonid":"treechildren"})'));
+controller.rightClick(new elementslib.Lookup(controller.window.document, path 
    + '/id("ltnSidebar")/id("calendar-panel")/id("calendar-list-pane")/id("calendar-listtree-pane")/'
    + 'id("calendar-list-tree-widget")/anon({"anonid":"tree"})/anon({"anonid":"treechildren"})'));
+  controller.click(new elementslib.ID(controller.window.document, "list-calendars-context-new"));

This is because doubleclick now specifies its position and 0,0 trigger default Home calendar modification dialog on this treeview and the test never finishes. It also needs either a name that will be alphabetically the first when running test folder commandline or run separately as it needs to be run first to create Mozmill calendar.

Comment 9

[cmtalbert]

Comment on attachment 387513 [details]
Day view test

Looks good.  You need to add yourself as a contributor on all these tests, so put two spaces to indent yourself under "Contributor:" in the license and type in your information in this format: "Clint Talbert <[e-mail]>" otherwise, looks great.

Comment 10

[cmtalbert]

Comment on attachment 387640 [details]
Week view test

r+ on the approach.  But, what do you mean by "Hangs the application" in the "Delete Event" comment?  If it is hanging the app, we need to know what is going on there.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 385167 [details]
Smoke test v1

Let me just throw in some general comments which will apply to all tests which looks indeed great.

>var mozmill = {}; Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
>var elementslib = {}; Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);

Both lines aren't needed anymore. The import will be made by Mozmill itself.

>  module.modalDialogAPI = collector.getModule('ModalDialogAPI');

The collector also assigns collected modules on its own. There is no need anymore for this line of code. You can access shared functions the same way as you already do.

>function handleDialog(controller) { 
>  let wizardPath = '/id("calendar-wizard")/anon({"anonid":"Buttons"})/'
>    + 'anon({"class":"wizard-buttons-box-1"})/{"class":"wizard-buttons-box-2"}/'
>    + 'anon({"anonid":"WizardButtonDeck"})/';
>  
>  // click next
>  controller.click(new elementslib.Lookup(controller.window.document, wizardPath
>    + '[1]/{"dlgtype":"next"}'));

Under some situations it can happen that you will be too fast here. Means wizardPath doesn't yet exist. For Firefox we are using waitThenClick just to make sure we really click an existent item.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 387513 [details]
Day view test

>  // pick day
>  controller.click(new elementslib.Lookup(controller.window.document, miniMonth
>    + 'anon({"anonid":"minimonth-calendar"})/[1]/{"value":"1"}'));
>  controller.sleep(5000);

Why do you need such a long delay?

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 388275 [details]
Test multiweek view

>  controller.assertJS(day.getNode().mDate.icalString == "20090101");

Doesn't it always pass true/false to the assertJS function? Do you use it to throw a failure instead of doing it on your own with throw? Looks to be nice way we should also adopt for our Firefox tests.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 388276 [details]
Test month view

>  // check if name was saved
>  controller.assertProperty(new elementslib.Lookup(controller.window.document, eventBox
>    + '/{"flex":"1"}/anon({"anonid":"event-name"})/anon({"anonid":"event-name-div"})'),
>    "textContent",
>    title2);

Isn't that simply assertText?

Comment 15

[Merike Sell (:merike)]

Attachment: Day view test v2

Fixed licence, odd sleep and changed from assertProperty to assertText. Added sleep after delete to be safer and removed imports.

And yes, assertJS gives you usual mozmill pass or fail, no need to throw yourself.

Comment 16

[Merike Sell (:merike)]

Attachment: Week view test v2

Fixed licence, removed imports, assertText. Hanging comment isn't valid any more, I had a mistake in lookup there.

Comment 17

[Merike Sell (:merike)]

Attachment: Today pane test

The test currently fails because click on - sign to collapse tomorrow or soon takes sometimes two clicks to work. Seems that this only happens when an event is focused right before the click. I'm unsure if this should be ui bug or not but I'd expect + and - signs to act on first click independent of what is focused.

Comment 18

[Merike Sell (:merike)]

Attachment: Multiweek view test v2

Apparently event start time logic is different if current time is between 23:00 and 00:00.

Comment 19

[Merike Sell (:merike)]

Attachment: Month view test v2

Also fixing event start time.

Comment 20

[Merike Sell (:merike)]

Attachment: Event dialog test v1

Once the attendee adding and popup closing issues get sorted it should run without problems. It may need adjustments to sleep durations though since I tested actions in smaller groups and not whole test at a time.

Comment 21

[cmtalbert]

Comment on attachment 389517 [details]
Today pane test

  
>  // verify tomorrow and soon collapsed
>  let tomorrow = (new elementslib.Lookup(controller.window.document,
>    '/id("messengerWindow")/id("tabmail-container")/id("today-pane-panel")/[1]/id("agenda-panel")/'
>    + '{"flex":"1"}/id("agenda-listbox")/[1]/anon({"class":"agenda-checkbox"})')).getNode();
>  let soon = (new elementslib.Lookup(controller.window.document,
>    '/id("messengerWindow")/id("tabmail-container")/id("today-pane-panel")/[1]/id("agenda-panel")/'
>    + '{"flex":"1"}/id("agenda-listbox")/[2]/anon({"class":"agenda-checkbox"})')).getNode();
>  controller.assertJS(tomorrow.getAttribute("checked") == "false");
>  controller.assertJS(soon.getAttribute("checked") == "false");

The way the checked attribute works is not exactly intuitive.  If a node as checked with a value of 'true' then the node is checked.  If the node has a value of checked with ANY other value OR if the checked attribute is removed, then the node is considered 'unchecked'.  So, this test runs great except these asserts fail because when the calendar code 'unchecks' the item, it removes the checked attribute rather than setting it to false.  So I think these should be changed to assert that the 'checked' attribute has been removed.

Also, the double-click needed on the ">" item to expand the node's selection seems to be a product bug.  I can make that happen just manually.  Would you mind filing a Calendar/UI bug for that?

Otherwise, this is an awesome test. I like the way you handled the scrolling of the day view to create the events.  Very nicely done.

r=ctalbert with those changes.

Comment 22

[Merike Sell (:merike)]

Attachment: Today pane test v2

Comment 23

[cmtalbert]

Comment on attachment 390184 [details]
Event dialog test v1


>var sleep = 500;
>var calendar = "Mozmill";
I think you should use "Home" as that is the default name of the default calendar.


>var testEventDialog = function () {

What follows is all very date intensive.  I wonder if there is a way to generalize this type of code such that other people writing calendar tests can make use of it.  I'm not sure there is unless we wire the test into the view creation mechanism which is not something we'd want.  Merike, any ideas?  This is not too important, and is not required for r+, but it is something that is worth thinking about.
>  // 31st of January is Saturday so there's four more full rows to check
>  let date = 4;
>  for(row = 1; row < 5; row++){
>    for(col = 0; col < 7; col++){
>        controller.assertNode(new elementslib.Lookup(controller.window.document, 
>          eventBox.replace("rowNumber", row).replace("columnNumber", col)));
>        checkIcon(eventBox, row, col);
>        checkTooltip(monthView, row, col, date, startTime, endTime);
>        date++;
>    }
>  }
>  
>  // delete and verify deleted 2nd Jan
>  let md4 = new ModalDialogAPI.modalDialog(handleOcurrenceDeletion);
>  md4.start();
>  controller.click(new elementslib.Lookup(controller.window.document, 
>    eventBox.replace("rowNumber", "0").replace("columnNumber", "5")));
>  controller.keypress(new elementslib.ID(controller.window.document, "month-view"),
>    "VK_DELETE", {});
>  controller.sleep;
>  
>
The following handle* functions might all make for good calendar shared modules, but they are so specific to calendar, I'm not entirely sure how to package them up (and you'd have to use modal dialog as well). It might be something good to think about though.   Maybe we should put them in a calendar-utils file inside the shared-modules folder?

>function handleAttendees(attendees){
>
>function handleAttachment(attachment){
>
>function handleMail(dialog){
>
>function handleOcurrenceDeletion(dialog){

>function handleParentDeletion(dialog){

>function checkIcon(eventBox, row, col){

>function checkTooltip(monthView, row, col, date, startTime, endTime){

It looks good; however, the test will simply not run for me all the way through, and I'm not sure what is wrong.  It keeps hanging on the attendee creation step.

I'm going to r- on this one for now as it doesn't run.  For review, you'll need to make the change to the "home" calendar, repack the handle* functions into a shared module.  I don't want other people to have to copy/paste or worse, re-write, these functions when they are writing tests. And of course, we have to figure out why it isn't running all the way through.

Comment 24

[Merike Sell (:merike)]

(In reply to comment #23)
> (From update of attachment 390184 [details])
> I think you should use "Home" as that is the default name of the default
> calendar.
And not specifically created Mozmill calendar as in all other tests? Why?

> What follows is all very date intensive.  I wonder if there is a way to
> generalize this type of code such that other people writing calendar tests can
> make use of it.  I'm not sure there is unless we wire the test into the view
> creation mechanism which is not something we'd want.  Merike, any ideas?  This
> is not too important, and is not required for r+, but it is something that is
> worth thinking about.
I can't see a good solution to this. The combinations of day boxes that people need to test will vary and so will the information in tooltips and event boxes.
It would be nice if there were a function along the lines "verify event present/not present from x to y" but it still needs to know about the layout of days and months and if that part would use calendar code it would bypass its possible errors also.

> The following handle* functions might all make for good calendar shared
> modules, but they are so specific to calendar, I'm not entirely sure how to
> package them up (and you'd have to use modal dialog as well). It might be
> something good to think about though.   Maybe we should put them in a
> calendar-utils file inside the shared-modules folder?
I'm all for it, except perhaps attendees part. This test adds just one, but others might want to add several and use additional actions available in that dialog. If someone would want to use this function to add multiple attendees it wouldn't work since it always expects to type email into second row in the list of attendees. Do you think it should be generalised to recognise the first empty row and type email in there? This would of course depend on bug #505336.

> It looks good; however, the test will simply not run for me all the way
> through, and I'm not sure what is wrong.  It keeps hanging on the attendee
> creation step.
That's because of bug #505336. If it is not solved then the part that creates element for textbox and type call need to be commented out to make it run.

Comment 25

[cmtalbert]

(In reply to comment #24)
> (In reply to comment #23)
> > (From update of attachment 390184 [details] [details])
> > I think you should use "Home" as that is the default name of the default
> > calendar.
> And not specifically created Mozmill calendar as in all other tests? Why?

Oops because I forgot that you create the "mozmill" calendar when you first start out on the first test.  Never mind this comment.

> > What follows is all very date intensive.  I wonder if there is a way to
> > generalize this type of code such that other people writing calendar tests can
> > make use of it.  I'm not sure there is unless we wire the test into the view
> > creation mechanism which is not something we'd want.  Merike, any ideas?  This
> > is not too important, and is not required for r+, but it is something that is
> > worth thinking about.
> I can't see a good solution to this. The combinations of day boxes that people
> need to test will vary and so will the information in tooltips and event boxes.
> It would be nice if there were a function along the lines "verify event
> present/not present from x to y" but it still needs to know about the layout of
> days and months and if that part would use calendar code it would bypass its
> possible errors also.
> 
True.  That's pretty much where I left it as well when I looked at your code a long while.  Oh well.  At least everyone will have an example of how to do it in your test here.

> > The following handle* functions might all make for good calendar shared
> > modules, but they are so specific to calendar, I'm not entirely sure how to
> > package them up (and you'd have to use modal dialog as well). It might be
> > something good to think about though.   Maybe we should put them in a
> > calendar-utils file inside the shared-modules folder?
> I'm all for it, except perhaps attendees part. This test adds just one, but
> others might want to add several and use additional actions available in that
> dialog. If someone would want to use this function to add multiple attendees it
> wouldn't work since it always expects to type email into second row in the list
> of attendees. Do you think it should be generalised to recognise the first
> empty row and type email in there? This would of course depend on bug #505336.
> 
Yeah, I agree.  Let's make all the other handle functions shared module items.  If it weren't for the fact that it is a modal dialog, we could parameterize the list of attendees and solve the problem that way, but alas, we can't.  So, let's leave the handleAttendees out, but make the rest into a utility module.

> > It looks good; however, the test will simply not run for me all the way
> > through, and I'm not sure what is wrong.  It keeps hanging on the attendee
> > creation step.
> That's because of bug #505336. If it is not solved then the part that creates
> element for textbox and type call need to be commented out to make it run.
Ah gotcha.  I'll ping Mikeal again to see if he can figure that one out.

Comment 26

[Merike Sell (:merike)]

Attachment: Event dialog test v2

Comment 27

[Merike Sell (:merike)]

Attachment: CalendarUtils module

Comment 28

[cmtalbert]

Both these files look good, Merike.  Only the event dialog test doesn't run all the way through for me.  Does it run all the way through for you?  Perhaps I should try on Windows?

The only feedback I have on the patches is that we should add the bug reference for the attendee panel bug so that other observers will know why those two lines are commented out.  Otherwise r=ctalbert, except that they don't run :/

I'll try them on windows.

Comment 29

[Merike Sell (:merike)]

(In reply to comment #28)
> Both these files look good, Merike.  Only the event dialog test doesn't run all
> the way through for me.  Does it run all the way through for you?  Perhaps I
> should try on Windows?
Hmm, it does run for me, both with a calendar having email set and not set. The latter case runs longer though, I believe it's because of waiting for email dialog, which won't appear in such case.

Comment 30

[Merike Sell (:merike)]

Attachment: Task view test

Comment 31

[cmtalbert]

Comment on attachment 391169 [details]
Event dialog test v2

r=ctalbert

Comment 32

[cmtalbert]

Comment on attachment 391170 [details]
CalendarUtils module

r=ctalbert, looks great, thanks

Comment 33

[cmtalbert]

Comment on attachment 391640 [details]
Task view test

>  // verify that tooltip shows status, priority and percent complete
>  let toolTipNode = new elementslib.Lookup(controller.window.document, toolTip).getNode();
>  toolTipNode.ownerDocument.defaultView.showToolTip(toolTipNode, taskTreeNode.getTaskAtRow(0));
>  
>  let toolTipName = new elementslib.Lookup(controller.window.document, toolTipGrid + '[1]/[0]/[1]');
>  let toolTipPriority = new elementslib.Lookup(controller.window.document, toolTipGrid + '[1]/[1]/[1]');
>  let toolTipStatus = new elementslib.Lookup(controller.window.document, toolTipGrid + '[1]/[2]/[1]');
>  let toolTipComplete = new elementslib.Lookup(controller.window.document, toolTipGrid + '[1]/[3]/[1]');
>  let priority = UtilsAPI.getProperty("chrome://calendar/locale/calendar.properties",
>    "highPriority");
>  
Merike, I can't believe this actually works.  It seems to on run.

I had a hiccup with the lookup at the "verify description and status in detail pane" segment (line 147).  But then it worked. Strange.  

r=ctalbert  

We will have to go back through these tests on each platform and be sure they all run end-to-end.  They have all worked on mac, but these hiccups still exist.  But I think some of this might be mozmill related.  

Great work getting these smoke tests done.  On to phase II.

And I have to figure out how to check these in.

Comment 34

[cmtalbert]

Ok.  Martin and I conferred this morning and got these checked in.  I checked in the latest version of each of the above patches.  For our discussion on the repo structure it was here in IRC:
mschroeder was promoted to operator by ChanServ.
[10:03am] mschroeder: ctalbert: hey. regarding merike's mozmill tests: I think you are go to check them in on comm-central/calendar/test/mozmill/. maybe i'll have a look at needed changes for client.py later, so we can proceed with getting them in the tree and running.
[10:04am] ctalbert: mschroeder: thank you!
[10:04am] ctalbert: mschroeder: this is something in the back of my mind constantly, but I haven't had a chance to do anything about it.
[10:04am] ctalbert: I'll try to get those checked in today
[10:06am] mschroeder: ctalbert: I read through the emails again, and decided this has to be done now. 
[10:06am] mschroeder: ctalbert: maybe the relative path to the shared modules has to be adjusted later
[10:08am] ctalbert: yeah, me too.  Yeah I think once you figure out how to tweak client.py to pull the proper stuff from hg.mozilla.org/qa/mozmill-tests and merge the shared modules directory together, it will just work.
[10:08am] ctalbert: So I'll put the tests in calendar/test/mozmill
[10:09am] ctalbert: and the shared modules in calendar/test/mozmill/shared_modules
[10:09am] ctalbert: sound good?
[10:11am] mschroeder: perfect 

Pushed ae4488b6fe3f --> FIXED