479314 - Mozmill test for starting/stopping private browsing mode

Status: VERIFIED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This bug will cover the work which is needed to create a MozMill test for https://litmus.mozilla.org/show_test.cgi?id=7394

Comment 1

[Nochum Sossonko [:Natch]]

This can be (and is afaik) done with automated tests... (especially now that the tests can be run without keep_current_session)

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We probably already have automated tests here but these are unit tests and not functional tests. The way how MozMill handles the actions is totally different. We wanna detect failures while interacting with the ui.

Comment 3

[Nochum Sossonko [:Natch]]

FTR private browsing has browser-chrome tests as well. I'm not saying not to do this, just casually mentioning.

Comment 4

[(no longer active)]

I think Natch is correct in comment 1, and this _can_ be written as a browser chrome test now, but from what I hear, MozMill is much better at handling these kinds of scenarios, so I'm willing to see what comes up here.  :-)

And as a general note, Henrik, I think with bug 476463 fixed, we can convert some of the current PB Litmus tests to automated tests.  Do you have a list of those tests which seem appropriate for this job?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #4)
> I think Natch is correct in comment 1, and this _can_ be written as a browser
> chrome test now, but from what I hear, MozMill is much better at handling these
> kinds of scenarios, so I'm willing to see what comes up here.  :-)

If you can come up with such a chrome test it would be really helpful, because MozMill isn't integrated into the build system right now. Means all the tests aren't run automatically on each check-in. Regressions will be find to late in the circle.

> And as a general note, Henrik, I think with bug 476463 fixed, we can convert
> some of the current PB Litmus tests to automated tests.  Do you have a list of
> those tests which seem appropriate for this job?

You mean tests which can be converted to automated tests? I think that Marcia can help out here. Marcia, do we have a private browsing subgroup in Litmus?

Comment 6

[(no longer active)]

(In reply to comment #5)
> (In reply to comment #4)
> > I think Natch is correct in comment 1, and this _can_ be written as a browser
> > chrome test now, but from what I hear, MozMill is much better at handling these
> > kinds of scenarios, so I'm willing to see what comes up here.  :-)
> 
> If you can come up with such a chrome test it would be really helpful, because
> MozMill isn't integrated into the build system right now. Means all the tests
> aren't run automatically on each check-in. Regressions will be find to late in
> the circle.

Ah, OK.  In that case, would you mind filing a new bug for that, please?  :-)

> > And as a general note, Henrik, I think with bug 476463 fixed, we can convert
> > some of the current PB Litmus tests to automated tests.  Do you have a list of
> > those tests which seem appropriate for this job?
> 
> You mean tests which can be converted to automated tests?

Yup.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #6)
> Ah, OK.  In that case, would you mind filing a new bug for that, please?  :-)

Done. We can head over to bug 479720 for the automated tests.

Comment 8

[(no longer active)]

Thanks for filing that bug!

BTW, I assume that you added the dependency on bug 479571 by mistake, right?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

No. That was intended. I've only forgotten to give a comment for that. Sorry.

Comment 10

[(no longer active)]

OK then, sorry for messing with your work.  :-)

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mike, this bug handles the MozMill test creation for the given Litmus test. It has nothing to do with Ehsans work on mc or 1.9.1. For the future just check the summary and if it contains "[mozmill]".

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Clint, shall we checkin the patch even when it will not run via jsbridge? I don't think so because it will completely break mozmill. As what I can see now, it hangs and there is no progress. Probably it's worth filing a bug for Mozmill here too. I would at least attach the patch on this bug. Then we have to wait until bug 488772 has been fixed for checkin.

Comment 13

[cmtalbert]

(In reply to comment #12)
> Clint, shall we checkin the patch even when it will not run via jsbridge? 

What patch? It's not attached here :/ I don't see any issue with checking it in now so that we can get it into source control, and just put a comment that it shouldn't be run automatically once that gets hooked up.  Since no test on build stuff is hooked up yet, there is no problem with checking it in now.  I'd rather have it in source control than wait for some bug to clear the way to run it with functionality we don't currently have. :)

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1.0

Clint, this is the first patch for review. It will run successfully with the latest trunk build of Mozmill. Could you please have a look into that?

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1.1

It would be nice to not forget to qrefresh first. :)

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1.2

Please blame me. That one should be fine. I'll stop working on this patch for today. :)

Comment 17

[cmtalbert]

Comment on attachment 382293 [details] [diff] [review]
Patch v1.2

r=ctalbert  Awesome tests whimboo, I also see in your tests what I was doing wrong when trying to generalize the modal dialog for the preferences utils.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1.3

I had a failure in checking for the titlemodifier. I have to use getAttribute to get the value. I've updated the patch. Taking over r+.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Test checked in: http://hg.mozilla.org/qa/mozmill-tests/rev/253e06411e8e

It will add tests for following litmus tests: 7394, 7395, 7443, 7463, 7643, and 7714.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I had to do a refactoring for those tests to match the style we need for reports of Mozmill. Lets reuse this bug because it applies to all written Private Browsing tests. Patch is upcoming.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Follow-up (refactoring)

Complete refactoring of the private browsing tests so that we have mapped one Litmus test to one test function or module in Mozmill.

Comment 22

[Aakash Desai [:aakashd]]

Comment on attachment 400099 [details] [diff] [review]
Follow-up (refactoring)

This fails on the latest mozmill-trunk, but passes on multiple attempts on OSX and XP using the latest MozMill release.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed as:

http://hg.mozilla.org/qa/mozmill-tests/rev/acc03852192d
http://hg.mozilla.org/qa/mozmill-tests/rev/373633409461

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mass move of Mozmill Test related project bugs to newly created components. You can filter out those emails by using "Mozmill-Tests-to-MozillaQA" as criteria.