1198449 - Create Best Practice Document for Automation in Firefox OS

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Comment 1

[Johan Lorenzo [:jlorenzo]]

We could also put the process of a patch (commit, push, PR, etc.). There is no need to rewrite the whole process.  A pointer to [1] would be okay + an extra comment that we need 2 reviewers and adhoc jobs.

[1] https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Submitting_a_Gaia_patch

Comment 2

[Johan Lorenzo [:jlorenzo]]

On top of my head. We could put one rule regarding the usage of assertion:

> Putting a custom error message only when it provides more information than the one given by default.
> For example `self.asserEqual()` is a nice assertion method: if it fails, it'll give you the expected 
> value and the actual. For example:
> This piece of code: self.assertEqual(displayed_phone_number, expected_phone_number), will fail with:
> ```
> AssertionError: u'+34...' != u'+33...'
> Stacktrace:
> ```
> If you provide an error message:
> self.assertEqual(displayed_phone_number, expected_phone_number, msg='Phone numbers are not the same')
> you'll have something like:
> ```
> AssertionError: Phone numbers are not the same
> Stacktrace:
> ```
> In other words, by willing to add more context, you might remove some useful debug data.

Comment 3

[Johan Lorenzo [:jlorenzo]]

> Testing strings that comes from the system (like "Missed call", or "Send"): Use `data-l10n-id` inside the Page class. For instance:
> ```
> class SomePageClass()
>     @property
>     def is_labeled_as_today(self)
>         return element.get_attribute('data-l10n-id') == 'today'
> ```

Comment 4

[No-Jun Park [:njpark]]

When should PageRegion class be used?  is it that when locating everything under the regions/ directory, or is it for the different regions in the same class?

Also, it should be noted that root_element will become stale when once the frame is changed.

Comment 5

[No-Jun Park [:njpark]]

The location of forked style guide from Web QA is here:
https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide

Comment 6

[No-Jun Park [:njpark]]

https://public.etherpad-mozilla.org/p/Automation_BestPractice is up.  Could you take a look and add anything that we should add to this list?

Comment 7

[Johan Lorenzo [:jlorenzo]]

I added some elements in the list. I think the list is big enough to be split into two. Hence, we could have a clear distinction between this bug and bug 1213243.

At a fist glance, the part called "Need to mention following concepts" seems to have the same size as every other combined. What do you guys think?

Comment 8

[Johan Lorenzo [:jlorenzo]]

Attachment: https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide

R+ on No-Jun's parts. These pages are in good shape for being our first revision.

Comment 9

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8686615 [details]
https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide

I filled up my parts and moved things around in:
* https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide/Best_Practices
* https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide/Avoiding_Duplication
* https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide/Submission_Review

I'm not sure the level of details is the right one. Asking No-Jun for review.

Comment 10

[No-Jun Park [:njpark]]

Comment on attachment 8686615 [details]
https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide

Johan is working on General Concepts section, but otherwise everything else looks good to me.

Comment 11

[Martijn Wargers (dead)]

Comment on attachment 8686615 [details]
https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide

I don't see a Wiki link here.

Comment 12

[No-Jun Park [:njpark]]

Comment on attachment 8686615 [details]
https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide

https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide

Comment 13

[Martijn Wargers (dead)]

Comment on attachment 8686615 [details]
https://wiki.mozilla.org/B2G/QA/Automation/Style_Guide

Looks good, currently I don't really know what to add. 

I seem some mentions of b2g deskop, but afaik, that doesn't exist anymore or isn't used anymore. Or does MarionetteJS use b2g desktop? (I thought it uses Mulet or something)

https://wiki.mozilla.org/B2G/QA/Automation/UI/Integration_suite_criteria#Firefox_OS_current_specific_considerations
This makes the assumption that integration tests are only run on desktop? If the integration tests would run on device, some of these constraints wouldn't exist.

Comment 14

[No-Jun Park [:njpark]]

I think MarionetteJS uses b2g desktop for the current integration testing.  I hear the plan is to run those same tests on device for the integration testing.