962795 - Reenable e.me UI automated tests

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Kevin Grandon :kgrandon]

Rocketbar will be turned on soon, and this will break a few tests. I'm going to disable them for now and use this to track either re-implementing them, or removing entirely.

Comment 1

[Zac C (:zac)]

Kevin, similarly let us know (via bugs) where you need extra coverage for device, ie where Travis/desktopb2g can't reach.

Comment 2

[Jason Smith [:jsmith]]

(In reply to Zac C (:zac) from comment #1)
> Kevin, similarly let us know (via bugs) where you need extra coverage for
> device, ie where Travis/desktopb2g can't reach.

What I'd do here is port over the existing tests to follow the new rocketbar flow in the area we know won't change to avoid UX churn. So I'd do the following:

1. Open the rocketbar from the homescreen by clicking the search bar
2. Search for the e.me term you would normally search for in the test
3. Select the e.me search term in the search suggestions
4. Continue the test as normal

There are some tests however in e.me test directory that shouldn't be dependent on the rocketbar changes though - they should be possible to run right now. Namely:

* test_everythingme_add_collection.py
* test_everythingme_app_install.py

Comment 3

[Jason Smith [:jsmith]]

Kevin - We need to get unblocked here. Is there a solution we can throw together here to be able to get the e.me tests back up and running?

Comment 4

[Kevin Grandon :kgrandon]

The problem is that dom selectors are rapidly changing for rocketbar, and I'm not sure if all of the existing use cases are there yet, or will be there in the end.

For now, we may be able to spin up a new travis target, which disables rocketbar and tests these old flows. I think that this would be a good approach, but it seems like we may need a new set of manifest files, or some way to skip-if based on a preference.

Comment 5

[Jason Smith [:jsmith]]

removing xfail here - we're going to disable rocketbar temporarily to get these tests back on. I'm going to file a new bug to track turning the tests on post bug 972034 landing.

Comment 7

[Kevin Grandon :kgrandon]

Zac - sorry about the limbo state of these tests! We now have E.me landed in rocketbar, so we are good to go here on testing. One thing that we should think about is that we may actually end up shipping with this feature turned off. Since that is the case, we may consider leaving the old tests in the codebase for now, and just turned off. If you have smarter switching mechanisms, that would be fine too. I'll leave the final implementation decision up to you.

Please let me know how you want to proceed re-enabling/implementing these tests, and if there's any way I can help. If you need me to pitch in for a day or two on the UI tests I'd be more than happy to do so. Thanks!

Comment 8

[Zac C (:zac)]

(In reply to Kevin Grandon :kgrandon from comment #7)
> Zac - sorry about the limbo state of these tests! We now have E.me landed in
> rocketbar, so we are good to go here on testing. One thing that we should
> think about is that we may actually end up shipping with this feature turned
> off. Since that is the case, we may consider leaving the old tests in the
> codebase for now, and just turned off. If you have smarter switching
> mechanisms, that would be fine too. I'll leave the final implementation
> decision up to you.
> 
> Please let me know how you want to proceed re-enabling/implementing these
> tests, and if there's any way I can help. If you need me to pitch in for a
> day or two on the UI tests I'd be more than happy to do so. Thanks!

We can switch it on and off using a pref (although I haven't checked properly) so it's no problem to run one or the other or both, running time costs aside.

For rocketbar, is there some coverage we can add for TBPL or Device that you haven't already got in the js tests?

Comment 9

[Jason Smith [:jsmith]]

(In reply to Zac C (:zac) from comment #8)
> (In reply to Kevin Grandon :kgrandon from comment #7)
> > Zac - sorry about the limbo state of these tests! We now have E.me landed in
> > rocketbar, so we are good to go here on testing. One thing that we should
> > think about is that we may actually end up shipping with this feature turned
> > off. Since that is the case, we may consider leaving the old tests in the
> > codebase for now, and just turned off. If you have smarter switching
> > mechanisms, that would be fine too. I'll leave the final implementation
> > decision up to you.
> > 
> > Please let me know how you want to proceed re-enabling/implementing these
> > tests, and if there's any way I can help. If you need me to pitch in for a
> > day or two on the UI tests I'd be more than happy to do so. Thanks!
> 
> We can switch it on and off using a pref (although I haven't checked
> properly) so it's no problem to run one or the other or both, running time
> costs aside.
> 
> For rocketbar, is there some coverage we can add for TBPL or Device that you
> haven't already got in the js tests?

Note - the coverage in Marionette JS won't guarantee test coverage on OOP workflows & on device impacts. We've already seen at least on example of a failure that only happened on device (e.me results worked on b2g desktop, failed on device). So we're going to need coverage for the tests that happen to model our manual smoketests for rocketbar - definition of this coming soon.

Comment 10

[Jason Smith [:jsmith]]

Turns out we're no longer pursuing rocketbar in 1.4, so I've filed bug 974163 to pref it off at build time. After that lands, we should be able to turn the existing tests back on.

Comment 11

[Zac C (:zac)]

I'll set it off in the test until 974163 is in.

Comment 12

[Zac C (:zac)]

Attachment: github pr

Askeing if the tree is open tomorrow morning feel free to merge this when you r+ it

Comment 13

[Zac C (:zac)]

Merged:
https://github.com/mozilla-b2g/gaia/commit/e450dff4fb9e7b1ad83be29f559817252b8f55f4