Closed Bug 962795 Opened 6 years ago Closed 6 years ago

Reenable e.me UI automated tests

Categories

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

x86
macOS
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kgrandon, Assigned: zcampbell)

References

Details

(Whiteboard: [xfail])

Attachments

(1 file)

46 bytes, text/x-github-pull-request
askeing
: review+
Bebe
: review+
Details | Review
Rocketbar will be turned on soon, and this will break a few tests. I'm going to disable them for now and use this to track either re-implementing them, or removing entirely.
Kevin, similarly let us know (via bugs) where you need extra coverage for device, ie where Travis/desktopb2g can't reach.
Whiteboard: [xfail]
(In reply to Zac C (:zac) from comment #1)
> Kevin, similarly let us know (via bugs) where you need extra coverage for
> device, ie where Travis/desktopb2g can't reach.

What I'd do here is port over the existing tests to follow the new rocketbar flow in the area we know won't change to avoid UX churn. So I'd do the following:

1. Open the rocketbar from the homescreen by clicking the search bar
2. Search for the e.me term you would normally search for in the test
3. Select the e.me search term in the search suggestions
4. Continue the test as normal

There are some tests however in e.me test directory that shouldn't be dependent on the rocketbar changes though - they should be possible to run right now. Namely:

* test_everythingme_add_collection.py
* test_everythingme_app_install.py
Depends on: 964698
Kevin - We need to get unblocked here. Is there a solution we can throw together here to be able to get the e.me tests back up and running?
Flags: needinfo?(kgrandon)
The problem is that dom selectors are rapidly changing for rocketbar, and I'm not sure if all of the existing use cases are there yet, or will be there in the end.

For now, we may be able to spin up a new travis target, which disables rocketbar and tests these old flows. I think that this would be a good approach, but it seems like we may need a new set of manifest files, or some way to skip-if based on a preference.
Flags: needinfo?(kgrandon)
removing xfail here - we're going to disable rocketbar temporarily to get these tests back on. I'm going to file a new bug to track turning the tests on post bug 972034 landing.
Whiteboard: [xfail]
Duplicate of this bug: 972092
Zac - sorry about the limbo state of these tests! We now have E.me landed in rocketbar, so we are good to go here on testing. One thing that we should think about is that we may actually end up shipping with this feature turned off. Since that is the case, we may consider leaving the old tests in the codebase for now, and just turned off. If you have smarter switching mechanisms, that would be fine too. I'll leave the final implementation decision up to you.

Please let me know how you want to proceed re-enabling/implementing these tests, and if there's any way I can help. If you need me to pitch in for a day or two on the UI tests I'd be more than happy to do so. Thanks!
Flags: needinfo?(zcampbell)
Summary: UI testing updates for Rocketbar → Reenable e.me UI automated tests
Whiteboard: [xfail]
Depends on: 971268
(In reply to Kevin Grandon :kgrandon from comment #7)
> Zac - sorry about the limbo state of these tests! We now have E.me landed in
> rocketbar, so we are good to go here on testing. One thing that we should
> think about is that we may actually end up shipping with this feature turned
> off. Since that is the case, we may consider leaving the old tests in the
> codebase for now, and just turned off. If you have smarter switching
> mechanisms, that would be fine too. I'll leave the final implementation
> decision up to you.
> 
> Please let me know how you want to proceed re-enabling/implementing these
> tests, and if there's any way I can help. If you need me to pitch in for a
> day or two on the UI tests I'd be more than happy to do so. Thanks!

We can switch it on and off using a pref (although I haven't checked properly) so it's no problem to run one or the other or both, running time costs aside.

For rocketbar, is there some coverage we can add for TBPL or Device that you haven't already got in the js tests?
Flags: needinfo?(zcampbell)
(In reply to Zac C (:zac) from comment #8)
> (In reply to Kevin Grandon :kgrandon from comment #7)
> > Zac - sorry about the limbo state of these tests! We now have E.me landed in
> > rocketbar, so we are good to go here on testing. One thing that we should
> > think about is that we may actually end up shipping with this feature turned
> > off. Since that is the case, we may consider leaving the old tests in the
> > codebase for now, and just turned off. If you have smarter switching
> > mechanisms, that would be fine too. I'll leave the final implementation
> > decision up to you.
> > 
> > Please let me know how you want to proceed re-enabling/implementing these
> > tests, and if there's any way I can help. If you need me to pitch in for a
> > day or two on the UI tests I'd be more than happy to do so. Thanks!
> 
> We can switch it on and off using a pref (although I haven't checked
> properly) so it's no problem to run one or the other or both, running time
> costs aside.
> 
> For rocketbar, is there some coverage we can add for TBPL or Device that you
> haven't already got in the js tests?

Note - the coverage in Marionette JS won't guarantee test coverage on OOP workflows & on device impacts. We've already seen at least on example of a failure that only happened on device (e.me results worked on b2g desktop, failed on device). So we're going to need coverage for the tests that happen to model our manual smoketests for rocketbar - definition of this coming soon.
Turns out we're no longer pursuing rocketbar in 1.4, so I've filed bug 974163 to pref it off at build time. After that lands, we should be able to turn the existing tests back on.
Depends on: 974163
No longer depends on: 964698, 971268
I'll set it off in the test until 974163 is in.
Assignee: nobody → zcampbell
Attached file github pr
Askeing if the tree is open tomorrow morning feel free to merge this when you r+ it
Attachment #8378404 - Flags: review?(fyen)
Attachment #8378404 - Flags: review?(florin.strugariu)
Attachment #8378404 - Flags: review?(florin.strugariu) → review+
Attachment #8378404 - Flags: review?(fyen) → review+
Merged:
https://github.com/mozilla-b2g/gaia/commit/e450dff4fb9e7b1ad83be29f559817252b8f55f4
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.