Closed Bug 1417255 Opened 7 years ago Closed 3 years ago

Use Espresso for UI Testing

Categories

(Firefox for Android Graveyard :: Testing, enhancement, P2)

Product:
Firefox for Android Graveyard
Component:
Testing
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: cnevinchen, Assigned: cnevinchen)

References

Details

Attachments

(3 files, 10 obsolete files)

Bug 1417255 - Add espresso to Fennec.
59 bytes, text/x-review-board-request
nalexander
: review-
Details
Bug 1417255 - Turn off animation while running UI Test.
59 bytes, text/x-review-board-request
nalexander
: review-
Details
Bug 1417255 - Add test when PAW badge should be displayed or hidden.
59 bytes, text/x-review-board-request
nalexander
: review-
Details 
Since we are moving to full gradle build, we can do this more easily.
The idea is to add new test in Espresso and leave existing robocop tests as is.
Here are some action items
1. If there can be two test runner at the same time.
2. Configure the build process to make Espresso work on CI.
3. The first target of the test is PWA prompt and onboarding
Assignee: nobody → cnevinchen
Priority: -- → P2
I'm not sure if we can use both InstrumentationTestRunner and AndroidJUnitRunner with Robotium and Espresso at the same time.

Using adb :

adb shell am instrument -w -r   -e debug false -e class org.mozilla.gecko.HelloWorldEspressoTest org.mozilla.roboexample.test/android.support.test.runner.AndroidJUnitRunner

will work but I want to uses gradle and not specify the test to run.

But running ./gradlew app:connectedLocalPhotonDebugAndroidTest or just ./gradlew cAt will have some test fail :

org.mozilla.gecko.background.db.TestAndroidBrowserBookmarksRepository > testFetchSinceOneRecord[Nexus_5_API_27(AVD) - 8.1.0] FAILED 
        junit.framework.AssertionFailedError: Caught error in performWait
        at org.mozilla.gecko.background.helpers.AndroidSyncTestCase.performWait(AndroidSyncTestCase.java:33)
....
org.mozilla.gecko.background.db.TestAndroidBrowserHistoryRepository > testAddMultipleVisits[Nexus_5_API_27(AVD) - 8.1.0] FAILED 

org.mozilla.gecko.background.db.TestAndroidBrowserHistoryRepository > testAddOneVisit[Nexus_5_API_27(AVD) - 8.1.0] FAILED 

org.mozilla.gecko.background.db.TestAndroidBrowserHistoryRepository > testCleanMultipleRecords[Nexus_5_API_27(AVD) - 8.1.0] FAILED 
org.mozilla.gecko.background.db.TestAndroidBrowserHistoryRepository > testDataAccessorBulkInsert[Nexus_5_API_27(AVD) - 8.1.0] FAILED 
org.mozilla.gecko.background.db.TestAndroidBrowserHistoryRepository > testDeleteLocalNewer[Nexus_5_API_27(AVD) - 8.1.0] FAILED 
org.mozilla.gecko.background.db.TestAndroidBrowserHistoryRepository > testFetchAll[Nexus_5_API_27(AVD) - 8.1.0] FAILED 
        junit.framework.AssertionFailedError: Caught error in performWait
        at org.mozilla.gecko.background.helpers.AndroidSyncTestCase.performWait(AndroidSyncTestCase.java:33)
org.mozilla.gecko.background.db.TestAndroidBrowserHistoryRepository > testFetchSinceOneRecord[Nexus_5_API_27(AVD) - 8.1.0] FAILED 
......

I'll look for solution either
1. filter and only run newly added espresso test using gradle
2. Use different test runner at the same time
3. others....
with the latest patch I can do ./mach gradle connectedEspressoPhotonDebugAndroidTest and run espresso test, and run ./mach robocop testNewTab locally without a problem :)

Will need to see how it works on try, and how to add the new task to CI
Whiteboard: [FNC][SPT59.2][BL]
I'll gave up the idea in  Comment 11 ( to make it work wtih CI) cause I want to make the scope smaller.
I'll reorg the patch and make it:

Part1. Add dependencies for Espresso.
Part2. Add a UI test.
Part3. Make the test run locally without bothering Robocop test.( Just like GeckoViewActivityTest in geckoview_example, but we need it to be in the app module!)
Attachment #8928321 - Attachment is obsolete: true
Attachment #8931296 - Attachment is obsolete: true
Attachment #8932319 - Attachment is obsolete: true
Attachment #8935304 - Attachment is obsolete: true
Attachment #8936051 - Attachment is obsolete: true
Attachment #8936052 - Attachment is obsolete: true
Attachment #8936053 - Attachment is obsolete: true
Attachment #8936110 - Attachment is obsolete: true
Whiteboard: [FNC][SPT59.2][BL]
Attachment #8937391 - Attachment is obsolete: true
Comment on attachment 8940143 [details]
Bug 1417255 - Add test when PAW badge should be displayed or hidden.

This patch :
1. Makes tearDown only clear the tabs the tabs has opened. 
2. Change the name of the test 
3. Add different CountingIdlingResource implementation for espresso and other flavors
Attachment #8940575 - Attachment is obsolete: true
Attachment #8940575 - Flags: review?(max)
Attachment #8930029 - Flags: review?(max) → review?(nalexander)
Attachment #8940142 - Flags: review?(max) → review?(nalexander)
Attachment #8940143 - Flags: review?(max) → review?(nalexander)
Attachment #8930029 - Flags: review?(nalexander)
Attachment #8940142 - Flags: review?(nalexander)
Attachment #8940143 - Flags: review?(nalexander)
Comment on attachment 8930029 [details]
Bug 1417255 - Add espresso to Fennec.

https://reviewboard.mozilla.org/r/201240/#review222872

I'm not going to give this a detailed review yet.  (And given that Nevin will likely not be workig on Fennec/PWAs, that might be ever.)

There are a lot of things to like here, but there are also some things to not like.

- there are many extraneous changes (whitespace, bogus includes, tooltool manifest changes).  Those need to go.
- there are several logically distinct things mashed together: Espresso (good!), PWA listener (good!), test definition and test data (good!), badge workarounds for tests (bad!).  Those good parts need to be separate patches.  The badge workaround (mixed content) for tests can be done better: an environment variable or a static variable set by the test code would be better than this very complicated system.  See below.
- having a flavor for Espresso is not the right approach for Gradle.  If we need a flag, we can do that -- put the flag in AppConstants or turn this off for --without-gradle.  But a product flavor isn't the right approach.
- the EspressoOnly piece is nice.

Overall, a great start!  But I think it will need to be rebased onto https://bugzilla.mozilla.org/show_bug.cgi?id=1411654, and will need reworking before landing.
Attachment #8930029 - Flags: review-
Comment on attachment 8940142 [details]
Bug 1417255 - Turn off animation while running UI Test.

https://reviewboard.mozilla.org/r/210464/#review222874

Further to my earlier review, this "is under test" API is pretty ugly.  A static variable would be better... or look at the Android code and see if we can get the "is developer option checked" value, just like the platform code does.  Maybe that's possible?

But in theory I think this is something we should do when testing.  And maybe when Robocop testing too!
Attachment #8940142 - Flags: review-
Comment on attachment 8940143 [details]
Bug 1417255 - Add test when PAW badge should be displayed or hidden.

https://reviewboard.mozilla.org/r/210466/#review222876

Fold the test resources into this patch -- the testing patch -- please.  This is a great start!

::: commit-message-50b93:1
(Diff revision 15)
> +Bug 1417255 - Add test when PAW badge should be displayed or hidden. r?maliu

nit: s/PAW/PWA/.

::: mobile/android/app/src/androidTest/java/org/mozilla/gecko/PwaBadgeTest.java:1
(Diff revision 15)
>  package org.mozilla.gecko;

nit: license header.

::: mobile/android/app/src/androidTest/java/org/mozilla/gecko/PwaBadgeTest.java:103
(Diff revision 15)
>  
>      }
>  
> +    @After
> +    public void tearDown() {
> +        onView(withId(R.id.counter_box)).perform(click());

There must be a better way to close all the tabs.  Can we not tell the browser to do that in some way that's _not_ clicking buttons?  (We shouldn't be interacting with the UI in teardown.)

Can we kill the Activity between tests instead?
Attachment #8940143 - Flags: review-
We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: