1184046 - Make open new private tab tests more robust

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: Testing, defect)

Description

[Martyn Haigh (:mhaigh)]

Bug 1175970 introduced a UI element on the first time a user opens a new tab, also it introduced a change to the testPrivateBrowsing test which cancels the new UI when any tab is opened.  This is pretty fragile as if two private tabs are open in a test, this will cause it to fail.

Lets make these tests a bit more robust.

Comment 1

[Martyn Haigh (:mhaigh)]

Attachment: Make open new private tab tests more robust

Have adjusted the tests as requested.  We now automatically track the state of if we've seen the TP UI before or not and handle it in PixelTest.  I've also added some verification in to testPrivateBrowing to test that we show UI on the first private tab, and also add a second private tab to ensure that this works as normal.

How's this looking to you?

Also - there's a bit of noise in there, my IDE decided to trim some spaces and everytime I save now it keeps trimming, no biggie but wanted to point it out.

Comment 2

[:Margaret Leibovic]

Comment on attachment 8634645 [details] [diff] [review]
Make open new private tab tests more robust

Review of attachment 8634645 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/tests/browser/robocop/PixelTest.java
@@ +77,5 @@
> +    protected void dismissTrackingProtectionUI() {
> +        waitForText(mStringHelper.TRACKING_PROTECTION_PROMPT_TITLE);
> +        mSolo.clickOnText(mStringHelper.TRACKING_PROTECTION_PROMPT_BUTTON);
> +        waitForText(mStringHelper.PRIVATE_BROWSING_TITLE);
> +    }

Instead of adding methods in here to deal with the tracking protection UI, I was thinking we would find a way to set PREFS_TRACKING_PROTECTION_PROMPT_SHOWN to true before any tests run.

We already have a way to do this for gecko prefs:
http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js

I don't think we currently have a way to do this for SharedPreferences, so maybe now would be a good time to figure that out. Or we could just set this pref in some central place, such as BaseTest.

It seems wrong to me for PixelTest to be dealing with this tracking protection UI.

::: mobile/android/tests/browser/robocop/testPrivateBrowsing.java
@@ +46,4 @@
>  
> +        // Add another private tab, this time the Tracking Protection UI shouldn't show.
> +        addTab(bigLinkUrl, true);
> +        verifyTabCount(2);

Does this actually verify that the tracking protection onboarding screen didn't show? Is there a way to check for the absence of some text on the screen?

Comment 3

[:Margaret Leibovic]

gbrown, do you know if there's any way for us to modify SharedPreferences in our testing profile, similarly to how we can set gecko prefs in testing/profiles/prefs_general.js?

If not, do you have a way to recommend setting a SharedPreference for all test runs? We're looking to disable a piece on onboarding UI that appears when you first open a new private tab.

Actually, liuche, what did you end up doing to solve this problem for the first run UI? I remember there were some test struggles with that at one point.

Comment 4

[Geoff Brown [:gbrown] (pto Apr 11-18)]

I am pretty sure we have no support for setting SharedPreferences in tests. You asked about "for all test runs" -- do you need this for all test suites, like mochitest and reftest? That seems difficult. If we can limit our aspirations to Robocop, I think some new code in BaseRobocopTest (hook into setUp()?) might be the way to go.

Comment 5

[Chenxia Liu [:liuche]]

We do all the test disabling on the Java side, actually. The robocop BaseTest starts the test with an Intent extra that disables first run.

http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/BrowserApp.java#908
http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/BaseRobocopTest.java#130

Comment 6

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Mass closing Firefox for Android :: Testing bugs with no progress in 2017.

If this bug is important to you, please re-open.