1184094 - Investigate: UI Tests for restricted profiles

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect)

Description

[Sebastian Kaspari (:sebastian; :pocmo)]

It would be nice to have (UI) tests for features relying on restricted profiles. We probably won't get much manual testing coverage. Since we are hiding and disabling features for restricted profiles this part of the code is great for introducing regressions. So let's investigate whether we can test these things somehow.

Comment 1

[Kevin Brosnan [Ex-Mozilla]]

If this ends up being too difficult to automate we can create a set of manual tests and define some critical steps that must be tested every release.

Comment 2

[Sebastian Kaspari (:sebastian; :pocmo)]

Having UI tests for restricted profiles seems to be out of scope for 42. But I'd like to have at least a simple test that verifies that all restrictions are not enforced (isAllowed(*) == true) whenever using a normal profile.

Comment 3

[Sebastian Kaspari (:sebastian; :pocmo)]

Attachment: 1184094-restrictions-test.patch

Simple test to verify that isAllowed() will return true for all restrictions using a normal profile.

Comment 4

[:Margaret Leibovic]

Comment on attachment 8640494 [details] [diff] [review]
1184094-restrictions-test.patch

Review of attachment 8640494 [details] [diff] [review]:
-----------------------------------------------------------------

Nice sanity check. Let's make sure it's green on try before landing, though, because automation can be flakey.

Comment 5

[Sebastian Kaspari (:sebastian; :pocmo)]

(In reply to :Margaret Leibovic from comment #4)
> Nice sanity check. Let's make sure it's green on try before landing, though,
> because automation can be flakey.

Yeah, I guess I'll push all "land-able" kidfox patches at once to try today.

Comment 6

[Sebastian Kaspari (:sebastian; :pocmo)]

(In reply to Sebastian Kaspari (:sebastian) from comment #5)
> Yeah, I guess I'll push all "land-able" kidfox patches at once to try today.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f1c1a95bb1c2

Comment 7

[Sebastian Kaspari (:sebastian; :pocmo)]

url:        https://hg.mozilla.org/integration/fx-team/rev/06ea5f2003a2484690df9a935050d23e1f32f4f7
changeset:  06ea5f2003a2484690df9a935050d23e1f32f4f7
user:       Sebastian Kaspari <[e-mail]>
date:       Fri Jul 31 09:49:04 2015 +0200
description:
Bug 1184094 - testRestrictions: No restrictions should be enforced when using a normal profile. r=margaret

Comment 8

[Sebastian Kaspari (:sebastian; :pocmo)]

(In reply to Kevin Brosnan [:kbrosnan] from comment #1)
> If this ends up being too difficult to automate we can create a set of
> manual tests and define some critical steps that must be tested every
> release.

Kevin: What would be the process to create this set of manual tests? Most v1 bugs are about hiding features and UI elements. So I guess most manual tests would verify that these features are still accessible with a normal profile and hidden with a restricted profile. Restricted profiles can be configured by the device admin so another test could be to verify that with all restrictions set to OFF the restricted profile behaves almost like a normal profile.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/06ea5f2003a2

Comment 10

[Allison Naaktgeboren :ally]

After talking the gbrown, the conclusion is that automated tests cannot happen until the migration to autophone occurs. the tablet emulator is not running a new enough version and the actual hardware are not tablets.

Comment 11

[Geoff Brown [:gbrown]]

To clarify..."Normal" automated unit tests run on emulated Android 2.3 tablets and emulated Android 4.3 phones, and neither configuration supports restricted profiles. 

Autophone runs a select group of tests against a variety of real devices. Theoretically, we could run new autophone tests for restricted profiles, limited to certain appropriate (4.3+) devices (perhaps manually pre-configured with a restricted profile?). This is not technically blocked by the project which is migrating Android Talos tests to autophone, but the people who are most familiar with autophone are currently focused on the Talos migration. autophone tests run today on most trees, but are hidden (they are "Tier-2" jobs). The obvious challenges for restricted profile autophone tests are simply:
 - writing the test(s)
 - finding appropriate devices
 - ensuring sufficient capacity for running the new test(s)

Comment 12

[Sebastian Kaspari (:sebastian; :pocmo)]

Detaching this bug from v1.

Comment 13

[Sebastian Kaspari (:sebastian; :pocmo)]

Closing this bug. The scope was investigating. That's what we did. We have a simple sanity check and /could/ write UI tests in the future.

Comment 14

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.