1145243 - Set default settings during full reset

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

At the moment we use Marionette to reset any default settings in our setUp. This has two main disadvantages: it's slow to call the settings API several times, and it prevents us from changing setting that affect how Gaia starts. A good example is that we can't avoid starting Gaia into the FTU app, and have to kill it for each test.

I've been able to successfully modify the default settings (for device) and current profile (for desktop) during the restart initiated by the --restart command line option. We will also likely see a performance improvement.

Comment 1

[GitHub Autolander Bot]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

Adhoc smoke test run:
http://jenkins1.qa.scl3.mozilla.com/view/Bitbar/job/flame-kk.ui.adhoc.bitbar/28/

Should be comparable to:
http://jenkins1.qa.scl3.mozilla.com/view/Bitbar/job/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke.bitbar/536/

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I triggered that adhoc run against master by mistake, use this one instead:
http://jenkins1.qa.scl3.mozilla.com/view/Bitbar/job/flame-kk.ui.adhoc.bitbar/29/

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

Removing review request until I have results I'm happy with. Here's the latest adhoc:
http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc.bitbar/30/

Comment 5

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Hm, yeah. That didn't look so good.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

Looking much better now, and about 10 minutes faster for a smoke suite:
http://jenkins1.qa.scl3.mozilla.com/view/Bitbar/job/flame-kk.ui.adhoc.bitbar/37/

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Geo mentioned wanting a way to override defaults on a per test basis. I've updated the pull request with a unit test that restores the FTU application. A similar approach could be used to skip resetting any defaults, however the approach we use of modifying the settings.json on the device means that only a flash will completely restore the original defaults.

Comment 8

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

I suggested a different method for the defaults override part, but this looks functionally fine as-is. We can always patch in the different method if you end up agreeing. r+.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

I liked your suggestions so have implemented them, please take another look. Before this merges I will rebase and squash the commits.

Comment 10

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

r+ pending passing tests on the re-trigger

Comment 11

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

OK, sounds like I blew it reconciling the original gaia_cleanup() code with the new DEFAULTS structure, and didn't notice we changed assumptions on lockscreen enabling.

Caused a test issue, so flipping to r- pending fix.

Aside from that and the ftu manifest pref, any other start state/behavior changes to highlight?

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Full suite adhoc:
http://jenkins1.qa.scl3.mozilla.com:8080/view/Bitbar/job/flame-kk.ui.adhoc.bitbar/38/

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The adhoc timed out (we have a ~4 hour limit for remote device sessions). I've rebased, pushed, and will re-request review when I have satisfactory results in a MV adhoc job.

Comment 14

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

Full suite adhoc looks pretty good, none of the failures look to be related, and some even look familiar:
http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc/748/

Comment 15

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

Revisiting this. Rebased and resolved conflicts in pull request. Also removing the dependency on Geo's availability to review.

Taskcluster: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=2e856a6bafd171cc98de02b3de10658f15deb542
Jenkins: http://jenkins1.qa.scl3.mozilla.com:8080/job/flame-kk.ui.adhoc/795/

Comment 16

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

I don't see anything major in the code. Treeherder is mostly green: Gip{6,12,19,20,22} are still running after 6 hours. There might be something wrong with the runner.

The naming of `modify_settings()` could be improved, to me. I wouldn't block on that, though.

Comment 17

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Thanks. I've added a docstring for modify_settings. I stopped and reran the TaskCluster builds and they turned green. Oliver: Could you comment on the failures in http://jenkins1.qa.scl3.mozilla.com:8080/job/flame-kk.ui.adhoc/795/HTML_Report/ to say if any are new to this patch?

Comment 18

[Oliver Nelson [:oliverthor]]

Compared the errors/fails observed from the above report:
http://jenkins1.qa.scl3.mozilla.com:8080/job/flame-kk.ui.adhoc/795/HTML_Report/

to b2g-inbound.tinderbox:
http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.tinderbox.ui.functional.smoke/3685/HTML_Report/

Though the adhoc job produced 11 combined errors/fails, only 2 of these were observable in the b2g-inbound.tinderbox pass:

6 fails:
test_sms_contact.py TestContacts.test_sms_contact
* test_dialer_miss_call_from_known_contact_notification.py TestReceiveCallFromKnownContactNotification.test_dialer_miss_call_from_known_contact_notification
** test_a11y_callscreen_visibility.py TestAccessibilityCallscreenVisibility.test_a11y_callscreen_visibility
** test_a11y_phone_call_log.py TestAccessibilityPhoneCallLog.test_phone_call_log
** test_a11y_screen_manager.py TestScreenManagerAccessibility.test_a11y_screen_manager

5 errors:
test_sms_with_attachments.py TestSmsWithAttachments.test_sms_send
* test_keyboard_predictive_key.py TestKeyboardPredictiveKey.test_keyboard_predictive_key
** test_a11y_calendar_settings_view.py TestCalendarSettingsViewAccessibility.test_a11y_calendar_settings_view
** test_a11y_calendar_views_visibility.py TestCalendarViewsVisibilityAccessibility.test_a11y_calendar_views_visibility
** test_a11y_ftu.py TestFtuAccessibility.test_a11y_ftu

* run in non-smoke tests
** not observed in smoke/non-smoke tests

Both observable tests [test_sms_contact && test_sms_with_attachments] passed in the above tinderbox run,
though a separate error was seen on the tinderbox in [test_sms_with_picture_attached] where the traceback is identical.

A separate error occurred in the tinderbox [test_sms_with_picture_attached.py] that was not observed in the adhoc.

Comment 19

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8580159 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/28986

I'm happy that the failures are unrelated. This has now landed in:
https://github.com/mozilla-b2g/gaia/commit/dac46a0369dccb7fd138aac503e2229c96d48e1a
https://github.com/mozilla-b2g/gaia/commit/4de43b6f58a63bd812afb80c0472827f3a6875f3

Comment 20

[Martijn Wargers (dead)]

Sorry for my delay in reviewing. I was looking at the patch, it seemed fine to me, so far I looked at it. But I hadn't tried it out myself yet.

Comment 21

[Dave Hunt [:davehunt] [he/him] ⌚BST]

No problem, I only needed one review but added a few people to increase visibility. So far this is looking okay, decreasing the sanity suite duration by around 2 minutes.