1140190 - [meta] Reset Gaia UI Test results back to green

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

We have a large number of failing tests on Gaia UI Test right now. 

In order to make the impending switchover to BitBar sane and understand if we're getting comparable results, we need to reset the test runs to a green state.

At this point, we should slash and burn with a low threshold: anything that either consistently fails, or which fails the majority of the time when re-run should be disabled/xfailed.

Once we're back at test-failure zero, we'll then immediately turn around into fixing those tests, getting them solid where the problem is the test, and getting them reintroduced again. 

If a given test turns out to be implicitly not-solid, we can either decide that test isn't a good idea, or use this as an opportunity to start planning out our Unstable test suite.

Here's a list of current suspects, but please refer to latest test results. 

If the existing results are pretty conclusive about frequency, we should just go off that for expediency's sake.

Definite disables/xfails:

[Not reproduced manually] test_ftu_status_bar - Intermittent failure, 9/10 passing with local automation.

[Reproduces manually] test_settings_from_inbox - https://bugzil.la/1138517 UI flow change for reaching Settings in Messages causing permafail; not a product failure

[Not reproduced manually] test_change_language_settings - https://bugzil.la/1137766 Exception finding Language. Reproduces via local automation; 10/10 failing

[Not reproduced manually] test_toggle_airplane_mode - https://bugzil.la/1131070 Device is not recognizing that airplane mode is enabled, even though it shows that it is (xfail?)

[Not reproduced manually] test_lockscreen_time_check - https://bugzil.la/1140027 test_lockscreen_time_check.py intermittently fails checking the time at the lockscreen. - 14/20 fail

[Not reproduced manually] test_change_language_settings_rtl - https://bugzil.la/1137766 Exception finding Language. Reproduces via local automation; 10/10 failing

Possible disable/xfails:

[Not reproduced manually] test_toggle_bluetooth_settings - https://bugzil.la/1134244 UI state of visibility changes after changing name during test (if it continues to fail)

[Not reproduced manually] test_sms_notification - Did not reproduce via local automation: 10/10 passing (if it continues to fail)

[Not reproduced manually] test_settings_change_time_format - https://bugzil.la/1136156 Lockscreen shows incorrect time format after it's modified for the first time. Bug is resolved fix, keeping an eye on it for the patch to land

[Not reproduced manually] test_dialer_clear_miss_call_notification -  https://bugzil.la/1140017 test_dialer_clear_miss_call_notification.py is timing out after receiving the missed call

[Not reproduced manually] test_add_to_new_contact_from_messages - Did not reproduce via local automation: 10/10 passing

[Not reproduced manually] test_sms_to_dialer - https://bugzil.la/1140039  test_sms_to_dialer.py is intermittently failing with a timeout.

Comment 1

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Doing history analysis on these.

[Not reproduced manually] test_ftu_status_bar - Intermittent failure, 9/10 passing with local automation:

I shouldn't have included this one in "definitely" above--think I read that as 9/10 failed. This seems to be a very infrequent failure on Jenkins. I'm surprised it was only 9/10 locally. I think we can let this go for the moment.

[Reproduces manually] test_settings_from_inbox - https://bugzil.la/1138517 UI flow change for reaching Settings in Messages causing permafail; not a product failure

This is now fixed, thanks to Martijn.

Comment 2

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

[Not reproduced manually] test_change_language_settings - https://bugzil.la/1137766 Exception finding Language. Reproduces via local automation; 10/10 failing

[Not reproduced manually] test_change_language_settings_rtl - https://bugzil.la/1137766 Exception finding Language. Reproduces via local automation; 10/10 failing

Consistent fails on-device only. These work on desktop, as we unfortunately found it in the initial xfail patch on 113776. This is probably related to the unit test failure for the same in Bug 1139215. I've filed a PR to xfail these on-device only in Bug 1140669.

Comment 3

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

[Not reproduced manually] test_toggle_airplane_mode - https://bugzil.la/1131070 Device is not recognizing that airplane mode is enabled, even though it shows that it is (xfail?)

This has been disabled by jlorenzo.

[Not reproduced manually] test_lockscreen_time_check - https://bugzil.la/1140027 test_lockscreen_time_check.py intermittently fails checking the time at the lockscreen. - 14/20 fail

Has also been disabled by jlorenzo.

That finishes out the "definitely" set above.

Comment 4

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

[Not reproduced manually] test_toggle_bluetooth_settings - https://bugzil.la/1134244 UI state of visibility changes after changing name during test (if it continues to fail)

Has a ~50% fail rate, which for a smoke/sanity test is pretty awful. Disabling for investigation in Bug 1141124

Comment 5

[BMO Automation]

Firefox OS is not being worked on