[meta] Reset Gaia UI Test results back to green

RESOLVED WONTFIX

Status

defect
RESOLVED WONTFIX
4 years ago
Last year

People

(Reporter: gmealer, Unassigned)

Tracking

({meta})

Firefox Tracking Flags

(Not tracked)

Details

We have a large number of failing tests on Gaia UI Test right now. 

In order to make the impending switchover to BitBar sane and understand if we're getting comparable results, we need to reset the test runs to a green state.

At this point, we should slash and burn with a low threshold: anything that either consistently fails, or which fails the majority of the time when re-run should be disabled/xfailed.

Once we're back at test-failure zero, we'll then immediately turn around into fixing those tests, getting them solid where the problem is the test, and getting them reintroduced again. 

If a given test turns out to be implicitly not-solid, we can either decide that test isn't a good idea, or use this as an opportunity to start planning out our Unstable test suite.

Here's a list of current suspects, but please refer to latest test results. 

If the existing results are pretty conclusive about frequency, we should just go off that for expediency's sake.

Definite disables/xfails:

[Not reproduced manually] test_ftu_status_bar - Intermittent failure, 9/10 passing with local automation.

[Reproduces manually] test_settings_from_inbox - https://bugzil.la/1138517 UI flow change for reaching Settings in Messages causing permafail; not a product failure

[Not reproduced manually] test_change_language_settings - https://bugzil.la/1137766 Exception finding Language. Reproduces via local automation; 10/10 failing

[Not reproduced manually] test_toggle_airplane_mode - https://bugzil.la/1131070 Device is not recognizing that airplane mode is enabled, even though it shows that it is (xfail?)

[Not reproduced manually] test_lockscreen_time_check - https://bugzil.la/1140027 test_lockscreen_time_check.py intermittently fails checking the time at the lockscreen. - 14/20 fail

[Not reproduced manually] test_change_language_settings_rtl - https://bugzil.la/1137766 Exception finding Language. Reproduces via local automation; 10/10 failing

Possible disable/xfails:

[Not reproduced manually] test_toggle_bluetooth_settings - https://bugzil.la/1134244 UI state of visibility changes after changing name during test (if it continues to fail)

[Not reproduced manually] test_sms_notification - Did not reproduce via local automation: 10/10 passing (if it continues to fail)

[Not reproduced manually] test_settings_change_time_format - https://bugzil.la/1136156 Lockscreen shows incorrect time format after it's modified for the first time. Bug is resolved fix, keeping an eye on it for the patch to land

[Not reproduced manually] test_dialer_clear_miss_call_notification -  https://bugzil.la/1140017 test_dialer_clear_miss_call_notification.py is timing out after receiving the missed call

[Not reproduced manually] test_add_to_new_contact_from_messages - Did not reproduce via local automation: 10/10 passing

[Not reproduced manually] test_sms_to_dialer - https://bugzil.la/1140039  test_sms_to_dialer.py is intermittently failing with a timeout.
Depends on: 1137766
Depends on: 1140027
Depends on: 1131070
Doing history analysis on these.

[Not reproduced manually] test_ftu_status_bar - Intermittent failure, 9/10 passing with local automation:

I shouldn't have included this one in "definitely" above--think I read that as 9/10 failed. This seems to be a very infrequent failure on Jenkins. I'm surprised it was only 9/10 locally. I think we can let this go for the moment.

[Reproduces manually] test_settings_from_inbox - https://bugzil.la/1138517 UI flow change for reaching Settings in Messages causing permafail; not a product failure

This is now fixed, thanks to Martijn.
Depends on: 1140669
No longer depends on: 1137766
[Not reproduced manually] test_change_language_settings - https://bugzil.la/1137766 Exception finding Language. Reproduces via local automation; 10/10 failing

[Not reproduced manually] test_change_language_settings_rtl - https://bugzil.la/1137766 Exception finding Language. Reproduces via local automation; 10/10 failing

Consistent fails on-device only. These work on desktop, as we unfortunately found it in the initial xfail patch on 113776. This is probably related to the unit test failure for the same in Bug 1139215. I've filed a PR to xfail these on-device only in Bug 1140669.
[Not reproduced manually] test_toggle_airplane_mode - https://bugzil.la/1131070 Device is not recognizing that airplane mode is enabled, even though it shows that it is (xfail?)

This has been disabled by jlorenzo.

[Not reproduced manually] test_lockscreen_time_check - https://bugzil.la/1140027 test_lockscreen_time_check.py intermittently fails checking the time at the lockscreen. - 14/20 fail

Has also been disabled by jlorenzo.

That finishes out the "definitely" set above.
[Not reproduced manually] test_toggle_bluetooth_settings - https://bugzil.la/1134244 UI state of visibility changes after changing name during test (if it continues to fail)

Has a ~50% fail rate, which for a smoke/sanity test is pretty awful. Disabling for investigation in Bug 1141124
Firefox OS is not being worked on
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.