1069998 - Better error messages for gaia UI tests when no SIM around?

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Martijn Wargers (dead)]

Attachment: gaiuitests.log

I accidentally ran all the Gaia UI tests without a SIM card in the Flame phone.
I guess it's logical to get errors in some of the tests.
But perhaps these tests need to get a better error message, like 'No SIM card found' or something?
And/or perhaps it would be good to have some kind of config var in testvars.json that let you say "without_sim" and the tests that need a SIM card would be skipped?
(Or perhaps I should just not be so stupid to forget to put a SIM card in?)

TEST-UNEXPECTED-ERROR | test_browser_cell_data.py TestBrowserCellData.test_browser_cell_data | ScriptTimeoutException: ScriptTimeoutException: timed out

TEST-UNEXPECTED-ERROR | test_call_contact.py TestContacts.test_call_contact | NoSuchElementException: NoSuchElementException: Unable to locate element: .handled-call.outgoing

TestImportContactsFromSIM.test_import_contacts_from_SIM | JavascriptException: JavascriptException: TypeError: icc.readContacts is not a function
stacktrace:
	execute_async_script @gaia_test.py, line 181
	inline javascript, line 284
	src: "undefined"

TEST-UNEXPECTED-ERROR | test_cost_control_ftu.py TestCostControlFTU.test_cost_control_ftu | TimeoutException: TimeoutException: Timed out after 10.0 seconds

TEST-UNEXPECTED-ERROR | test_cost_control_data_alert_mobile.py TestCostControlDataAlertMobile.test_cost_control_data_alert_mobile | ScriptTimeoutException: ScriptTimeoutException: timed out

TEST-UNEXPECTED-ERROR | test_cost_control_reset_wifi.py TestCostControlReset.test_cost_control_reset_wifi | TimeoutException: TimeoutException: Timed out after 10.0 seconds

TEST-UNEXPECTED-ERROR | test_dialer.py TestDialer.test_dialer_make_call | NoSuchElementException: NoSuchElementException: Unable to locate element: .handled-call.outgoing

TEST-UNEXPECTED-ERROR | test_call_log_all_calls.py TestCallLogAllCalls.test_call_log_all_calls | NoSuchElementException: NoSuchElementException: Unable to locate element: .handled-call.outgoing

TEST-UNEXPECTED-ERROR | test_fmradio_add_remove_from_favorites.py TestFMRadioAddRemoveFavorites.test_add_remove_from_favorites | TimeoutException: TimeoutException: Timed out after 10.1 seconds

TEST-UNEXPECTED-ERROR | test_fmradio_find_stations.py TestFMRadioFindStations.test_find_next_previous_station | TimeoutException: TimeoutException: Timed out after 10.0 seconds

TEST-UNEXPECTED-ERROR | test_fmradio_frequency_dialer.py TestFMRadioFreqDialer.test_radio_frequency_dialer | TimeoutException: TimeoutException: Timed out after 10.1 seconds

TEST-UNEXPECTED-ERROR | test_fmradio_turn_on_off.py TestFMRadioTurnOnOff.test_turn_radio_on_off | TimeoutException: TimeoutException: Timed out after 10.0 seconds

TEST-UNEXPECTED-ERROR | test_ftu_with_tour.py TestFtu.test_ftu_with_tour | IOError: Connection to Marionette server is lost. Check gecko.log (desktop firefox) or logcat (b2g) for errors.

TEST-UNEXPECTED-ERROR | test_add_to_an_existing_contact_from_messages.py TestSmsAddToExistingContact.test_sms_add_number_to_existing_contact | KeyError: 'carrier'

TEST-UNEXPECTED-ERROR | test_add_to_new_contact_from_messages.py TestSmsCreateContact.test_sms_create_new_contact | KeyError: 'carrier'

TEST-START | test_sms.py TestSms.test_sms_send
TEST-UNEXPECTED-ERROR | test_sms.py TestSms.test_sms_send | KeyError: 'carrier'

TEST-UNEXPECTED-ERROR | test_sms_with_attachments.py TestSmsWithAttachments.test_sms_send | ScriptTimeoutException: ScriptTimeoutException: timed out

TEST-UNEXPECTED-ERROR | test_sms_to_dialer.py TestDialerFromMessage.test_dialer_from_message | KeyError: 'carrier'

TEST-UNEXPECTED-ERROR | test_sms_notification.py TestSmsNotification.test_sms_notification | KeyError: 'carrier'

TEST-UNEXPECTED-ERROR | test_persona_app.py TestPersonaStandard.test_persona_standard_sign_in | IOError: Connection to Marionette server is lost. Check gecko.log (desktop firefox) or logcat (b2g) for errors.

It seems like the whole run didn't complete, because the Gaia UI tests end with a connection lost error message.

Comment 1

[Zac C (:zac)]

Yeah we'd like this ability too but the issue here is that we can't find out whether the phone has a SIM or not without using Firefox OS itself, so if Firefox OS were to erroneously report that it has no SIM (which it does do on occasion) then we'd skip tests or get a false positive which we don't want to risk.

The best way to avoid the SIM tests when you're running is to use the tag `--type=b2g-carrier` to skip all tests that are tagged with needing carrier/cell service. We've similar tags for wifi, sdcard, etc too. But generally we assume that the device has access to all of the hardware it needs to run every test because we want to run every test every time.

> It seems like the whole run didn't complete, because the Gaia UI tests end with a connection lost error message.
This is a device crash and unrelated to not having a SIM (well, maybe it's a bug that exists when you don't have a SIM in the phone!)

Comment 2

[Martijn Wargers (dead)]

(In reply to Zac C (:zac) from comment #1)
> The best way to avoid the SIM tests when you're running is to use the tag
> `--type=b2g-carrier` to skip all tests that are tagged with needing
> carrier/cell service. We've similar tags for wifi, sdcard, etc too. But
> generally we assume that the device has access to all of the hardware it
> needs to run every test because we want to run every test every time.

Ah, I see. This corresponds to the manifest files like this:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/messages/manifest.ini

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We can improve the error message by catching the KeyError when the carrier details are not in the testvars file, or avoid it entirely using a condition in the test - I think this would be a worthwhile enhancement.

As Zac says it's possible that if we check for a SIM programmatically that we can't trust the result so it's better to state the tests you want to run and expect to pass by filtering using the manifest tags.

Comment 4

[Martijn Wargers (dead)]

(In reply to Dave Hunt (:davehunt) from comment #3)
> As Zac says it's possible that if we check for a SIM programmatically that
> we can't trust the result so it's better to state the tests you want to run
> and expect to pass by filtering using the manifest tags.

Ok, but using something like 'type=b2g-carrier' might not work on sub manifest files correctly, see bug 1073048.

Comment 5

[Martijn Wargers (dead)]

Marking WONTFIX, sorry for the bug spam. If somebody still wants to work on this, please file a new bug for it.