1098998 - [MCTS] error on test "webapi_tests.mobile_message.test_sms_outgoing.TestSmsOutgoing.test_sms_outgoing"

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Certification Suite, defect)

Description

[pkj1020]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36

Steps to reproduce:

I have conducted MCTS using Frame device.



Actual results:

I found out the error on the test "webapi_tests.mobile_message.test_sms_outgoing.TestSmsOutgoing.test_sms_outgoing". SMS has been sent successfully, but the result is different depending on the delivery report option.

Settings -> Messaging Settings 

1) Delivery reports off -> PASS
2) Delivery reports on -> FAIL


Expected results:

Always "PASS" regardless of the option.

Comment 1

[Rachelle Yang [:ryang][[e-mail]]]

Hi Jonathan, would you please kindly help look into this ?

The issue happened on Flame 2.0 as well.

Thank you very much !

Comment 2

[Rachelle Yang [:ryang][[e-mail]]]

Hi Jongsoo,
would you please kindly provide the log and failed report for this SMS issue ?

and besides, would you please provide the Build Id, SW version of Flame and MCTS 2.0 version as well ?

Thank you very much !

Comment 3

[pkj1020]

Attachment: mcts_sms_log.txt

Comment 4

[pkj1020]

I attached the log file(mcts_sms_log.txt) and the failed report is below.

==================================================================================================
 1:14.38 TEST_START: MainThread webapi_tests.mobile_message.test_sms_outgoing.TestSmsOutgoing.test_sms_outgoing
 1:37.82 TEST_END: MainThread FAIL, expected PASS
Traceback (most recent call last):
  File "/root/Desktop/MCTS/fxos-certsuite/certsuite_venv/local/lib/python2.7/site-packages/fxos_certsuite-2.0.1-py2.7.egg/webapi_tests/mobile_message/test_sms_outgoing.py", line 39, in test_sms_outgoing
    "Sent SMS MozSmsMessage.deliveryStatus should be 'success' or 'not-applicable'")
AssertionError: Sent SMS MozSmsMessage.deliveryStatus should be 'success' or 'not-applicable'
==================================================================================================

Also, I have conducted MCTS using the source from github. 
(https://github.com/mozilla-b2g/fxos-certsuite)

Please refer Flame information below.

Software : Boot2Gecko 2.0.0.0-prerelease
Build ID : 20140930112111

Thank you.

Comment 5

[Jonathan Griffin (:jgriffin)]

Looks like we either need to:

1 - make having delivery reports enabled a prerequisite
2 - turn on delivery reports at the start of the test
or
3 - disable checks which can fail if delivery reports are enabled

I think delivery reports are somewhat sensitive to network and environment, so I might be inclined towards 3 if no one has an alternate opinion.

Comment 6

[Rachelle Yang [:ryang][[e-mail]]]

Hi Jonathan, thanks for your feedback and comments!

Since partner is very close to MP and also expect MCTS on 2.0 to be done ASAP.

Do you think partner should go for wavier process for this sms case or when the updated version of MCTS 2.0 with the option 3 in comment5 will be ready for them ?

Thank you very much !!

Comment 7

[Jonathan Griffin (:jgriffin)]

I think apply for a waiver for this case is appropriate if they need to submit results before we release a new version of MCTS.

Comment 8

[Rachelle Yang [:ryang][[e-mail]]]

Hi Faramarz, per comment 7,
Could you lease kindly help with the Waiver approval process ? 
Thank you very much!

Comment 9

[Rachelle Yang [:ryang][[e-mail]]]

Based on the comment 7 above, and also see comment 5 in Meta bug Bug1104617, this waiver is granted.
Set this bug to Resolved, WONTFIX

Comment 10

[Karen Ward [:kward]]

Update to MCTS acknowledged by Jonathon (see comment #5).
Also, response by faramarz no longer needed.