1127724 - [META] 2.2 Test SPEC review

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: General, defect)

Description

[Gerry Chang [:gchang]]

Each feature owner should attach the test spec and invite reviewers to sign off.

Comment 1

[Teri Wen [:twen]]

Attachment: Productivity Copy/Paste test cases

Shing, 
Please help review my test cases. Thanks.

Comment 2

[Hermes Cheng (inactive after July 27, 2015)]

Attachment: App-grouping Test Cases

Comment 3

[Shing Lyu [:shinglyu]]

Attachment: Metrics Test Cases

Comment 4

[Shing Lyu [:shinglyu]]

Comment on attachment 8558285 [details]
Metrics Test Cases

Hi Gerry,

Please review my test cases, the only difference in 2.2 is that there will be more information in the metrics payload.

Comment 5

[Shing Lyu [:shinglyu]]

Comment on attachment 8558285 [details]
Metrics Test Cases

<a href="https://moztrap.mozilla.org/manage/cases/?filter-suite=775">Metrics Test Cases (MozTrap)</a><br>/>
<a href="https://drive.google.com/a/mozilla.com/file/d/0B5hP4LvYzmF5WVVFY2dReWJHSnc/view?usp=sharing">Mind Map</a>

Comment 6

[Shing Lyu [:shinglyu]]

(Ignore Comment 5, I can't delete it)

@Teri: The test cases generally looks good to me. There are some minor issues though:

1. You can consider testing the non-selectable areas by long-tapping the high-risk areas that the RD may forget to mark as non-selectable.

2. Consider using the following Markdown syntax: `[Link Title](http://foo.com)` to create links in those test cases that links to the general cases.

Comment 7

[Eric Chang [:ericcc] [:echang]]

Comment on attachment 8557842 [details]
App-grouping Test Cases

1. Case to remove apps and make sure remaining icon are moved to the left?
2. Case to collapse/Expand in edit mode
3. Delete/move from a group with 1 item, make sure the grouping is gone.
4. What's the max number of icons in a group?

Comment 8

[Alison Shiue]

Attachment: NFC Test Cases

Comment 9

[Hermes Cheng (inactive after July 27, 2015)]

Eric, thank you!

(In reply to Eric Chang [:ericcc] [:echang] from comment #7)
> Comment on attachment 8557842 [details]
> App-grouping Test Cases
> 
> 1. Case to remove apps and make sure remaining icon are moved to the left?
I assume it would be covered by Homescreen old test cases, and will find it and update the suite.
> 2. Case to collapse/Expand in edit mode
covered by 15219, 15220, 15495, 15496
> 3. Delete/move from a group with 1 item, make sure the grouping is gone.
covered by 15290, 15291
> 4. What's the max number of icons in a group?
No, it is not covered now. I assume it is unlimited and would be a potential bug for OOM in the future.......need to think how to cover it......

Comment 10

[Teri Wen [:twen]]

(In reply to Shing Lyu [:slyu] from comment #6)
Thanks Shing for the review. 

> (Ignore Comment 5, I can't delete it)
> 
> @Teri: The test cases generally looks good to me. There are some minor
> issues though:
> 
> 1. You can consider testing the non-selectable areas by long-tapping the
> high-risk areas that the RD may forget to mark as non-selectable.
> 
I have bugs opened, and will add test cases to moztrap.

> 2. Consider using the following Markdown syntax: `[Link
> Title](http://foo.com)` to create links in those test cases that links to
> the general cases.
Changed formatting to include test case title, see https://moztrap.mozilla.org/manage/case/15207/.

Comment 11

[Hermes Cheng (inactive after July 27, 2015)]

Comment on attachment 8557842 [details]
App-grouping Test Cases

leave boundary test not covered...

Comment 12

[Hermes Cheng (inactive after July 27, 2015)]

Attachment: Bug 1001861 - short_name support.txt

Alison, since you have developed a webapp for FxOS, please help review this one about webapp manifest attribute. Thanks!

Comment 13

[Alison Shiue]

Comment on attachment 8558969 [details]
Bug 1001861 - short_name support.txt

Hi Hermes,
I suggest add some test cases about the correlation between "name" and "short_name". But I am not sure if they have clear defined since I could not find the design conclusion from the bug discussion.

Comment 14

[Hermes Cheng (inactive after July 27, 2015)]

(In reply to Alison Shiue from comment #13)
> Comment on attachment 8558969 [details]
> Bug 1001861 - short_name support.txt
> 
> Hi Hermes,
> I suggest add some test cases about the correlation between "name" and
> "short_name". But I am not sure if they have clear defined since I could not
> find the design conclusion from the bug discussion.

Thanks! asking below questions to enhance coverage...

1. Will short_name be used in search app results?
2. Is there any correlation between "name" and "short_name"? Do we have any rule for it? Are we going to use "app-validator" or some methods to verify its correctness?
3. What is the maximum length of "short_name"? It seems not be defined in W3C spec.

Comment 15

[Eric Chang [:ericcc] [:echang]]

Attachment: Bug 911055 - [Dialer]DuT doesn't allow to dial a <<pause>>

Hi Alison, Please help to review test case for bug 911055, thank you

Comment 16

[Eric Chang [:ericcc] [:echang]]

Comment on attachment 8557842 [details]
App-grouping Test Cases

r+, thank you.

Comment 17

[Hermes Cheng (inactive after July 27, 2015)]

Attachment: Rocketbar v2.2 - Search Provider (Bug 1098494) & part of Rocketbar refactor (Bug 1099157)

Gerry, help!

Comment 18

[Hermes Cheng (inactive after July 27, 2015)]

Comment on attachment 8558902 [details]
NFC Test Cases

Comments only for SE API:
4.1.1 without SIM card case?
4.3.* reconnect a logical/basic channel?
4.4.4 transmit with incorrect SECommand?
    a. right format but not exist command
    b. incorrect format

Test case to cover "SEIoError" and "SEGenericError"?

Comment 19

[Alison Shiue]

(In reply to Hermes Cheng[:hermescheng] from comment #18)
> Comment on attachment 8558902 [details]
> NFC Test Cases
> 
> Comments only for SE API:
> 4.1.1 without SIM card case?
https://moztrap.mozilla.org/manage/cases/?filter-id=15275 covered this case.

> 4.3.* reconnect a logical/basic channel?
I think if a channel has been closed, we just can open a new channel to the specific applet again, not reconnect.

> 4.4.4 transmit with incorrect SECommand?
>     a. right format but not exist comman
OK, I would try add a new test case such as "Open a channel to PPSE but send CRS APDU command".

>     b. incorrect format
I have ever created a test case "Command parameter shorter than 4 bytes" which already removed from test spec since developer suggest "SEChannel.transmit expects SECommand as an argument. SECommand is strongly typed and each of the 4 bytes is required in SECommand constructor. So it's not possible to transmit a shorter command." 
> 
> Test case to cover "SEIoError" and "SEGenericError"?
Until now, the error code still not defined well. (bug 1118102)

Comment 20

[Alison Shiue]

Comment on attachment 8560324 [details]
Bug 911055 - [Dialer]DuT doesn't allow to dial a <<pause>>

Looks good to me, r+, thank you.

Comment 21

[Hermes Cheng (inactive after July 27, 2015)]

Comment on attachment 8558902 [details]
NFC Test Cases

For Bug 1102019:
MozTrap ID - Comment
#15273  - similar with #15272, insert step 2: open a Contact app and then open NFC test app.
#15271  - do we need to add a similar case which use two phone, and send NDEF to each other?
#15270 - How do we verify the results written to tag?
#15269 - Suggest to tap the tag to show its content and then to format it. Make sure the tag contains correct info

Comment 22

[Alison Shiue]

(In reply to Hermes Cheng[:hermescheng] from comment #21)
> Comment on attachment 8558902 [details]
> NFC Test Cases
> 
> For Bug 1102019:
> MozTrap ID - Comment
> #15273  - similar with #15272, insert step 2: open a Contact app and then
> open NFC test app.
#15272 open Contact at step 4.
#15273 doesn't need to open Contact app since I want to test that System app would not dispatch event to suitable (Contact) app when current foreground app can handle it. The expected results at step 3 are: (1) NFC test app should show tag info (2) Contacts app should not import contact from tag

> #15271  - do we need to add a similar case which use two phone, and send
> NDEF to each other?
This case is exactly for testing send NDEF via phone to phone (peer to peer)

> #15270 - How do we verify the results written to tag?
We verify at Step 4: Go back to home screen and tap tag again - Browser would open URL which is written by NFC test app 

> #15269 - Suggest to tap the tag to show its content and then to format it.
> Make sure the tag contains correct info
OK, I've updated. (https://moztrap.mozilla.org/manage/cases/?filter-id=15269)

Thanks for your review!

Comment 23

[Eric Chang [:ericcc] [:echang]]

0. Talk to EPM about the process.

1. A meta bug 
2. QA test review bug links to 1.
3. Also add UX link in 2.
4. Reviewer: UX, RD, EPM, QA.. suggest to have 1 of each.

Comment 24

[Shing Lyu [:shinglyu]]

Comment on attachment 8557792 [details]
Productivity Copy/Paste test cases

Sorry for the late r+

Comment 25

[Shing Lyu [:shinglyu]]

Attachment: Metrics Test Cases 20150226

Updated test cases for new features added to 2.2 scope

Comment 26

[Gerry Chang [:gchang]]

Comment on attachment 8569627 [details]
Metrics Test Cases 20150226

I think one more category is to have background app running (ex: Music) and see if the background app will be calculated.

Comment 27

[BMO Automation]

Firefox OS is not being worked on