970615 - (b2g-cert-tests) [meta] Create B2G Certification Tests

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Certification Suite, defect)

Description

[Jonathan Griffin (:jgriffin)]

This is a tracking bug for the creation of a set of tests that can be used by OEM's to test B2G certification requirements.

Comment 1

[Jonathan Griffin (:jgriffin)]

Patrick, our current README is at https://github.com/mozilla-b2g/fxos-certsuite; can you review?

Comment 2

[Patrick Hundal]

(In reply to Jonathan Griffin (:jgriffin) from comment #1)
> Patrick, our current README is at
> https://github.com/mozilla-b2g/fxos-certsuite; can you review?

Jonathan,

These look great... These look simple enough for me to follow. ;)

One question... Just so I understand, the end user turns on remote debugging on device, joins the same wifi network as the machine running the tests, and that's it?  You don't have to tell the test machine any information about the Firefox OS device?  (IP, device name, etc...)  Does it do some type of port scan over the wifi network?

Comment 3

[Jonathan Griffin (:jgriffin)]

(In reply to Patrick Hundal from comment #2)
> (In reply to Jonathan Griffin (:jgriffin) from comment #1)
> > Patrick, our current README is at
> > https://github.com/mozilla-b2g/fxos-certsuite; can you review?
> 
> Jonathan,
> 
> These look great... These look simple enough for me to follow. ;)
> 
> One question... Just so I understand, the end user turns on remote debugging
> on device, joins the same wifi network as the machine running the tests, and
> that's it?  You don't have to tell the test machine any information about
> the Firefox OS device?  (IP, device name, etc...)  Does it do some type of
> port scan over the wifi network?

Once the tests start, they instruct the user to launch the browser app and navigate to a web page that's hosted by the test harness; e.g., something like http://192.168.1.1:8080/.

Comment 4

[Patrick Hundal]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> (In reply to Patrick Hundal from comment #2)
> > (In reply to Jonathan Griffin (:jgriffin) from comment #1)
> > > Patrick, our current README is at
> > > https://github.com/mozilla-b2g/fxos-certsuite; can you review?
> > 
> > Jonathan,
> > 
> > These look great... These look simple enough for me to follow. ;)
> > 
> > One question... Just so I understand, the end user turns on remote debugging
> > on device, joins the same wifi network as the machine running the tests, and
> > that's it?  You don't have to tell the test machine any information about
> > the Firefox OS device?  (IP, device name, etc...)  Does it do some type of
> > port scan over the wifi network?
> 
> Once the tests start, they instruct the user to launch the browser app and
> navigate to a web page that's hosted by the test harness; e.g., something
> like http://192.168.1.1:8080/.

Awesome. ;)

Comment 5

[Jonathan Griffin (:jgriffin)]

Attachment: fxos-MCTS-1.3.0.2.zip

This is the Sprint 1 release of the fxos-certsuite tool.  It contains the same content as the MWC release, plus the following:

- omni_diff.py to produce diffs of omni_analyzer results
- a metaharness which is used to run the tests and which will be refined in future sprints
- a WebIDL verifier which verifies top-level WebIDL's in an unprivileged context; more will be added in future sprints
- a small set of WebAPI tests which are designed to be run manually, and invoked separately from the main harness; these will be hooked up to a harness in the next sprint

Comment 6

[Karen Ward [:kward]]

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> Created attachment 8391576 [details]
> fxos-MCTS-1.3.0.2.zip
> 
> This is the Sprint 1 release of the fxos-certsuite tool.  It contains the
> same content as the MWC release, plus the following:
> 
> - omni_diff.py to produce diffs of omni_analyzer results
> - a metaharness which is used to run the tests and which will be refined in
> future sprints
> - a WebIDL verifier which verifies top-level WebIDL's in an unprivileged
> context; more will be added in future sprints
> - a small set of WebAPI tests which are designed to be run manually, and
> invoked separately from the main harness; these will be hooked up to a
> harness in the next sprint

Bug 984547 created to promote Sprint 1 deliverable to live partner site.

Comment 7

[Jonathan Griffin (:jgriffin)]

Attachment: fxos-MCTS-1.3.0.3.zip

This is the Sprint 2 release of the fxos-certsuite tool.  It contains the same content as the Sprint 1 release, plus the following:

* The data for the WebIDL comparisons is now generated by the in-tree WebIDL.py.
* The WebIDL comparisons now use a static reference to compare against; this eliminates false failures due to permissions issues or cases where our implementation of a WebIDL doesn't match the expectations set by testharness.js.
* The window and navigator name checkers are now recursive, and the navigator name checker checks for objects which are null on a reference version but not-null on the device, and vice-versa.  Failures of these null/not-null checks can be indicative of permissions problems on the device.
* The meta-harness now installs Marionette as an extension on the target device.
* A prototype of the new semi-automatic WebAPI test harness is included, along with a small set of tests; see RUNNING_WEBAPI_TESTS.rst in the archive for details.  This will be connected to the meta-harness in the next sprint.
* An automatic app generator/installer has been written and is used by the semi-automatic WebAPI harness, and will be used by the WebIDL tests in the next sprint.
* A document which describes how to interpret the results is included, see INTERPRETING_RESULTS.rst.

Comment 8

[Karen Ward [:kward]]

fxos-MCTS-1.3.0.3.zip posted on partner site.

Comment 9

[Jonathan Griffin (:jgriffin)]

Attachment: fxos-MCTS-1.3.0.4.zip

This is the Sprint 3 release of the fxos-certsuite tool.  It contains the same content as the Sprint 2 release, plus the following:

* The WebIDL tests and window and navigator object tests are now run three times, once in each of the following contexts:  unprivileged, privileged, and certified.
* New semi-auto WebAPI tests have been added for the following API's:  telephony, vibration, proximity, orientation, tcp_socket.  The existing SMS tests have been improved.
* Integration points have been added to the meta-harness which will allow it to be connected to the semi-auto harness and to the web-platform-tests; the actual connections will likely be implemented in Sprint 4.
* The existing suite has been broken into a set of subtests, and the meta-harness can now list and execute these individually.
* Several bugs in the semi-auto harness have been fixed.

Comment 10

[Karen Ward [:kward]]

Version fxos-MCTS-1.3.0.4.zip posted on Mobile Partner Site.

Comment 11

[Jonathan Griffin (:jgriffin)]

This is the 1.3.0.5 release of the fxos-certsuite tool.  It contains the same content as the 1.3.0.4 release, plus the following:

* New webapi tests have been added for fm_radio
* The semi-auto webapi tests now prevent the lockscreen from turning on and interrupting the tests
* A number of bug fixes have been made to the semi-auto test harness
* The meta-harness has been updated to better support multiple test suites

release url: http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.5.zip

(I'm no longer attaching the zip files to this bug, since they exceed the max size allowed by bugzilla.)

Comment 12

[Karen Ward [:kward]]

CTS V1.3.0.5 posted to the partner website.

Comment 13

[Robert Wood [:rwood]]

Created a tracking bug for the WebAPI test development (Bug 1008970)

Comment 14

[(not currently active) Ted Mielczarek]

Here's the 1.3.0.6. I didn't get it ready Friday because we hit a device-specific crash with the getUserMedia API and it took some time to narrow it down:
http://people.mozilla.org/~tmielczarek/fxos-MCTS-1.3.0.6.zip

New since the last release:
* We now run WebAPI tests in unprivileged, privileged, and certified apps to test app permissions
* Device state is backed-up before running tests and restored afterwards

Comment 15

[Karen Ward [:kward]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #14)
> Here's the 1.3.0.6. I didn't get it ready Friday because we hit a
> device-specific crash with the getUserMedia API and it took some time to
> narrow it down:
> http://people.mozilla.org/~tmielczarek/fxos-MCTS-1.3.0.6.zip
> 
> New since the last release:
> * We now run WebAPI tests in unprivileged, privileged, and certified apps to
> test app permissions
> * Device state is backed-up before running tests and restored afterwards

V1.3.0.6 has been uploaded to the Mobile Partner Website.

Comment 16

[Jonathan Griffin (:jgriffin)]

This is the 1.3.0.7 release of the fxos-certsuite tool.  It contains the same content as the previous release, plus the following:

* WebAPI verifier tests are completely automated now, no manual interaction required
* New online docs at http://fxos-certsuite.readthedocs.org
* The suite of semi-automatic WebAPI tests can now be invoked using webapi_tests/setup_and_run.sh
* Telephony tests no longer fail on Tarako devices
* Result files are now timestamped

release url: http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.7.zip

Comment 17

[Karen Ward [:kward]]

(In reply to Jonathan Griffin (:jgriffin) from comment #16)
> This is the 1.3.0.7 release of the fxos-certsuite tool.  It contains the
> same content as the previous release, plus the following:
> 
> * WebAPI verifier tests are completely automated now, no manual interaction
> required
> * New online docs at http://fxos-certsuite.readthedocs.org
> * The suite of semi-automatic WebAPI tests can now be invoked using
> webapi_tests/setup_and_run.sh
> * Telephony tests no longer fail on Tarako devices
> * Result files are now timestamped
> 
> release url: http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.7.zip

MCTS-1.3.0.7 uploaded today to https://mobilepartners.mozilla.org/market/certify-your-ffos-device/

Comment 18

[Jonathan Griffin (:jgriffin)]

This is the 1.3.0.8 release of the fxos-certsuite tool.  In contains the same content
as the previous release, plus the following:

* The first set of the web-platform-tests
* A suite of permissions model tests
* New semi-automated WebAPI tests for geolocation and bluetooth

release url:  http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.8.zip

Comment 19

[Karen Ward [:kward]]

Request: Will you change the "Readme.md" file in the "Mozilla Certification Test Suite [MCTS]" to a .pdf or other format that is compatible with Mac's and non-PC platforms. (Reference Comment 5.3 in Bug 1010043)

Comment 20

[Jonathan Griffin (:jgriffin)]

(In reply to Karen Ward [:kward] from comment #19)
> Request: Will you change the "Readme.md" file in the "Mozilla Certification
> Test Suite [MCTS]" to a .pdf or other format that is compatible with Mac's
> and non-PC platforms. (Reference Comment 5.3 in Bug 1010043)

I'll change this to a .txt file so it's point-and-click friendly.

Comment 21

[Andreas Tolfsen ❲:ato❳]

(In reply to Jonathan Griffin (:jgriffin) from comment #20)
> (In reply to Karen Ward [:kward] from comment #19)
> > Request: Will you change the "Readme.md" file in the "Mozilla Certification
> > Test Suite [MCTS]" to a .pdf or other format that is compatible with Mac's
> > and non-PC platforms. (Reference Comment 5.3 in Bug 1010043)
> 
> I'll change this to a .txt file so it's point-and-click friendly.

I think probably having Sphinx generate the documentation in docs/ to PDF would be more appropriate.  IIRC it's something like `make latex` or `make pdf`.

Comment 22

[Jonathan Griffin (:jgriffin)]

I think the issue here is that the .md file (which is just a minimal pointer to the readthedocs site) doesn't open in a text editor by default if you're double-clicking on it from the Finder, etc.

But, since we're going to branch the certsuite repo, it means different versions of the docs will be needed, so it probably makes sense to include the docs themselves in PDF form, like you say.

Comment 23

[Karen Ward [:kward]]

(In reply to Jonathan Griffin (:jgriffin) from comment #18)
> This is the 1.3.0.8 release of the fxos-certsuite tool.  In contains the
> same content
> as the previous release, plus the following:
> 
> * The first set of the web-platform-tests
> * A suite of permissions model tests
> * New semi-automated WebAPI tests for geolocation and bluetooth
> 
> release url:  http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.8.zip

Version 1.3.0.8 of the CTS posted to mobile partner site.

Comment 24

[Karen Ward [:kward]]

Hi.  Just got a request from Legal team (Geoff Piper) to also have the License file either in pdf or Notepad format in future versions. OK?

Comment 25

[Jonathan Griffin (:jgriffin)]

(In reply to Karen Ward [:kward] from comment #24)
> Hi.  Just got a request from Legal team (Geoff Piper) to also have the
> License file either in pdf or Notepad format in future versions. OK?

Sure thing!

Comment 26

[Jonathan Griffin (:jgriffin)]

This is the 1.3.0.9 release of the MCTS.  New features in this release:

* A new HTML report which contains the results of the web-platform-tests
* Documentation in PDF format is included within the suite
* The webapi tests are more tolerant of crashes and low-memory conditions
* The semi-auto tests are now run first, so that the tests can then finish unattended
* New semi-auto tests for several WebAPIs

Release url: http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.9.zip

Comment 27

[Karen Ward [:kward]]

Updated on the Mobile Partner site on July 2.  Do we anticipate any issues with downloading this file from the site given it's large size?

Comment 28

[Ben (:bensternthal)]

Not from my perspective. Double checking with Josh.

Comment 29

[Josh Mize [:jgmize]]

The attached files are all well under the 100 MB limit currently set on the server.

Comment 30

[Jonathan Griffin (:jgriffin)]

This is the 1.3.0.10 release of the MCTS.  New features in this release:

* An updated format for the HTML report that is more complete and usable
* We don't show web-platform-test failures if they're expected failures
* Better handling of wifi connection problems during rebooting while running web-platform-tests
* Many bug fixes

Release url: http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.10.zip

Comment 31

[Karen Ward [:kward]]

Version fxos-MCTS-1.3.0.10.zip posted to the Mobile Partners site.

Comment 32

[Jonathan Griffin (:jgriffin)]

This is the 1.3.0.11 release of the MCTS, for devices running Firefox OS v1.3.  New features in this release:

* An updated format for the HTML report
* Telephony tests disable themselves automatically if the harness can't disable the dialer agent
* Better timeout handling for permissions tests
* Several logging improvements, including full adb commands and outputs
* Python dependencies are now pinned to prevent future package releases from affecting the suite
* Android build properties with '=' are now handled correctly
* Many other bug fixes

Release url: http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.11.zip

Comment 33

[Karen Ward [:kward]]

(In reply to Jonathan Griffin (:jgriffin) from comment #32)
> This is the 1.3.0.11 release of the MCTS, for devices running Firefox OS
> v1.3.  New features in this release:
> 
> * An updated format for the HTML report
> * Telephony tests disable themselves automatically if the harness can't
> disable the dialer agent
> * Better timeout handling for permissions tests
> * Several logging improvements, including full adb commands and outputs
> * Python dependencies are now pinned to prevent future package releases from
> affecting the suite
> * Android build properties with '=' are now handled correctly
> * Many other bug fixes
> 
> Release url: http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.11.zip

KW: Version fxos-MCTS-1.3.0.1a.zip posted to the Mobile Partners site.

Comment 34

[Jonathan Griffin (:jgriffin)]

This is the 1.4.0.1 release of the MCTS, for devices running FirefoxOS v1.4.  Note that devices running FirefoxOS v1.3 should continue to use releases in the 1.3.0.x version line.

This is the initial MCTS release for v1.4; it omits web-platform-tests pending the resolution of bug 1036708; a future release will contain these tests.

Release url:  http://people.mozilla.org/~jgriffin/fxos-MCTS-1.4.0.1.zip

Comment 35

[Karen Ward [:kward]]

(In reply to Jonathan Griffin (:jgriffin) from comment #34)
> This is the 1.4.0.1 release of the MCTS, for devices running FirefoxOS v1.4.
> Note that devices running FirefoxOS v1.3 should continue to use releases in
> the 1.3.0.x version line.
> 
> This is the initial MCTS release for v1.4; it omits web-platform-tests
> pending the resolution of bug 1036708; a future release will contain these
> tests.
> 
> Release url:  http://people.mozilla.org/~jgriffin/fxos-MCTS-1.4.0.1.zip

KW: Version fxos-MCTS-1.4.0.1.zip posted to the Mobile Partners site.

Comment 36

[Jonathan Griffin (:jgriffin)]

This is the 1.3.0.12 release of the MCTS, for devices running Firefox v1.3.  It replaces version 1.3.0.11 of the suite.  Devices running FirefoxOS v1.4 should use a release on the 1.4.0.x version line.

This release fixes the way we identify missing WebAPI methods; see bug 1053174.

Release url:  http://people.mozilla.org/~jgriffin/fxos-MCTS-1.3.0.12.zip

Comment 37

[Jonathan Griffin (:jgriffin)]

This is the 1.4.0.2 release of the MCTS, for devices running FirefoxOS v1.4.  Note that devices running FirefoxOS v1.3 should continue to use releases in the 1.3.0.x version line.

New in this release:

* A fix in the way we identify missing WebAPI methods; see bug 1053174.
* Removed some checks for sender and receiver addresses in SMS tests, since they can be blank under certain conditions.

Release url: http://people.mozilla.org/~jgriffin/fxos-MCTS-1.4.0.2.zip

Comment 38

[Jonathan Griffin (:jgriffin)]

This is the 2.0.1 release of the MCTS, for devices running FirefoxOS v2.0.  Note that devices running earlier versions of FirefoxOS should use a release of the MCTS specifically targeted for that version.

New in this release:

* Initial release for FirefoxOS 2.0.

Release url:  http://people.mozilla.org/~jgriffin/fxos-MCTS-2.0.1.zip

Comment 39

[Karen Ward [:kward]]

KW: fxos-MCTS-2.0.1.zip, fxos-MCTS-1.4.0.2.zip and fxos-MCTS-1.3.0.12.zip all uploaded to Mobile Partner site.

Comment 40

[Karen Ward [:kward]]

Jonathan - Will there be a test suite for 1.3T or does 1.3 test suite cover both releases?

Comment 41

[Jonathan Griffin (:jgriffin)]

(In reply to Karen Ward [:kward] from comment #40)
> Jonathan - Will there be a test suite for 1.3T or does 1.3 test suite cover
> both releases?

1.3 covers both releases.

Comment 42

[Karen Ward [:kward]]

When do you expect the 2.1 version to be ready?  Will it be attached here per our 'usual' process?

Comment 43

[Jonathan Griffin (:jgriffin)]

I expect 2.1 to be ready sometime in early January, and yes it will be added to this bug in the usual way.

Comment 44

[Jonathan Griffin (:jgriffin)]

This is version 2.1.0 of the MCTS, supporting Firefox OS version 2.1:  http://people.mozilla.org/~jgriffin/fxos-MCTS-2.1.0.zip

Note that devices running earlier versions of FirefoxOS should use a release of the MCTS specifically targeted for that version.

New in this release:

* Initial release for FirefoxOS 2.0.

Known issues:

* Bluetooth tests fail with "app = null" - bug 1116878
* Apps webapi tests are disabled because they time out - bug 1110545
* web-platform-tests are disabled - bug 1086171

Comment 45

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #44)
> This is version 2.1.0 of the MCTS, supporting Firefox OS version 2.1: 
> http://people.mozilla.org/~jgriffin/fxos-MCTS-2.1.0.zip
> 
> Note that devices running earlier versions of FirefoxOS should use a release
> of the MCTS specifically targeted for that version.
> 
> New in this release:
> 
> * Initial release for FirefoxOS 2.0.
> 
And of course that should be "Initial release for Firefox 2.1".

Comment 46

[Jonathan Griffin (:jgriffin)]

This is version 2.1.0 of the MCTS, supporting Firefox OS version 2.1:  http://people.mozilla.org/~jgriffin/fxos-MCTS-2.1.1.zip

Note that devices running earlier versions of FirefoxOS should use a release of the MCTS specifically targeted for that version.

New in this release:

* Fix 'apps' webapi tests to run correctly with Firefox OS 2.1
* Fix 'telephony' webapi tests to run correctly with Firefox OS 2.1
* Prevent errors when logs contain non-ascii characters

Known issues:

* Bluetooth tests fail with "app = null" - bug 1116878
* web-platform-tests are disabled - bug 1086171

Comment 47

[oo[oouyang]]

For telephony, I still got 2 failed cases.
- webapi_tests.telephony.test_telephony_incoming_hold_outgoing.TestTelephonyIncomingHoldOutgoing.test_telephony_incoming_hold_outgoing

   AssertionError: Failed to hangup the second call or change the state of first call

- webapi_tests.telephony.test_telephony_incoming_multiple.TestTelephonyIncomingMultiple.test_telephony_incoming_multiple
   AssertionError: Failed to hangup the second call or change the state of first call

(In reply to Jonathan Griffin (:jgriffin) from comment #46)
> This is version 2.1.0 of the MCTS, supporting Firefox OS version 2.1: 
> http://people.mozilla.org/~jgriffin/fxos-MCTS-2.1.1.zip
> 
> Note that devices running earlier versions of FirefoxOS should use a release
> of the MCTS specifically targeted for that version.
> 
> New in this release:
> 
> * Fix 'apps' webapi tests to run correctly with Firefox OS 2.1
> * Fix 'telephony' webapi tests to run correctly with Firefox OS 2.1
> * Prevent errors when logs contain non-ascii characters
> 
> Known issues:
> 
> * Bluetooth tests fail with "app = null" - bug 1116878
> * web-platform-tests are disabled - bug 1086171

(In reply to Jonathan Griffin (:jgriffin) from comment #46)
> This is version 2.1.0 of the MCTS, supporting Firefox OS version 2.1: 
> http://people.mozilla.org/~jgriffin/fxos-MCTS-2.1.1.zip
> 
> Note that devices running earlier versions of FirefoxOS should use a release
> of the MCTS specifically targeted for that version.
> 
> New in this release:
> 
> * Fix 'apps' webapi tests to run correctly with Firefox OS 2.1
> * Fix 'telephony' webapi tests to run correctly with Firefox OS 2.1
> * Prevent errors when logs contain non-ascii characters
> 
> Known issues:
> 
> * Bluetooth tests fail with "app = null" - bug 1116878
> * web-platform-tests are disabled - bug 1086171

Comment 48

[Jonathan Griffin (:jgriffin)]

Moving discussion of the 2.1 telephony issues to bug 1124899.

Comment 49

[Karen Ward [:kward]]

(In reply to Jonathan Griffin (:jgriffin) from comment #46)
> This is version 2.1.0 of the MCTS, supporting Firefox OS version 2.1: 
> http://people.mozilla.org/~jgriffin/fxos-MCTS-2.1.1.zip
> 
> Note that devices running earlier versions of FirefoxOS should use a release
> of the MCTS specifically targeted for that version.
> 
> New in this release:
> 
> * Fix 'apps' webapi tests to run correctly with Firefox OS 2.1
> * Fix 'telephony' webapi tests to run correctly with Firefox OS 2.1
> * Prevent errors when logs contain non-ascii characters
> 
> Known issues:
> 
> * Bluetooth tests fail with "app = null" - bug 1116878
> * web-platform-tests are disabled - bug 1086171

Version 2.1.1 zip file posted to Mobile Partner site (https://mobilepartners.mozilla.org/market/certify-your-ffos-device/)

Comment 50

[Karen Ward [:kward]]

Hi Brian, when do you expect the MCTS for 2.2 to be ready?

Comment 51

[Brian Huang [:brianhuang][:brhuang]]

(In reply to Karen Ward [:kward] from comment #50)
> Hi Brian, when do you expect the MCTS for 2.2 to be ready?

Hi Karen,

The first draft could be ready by the mid of the month (March). The official release will be the end of April.

Brian

Comment 52

[Karen Ward [:kward]]

Anything ready to post on the Mobile Partner site?

Comment 53

[Brian Huang [:brianhuang][:brhuang]]

(In reply to Karen Ward [:kward] from comment #52)
> Anything ready to post on the Mobile Partner site?

We just finish the first draft and doing test. We can release soon. I include Al here to update when we are ready.

Comment 54

[Brian Huang [:brianhuang][:brhuang]]

(In reply to Karen Ward [:kward] from comment #52)
> Anything ready to post on the Mobile Partner site?

We just finish the first draft and doing test. We can release soon. I include Al here to update when we are ready.

Comment 55

[oo[oouyang]]

Attachment: fxos-mcts-2.0.2.zip

Comment 56

[Al Tsai [:atsai]]

Hi Karen,

Would you mind to publish the v2.0 release to the partner website for us? The one on the website has some failures with adb connection and package version. fxos-mcts-2.0.2.zip should with the fixes.

Comment 57

[Al Tsai [:atsai]]

(In reply to Brian Huang [:brianhuang][:brhuang] from comment #53)
> (In reply to Karen Ward [:kward] from comment #52)
> > Anything ready to post on the Mobile Partner site?
> 
> We just finish the first draft and doing test. We can release soon. I
> include Al here to update when we are ready.

For v2.2 release, since FC was postponed to April 29th. I'll prefer to announce MCTS v2.2 on April 29th as well. If there's any partners would like to have a pre-test, we can discuss this offline. Please feel free to let me know if there's any concerns.

Comment 58

[Karen Ward [:kward]]

(In reply to oo[oouyang] from comment #55)
> Created attachment 8592662 [details]
> fxos-mcts-2.0.2.zip

The zip file only has 77 Bytes and is not downloading correctly.  Please try to upload again or give me a url where I can access the file so that I can update the Mobile Partner site.

Comment 59

[oo[oouyang]]

There is file size limit with bugzilla.
So I gave a google doc share link as attachment.
Please check it again.

https://drive.google.com/file/d/0B3lbWXmgZ-UAV0E4U1gtdGpjY3c/view?usp=sharing

Comment 60

[Karen Ward [:kward]]

(In reply to oo[oouyang] from comment #59)
> There is file size limit with bugzilla.
> So I gave a google doc share link as attachment.
> Please check it again.
> 
> https://drive.google.com/file/d/0B3lbWXmgZ-UAV0E4U1gtdGpjY3c/view?usp=sharing

The English version of the site has been changed.  The Certify your Device page now has MCTS 2.0.2 test suite.

Comment 61

[Karen Ward [:kward]]

Wanting or Lufei - Please update the Chinese version of the "Certify Your Device" Page to link to Version 2.0.2 of the MCTS test suite (for Version 2.0).  Thanks!

Comment 62

[Karen Ward [:kward]]

Any update?

Comment 63

[Wanting Xiong]

updates done for Chinese version.

Comment 64

[Karen Ward [:kward]]

We keep this bug open for future versions of the MCTS

Comment 65

[Karen Ward [:kward]]

Updated mobile partner site with https://github.com/mozilla-b2g/fxos-certsuite/archive/2.1.3.zip 
Release Notes for this version:
* Bug 1116418 - Try harder to fix unicode decoding error
* Bug 1124899 - fix not hangup all phone when some cases failed
* Fix diff CLI args
* Bug 1000937 - semiauto web server should not cache static resources
* Bug 1116878 - add resume function for telephony behaviour change.
* bug 1116978 - return bt_adapter's attributes as an object
* add securitysuite into v2.1
* 2.1 ssl stingray
@Wanting, please update the Chinese version of the site too.

Comment 66

[Wanting Xiong]

updates done.

Comment 67

[Karen Ward [:kward]]

Hi Wanting, Please update the Chinese version of the mobile partner site to show the MCTS version for Release 2.2.  The 2.2 test suite is already loaded into the media library on the site. Thanks!

Comment 68

[Wanting Xiong]

Chinese site is updated accordingly.