972979 - Create initial set of b2g-cert-tests for MWC

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We're going to create a minimal initial set of tests as part of the b2g-cert-tests; this set will be delivered EOD Feb 19 (PST), and consist of a zip file that can be extracted and used to run a series of tests that produce some informative output about the system.

We're tracking the details of this at https://etherpad.mozilla.org/b2g-certification-prototype; code will be checked into https://github.com/mozilla-b2g/fxos-certsuite.

Comment 1

[William Lachance (:wlach)]

There seems to be a permissions problem with the mozilla-b2g/fxos-certsuite repository. Until that's addressed, mdas and I have been committing work here: https://github.com/wlach/fxos-certsuite

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: results.json - Example script output

This shows an example of the output provided by this tool in its current form.

Comment 3

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> Created attachment 8378556 [details]
> results.json - Example script output
> 
> This shows an example of the output provided by this tool in its current
> form.

Btw, this was produced by running the tool against a very old build (gecko 28) in order to verify it could handle a significant amount of divergence from the expected.

Comment 4

[Jonathan Griffin (:jgriffin)]

Attachment: fxos-certsuite-0.1.zip

v0.1 of the fxos-certsuite.

Comment 5

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8378657 [details]
fxos-certsuite-0.1.zip

Patrick, let us know if you need any changes.

Comment 6

[Jonathan Griffin (:jgriffin)]

Tagged in github as https://github.com/mozilla-b2g/fxos-certsuite/tree/0.1

Comment 7

[Patrick Hundal]

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> Comment on attachment 8378657 [details]
> fxos-certsuite-0.1.zip
> 
> Patrick, let us know if you need any changes.

Guys, this looks fantastic.  One (very small) question: do we want the version of this cert test to reflect the "latest" version of OS it was designed for?

Otherwise, I'm ready to post this. ;)

Comment 8

[Jonathan Griffin (:jgriffin)]

(In reply to Patrick Hundal from comment #7)
> (In reply to Jonathan Griffin (:jgriffin) from comment #5)
> > Comment on attachment 8378657 [details]
> > fxos-certsuite-0.1.zip
> > 
> > Patrick, let us know if you need any changes.
> 
> Guys, this looks fantastic.  One (very small) question: do we want the
> version of this cert test to reflect the "latest" version of OS it was
> designed for?
> 
> Otherwise, I'm ready to post this. ;)

That's probably mostly a question for you.  We'll eventually need to branch this and maintain separate versions according to branch.  We don't absolutely need to do this today, but we could.

Should we use 1.3.0.1, where 1.3.0 reflects the B2G version and the extra .1 is the version of the suite?

Comment 9

[Jonathan Griffin (:jgriffin)]

Attachment: fxos-certsuite-1.3.0.1.zip

This is exactly the same as the 0.1 version, but made from a 1.3.0.1 branch and versioned accordingly.  You can take whichever you want, just let us know which you publish so we can handle subsequent revisions correctly.

Comment 10

[Patrick Hundal]

(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> Created attachment 8378680 [details]
> fxos-certsuite-1.3.0.1.zip
> 
> This is exactly the same as the 0.1 version, but made from a 1.3.0.1 branch
> and versioned accordingly.  You can take whichever you want, just let us
> know which you publish so we can handle subsequent revisions correctly.

Awesome - I'll take the 1.3.0.1....  That is perfect.

Comment 11

[Patrick Hundal]

(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> Created attachment 8378680 [details]
> fxos-certsuite-1.3.0.1.zip
> 
> This is exactly the same as the 0.1 version, but made from a 1.3.0.1 branch
> and versioned accordingly.  You can take whichever you want, just let us
> know which you publish so we can handle subsequent revisions correctly.

Hi Jonathan,

One small ask... For consistency with some other areas on the site, can we change the first part of the naming of this file to "fxos-MCTS-1.3..."  MCTS= Mozilla Certification Test Suite... :)

Comment 12

[Jonathan Griffin (:jgriffin)]

Attachment: fxos-MCTS-1.3.0.1.zip

Sure thing, here it is.

Comment 13

[Patrick Hundal]

(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> Created attachment 8379202 [details]
> fxos-MCTS-1.3.0.1.zip
> 
> Sure thing, here it is.

awesome... Just wanted to make sure branch was aligned with naming conventions... Perfect!  Thanks for the quick turnaround. :)

Comment 14

[Jonathan Griffin (:jgriffin)]

Will address any follow-up issues in separate bugs.