874335 - [meta] B2G MMS: MMS test framework.

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: RIL, defect)

Description

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

We need a test framework for MMS. Might start from surveying mochitest first.

Comment 1

[Gene Lian (I already quit Mozilla)]

We often use lower cases and put it in the front of title.

Comment 2

[Kai-Chih Hu [:khu]]

So, this will be used for developers and QA both, right? Or, this is something like unit testing?

Comment 3

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

Kind of unit test first....Try to test the DOM API first. Not focus on GAIA Messaging AP.

Comment 4

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

SMS does some level of unit testing (dispatcher, etc.) in Mochitest, but much of the SMS testing is run via Marionette--the API/core JS framework, not the Gaia frameworks.

This is mainly because Marionette can launch multiple emulators and have them message each other.

However, it also keeps us from having tests in the b2g.json mochitest file which are only valid in emulator. There's no clean way to include/exclude them on that basis right now, I don't think.

You might want to check out what was done there prior to doing something different for MMS. 

http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/tests/marionette/

Comment 5

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

Attachment: WIP-bug-874335-1.patch

Force emulator mochitest run in chrome process.

Comment 6

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

Attachment: WIP-bug-874335-2.patch

MMS mochitets.

Comment 7

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Geo Mealer [:geo] from comment #4)
> This is mainly because Marionette can launch multiple emulators and have
> them message each other.

This is no longer true after bug 759529.  Actually, that two emulators test case was deprecated long ago for it caused problems often.

> However, it also keeps us from having tests in the b2g.json mochitest file
> which are only valid in emulator. There's no clean way to include/exclude
> them on that basis right now, I don't think.

Don't quite follow you.  Do you mean we can't have mochitest test cases for B2G?

The reason Chia-Hung is having experiments on mochitest is |httpd.js| and the access to chrome services.  Maybe you have other ideas?

Comment 8

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

You can run the test by below command on emulator build.
./test.sh mochitest  --emulator=arm --test-path dom/mobilemessage/tests/mochitest/wrapped_mms_connection.html


(In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #6)
> Created attachment 788848 [details] [diff] [review]
> WIP-bug-874335-2.patch
> 
> MMS mochitets.

Comment 9

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

In this MMS test framework, we need a httpd to simulate the behaviour of MMSC.
For example, sending MMS:
The client send a http request to the httpd(MMS) with MMS data. The httpd parse the MMS data and judge the result.

Currently, I use mochitest on emulator for MMS test framework. The reason why I use mochitest on emulator is MMS only work on B2G backend. The desktop version of Gecko doesn't have MMS.
Now I have some questions want to raise. The problems I faced are below:

1. Can we run the mochitest on emulator in chrome process? The way I used is in WIP-bug-874335-1.patch. Is there any better way to run mochitest on emulator in chrome process?

2. I want to put those test on try server. But I also use sjs file for the logic of httpd(MMSC). Does anyone know how to overcome the problem for setting Desktop xpcshell on try server?

Every comment is welcome. Thanks.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I would like to get :ahal to weigh on on this as he knows a lot more about b2g unit tests.

My understanding of mochitests on b2g is that they use marionette (as indicated above) and with marionette you can switch to the chrome context to run code.

As for httpd and .sjs files, if it runs for you locally, it should work on try server.  Make sure you add the .sjs file to the Makefile so it is packaged up correctly.  Also make sure you are not using any hardcoded ports which might be changed upon running in automation.

Comment 11

[Andrew Halberstadt [:ahal]]

(In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #9)
> 1. Can we run the mochitest on emulator in chrome process? The way I used is
> in WIP-bug-874335-1.patch. Is there any better way to run mochitest on
> emulator in chrome process?

We wouldn't be able to accept your WIP patch as is because that would make all of the other mochitests run in the wrong context. We could potentially create a mochitest-chrome job (similar to desktop) that runs with out_of_process=False.

> 2. I want to put those test on try server. But I also use sjs file for the
> logic of httpd(MMSC). Does anyone know how to overcome the problem for
> setting Desktop xpcshell on try server?

This should already be solved. In production there is a static copy of desktop xpcshell that the tests slaves download and use.

Re: Joel's comment about marionette. We only really use marionette to start up the emulator and get tests running. Tests themselves don't have access to marionette, so we'd probably need to create a separate job.

Comment 12

[Rob Campbell [:rc] (:robcee)]

sorry, I'm not much help here. Passing the buck onto someone who understands mochitest under fxos emulation.

Comment 13

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

Hi, ahal,
Could you please help to point out or provide some information of how to use a static copy of desktop xpcshell on try server?
I have tried to search those information on MDN and bugzilla. But I still can't find any useful information.
In my case, I will try to run my mochitest on B2G emulator and use the desktop xpcshell for the sjs file to simulate a MMSC(Multimedia Messaging Service Center) as a HTTP daemon.

Thanks.


(In reply to Andrew Halberstadt [:ahal] from comment #11)
> (In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #9)
> > 1. Can we run the mochitest on emulator in chrome process? The way I used is
> > in WIP-bug-874335-1.patch. Is there any better way to run mochitest on
> > emulator in chrome process?
> 
> We wouldn't be able to accept your WIP patch as is because that would make
> all of the other mochitests run in the wrong context. We could potentially
> create a mochitest-chrome job (similar to desktop) that runs with
> out_of_process=False.
> 
> > 2. I want to put those test on try server. But I also use sjs file for the
> > logic of httpd(MMSC). Does anyone know how to overcome the problem for
> > setting Desktop xpcshell on try server?
> 
> This should already be solved. In production there is a static copy of
> desktop xpcshell that the tests slaves download and use.
> 
> Re: Joel's comment about marionette. We only really use marionette to start
> up the emulator and get tests running. Tests themselves don't have access to
> marionette, so we'd probably need to create a separate job.

Comment 14

[Andrew Halberstadt [:ahal]]

(In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #13)
> Hi, ahal,
> Could you please help to point out or provide some information of how to use
> a static copy of desktop xpcshell on try server?
> I have tried to search those information on MDN and bugzilla. But I still
> can't find any useful information.
> In my case, I will try to run my mochitest on B2G emulator and use the
> desktop xpcshell for the sjs file to simulate a MMSC(Multimedia Messaging
> Service Center) as a HTTP daemon.
> 
> Thanks.

Sure, the try documentation lives here: https://wiki.mozilla.org/ReleaseEngineering/TryServer

Basically in your hg mozilla-central repo create a patch with the changes you need, then change the commit message to "try: -b do -p emulator -u mochitest -t none" and then run "hg push -f try" (make sure the patch is applied). It'll give you a link you can use to wait for the results to come in. You don't need to worry about the desktop xpcshell, the automation handles all that for you :)

Comment 15

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

Thanks, Andrew.
I will try it.


(In reply to Andrew Halberstadt [:ahal] from comment #14)
> (In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #13)
> > Hi, ahal,
> > Could you please help to point out or provide some information of how to use
> > a static copy of desktop xpcshell on try server?
> > I have tried to search those information on MDN and bugzilla. But I still
> > can't find any useful information.
> > In my case, I will try to run my mochitest on B2G emulator and use the
> > desktop xpcshell for the sjs file to simulate a MMSC(Multimedia Messaging
> > Service Center) as a HTTP daemon.
> > 
> > Thanks.
> 
> Sure, the try documentation lives here:
> https://wiki.mozilla.org/ReleaseEngineering/TryServer
> 
> Basically in your hg mozilla-central repo create a patch with the changes
> you need, then change the commit message to "try: -b do -p emulator -u
> mochitest -t none" and then run "hg push -f try" (make sure the patch is
> applied). It'll give you a link you can use to wait for the results to come
> in. You don't need to worry about the desktop xpcshell, the automation
> handles all that for you :)

Comment 16

[Ken Chang[:kenkai]]

Put this bug into backlog.

Comment 17

[Bevis Tseng[:bevis][:btseng](Exited)]

no longer following up this bug.

Comment 18

[BMO Automation]

Firefox OS is not being worked on