Closed Bug 874335 Opened 10 years ago Closed 5 years ago

[meta] B2G MMS: MMS test framework.

Categories

(Firefox OS Graveyard :: RIL, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(tracking-b2g:backlog)

RESOLVED WONTFIX
tracking-b2g backlog

People

(Reporter: ctai, Unassigned)

References

Details

(Whiteboard: [grooming])

Attachments

(2 files)

We need a test framework for MMS. Might start from surveying mochitest first.
Summary: B2G MMS: MMS test framework. → B2G MMS: [META] MMS test framework.
We often use lower cases and put it in the front of title.
Summary: B2G MMS: [META] MMS test framework. → [meta] B2G MMS: MMS test framework.
So, this will be used for developers and QA both, right? Or, this is something like unit testing?
Kind of unit test first....Try to test the DOM API first. Not focus on GAIA Messaging AP.
Whiteboard: RN5/29
Whiteboard: RN5/29
Depends on: 897817
SMS does some level of unit testing (dispatcher, etc.) in Mochitest, but much of the SMS testing is run via Marionette--the API/core JS framework, not the Gaia frameworks.

This is mainly because Marionette can launch multiple emulators and have them message each other.

However, it also keeps us from having tests in the b2g.json mochitest file which are only valid in emulator. There's no clean way to include/exclude them on that basis right now, I don't think.

You might want to check out what was done there prior to doing something different for MMS. 

http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/tests/marionette/
Force emulator mochitest run in chrome process.
MMS mochitets.
Attachment #788848 - Attachment description: bug-874335-2.patch → WIP-bug-874335-2.patch
(In reply to Geo Mealer [:geo] from comment #4)
> This is mainly because Marionette can launch multiple emulators and have
> them message each other.

This is no longer true after bug 759529.  Actually, that two emulators test case was deprecated long ago for it caused problems often.

> However, it also keeps us from having tests in the b2g.json mochitest file
> which are only valid in emulator. There's no clean way to include/exclude
> them on that basis right now, I don't think.

Don't quite follow you.  Do you mean we can't have mochitest test cases for B2G?

The reason Chia-Hung is having experiments on mochitest is |httpd.js| and the access to chrome services.  Maybe you have other ideas?
Depends on: 900438
You can run the test by below command on emulator build.
./test.sh mochitest  --emulator=arm --test-path dom/mobilemessage/tests/mochitest/wrapped_mms_connection.html


(In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #6)
> Created attachment 788848 [details] [diff] [review]
> WIP-bug-874335-2.patch
> 
> MMS mochitets.
In this MMS test framework, we need a httpd to simulate the behaviour of MMSC.
For example, sending MMS:
The client send a http request to the httpd(MMS) with MMS data. The httpd parse the MMS data and judge the result.

Currently, I use mochitest on emulator for MMS test framework. The reason why I use mochitest on emulator is MMS only work on B2G backend. The desktop version of Gecko doesn't have MMS.
Now I have some questions want to raise. The problems I faced are below:

1. Can we run the mochitest on emulator in chrome process? The way I used is in WIP-bug-874335-1.patch. Is there any better way to run mochitest on emulator in chrome process?

2. I want to put those test on try server. But I also use sjs file for the logic of httpd(MMSC). Does anyone know how to overcome the problem for setting Desktop xpcshell on try server?

Every comment is welcome. Thanks.
Flags: needinfo?(rcampbell)
Flags: needinfo?(jmaher)
Flags: needinfo?(gmealer)
I would like to get :ahal to weigh on on this as he knows a lot more about b2g unit tests.

My understanding of mochitests on b2g is that they use marionette (as indicated above) and with marionette you can switch to the chrome context to run code.

As for httpd and .sjs files, if it runs for you locally, it should work on try server.  Make sure you add the .sjs file to the Makefile so it is packaged up correctly.  Also make sure you are not using any hardcoded ports which might be changed upon running in automation.
Flags: needinfo?(jmaher) → needinfo?(ahalberstadt)
(In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #9)
> 1. Can we run the mochitest on emulator in chrome process? The way I used is
> in WIP-bug-874335-1.patch. Is there any better way to run mochitest on
> emulator in chrome process?

We wouldn't be able to accept your WIP patch as is because that would make all of the other mochitests run in the wrong context. We could potentially create a mochitest-chrome job (similar to desktop) that runs with out_of_process=False.

> 2. I want to put those test on try server. But I also use sjs file for the
> logic of httpd(MMSC). Does anyone know how to overcome the problem for
> setting Desktop xpcshell on try server?

This should already be solved. In production there is a static copy of desktop xpcshell that the tests slaves download and use.

Re: Joel's comment about marionette. We only really use marionette to start up the emulator and get tests running. Tests themselves don't have access to marionette, so we'd probably need to create a separate job.
Flags: needinfo?(ahalberstadt)
sorry, I'm not much help here. Passing the buck onto someone who understands mochitest under fxos emulation.
Flags: needinfo?(rcampbell)
Hi, ahal,
Could you please help to point out or provide some information of how to use a static copy of desktop xpcshell on try server?
I have tried to search those information on MDN and bugzilla. But I still can't find any useful information.
In my case, I will try to run my mochitest on B2G emulator and use the desktop xpcshell for the sjs file to simulate a MMSC(Multimedia Messaging Service Center) as a HTTP daemon.

Thanks.


(In reply to Andrew Halberstadt [:ahal] from comment #11)
> (In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #9)
> > 1. Can we run the mochitest on emulator in chrome process? The way I used is
> > in WIP-bug-874335-1.patch. Is there any better way to run mochitest on
> > emulator in chrome process?
> 
> We wouldn't be able to accept your WIP patch as is because that would make
> all of the other mochitests run in the wrong context. We could potentially
> create a mochitest-chrome job (similar to desktop) that runs with
> out_of_process=False.
> 
> > 2. I want to put those test on try server. But I also use sjs file for the
> > logic of httpd(MMSC). Does anyone know how to overcome the problem for
> > setting Desktop xpcshell on try server?
> 
> This should already be solved. In production there is a static copy of
> desktop xpcshell that the tests slaves download and use.
> 
> Re: Joel's comment about marionette. We only really use marionette to start
> up the emulator and get tests running. Tests themselves don't have access to
> marionette, so we'd probably need to create a separate job.
Flags: needinfo?(ahalberstadt)
(In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #13)
> Hi, ahal,
> Could you please help to point out or provide some information of how to use
> a static copy of desktop xpcshell on try server?
> I have tried to search those information on MDN and bugzilla. But I still
> can't find any useful information.
> In my case, I will try to run my mochitest on B2G emulator and use the
> desktop xpcshell for the sjs file to simulate a MMSC(Multimedia Messaging
> Service Center) as a HTTP daemon.
> 
> Thanks.

Sure, the try documentation lives here: https://wiki.mozilla.org/ReleaseEngineering/TryServer

Basically in your hg mozilla-central repo create a patch with the changes you need, then change the commit message to "try: -b do -p emulator -u mochitest -t none" and then run "hg push -f try" (make sure the patch is applied). It'll give you a link you can use to wait for the results to come in. You don't need to worry about the desktop xpcshell, the automation handles all that for you :)
Flags: needinfo?(ahalberstadt)
Thanks, Andrew.
I will try it.


(In reply to Andrew Halberstadt [:ahal] from comment #14)
> (In reply to Chia-hung Tai [:ctai :ctai_mozilla :cht] from comment #13)
> > Hi, ahal,
> > Could you please help to point out or provide some information of how to use
> > a static copy of desktop xpcshell on try server?
> > I have tried to search those information on MDN and bugzilla. But I still
> > can't find any useful information.
> > In my case, I will try to run my mochitest on B2G emulator and use the
> > desktop xpcshell for the sjs file to simulate a MMSC(Multimedia Messaging
> > Service Center) as a HTTP daemon.
> > 
> > Thanks.
> 
> Sure, the try documentation lives here:
> https://wiki.mozilla.org/ReleaseEngineering/TryServer
> 
> Basically in your hg mozilla-central repo create a patch with the changes
> you need, then change the commit message to "try: -b do -p emulator -u
> mochitest -t none" and then run "hg push -f try" (make sure the patch is
> applied). It'll give you a link you can use to wait for the results to come
> in. You don't need to worry about the desktop xpcshell, the automation
> handles all that for you :)
Flags: needinfo?(gmealer)
Component: DOM: Device Interfaces → RIL
Product: Core → Firefox OS
Depends on: 942066
Assignee: ctai → btseng
Put this bug into backlog.
blocking-b2g: --- → backlog
Whiteboard: [grooming]
Blocks: b2g-mms
No longer depends on: b2g-mms
blocking-b2g: backlog → ---
no longer following up this bug.
Assignee: btseng → nobody
Firefox OS is not being worked on
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.