864246 - [Test Agent] Can not run unit tests on Device.

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia, defect)

Description

[Pin Zhang [:pzhang] (inactive)]

I tried to run unit tests for FM Radio on the Deivce, it used to work before, and it failed now, the test-agent server can't establish connection with the test-agent-server, I do think it's currently broken for all the unit test.

Here is what I tried:
- Setup web server for test-agent.gaiamobile.org on PC
- Map test-agent.gaiamobile.org to my PC in the Unagi's /etc/hosts
- Connect Unagi to LAN
- Run `make test-agent-server` in the terminal
- Start the TestAgent app on Unagi
- Run `make test-agent-test APP=fm`

Nothing happened, I checked the log, and it told me:

    E/GeckoConsole(  708): [JavaScript Error: "Firefox can't establish a connection to the server at ws://test-agent.gaiamobile.org:8789/." {file: "app://test-agent.gaiamobile.org/common/vendor/test-agent/test-agent.js" line: 1218}]

I tried to open `test-agent.gaiamobile.org` in browser on Unagi, it also told me no connection.

Then I tried install production Gaia `PRODUCTION=1 make reset-gaia`, and opened the test-agent in the browser on Unagi, yes, the page is displayed, however,  navigator.mozFMRadio is undefined.

I thought the reason might be no permissions for test-agent on Gaia with PRODUCTION=1, then I pulled the /data/local/permissions.sqlite and added fmradio permission to test-agent.gaiamobile.org, and pushed the permissions.sqlite back, and restart b2g on Unagi, retried it on the browser, still no navigator.mozFMRadio is defined.

Comment 1

[Marco Chen [:mchen]]

(In reply to Pin Zhang [:pzhang] from comment #0)
> I thought the reason might be no permissions for test-agent on Gaia with
> PRODUCTION=1, then I pulled the /data/local/permissions.sqlite and added
> fmradio permission to test-agent.gaiamobile.org, and pushed the
> permissions.sqlite back, and restart b2g on Unagi, retried it on the
> browser, still no navigator.mozFMRadio is defined.

Hi Pin,

FMRadio can be used by app with privileged type only, please see link as below. 
http://mxr.mozilla.org/mozilla-central/source/dom/apps/src/PermissionsTable.jsm#144

So maybe except to add the permission of fmradio, you also need to modify test-agent as a privileged app.

Comment 2

[Pin Zhang [:pzhang] (inactive)]

(In reply to Marco Chen [:mchen] from comment #1)
> (In reply to Pin Zhang [:pzhang] from comment #0)
> > I thought the reason might be no permissions for test-agent on Gaia with
> > PRODUCTION=1, then I pulled the /data/local/permissions.sqlite and added
> > fmradio permission to test-agent.gaiamobile.org, and pushed the
> > permissions.sqlite back, and restart b2g on Unagi, retried it on the
> > browser, still no navigator.mozFMRadio is defined.
> 
> Hi Pin,
> 
> FMRadio can be used by app with privileged type only, please see link as
> below. 
> http://mxr.mozilla.org/mozilla-central/source/dom/apps/src/PermissionsTable.
> jsm#144
> 
> So maybe except to add the permission of fmradio, you also need to modify
> test-agent as a privileged app.

OK, I will have a try, but I don't think it will fix this:
  E/GeckoConsole(  708): [JavaScript Error: "Firefox can't establish a connection to the server at ws://test-agent.gaiamobile.org:8789/." {file: "app://test-agent.gaiamobile.org/common/vendor/test-agent/test-agent.js" line: 1218}]

if no connection to test-agent-server could be established, no tests will be ran.

Comment 3

[Pin Zhang [:pzhang] (inactive)]

(In reply to Marco Chen [:mchen] from comment #1)
> (In reply to Pin Zhang [:pzhang] from comment #0)
> > I thought the reason might be no permissions for test-agent on Gaia with
> > PRODUCTION=1, then I pulled the /data/local/permissions.sqlite and added
> > fmradio permission to test-agent.gaiamobile.org, and pushed the
> > permissions.sqlite back, and restart b2g on Unagi, retried it on the
> > browser, still no navigator.mozFMRadio is defined.
> 
> Hi Pin,
> 
> FMRadio can be used by app with privileged type only, please see link as
> below. 
> http://mxr.mozilla.org/mozilla-central/source/dom/apps/src/PermissionsTable.
> jsm#144
> 
> So maybe except to add the permission of fmradio, you also need to modify
> test-agent as a privileged app.

I checked that I have added `fmradio` permission to test-agent/manifest.webapp, is this the right way to promote the test-agent as a privileged app?

Comment 4

[Marco Chen [:mchen]]

Hi Pin,

You need to add {"type": "certified"} into manifest or test-agent can't access FM even permission is added.


Please refer to https://github.com/mozilla-b2g/gaia/blob/master/apps/fm/manifest.webapp#L4

Thanks Randy's help.

Comment 5

[Pin Zhang [:pzhang] (inactive)]

(In reply to Marco Chen [:mchen] from comment #4)
> Hi Pin,
> 
> You need to add {"type": "certified"} into manifest or test-agent can't
> access FM even permission is added.
> 
> 
> Please refer to
> https://github.com/mozilla-b2g/gaia/blob/master/apps/fm/manifest.webapp#L4
> 
> Thanks Randy's help.

OK, thanks.

I found that there are some marionette tests for Telephony now, here is the definition in the test/marionette/manifest.ini:
  [default]
  b2g = true
  browser = false
  qemu = true
it seems that testing on Device is supported now in Tinder box, and I think I should migrate the test, i.e. api_test.js, from Gaia to Gecko.

Comment 7

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Let's CC James Lal...

Comment 8

[James Lal [:lightsofapollo]]

We likely remove the tests from the package so there is no way to run these on device right now. I would say that integration tests are better candidate for this kind of thing.. unit tests typically don't touch device specific api's.

Comment 9

[Leo]

I tried install production Gaia `PRODUCTION=1 make reset-gaia`, and opened the test-agent in the browser on device, yes, the page is displayed,and i can see the list of testcases.

But in that i am able to execute by clicking only below two test cases.
"test-agent/test/unit/mocha_task_test.js",
"test-agent/test/unit/synthetic_gestures_test.js"

and remaining all testcases are giving  error 

E/GeckoConsole(  708): [JavaScript Error: "Firefox can't establish a connection to the server at ws://test-agent.gaiamobile.org:8789/." {file: "app://test-agent.gaiamobile.org/common/vendor/test-agent/test-agent.js" line: 1218}]

Comment 10

[Julien Wajsberg [:julienw]]

We should not need to add permissions to the test-agent. Rather you should simulate the API in your unit tests, so that you can really control all cases. If you would use the real API you would not be able to test all cases.

So please don't add any permissions to the test-agent, this is useless.

Comment 11

[BMO Automation]

Firefox OS is not being worked on