1006962 - no error message when test agent runs with a non-existed app.

Status: RESOLVED INVALID

(Firefox OS Graveyard :: Gaia::TestAgent, defect)

Description

[John Hu [:johnhu][:johu][:醬糊小弟]]

+++ This bug was initially created as a clone of Bug #1006357 +++

Stingray is the TV project which utilize GAIA_DEVICE_TYPE=tv as its build configuration. It contains an app called homescreen-stingray which is a homescreen support widgets at dev_apps.

During the development of homescreen-stingray, we found an app only existed in tv build instead of phone build. The test agent will get stuck at the end of unit test.

Test Agent got stuck with the following cases:
1. if we run: make test-agent-test APP=non-existed-app
   output:
Running tests for non-existed-app
./node_modules/test-agent/bin/js-test-agent test  --server ws://localhost:8789 --reporter spec
  <got stuck here>

2. if we run homescreen-stingray's tests with phone's build: make test-agent-test APP=homescreen-stingray
   output:
Running tests for homescreen-stingray
./node_modules/test-agent/bin/js-test-agent test  --server ws://localhost:8789 --reporter spec dev_apps/homescreen-stingray/test/unit/layout_editor_test.js
  <got stuck here>

3. if we run homescreen-stingray's tests with tv's build, make DEBUG=1 GAIA_DEVICE_TYPE=tv: make test-agent-test APP=homescreen-stingray
   everything works.

Expected: Test agent should reply error message with the case 1 and 2.

Actually: Test agent got stuck at case 1 and 2.

Comment 1

[Ricky Chien [:rickychien] (inactive)]

If we run a non-existed app in previous version of test agent, test agent will skip the non-existed app and start to run all apps. But now our new version will get stuck for no reason, even output nothing on console.

So I'm going to add error message and raise error to stop our tests.

Comment 2

[Ricky Chien [:rickychien] (inactive)]

Attachment: Gaia PR

Comment 3

[Ricky Chien [:rickychien] (inactive)]

Attachment: Js-test-agent PR

Comment 4

[Ricky Chien [:rickychien] (inactive)]

Attachment: Gaia-node-modules PR

Comment 5

[Julien Wajsberg [:julienw]]

Ricky, can you help me reproducing the original bug? Even without your patch (which looks fine!), the test-agent doesn't seem to wait for me...

I tried rebuilding my profile, but the same: when I run "APP=plop make test-agent-test", all tests are run (like it used to). I'm not saying it is what it should do, but at least it doesn't stop.

Comment 6

[Ricky Chien [:rickychien] (inactive)]

Rebase Gaia to current master and change _test_ to test in dev_apps/homescreen-stingray/ for enabling homescreen-stingray test.

After then, you can try "make test-agent-test APP=homescreen-stingray" to see error message.

Sorry the Comment 1 was wrong. Test agent will run all tests while giving a non-exist-app in current version. However, it will get stuck if we give a exist-app but haven't built into profile.

Comment 7

[Julien Wajsberg [:julienw]]

Ok, I see the issue now, but how do you explain that "APP=test-agent make test-agent-test" works fine?

Comment 8

[Julien Wajsberg [:julienw]]

Maybe I'm doing something wrong, but with your patch:

* "make test-agent-test APP=homescreen-stingray" shows the error message
* but then "make test-agent-test APP=sms" still doesn't work

Comment 9

[Ricky Chien [:rickychien] (inactive)]

Oh, thanks for reporting this bug!

PR has been updated. Test agent will get stuck if we aren't giving "this._running = false" in [1] while iframe access error occur.

[1] https://github.com/RickyChien/gaia/blob/bug-1006962/dev_apps/test-agent/common/vendor/test-agent/test-agent.js#L2736

Comment 10

[Julien Wajsberg [:julienw]]

So, now, here is what I have:

* "make test-agent-test APP=homescreen-stingray" shows the error message:
  Cannot access homescreen-stingray.gaiamobile.org

* then "make test-agent-test APP=sms" shows the exact same error message:
  Cannot access homescreen-stingray.gaiamobile.org

* then "make test-agent-test APP=sms" hangs.

Comment 11

[Julien Wajsberg [:julienw]]

(please request review only on the js-test-agent PR :) )

Comment 12

[Ricky Chien [:rickychien] (inactive)]

Oops! I tried your steps mentioned on comment 10 but it works fine for me.

While trying step 2 in "make test-agent-test APP=sms", I can run sms tests successfully.

Could you please try it again and make sure all of these setups are done.

Comment 13

[Julien Wajsberg [:julienw]]

I'll try again

Comment 14

[Julien Wajsberg [:julienw]]

I'm sorry Ricky, I get the exact same issue.

here is what I do to apply the patch (like I always do):

in js-test-agent:
* apply the patch from https://github.com/mozilla-b2g/js-test-agent/pull/73 to master
* run "make package"
* run "npm link"

Then in gaia:
* run "make clean && rm -rf node_modules"
* run "FIREFOX=~/firefox-nightly/firefox bin/gaia-test"
-> this will download the latest node_modules and generate the test profile
* run "npm link test-agent"
* run "make update-common"
=> I see that test-agent.js has been replaced
* run "FIREFOX=~/firefox-nightly/firefox bin/gaia-test" again

Then the steps from comment 10.

Is it possible that you didn't update properly the code in js-test-agent?

Comment 15

[Ricky Chien [:rickychien] (inactive)]

I alaways apply the patch on Gaia and because my Gaia patch used new version test-agent which was published to npm.
Therefore, we could remove node_modules and run "npm install" again after applying Gaia patch.


**Important**
On the other hand, I think this bug would be useless because of Bug 1006357 has landed.

Let me explain more details:

When running "make test-agent-test APP=xxx" before Bug 1006357 has landed,
  1. if xxx doesn't exist in apps/ and dev_apps/, test-agent will skip it and run all tests.
  2. if xxx do exist in apps/ or dev_apps/, test-agent will try to run test for xxx app. But test-agent will get stuck since we couldn't access the domain of homescreen-stingray.

The homescreen-stingray belongs to situation 2, test-agent try to run test for homescreen-stingray. However, httpd server cannot resolve domain for homescreen-stingray since we didn't build it. Therefore, Bug 1006357 had fixed this issue and allow httpd resolve all domains without building all apps. Test-agent can access any domain properly if it exist in apps/ or dev_apps/. 

Finally, the situation 2 is absent after Bug 1006357 was landed, we could run "test-agent-test APP=homescreen-stingray" properly now. (You could update Gaia master and try again.)

Comment 16

[Julien Wajsberg [:julienw]]

I'm fine with closing it if bug 1006357 made it work fine :)


I still don't understand why my way of testing the patch didn't work here, but it's fine...

Comment 17

[Ricky Chien [:rickychien] (inactive)]

OK! Close this bug since bug 1006357 made it work now.