964117 - [TestAgent][Travis] AppWindow hold different element in unit tests on Travis and localhost

Status: RESOLVED DUPLICATE of bug 864178

(Firefox OS Graveyard :: Gaia::TestAgent, defect)

Description

[Greg Weng [:snowmantw][:gweng][:λ]]

As I mentioned in:

https://groups.google.com/forum/#!topic/mozilla.dev.gaia/lISYI1SVjnw

The AppWindow.prototype.element would be different when the test runs on Travis and local environment, and it's hardly to discover because in the test file there's no clue about this issue. 

It's really bad that we seems just avoiding this problem luckily because we never wrote any test related to this strange error(?), until I got stuck and spent lots of time to make sure that the test is correct, and the issue is caused by the environments. Nevertheless, I still don't know why this would happen.

If we can't make our Travis environment as the same with local environment as possible, we may encounter similar problems, and the local tests would be no longer trustable, or even worse, become meaningless. A complete way to eliminate these issues is to tell developers not to run unit test locally by their Firefox(Nightly), but build and deploy a Travis (image) to get the same environment and run the test locally. However I don't know whether this is possible or not.

Comment 1

[Julien Wajsberg [:julienw]]

Hey,

I don't see how this is possible.

The main difference between Travis and your machine (if you use a DEBUG=1 DESKTOP=0 profile) is the speed of the machine, so maybe that means your test (or file) is sensible to race conditions.

I can't tell much more without looking at actual code, so you should share your code (even if this is just a debugging patch) with us :)

Comment 2

[Julien Wajsberg [:julienw]]

Oh, I also saw Etienne's answer, and it's really possible.

Funny fact: I actually tried to sort the files so that it would be executed in the same order, but it seems to not work correctly.

Comment 3

[Greg Weng [:snowmantw][:gweng][:λ]]

You can test this with simple 'console' code. In the 'system/test/unit/app_window_test.js', line 99 or 100, insert:

    console.log('app1.element', app1.element.outerHTML)

It would print out the result I posted on the discussion thread (as below).
I didn't change any other code except the line.

# On Travis: 

<div data-manifest-name="undefined"><iframe 
data-frame-origin="app://www.fake" data-frame-type="window" class="browser" 
id="browser39" data-url="app://www.fake/index.html" 
src="app://www.fake/index.html" 
expecting-system-message="expecting-system-message" 
mozapp="app://wwww.fake/ManifestURL" name="main" mozbrowser="true" 
mozallowfullscreen="true"></iframe><iframe 
data-frame-origin="app://www.fake2" data-frame-type="window" 
class="browser" id="browser40" data-url="app://www.fake2/index.html" 
src="app://www.fake2/index.html" 
expecting-system-message="expecting-system-message" 
mozapp="app://wwww.fake2/ManifestURL" name="main" mozbrowser="true" 
mozallowfullscreen="true"></iframe></div> 

# On my local machine: 

<div data-manifest-name="undefined"><div class=" appWindow " 
id="AppWindow-0" transition-state="closed"><div 
class="screenshot-overlay"></div><div 
class="fade-overlay"></div></div><iframe data-frame-origin="app://www.fake" 
data-frame-type="window" class="browser" id="browser0" 
data-url="app://www.fake/index.html" src="app://www.fake/index.html" 
expecting-system-message="expecting-system-message" 
mozapp="app://wwww.fake/ManifestURL" name="main" mozbrowser="true" 
mozallowfullscreen="true"></iframe><div class=" appWindow " 
id="AppWindow-1" transition-state="closed"><div 
class="screenshot-overlay"></div><div 
class="fade-overlay"></div></div><iframe 
data-frame-origin="app://www.fake2" data-frame-type="window" 
class="browser" id="browser1" data-url="app://www.fake2/index.html" 
src="app://www.fake2/index.html" 
expecting-system-message="expecting-system-message" 
mozapp="app://wwww.fake2/ManifestURL" name="main" mozbrowser="true" 
mozallowfullscreen="true"></iframe></div>

Comment 4

[Kevin Grandon :kgrandon]

It's probably just a previous test not cleaning up state properly.

Just checking because I haven't seen it mentioned, have you tried running *all* of the system tests locally? This is, instead of running the single test, run them all with make test-agent-test APP=system? I would assume that this would show a similar condition to travis.

Heh, funnily enough I just tried and it seems the system app tests don't like to all be run locally =| We should fix this.

Comment 5

[Julien Wajsberg [:julienw]]

(In reply to Kevin Grandon :kgrandon from comment #4)

> Heh, funnily enough I just tried and it seems the system app tests don't
> like to all be run locally =| We should fix this.

Yeah, I've seen this as well.

FTR, bug 899725 is where I tried to sort the test files but failed.

Comment 6

[Julien Wajsberg [:julienw]]

I'd also like to fix bug 864178, should be reasonnably easy, and should fix this properly.

Is it ok if I just dupe to bug 864178?

Comment 7

[Ricky Chien [:rickychien] (inactive)]

(In reply to Greg Weng [:snowmantw][:gweng][:λ] from comment #3)
> You can test this with simple 'console' code. In the
> 'system/test/unit/app_window_test.js', line 99 or 100, insert:
> 
>     console.log('app1.element', app1.element.outerHTML)
> 
> It would print out the result I posted on the discussion thread (as below).
> I didn't change any other code except the line.
> 
> # On Travis: 
> 
> <div data-manifest-name="undefined"><iframe 
> data-frame-origin="app://www.fake" data-frame-type="window" class="browser" 
> id="browser39" data-url="app://www.fake/index.html" 
> src="app://www.fake/index.html" 
> expecting-system-message="expecting-system-message" 
> mozapp="app://wwww.fake/ManifestURL" name="main" mozbrowser="true" 
> mozallowfullscreen="true"></iframe><iframe 
> data-frame-origin="app://www.fake2" data-frame-type="window" 
> class="browser" id="browser40" data-url="app://www.fake2/index.html" 
> src="app://www.fake2/index.html" 
> expecting-system-message="expecting-system-message" 
> mozapp="app://wwww.fake2/ManifestURL" name="main" mozbrowser="true" 
> mozallowfullscreen="true"></iframe></div> 
> 
> # On my local machine: 
> 
> <div data-manifest-name="undefined"><div class=" appWindow " 
> id="AppWindow-0" transition-state="closed"><div 
> class="screenshot-overlay"></div><div 
> class="fade-overlay"></div></div><iframe data-frame-origin="app://www.fake" 
> data-frame-type="window" class="browser" id="browser0" 
> data-url="app://www.fake/index.html" src="app://www.fake/index.html" 
> expecting-system-message="expecting-system-message" 
> mozapp="app://wwww.fake/ManifestURL" name="main" mozbrowser="true" 
> mozallowfullscreen="true"></iframe><div class=" appWindow " 
> id="AppWindow-1" transition-state="closed"><div 
> class="screenshot-overlay"></div><div 
> class="fade-overlay"></div></div><iframe 
> data-frame-origin="app://www.fake2" data-frame-type="window" 
> class="browser" id="browser1" data-url="app://www.fake2/index.html" 
> src="app://www.fake2/index.html" 
> expecting-system-message="expecting-system-message" 
> mozapp="app://wwww.fake2/ManifestURL" name="main" mozbrowser="true" 
> mozallowfullscreen="true"></iframe></div>

Just tried to test this sample code again. Fortunately, both outputs on local and on Travis are same now.

So I think maybe this bug has been fixed by bug 864178. Can we close it if the problem doesn't exist?

Comment 8

[Julien Wajsberg [:julienw]]

let's do this :)