980944 - A lot of Gaia UI tests are failing

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Anthony Ricaud (:rik)]

Last good Travis build: https://travis-ci.org/mozilla-b2g/gaia/jobs/20282570
First failing build: https://travis-ci.org/mozilla-b2g/gaia/jobs/20284800

The gaia change looks unrelated but we've changed b2g-desktop versions between those two builds

Last good b2g-desktop: http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/1394193723/b2g-30.0a1.multi.linux-x86_64.tar.bz2
First bad b2g-desktop: http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/1394194076/b2g-30.0a1.multi.linux-x86_64.tar.bz2

Comment 1

[Ed Morley [:emorley]]

b2g-inbound closed per request in #gaia

Comment 2

[Zac C (:zac)]

This actually did fail on TBPL too but showed up orange 

https://tbpl.mozilla.org/?showall=1&rev=0dc1be930880

Comment 3

[Zac C (:zac)]

Here is the Gu HTML Report:
http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-central/sha512/685759b961193c83d980453c2daa8847e55fe1bc7b961ddc4d4c03e7c1673d508c208dd0670ff7fb7c9c442b6624ca66d30e0ceb6c3ff61f4819620c41edecad

I ran the tests locally and there's all sorts of weird stuff going on for example the AppWindowManager returning the wrong app as the displayed app (this is the main thing that is disrupting the tests).

Comment 4

[Ed Morley [:emorley]]

This seems to be the range:
https://tbpl.mozilla.org/?showall=1&tree=Mozilla-Inbound&jobname=b2g_ubuntu64_vm%20mozilla-inbound%20opt%20test%20gaia-ui-test&tochange=46e56e65ecc7&fromchange=d8015367ac9b

-> Caused by https://hg.mozilla.org/integration/mozilla-inbound/rev/f94ee00aa4d6

Comment 5

[Ed Morley [:emorley]]

Should be fixed by the backout of bug 974197.

Comment 6

[Matt Woodrow (:mattwoodrow)]

It looks like the same tests passed on the OSX desktop builds. Does anyone know of any relevant differences between these two configurations?

Comment 7

[Zac C (:zac)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #6)
> It looks like the same tests passed on the OSX desktop builds. Does anyone
> know of any relevant differences between these two configurations?

When I ran this locally I found the AppWindowManager was reporting odd results like the wrong displayed app.

I'll ni? Alive for you and see if he can help.

Comment 8

[(inactive after 6/18) Alive Kuo [:alive]]

Hey, I don't see what's wrong here, isn't this fixed by gecko backout?

Comment 9

[Zac C (:zac)]

Matt needs help debugging it.

Comment 10

[(inactive after 6/18) Alive Kuo [:alive]]

Hm, but I don't know what does "the wrong displayed app" means, any log about this?

Comment 11

[Matt Woodrow (:mattwoodrow)]

I can't get the gaia ui tests to run locally, gaiatest fails with "Timed out waiting for port!".

Comment 12

[Matt Woodrow (:mattwoodrow)]

Ok, managed to get the tests to work using the 'Testing locally with TBPL configuration' section.

But the tests pass locally for me, on ubuntu.

Comment 13

[Matt Woodrow (:mattwoodrow)]

(In reply to Zac C (:zac) from comment #7)
> 
> When I ran this locally I found the AppWindowManager was reporting odd
> results like the wrong displayed app.
> 
> I'll ni? Alive for you and see if he can help.

Can you please walk me through what you had to do to reproduce this locally?

I can't get the tests to fail at all :(

Comment 14

[Zac C (:zac)]

1) Download the exact binary that TBPL build (not your self-built one)
2) Build a Gaia profile from a Git gaia clone using "DESKTOP_SHIMS=1 make"
3) The rest is as per MDN and you've got that working already it seems. I would run the browser tests in tests/functional/browser/ directory first.

Comment 15

[Matt Woodrow (:mattwoodrow)]

Everything in functional/browser gets skipped because I'm testing on desktop?

Running test_kill.py (one of the ones that failed on tpbl) still passes just fine with the downloaded build.

My patches should only have changed the timing of when the 'MozAfterPaint' event is fired, but I can't find any references to that in gaia.

Comment 16

[Zac C (:zac)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #15)
> Everything in functional/browser gets skipped because I'm testing on desktop?
> 
Yes most tests are for running on real devices, we have to skip them for desktop.

> Running test_kill.py (one of the ones that failed on tpbl) still passes just
> fine with the downloaded build.
> 
> My patches should only have changed the timing of when the 'MozAfterPaint'
> event is fired, but I can't find any references to that in gaia.

I won't have the chance to run this again until Thursday, in the meantime perhaps do another Try run with all suites?

Comment 17

[Zac C (:zac)]

Matt, I ran this again (although on a different computer) and it was passing. I don't know what to try apart from another Try run!