1008414 - Gaia UI and Integration notification tests consistently failing on Gaia-Try

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

This test is done against gaia@15ac34804eb8b3c9b2582d7cf754c57e23182df6 (git).

The same code is used the mozilla-central build and the gaia-try build.

Mozilla-Central: https://tbpl.mozilla.org/?rev=373e31d211f3
Gaia-Try: https://tbpl.mozilla.org/?tree=Gaia-Try&rev=976dc0d512ff

Logs for the relevant failures are here: http://pastebin.mozilla.org/5114041

The names of the failing UI Tests are:

1. test_lockscreen_notification.TestLockScreen.test_lock_screen_notification
2. test_lockscreen_wake_with_notification.TestLockScreenNotification.test_lock_screen_wake_with_notification
3. test_system_notification_bar.TestNotificationBar.test_notification_bar

The names of the failing Integration Tests are:
1. notification_test.js | notification tests fire notification
2. notification_test.js | notification tests system replace notification
3. notification_test.js | notification tests close notification
4. desktop_notification_test.js | Desktop Notifications should be cleared after opening app

The fourth failing integration test also has a screen shot: http://i.imgur.com/0Qhler3.png.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

I've confirmed with releng that the machines are identical between try and production.  I've diffed the logs of a couple other builds and the only differences were how the Gaia sources got onto the machine and a couple commands about how Gaia is built.

Comment 2

[Kevin Grandon :kgrandon]

Weird. What is the difference in commands in how we build gaia?

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

According to code and data in https://gist.github.com/jhford/b6f87fa95ee9c38741f4, the pre-failure differences are:

1. we obtain gaia from the hg mirror instead of git.  I've verified that the contents of the working directory is identical
2. the file used by the settings app doesn't have a git commit because of above.  I'd be shocked if this caused an issue.
3. We invoke gaia_integration.py and marionett.py mozharness scripts with the extra parameters:

-c b2g/gaia_try.py --installer-url https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-32.0a1.en-US.linux-x86_64.tar.bz2 --test-url https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest/en-US/b2g-32.0a1.en-US.linux-x86_64.tests.zip

The gaia_try.py is an extra config file that tells mozharness where to find the gaia.json file used to grab the gaia sources.  The other arguments seem to be to tell mozharness where to download the blobs.

5. We are downloading gecko builds from the /latest/ directory instead of a specific tinderbox build.  That is desired behaviour and /latest/ is a symlink to the latest gecko.  We download the gecko build, gecko tests and gecko symbols.  There's a chance that there is a race here, with the latest build leapfroging the test, but it would be hard to explain that *every* gaia-try run happened during this race.

The actual commands that are invoked by each set of tests are the same between production and try.

(oh and pastebin from comment 0 is now at http://pastebin.mozilla.org/5114965, which is now a forever paste)

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

There just aren't differences in how the code is being called or how the makefiles are being invoked.  The Gaia-Try jobs are the same commands as the Production and normal Gecko Try jobs.  If you go back far enough, there are old changesets that are completely green.

The newest base that I found which produced a green Gu was May 5.  This was https://github.com/crh0716/gaia/commit/cbdbb5f150627d391e92df20ec54e57ad3143f79

This means that there is some code change between May 5 and now that either causes or triggers these failures.

I'm going to trigger a set of jobs to see if I can narrow down the commit where things started breaking.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

So, going through the git log, I decided to filter all the commits from the commit in comment 4 and see if there were any changes related to notifications.  There is exactly one.

$ git log --first-parent ^cbdbb5f150627d391e92df20ec54e57ad3143f79 master --oneline --grep notifications
77e9b2f Merge pull request #18856 from lissyx/bug1003063

So, I decided to submit that changeset to try and it fails integration and ui-tests: 
     https://tbpl.mozilla.org/?tree=Gaia-Try&rev=49dca930d6bd

I thought I'd submit the parent commit to see if this was green:
     https://tbpl.mozilla.org/?tree=Gaia-Try&rev=add9643d3711

Turns out that it's 100% green.  Looking at the contents of the patch, the issue is regarding a change for bug 963234 landing on Gecko.  Given that this change has landed on mozilla-central [1] and is present on b2g-inbound. 

I pushed a commit to try that consists of mozilla-b2g:master with 77e9b2f reverted:
     https://tbpl.mozilla.org/?tree=Gaia-Try&rev=f3daa189c2da

If the Gu and Gi turn green, it's pretty clear to me that the issue here is that the copy of Gecko being used is out of date.



[1] https://bugzilla.mozilla.org/show_bug.cgi?id=963234#c52

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

Yikes!  I just loaded up the directory that stores the Gecko we use for these tests[1].  They are all from May 1!  The required changeset to Gecko that allows 77e9b2f to work landed on May 2, so no wonder things are broken.

The regular trees (b2g-inbound, try, m-c) all work fine because they are triggered against specific builds of Gecko instead of relying on the /latest link.

There are two problems here:

1. The way we determine which Gecko to download is busted.  Either /latest should be fixed or we should stop using it.

2. Gecko hadn't updated in 8 days and alarm bells weren't ringing.

Gaia Developers: nothing to do here anymore, the problem's been found, but thanks for the help you've already provided.



https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-linux64_gecko/latest

Comment 7

[BMO Automation]

Firefox OS is not being worked on