Gij tests are orange on Treeherder

RESOLVED FIXED

Status

defect
--
blocker
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: julienw, Assigned: aus)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 7 obsolete attachments)

Locally, I reproduce fairly well but only if I use VERBOSE=1. My full command line is:

VERBOSE=1 DISPLAY=:1 TEST_FILES=apps/calendar/test/marionette/modify_event_view_test.js make test-integration-test

(I predownloaded b2g, precreated the profile, and preexecuted the in-memory X server Xephyr).

And I noticed this message:

  message: 'JavaScriptError: TypeError: window.MARIONETTE_LOG_GRABBER is undefined\nRemote Stack:\n\ninline javascript, line 65\nsrc: "      return window.MARIONETTE_LOG_GRABBER.grabAndClearLogs();"' }
(In reply to Julien Wajsberg [:julienw] from comment #1)
> Current investigations:
> 
> Last working:
> https://treeherder.mozilla.org/#/jobs?repo=gaia-
> master&revision=cddb9f610cbe03d0ca39d81bbdce46a0fca841ab
> 
> B2G comes from:
> Task: FXGKxu6MRoaLMp7MkKCZww
> https://tools.taskcluster.net/task-inspector/#FXGKxu6MRoaLMp7MkKCZww/
> gecko: 22c34579ae0720e7d3dc39a22b9d33f13bc0198b
> http://hg.mozilla.org/mozilla-central/rev/
> 22c34579ae0720e7d3dc39a22b9d33f13bc0198b
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&revision=22c34579ae07
> 
> 
> 
> First not working:
> https://tools.taskcluster.net/task-inspector/#uqg8vcjOSIihPKp76Tp6tw/4

Grrr, bad copy/paste
First not working: https://treeherder.mozilla.org/#/jobs?repo=gaia-master&revision=22fba58ea99fc8212de8d4d44f366773b439dd20

> 
> B2G comes from:
> Task: 0jnopxNmTWOyauKICIhGww
> https://tools.taskcluster.net/task-inspector/#0jnopxNmTWOyauKICIhGww/
> gecko: 8a6045d14d6bd348a3b5bfeb55a9321e680cc93e
> http://hg.mozilla.org/mozilla-central/rev/
> 8a6045d14d6bd348a3b5bfeb55a9321e680cc93e
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&revision=8a6045d14d6b
In nearly all failures we have errors like this:

TEST-START | apps/system/test/marionette/software_home_lockscreen_power_menu_test.js | Software Home Button - Lockscreen Power Menu Covers entire screen
TEST-UNEXPECTED-FAIL | apps/system/test/marionette/software_home_lockscreen_power_menu_test.js | Software Home Button - Lockscreen Power Menu "before each" hook
Error: timeout of 180000ms exceeded
    at null.<anonymous> (/home/tester/git_checkout/node_modules/mocha/lib/runnable.js:139:19)
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
TEST-UNEXPECTED-FAIL | apps/system/test/marionette/software_home_lockscreen_power_menu_test.js | Software Home Button - Lockscreen Power Menu "after each" hook
TypeError: Cannot call method 'send' of undefined
    at Object.send (/home/tester/git_checkout/node_modules/marionette-client/lib/marionette/client.js:463:36)
    at Object.Client._sendCommand (/home/tester/git_checkout/node_modules/marionette-client/lib/marionette/client.js:512:21)
    at Object.closeDriver (/home/tester/git_checkout/node_modules/marionette-client/lib/marionette/client.js:774:14)
    at process._tickCallback (node.js:419:13)
Exit code 2


So first is a timeout in "before each", and then the infamous "Cannot call method 'send' of undefined" error in "after each" (but I guess it's more a consequence of the previous issue).

That's why I tried to increase the timeout when waiting for the boot in attachment 8652333 [details] [review], because I saw the related error "Never saw webapps-registry-ready yes".

But I still see issues. Should we increase it even more ? Maybe in TBPL environment 5 sec is too low ? I admit I don't really know how it's used.
Severity: normal → blocker
So, I've been approaching this issue from various angles and here's what I've ruled out so far.

* random glib assertions are red herrings, they are apparently _not_ critical. :/
* process-content.js not loading is apparently ok.
* backing out the apparent gaia checkin that first was broken doesn't change anything. this isn't the changeset we're looking for if it's coming from a gaia commit.
* reproducing this locally on ubuntu 14 is not easy, reproducing on mac os x is easier as julien pointed out, with or without using Xephyr. Why that is? No idea yet.
Appears to be caused by a gecko change afaict. I have a clean backout with tests that are passing, going to go ahead and push that to b2g-inbound to see if it has any effect there.
Assignee: nobody → aus
Status: NEW → ASSIGNED
Here's my try run that I'm basing my assumptions on -- https://treeherder.mozilla.org/#/jobs?repo=try&revision=da5d408ba9a9

Using custom gaia revision (latest off of master).
Comment on attachment 8652516 [details] [review]
[gaia] nullaus:bug1198172 > mozilla-b2g:master

landed to help with debugging.
Attachment #8652516 - Flags: review+
Attachment #8652820 - Attachment is obsolete: true
OK, the best line of thinking now is to simply standardize the environment between gecko and gaia. I have a pull request to update gaia-taskenv to use tester:0.0.5 as it's base image and customize only the bits we require since we don't use mozharness.

There's still some hiccups to be worked out since there was overlap and differences in how these two environments were set-up and their expectations of where they may be running things.

Hopefully these issues will get worked out and it will enable us to re-open. *fingers crossed*
Things are looking positive, I'm tentatively saying we should be re-opening in the next hour.
Attachment #8652693 - Attachment is obsolete: true
Attachment #8652840 - Attachment is obsolete: true
Attachment #8652858 - Attachment is obsolete: true
Attachment #8652865 - Attachment is obsolete: true
Attachment #8652972 - Attachment is obsolete: true
Attachment #8652995 - Attachment is obsolete: true
Commit gaia-master: https://github.com/mozilla-b2g/gaia/commit/4bf0cf91219175304ef906448f7dfbcf269fc033

Green things a plenty!
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.