Closed Bug 1047244 Opened 11 years ago Closed 11 years ago

gaia-build jobs failing with "build.test.js | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet" or "multilocale.test.js | multilocale Integration tests "before all" hook"

Categories

(Firefox OS Graveyard :: Gaia::Build, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::Build
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(b2g-v2.0 fixed, b2g-v2.1 fixed)

Status:
RESOLVED FIXED
Milestone:
2.1 S1 (1aug)
Tracking Flags:
Tracking Status
b2g-v2.0 --- fixed
b2g-v2.1 --- fixed

People

(Reporter: emorley, Assigned: yurenju)

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

github PR: https://github.com/mozilla-b2g/gaia/pull/22400
46 bytes, text/x-github-pull-request
Details | Review
Jobs are failing on multiple trees, and previously passing jobs are failing when retriggered. There were some recent substantial mozharness changes bug 1017759 (which landed and was then backed out, but the backout might not have been complete, given several followups landed with it) - or else perhaps the gaia-build job relies on third party repos that are not connected to sources.xml eg: https://tbpl.mozilla.org/php/getParsedLog.php?id=45029804&tree=Mozilla-Inbound 23:20:39 INFO - Calling ['make', 'update-common'] with output_timeout 300 23:20:40 INFO - # common testing tools 23:20:40 INFO - mkdir -p dev_apps/test-agent/common/vendor/test-agent/ 23:20:40 INFO - rm -f dev_apps/test-agent/common/vendor/test-agent/test-agent.js 23:20:40 INFO - rm -f dev_apps/test-agent/common/vendor/test-agent/test-agent.css 23:20:40 INFO - cp node_modules/test-agent/test-agent.js dev_apps/test-agent/common/vendor/test-agent/ 23:20:40 INFO - cp node_modules/test-agent/test-agent.css dev_apps/test-agent/common/vendor/test-agent/ 23:20:40 INFO - Return code: 0 23:20:40 INFO - Running command: ['make', 'build-test-integration', 'NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org', 'REPORTER=mocha-tbpl-reporter'] in /builds/slave/test/gaia 23:20:40 INFO - Copy/paste: make build-test-integration NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org REPORTER=mocha-tbpl-reporter 23:20:40 INFO - Calling ['make', 'build-test-integration', 'NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org', 'REPORTER=mocha-tbpl-reporter'] with output_timeout 330 23:20:41 INFO - TEST-START | ADB tests make install-test-media 23:20:42 INFO - TEST-PASS | ADB tests make install-test-media 23:20:42 INFO - TEST-END | ADB tests make install-test-media 23:20:42 INFO - TEST-START | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make 23:23:42 INFO - TEST-UNEXPECTED-FAIL | /builds/slave/test/gaia/build/test/integration/build.test.js | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make 23:23:42 INFO - TEST-END | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make -> All trunk trees closed.
Blocks: 1017759
All patches in bug 1017759 are mozharness patches - so no other obvious source code to pull out related to this bug, since all mozharness changes were already reverted.
Ed and I are going to hop on vidyo now to discuss. Anyone welcome to join us in Release Engineering vidyo room...
There's a few different failure modes appearing (though this job does have a number of intermittent failures already, so some may just be these). Some pushes that were previously green, but for which the retriggers are now red: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=b2g_ubuntu64_vm%20mozilla-inbound%20opt%20test%20gaia-build&rev=81ce8a74aba1 https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=b2g_ubuntu64_vm%20mozilla-inbound%20opt%20test%20gaia-build&rev=789665f36a74 https://tbpl.mozilla.org/?tree=B2g-Inbound&jobname=b2g_ubuntu64_vm%20b2g-inbound%20opt%20test%20gaia-build&rev=bdd901635243 Example logs... Green: https://tbpl.mozilla.org/php/getParsedLog.php?id=44995028&tree=B2g-Inbound Red retriggers on same push: https://tbpl.mozilla.org/php/getParsedLog.php?id=45033620&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=45033610&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=45033952&tree=B2g-Inbound Green: https://tbpl.mozilla.org/php/getParsedLog.php?id=44993938&tree=Mozilla-Inbound Red retriggers on same push: https://tbpl.mozilla.org/php/getParsedLog.php?id=45033381&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=45033134&tree=Mozilla-Inbound Green: https://tbpl.mozilla.org/php/getParsedLog.php?id=44993905&tree=Mozilla-Inbound Red retriggers on same push: https://tbpl.mozilla.org/php/getParsedLog.php?id=45025726&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=45027074&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=45027508&tree=Mozilla-Inbound
Given there are several failures modes, mozharness seems less like the cause & this wasn't caused by a gecko commit, that gaia-build tests are relatively new & have several intermittent bugs filed already - pmoore and I decided that we should just hide the job on all trees for now & leave it to the owner of this job to follow up. Reopening trees.
Severity: blocker → critical
Summary: gaia-build jobs failing with "build.test.js | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet" → gaia-build jobs failing with "build.test.js | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet" or "multilocale.test.js | multilocale Integration tests "before all" hook"
Would you mind taking a look at this? The trees are no longer closed over this, but the gaia-build job is hidden on all trees.
Flags: needinfo?(yurenju.mozilla)
It's also failing on Mozilla-B2g32-v2.0, hidden there too.
reproducable on local box, take it.
Assignee: nobody → yurenju.mozilla
Flags: needinfo?(yurenju.mozilla)
http://hg.mozilla.org/gaia-l10n/en-US/archive/tip.tar.gz 404 not found which we use this file for testing.
hg.mozilla.org removed that file, why :(
(In reply to Yuren [:yurenju] from comment #9) > http://hg.mozilla.org/gaia-l10n/en-US/archive/tip.tar.gz 404 not found which > we use this file for testing. I can't see that in the logs - is it present? We should fix the logs and/or parsing in addition to the root cause here. Also, we shouldn't be pulling in the tip of another repo as part of a job :-/
I'm adding those files for testing to gaia repository.
Ed, this pr can be landed with r=testonly, can you land this pr if it pass on try server? I'm away from keyboard for few hours.
Flags: needinfo?(emorley)
Will do, thank you :-)
Was green; merged: https://github.com/mozilla-b2g/gaia/commit/c8063168fd0206bde8609c16fa2448009e965472 However can you also fix the output not appearing in the logs? I notice in the PR we're redirecting even more output to /dev/null. Happy for the followup fix to either be in this bug or in a new one if you prefer.
Flags: needinfo?(emorley) → needinfo?(yurenju.mozilla)
No longer blocks: 1017759
v2.0: https://github.com/mozilla-b2g/gaia/commit/fbe8182432d57f98a1f339b8b99b207fb0a6966c
Status: NEW → RESOLVED
Closed: 11 years ago
status-b2g-v2.0: --- → fixed
status-b2g-v2.1: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 2.1 S1 (1aug)
Leaving gaia-build hidden until the log output fixed (comment 17) and we also stop relying on github (see the intermittent failure bugs filed).
the reason I redirect to null is I got too much standard output and reach maximum buffer on nodejs, and the root cause is I only use settings l10n files in test-locales and the issue can be resovled if we use full l10n packages for testing. I will send another pr to fix it.
Flags: needinfo?(yurenju.mozilla)
Ah great - thank you :-)
Gij is red but it is not related to build system test.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: