Closed Bug 1013585 Opened 10 years ago Closed 9 years ago

Intermittent gaia-build "Automation Error: mozprocess timed out after 330 seconds running ['make', 'build-test-integration', 'NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org', 'REPORTER=mocha-tbpl-reporter'] | timed out after 330 seconds of no output

Categories

(Firefox OS Graveyard :: Gaia::Build, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::Build
Platform:
x86_64
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
2.2 S5 (6feb)

People

(Reporter: KWierso, Assigned: gduan)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files)

github PR: https://github.com/mozilla-b2g/gaia/pull/20071
46 bytes, text/x-github-pull-request
Details | Review
extend timeout for gaia_build_integration test
906 bytes, patch
jgriffin
: review+
Details | Diff | Splinter Review 
https://tbpl.mozilla.org/php/getParsedLog.php?id=40034987&tree=Mozilla-Inbound
b2g_ubuntu64_vm mozilla-inbound opt test gaia-build on 2014-05-20 12:55:34 PDT for push 37e1dd2155b6

slave: tst-linux64-spot-368


13:05:18     INFO -  touch -c node_modules
13:05:18     INFO -  TEST-START | ADB tests make install-test-media
13:05:19     INFO -  TEST-PASS | ADB tests make install-test-media
13:05:19     INFO -  TEST-END | ADB tests make install-test-media
13:05:19     INFO -  TEST-START | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make
13:06:56     INFO -  TEST-PASS | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make
13:06:56     INFO -  TEST-END | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make
13:06:57     INFO -  TEST-START | Build GAIA from differece app list GAIA_DEVICE_TYPE=phone make
13:07:48     INFO -  TEST-PASS | Build GAIA from differece app list GAIA_DEVICE_TYPE=phone make
13:07:48     INFO -  TEST-END | Build GAIA from differece app list GAIA_DEVICE_TYPE=phone make
13:07:49     INFO -  TEST-START | Node modules tests make node_modules from git mirror
13:08:10     INFO -  TEST-PASS | Node modules tests make node_modules from git mirror
13:08:10     INFO -  TEST-END | Node modules tests make node_modules from git mirror
13:08:10     INFO -  TEST-START | Node modules tests make node_modules from github
13:13:40     INFO - Automation Error: mozprocess timed out after 330 seconds running ['make', 'build-test-integration', 'NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org', 'REPORTER=mocha-tbpl-reporter']
13:13:40    ERROR - timed out after 330 seconds of no output
13:13:40    ERROR - Return code: -9
13:13:40     INFO - TinderboxPrint: gaia-build-integration-tests: <em class="testfail">T-FAIL</em>
13:13:40    ERROR - Tests exited with return code -9: harness failures
13:13:40    ERROR - # TBPL FAILURE #
13:13:40     INFO - Running post-action listener: _resource_record_post_action
13:13:40     INFO - Running post-run listener: _resource_record_post_run
Blocks: 989126
Flags: needinfo?(jgriffin)
Summary: Intermittent Automation Error: mozprocess timed out after 330 seconds running ['make', 'build-test-integration', 'NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org', 'REPORTER=mocha-tbpl-reporter'] | timed out after 330 seconds of no output → Intermittent gaia-build "Automation Error: mozprocess timed out after 330 seconds running ['make', 'build-test-integration', 'NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org', 'REPORTER=mocha-tbpl-reporter'] | timed out after 330 seconds of no output
Not sure what's happening here, so passing the buck.

Note that the problem
Flags: needinfo?(mhenretty)
Flags: needinfo?(kgrandon)
Flags: needinfo?(jgriffin)
Flags: needinfo?(gaye)
The tests are dying here:

09:24:30     INFO -  TEST-START | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make

The integration test framework isn't handling timeouts.
I think this may have been fixed by https://github.com/mozilla-b2g/gaia/commit/514cc1e245e38efe2ba3111535b91efa89b79ae9; Gb is green on b2g-inbound with that.  Let's see what it looks like in another hour or two.
The robot notifications have seemed to stop? Loop me in if it's still broken and there's anything I can do to help.
Flags: needinfo?(kgrandon)
Can we close this? Looks like it's not happening anymore with the backout mentioned in comment #15.
Flags: needinfo?(mhenretty)
(In reply to Jonathan Griffin (:jgriffin) from comment #15)
> The tests are dying here:
> 
> 09:24:30     INFO -  TEST-START | Build GAIA from differece app list
> GAIA_DEVICE_TYPE=tablet make
> 
> The integration test framework isn't handling timeouts.

The "handle timeouts correctly" part is still outstanding, afaict.
Assignee: nobody → yurenju.mozilla
(In reply to Yuren [:yurenju] from comment #21)
> add timeout back.
> 
> travis:
> https://travis-ci.org/mozilla-b2g/gaia/builds/26842549
> 
> try server:
> https://tbpl.mozilla.org/?tree=Gaia-Try&rev=c2269867f7a1

For anyone else curious about the origin of "--timeout 0", I tracked it back to:
https://github.com/mozilla-b2g/gaia/commit/4cc6b3ca16c31d7dcdcdf074e4876841bba2c62a#diff-b67911656ef5d18c4ae36cb6741b7965R385

...looks like it's always been 0.
yes, because we take too long time for running build integration test and it works fine on travis-ci but we should have a reasonable timeout.
the reason for 3min (180000ms) is the longest time of build integration test case is about 1.5min so set timeout to 3min is safe for current test case and also can avoid this intermittent error after 330 seconds.
Ed, looks doens't happen again, should we close it?
Flags: needinfo?(gaye) → needinfo?(emorley)
Yup, lgtm - thank you :-0
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Flags: needinfo?(emorley)
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Flags: needinfo?(gduan)
Resolution: FIXED → ---
Looks like I need to either raise the build performance or extend the timeout here
https://github.com/escapewindow/mozharness/blob/master/scripts/gaia_build_integration.py#L49
Hi  Jonathan,
Could I have your review on this patch?
The patch would extend the timeout when execute gaia_build_integration test. I hope the modification is correct.

Currently our apps are growing bigger, it would take longer time when build gaia and exceed the timeout and fail the test.
We'll try to raise the build performance in next refactoring plan.
Flags: needinfo?(gduan)
Attachment #8558892 - Flags: review?(jgriffin)
Comment on attachment 8558892 [details] [diff] [review]
extend timeout for gaia_build_integration test

Review of attachment 8558892 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!
Attachment #8558892 - Flags: review?(jgriffin) → review+
Assignee: yurenju → gduan
Status: REOPENED → ASSIGNED
Whiteboard: checkin-needed
Hi George - I believe checkin-needed might only be a keyword. Not sure if it's going to get landed as a whiteboard tag..
Our queries look for both :)
https://hg.mozilla.org/build/mozharness/rev/494baf51b077
Whiteboard: checkin-needed
Target Milestone: --- → 2.2 S5 (6feb)
Status: ASSIGNED → RESOLVED
Closed: 10 years ago → 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: