1126945 - Frequent B2G device image Automation Error: mozprocess timed out after 3300 seconds running ['./config.sh', '-q', 'flame-kk', '/builds/slave/b2g_b2g-in_flm-kk_eng_dep-0000/build/tmp_manifest/flame-kk.xml']

Reporter

Description

•

10 years ago

06:41:08 INFO - Running command: ['./config.sh', '-q', 'flame-kk', '/builds/slave/b2g_b2g-in_flm-kk_eng_dep-0000/build/tmp_manifest/flame-kk.xml'] in /builds/slave/b2g_b2g-in_flm-kk_eng_dep-0000/build 06:41:08 INFO - Copy/paste: ./config.sh -q flame-kk /builds/slave/b2g_b2g-in_flm-kk_eng_dep-0000/build/tmp_manifest/flame-kk.xml 06:41:08 INFO - Calling ['./config.sh', '-q', 'flame-kk', '/builds/slave/b2g_b2g-in_flm-kk_eng_dep-0000/build/tmp_manifest/flame-kk.xml'] with output_timeout 3300 06:41:08 INFO - Initialized empty Git repository in /builds/slave/b2g_b2g-in_flm-kk_eng_dep-0000/build/tmp_manifest_repo/.git/ 06:41:08 INFO - [master (root-commit) ccd1878] manifest 06:41:08 INFO - 1 file changed, 153 insertions(+) 06:41:08 INFO - create mode 100644 flame-kk.xml 06:41:08 INFO - Get tmp_manifest_repo 06:41:08 INFO - From tmp_manifest_repo 06:41:08 INFO - * [new branch] master -> origin/master 06:41:08 INFO - repo has been initialized in /builds/slave/b2g_b2g-in_flm-kk_eng_dep-0000/build 07:36:09 INFO - Automation Error: mozprocess timed out after 3300 seconds running ['./config.sh', '-q', 'flame-kk', '/builds/slave/b2g_b2g-in_flm-kk_eng_dep-0000/build/tmp_manifest/flame-kk.xml'] 07:36:09 ERROR - timed out after 3300 seconds of no output 07:36:09 ERROR - Return code: -9

Comment hidden (Legacy TBPL/Treeherder Robot)

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=11000105 repository: mozilla-inbound start_time: 2015-06-20T23:02:07 who: rvandermeulen[at]mozilla[dot]com machine: bld-linux64-spot-034 buildname: b2g_mozilla-inbound_flame-kk_eng_dep revision: ed1115ca4083 "comments": "Back out changesets ed293fc9596c and f18cb4c41578 (bug 1174700) for fatal assertions in all Windows debug reftest runs.\n\nCLOSED TREE\n\nAssertion failure: origKidNormalPosition.B(wm) == 0, at c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/src/layout/tables/nsTableRowFrame.cpp:861\nTEST-UNEXPECTED-FAIL | file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/writing-mode/ua-style-sheet-size-1.html | application terminated with "comments": "Back out changeset cd741d3ae78a (bug 1168737) for mochitest failures on Mac OS X 10.10.\n\nCLOSED TREE\n\nFailures were:\n2358 INFO TEST-UNEXPECTED-FAIL | toolkit/webapps/tests/test_webapp_runtime_executable_update.xul | Test timed out. - expected PASS\n2359 INFO TEST-UNEXPECTED-FAIL | toolkit/webapps/tests/test_webapp_runtime_executable_update.xul | Webapp runtime executable has been replaced - got 256, expected 42\n2362 INFO TEST-UNEXPECTED-ERRO Return code: 1 Automation Error: mozprocess timed out after 3300 seconds running ['./config.sh', '-q', 'flame-kk', '/builds/slave/b2g_m-in_flm-kk_eng_dep-000000/build/tmp_manifest/flame-kk.xml'] timed out after 3300 seconds of no output Return code: -9 Automation Error: mozprocess timed out after 3300 seconds running ['./config.sh', '-q', 'flame-kk', '/builds/slave/b2g_m-in_flm-kk_eng_dep-000000/build/tmp_manifest/flame-kk.xml'] timed out after 3300 seconds of no output Return code: -9 Automation Error: mozprocess timed out after 3300 seconds running ['./config.sh', '-q', 'flame-kk', '/builds/slave/b2g_m-in_flm-kk_eng_dep-000000/build/tmp_manifest/flame-kk.xml'] timed out after 3300 seconds of no output Return code: -9 Automation Error: mozprocess timed out after 3300 seconds running ['./config.sh', '-q', 'flame-kk', '/builds/slave/b2g_m-in_flm-kk_eng_dep-000000/build/tmp_manifest/flame-kk.xml'] timed out after 3300 seconds of no output Return code: -9 command timed out: 14400 seconds elapsed running ['scripts/scripts/b2g_build.py', '--target', 'flame-kk', '--config', 'b2g/releng-otoro-eng.py', '--gaia-languages-file', 'locales/languages_all.json', '--gecko-languages-file', 'gecko/b2g/locales/all-locales', '--repotool-revision', 'v1.12.16'], attempting to kill

Comment hidden (Legacy TBPL/Treeherder Robot)

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=10997474 repository: mozilla-inbound start_time: 2015-06-20T16:30:10 who: rvandermeulen[at]mozilla[dot]com machine: bld-lion-r5-073 buildname: b2g_mozilla-inbound_macosx64_gecko-debug build revision: 8a25cd69c4ae CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 Automation Error: Can't checkout https://git.mozilla.org/releases/gaia.git! Caught exception: Unable to checkout Caught exception: Unable to checkout Caught exception: Unable to checkout Caught exception: Unable to checkout Caught exception: Unable to checkout Running post_fatal callback... setting return code to 2 because fatal was called Exiting -1 # TBPL FAILURE # # TBPL FAILURE #

Comment hidden (Legacy TBPL/Treeherder Robot)

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=10990917 repository: mozilla-inbound start_time: 2015-06-20T10:30:11 who: rvandermeulen[at]mozilla[dot]com machine: bld-lion-r5-073 buildname: b2g_mozilla-inbound_macosx64_gecko-debug build revision: 23684898a457 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 subprocess.CalledProcessError: Command '['git', 'checkout', '-q', '-f', 'd8bac9577a537b5e6bafc3f7ed088af5ea60c99e^0']' returned non-zero exit status 128 Return code: 1 Automation Error: Can't checkout https://git.mozilla.org/releases/gaia.git! Caught exception: Unable to checkout Caught exception: Unable to checkout Caught exception: Unable to checkout Caught exception: Unable to checkout Caught exception: Unable to checkout Running post_fatal callback... setting return code to 2 because fatal was called Exiting -1 # TBPL FAILURE # # TBPL FAILURE #

Comment hidden (Legacy TBPL/Treeherder Robot)

Selena Deckelmann :selenamarie :selena

Assignee

Updated

•

10 years ago

Blocks: gitperf

Comment hidden (Legacy TBPL/Treeherder Robot)

Phil Ringnalda (:philor)

Updated

•

10 years ago

Depends on: 1177322

Comment hidden (Legacy TBPL/Treeherder Robot)

Selena Deckelmann :selenamarie :selena

Assignee

Updated

•

10 years ago

Depends on: 1178927

Comment hidden (Legacy TBPL/Treeherder Robot)

Selena Deckelmann :selenamarie :selena

Assignee

Comment 1291

•

10 years ago

Repo launches many git actions in parallel so the final 'git' command in the log was different each time a job hung and ultimately failed. I'd heard the previous week that a corrupt git repo existed, but didn't have evidence of which one from the logs, so on a hunch :garndt and I started looking into exactly which sync was failing. To diagnose, we had to find and login to several systems that were hung (a trivial but slightly time consuming process), look at what processes were running and note that we had the same repo stuck. Then we needed help from :bkero to look into the repo that was problematic on git.mo. He found the corruption and then after some discussion decided to purge and refresh the repo. This is done with a tool called 'vcssync'. After that, :garndt killed our in-process device image and emulator builders. Then we watched some builds. Things seemed to be ok, although unrelated bustage prevented us from confirming things were ok for about an hour. We suspect this all comes back to an issue with git.mo (bug 1176192) that caused a ton of problems a couple weeks ago. :bkero kept a copy of the busted repo for analysis later.

Selena Deckelmann :selenamarie :selena

Assignee

Updated

•

10 years ago

Assignee: nobody → sdeckelmann

Status: NEW → RESOLVED

Closed: 10 years ago

Resolution: --- → FIXED

Nobody; OK to take it and work on it

Updated

•

7 years ago

Component: General Automation → General