1567838 - Intermittent [taskcluster:error] Task timeout after 1800 seconds. Force killing container. due to hang after Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)

Status: RESOLVED INCOMPLETE

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=257164737&repo=mozilla-beta
Full log: https://queue.taskcluster.net/v1/task/NTcr1M7MT-C5F_5TyZ9fag/runs/0/artifacts/public/logs/live_backing.log

---

[vcs 2019-07-18T14:26:55.622Z] 279343 files updated, 0 files merged, 0 files removed, 0 files unresolved
[vcs 2019-07-18T14:26:56.324Z] updated to 067e54510790223f29c1c10d8f488e10d679ae6c
[vcs 2019-07-18T14:26:56.537Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["c3.xlarge"], "lowerIsBetter": true, "name": "clone", "serverUrl": "us-west-1.hgmointernal.net", "shouldAlert": false, "subtests": [], "value": 138.5182399749756}, {"extraOptions": ["c3.xlarge"], "lowerIsBetter": true, "name": "update", "serverUrl": "us-west-1.hgmointernal.net", "shouldAlert": false, "subtests": [], "value": 348.5316171646118}, {"extraOptions": ["c3.xlarge"], "lowerIsBetter": true, "name": "overall", "serverUrl": "us-west-1.hgmointernal.net", "shouldAlert": false, "subtests": [], "value": 487.2312319278717}, {"extraOptions": ["c3.xlarge"], "lowerIsBetter": true, "name": "overall_clone", "serverUrl": "us-west-1.hgmointernal.net", "shouldAlert": false, "subtests": [], "value": 487.2312319278717}, {"extraOptions": ["c3.xlarge"], "lowerIsBetter": true, "name": "overall_clone_fullcheckout", "serverUrl": "us-west-1.hgmointernal.net", "shouldAlert": false, "subtests": [], "value": 487.2312319278717}]}
[vcs 2019-07-18T14:26:56.909Z] TinderboxPrint:<a href=https://us-west-1.hgmointernal.net/releases/mozilla-beta/rev/067e54510790223f29c1c10d8f488e10d679ae6c title='Built from mozilla-beta revision 067e54510790223f29c1c10d8f488e10d679ae6c'>067e54510790223f29c1c10d8f488e10d679ae6c</a>
[task 2019-07-18T14:26:56.910Z] executing ['bash', '-cx', 'cd /builds/worker/checkouts/gecko/ && cp -r /build/node_modules_eslint node_modules && ln -s ../tools/lint/eslint/eslint-plugin-mozilla node_modules && ln -s ../tools/lint/eslint/eslint-plugin-spidermonkey-js node_modules && ./mach lint -l eslint -f treeherder --quiet -f json:/builds/worker/mozlint.json\n']
[task 2019-07-18T14:26:56.919Z] + cd /builds/worker/checkouts/gecko/
[task 2019-07-18T14:26:56.919Z] + cp -r /build/node_modules_eslint node_modules
[task 2019-07-18T14:26:59.056Z] + ln -s ../tools/lint/eslint/eslint-plugin-mozilla node_modules
[task 2019-07-18T14:26:59.057Z] + ln -s ../tools/lint/eslint/eslint-plugin-spidermonkey-js node_modules
[task 2019-07-18T14:26:59.058Z] + ./mach lint -l eslint -f treeherder --quiet -f json:/builds/worker/mozlint.json
[task 2019-07-18T14:27:00.814Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2019-07-18T14:27:00.814Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2019-07-18T14:27:02.448Z] Installing setuptools, pip, wheel...done.
[task 2019-07-18T14:27:04.708Z] running build_ext
[task 2019-07-18T14:27:04.708Z] building 'psutil._psutil_linux' extension
[task 2019-07-18T14:27:04.708Z] creating build
[task 2019-07-18T14:27:04.708Z] creating build/temp.linux-x86_64-2.7
[task 2019-07-18T14:27:04.708Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2019-07-18T14:27:04.708Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-07-18T14:27:04.708Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-07-18T14:27:04.708Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2019-07-18T14:27:04.708Z] creating build/lib.linux-x86_64-2.7
[task 2019-07-18T14:27:04.708Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2019-07-18T14:27:04.708Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2019-07-18T14:27:04.708Z] building 'psutil._psutil_posix' extension
[task 2019-07-18T14:27:04.708Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-07-18T14:27:04.708Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-07-18T14:27:04.709Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-07-18T14:27:04.709Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-07-18T14:27:04.709Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-07-18T14:27:04.709Z]
[task 2019-07-18T14:27:04.709Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[taskcluster:error] Task timeout after 1800 seconds. Force killing container.

Comment 1

[Intermittent Failures Robot]

1 failures in 4608 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• lint: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1567838&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 2

[Ionuț Goldan [:igoldan]]

I'm experiencing this problem, especially when backfilling last 25 commits.
Somehow, this is making the AC(Bk) backfill task run over 30 minutes. For each commit we require, it needs some extra minutes.

For example, backfilling 5 commits requires us 8 minutes of runtime
This roughly implies that backfilling 25 commits would require us roughly 40 minutes.

I suggest increasing the timeout limit to 45 minutes.

Comment 3

[Intermittent Failures Robot]

1 failures in 4439 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• lint: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1567838&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 4

[Intermittent Failures Robot]

2 failures in 4666 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• lint: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1567838&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 5

[Intermittent Failures Robot]

1 failures in 5073 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• lint: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1567838&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 6

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Comment 7

[Intermittent Failures Robot]

1 failures in 4350 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1567838&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 8

[Andreea Pavel [:apavel]]

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=278136997&repo=autoland&lineNumber=351

[task 2019-11-26T01:17:35.240Z] Installing setuptools, pip, wheel...done.
[task 2019-11-26T01:17:36.056Z] running build_ext
[task 2019-11-26T01:17:36.056Z] building 'psutil._psutil_linux' extension
[task 2019-11-26T01:17:36.056Z] creating build
[task 2019-11-26T01:17:36.056Z] creating build/temp.linux-x86_64-2.7
[task 2019-11-26T01:17:36.056Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2019-11-26T01:17:36.056Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-11-26T01:17:36.056Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-11-26T01:17:36.056Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2019-11-26T01:17:36.056Z] creating build/lib.linux-x86_64-2.7
[task 2019-11-26T01:17:36.056Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2019-11-26T01:17:36.056Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2019-11-26T01:17:36.056Z] building 'psutil._psutil_posix' extension
[task 2019-11-26T01:17:36.056Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-11-26T01:17:36.056Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=563 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-11-26T01:17:36.056Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-11-26T01:17:36.056Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-11-26T01:17:36.056Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-11-26T01:17:36.056Z]
[task 2019-11-26T01:17:36.056Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[taskcluster:error] Task timeout after 1200 seconds. Force killing container.
[taskcluster 2019-11-26 01:36:30.455Z] === Task Finished ===
[taskcluster 2019-11-26 01:36:30.455Z] Unsuccessful task run with exit code: -1 completed in 1223.898 seconds

Comment 9

[Intermittent Failures Robot]

2 failures in 4331 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux32-shippable: 1
• lint: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1567838&startday=2019-11-25&endday=2019-12-01&tree=all

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.