Closed Bug 1024962 Opened 6 years ago Closed 6 years ago

try-bld-spot in us-west-2 are failing to init mock environment


(Release Engineering :: General, defect, critical)

Not set


(Not tracked)



(Reporter: nthomas, Assigned: rail)




========= Started mock-tgt mozilla-centos6-x86_64 failed (results: 2, elapsed: 2 secs) (at 2014-06-13 02:51:49.988385) =========
mock_mozilla -r mozilla-centos6-x86_64 --init
 in dir /builds/slave/try-l64-asan-00000000000000000/build (timeout 1200 secs)
 watching logfiles {}
 argv: mock_mozilla -r mozilla-centos6-x86_64 --init
INFO: version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: clean
INFO: chroot (/builds/mock_mozilla/mozilla-centos6-x86_64) unlocked and deleted
State Changed: unlock buildroot
State Changed: init
State Changed: lock buildroot
Mock Version: 1.0.3
INFO: Mock Version: 1.0.3
INFO: calling preinit hooks
INFO: enabled root cache
State Changed: unpacking root cache
ERROR: Command failed. See logs for output.
 # ['tar', '--use-compress-program', 'pigz', '-xf', '/builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/cache.tar.gz', '-C', '/builds/mock_mozilla/mozilla-centos6-x86_64/root/']
program finished with exit code 2

Looking at try-linux64-spot-308:/builds/mock_mozilla/mozilla-centos6-x86_64/result/root.log

INFO  Mock Version: 1.0.3
DEBUG  rootdir = /builds/mock_mozilla/mozilla-centos6-x86_64/root/
DEBUG  resultdir = /builds/mock_mozilla/mozilla-centos6-x86_64/result
INFO  calling preinit hooks
DEBUG  ensuring that dir exists: /builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/
DEBUG  Executing command: ['tar', '--use-compress-program', 'pigz', '-xf', '/builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/cache.tar.gz', '-C', '/builds/mock_mozilla/mozilla-centos6-x86_64/root/']
DEBUG  tar: ./etc/resolv.conf: time stamp 2014-06-13 11:15:19 is 19255.584940646 s in the future
DEBUG  tar: ./etc/hosts: time stamp 2014-06-13 11:15:45 is 19281.574152936 s in the future
DEBUG  pigz abort: corrupted input -- invalid deflate data: <stdin>
DEBUG  tar: Unexpected EOF in archive
DEBUG  tar: Unexpected EOF in archive
DEBUG  tar: Error is not recoverable: exiting now
DEBUG  Child return code was: 2
DEBUG  kill orphans

Possibly a bad AMI, need to confirm other slaves have same issue. is the relevant doc, if I'm on the right track.

[cltbld@try-linux64-spot-308.try.releng.usw2 result]$ curl

[buildduty@aws-manager1 aws_manager]$ bin/python cloud-tools/scripts/  | grep ^try
try-linux64, us-east-1: ami-9055a8f8
try-linux64, us-west-2: ami-23017c13
try-linux64-spot-385 has the same symptoms, but is doing an 'Android 2.2 Debug try build' which didn't fail. Looks from result/root.log that it built from scratch.

try-linux64-spot-316 looks like 308, so lets consider that confirmed.

I don't actually know the mechanics of where
comes from, or how do remove the AMI & snapshots per the wiki doc in the previous comment. Rail is our expert here.
It's possible this is due to bug 1023477 (tweaking FS parameters).
Blocks: 1023477
We should be using older AMIs now for fresh spot instances.
Assignee: nobody → rail
Closed: 6 years ago
Resolution: --- → FIXED
Depends on: 1025842
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.