1024962 - try-bld-spot in us-west-2 are failing to init mock environment

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

eg https://tbpl.mozilla.org/php/getParsedLog.php?id=41664877&tree=Try

========= Started mock-tgt mozilla-centos6-x86_64 failed (results: 2, elapsed: 2 secs) (at 2014-06-13 02:51:49.988385) =========
mock_mozilla -r mozilla-centos6-x86_64 --init
 in dir /builds/slave/try-l64-asan-00000000000000000/build (timeout 1200 secs)
 watching logfiles {}
 argv: mock_mozilla -r mozilla-centos6-x86_64 --init
 environment:
...
INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: clean
INFO: chroot (/builds/mock_mozilla/mozilla-centos6-x86_64) unlocked and deleted
State Changed: unlock buildroot
State Changed: init
State Changed: lock buildroot
Mock Version: 1.0.3
INFO: Mock Version: 1.0.3
INFO: calling preinit hooks
INFO: enabled root cache
State Changed: unpacking root cache
ERROR: Command failed. See logs for output.
 # ['tar', '--use-compress-program', 'pigz', '-xf', '/builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/cache.tar.gz', '-C', '/builds/mock_mozilla/mozilla-centos6-x86_64/root/']
program finished with exit code 2
elapsedTime=2.710941

Looking at try-linux64-spot-308:/builds/mock_mozilla/mozilla-centos6-x86_64/result/root.log

INFO backend.py:924:  Mock Version: 1.0.3
DEBUG backend.py:276:  rootdir = /builds/mock_mozilla/mozilla-centos6-x86_64/root/
DEBUG backend.py:277:  resultdir = /builds/mock_mozilla/mozilla-centos6-x86_64/result
INFO backend.py:280:  calling preinit hooks
DEBUG util.py:57:  ensuring that dir exists: /builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/
DEBUG util.py:303:  Executing command: ['tar', '--use-compress-program', 'pigz', '-xf', '/builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/cache.tar.gz', '-C', '/builds/mock_mozilla/mozilla-centos6-x86_64/root/']
DEBUG util.py:259:  tar: ./etc/resolv.conf: time stamp 2014-06-13 11:15:19 is 19255.584940646 s in the future
DEBUG util.py:259:  tar: ./etc/hosts: time stamp 2014-06-13 11:15:45 is 19281.574152936 s in the future
DEBUG util.py:259:  pigz abort: corrupted input -- invalid deflate data: <stdin>
DEBUG util.py:259:  tar: Unexpected EOF in archive
DEBUG util.py:259:  tar: Unexpected EOF in archive
DEBUG util.py:259:  tar: Error is not recoverable: exiting now
DEBUG util.py:343:  Child return code was: 2
DEBUG util.py:111:  kill orphans

Possibly a bad AMI, need to confirm other slaves have same issue.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_spot_AMIs is the relevant doc, if I'm on the right track.

[[e-mail]2 result]$ curl http://169.254.169.254/latest/meta-data/ami-id
ami-23017c13

[buildduty@aws-manager1 aws_manager]$ bin/python cloud-tools/scripts/get_spot_amis.py  | grep ^try
try-linux64, us-east-1: ami-9055a8f8
try-linux64, us-west-2: ami-23017c13

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

try-linux64-spot-385 has the same symptoms, but is doing an 'Android 2.2 Debug try build' which didn't fail. Looks from result/root.log that it built from scratch.

try-linux64-spot-316 looks like 308, so lets consider that confirmed.

I don't actually know the mechanics of where
 /builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/cache.tar.gz
comes from, or how do remove the AMI & snapshots per the wiki doc in the previous comment. Rail is our expert here.

Comment 3

[Chris AtLee [:catlee]]

It's possible this is due to bug 1023477 (tweaking FS parameters).

Comment 4

[Rail Aliiev [:rail]]

We should be using older AMIs now for fresh spot instances.