The default bug view has changed. See this FAQ.

try-bld-spot in us-west-2 are failing to init mock environment

RESOLVED FIXED

Status

Release Engineering
General Automation
--
critical
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: nthomas, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
eg https://tbpl.mozilla.org/php/getParsedLog.php?id=41664877&tree=Try

========= Started mock-tgt mozilla-centos6-x86_64 failed (results: 2, elapsed: 2 secs) (at 2014-06-13 02:51:49.988385) =========
mock_mozilla -r mozilla-centos6-x86_64 --init
 in dir /builds/slave/try-l64-asan-00000000000000000/build (timeout 1200 secs)
 watching logfiles {}
 argv: mock_mozilla -r mozilla-centos6-x86_64 --init
 environment:
...
INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: clean
INFO: chroot (/builds/mock_mozilla/mozilla-centos6-x86_64) unlocked and deleted
State Changed: unlock buildroot
State Changed: init
State Changed: lock buildroot
Mock Version: 1.0.3
INFO: Mock Version: 1.0.3
INFO: calling preinit hooks
INFO: enabled root cache
State Changed: unpacking root cache
ERROR: Command failed. See logs for output.
 # ['tar', '--use-compress-program', 'pigz', '-xf', '/builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/cache.tar.gz', '-C', '/builds/mock_mozilla/mozilla-centos6-x86_64/root/']
program finished with exit code 2
elapsedTime=2.710941

Looking at try-linux64-spot-308:/builds/mock_mozilla/mozilla-centos6-x86_64/result/root.log

INFO backend.py:924:  Mock Version: 1.0.3
DEBUG backend.py:276:  rootdir = /builds/mock_mozilla/mozilla-centos6-x86_64/root/
DEBUG backend.py:277:  resultdir = /builds/mock_mozilla/mozilla-centos6-x86_64/result
INFO backend.py:280:  calling preinit hooks
DEBUG util.py:57:  ensuring that dir exists: /builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/
DEBUG util.py:303:  Executing command: ['tar', '--use-compress-program', 'pigz', '-xf', '/builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/cache.tar.gz', '-C', '/builds/mock_mozilla/mozilla-centos6-x86_64/root/']
DEBUG util.py:259:  tar: ./etc/resolv.conf: time stamp 2014-06-13 11:15:19 is 19255.584940646 s in the future
DEBUG util.py:259:  tar: ./etc/hosts: time stamp 2014-06-13 11:15:45 is 19281.574152936 s in the future
DEBUG util.py:259:  pigz abort: corrupted input -- invalid deflate data: <stdin>
DEBUG util.py:259:  tar: Unexpected EOF in archive
DEBUG util.py:259:  tar: Unexpected EOF in archive
DEBUG util.py:259:  tar: Error is not recoverable: exiting now
DEBUG util.py:343:  Child return code was: 2
DEBUG util.py:111:  kill orphans

Possibly a bad AMI, need to confirm other slaves have same issue.
(Reporter)

Comment 1

3 years ago
https://wiki.mozilla.org/ReleaseEngineering/How_To/Manage_spot_AMIs is the relevant doc, if I'm on the right track.

[cltbld@try-linux64-spot-308.try.releng.usw2 result]$ curl http://169.254.169.254/latest/meta-data/ami-id
ami-23017c13

[buildduty@aws-manager1 aws_manager]$ bin/python cloud-tools/scripts/get_spot_amis.py  | grep ^try
try-linux64, us-east-1: ami-9055a8f8
try-linux64, us-west-2: ami-23017c13
(Reporter)

Comment 2

3 years ago
try-linux64-spot-385 has the same symptoms, but is doing an 'Android 2.2 Debug try build' which didn't fail. Looks from result/root.log that it built from scratch.

try-linux64-spot-316 looks like 308, so lets consider that confirmed.

I don't actually know the mechanics of where
 /builds/mock_mozilla/cache/mozilla-centos6-x86_64/root_cache/cache.tar.gz
comes from, or how do remove the AMI & snapshots per the wiki doc in the previous comment. Rail is our expert here.
It's possible this is due to bug 1023477 (tweaking FS parameters).
Blocks: 1023477
(Assignee)

Comment 4

3 years ago
We should be using older AMIs now for fresh spot instances.

Updated

3 years ago
Assignee: nobody → rail
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Depends on: 1025842
You need to log in before you can comment on or make changes to this bug.