Closed Bug 1190860 Opened 10 years ago Closed 10 years ago

Run tooltool inside mock in buildbot jobs

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cbook, Assigned: nthomas)

References

Details

(Whiteboard: tooltool mock)

Attachments

(4 files, 1 obsolete file)

[buildbotcustom] Fix up buildbot style builds
2.45 KB, patch
Callek
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
Buildbot builders modified
7.82 KB, text/plain
Details
[buildbotcustom] Fix valgrind
4.42 KB, patch
jlund
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
[buildbot-configs] Copy in tooltool.py, and run it from in-mock location (with TB too)
25.74 KB, patch
jlund
: review+
nthomas
: checked-in+
Details | Diff | Splinter Review
Linux build seems to fail across the trees with: https://treeherder.mozilla.org/logviewer.html#?job_id=12472397&repo=mozilla-inbound Halting on failure while running ['sh', '/builds/slave/b2g-in-lx-00000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh', '/builds/slave/b2g-in-lx-00000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', '/builds/tooltool.py', '--authentication-file', '/builds/relengapi.tok'] closing integration trees for this
From the log 06:50:08 INFO - Executing: ['/builds/tooltool.py', '--authentication-file', '/builds/relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', '/builds/slave/m-in-lx-0000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'] 06:50:08 INFO - INFO - File gcc.tar.xz not present in local cache folder /builds/tooltool_cache 06:50:08 INFO - INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... 06:50:10 INFO - INFO - File gcc.tar.xz fetched from https://api.pub.build.mozilla.org/tooltool/ as /builds/slave/m-in-lx-0000000000000000000000/build/src/tmpoajZpM 06:50:10 INFO - INFO - File gtk3.tar.xz not present in local cache folder /builds/tooltool_cache 06:50:10 INFO - INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... 06:50:10 INFO - INFO - File gtk3.tar.xz fetched from https://api.pub.build.mozilla.org/tooltool/ as /builds/slave/m-in-lx-0000000000000000000000/build/src/tmpF39b8R 06:50:10 INFO - INFO - File sccache.tar.bz2 not present in local cache folder /builds/tooltool_cache 06:50:10 INFO - INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... 06:50:11 INFO - INFO - File sccache.tar.bz2 fetched from https://api.pub.build.mozilla.org/tooltool/ as /builds/slave/m-in-lx-0000000000000000000000/build/src/tmp_K1CNf 06:50:11 INFO - INFO - File integrity verified, renaming tmpoajZpM to gcc.tar.xz 06:50:11 INFO - INFO - Updating local cache /builds/tooltool_cache... 06:50:11 INFO - INFO - Local cache /builds/tooltool_cache updated with gcc.tar.xz 06:50:11 INFO - INFO - File integrity verified, renaming tmpF39b8R to gtk3.tar.xz 06:50:11 INFO - INFO - Updating local cache /builds/tooltool_cache... 06:50:11 INFO - INFO - Local cache /builds/tooltool_cache updated with gtk3.tar.xz 06:50:11 INFO - INFO - File integrity verified, renaming tmp_K1CNf to sccache.tar.bz2 06:50:11 INFO - INFO - Updating local cache /builds/tooltool_cache... 06:50:11 INFO - INFO - Local cache /builds/tooltool_cache updated with sccache.tar.bz2 06:50:11 INFO - INFO - untarring "gcc.tar.xz" 06:50:21 INFO - INFO - untarring "gtk3.tar.xz" 06:50:22 INFO - gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory 06:50:22 INFO - gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory 06:50:22 INFO - gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory 06:50:22 INFO - INFO - untarring "sccache.tar.bz2" 06:50:22 INFO - ERROR - The following files failed: 'gtk3.tar.xz' 06:50:51 INFO - retry: Calling <function run_with_timeout at 0x7f76d3f6c938> with args: (['/builds/tooltool.py', '--authentication-file', '/builds/relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', '/builds/slave/m-in-lx-0000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #2 06:50:51 INFO - Executing: ['/builds/tooltool.py', '--authentication-file', '/builds/relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', '/builds/slave/m-in-lx-0000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'] 06:50:51 INFO - INFO - rm tree: gcc 06:50:52 INFO - INFO - untarring "gcc.tar.xz" 06:51:01 INFO - INFO - rm tree: gtk3 06:51:01 INFO - INFO - untarring "gtk3.tar.xz" 06:51:02 INFO - gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory 06:51:02 INFO - gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory 06:51:02 INFO - gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
Sheriffs are pinging glandium to see see if this is fallout from a recent landing Tomcat|sheriffduty maybe related to bug 1186748 but that was on inbound before and was fine Tomcat|sheriffduty RyanVM: does this maybe need a clobber ? Tomcat|sheriffduty oh there a clobber change RyanVM i thought he landed a clobber touch RyanVM yeah Tomcat|sheriffduty also i guess this would have failed before and not just 1 push after the re-merge Callek_cloud thats definitely a glandium fallout Callek_cloud based on my skim of bugmail last night =-= hwine-ooo is now known as hwine Tomcat|sheriffduty Callek_cloud: yeah but thats a strange fallout Callek_cloud I don't disagree Tomcat|sheriffduty https://treeherder.mozilla.org/#/jobs?repo=fx-team&revision=256463501f11 has green builds and was the merge back from m-c Tomcat|sheriffduty and then the next push fails Callek_cloud but if it leaves behind state in-tree, especially as it relates to setup.sh that he is executing from the tooltool package, I not totally shocked that there is clobber-related bustage involved Callek_cloud admits he hasn't looked at the setup.sh or any of the code in depth, but #releng or #build is likely a better venue to discuss, since thats where glandium is likely to be Tomcat|sheriffduty Callek_cloud: somehow even the mozilla-central push has green builds Tomcat|sheriffduty so wonder whats going on Callek_cloud Tomcat|sheriffduty: ooooo I wonder.
Blocks: 1180624
Based on the above logs, and some retriggers, going all the way back to some greens on CLOBBER file touches with glandiums gtk3 push, I came to the conclusion that its a bustage caused by Bug 1189948 due to executing setup.sh from tooltool I pushed to puppet to back out those changes... https://hg.mozilla.org/build/puppet/rev/6e7de1c8f6a3 https://hg.mozilla.org/build/puppet/rev/483ee5bd5b8d I then enumerated all current AMI id's and deleted them from aws, and then ran the terminate script for all bld-* and try-* hosts using said ID's. Log of that below. I think we are good now, but leaving open pending retrigger results: (aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/get_spot_amis.py bld-linux64, us-east-1: ami-7108a81a (spot-bld-linux64-2015-08-04-09-06, ebs) try-linux64, us-east-1: ami-1b04a470 (spot-try-linux64-2015-08-04-08-48, ebs) tst-linux64, us-east-1: ami-e906a682 (spot-tst-linux64-2015-08-04-09-05, ebs) tst-linux32, us-east-1: ami-d506a6be (spot-tst-linux32-2015-08-04-09-08, ebs) tst-emulator64, us-east-1: ami-d108a8ba (spot-tst-emulator64-2015-08-04-09-26, ebs) bld-linux64, us-west-2: ami-9fb4bdaf (spot-bld-linux64-2015-08-04-09-06, ebs) try-linux64, us-west-2: ami-97b5bca7 (spot-try-linux64-2015-08-04-08-48, ebs) tst-linux64, us-west-2: ami-3db4bd0d (spot-tst-linux64-2015-08-04-09-05, ebs) tst-linux32, us-west-2: ami-8fe9e0bf (spot-tst-linux32-2015-08-03-09-11, ebs) tst-emulator64, us-west-2: ami-d5b4bde5 (spot-tst-emulator64-2015-08-04-09-26, ebs) (aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/aws_terminate_by_ami_id.py --help usage: aws_terminate_by_ami_id.py [-h] [-r REGIONS] [-v] AMI [AMI ...] positional arguments: AMI AMI IDs optional arguments: -h, --help show this help message and exit -r REGIONS, --region REGIONS -v, --verbose Verbose logging (aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/aws_terminate_by_ami_id.py -r us-e ast-1 ami-7108a81a ami-1b04a470 2015-08-04 10:02:12,359 - INFO - Preparing to terminate the following 60 instances: 2015-08-04 10:02:12,359 - INFO - i-ff3df52d (bld-linux64-spot-129) 2015-08-04 10:02:12,359 - INFO - i-fe3df52c (bld-linux64-spot-142) 2015-08-04 10:02:12,359 - INFO - i-663cf4b4 (bld-linux64-spot-159) 2015-08-04 10:02:12,362 - INFO - i-d23cf400 (bld-linux64-spot-139) 2015-08-04 10:02:12,362 - INFO - i-343bf3e6 (bld-linux64-spot-131) 2015-08-04 10:02:12,362 - INFO - i-6d3df5bf (bld-linux64-spot-122) 2015-08-04 10:02:12,362 - INFO - i-ceb77f1c (bld-linux64-spot-081) 2015-08-04 10:02:12,362 - INFO - i-22ab63f0 (bld-linux64-spot-080) 2015-08-04 10:02:12,362 - INFO - i-c13cf413 (bld-linux64-spot-028) 2015-08-04 10:02:12,362 - INFO - i-7fc30bad (bld-linux64-spot-094) 2015-08-04 10:02:12,362 - INFO - i-bc3cf46e (bld-linux64-spot-199) 2015-08-04 10:02:12,362 - INFO - i-793bf3ab (bld-linux64-spot-228) 2015-08-04 10:02:12,362 - INFO - i-6b3cf4b9 (bld-linux64-spot-034) 2015-08-04 10:02:12,362 - INFO - i-c63df514 (bld-linux64-spot-195) 2015-08-04 10:02:12,362 - INFO - i-803cf452 (bld-linux64-spot-138) 2015-08-04 10:02:12,362 - INFO - i-303cf4e2 (bld-linux64-spot-144) 2015-08-04 10:02:12,362 - INFO - i-3e3cf4ec (bld-linux64-spot-267) 2015-08-04 10:02:12,362 - INFO - i-6a3cf4b8 (bld-linux64-spot-023) 2015-08-04 10:02:12,365 - INFO - i-c93cf41b (bld-linux64-spot-146) 2015-08-04 10:02:12,365 - INFO - i-9a3cf448 (bld-linux64-spot-027) 2015-08-04 10:02:12,365 - INFO - i-393bf3eb (bld-linux64-spot-137) 2015-08-04 10:02:12,365 - INFO - i-683cf4ba (bld-linux64-spot-148) 2015-08-04 10:02:12,365 - INFO - i-e63df534 (bld-linux64-spot-133) 2015-08-04 10:02:12,365 - INFO - i-633cf4b1 (bld-linux64-spot-071) 2015-08-04 10:02:12,365 - INFO - i-3c3cf4ee (bld-linux64-spot-226) 2015-08-04 10:02:12,365 - INFO - i-363cf4e4 (bld-linux64-spot-250) 2015-08-04 10:02:12,365 - INFO - i-313bf3e3 (bld-linux64-spot-019) 2015-08-04 10:02:12,365 - INFO - i-243cf4f6 (bld-linux64-spot-017) 2015-08-04 10:02:12,365 - INFO - i-bd3cf46f (bld-linux64-spot-252) 2015-08-04 10:02:12,365 - INFO - i-773cf4a5 (bld-linux64-spot-029) 2015-08-04 10:02:12,365 - INFO - i-dc3cf40e (bld-linux64-spot-209) 2015-08-04 10:02:12,368 - INFO - i-c43df516 (bld-linux64-spot-090) 2015-08-04 10:02:12,368 - INFO - i-c23cf410 (bld-linux64-spot-035) 2015-08-04 10:02:12,368 - INFO - i-ad3cf47f (bld-linux64-spot-152) 2015-08-04 10:02:12,368 - INFO - i-8b3cf459 (bld-linux64-spot-145) 2015-08-04 10:02:12,368 - INFO - i-f73df525 (bld-linux64-spot-093) 2015-08-04 10:02:12,368 - INFO - i-7c3cf4ae (bld-linux64-spot-258) 2015-08-04 10:02:12,368 - INFO - i-813cf453 (bld-linux64-spot-245) 2015-08-04 10:02:12,368 - INFO - i-a03cf472 (bld-linux64-spot-147) 2015-08-04 10:02:12,368 - INFO - i-423cf490 (bld-linux64-spot-155) 2015-08-04 10:02:12,368 - INFO - i-85657a2d (bld-linux64-spot-156) 2015-08-04 10:02:12,368 - INFO - i-cd3df51f (bld-linux64-spot-099) 2015-08-04 10:02:12,368 - INFO - i-d33df501 (bld-linux64-spot-130) 2015-08-04 10:02:12,368 - INFO - i-943cf446 (bld-linux64-spot-270) 2015-08-04 10:02:12,371 - INFO - i-623cf4b0 (try-linux64-spot-145) 2015-08-04 10:02:12,371 - INFO - i-b23df560 (bld-linux64-spot-157) 2015-08-04 10:02:12,371 - INFO - i-873cf455 (bld-linux64-spot-150) 2015-08-04 10:02:12,371 - INFO - i-383cf4ea (bld-linux64-spot-242) 2015-08-04 10:02:12,371 - INFO - i-96dc1444 (bld-linux64-spot-158) 2015-08-04 10:02:12,371 - INFO - i-2c3bf3fe (bld-linux64-spot-038) 2015-08-04 10:02:12,371 - INFO - i-3d8c47ef (bld-linux64-spot-024) 2015-08-04 10:02:12,371 - INFO - i-3b3cf4e9 (bld-linux64-spot-240) 2015-08-04 10:02:12,371 - INFO - i-b23cf460 (bld-linux64-spot-153) 2015-08-04 10:02:12,371 - INFO - i-bb3cf469 (bld-linux64-spot-143) 2015-08-04 10:02:12,371 - INFO - i-303bf3e2 (bld-linux64-spot-033) 2015-08-04 10:02:12,374 - INFO - i-f53df527 (bld-linux64-spot-192) 2015-08-04 10:02:12,374 - INFO - i-e466794c (bld-linux64-spot-011) 2015-08-04 10:02:12,374 - INFO - i-f83df52a (bld-linux64-spot-244) 2015-08-04 10:02:12,374 - INFO - i-2b3bf3f9 (try-linux64-spot-148) 2015-08-04 10:02:12,374 - INFO - i-3f3bf3ed (bld-linux64-spot-227) Are you sure you want to kill these? ^ y/N >y ARE YOU SURE YOU WANT TO KILL THESE? ^ LAST WARNING!!! y/N >y 2015-08-04 10:02:19,541 - WARNING - The instances mentioned above are about to be terminated 2015-08-04 10:02:19,541 - WARNING - Waiting extra 60 seconds to make sure... 2015-08-04 10:03:19,601 - WARNING - Starting... 2015-08-04 10:03:19,601 - WARNING - Terminating Instance:i-ff3df52d... 2015-08-04 10:03:20,257 - WARNING - Done. 2015-08-04 10:03:20,257 - WARNING - Terminating Instance:i-fe3df52c... 2015-08-04 10:03:20,637 - WARNING - Done. 2015-08-04 10:03:20,637 - WARNING - Terminating Instance:i-663cf4b4... 2015-08-04 10:03:21,163 - WARNING - Done. 2015-08-04 10:03:21,163 - WARNING - Terminating Instance:i-d23cf400... 2015-08-04 10:03:21,504 - WARNING - Done. 2015-08-04 10:03:21,504 - WARNING - Terminating Instance:i-343bf3e6... 2015-08-04 10:03:22,103 - WARNING - Done. 2015-08-04 10:03:22,103 - WARNING - Terminating Instance:i-6d3df5bf... 2015-08-04 10:03:22,734 - WARNING - Done. 2015-08-04 10:03:22,734 - WARNING - Terminating Instance:i-ceb77f1c... 2015-08-04 10:03:23,094 - WARNING - Done. 2015-08-04 10:03:23,094 - WARNING - Terminating Instance:i-22ab63f0... 2015-08-04 10:03:23,437 - WARNING - Done. 2015-08-04 10:03:23,437 - WARNING - Terminating Instance:i-c13cf413... 2015-08-04 10:03:23,791 - WARNING - Done. 2015-08-04 10:03:23,791 - WARNING - Terminating Instance:i-7fc30bad... 2015-08-04 10:03:24,517 - WARNING - Done. 2015-08-04 10:03:24,517 - WARNING - Terminating Instance:i-bc3cf46e... 2015-08-04 10:03:24,897 - WARNING - Done. 2015-08-04 10:03:24,897 - WARNING - Terminating Instance:i-793bf3ab... 2015-08-04 10:03:25,231 - WARNING - Done. 2015-08-04 10:03:25,231 - WARNING - Terminating Instance:i-6b3cf4b9... 2015-08-04 10:03:25,613 - WARNING - Done. 2015-08-04 10:03:25,613 - WARNING - Terminating Instance:i-c63df514... 2015-08-04 10:03:25,999 - WARNING - Done. 2015-08-04 10:03:25,999 - WARNING - Terminating Instance:i-803cf452... 2015-08-04 10:03:26,318 - WARNING - Done. 2015-08-04 10:03:26,318 - WARNING - Terminating Instance:i-303cf4e2... 2015-08-04 10:03:26,694 - WARNING - Done. 2015-08-04 10:03:26,694 - WARNING - Terminating Instance:i-3e3cf4ec... 2015-08-04 10:03:27,190 - WARNING - Done. 2015-08-04 10:03:27,190 - WARNING - Terminating Instance:i-6a3cf4b8... 2015-08-04 10:03:27,508 - WARNING - Done. 2015-08-04 10:03:27,509 - WARNING - Terminating Instance:i-c93cf41b... 2015-08-04 10:03:27,883 - WARNING - Done. 2015-08-04 10:03:27,883 - WARNING - Terminating Instance:i-9a3cf448... 2015-08-04 10:03:28,336 - WARNING - Done. 2015-08-04 10:03:28,337 - WARNING - Terminating Instance:i-393bf3eb... 2015-08-04 10:03:28,845 - WARNING - Done. 2015-08-04 10:03:28,845 - WARNING - Terminating Instance:i-683cf4ba... 2015-08-04 10:03:29,233 - WARNING - Done. 2015-08-04 10:03:29,233 - WARNING - Terminating Instance:i-e63df534... 2015-08-04 10:03:29,540 - WARNING - Done. 2015-08-04 10:03:29,540 - WARNING - Terminating Instance:i-633cf4b1... 2015-08-04 10:03:29,849 - WARNING - Done. 2015-08-04 10:03:29,849 - WARNING - Terminating Instance:i-3c3cf4ee... 2015-08-04 10:03:30,145 - WARNING - Done. 2015-08-04 10:03:30,145 - WARNING - Terminating Instance:i-363cf4e4... 2015-08-04 10:03:30,469 - WARNING - Done. 2015-08-04 10:03:30,469 - WARNING - Terminating Instance:i-313bf3e3... 2015-08-04 10:03:30,808 - WARNING - Done. 2015-08-04 10:03:30,808 - WARNING - Terminating Instance:i-243cf4f6... 2015-08-04 10:03:31,147 - WARNING - Done. 2015-08-04 10:03:31,147 - WARNING - Terminating Instance:i-bd3cf46f... 2015-08-04 10:03:31,502 - WARNING - Done. 2015-08-04 10:03:31,502 - WARNING - Terminating Instance:i-773cf4a5... 2015-08-04 10:03:31,864 - WARNING - Done. 2015-08-04 10:03:31,864 - WARNING - Terminating Instance:i-dc3cf40e... 2015-08-04 10:03:32,343 - WARNING - Done. 2015-08-04 10:03:32,343 - WARNING - Terminating Instance:i-c43df516... 2015-08-04 10:03:33,002 - WARNING - Done. 2015-08-04 10:03:33,002 - WARNING - Terminating Instance:i-c23cf410... 2015-08-04 10:03:33,333 - WARNING - Done. 2015-08-04 10:03:33,333 - WARNING - Terminating Instance:i-ad3cf47f... 2015-08-04 10:03:33,643 - WARNING - Done. 2015-08-04 10:03:33,643 - WARNING - Terminating Instance:i-8b3cf459... 2015-08-04 10:03:34,064 - WARNING - Done. 2015-08-04 10:03:34,064 - WARNING - Terminating Instance:i-f73df525... 2015-08-04 10:03:34,389 - WARNING - Done. 2015-08-04 10:03:34,389 - WARNING - Terminating Instance:i-7c3cf4ae... 2015-08-04 10:03:34,816 - WARNING - Done. 2015-08-04 10:03:34,816 - WARNING - Terminating Instance:i-813cf453... 2015-08-04 10:03:35,226 - WARNING - Done. 2015-08-04 10:03:35,226 - WARNING - Terminating Instance:i-a03cf472... 2015-08-04 10:03:35,559 - WARNING - Done. 2015-08-04 10:03:35,559 - WARNING - Terminating Instance:i-423cf490... 2015-08-04 10:03:35,905 - WARNING - Done. 2015-08-04 10:03:35,905 - WARNING - Terminating Instance:i-85657a2d... 2015-08-04 10:03:36,274 - WARNING - Done. 2015-08-04 10:03:36,275 - WARNING - Terminating Instance:i-cd3df51f... 2015-08-04 10:03:36,612 - WARNING - Done. 2015-08-04 10:03:36,612 - WARNING - Terminating Instance:i-d33df501... 2015-08-04 10:03:36,931 - WARNING - Done. 2015-08-04 10:03:36,931 - WARNING - Terminating Instance:i-943cf446... 2015-08-04 10:03:37,322 - WARNING - Done. 2015-08-04 10:03:37,322 - WARNING - Terminating Instance:i-623cf4b0... 2015-08-04 10:03:37,726 - WARNING - Done. 2015-08-04 10:03:37,726 - WARNING - Terminating Instance:i-b23df560... 2015-08-04 10:03:38,223 - WARNING - Done. 2015-08-04 10:03:38,223 - WARNING - Terminating Instance:i-873cf455... 2015-08-04 10:03:38,586 - WARNING - Done. 2015-08-04 10:03:38,586 - WARNING - Terminating Instance:i-383cf4ea... 2015-08-04 10:03:38,948 - WARNING - Done. 2015-08-04 10:03:38,948 - WARNING - Terminating Instance:i-96dc1444... 2015-08-04 10:03:39,371 - WARNING - Done. 2015-08-04 10:03:39,371 - WARNING - Terminating Instance:i-2c3bf3fe... 2015-08-04 10:03:39,828 - WARNING - Done. 2015-08-04 10:03:39,828 - WARNING - Terminating Instance:i-3d8c47ef... 2015-08-04 10:03:40,216 - WARNING - Done. 2015-08-04 10:03:40,216 - WARNING - Terminating Instance:i-3b3cf4e9... 2015-08-04 10:03:40,683 - WARNING - Done. 2015-08-04 10:03:40,683 - WARNING - Terminating Instance:i-b23cf460... 2015-08-04 10:03:41,024 - WARNING - Done. 2015-08-04 10:03:41,024 - WARNING - Terminating Instance:i-bb3cf469... 2015-08-04 10:03:41,342 - WARNING - Done. 2015-08-04 10:03:41,343 - WARNING - Terminating Instance:i-303bf3e2... 2015-08-04 10:03:41,650 - WARNING - Done. 2015-08-04 10:03:41,650 - WARNING - Terminating Instance:i-f53df527... 2015-08-04 10:03:42,049 - WARNING - Done. 2015-08-04 10:03:42,049 - WARNING - Terminating Instance:i-e466794c... 2015-08-04 10:03:42,387 - WARNING - Done. 2015-08-04 10:03:42,387 - WARNING - Terminating Instance:i-f83df52a... 2015-08-04 10:03:42,778 - WARNING - Done. 2015-08-04 10:03:42,778 - WARNING - Terminating Instance:i-2b3bf3f9... 2015-08-04 10:03:43,136 - WARNING - Done. 2015-08-04 10:03:43,136 - WARNING - Terminating Instance:i-3f3bf3ed... 2015-08-04 10:03:43,496 - WARNING - Done. (aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/aws_terminate_by_ami_id.py -r us-w est-2 ami-9fb4bdaf ami-97b5bca7 2015-08-04 10:05:18,582 - INFO - Preparing to terminate the following 220 instances: 2015-08-04 10:05:18,582 - INFO - i-c1389408 (try-linux64-spot-421) 2015-08-04 10:05:18,582 - INFO - i-e7842b22 (try-linux64-spot-1099) 2015-08-04 10:05:18,582 - INFO - i-a50ba76c (try-linux64-spot-499) 2015-08-04 10:05:18,582 - INFO - i-fa0b6d32 (bld-linux64-spot-542) 2015-08-04 10:05:18,582 - INFO - i-f20b6d3a (bld-linux64-spot-424) 2015-08-04 10:05:18,582 - INFO - i-450b6d8d (bld-linux64-spot-425) 2015-08-04 10:05:18,582 - INFO - i-7a0b6db2 (bld-linux64-spot-545) 2015-08-04 10:05:18,583 - INFO - i-b5a80770 (bld-linux64-spot-590) 2015-08-04 10:05:18,583 - INFO - i-aa5ff363 (try-linux64-spot-494) 2015-08-04 10:05:18,583 - INFO - i-b4a80771 (bld-linux64-spot-1062) 2015-08-04 10:05:18,583 - INFO - i-fb238f32 (try-linux64-spot-402) 2015-08-04 10:05:18,583 - INFO - i-dfa20d1a (bld-linux64-spot-364) 2015-08-04 10:05:18,583 - INFO - i-62268aab (try-linux64-spot-437) 2015-08-04 10:05:18,583 - INFO - i-4f0b6d87 (bld-linux64-spot-426) 2015-08-04 10:05:18,583 - INFO - i-a55ff36c (bld-linux64-spot-1063) 2015-08-04 10:05:18,583 - INFO - i-778629b2 (try-linux64-spot-489) 2015-08-04 10:05:18,583 - INFO - i-39278bf0 (try-linux64-spot-387) 2015-08-04 10:05:18,583 - INFO - i-385ff3f1 (try-linux64-spot-439) 2015-08-04 10:05:18,583 - INFO - i-cc0b6d04 (try-linux64-spot-432) 2015-08-04 10:05:18,584 - INFO - i-b0a80775 (bld-linux64-spot-1064) 2015-08-04 10:05:18,584 - INFO - i-420b6d8a (bld-linux64-spot-369) 2015-08-04 10:05:18,584 - INFO - i-7bab04be (bld-linux64-spot-1065) 2015-08-04 10:05:18,584 - INFO - i-010aa6c8 (try-linux64-spot-424) 2015-08-04 10:05:18,584 - INFO - i-56852a93 (try-linux64-spot-1094) 2015-08-04 10:05:18,584 - INFO - i-ca086e02 (bld-linux64-spot-1080) 2015-08-04 10:05:18,584 - INFO - i-8704a84e (try-linux64-spot-498) 2015-08-04 10:05:18,584 - INFO - i-150fa3dc (try-linux64-spot-417) 2015-08-04 10:05:18,584 - INFO - i-080a6cc0 (bld-linux64-spot-1069) 2015-08-04 10:05:18,584 - INFO - i-d7086e1f (bld-linux64-spot-428) 2015-08-04 10:05:18,584 - INFO - i-afa30c6a (bld-linux64-spot-361) 2015-08-04 10:05:18,584 - INFO - i-3e832cfb (try-linux64-spot-1093) 2015-08-04 10:05:18,584 - INFO - i-ce58f407 (bld-linux64-spot-458) 2015-08-04 10:05:18,585 - INFO - i-b1a30c74 (bld-linux64-spot-1059) 2015-08-04 10:05:18,585 - INFO - i-400aa689 (try-linux64-spot-423) 2015-08-04 10:05:18,585 - INFO - i-a60ba76f (try-linux64-spot-340) 2015-08-04 10:05:18,585 - INFO - i-3e278bf7 (try-linux64-spot-386) 2015-08-04 10:05:18,585 - INFO - i-56842b93 (try-linux64-spot-422) 2015-08-04 10:05:18,585 - INFO - i-db600613 (bld-linux64-spot-302) 2015-08-04 10:05:18,585 - INFO - i-76238fbf (try-linux64-spot-403) 2015-08-04 10:05:18,585 - INFO - i-565ff39f (bld-linux64-spot-319) 2015-08-04 10:05:18,585 - INFO - i-df086e17 (try-linux64-spot-433) 2015-08-04 10:05:18,585 - INFO - i-e407ab2d (try-linux64-spot-1083) 2015-08-04 10:05:18,585 - INFO - i-daa20d1f (bld-linux64-spot-360) 2015-08-04 10:05:18,585 - INFO - i-105df1d9 (bld-linux64-spot-532) 2015-08-04 10:05:18,586 - INFO - i-5b3a9692 (try-linux64-spot-481) 2015-08-04 10:05:18,586 - INFO - i-3aa30cff (bld-linux64-spot-1053) 2015-08-04 10:05:18,586 - INFO - i-9b05a952 (try-linux64-spot-1080) 2015-08-04 10:05:18,586 - INFO - i-51258998 (try-linux64-spot-431) 2015-08-04 10:05:18,586 - INFO - i-5941ed90 (bld-linux64-spot-431) 2015-08-04 10:05:18,586 - INFO - i-e707ab2e (try-linux64-spot-1085) 2015-08-04 10:05:18,586 - INFO - i-a30ba76a (try-linux64-spot-497) 2015-08-04 10:05:18,586 - INFO - i-580ba791 (try-linux64-spot-364) 2015-08-04 10:05:18,586 - INFO - i-ab5ff362 (bld-linux64-spot-312) 2015-08-04 10:05:18,586 - INFO - i-bfa9067a (bld-linux64-spot-435) 2015-08-04 10:05:18,586 - INFO - i-3a42eef3 (bld-linux64-spot-506) 2015-08-04 10:05:18,586 - INFO - i-ee3a9627 (try-linux64-spot-485) 2015-08-04 10:05:18,586 - INFO - i-7aab04bf (bld-linux64-spot-1057) 2015-08-04 10:05:18,587 - INFO - i-5d04a894 (try-linux64-spot-362) 2015-08-04 10:05:18,587 - INFO - i-805df149 (bld-linux64-spot-518) 2015-08-04 10:05:18,587 - INFO - i-135df1da (try-linux64-spot-380) 2015-08-04 10:05:18,587 - INFO - i-3ea30cfb (bld-linux64-spot-383) 2015-08-04 10:05:18,587 - INFO - i-0d5df1c4 (bld-linux64-spot-531) 2015-08-04 10:05:18,587 - INFO - i-f8238f31 (try-linux64-spot-388) 2015-08-04 10:05:18,587 - INFO - i-5a0ba793 (try-linux64-spot-366) 2015-08-04 10:05:18,587 - INFO - i-76268abf (try-linux64-spot-300) 2015-08-04 10:05:18,587 - INFO - i-dd842b18 (try-linux64-spot-1097) 2015-08-04 10:05:18,587 - INFO - i-b2832c77 (try-linux64-spot-414) 2015-08-04 10:05:18,587 - INFO - i-59208c90 (try-linux64-spot-408) 2015-08-04 10:05:18,587 - INFO - i-0b0a6cc3 (bld-linux64-spot-498) 2015-08-04 10:05:18,588 - INFO - i-b7a80772 (bld-linux64-spot-377) 2015-08-04 10:05:18,588 - INFO - i-a6862963 (try-linux64-spot-1096) 2015-08-04 10:05:18,588 - INFO - i-400b6d88 (bld-linux64-spot-541) 2015-08-04 10:05:18,588 - INFO - i-e5aa0520 (bld-linux64-spot-305) 2015-08-04 10:05:18,588 - INFO - i-d9a20d1c (bld-linux64-spot-493) 2015-08-04 10:05:18,588 - INFO - i-d0086e18 (bld-linux64-spot-1096) 2015-08-04 10:05:18,588 - INFO - i-5f852a9a (try-linux64-spot-379) 2015-08-04 10:05:18,588 - INFO - i-9238945b (try-linux64-spot-308) 2015-08-04 10:05:18,588 - INFO - i-b8a9067d (bld-linux64-spot-304) 2015-08-04 10:05:18,588 - INFO - i-eb07ab22 (try-linux64-spot-1082) 2015-08-04 10:05:18,588 - INFO - i-75ab04b0 (bld-linux64-spot-336) 2015-08-04 10:05:18,588 - INFO - i-7f09a5b6 (try-linux64-spot-445) 2015-08-04 10:05:18,588 - INFO - i-410b6d89 (bld-linux64-spot-368) 2015-08-04 10:05:18,589 - INFO - i-420aa68b (try-linux64-spot-1090) 2015-08-04 10:05:18,589 - INFO - i-9c399555 (try-linux64-spot-482) 2015-08-04 10:05:18,589 - INFO - i-c958f400 (bld-linux64-spot-310) 2015-08-04 10:05:18,589 - INFO - i-6b5ef2a2 (try-linux64-spot-385) 2015-08-04 10:05:18,589 - INFO - i-a3228e6a (try-linux64-spot-405) 2015-08-04 10:05:18,589 - INFO - i-df842b1a (try-linux64-spot-411) 2015-08-04 10:05:18,589 - INFO - i-5e04a897 (try-linux64-spot-425) 2015-08-04 10:05:18,589 - INFO - i-a542ee6c (bld-linux64-spot-1054) 2015-08-04 10:05:18,589 - INFO - i-51862994 (try-linux64-spot-419) 2015-08-04 10:05:18,589 - INFO - i-d8389411 (try-linux64-spot-376) 2015-08-04 10:05:18,589 - INFO - i-d1086e19 (bld-linux64-spot-421) 2015-08-04 10:05:18,589 - INFO - i-05238fcc (try-linux64-spot-404) 2015-08-04 10:05:18,590 - INFO - i-4b238f82 (try-linux64-spot-496) 2015-08-04 10:05:18,590 - INFO - i-195df1d0 (bld-linux64-spot-439) 2015-08-04 10:05:18,590 - INFO - i-490b6d81 (bld-linux64-spot-560) 2015-08-04 10:05:18,590 - INFO - i-bd208c74 (try-linux64-spot-361) 2015-08-04 10:05:18,590 - INFO - i-4c5ef285 (try-linux64-spot-436) 2015-08-04 10:05:18,590 - INFO - i-9c05a955 (try-linux64-spot-1087) 2015-08-04 10:05:18,590 - INFO - i-d8842b1d (try-linux64-spot-418) 2015-08-04 10:05:18,590 - INFO - i-a00ba769 (try-linux64-spot-495) 2015-08-04 10:05:18,590 - INFO - i-ae0fa367 (try-linux64-spot-1091) 2015-08-04 10:05:18,590 - INFO - i-1f2589d6 (try-linux64-spot-303) 2015-08-04 10:05:18,590 - INFO - i-f00b6d38 (bld-linux64-spot-598) 2015-08-04 10:05:18,590 - INFO - i-3b42eef2 (bld-linux64-spot-534) 2015-08-04 10:05:18,590 - INFO - i-9d05a954 (try-linux64-spot-440) 2015-08-04 10:05:18,591 - INFO - i-b3a80776 (bld-linux64-spot-1051) 2015-08-04 10:05:18,591 - INFO - i-5d0ba794 (try-linux64-spot-360) 2015-08-04 10:05:18,591 - INFO - i-1004a8d9 (try-linux64-spot-1081) 2015-08-04 10:05:18,591 - INFO - i-99399550 (try-linux64-spot-373) 2015-08-04 10:05:18,591 - INFO - i-baa20d7f (bld-linux64-spot-491) 2015-08-04 10:05:18,591 - INFO - i-8b238f42 (try-linux64-spot-371) 2015-08-04 10:05:18,591 - INFO - i-79ab04bc (bld-linux64-spot-592) 2015-08-04 10:05:18,591 - INFO - i-b8a20d7d (bld-linux64-spot-373) 2015-08-04 10:05:18,591 - INFO - i-5b0ba792 (try-linux64-spot-426) 2015-08-04 10:05:18,591 - INFO - i-cc04a805 (try-linux64-spot-365) 2015-08-04 10:05:18,591 - INFO - i-5a04a893 (try-linux64-spot-1088) 2015-08-04 10:05:18,592 - INFO - i-070a6ccf (bld-linux64-spot-422) 2015-08-04 10:05:18,592 - INFO - i-2d3f93e4 (try-linux64-spot-367) 2015-08-04 10:05:18,592 - INFO - i-1f3894d6 (try-linux64-spot-486) 2015-08-04 10:05:18,592 - INFO - i-fc42ee35 (bld-linux64-spot-432) 2015-08-04 10:05:18,592 - INFO - i-17a20dd2 (bld-linux64-spot-1090) 2015-08-04 10:05:18,592 - INFO - i-c9086e01 (bld-linux64-spot-547) 2015-08-04 10:05:18,592 - INFO - i-cb086e03 (bld-linux64-spot-499) 2015-08-04 10:05:18,592 - INFO - i-4f5ff386 (bld-linux64-spot-313) 2015-08-04 10:05:18,592 - INFO - i-a85ff361 (try-linux64-spot-389) 2015-08-04 10:05:18,592 - INFO - i-0aa20dcf (bld-linux64-spot-388) 2015-08-04 10:05:18,592 - INFO - i-523a969b (try-linux64-spot-372) 2015-08-04 10:05:18,592 - INFO - i-0f238fc6 (try-linux64-spot-400) 2015-08-04 10:05:18,593 - INFO - i-dba20d1e (bld-linux64-spot-362) 2015-08-04 10:05:18,593 - INFO - i-cf0b6d07 (bld-linux64-spot-540) 2015-08-04 10:05:18,593 - INFO - i-f70b6d3f (bld-linux64-spot-543) 2015-08-04 10:05:18,593 - INFO - i-c90b6d01 (bld-linux64-spot-429) 2015-08-04 10:05:18,593 - INFO - i-b338947a (try-linux64-spot-378) 2015-08-04 10:05:18,593 - INFO - i-042589cd (try-linux64-spot-336) 2015-08-04 10:05:18,593 - INFO - i-74ab04b1 (bld-linux64-spot-1067) 2015-08-04 10:05:18,593 - INFO - i-515ff398 (bld-linux64-spot-328) 2015-08-04 10:05:18,593 - INFO - i-a208a46b (try-linux64-spot-416) 2015-08-04 10:05:18,593 - INFO - i-c258f40b (bld-linux64-spot-1060) 2015-08-04 10:05:18,593 - INFO - i-e2842b27 (try-linux64-spot-1098) 2015-08-04 10:05:18,593 - INFO - i-e3842b26 (try-linux64-spot-410) 2015-08-04 10:05:18,594 - INFO - i-aa208c63 (try-linux64-spot-382) 2015-08-04 10:05:18,594 - INFO - i-5641ed9f (bld-linux64-spot-535) 2015-08-04 10:05:18,594 - INFO - i-31a30cf4 (bld-linux64-spot-365) 2015-08-04 10:05:18,594 - INFO - i-d2086e1a (bld-linux64-spot-566) 2015-08-04 10:05:18,594 - INFO - i-71aa05b4 (bld-linux64-spot-558) 2015-08-04 10:05:18,594 - INFO - i-4b086e83 (bld-linux64-spot-427) 2015-08-04 10:05:18,594 - INFO - i-02238fcb (try-linux64-spot-407) 2015-08-04 10:05:18,594 - INFO - i-c042ee09 (bld-linux64-spot-433) 2015-08-04 10:05:18,594 - INFO - i-a10ba768 (try-linux64-spot-427) 2015-08-04 10:05:18,594 - INFO - i-b4268a7d (try-linux64-spot-435) 2015-08-04 10:05:18,594 - INFO - i-a40ba76d (try-linux64-spot-491) 2015-08-04 10:05:18,594 - INFO - i-ea3a9623 (try-linux64-spot-483) 2015-08-04 10:05:18,594 - INFO - i-460b6d8e (bld-linux64-spot-423) 2015-08-04 10:05:18,595 - INFO - i-5b208c92 (try-linux64-spot-413) 2015-08-04 10:05:18,595 - INFO - i-5225899b (try-linux64-spot-307) 2015-08-04 10:05:18,595 - INFO - i-75238fbc (try-linux64-spot-469) 2015-08-04 10:05:18,595 - INFO - i-4f5ef286 (try-linux64-spot-306) 2015-08-04 10:05:18,595 - INFO - i-1e3894d7 (try-linux64-spot-487) 2015-08-04 10:05:18,595 - INFO - i-4a5ef283 (bld-linux64-spot-537) 2015-08-04 10:05:18,595 - INFO - i-c85ef201 (bld-linux64-spot-438) 2015-08-04 10:05:18,595 - INFO - i-4b5ff382 (try-linux64-spot-363) 2015-08-04 10:05:18,595 - INFO - i-5c04a895 (try-linux64-spot-368) 2015-08-04 10:05:18,595 - INFO - i-38a20dfd (bld-linux64-spot-490) 2015-08-04 10:05:18,595 - INFO - i-ee06aa27 (try-linux64-spot-1084) 2015-08-04 10:05:18,595 - INFO - i-505ff399 (bld-linux64-spot-510) 2015-08-04 10:05:18,595 - INFO - i-4e5ff387 (bld-linux64-spot-492) 2015-08-04 10:05:18,596 - INFO - i-090a6cc1 (bld-linux64-spot-544) 2015-08-04 10:05:18,596 - INFO - i-cb0b6d03 (bld-linux64-spot-546) 2015-08-04 10:05:18,596 - INFO - i-aaa30c6f (bld-linux64-spot-1094) 2015-08-04 10:05:18,596 - INFO - i-d6218d1f (try-linux64-spot-383) 2015-08-04 10:05:18,596 - INFO - i-e607ab2f (try-linux64-spot-443) 2015-08-04 10:05:18,596 - INFO - i-65238fac (try-linux64-spot-401) 2015-08-04 10:05:18,596 - INFO - i-d4086e1c (bld-linux64-spot-1068) 2015-08-04 10:05:18,596 - INFO - i-efa9062a (bld-linux64-spot-1083) 2015-08-04 10:05:18,596 - INFO - i-85309c4c (bld-linux64-spot-315) 2015-08-04 10:05:18,596 - INFO - i-0c5df1c5 (bld-linux64-spot-536) 2015-08-04 10:05:18,596 - INFO - i-6a5ef2a3 (try-linux64-spot-384) 2015-08-04 10:05:18,596 - INFO - i-b6a80773 (bld-linux64-spot-434) 2015-08-04 10:05:18,596 - INFO - i-9a05a953 (try-linux64-spot-1089) 2015-08-04 10:05:18,597 - INFO - i-c50b6d0d (bld-linux64-spot-337) 2015-08-04 10:05:18,597 - INFO - i-3da30cf8 (bld-linux64-spot-363) 2015-08-04 10:05:18,597 - INFO - i-7e268ab7 (try-linux64-spot-434) 2015-08-04 10:05:18,597 - INFO - i-1704a8de (try-linux64-spot-442) 2015-08-04 10:05:18,597 - INFO - i-c858f401 (bld-linux64-spot-318) 2015-08-04 10:05:18,597 - INFO - i-125df1db (try-linux64-spot-304) 2015-08-04 10:05:18,597 - INFO - i-708629b5 (try-linux64-spot-1095) 2015-08-04 10:05:18,597 - INFO - i-b0a20d75 (bld-linux64-spot-379) 2015-08-04 10:05:18,597 - INFO - i-5b04a892 (try-linux64-spot-369) 2015-08-04 10:05:18,597 - INFO - i-c0389409 (try-linux64-spot-375) 2015-08-04 10:05:18,597 - INFO - i-d7218d1e (try-linux64-spot-409) 2015-08-04 10:05:18,598 - INFO - i-dc852a19 (try-linux64-spot-488) 2015-08-04 10:05:18,598 - INFO - i-020a6cca (bld-linux64-spot-548) 2015-08-04 10:05:18,598 - INFO - i-c5218d0c (try-linux64-spot-492) 2015-08-04 10:05:18,598 - INFO - i-ada30c68 (bld-linux64-spot-1055) 2015-08-04 10:05:18,598 - INFO - i-70aa05b5 (bld-linux64-spot-1066) 2015-08-04 10:05:18,598 - INFO - i-a75df16e (bld-linux64-spot-530) 2015-08-04 10:05:18,598 - INFO - i-5a3a9693 (try-linux64-spot-370) 2015-08-04 10:05:18,598 - INFO - i-5c852a99 (try-linux64-spot-415) 2015-08-04 10:05:18,598 - INFO - i-34278bfd (try-linux64-spot-406) 2015-08-04 10:05:18,598 - INFO - i-b238947b (try-linux64-spot-480) 2015-08-04 10:05:18,598 - INFO - i-eca90629 (bld-linux64-spot-419) 2015-08-04 10:05:18,598 - INFO - i-e93a9620 (try-linux64-spot-374) 2015-08-04 10:05:18,598 - INFO - i-d6852a13 (try-linux64-spot-316) 2015-08-04 10:05:18,599 - INFO - i-3ba20dfe (try-linux64-spot-301) 2015-08-04 10:05:18,599 - INFO - i-46319d8f (bld-linux64-spot-314) 2015-08-04 10:05:18,599 - INFO - i-555ff39c (bld-linux64-spot-311) 2015-08-04 10:05:18,599 - INFO - i-d5086e1d (bld-linux64-spot-549) 2015-08-04 10:05:18,599 - INFO - i-b2a80777 (bld-linux64-spot-1058) 2015-08-04 10:05:18,599 - INFO - i-513a9698 (try-linux64-spot-377) 2015-08-04 10:05:18,599 - INFO - i-430b6d8b (bld-linux64-spot-420) 2015-08-04 10:05:18,599 - INFO - i-b5268a7c (try-linux64-spot-430) 2015-08-04 10:05:18,599 - INFO - i-53208c9a (try-linux64-spot-302) 2015-08-04 10:05:18,599 - INFO - i-bea9067b (bld-linux64-spot-1061) 2015-08-04 10:05:18,599 - INFO - i-58842b9d (try-linux64-spot-1092) 2015-08-04 10:05:18,599 - INFO - i-9905a950 (try-linux64-spot-1086) 2015-08-04 10:05:18,599 - INFO - i-e507ab2c (try-linux64-spot-466) 2015-08-04 10:05:18,600 - INFO - i-733995ba (try-linux64-spot-484) 2015-08-04 10:05:18,600 - INFO - i-e4842b21 (try-linux64-spot-412) Are you sure you want to kill these? ^ y/N >y ARE YOU SURE YOU WANT TO KILL THESE? ^ LAST WARNING!!! y/N >y 2015-08-04 10:05:24,102 - WARNING - The instances mentioned above are about to be terminated 2015-08-04 10:05:24,102 - WARNING - Waiting extra 60 seconds to make sure... 2015-08-04 10:06:24,108 - WARNING - Starting... 2015-08-04 10:06:24,109 - WARNING - Terminating Instance:i-c1389408... 2015-08-04 10:06:24,419 - WARNING - Done. 2015-08-04 10:06:24,419 - WARNING - Terminating Instance:i-e7842b22... 2015-08-04 10:06:24,660 - WARNING - Done. 2015-08-04 10:06:24,660 - WARNING - Terminating Instance:i-a50ba76c... 2015-08-04 10:06:24,871 - WARNING - Done. 2015-08-04 10:06:24,871 - WARNING - Terminating Instance:i-fa0b6d32... 2015-08-04 10:06:25,111 - WARNING - Done. 2015-08-04 10:06:25,112 - WARNING - Terminating Instance:i-f20b6d3a... 2015-08-04 10:06:25,292 - WARNING - Done. 2015-08-04 10:06:25,292 - WARNING - Terminating Instance:i-450b6d8d... 2015-08-04 10:06:25,500 - WARNING - Done. 2015-08-04 10:06:25,500 - WARNING - Terminating Instance:i-7a0b6db2... 2015-08-04 10:06:25,730 - WARNING - Done. 2015-08-04 10:06:25,730 - WARNING - Terminating Instance:i-b5a80770... 2015-08-04 10:06:25,951 - WARNING - Done. 2015-08-04 10:06:25,952 - WARNING - Terminating Instance:i-aa5ff363... 2015-08-04 10:06:26,192 - WARNING - Done. 2015-08-04 10:06:26,192 - WARNING - Terminating Instance:i-b4a80771... 2015-08-04 10:06:26,404 - WARNING - Done. 2015-08-04 10:06:26,404 - WARNING - Terminating Instance:i-fb238f32... 2015-08-04 10:06:26,630 - WARNING - Done. 2015-08-04 10:06:26,630 - WARNING - Terminating Instance:i-dfa20d1a... 2015-08-04 10:06:26,853 - WARNING - Done. 2015-08-04 10:06:26,853 - WARNING - Terminating Instance:i-62268aab... 2015-08-04 10:06:27,090 - WARNING - Done. 2015-08-04 10:06:27,090 - WARNING - Terminating Instance:i-4f0b6d87... 2015-08-04 10:06:27,286 - WARNING - Done. 2015-08-04 10:06:27,286 - WARNING - Terminating Instance:i-a55ff36c... 2015-08-04 10:06:27,518 - WARNING - Done. 2015-08-04 10:06:27,519 - WARNING - Terminating Instance:i-778629b2... 2015-08-04 10:06:27,750 - WARNING - Done. 2015-08-04 10:06:27,750 - WARNING - Terminating Instance:i-39278bf0... 2015-08-04 10:06:27,979 - WARNING - Done. 2015-08-04 10:06:27,980 - WARNING - Terminating Instance:i-385ff3f1... 2015-08-04 10:06:28,194 - WARNING - Done. 2015-08-04 10:06:28,195 - WARNING - Terminating Instance:i-cc0b6d04... 2015-08-04 10:06:28,402 - WARNING - Done. 2015-08-04 10:06:28,402 - WARNING - Terminating Instance:i-b0a80775... 2015-08-04 10:06:28,622 - WARNING - Done. 2015-08-04 10:06:28,622 - WARNING - Terminating Instance:i-420b6d8a... 2015-08-04 10:06:28,814 - WARNING - Done. 2015-08-04 10:06:28,814 - WARNING - Terminating Instance:i-7bab04be... 2015-08-04 10:06:29,039 - WARNING - Done. 2015-08-04 10:06:29,039 - WARNING - Terminating Instance:i-010aa6c8... 2015-08-04 10:06:29,260 - WARNING - Done. 2015-08-04 10:06:29,260 - WARNING - Terminating Instance:i-56852a93... 2015-08-04 10:06:29,479 - WARNING - Done. 2015-08-04 10:06:29,479 - WARNING - Terminating Instance:i-ca086e02... 2015-08-04 10:06:29,703 - WARNING - Done. 2015-08-04 10:06:29,703 - WARNING - Terminating Instance:i-8704a84e... 2015-08-04 10:06:30,056 - WARNING - Done. 2015-08-04 10:06:30,056 - WARNING - Terminating Instance:i-150fa3dc... 2015-08-04 10:06:30,284 - WARNING - Done. 2015-08-04 10:06:30,284 - WARNING - Terminating Instance:i-080a6cc0... 2015-08-04 10:06:30,479 - WARNING - Done. 2015-08-04 10:06:30,479 - WARNING - Terminating Instance:i-d7086e1f... 2015-08-04 10:06:30,813 - WARNING - Done. 2015-08-04 10:06:30,813 - WARNING - Terminating Instance:i-afa30c6a... 2015-08-04 10:06:31,025 - WARNING - Done. 2015-08-04 10:06:31,025 - WARNING - Terminating Instance:i-3e832cfb... 2015-08-04 10:06:31,232 - WARNING - Done. 2015-08-04 10:06:31,232 - WARNING - Terminating Instance:i-ce58f407... 2015-08-04 10:06:31,618 - WARNING - Done. 2015-08-04 10:06:31,618 - WARNING - Terminating Instance:i-b1a30c74... 2015-08-04 10:06:31,936 - WARNING - Done. 2015-08-04 10:06:31,936 - WARNING - Terminating Instance:i-400aa689... 2015-08-04 10:06:32,226 - WARNING - Done. 2015-08-04 10:06:32,227 - WARNING - Terminating Instance:i-a60ba76f... 2015-08-04 10:06:32,466 - WARNING - Done. 2015-08-04 10:06:32,466 - WARNING - Terminating Instance:i-3e278bf7... 2015-08-04 10:06:32,721 - WARNING - Done. 2015-08-04 10:06:32,721 - WARNING - Terminating Instance:i-56842b93... 2015-08-04 10:06:32,996 - WARNING - Done. 2015-08-04 10:06:32,996 - WARNING - Terminating Instance:i-db600613... 2015-08-04 10:06:33,231 - WARNING - Done. 2015-08-04 10:06:33,232 - WARNING - Terminating Instance:i-76238fbf... 2015-08-04 10:06:33,547 - WARNING - Done. 2015-08-04 10:06:33,547 - WARNING - Terminating Instance:i-565ff39f... 2015-08-04 10:06:33,798 - WARNING - Done. 2015-08-04 10:06:33,798 - WARNING - Terminating Instance:i-df086e17... 2015-08-04 10:06:34,056 - WARNING - Done. 2015-08-04 10:06:34,056 - WARNING - Terminating Instance:i-e407ab2d... 2015-08-04 10:06:34,289 - WARNING - Done. 2015-08-04 10:06:34,289 - WARNING - Terminating Instance:i-daa20d1f... 2015-08-04 10:06:34,526 - WARNING - Done. 2015-08-04 10:06:34,526 - WARNING - Terminating Instance:i-105df1d9... 2015-08-04 10:06:34,786 - WARNING - Done. 2015-08-04 10:06:34,786 - WARNING - Terminating Instance:i-5b3a9692... 2015-08-04 10:06:35,131 - WARNING - Done. 2015-08-04 10:06:35,131 - WARNING - Terminating Instance:i-3aa30cff... 2015-08-04 10:06:35,342 - WARNING - Done. 2015-08-04 10:06:35,343 - WARNING - Terminating Instance:i-9b05a952... 2015-08-04 10:06:35,595 - WARNING - Done. 2015-08-04 10:06:35,595 - WARNING - Terminating Instance:i-51258998... 2015-08-04 10:06:35,858 - WARNING - Done. 2015-08-04 10:06:35,858 - WARNING - Terminating Instance:i-5941ed90... 2015-08-04 10:06:36,076 - WARNING - Done. 2015-08-04 10:06:36,077 - WARNING - Terminating Instance:i-e707ab2e... 2015-08-04 10:06:36,394 - WARNING - Done. 2015-08-04 10:06:36,394 - WARNING - Terminating Instance:i-a30ba76a... 2015-08-04 10:06:36,650 - WARNING - Done. 2015-08-04 10:06:36,650 - WARNING - Terminating Instance:i-580ba791... 2015-08-04 10:06:36,894 - WARNING - Done. 2015-08-04 10:06:36,894 - WARNING - Terminating Instance:i-ab5ff362... 2015-08-04 10:06:37,109 - WARNING - Done. 2015-08-04 10:06:37,109 - WARNING - Terminating Instance:i-bfa9067a... 2015-08-04 10:06:37,320 - WARNING - Done. 2015-08-04 10:06:37,320 - WARNING - Terminating Instance:i-3a42eef3... 2015-08-04 10:06:37,554 - WARNING - Done. 2015-08-04 10:06:37,554 - WARNING - Terminating Instance:i-ee3a9627... 2015-08-04 10:06:37,776 - WARNING - Done. 2015-08-04 10:06:37,776 - WARNING - Terminating Instance:i-7aab04bf... 2015-08-04 10:06:38,134 - WARNING - Done. 2015-08-04 10:06:38,134 - WARNING - Terminating Instance:i-5d04a894... 2015-08-04 10:06:38,361 - WARNING - Done. 2015-08-04 10:06:38,361 - WARNING - Terminating Instance:i-805df149... 2015-08-04 10:06:38,563 - WARNING - Done. 2015-08-04 10:06:38,564 - WARNING - Terminating Instance:i-135df1da... 2015-08-04 10:06:38,860 - WARNING - Done. 2015-08-04 10:06:38,860 - WARNING - Terminating Instance:i-3ea30cfb... 2015-08-04 10:06:39,106 - WARNING - Done. 2015-08-04 10:06:39,107 - WARNING - Terminating Instance:i-0d5df1c4... 2015-08-04 10:06:39,330 - WARNING - Done. 2015-08-04 10:06:39,331 - WARNING - Terminating Instance:i-f8238f31... 2015-08-04 10:06:39,625 - WARNING - Done. 2015-08-04 10:06:39,625 - WARNING - Terminating Instance:i-5a0ba793... 2015-08-04 10:06:39,969 - WARNING - Done. 2015-08-04 10:06:39,970 - WARNING - Terminating Instance:i-76268abf... 2015-08-04 10:06:40,228 - WARNING - Done. 2015-08-04 10:06:40,228 - WARNING - Terminating Instance:i-dd842b18... 2015-08-04 10:06:40,447 - WARNING - Done. 2015-08-04 10:06:40,447 - WARNING - Terminating Instance:i-b2832c77... 2015-08-04 10:06:40,778 - WARNING - Done. 2015-08-04 10:06:40,779 - WARNING - Terminating Instance:i-59208c90... 2015-08-04 10:06:41,118 - WARNING - Done. 2015-08-04 10:06:41,118 - WARNING - Terminating Instance:i-0b0a6cc3... 2015-08-04 10:06:41,330 - WARNING - Done. 2015-08-04 10:06:41,330 - WARNING - Terminating Instance:i-b7a80772... 2015-08-04 10:06:41,575 - WARNING - Done. 2015-08-04 10:06:41,575 - WARNING - Terminating Instance:i-a6862963... 2015-08-04 10:06:41,839 - WARNING - Done. 2015-08-04 10:06:41,840 - WARNING - Terminating Instance:i-400b6d88... 2015-08-04 10:06:42,158 - WARNING - Done. 2015-08-04 10:06:42,159 - WARNING - Terminating Instance:i-e5aa0520... 2015-08-04 10:06:42,416 - WARNING - Done. 2015-08-04 10:06:42,416 - WARNING - Terminating Instance:i-d9a20d1c... 2015-08-04 10:06:42,707 - WARNING - Done. 2015-08-04 10:06:42,707 - WARNING - Terminating Instance:i-d0086e18... 2015-08-04 10:06:42,921 - WARNING - Done. 2015-08-04 10:06:42,921 - WARNING - Terminating Instance:i-5f852a9a... 2015-08-04 10:06:43,193 - WARNING - Done. 2015-08-04 10:06:43,194 - WARNING - Terminating Instance:i-9238945b... 2015-08-04 10:06:43,416 - WARNING - Done. 2015-08-04 10:06:43,417 - WARNING - Terminating Instance:i-b8a9067d... 2015-08-04 10:06:43,647 - WARNING - Done. 2015-08-04 10:06:43,647 - WARNING - Terminating Instance:i-eb07ab22... 2015-08-04 10:06:43,852 - WARNING - Done. 2015-08-04 10:06:43,852 - WARNING - Terminating Instance:i-75ab04b0... 2015-08-04 10:06:44,069 - WARNING - Done. 2015-08-04 10:06:44,069 - WARNING - Terminating Instance:i-7f09a5b6... 2015-08-04 10:06:44,289 - WARNING - Done. 2015-08-04 10:06:44,289 - WARNING - Terminating Instance:i-410b6d89... 2015-08-04 10:06:44,593 - WARNING - Done. 2015-08-04 10:06:44,593 - WARNING - Terminating Instance:i-420aa68b... 2015-08-04 10:06:44,979 - WARNING - Done. 2015-08-04 10:06:44,979 - WARNING - Terminating Instance:i-9c399555... 2015-08-04 10:06:45,281 - WARNING - Done. 2015-08-04 10:06:45,281 - WARNING - Terminating Instance:i-c958f400... 2015-08-04 10:06:45,484 - WARNING - Done. 2015-08-04 10:06:45,484 - WARNING - Terminating Instance:i-6b5ef2a2... 2015-08-04 10:06:45,701 - WARNING - Done. 2015-08-04 10:06:45,701 - WARNING - Terminating Instance:i-a3228e6a... 2015-08-04 10:06:45,921 - WARNING - Done. 2015-08-04 10:06:45,922 - WARNING - Terminating Instance:i-df842b1a... 2015-08-04 10:06:46,164 - WARNING - Done. 2015-08-04 10:06:46,164 - WARNING - Terminating Instance:i-5e04a897... 2015-08-04 10:06:46,372 - WARNING - Done. 2015-08-04 10:06:46,372 - WARNING - Terminating Instance:i-a542ee6c... 2015-08-04 10:06:46,589 - WARNING - Done. 2015-08-04 10:06:46,589 - WARNING - Terminating Instance:i-51862994... 2015-08-04 10:06:46,804 - WARNING - Done. 2015-08-04 10:06:46,804 - WARNING - Terminating Instance:i-d8389411... 2015-08-04 10:06:47,026 - WARNING - Done. 2015-08-04 10:06:47,026 - WARNING - Terminating Instance:i-d1086e19... 2015-08-04 10:06:47,247 - WARNING - Done. 2015-08-04 10:06:47,247 - WARNING - Terminating Instance:i-05238fcc... 2015-08-04 10:06:47,517 - WARNING - Done. 2015-08-04 10:06:47,517 - WARNING - Terminating Instance:i-4b238f82... 2015-08-04 10:06:47,745 - WARNING - Done. 2015-08-04 10:06:47,746 - WARNING - Terminating Instance:i-195df1d0... 2015-08-04 10:06:48,075 - WARNING - Done. 2015-08-04 10:06:48,075 - WARNING - Terminating Instance:i-490b6d81... 2015-08-04 10:06:48,380 - WARNING - Done. 2015-08-04 10:06:48,380 - WARNING - Terminating Instance:i-bd208c74... 2015-08-04 10:06:49,167 - WARNING - Done. 2015-08-04 10:06:49,167 - WARNING - Terminating Instance:i-4c5ef285... 2015-08-04 10:06:49,415 - WARNING - Done. 2015-08-04 10:06:49,415 - WARNING - Terminating Instance:i-9c05a955... 2015-08-04 10:06:49,677 - WARNING - Done. 2015-08-04 10:06:49,677 - WARNING - Terminating Instance:i-d8842b1d... 2015-08-04 10:06:49,900 - WARNING - Done. 2015-08-04 10:06:49,900 - WARNING - Terminating Instance:i-a00ba769... 2015-08-04 10:06:50,114 - WARNING - Done. 2015-08-04 10:06:50,114 - WARNING - Terminating Instance:i-ae0fa367... 2015-08-04 10:06:50,345 - WARNING - Done. 2015-08-04 10:06:50,345 - WARNING - Terminating Instance:i-1f2589d6... 2015-08-04 10:06:50,573 - WARNING - Done. 2015-08-04 10:06:50,573 - WARNING - Terminating Instance:i-f00b6d38... 2015-08-04 10:06:50,783 - WARNING - Done. 2015-08-04 10:06:50,783 - WARNING - Terminating Instance:i-3b42eef2... 2015-08-04 10:06:51,000 - WARNING - Done. 2015-08-04 10:06:51,000 - WARNING - Terminating Instance:i-9d05a954... 2015-08-04 10:06:51,232 - WARNING - Done. 2015-08-04 10:06:51,232 - WARNING - Terminating Instance:i-b3a80776... 2015-08-04 10:06:51,421 - WARNING - Done. 2015-08-04 10:06:51,421 - WARNING - Terminating Instance:i-5d0ba794... 2015-08-04 10:06:51,637 - WARNING - Done. 2015-08-04 10:06:51,638 - WARNING - Terminating Instance:i-1004a8d9... 2015-08-04 10:06:51,845 - WARNING - Done. 2015-08-04 10:06:51,845 - WARNING - Terminating Instance:i-99399550... 2015-08-04 10:06:52,060 - WARNING - Done. 2015-08-04 10:06:52,061 - WARNING - Terminating Instance:i-baa20d7f... 2015-08-04 10:06:52,289 - WARNING - Done. 2015-08-04 10:06:52,289 - WARNING - Terminating Instance:i-8b238f42... 2015-08-04 10:06:52,499 - WARNING - Done. 2015-08-04 10:06:52,499 - WARNING - Terminating Instance:i-79ab04bc... 2015-08-04 10:06:52,738 - WARNING - Done. 2015-08-04 10:06:52,738 - WARNING - Terminating Instance:i-b8a20d7d... 2015-08-04 10:06:52,937 - WARNING - Done. 2015-08-04 10:06:52,937 - WARNING - Terminating Instance:i-5b0ba792... 2015-08-04 10:06:53,166 - WARNING - Done. 2015-08-04 10:06:53,166 - WARNING - Terminating Instance:i-cc04a805... 2015-08-04 10:06:53,576 - WARNING - Done. 2015-08-04 10:06:53,576 - WARNING - Terminating Instance:i-5a04a893... 2015-08-04 10:06:53,802 - WARNING - Done. 2015-08-04 10:06:53,802 - WARNING - Terminating Instance:i-070a6ccf... 2015-08-04 10:06:54,023 - WARNING - Done. 2015-08-04 10:06:54,023 - WARNING - Terminating Instance:i-2d3f93e4... 2015-08-04 10:06:54,237 - WARNING - Done. 2015-08-04 10:06:54,237 - WARNING - Terminating Instance:i-1f3894d6... 2015-08-04 10:06:54,502 - WARNING - Done. 2015-08-04 10:06:54,502 - WARNING - Terminating Instance:i-fc42ee35... 2015-08-04 10:06:54,736 - WARNING - Done. 2015-08-04 10:06:54,736 - WARNING - Terminating Instance:i-17a20dd2... 2015-08-04 10:06:54,970 - WARNING - Done. 2015-08-04 10:06:54,970 - WARNING - Terminating Instance:i-c9086e01... 2015-08-04 10:06:55,187 - WARNING - Done. 2015-08-04 10:06:55,187 - WARNING - Terminating Instance:i-cb086e03... 2015-08-04 10:06:55,388 - WARNING - Done. 2015-08-04 10:06:55,388 - WARNING - Terminating Instance:i-4f5ff386... 2015-08-04 10:06:55,598 - WARNING - Done. 2015-08-04 10:06:55,598 - WARNING - Terminating Instance:i-a85ff361... 2015-08-04 10:06:55,884 - WARNING - Done. 2015-08-04 10:06:55,884 - WARNING - Terminating Instance:i-0aa20dcf... 2015-08-04 10:06:56,121 - WARNING - Done. 2015-08-04 10:06:56,121 - WARNING - Terminating Instance:i-523a969b... 2015-08-04 10:06:56,390 - WARNING - Done. 2015-08-04 10:06:56,390 - WARNING - Terminating Instance:i-0f238fc6... 2015-08-04 10:06:56,608 - WARNING - Done. 2015-08-04 10:06:56,608 - WARNING - Terminating Instance:i-dba20d1e... 2015-08-04 10:06:56,834 - WARNING - Done. 2015-08-04 10:06:56,835 - WARNING - Terminating Instance:i-cf0b6d07... 2015-08-04 10:06:57,035 - WARNING - Done. 2015-08-04 10:06:57,035 - WARNING - Terminating Instance:i-f70b6d3f... 2015-08-04 10:06:57,216 - WARNING - Done. 2015-08-04 10:06:57,217 - WARNING - Terminating Instance:i-c90b6d01... 2015-08-04 10:06:57,410 - WARNING - Done. 2015-08-04 10:06:57,410 - WARNING - Terminating Instance:i-b338947a... 2015-08-04 10:06:57,623 - WARNING - Done. 2015-08-04 10:06:57,623 - WARNING - Terminating Instance:i-042589cd... 2015-08-04 10:06:57,835 - WARNING - Done. 2015-08-04 10:06:57,835 - WARNING - Terminating Instance:i-74ab04b1... 2015-08-04 10:06:58,058 - WARNING - Done. 2015-08-04 10:06:58,058 - WARNING - Terminating Instance:i-515ff398... 2015-08-04 10:06:58,268 - WARNING - Done. 2015-08-04 10:06:58,268 - WARNING - Terminating Instance:i-a208a46b... 2015-08-04 10:06:58,492 - WARNING - Done. 2015-08-04 10:06:58,493 - WARNING - Terminating Instance:i-c258f40b... 2015-08-04 10:06:58,706 - WARNING - Done. 2015-08-04 10:06:58,706 - WARNING - Terminating Instance:i-e2842b27... 2015-08-04 10:06:58,944 - WARNING - Done. 2015-08-04 10:06:58,944 - WARNING - Terminating Instance:i-e3842b26... 2015-08-04 10:07:00,093 - WARNING - Done. 2015-08-04 10:07:00,093 - WARNING - Terminating Instance:i-aa208c63... 2015-08-04 10:07:01,303 - WARNING - Done. 2015-08-04 10:07:01,303 - WARNING - Terminating Instance:i-5641ed9f... 2015-08-04 10:07:01,538 - WARNING - Done. 2015-08-04 10:07:01,538 - WARNING - Terminating Instance:i-31a30cf4... 2015-08-04 10:07:02,907 - WARNING - Done. 2015-08-04 10:07:02,907 - WARNING - Terminating Instance:i-d2086e1a... 2015-08-04 10:07:03,162 - WARNING - Done. 2015-08-04 10:07:03,162 - WARNING - Terminating Instance:i-71aa05b4... 2015-08-04 10:07:03,440 - WARNING - Done. 2015-08-04 10:07:03,440 - WARNING - Terminating Instance:i-4b086e83... 2015-08-04 10:07:03,843 - WARNING - Done. 2015-08-04 10:07:03,844 - WARNING - Terminating Instance:i-02238fcb... 2015-08-04 10:07:09,299 - WARNING - Done. 2015-08-04 10:07:09,299 - WARNING - Terminating Instance:i-c042ee09... 2015-08-04 10:07:09,617 - WARNING - Done. 2015-08-04 10:07:09,617 - WARNING - Terminating Instance:i-a10ba768... 2015-08-04 10:07:09,833 - WARNING - Done. 2015-08-04 10:07:09,833 - WARNING - Terminating Instance:i-b4268a7d... 2015-08-04 10:07:10,056 - WARNING - Done. 2015-08-04 10:07:10,057 - WARNING - Terminating Instance:i-a40ba76d... 2015-08-04 10:07:10,379 - WARNING - Done. 2015-08-04 10:07:10,379 - WARNING - Terminating Instance:i-ea3a9623... 2015-08-04 10:07:12,146 - WARNING - Done. 2015-08-04 10:07:12,146 - WARNING - Terminating Instance:i-460b6d8e... 2015-08-04 10:07:12,457 - WARNING - Done. 2015-08-04 10:07:12,457 - WARNING - Terminating Instance:i-5b208c92... 2015-08-04 10:07:12,673 - WARNING - Done. 2015-08-04 10:07:12,673 - WARNING - Terminating Instance:i-5225899b... 2015-08-04 10:07:12,967 - WARNING - Done. 2015-08-04 10:07:12,967 - WARNING - Terminating Instance:i-75238fbc... 2015-08-04 10:07:13,966 - WARNING - Done. 2015-08-04 10:07:13,966 - WARNING - Terminating Instance:i-4f5ef286... 2015-08-04 10:07:14,171 - WARNING - Done. 2015-08-04 10:07:14,171 - WARNING - Terminating Instance:i-1e3894d7... 2015-08-04 10:07:23,500 - WARNING - Done. 2015-08-04 10:07:23,501 - WARNING - Terminating Instance:i-4a5ef283... 2015-08-04 10:07:23,729 - WARNING - Done. 2015-08-04 10:07:23,730 - WARNING - Terminating Instance:i-c85ef201... 2015-08-04 10:07:23,952 - WARNING - Done. 2015-08-04 10:07:23,953 - WARNING - Terminating Instance:i-4b5ff382... 2015-08-04 10:07:24,208 - WARNING - Done. 2015-08-04 10:07:24,209 - WARNING - Terminating Instance:i-5c04a895... 2015-08-04 10:07:24,505 - WARNING - Done. 2015-08-04 10:07:24,505 - WARNING - Terminating Instance:i-38a20dfd... 2015-08-04 10:07:24,750 - WARNING - Done. 2015-08-04 10:07:24,750 - WARNING - Terminating Instance:i-ee06aa27... 2015-08-04 10:07:25,050 - WARNING - Done. 2015-08-04 10:07:25,051 - WARNING - Terminating Instance:i-505ff399... 2015-08-04 10:07:25,249 - WARNING - Done. 2015-08-04 10:07:25,249 - WARNING - Terminating Instance:i-4e5ff387... 2015-08-04 10:07:25,461 - WARNING - Done. 2015-08-04 10:07:25,461 - WARNING - Terminating Instance:i-090a6cc1... 2015-08-04 10:07:25,657 - WARNING - Done. 2015-08-04 10:07:25,657 - WARNING - Terminating Instance:i-cb0b6d03... 2015-08-04 10:07:25,838 - WARNING - Done. 2015-08-04 10:07:25,838 - WARNING - Terminating Instance:i-aaa30c6f... 2015-08-04 10:07:27,024 - WARNING - Done. 2015-08-04 10:07:27,024 - WARNING - Terminating Instance:i-d6218d1f... 2015-08-04 10:07:27,308 - WARNING - Done. 2015-08-04 10:07:27,309 - WARNING - Terminating Instance:i-e607ab2f... 2015-08-04 10:07:27,523 - WARNING - Done. 2015-08-04 10:07:27,523 - WARNING - Terminating Instance:i-65238fac... 2015-08-04 10:07:27,743 - WARNING - Done. 2015-08-04 10:07:27,743 - WARNING - Terminating Instance:i-d4086e1c... 2015-08-04 10:07:28,969 - WARNING - Done. 2015-08-04 10:07:28,969 - WARNING - Terminating Instance:i-efa9062a... 2015-08-04 10:07:29,167 - WARNING - Done. 2015-08-04 10:07:29,167 - WARNING - Terminating Instance:i-85309c4c... 2015-08-04 10:07:29,420 - WARNING - Done. 2015-08-04 10:07:29,420 - WARNING - Terminating Instance:i-0c5df1c5... 2015-08-04 10:07:29,658 - WARNING - Done. 2015-08-04 10:07:29,659 - WARNING - Terminating Instance:i-6a5ef2a3... 2015-08-04 10:07:30,368 - WARNING - Done. 2015-08-04 10:07:30,368 - WARNING - Terminating Instance:i-b6a80773... 2015-08-04 10:07:31,348 - WARNING - Done. 2015-08-04 10:07:31,348 - WARNING - Terminating Instance:i-9a05a953... 2015-08-04 10:07:31,618 - WARNING - Done. 2015-08-04 10:07:31,618 - WARNING - Terminating Instance:i-c50b6d0d... 2015-08-04 10:07:31,842 - WARNING - Done. 2015-08-04 10:07:31,842 - WARNING - Terminating Instance:i-3da30cf8... 2015-08-04 10:07:32,085 - WARNING - Done. 2015-08-04 10:07:32,085 - WARNING - Terminating Instance:i-7e268ab7... 2015-08-04 10:07:32,312 - WARNING - Done. 2015-08-04 10:07:32,312 - WARNING - Terminating Instance:i-1704a8de... 2015-08-04 10:07:36,325 - WARNING - Done. 2015-08-04 10:07:36,325 - WARNING - Terminating Instance:i-c858f401... 2015-08-04 10:07:36,545 - WARNING - Done. 2015-08-04 10:07:36,545 - WARNING - Terminating Instance:i-125df1db... 2015-08-04 10:07:37,234 - WARNING - Done. 2015-08-04 10:07:37,234 - WARNING - Terminating Instance:i-708629b5... 2015-08-04 10:07:37,468 - WARNING - Done. 2015-08-04 10:07:37,469 - WARNING - Terminating Instance:i-b0a20d75... 2015-08-04 10:07:37,692 - WARNING - Done. 2015-08-04 10:07:37,693 - WARNING - Terminating Instance:i-5b04a892... 2015-08-04 10:07:37,920 - WARNING - Done. 2015-08-04 10:07:37,920 - WARNING - Terminating Instance:i-c0389409... 2015-08-04 10:07:39,156 - WARNING - Done. 2015-08-04 10:07:39,157 - WARNING - Terminating Instance:i-d7218d1e... 2015-08-04 10:07:39,369 - WARNING - Done. 2015-08-04 10:07:39,369 - WARNING - Terminating Instance:i-dc852a19... 2015-08-04 10:07:39,603 - WARNING - Done. 2015-08-04 10:07:39,603 - WARNING - Terminating Instance:i-020a6cca... 2015-08-04 10:07:54,374 - WARNING - Done. 2015-08-04 10:07:54,375 - WARNING - Terminating Instance:i-c5218d0c... 2015-08-04 10:07:54,577 - WARNING - Done. 2015-08-04 10:07:54,577 - WARNING - Terminating Instance:i-ada30c68... 2015-08-04 10:07:54,796 - WARNING - Done. 2015-08-04 10:07:54,796 - WARNING - Terminating Instance:i-70aa05b5... 2015-08-04 10:07:55,064 - WARNING - Done. 2015-08-04 10:07:55,064 - WARNING - Terminating Instance:i-a75df16e... 2015-08-04 10:07:55,276 - WARNING - Done. 2015-08-04 10:07:55,277 - WARNING - Terminating Instance:i-5a3a9693... 2015-08-04 10:07:55,518 - WARNING - Done. 2015-08-04 10:07:55,518 - WARNING - Terminating Instance:i-5c852a99... 2015-08-04 10:07:55,723 - WARNING - Done. 2015-08-04 10:07:55,724 - WARNING - Terminating Instance:i-34278bfd... 2015-08-04 10:07:55,982 - WARNING - Done. 2015-08-04 10:07:55,982 - WARNING - Terminating Instance:i-b238947b... 2015-08-04 10:07:56,223 - WARNING - Done. 2015-08-04 10:07:56,223 - WARNING - Terminating Instance:i-eca90629... 2015-08-04 10:07:56,436 - WARNING - Done. 2015-08-04 10:07:56,437 - WARNING - Terminating Instance:i-e93a9620... 2015-08-04 10:07:56,651 - WARNING - Done. 2015-08-04 10:07:56,652 - WARNING - Terminating Instance:i-d6852a13... 2015-08-04 10:07:56,909 - WARNING - Done. 2015-08-04 10:07:56,909 - WARNING - Terminating Instance:i-3ba20dfe... 2015-08-04 10:07:57,274 - WARNING - Done. 2015-08-04 10:07:57,274 - WARNING - Terminating Instance:i-46319d8f... 2015-08-04 10:07:57,495 - WARNING - Done. 2015-08-04 10:07:57,495 - WARNING - Terminating Instance:i-555ff39c... 2015-08-04 10:07:57,746 - WARNING - Done. 2015-08-04 10:07:57,746 - WARNING - Terminating Instance:i-d5086e1d... 2015-08-04 10:07:57,960 - WARNING - Done. 2015-08-04 10:07:57,961 - WARNING - Terminating Instance:i-b2a80777... 2015-08-04 10:07:58,217 - WARNING - Done. 2015-08-04 10:07:58,217 - WARNING - Terminating Instance:i-513a9698... 2015-08-04 10:07:58,445 - WARNING - Done. 2015-08-04 10:07:58,445 - WARNING - Terminating Instance:i-430b6d8b... 2015-08-04 10:07:58,667 - WARNING - Done. 2015-08-04 10:07:58,668 - WARNING - Terminating Instance:i-b5268a7c... 2015-08-04 10:07:58,888 - WARNING - Done. 2015-08-04 10:07:58,888 - WARNING - Terminating Instance:i-53208c9a... 2015-08-04 10:07:59,215 - WARNING - Done. 2015-08-04 10:07:59,215 - WARNING - Terminating Instance:i-bea9067b... 2015-08-04 10:07:59,445 - WARNING - Done. 2015-08-04 10:07:59,445 - WARNING - Terminating Instance:i-58842b9d... 2015-08-04 10:07:59,705 - WARNING - Done. 2015-08-04 10:07:59,706 - WARNING - Terminating Instance:i-9905a950... 2015-08-04 10:08:00,120 - WARNING - Done. 2015-08-04 10:08:00,120 - WARNING - Terminating Instance:i-e507ab2c... 2015-08-04 10:08:08,217 - WARNING - Done. 2015-08-04 10:08:08,217 - WARNING - Terminating Instance:i-733995ba... 2015-08-04 10:08:08,449 - WARNING - Done. 2015-08-04 10:08:08,449 - WARNING - Terminating Instance:i-e4842b21... 2015-08-04 10:08:08,662 - WARNING - Done. (aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/get_spot_amis.py bld-linux64, us-east-1: ami-d73b9abc (spot-bld-linux64-2015-08-03-09-06, ebs) try-linux64, us-east-1: ami-bd3796d6 (spot-try-linux64-2015-08-03-08-46, ebs) tst-linux64, us-east-1: ami-03389968 (spot-tst-linux64-2015-08-03-09-06, ebs) tst-linux32, us-east-1: ami-f9389992 (spot-tst-linux32-2015-08-03-09-11, ebs) tst-emulator64, us-east-1: ami-5b3a9b30 (spot-tst-emulator64-2015-08-03-09-28, ebs) bld-linux64, us-west-2: ami-5f2b226f (spot-bld-linux64-2015-08-02-09-11, ebs) try-linux64, us-west-2: ami-99d6dfa9 (spot-try-linux64-2015-08-03-08-46, ebs) tst-linux64, us-west-2: ami-95e9e0a5 (spot-tst-linux64-2015-08-03-09-06, ebs) tst-linux32, us-west-2: ami-67b4bd57 (spot-tst-linux32-2015-08-03-09-11, ebs) tst-emulator64, us-west-2: ami-5fe8e16f (spot-tst-emulator64-2015-08-03-09-28, ebs)
Blocks: 1189948
Looks like reverting the AMI worked. I've reopened Gaia, but I'm holding off on the other Gecko trees until more AWS instances have spun up to avoid huge backlogs off the bat.
Yeah, the correct fix here was to backout the change to the gtk tarball that added setup.sh, as the setup.sh script is broken. Apparently that change landed before the tooltool changes were deployed, so the bug was hidden until the tooltool changes landed. The change that should be backed out is for bug 1188780: http://hg.mozilla.org/mozilla-central/rev/a9d3df6e14e0 and we can re-land this puppet change once that's backed out. I suspect that setup.sh just needs more linker paths set?
All trees reopened.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Let's repurpose this bug to fix the underlying issue that is that tooltool is not run in the mock environment.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Severity: blocker → normal
No longer blocks: 1180624
Component: Buildduty → General Automation
QA Contact: bugspam.Callek → catlee
Whiteboard: tooltool mock
Blocks: 1188571
Blocks: 1242641
Assignee: nobody → nthomas
Summary: Builds fail with Halting on failure while running ['sh', '/builds/slave/b2g-in-lx-00000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh' → Run tooltool inside mock
This is not tested, because that's what production is for (one less new year's resolution to stick to). There is the possibility that we'll be running mock_mozilla in the wrong cwd, and get the --cwd wrong, but I've tried to avoid that. List of affected builders to follow.
Attachment #8712421 - Flags: review?(bugspam.Callek)
The mac and windows builders change to MockCommand, but mock is disabled for them so they fall back to ShellCommand. Valgrind builds use ScriptFactory, otherwise it's all subclasses of MercurialBuildFactory.
Comment on attachment 8712421 [details] [diff] [review] [buildbotcustom] Fix up buildbot style builds Review of attachment 8712421 [details] [diff] [review]: ----------------------------------------------------------------- Lets give this a shot
Attachment #8712421 - Flags: review?(bugspam.Callek) → review+
In production.
Followup fix for bogus use of mock_workdir which caused failures when not in mock: https://hg.mozilla.org/build/buildbotcustom/rev/a4bc25f76f2d https://hg.mozilla.org/build/buildbotcustom/rev/a212ce1d8529
This extends what we already have for some linux platforms across all linux and android (everywhere we have use_mock True).
Attachment #8712477 - Flags: review?(jlund)
Philor reported valgrinds hitting issues trying to run tooltool, ie ERROR: chroot /builds/mock_mozilla/mozilla-centos6-x86_64/root/ not initialized! Turns out we're trying to run tooltool in mock before setting up mock, hence this change. This should fix it up.
Attachment #8712478 - Flags: review?(jlund)
Comment on attachment 8712477 [details] [diff] [review] [buildbot-configs] Copy in tooltool.py, and run it from in-mock location Review of attachment 8712477 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla/config.py @@ +1821,2 @@ > 'tooltool_manifest_src': 'mobile/android/config/tooltool-manifests/android-x86/releng.manifest', > + 'tooltool_script': ['/builds/tooltool.py'], might be getting late for me but this looks like double entries in the same builder
Comment on attachment 8712478 [details] [diff] [review] [buildbotcustom] Fix valgrind Review of attachment 8712478 [details] [diff] [review]: ----------------------------------------------------------------- sigh. :)
Attachment #8712478 - Flags: review?(jlund) → review+
Comment on attachment 8712487 [details] [diff] [review] [buildbot-configs] Copy in tooltool.py, and run it from in-mock location (with TB too) Review of attachment 8712487 [details] [diff] [review]: ----------------------------------------------------------------- lgtm :)
Attachment #8712487 - Flags: review?(jlund) → review+
Comment on attachment 8712487 [details] [diff] [review] [buildbot-configs] Copy in tooltool.py, and run it from in-mock location (with TB too) https://hg.mozilla.org/build/buildbot-configs/rev/10af9c3220aa https://hg.mozilla.org/build/buildbot-configs/rev/99bb075221a7
Attachment #8712487 - Flags: checked-in+
Depends on: 1243251
This is working on Firefox m-b, and there are some running on m-r and m-esr38 and the comm branches to double check the status there is OK. TODO - backport changes to mozharness-builds on m-a and newer
Blocks: 1245828
Morphing this to be buildbot-jobs only, as work for mozharness is happening in bug 1242641.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Summary: Run tooltool inside mock → Run tooltool inside mock in buildbot jobs
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: