Closed Bug 1190860 Opened 4 years ago Closed 4 years ago

Run tooltool inside mock in buildbot jobs

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: cbook, Assigned: nthomas)

References

Details

(Whiteboard: tooltool mock)

Attachments

(4 files, 1 obsolete file)

Linux build seems to fail across the trees with:

https://treeherder.mozilla.org/logviewer.html#?job_id=12472397&repo=mozilla-inbound

 Halting on failure while running ['sh', '/builds/slave/b2g-in-lx-00000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh', '/builds/slave/b2g-in-lx-00000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', '/builds/tooltool.py', '--authentication-file', '/builds/relengapi.tok'] 

closing integration trees for this
From the log


06:50:08     INFO -  Executing: ['/builds/tooltool.py', '--authentication-file', '/builds/relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', '/builds/slave/m-in-lx-0000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache']
06:50:08     INFO -  INFO - File gcc.tar.xz not present in local cache folder /builds/tooltool_cache
06:50:08     INFO -  INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
06:50:10     INFO -  INFO - File gcc.tar.xz fetched from https://api.pub.build.mozilla.org/tooltool/ as /builds/slave/m-in-lx-0000000000000000000000/build/src/tmpoajZpM
06:50:10     INFO -  INFO - File gtk3.tar.xz not present in local cache folder /builds/tooltool_cache
06:50:10     INFO -  INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
06:50:10     INFO -  INFO - File gtk3.tar.xz fetched from https://api.pub.build.mozilla.org/tooltool/ as /builds/slave/m-in-lx-0000000000000000000000/build/src/tmpF39b8R
06:50:10     INFO -  INFO - File sccache.tar.bz2 not present in local cache folder /builds/tooltool_cache
06:50:10     INFO -  INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
06:50:11     INFO -  INFO - File sccache.tar.bz2 fetched from https://api.pub.build.mozilla.org/tooltool/ as /builds/slave/m-in-lx-0000000000000000000000/build/src/tmp_K1CNf
06:50:11     INFO -  INFO - File integrity verified, renaming tmpoajZpM to gcc.tar.xz
06:50:11     INFO -  INFO - Updating local cache /builds/tooltool_cache...
06:50:11     INFO -  INFO - Local cache /builds/tooltool_cache updated with gcc.tar.xz
06:50:11     INFO -  INFO - File integrity verified, renaming tmpF39b8R to gtk3.tar.xz
06:50:11     INFO -  INFO - Updating local cache /builds/tooltool_cache...
06:50:11     INFO -  INFO - Local cache /builds/tooltool_cache updated with gtk3.tar.xz
06:50:11     INFO -  INFO - File integrity verified, renaming tmp_K1CNf to sccache.tar.bz2
06:50:11     INFO -  INFO - Updating local cache /builds/tooltool_cache...
06:50:11     INFO -  INFO - Local cache /builds/tooltool_cache updated with sccache.tar.bz2
06:50:11     INFO -  INFO - untarring "gcc.tar.xz"
06:50:21     INFO -  INFO - untarring "gtk3.tar.xz"
06:50:22     INFO -  gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
06:50:22     INFO -  gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
06:50:22     INFO -  gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
06:50:22     INFO -  INFO - untarring "sccache.tar.bz2"
06:50:22     INFO -  ERROR - The following files failed: 'gtk3.tar.xz'
06:50:51     INFO -  retry: Calling <function run_with_timeout at 0x7f76d3f6c938> with args: (['/builds/tooltool.py', '--authentication-file', '/builds/relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', '/builds/slave/m-in-lx-0000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #2
06:50:51     INFO -  Executing: ['/builds/tooltool.py', '--authentication-file', '/builds/relengapi.tok', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', '/builds/slave/m-in-lx-0000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache']
06:50:51     INFO -  INFO - rm tree: gcc
06:50:52     INFO -  INFO - untarring "gcc.tar.xz"
06:51:01     INFO -  INFO - rm tree: gtk3
06:51:01     INFO -  INFO - untarring "gtk3.tar.xz"
06:51:02     INFO -  gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
06:51:02     INFO -  gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
06:51:02     INFO -  gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
Sheriffs are pinging glandium to see see if this is fallout from a recent landing

	Tomcat|sheriffduty	maybe related to bug 1186748 but that was on inbound before and was fine 
	Tomcat|sheriffduty	RyanVM: does this maybe need a clobber ?
	Tomcat|sheriffduty	oh there a clobber change
	RyanVM	i thought he landed a clobber touch
	RyanVM	yeah
	Tomcat|sheriffduty	also i guess this would have failed before and not just 1 push after the re-merge
	Callek_cloud	thats definitely a glandium fallout
	Callek_cloud	based on my skim of bugmail last night
	=-=	hwine-ooo is now known as hwine
	Tomcat|sheriffduty	Callek_cloud: yeah but thats a strange fallout
	Callek_cloud	I don't disagree
	Tomcat|sheriffduty	https://treeherder.mozilla.org/#/jobs?repo=fx-team&revision=256463501f11 has green builds and was the merge back from m-c
	Tomcat|sheriffduty	and then the next push fails
	Callek_cloud	but if it leaves behind state in-tree, especially as it relates to setup.sh that he is executing from the tooltool package, I not totally shocked that there is clobber-related bustage involved
	Callek_cloud	admits he hasn't looked at the setup.sh or any of the code in depth, but #releng or #build is likely a better venue to discuss, since thats where glandium is likely to be
	Tomcat|sheriffduty	Callek_cloud: somehow even the mozilla-central push has green builds
	Tomcat|sheriffduty	so wonder whats going on
	Callek_cloud	Tomcat|sheriffduty: ooooo I wonder.
Based on the above logs, and some retriggers, going all the way back to some greens on CLOBBER file touches with glandiums gtk3 push, I came to the conclusion that its a bustage caused by Bug 1189948 due to executing setup.sh from tooltool

I pushed to puppet to back out those changes...

https://hg.mozilla.org/build/puppet/rev/6e7de1c8f6a3
https://hg.mozilla.org/build/puppet/rev/483ee5bd5b8d

I then enumerated all current AMI id's and deleted them from aws, and then ran the terminate script for all bld-* and try-* hosts using said ID's. Log of that below.

I think we are good now, but leaving open pending retrigger results:


(aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/get_spot_amis.py
bld-linux64, us-east-1: ami-7108a81a (spot-bld-linux64-2015-08-04-09-06, ebs)
try-linux64, us-east-1: ami-1b04a470 (spot-try-linux64-2015-08-04-08-48, ebs)
tst-linux64, us-east-1: ami-e906a682 (spot-tst-linux64-2015-08-04-09-05, ebs)
tst-linux32, us-east-1: ami-d506a6be (spot-tst-linux32-2015-08-04-09-08, ebs)
tst-emulator64, us-east-1: ami-d108a8ba (spot-tst-emulator64-2015-08-04-09-26, ebs)
bld-linux64, us-west-2: ami-9fb4bdaf (spot-bld-linux64-2015-08-04-09-06, ebs)
try-linux64, us-west-2: ami-97b5bca7 (spot-try-linux64-2015-08-04-08-48, ebs)
tst-linux64, us-west-2: ami-3db4bd0d (spot-tst-linux64-2015-08-04-09-05, ebs)
tst-linux32, us-west-2: ami-8fe9e0bf (spot-tst-linux32-2015-08-03-09-11, ebs)
tst-emulator64, us-west-2: ami-d5b4bde5 (spot-tst-emulator64-2015-08-04-09-26, ebs)
(aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/aws_terminate_by_ami_id.py --help
usage: aws_terminate_by_ami_id.py [-h] [-r REGIONS] [-v] AMI [AMI ...]

positional arguments:
  AMI                   AMI IDs

optional arguments:
  -h, --help            show this help message and exit
  -r REGIONS, --region REGIONS
  -v, --verbose         Verbose logging
(aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/aws_terminate_by_ami_id.py -r us-e
ast-1 ami-7108a81a ami-1b04a470
2015-08-04 10:02:12,359 - INFO - Preparing to terminate the following 60 instances:
2015-08-04 10:02:12,359 - INFO - i-ff3df52d (bld-linux64-spot-129)
2015-08-04 10:02:12,359 - INFO - i-fe3df52c (bld-linux64-spot-142)
2015-08-04 10:02:12,359 - INFO - i-663cf4b4 (bld-linux64-spot-159)
2015-08-04 10:02:12,362 - INFO - i-d23cf400 (bld-linux64-spot-139)
2015-08-04 10:02:12,362 - INFO - i-343bf3e6 (bld-linux64-spot-131)
2015-08-04 10:02:12,362 - INFO - i-6d3df5bf (bld-linux64-spot-122)
2015-08-04 10:02:12,362 - INFO - i-ceb77f1c (bld-linux64-spot-081)
2015-08-04 10:02:12,362 - INFO - i-22ab63f0 (bld-linux64-spot-080)
2015-08-04 10:02:12,362 - INFO - i-c13cf413 (bld-linux64-spot-028)
2015-08-04 10:02:12,362 - INFO - i-7fc30bad (bld-linux64-spot-094)
2015-08-04 10:02:12,362 - INFO - i-bc3cf46e (bld-linux64-spot-199)
2015-08-04 10:02:12,362 - INFO - i-793bf3ab (bld-linux64-spot-228)
2015-08-04 10:02:12,362 - INFO - i-6b3cf4b9 (bld-linux64-spot-034)
2015-08-04 10:02:12,362 - INFO - i-c63df514 (bld-linux64-spot-195)
2015-08-04 10:02:12,362 - INFO - i-803cf452 (bld-linux64-spot-138)
2015-08-04 10:02:12,362 - INFO - i-303cf4e2 (bld-linux64-spot-144)
2015-08-04 10:02:12,362 - INFO - i-3e3cf4ec (bld-linux64-spot-267)
2015-08-04 10:02:12,362 - INFO - i-6a3cf4b8 (bld-linux64-spot-023)
2015-08-04 10:02:12,365 - INFO - i-c93cf41b (bld-linux64-spot-146)
2015-08-04 10:02:12,365 - INFO - i-9a3cf448 (bld-linux64-spot-027)
2015-08-04 10:02:12,365 - INFO - i-393bf3eb (bld-linux64-spot-137)
2015-08-04 10:02:12,365 - INFO - i-683cf4ba (bld-linux64-spot-148)
2015-08-04 10:02:12,365 - INFO - i-e63df534 (bld-linux64-spot-133)
2015-08-04 10:02:12,365 - INFO - i-633cf4b1 (bld-linux64-spot-071)
2015-08-04 10:02:12,365 - INFO - i-3c3cf4ee (bld-linux64-spot-226)
2015-08-04 10:02:12,365 - INFO - i-363cf4e4 (bld-linux64-spot-250)
2015-08-04 10:02:12,365 - INFO - i-313bf3e3 (bld-linux64-spot-019)
2015-08-04 10:02:12,365 - INFO - i-243cf4f6 (bld-linux64-spot-017)
2015-08-04 10:02:12,365 - INFO - i-bd3cf46f (bld-linux64-spot-252)
2015-08-04 10:02:12,365 - INFO - i-773cf4a5 (bld-linux64-spot-029)
2015-08-04 10:02:12,365 - INFO - i-dc3cf40e (bld-linux64-spot-209)
2015-08-04 10:02:12,368 - INFO - i-c43df516 (bld-linux64-spot-090)
2015-08-04 10:02:12,368 - INFO - i-c23cf410 (bld-linux64-spot-035)
2015-08-04 10:02:12,368 - INFO - i-ad3cf47f (bld-linux64-spot-152)
2015-08-04 10:02:12,368 - INFO - i-8b3cf459 (bld-linux64-spot-145)
2015-08-04 10:02:12,368 - INFO - i-f73df525 (bld-linux64-spot-093)
2015-08-04 10:02:12,368 - INFO - i-7c3cf4ae (bld-linux64-spot-258)
2015-08-04 10:02:12,368 - INFO - i-813cf453 (bld-linux64-spot-245)
2015-08-04 10:02:12,368 - INFO - i-a03cf472 (bld-linux64-spot-147)
2015-08-04 10:02:12,368 - INFO - i-423cf490 (bld-linux64-spot-155)
2015-08-04 10:02:12,368 - INFO - i-85657a2d (bld-linux64-spot-156)
2015-08-04 10:02:12,368 - INFO - i-cd3df51f (bld-linux64-spot-099)
2015-08-04 10:02:12,368 - INFO - i-d33df501 (bld-linux64-spot-130)
2015-08-04 10:02:12,368 - INFO - i-943cf446 (bld-linux64-spot-270)
2015-08-04 10:02:12,371 - INFO - i-623cf4b0 (try-linux64-spot-145)
2015-08-04 10:02:12,371 - INFO - i-b23df560 (bld-linux64-spot-157)
2015-08-04 10:02:12,371 - INFO - i-873cf455 (bld-linux64-spot-150)
2015-08-04 10:02:12,371 - INFO - i-383cf4ea (bld-linux64-spot-242)
2015-08-04 10:02:12,371 - INFO - i-96dc1444 (bld-linux64-spot-158)
2015-08-04 10:02:12,371 - INFO - i-2c3bf3fe (bld-linux64-spot-038)
2015-08-04 10:02:12,371 - INFO - i-3d8c47ef (bld-linux64-spot-024)
2015-08-04 10:02:12,371 - INFO - i-3b3cf4e9 (bld-linux64-spot-240)
2015-08-04 10:02:12,371 - INFO - i-b23cf460 (bld-linux64-spot-153)
2015-08-04 10:02:12,371 - INFO - i-bb3cf469 (bld-linux64-spot-143)
2015-08-04 10:02:12,371 - INFO - i-303bf3e2 (bld-linux64-spot-033)
2015-08-04 10:02:12,374 - INFO - i-f53df527 (bld-linux64-spot-192)
2015-08-04 10:02:12,374 - INFO - i-e466794c (bld-linux64-spot-011)
2015-08-04 10:02:12,374 - INFO - i-f83df52a (bld-linux64-spot-244)
2015-08-04 10:02:12,374 - INFO - i-2b3bf3f9 (try-linux64-spot-148)
2015-08-04 10:02:12,374 - INFO - i-3f3bf3ed (bld-linux64-spot-227)
Are you sure you want to kill these? ^ y/N >y
ARE YOU SURE YOU WANT TO KILL THESE? ^ LAST WARNING!!! y/N >y
2015-08-04 10:02:19,541 - WARNING - The instances mentioned above are about to be terminated
2015-08-04 10:02:19,541 - WARNING - Waiting extra 60 seconds to make sure...
2015-08-04 10:03:19,601 - WARNING - Starting...
2015-08-04 10:03:19,601 - WARNING - Terminating Instance:i-ff3df52d...
2015-08-04 10:03:20,257 - WARNING - Done.
2015-08-04 10:03:20,257 - WARNING - Terminating Instance:i-fe3df52c...
2015-08-04 10:03:20,637 - WARNING - Done.
2015-08-04 10:03:20,637 - WARNING - Terminating Instance:i-663cf4b4...
2015-08-04 10:03:21,163 - WARNING - Done.
2015-08-04 10:03:21,163 - WARNING - Terminating Instance:i-d23cf400...
2015-08-04 10:03:21,504 - WARNING - Done.
2015-08-04 10:03:21,504 - WARNING - Terminating Instance:i-343bf3e6...
2015-08-04 10:03:22,103 - WARNING - Done.
2015-08-04 10:03:22,103 - WARNING - Terminating Instance:i-6d3df5bf...
2015-08-04 10:03:22,734 - WARNING - Done.
2015-08-04 10:03:22,734 - WARNING - Terminating Instance:i-ceb77f1c...
2015-08-04 10:03:23,094 - WARNING - Done.
2015-08-04 10:03:23,094 - WARNING - Terminating Instance:i-22ab63f0...
2015-08-04 10:03:23,437 - WARNING - Done.
2015-08-04 10:03:23,437 - WARNING - Terminating Instance:i-c13cf413...
2015-08-04 10:03:23,791 - WARNING - Done.
2015-08-04 10:03:23,791 - WARNING - Terminating Instance:i-7fc30bad...
2015-08-04 10:03:24,517 - WARNING - Done.
2015-08-04 10:03:24,517 - WARNING - Terminating Instance:i-bc3cf46e...
2015-08-04 10:03:24,897 - WARNING - Done.
2015-08-04 10:03:24,897 - WARNING - Terminating Instance:i-793bf3ab...
2015-08-04 10:03:25,231 - WARNING - Done.
2015-08-04 10:03:25,231 - WARNING - Terminating Instance:i-6b3cf4b9...
2015-08-04 10:03:25,613 - WARNING - Done.
2015-08-04 10:03:25,613 - WARNING - Terminating Instance:i-c63df514...
2015-08-04 10:03:25,999 - WARNING - Done.
2015-08-04 10:03:25,999 - WARNING - Terminating Instance:i-803cf452...
2015-08-04 10:03:26,318 - WARNING - Done.
2015-08-04 10:03:26,318 - WARNING - Terminating Instance:i-303cf4e2...
2015-08-04 10:03:26,694 - WARNING - Done.
2015-08-04 10:03:26,694 - WARNING - Terminating Instance:i-3e3cf4ec...
2015-08-04 10:03:27,190 - WARNING - Done.
2015-08-04 10:03:27,190 - WARNING - Terminating Instance:i-6a3cf4b8...
2015-08-04 10:03:27,508 - WARNING - Done.
2015-08-04 10:03:27,509 - WARNING - Terminating Instance:i-c93cf41b...
2015-08-04 10:03:27,883 - WARNING - Done.
2015-08-04 10:03:27,883 - WARNING - Terminating Instance:i-9a3cf448...
2015-08-04 10:03:28,336 - WARNING - Done.
2015-08-04 10:03:28,337 - WARNING - Terminating Instance:i-393bf3eb...
2015-08-04 10:03:28,845 - WARNING - Done.
2015-08-04 10:03:28,845 - WARNING - Terminating Instance:i-683cf4ba...
2015-08-04 10:03:29,233 - WARNING - Done.
2015-08-04 10:03:29,233 - WARNING - Terminating Instance:i-e63df534...
2015-08-04 10:03:29,540 - WARNING - Done.
2015-08-04 10:03:29,540 - WARNING - Terminating Instance:i-633cf4b1...
2015-08-04 10:03:29,849 - WARNING - Done.
2015-08-04 10:03:29,849 - WARNING - Terminating Instance:i-3c3cf4ee...
2015-08-04 10:03:30,145 - WARNING - Done.
2015-08-04 10:03:30,145 - WARNING - Terminating Instance:i-363cf4e4...
2015-08-04 10:03:30,469 - WARNING - Done.
2015-08-04 10:03:30,469 - WARNING - Terminating Instance:i-313bf3e3...
2015-08-04 10:03:30,808 - WARNING - Done.
2015-08-04 10:03:30,808 - WARNING - Terminating Instance:i-243cf4f6...
2015-08-04 10:03:31,147 - WARNING - Done.
2015-08-04 10:03:31,147 - WARNING - Terminating Instance:i-bd3cf46f...
2015-08-04 10:03:31,502 - WARNING - Done.
2015-08-04 10:03:31,502 - WARNING - Terminating Instance:i-773cf4a5...
2015-08-04 10:03:31,864 - WARNING - Done.
2015-08-04 10:03:31,864 - WARNING - Terminating Instance:i-dc3cf40e...
2015-08-04 10:03:32,343 - WARNING - Done.
2015-08-04 10:03:32,343 - WARNING - Terminating Instance:i-c43df516...
2015-08-04 10:03:33,002 - WARNING - Done.
2015-08-04 10:03:33,002 - WARNING - Terminating Instance:i-c23cf410...
2015-08-04 10:03:33,333 - WARNING - Done.
2015-08-04 10:03:33,333 - WARNING - Terminating Instance:i-ad3cf47f...
2015-08-04 10:03:33,643 - WARNING - Done.
2015-08-04 10:03:33,643 - WARNING - Terminating Instance:i-8b3cf459...
2015-08-04 10:03:34,064 - WARNING - Done.
2015-08-04 10:03:34,064 - WARNING - Terminating Instance:i-f73df525...
2015-08-04 10:03:34,389 - WARNING - Done.
2015-08-04 10:03:34,389 - WARNING - Terminating Instance:i-7c3cf4ae...
2015-08-04 10:03:34,816 - WARNING - Done.
2015-08-04 10:03:34,816 - WARNING - Terminating Instance:i-813cf453...
2015-08-04 10:03:35,226 - WARNING - Done.
2015-08-04 10:03:35,226 - WARNING - Terminating Instance:i-a03cf472...
2015-08-04 10:03:35,559 - WARNING - Done.
2015-08-04 10:03:35,559 - WARNING - Terminating Instance:i-423cf490...
2015-08-04 10:03:35,905 - WARNING - Done.
2015-08-04 10:03:35,905 - WARNING - Terminating Instance:i-85657a2d...
2015-08-04 10:03:36,274 - WARNING - Done.
2015-08-04 10:03:36,275 - WARNING - Terminating Instance:i-cd3df51f...
2015-08-04 10:03:36,612 - WARNING - Done.
2015-08-04 10:03:36,612 - WARNING - Terminating Instance:i-d33df501...
2015-08-04 10:03:36,931 - WARNING - Done.
2015-08-04 10:03:36,931 - WARNING - Terminating Instance:i-943cf446...
2015-08-04 10:03:37,322 - WARNING - Done.
2015-08-04 10:03:37,322 - WARNING - Terminating Instance:i-623cf4b0...
2015-08-04 10:03:37,726 - WARNING - Done.
2015-08-04 10:03:37,726 - WARNING - Terminating Instance:i-b23df560...
2015-08-04 10:03:38,223 - WARNING - Done.
2015-08-04 10:03:38,223 - WARNING - Terminating Instance:i-873cf455...
2015-08-04 10:03:38,586 - WARNING - Done.
2015-08-04 10:03:38,586 - WARNING - Terminating Instance:i-383cf4ea...
2015-08-04 10:03:38,948 - WARNING - Done.
2015-08-04 10:03:38,948 - WARNING - Terminating Instance:i-96dc1444...
2015-08-04 10:03:39,371 - WARNING - Done.
2015-08-04 10:03:39,371 - WARNING - Terminating Instance:i-2c3bf3fe...
2015-08-04 10:03:39,828 - WARNING - Done.
2015-08-04 10:03:39,828 - WARNING - Terminating Instance:i-3d8c47ef...
2015-08-04 10:03:40,216 - WARNING - Done.
2015-08-04 10:03:40,216 - WARNING - Terminating Instance:i-3b3cf4e9...
2015-08-04 10:03:40,683 - WARNING - Done.
2015-08-04 10:03:40,683 - WARNING - Terminating Instance:i-b23cf460...
2015-08-04 10:03:41,024 - WARNING - Done.
2015-08-04 10:03:41,024 - WARNING - Terminating Instance:i-bb3cf469...
2015-08-04 10:03:41,342 - WARNING - Done.
2015-08-04 10:03:41,343 - WARNING - Terminating Instance:i-303bf3e2...
2015-08-04 10:03:41,650 - WARNING - Done.
2015-08-04 10:03:41,650 - WARNING - Terminating Instance:i-f53df527...
2015-08-04 10:03:42,049 - WARNING - Done.
2015-08-04 10:03:42,049 - WARNING - Terminating Instance:i-e466794c...
2015-08-04 10:03:42,387 - WARNING - Done.
2015-08-04 10:03:42,387 - WARNING - Terminating Instance:i-f83df52a...
2015-08-04 10:03:42,778 - WARNING - Done.
2015-08-04 10:03:42,778 - WARNING - Terminating Instance:i-2b3bf3f9...
2015-08-04 10:03:43,136 - WARNING - Done.
2015-08-04 10:03:43,136 - WARNING - Terminating Instance:i-3f3bf3ed...
2015-08-04 10:03:43,496 - WARNING - Done.
(aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/aws_terminate_by_ami_id.py -r us-w
est-2 ami-9fb4bdaf ami-97b5bca7
2015-08-04 10:05:18,582 - INFO - Preparing to terminate the following 220 instances:
2015-08-04 10:05:18,582 - INFO - i-c1389408 (try-linux64-spot-421)
2015-08-04 10:05:18,582 - INFO - i-e7842b22 (try-linux64-spot-1099)
2015-08-04 10:05:18,582 - INFO - i-a50ba76c (try-linux64-spot-499)
2015-08-04 10:05:18,582 - INFO - i-fa0b6d32 (bld-linux64-spot-542)
2015-08-04 10:05:18,582 - INFO - i-f20b6d3a (bld-linux64-spot-424)
2015-08-04 10:05:18,582 - INFO - i-450b6d8d (bld-linux64-spot-425)
2015-08-04 10:05:18,582 - INFO - i-7a0b6db2 (bld-linux64-spot-545)
2015-08-04 10:05:18,583 - INFO - i-b5a80770 (bld-linux64-spot-590)
2015-08-04 10:05:18,583 - INFO - i-aa5ff363 (try-linux64-spot-494)
2015-08-04 10:05:18,583 - INFO - i-b4a80771 (bld-linux64-spot-1062)
2015-08-04 10:05:18,583 - INFO - i-fb238f32 (try-linux64-spot-402)
2015-08-04 10:05:18,583 - INFO - i-dfa20d1a (bld-linux64-spot-364)
2015-08-04 10:05:18,583 - INFO - i-62268aab (try-linux64-spot-437)
2015-08-04 10:05:18,583 - INFO - i-4f0b6d87 (bld-linux64-spot-426)
2015-08-04 10:05:18,583 - INFO - i-a55ff36c (bld-linux64-spot-1063)
2015-08-04 10:05:18,583 - INFO - i-778629b2 (try-linux64-spot-489)
2015-08-04 10:05:18,583 - INFO - i-39278bf0 (try-linux64-spot-387)
2015-08-04 10:05:18,583 - INFO - i-385ff3f1 (try-linux64-spot-439)
2015-08-04 10:05:18,583 - INFO - i-cc0b6d04 (try-linux64-spot-432)
2015-08-04 10:05:18,584 - INFO - i-b0a80775 (bld-linux64-spot-1064)
2015-08-04 10:05:18,584 - INFO - i-420b6d8a (bld-linux64-spot-369)
2015-08-04 10:05:18,584 - INFO - i-7bab04be (bld-linux64-spot-1065)
2015-08-04 10:05:18,584 - INFO - i-010aa6c8 (try-linux64-spot-424)
2015-08-04 10:05:18,584 - INFO - i-56852a93 (try-linux64-spot-1094)
2015-08-04 10:05:18,584 - INFO - i-ca086e02 (bld-linux64-spot-1080)
2015-08-04 10:05:18,584 - INFO - i-8704a84e (try-linux64-spot-498)
2015-08-04 10:05:18,584 - INFO - i-150fa3dc (try-linux64-spot-417)
2015-08-04 10:05:18,584 - INFO - i-080a6cc0 (bld-linux64-spot-1069)
2015-08-04 10:05:18,584 - INFO - i-d7086e1f (bld-linux64-spot-428)
2015-08-04 10:05:18,584 - INFO - i-afa30c6a (bld-linux64-spot-361)
2015-08-04 10:05:18,584 - INFO - i-3e832cfb (try-linux64-spot-1093)
2015-08-04 10:05:18,584 - INFO - i-ce58f407 (bld-linux64-spot-458)
2015-08-04 10:05:18,585 - INFO - i-b1a30c74 (bld-linux64-spot-1059)
2015-08-04 10:05:18,585 - INFO - i-400aa689 (try-linux64-spot-423)
2015-08-04 10:05:18,585 - INFO - i-a60ba76f (try-linux64-spot-340)
2015-08-04 10:05:18,585 - INFO - i-3e278bf7 (try-linux64-spot-386)
2015-08-04 10:05:18,585 - INFO - i-56842b93 (try-linux64-spot-422)
2015-08-04 10:05:18,585 - INFO - i-db600613 (bld-linux64-spot-302)
2015-08-04 10:05:18,585 - INFO - i-76238fbf (try-linux64-spot-403)
2015-08-04 10:05:18,585 - INFO - i-565ff39f (bld-linux64-spot-319)
2015-08-04 10:05:18,585 - INFO - i-df086e17 (try-linux64-spot-433)
2015-08-04 10:05:18,585 - INFO - i-e407ab2d (try-linux64-spot-1083)
2015-08-04 10:05:18,585 - INFO - i-daa20d1f (bld-linux64-spot-360)
2015-08-04 10:05:18,585 - INFO - i-105df1d9 (bld-linux64-spot-532)
2015-08-04 10:05:18,586 - INFO - i-5b3a9692 (try-linux64-spot-481)
2015-08-04 10:05:18,586 - INFO - i-3aa30cff (bld-linux64-spot-1053)
2015-08-04 10:05:18,586 - INFO - i-9b05a952 (try-linux64-spot-1080)
2015-08-04 10:05:18,586 - INFO - i-51258998 (try-linux64-spot-431)
2015-08-04 10:05:18,586 - INFO - i-5941ed90 (bld-linux64-spot-431)
2015-08-04 10:05:18,586 - INFO - i-e707ab2e (try-linux64-spot-1085)
2015-08-04 10:05:18,586 - INFO - i-a30ba76a (try-linux64-spot-497)
2015-08-04 10:05:18,586 - INFO - i-580ba791 (try-linux64-spot-364)
2015-08-04 10:05:18,586 - INFO - i-ab5ff362 (bld-linux64-spot-312)
2015-08-04 10:05:18,586 - INFO - i-bfa9067a (bld-linux64-spot-435)
2015-08-04 10:05:18,586 - INFO - i-3a42eef3 (bld-linux64-spot-506)
2015-08-04 10:05:18,586 - INFO - i-ee3a9627 (try-linux64-spot-485)
2015-08-04 10:05:18,586 - INFO - i-7aab04bf (bld-linux64-spot-1057)
2015-08-04 10:05:18,587 - INFO - i-5d04a894 (try-linux64-spot-362)
2015-08-04 10:05:18,587 - INFO - i-805df149 (bld-linux64-spot-518)
2015-08-04 10:05:18,587 - INFO - i-135df1da (try-linux64-spot-380)
2015-08-04 10:05:18,587 - INFO - i-3ea30cfb (bld-linux64-spot-383)
2015-08-04 10:05:18,587 - INFO - i-0d5df1c4 (bld-linux64-spot-531)
2015-08-04 10:05:18,587 - INFO - i-f8238f31 (try-linux64-spot-388)
2015-08-04 10:05:18,587 - INFO - i-5a0ba793 (try-linux64-spot-366)
2015-08-04 10:05:18,587 - INFO - i-76268abf (try-linux64-spot-300)
2015-08-04 10:05:18,587 - INFO - i-dd842b18 (try-linux64-spot-1097)
2015-08-04 10:05:18,587 - INFO - i-b2832c77 (try-linux64-spot-414)
2015-08-04 10:05:18,587 - INFO - i-59208c90 (try-linux64-spot-408)
2015-08-04 10:05:18,587 - INFO - i-0b0a6cc3 (bld-linux64-spot-498)
2015-08-04 10:05:18,588 - INFO - i-b7a80772 (bld-linux64-spot-377)
2015-08-04 10:05:18,588 - INFO - i-a6862963 (try-linux64-spot-1096)
2015-08-04 10:05:18,588 - INFO - i-400b6d88 (bld-linux64-spot-541)
2015-08-04 10:05:18,588 - INFO - i-e5aa0520 (bld-linux64-spot-305)
2015-08-04 10:05:18,588 - INFO - i-d9a20d1c (bld-linux64-spot-493)
2015-08-04 10:05:18,588 - INFO - i-d0086e18 (bld-linux64-spot-1096)
2015-08-04 10:05:18,588 - INFO - i-5f852a9a (try-linux64-spot-379)
2015-08-04 10:05:18,588 - INFO - i-9238945b (try-linux64-spot-308)
2015-08-04 10:05:18,588 - INFO - i-b8a9067d (bld-linux64-spot-304)
2015-08-04 10:05:18,588 - INFO - i-eb07ab22 (try-linux64-spot-1082)
2015-08-04 10:05:18,588 - INFO - i-75ab04b0 (bld-linux64-spot-336)
2015-08-04 10:05:18,588 - INFO - i-7f09a5b6 (try-linux64-spot-445)
2015-08-04 10:05:18,588 - INFO - i-410b6d89 (bld-linux64-spot-368)
2015-08-04 10:05:18,589 - INFO - i-420aa68b (try-linux64-spot-1090)
2015-08-04 10:05:18,589 - INFO - i-9c399555 (try-linux64-spot-482)
2015-08-04 10:05:18,589 - INFO - i-c958f400 (bld-linux64-spot-310)
2015-08-04 10:05:18,589 - INFO - i-6b5ef2a2 (try-linux64-spot-385)
2015-08-04 10:05:18,589 - INFO - i-a3228e6a (try-linux64-spot-405)
2015-08-04 10:05:18,589 - INFO - i-df842b1a (try-linux64-spot-411)
2015-08-04 10:05:18,589 - INFO - i-5e04a897 (try-linux64-spot-425)
2015-08-04 10:05:18,589 - INFO - i-a542ee6c (bld-linux64-spot-1054)
2015-08-04 10:05:18,589 - INFO - i-51862994 (try-linux64-spot-419)
2015-08-04 10:05:18,589 - INFO - i-d8389411 (try-linux64-spot-376)
2015-08-04 10:05:18,589 - INFO - i-d1086e19 (bld-linux64-spot-421)
2015-08-04 10:05:18,589 - INFO - i-05238fcc (try-linux64-spot-404)
2015-08-04 10:05:18,590 - INFO - i-4b238f82 (try-linux64-spot-496)
2015-08-04 10:05:18,590 - INFO - i-195df1d0 (bld-linux64-spot-439)
2015-08-04 10:05:18,590 - INFO - i-490b6d81 (bld-linux64-spot-560)
2015-08-04 10:05:18,590 - INFO - i-bd208c74 (try-linux64-spot-361)
2015-08-04 10:05:18,590 - INFO - i-4c5ef285 (try-linux64-spot-436)
2015-08-04 10:05:18,590 - INFO - i-9c05a955 (try-linux64-spot-1087)
2015-08-04 10:05:18,590 - INFO - i-d8842b1d (try-linux64-spot-418)
2015-08-04 10:05:18,590 - INFO - i-a00ba769 (try-linux64-spot-495)
2015-08-04 10:05:18,590 - INFO - i-ae0fa367 (try-linux64-spot-1091)
2015-08-04 10:05:18,590 - INFO - i-1f2589d6 (try-linux64-spot-303)
2015-08-04 10:05:18,590 - INFO - i-f00b6d38 (bld-linux64-spot-598)
2015-08-04 10:05:18,590 - INFO - i-3b42eef2 (bld-linux64-spot-534)
2015-08-04 10:05:18,590 - INFO - i-9d05a954 (try-linux64-spot-440)
2015-08-04 10:05:18,591 - INFO - i-b3a80776 (bld-linux64-spot-1051)
2015-08-04 10:05:18,591 - INFO - i-5d0ba794 (try-linux64-spot-360)
2015-08-04 10:05:18,591 - INFO - i-1004a8d9 (try-linux64-spot-1081)
2015-08-04 10:05:18,591 - INFO - i-99399550 (try-linux64-spot-373)
2015-08-04 10:05:18,591 - INFO - i-baa20d7f (bld-linux64-spot-491)
2015-08-04 10:05:18,591 - INFO - i-8b238f42 (try-linux64-spot-371)
2015-08-04 10:05:18,591 - INFO - i-79ab04bc (bld-linux64-spot-592)
2015-08-04 10:05:18,591 - INFO - i-b8a20d7d (bld-linux64-spot-373)
2015-08-04 10:05:18,591 - INFO - i-5b0ba792 (try-linux64-spot-426)
2015-08-04 10:05:18,591 - INFO - i-cc04a805 (try-linux64-spot-365)
2015-08-04 10:05:18,591 - INFO - i-5a04a893 (try-linux64-spot-1088)
2015-08-04 10:05:18,592 - INFO - i-070a6ccf (bld-linux64-spot-422)
2015-08-04 10:05:18,592 - INFO - i-2d3f93e4 (try-linux64-spot-367)
2015-08-04 10:05:18,592 - INFO - i-1f3894d6 (try-linux64-spot-486)
2015-08-04 10:05:18,592 - INFO - i-fc42ee35 (bld-linux64-spot-432)
2015-08-04 10:05:18,592 - INFO - i-17a20dd2 (bld-linux64-spot-1090)
2015-08-04 10:05:18,592 - INFO - i-c9086e01 (bld-linux64-spot-547)
2015-08-04 10:05:18,592 - INFO - i-cb086e03 (bld-linux64-spot-499)
2015-08-04 10:05:18,592 - INFO - i-4f5ff386 (bld-linux64-spot-313)
2015-08-04 10:05:18,592 - INFO - i-a85ff361 (try-linux64-spot-389)
2015-08-04 10:05:18,592 - INFO - i-0aa20dcf (bld-linux64-spot-388)
2015-08-04 10:05:18,592 - INFO - i-523a969b (try-linux64-spot-372)
2015-08-04 10:05:18,592 - INFO - i-0f238fc6 (try-linux64-spot-400)
2015-08-04 10:05:18,593 - INFO - i-dba20d1e (bld-linux64-spot-362)
2015-08-04 10:05:18,593 - INFO - i-cf0b6d07 (bld-linux64-spot-540)
2015-08-04 10:05:18,593 - INFO - i-f70b6d3f (bld-linux64-spot-543)
2015-08-04 10:05:18,593 - INFO - i-c90b6d01 (bld-linux64-spot-429)
2015-08-04 10:05:18,593 - INFO - i-b338947a (try-linux64-spot-378)
2015-08-04 10:05:18,593 - INFO - i-042589cd (try-linux64-spot-336)
2015-08-04 10:05:18,593 - INFO - i-74ab04b1 (bld-linux64-spot-1067)
2015-08-04 10:05:18,593 - INFO - i-515ff398 (bld-linux64-spot-328)
2015-08-04 10:05:18,593 - INFO - i-a208a46b (try-linux64-spot-416)
2015-08-04 10:05:18,593 - INFO - i-c258f40b (bld-linux64-spot-1060)
2015-08-04 10:05:18,593 - INFO - i-e2842b27 (try-linux64-spot-1098)
2015-08-04 10:05:18,593 - INFO - i-e3842b26 (try-linux64-spot-410)
2015-08-04 10:05:18,594 - INFO - i-aa208c63 (try-linux64-spot-382)
2015-08-04 10:05:18,594 - INFO - i-5641ed9f (bld-linux64-spot-535)
2015-08-04 10:05:18,594 - INFO - i-31a30cf4 (bld-linux64-spot-365)
2015-08-04 10:05:18,594 - INFO - i-d2086e1a (bld-linux64-spot-566)
2015-08-04 10:05:18,594 - INFO - i-71aa05b4 (bld-linux64-spot-558)
2015-08-04 10:05:18,594 - INFO - i-4b086e83 (bld-linux64-spot-427)
2015-08-04 10:05:18,594 - INFO - i-02238fcb (try-linux64-spot-407)
2015-08-04 10:05:18,594 - INFO - i-c042ee09 (bld-linux64-spot-433)
2015-08-04 10:05:18,594 - INFO - i-a10ba768 (try-linux64-spot-427)
2015-08-04 10:05:18,594 - INFO - i-b4268a7d (try-linux64-spot-435)
2015-08-04 10:05:18,594 - INFO - i-a40ba76d (try-linux64-spot-491)
2015-08-04 10:05:18,594 - INFO - i-ea3a9623 (try-linux64-spot-483)
2015-08-04 10:05:18,594 - INFO - i-460b6d8e (bld-linux64-spot-423)
2015-08-04 10:05:18,595 - INFO - i-5b208c92 (try-linux64-spot-413)
2015-08-04 10:05:18,595 - INFO - i-5225899b (try-linux64-spot-307)
2015-08-04 10:05:18,595 - INFO - i-75238fbc (try-linux64-spot-469)
2015-08-04 10:05:18,595 - INFO - i-4f5ef286 (try-linux64-spot-306)
2015-08-04 10:05:18,595 - INFO - i-1e3894d7 (try-linux64-spot-487)
2015-08-04 10:05:18,595 - INFO - i-4a5ef283 (bld-linux64-spot-537)
2015-08-04 10:05:18,595 - INFO - i-c85ef201 (bld-linux64-spot-438)
2015-08-04 10:05:18,595 - INFO - i-4b5ff382 (try-linux64-spot-363)
2015-08-04 10:05:18,595 - INFO - i-5c04a895 (try-linux64-spot-368)
2015-08-04 10:05:18,595 - INFO - i-38a20dfd (bld-linux64-spot-490)
2015-08-04 10:05:18,595 - INFO - i-ee06aa27 (try-linux64-spot-1084)
2015-08-04 10:05:18,595 - INFO - i-505ff399 (bld-linux64-spot-510)
2015-08-04 10:05:18,595 - INFO - i-4e5ff387 (bld-linux64-spot-492)
2015-08-04 10:05:18,596 - INFO - i-090a6cc1 (bld-linux64-spot-544)
2015-08-04 10:05:18,596 - INFO - i-cb0b6d03 (bld-linux64-spot-546)
2015-08-04 10:05:18,596 - INFO - i-aaa30c6f (bld-linux64-spot-1094)
2015-08-04 10:05:18,596 - INFO - i-d6218d1f (try-linux64-spot-383)
2015-08-04 10:05:18,596 - INFO - i-e607ab2f (try-linux64-spot-443)
2015-08-04 10:05:18,596 - INFO - i-65238fac (try-linux64-spot-401)
2015-08-04 10:05:18,596 - INFO - i-d4086e1c (bld-linux64-spot-1068)
2015-08-04 10:05:18,596 - INFO - i-efa9062a (bld-linux64-spot-1083)
2015-08-04 10:05:18,596 - INFO - i-85309c4c (bld-linux64-spot-315)
2015-08-04 10:05:18,596 - INFO - i-0c5df1c5 (bld-linux64-spot-536)
2015-08-04 10:05:18,596 - INFO - i-6a5ef2a3 (try-linux64-spot-384)
2015-08-04 10:05:18,596 - INFO - i-b6a80773 (bld-linux64-spot-434)
2015-08-04 10:05:18,596 - INFO - i-9a05a953 (try-linux64-spot-1089)
2015-08-04 10:05:18,597 - INFO - i-c50b6d0d (bld-linux64-spot-337)
2015-08-04 10:05:18,597 - INFO - i-3da30cf8 (bld-linux64-spot-363)
2015-08-04 10:05:18,597 - INFO - i-7e268ab7 (try-linux64-spot-434)
2015-08-04 10:05:18,597 - INFO - i-1704a8de (try-linux64-spot-442)
2015-08-04 10:05:18,597 - INFO - i-c858f401 (bld-linux64-spot-318)
2015-08-04 10:05:18,597 - INFO - i-125df1db (try-linux64-spot-304)
2015-08-04 10:05:18,597 - INFO - i-708629b5 (try-linux64-spot-1095)
2015-08-04 10:05:18,597 - INFO - i-b0a20d75 (bld-linux64-spot-379)
2015-08-04 10:05:18,597 - INFO - i-5b04a892 (try-linux64-spot-369)
2015-08-04 10:05:18,597 - INFO - i-c0389409 (try-linux64-spot-375)
2015-08-04 10:05:18,597 - INFO - i-d7218d1e (try-linux64-spot-409)
2015-08-04 10:05:18,598 - INFO - i-dc852a19 (try-linux64-spot-488)
2015-08-04 10:05:18,598 - INFO - i-020a6cca (bld-linux64-spot-548)
2015-08-04 10:05:18,598 - INFO - i-c5218d0c (try-linux64-spot-492)
2015-08-04 10:05:18,598 - INFO - i-ada30c68 (bld-linux64-spot-1055)
2015-08-04 10:05:18,598 - INFO - i-70aa05b5 (bld-linux64-spot-1066)
2015-08-04 10:05:18,598 - INFO - i-a75df16e (bld-linux64-spot-530)
2015-08-04 10:05:18,598 - INFO - i-5a3a9693 (try-linux64-spot-370)
2015-08-04 10:05:18,598 - INFO - i-5c852a99 (try-linux64-spot-415)
2015-08-04 10:05:18,598 - INFO - i-34278bfd (try-linux64-spot-406)
2015-08-04 10:05:18,598 - INFO - i-b238947b (try-linux64-spot-480)
2015-08-04 10:05:18,598 - INFO - i-eca90629 (bld-linux64-spot-419)
2015-08-04 10:05:18,598 - INFO - i-e93a9620 (try-linux64-spot-374)
2015-08-04 10:05:18,598 - INFO - i-d6852a13 (try-linux64-spot-316)
2015-08-04 10:05:18,599 - INFO - i-3ba20dfe (try-linux64-spot-301)
2015-08-04 10:05:18,599 - INFO - i-46319d8f (bld-linux64-spot-314)
2015-08-04 10:05:18,599 - INFO - i-555ff39c (bld-linux64-spot-311)
2015-08-04 10:05:18,599 - INFO - i-d5086e1d (bld-linux64-spot-549)
2015-08-04 10:05:18,599 - INFO - i-b2a80777 (bld-linux64-spot-1058)
2015-08-04 10:05:18,599 - INFO - i-513a9698 (try-linux64-spot-377)
2015-08-04 10:05:18,599 - INFO - i-430b6d8b (bld-linux64-spot-420)
2015-08-04 10:05:18,599 - INFO - i-b5268a7c (try-linux64-spot-430)
2015-08-04 10:05:18,599 - INFO - i-53208c9a (try-linux64-spot-302)
2015-08-04 10:05:18,599 - INFO - i-bea9067b (bld-linux64-spot-1061)
2015-08-04 10:05:18,599 - INFO - i-58842b9d (try-linux64-spot-1092)
2015-08-04 10:05:18,599 - INFO - i-9905a950 (try-linux64-spot-1086)
2015-08-04 10:05:18,599 - INFO - i-e507ab2c (try-linux64-spot-466)
2015-08-04 10:05:18,600 - INFO - i-733995ba (try-linux64-spot-484)
2015-08-04 10:05:18,600 - INFO - i-e4842b21 (try-linux64-spot-412)
Are you sure you want to kill these? ^ y/N >y
ARE YOU SURE YOU WANT TO KILL THESE? ^ LAST WARNING!!! y/N >y
2015-08-04 10:05:24,102 - WARNING - The instances mentioned above are about to be terminated
2015-08-04 10:05:24,102 - WARNING - Waiting extra 60 seconds to make sure...
2015-08-04 10:06:24,108 - WARNING - Starting...
2015-08-04 10:06:24,109 - WARNING - Terminating Instance:i-c1389408...
2015-08-04 10:06:24,419 - WARNING - Done.
2015-08-04 10:06:24,419 - WARNING - Terminating Instance:i-e7842b22...
2015-08-04 10:06:24,660 - WARNING - Done.
2015-08-04 10:06:24,660 - WARNING - Terminating Instance:i-a50ba76c...
2015-08-04 10:06:24,871 - WARNING - Done.
2015-08-04 10:06:24,871 - WARNING - Terminating Instance:i-fa0b6d32...
2015-08-04 10:06:25,111 - WARNING - Done.
2015-08-04 10:06:25,112 - WARNING - Terminating Instance:i-f20b6d3a...
2015-08-04 10:06:25,292 - WARNING - Done.
2015-08-04 10:06:25,292 - WARNING - Terminating Instance:i-450b6d8d...
2015-08-04 10:06:25,500 - WARNING - Done.
2015-08-04 10:06:25,500 - WARNING - Terminating Instance:i-7a0b6db2...
2015-08-04 10:06:25,730 - WARNING - Done.
2015-08-04 10:06:25,730 - WARNING - Terminating Instance:i-b5a80770...
2015-08-04 10:06:25,951 - WARNING - Done.
2015-08-04 10:06:25,952 - WARNING - Terminating Instance:i-aa5ff363...
2015-08-04 10:06:26,192 - WARNING - Done.
2015-08-04 10:06:26,192 - WARNING - Terminating Instance:i-b4a80771...
2015-08-04 10:06:26,404 - WARNING - Done.
2015-08-04 10:06:26,404 - WARNING - Terminating Instance:i-fb238f32...
2015-08-04 10:06:26,630 - WARNING - Done.
2015-08-04 10:06:26,630 - WARNING - Terminating Instance:i-dfa20d1a...
2015-08-04 10:06:26,853 - WARNING - Done.
2015-08-04 10:06:26,853 - WARNING - Terminating Instance:i-62268aab...
2015-08-04 10:06:27,090 - WARNING - Done.
2015-08-04 10:06:27,090 - WARNING - Terminating Instance:i-4f0b6d87...
2015-08-04 10:06:27,286 - WARNING - Done.
2015-08-04 10:06:27,286 - WARNING - Terminating Instance:i-a55ff36c...
2015-08-04 10:06:27,518 - WARNING - Done.
2015-08-04 10:06:27,519 - WARNING - Terminating Instance:i-778629b2...
2015-08-04 10:06:27,750 - WARNING - Done.
2015-08-04 10:06:27,750 - WARNING - Terminating Instance:i-39278bf0...
2015-08-04 10:06:27,979 - WARNING - Done.
2015-08-04 10:06:27,980 - WARNING - Terminating Instance:i-385ff3f1...
2015-08-04 10:06:28,194 - WARNING - Done.
2015-08-04 10:06:28,195 - WARNING - Terminating Instance:i-cc0b6d04...
2015-08-04 10:06:28,402 - WARNING - Done.
2015-08-04 10:06:28,402 - WARNING - Terminating Instance:i-b0a80775...
2015-08-04 10:06:28,622 - WARNING - Done.
2015-08-04 10:06:28,622 - WARNING - Terminating Instance:i-420b6d8a...
2015-08-04 10:06:28,814 - WARNING - Done.
2015-08-04 10:06:28,814 - WARNING - Terminating Instance:i-7bab04be...
2015-08-04 10:06:29,039 - WARNING - Done.
2015-08-04 10:06:29,039 - WARNING - Terminating Instance:i-010aa6c8...
2015-08-04 10:06:29,260 - WARNING - Done.
2015-08-04 10:06:29,260 - WARNING - Terminating Instance:i-56852a93...
2015-08-04 10:06:29,479 - WARNING - Done.
2015-08-04 10:06:29,479 - WARNING - Terminating Instance:i-ca086e02...
2015-08-04 10:06:29,703 - WARNING - Done.
2015-08-04 10:06:29,703 - WARNING - Terminating Instance:i-8704a84e...
2015-08-04 10:06:30,056 - WARNING - Done.
2015-08-04 10:06:30,056 - WARNING - Terminating Instance:i-150fa3dc...
2015-08-04 10:06:30,284 - WARNING - Done.
2015-08-04 10:06:30,284 - WARNING - Terminating Instance:i-080a6cc0...
2015-08-04 10:06:30,479 - WARNING - Done.
2015-08-04 10:06:30,479 - WARNING - Terminating Instance:i-d7086e1f...
2015-08-04 10:06:30,813 - WARNING - Done.
2015-08-04 10:06:30,813 - WARNING - Terminating Instance:i-afa30c6a...
2015-08-04 10:06:31,025 - WARNING - Done.
2015-08-04 10:06:31,025 - WARNING - Terminating Instance:i-3e832cfb...
2015-08-04 10:06:31,232 - WARNING - Done.
2015-08-04 10:06:31,232 - WARNING - Terminating Instance:i-ce58f407...
2015-08-04 10:06:31,618 - WARNING - Done.
2015-08-04 10:06:31,618 - WARNING - Terminating Instance:i-b1a30c74...
2015-08-04 10:06:31,936 - WARNING - Done.
2015-08-04 10:06:31,936 - WARNING - Terminating Instance:i-400aa689...
2015-08-04 10:06:32,226 - WARNING - Done.
2015-08-04 10:06:32,227 - WARNING - Terminating Instance:i-a60ba76f...
2015-08-04 10:06:32,466 - WARNING - Done.
2015-08-04 10:06:32,466 - WARNING - Terminating Instance:i-3e278bf7...
2015-08-04 10:06:32,721 - WARNING - Done.
2015-08-04 10:06:32,721 - WARNING - Terminating Instance:i-56842b93...
2015-08-04 10:06:32,996 - WARNING - Done.
2015-08-04 10:06:32,996 - WARNING - Terminating Instance:i-db600613...
2015-08-04 10:06:33,231 - WARNING - Done.
2015-08-04 10:06:33,232 - WARNING - Terminating Instance:i-76238fbf...
2015-08-04 10:06:33,547 - WARNING - Done.
2015-08-04 10:06:33,547 - WARNING - Terminating Instance:i-565ff39f...
2015-08-04 10:06:33,798 - WARNING - Done.
2015-08-04 10:06:33,798 - WARNING - Terminating Instance:i-df086e17...
2015-08-04 10:06:34,056 - WARNING - Done.
2015-08-04 10:06:34,056 - WARNING - Terminating Instance:i-e407ab2d...
2015-08-04 10:06:34,289 - WARNING - Done.
2015-08-04 10:06:34,289 - WARNING - Terminating Instance:i-daa20d1f...
2015-08-04 10:06:34,526 - WARNING - Done.
2015-08-04 10:06:34,526 - WARNING - Terminating Instance:i-105df1d9...
2015-08-04 10:06:34,786 - WARNING - Done.
2015-08-04 10:06:34,786 - WARNING - Terminating Instance:i-5b3a9692...
2015-08-04 10:06:35,131 - WARNING - Done.
2015-08-04 10:06:35,131 - WARNING - Terminating Instance:i-3aa30cff...
2015-08-04 10:06:35,342 - WARNING - Done.
2015-08-04 10:06:35,343 - WARNING - Terminating Instance:i-9b05a952...
2015-08-04 10:06:35,595 - WARNING - Done.
2015-08-04 10:06:35,595 - WARNING - Terminating Instance:i-51258998...
2015-08-04 10:06:35,858 - WARNING - Done.
2015-08-04 10:06:35,858 - WARNING - Terminating Instance:i-5941ed90...
2015-08-04 10:06:36,076 - WARNING - Done.
2015-08-04 10:06:36,077 - WARNING - Terminating Instance:i-e707ab2e...
2015-08-04 10:06:36,394 - WARNING - Done.
2015-08-04 10:06:36,394 - WARNING - Terminating Instance:i-a30ba76a...
2015-08-04 10:06:36,650 - WARNING - Done.
2015-08-04 10:06:36,650 - WARNING - Terminating Instance:i-580ba791...
2015-08-04 10:06:36,894 - WARNING - Done.
2015-08-04 10:06:36,894 - WARNING - Terminating Instance:i-ab5ff362...
2015-08-04 10:06:37,109 - WARNING - Done.
2015-08-04 10:06:37,109 - WARNING - Terminating Instance:i-bfa9067a...
2015-08-04 10:06:37,320 - WARNING - Done.
2015-08-04 10:06:37,320 - WARNING - Terminating Instance:i-3a42eef3...
2015-08-04 10:06:37,554 - WARNING - Done.
2015-08-04 10:06:37,554 - WARNING - Terminating Instance:i-ee3a9627...
2015-08-04 10:06:37,776 - WARNING - Done.
2015-08-04 10:06:37,776 - WARNING - Terminating Instance:i-7aab04bf...
2015-08-04 10:06:38,134 - WARNING - Done.
2015-08-04 10:06:38,134 - WARNING - Terminating Instance:i-5d04a894...
2015-08-04 10:06:38,361 - WARNING - Done.
2015-08-04 10:06:38,361 - WARNING - Terminating Instance:i-805df149...
2015-08-04 10:06:38,563 - WARNING - Done.
2015-08-04 10:06:38,564 - WARNING - Terminating Instance:i-135df1da...
2015-08-04 10:06:38,860 - WARNING - Done.
2015-08-04 10:06:38,860 - WARNING - Terminating Instance:i-3ea30cfb...
2015-08-04 10:06:39,106 - WARNING - Done.
2015-08-04 10:06:39,107 - WARNING - Terminating Instance:i-0d5df1c4...
2015-08-04 10:06:39,330 - WARNING - Done.
2015-08-04 10:06:39,331 - WARNING - Terminating Instance:i-f8238f31...
2015-08-04 10:06:39,625 - WARNING - Done.
2015-08-04 10:06:39,625 - WARNING - Terminating Instance:i-5a0ba793...
2015-08-04 10:06:39,969 - WARNING - Done.
2015-08-04 10:06:39,970 - WARNING - Terminating Instance:i-76268abf...
2015-08-04 10:06:40,228 - WARNING - Done.
2015-08-04 10:06:40,228 - WARNING - Terminating Instance:i-dd842b18...
2015-08-04 10:06:40,447 - WARNING - Done.
2015-08-04 10:06:40,447 - WARNING - Terminating Instance:i-b2832c77...
2015-08-04 10:06:40,778 - WARNING - Done.
2015-08-04 10:06:40,779 - WARNING - Terminating Instance:i-59208c90...
2015-08-04 10:06:41,118 - WARNING - Done.
2015-08-04 10:06:41,118 - WARNING - Terminating Instance:i-0b0a6cc3...
2015-08-04 10:06:41,330 - WARNING - Done.
2015-08-04 10:06:41,330 - WARNING - Terminating Instance:i-b7a80772...
2015-08-04 10:06:41,575 - WARNING - Done.
2015-08-04 10:06:41,575 - WARNING - Terminating Instance:i-a6862963...
2015-08-04 10:06:41,839 - WARNING - Done.
2015-08-04 10:06:41,840 - WARNING - Terminating Instance:i-400b6d88...
2015-08-04 10:06:42,158 - WARNING - Done.
2015-08-04 10:06:42,159 - WARNING - Terminating Instance:i-e5aa0520...
2015-08-04 10:06:42,416 - WARNING - Done.
2015-08-04 10:06:42,416 - WARNING - Terminating Instance:i-d9a20d1c...
2015-08-04 10:06:42,707 - WARNING - Done.
2015-08-04 10:06:42,707 - WARNING - Terminating Instance:i-d0086e18...
2015-08-04 10:06:42,921 - WARNING - Done.
2015-08-04 10:06:42,921 - WARNING - Terminating Instance:i-5f852a9a...
2015-08-04 10:06:43,193 - WARNING - Done.
2015-08-04 10:06:43,194 - WARNING - Terminating Instance:i-9238945b...
2015-08-04 10:06:43,416 - WARNING - Done.
2015-08-04 10:06:43,417 - WARNING - Terminating Instance:i-b8a9067d...
2015-08-04 10:06:43,647 - WARNING - Done.
2015-08-04 10:06:43,647 - WARNING - Terminating Instance:i-eb07ab22...
2015-08-04 10:06:43,852 - WARNING - Done.
2015-08-04 10:06:43,852 - WARNING - Terminating Instance:i-75ab04b0...
2015-08-04 10:06:44,069 - WARNING - Done.
2015-08-04 10:06:44,069 - WARNING - Terminating Instance:i-7f09a5b6...
2015-08-04 10:06:44,289 - WARNING - Done.
2015-08-04 10:06:44,289 - WARNING - Terminating Instance:i-410b6d89...
2015-08-04 10:06:44,593 - WARNING - Done.
2015-08-04 10:06:44,593 - WARNING - Terminating Instance:i-420aa68b...
2015-08-04 10:06:44,979 - WARNING - Done.
2015-08-04 10:06:44,979 - WARNING - Terminating Instance:i-9c399555...
2015-08-04 10:06:45,281 - WARNING - Done.
2015-08-04 10:06:45,281 - WARNING - Terminating Instance:i-c958f400...
2015-08-04 10:06:45,484 - WARNING - Done.
2015-08-04 10:06:45,484 - WARNING - Terminating Instance:i-6b5ef2a2...
2015-08-04 10:06:45,701 - WARNING - Done.
2015-08-04 10:06:45,701 - WARNING - Terminating Instance:i-a3228e6a...
2015-08-04 10:06:45,921 - WARNING - Done.
2015-08-04 10:06:45,922 - WARNING - Terminating Instance:i-df842b1a...
2015-08-04 10:06:46,164 - WARNING - Done.
2015-08-04 10:06:46,164 - WARNING - Terminating Instance:i-5e04a897...
2015-08-04 10:06:46,372 - WARNING - Done.
2015-08-04 10:06:46,372 - WARNING - Terminating Instance:i-a542ee6c...
2015-08-04 10:06:46,589 - WARNING - Done.
2015-08-04 10:06:46,589 - WARNING - Terminating Instance:i-51862994...
2015-08-04 10:06:46,804 - WARNING - Done.
2015-08-04 10:06:46,804 - WARNING - Terminating Instance:i-d8389411...
2015-08-04 10:06:47,026 - WARNING - Done.
2015-08-04 10:06:47,026 - WARNING - Terminating Instance:i-d1086e19...
2015-08-04 10:06:47,247 - WARNING - Done.
2015-08-04 10:06:47,247 - WARNING - Terminating Instance:i-05238fcc...
2015-08-04 10:06:47,517 - WARNING - Done.
2015-08-04 10:06:47,517 - WARNING - Terminating Instance:i-4b238f82...
2015-08-04 10:06:47,745 - WARNING - Done.
2015-08-04 10:06:47,746 - WARNING - Terminating Instance:i-195df1d0...
2015-08-04 10:06:48,075 - WARNING - Done.
2015-08-04 10:06:48,075 - WARNING - Terminating Instance:i-490b6d81...
2015-08-04 10:06:48,380 - WARNING - Done.
2015-08-04 10:06:48,380 - WARNING - Terminating Instance:i-bd208c74...
2015-08-04 10:06:49,167 - WARNING - Done.
2015-08-04 10:06:49,167 - WARNING - Terminating Instance:i-4c5ef285...
2015-08-04 10:06:49,415 - WARNING - Done.
2015-08-04 10:06:49,415 - WARNING - Terminating Instance:i-9c05a955...
2015-08-04 10:06:49,677 - WARNING - Done.
2015-08-04 10:06:49,677 - WARNING - Terminating Instance:i-d8842b1d...
2015-08-04 10:06:49,900 - WARNING - Done.
2015-08-04 10:06:49,900 - WARNING - Terminating Instance:i-a00ba769...
2015-08-04 10:06:50,114 - WARNING - Done.
2015-08-04 10:06:50,114 - WARNING - Terminating Instance:i-ae0fa367...
2015-08-04 10:06:50,345 - WARNING - Done.
2015-08-04 10:06:50,345 - WARNING - Terminating Instance:i-1f2589d6...
2015-08-04 10:06:50,573 - WARNING - Done.
2015-08-04 10:06:50,573 - WARNING - Terminating Instance:i-f00b6d38...
2015-08-04 10:06:50,783 - WARNING - Done.
2015-08-04 10:06:50,783 - WARNING - Terminating Instance:i-3b42eef2...
2015-08-04 10:06:51,000 - WARNING - Done.
2015-08-04 10:06:51,000 - WARNING - Terminating Instance:i-9d05a954...
2015-08-04 10:06:51,232 - WARNING - Done.
2015-08-04 10:06:51,232 - WARNING - Terminating Instance:i-b3a80776...
2015-08-04 10:06:51,421 - WARNING - Done.
2015-08-04 10:06:51,421 - WARNING - Terminating Instance:i-5d0ba794...
2015-08-04 10:06:51,637 - WARNING - Done.
2015-08-04 10:06:51,638 - WARNING - Terminating Instance:i-1004a8d9...
2015-08-04 10:06:51,845 - WARNING - Done.
2015-08-04 10:06:51,845 - WARNING - Terminating Instance:i-99399550...
2015-08-04 10:06:52,060 - WARNING - Done.
2015-08-04 10:06:52,061 - WARNING - Terminating Instance:i-baa20d7f...
2015-08-04 10:06:52,289 - WARNING - Done.
2015-08-04 10:06:52,289 - WARNING - Terminating Instance:i-8b238f42...
2015-08-04 10:06:52,499 - WARNING - Done.
2015-08-04 10:06:52,499 - WARNING - Terminating Instance:i-79ab04bc...
2015-08-04 10:06:52,738 - WARNING - Done.
2015-08-04 10:06:52,738 - WARNING - Terminating Instance:i-b8a20d7d...
2015-08-04 10:06:52,937 - WARNING - Done.
2015-08-04 10:06:52,937 - WARNING - Terminating Instance:i-5b0ba792...
2015-08-04 10:06:53,166 - WARNING - Done.
2015-08-04 10:06:53,166 - WARNING - Terminating Instance:i-cc04a805...
2015-08-04 10:06:53,576 - WARNING - Done.
2015-08-04 10:06:53,576 - WARNING - Terminating Instance:i-5a04a893...
2015-08-04 10:06:53,802 - WARNING - Done.
2015-08-04 10:06:53,802 - WARNING - Terminating Instance:i-070a6ccf...
2015-08-04 10:06:54,023 - WARNING - Done.
2015-08-04 10:06:54,023 - WARNING - Terminating Instance:i-2d3f93e4...
2015-08-04 10:06:54,237 - WARNING - Done.
2015-08-04 10:06:54,237 - WARNING - Terminating Instance:i-1f3894d6...
2015-08-04 10:06:54,502 - WARNING - Done.
2015-08-04 10:06:54,502 - WARNING - Terminating Instance:i-fc42ee35...
2015-08-04 10:06:54,736 - WARNING - Done.
2015-08-04 10:06:54,736 - WARNING - Terminating Instance:i-17a20dd2...
2015-08-04 10:06:54,970 - WARNING - Done.
2015-08-04 10:06:54,970 - WARNING - Terminating Instance:i-c9086e01...
2015-08-04 10:06:55,187 - WARNING - Done.
2015-08-04 10:06:55,187 - WARNING - Terminating Instance:i-cb086e03...
2015-08-04 10:06:55,388 - WARNING - Done.
2015-08-04 10:06:55,388 - WARNING - Terminating Instance:i-4f5ff386...
2015-08-04 10:06:55,598 - WARNING - Done.
2015-08-04 10:06:55,598 - WARNING - Terminating Instance:i-a85ff361...
2015-08-04 10:06:55,884 - WARNING - Done.
2015-08-04 10:06:55,884 - WARNING - Terminating Instance:i-0aa20dcf...
2015-08-04 10:06:56,121 - WARNING - Done.
2015-08-04 10:06:56,121 - WARNING - Terminating Instance:i-523a969b...
2015-08-04 10:06:56,390 - WARNING - Done.
2015-08-04 10:06:56,390 - WARNING - Terminating Instance:i-0f238fc6...
2015-08-04 10:06:56,608 - WARNING - Done.
2015-08-04 10:06:56,608 - WARNING - Terminating Instance:i-dba20d1e...
2015-08-04 10:06:56,834 - WARNING - Done.
2015-08-04 10:06:56,835 - WARNING - Terminating Instance:i-cf0b6d07...
2015-08-04 10:06:57,035 - WARNING - Done.
2015-08-04 10:06:57,035 - WARNING - Terminating Instance:i-f70b6d3f...
2015-08-04 10:06:57,216 - WARNING - Done.
2015-08-04 10:06:57,217 - WARNING - Terminating Instance:i-c90b6d01...
2015-08-04 10:06:57,410 - WARNING - Done.
2015-08-04 10:06:57,410 - WARNING - Terminating Instance:i-b338947a...
2015-08-04 10:06:57,623 - WARNING - Done.
2015-08-04 10:06:57,623 - WARNING - Terminating Instance:i-042589cd...
2015-08-04 10:06:57,835 - WARNING - Done.
2015-08-04 10:06:57,835 - WARNING - Terminating Instance:i-74ab04b1...
2015-08-04 10:06:58,058 - WARNING - Done.
2015-08-04 10:06:58,058 - WARNING - Terminating Instance:i-515ff398...
2015-08-04 10:06:58,268 - WARNING - Done.
2015-08-04 10:06:58,268 - WARNING - Terminating Instance:i-a208a46b...
2015-08-04 10:06:58,492 - WARNING - Done.
2015-08-04 10:06:58,493 - WARNING - Terminating Instance:i-c258f40b...
2015-08-04 10:06:58,706 - WARNING - Done.
2015-08-04 10:06:58,706 - WARNING - Terminating Instance:i-e2842b27...
2015-08-04 10:06:58,944 - WARNING - Done.
2015-08-04 10:06:58,944 - WARNING - Terminating Instance:i-e3842b26...
2015-08-04 10:07:00,093 - WARNING - Done.
2015-08-04 10:07:00,093 - WARNING - Terminating Instance:i-aa208c63...
2015-08-04 10:07:01,303 - WARNING - Done.
2015-08-04 10:07:01,303 - WARNING - Terminating Instance:i-5641ed9f...
2015-08-04 10:07:01,538 - WARNING - Done.
2015-08-04 10:07:01,538 - WARNING - Terminating Instance:i-31a30cf4...
2015-08-04 10:07:02,907 - WARNING - Done.
2015-08-04 10:07:02,907 - WARNING - Terminating Instance:i-d2086e1a...
2015-08-04 10:07:03,162 - WARNING - Done.
2015-08-04 10:07:03,162 - WARNING - Terminating Instance:i-71aa05b4...
2015-08-04 10:07:03,440 - WARNING - Done.
2015-08-04 10:07:03,440 - WARNING - Terminating Instance:i-4b086e83...
2015-08-04 10:07:03,843 - WARNING - Done.
2015-08-04 10:07:03,844 - WARNING - Terminating Instance:i-02238fcb...
2015-08-04 10:07:09,299 - WARNING - Done.
2015-08-04 10:07:09,299 - WARNING - Terminating Instance:i-c042ee09...
2015-08-04 10:07:09,617 - WARNING - Done.
2015-08-04 10:07:09,617 - WARNING - Terminating Instance:i-a10ba768...
2015-08-04 10:07:09,833 - WARNING - Done.
2015-08-04 10:07:09,833 - WARNING - Terminating Instance:i-b4268a7d...
2015-08-04 10:07:10,056 - WARNING - Done.
2015-08-04 10:07:10,057 - WARNING - Terminating Instance:i-a40ba76d...
2015-08-04 10:07:10,379 - WARNING - Done.
2015-08-04 10:07:10,379 - WARNING - Terminating Instance:i-ea3a9623...
2015-08-04 10:07:12,146 - WARNING - Done.
2015-08-04 10:07:12,146 - WARNING - Terminating Instance:i-460b6d8e...
2015-08-04 10:07:12,457 - WARNING - Done.
2015-08-04 10:07:12,457 - WARNING - Terminating Instance:i-5b208c92...
2015-08-04 10:07:12,673 - WARNING - Done.
2015-08-04 10:07:12,673 - WARNING - Terminating Instance:i-5225899b...
2015-08-04 10:07:12,967 - WARNING - Done.
2015-08-04 10:07:12,967 - WARNING - Terminating Instance:i-75238fbc...
2015-08-04 10:07:13,966 - WARNING - Done.
2015-08-04 10:07:13,966 - WARNING - Terminating Instance:i-4f5ef286...
2015-08-04 10:07:14,171 - WARNING - Done.
2015-08-04 10:07:14,171 - WARNING - Terminating Instance:i-1e3894d7...
2015-08-04 10:07:23,500 - WARNING - Done.
2015-08-04 10:07:23,501 - WARNING - Terminating Instance:i-4a5ef283...
2015-08-04 10:07:23,729 - WARNING - Done.
2015-08-04 10:07:23,730 - WARNING - Terminating Instance:i-c85ef201...
2015-08-04 10:07:23,952 - WARNING - Done.
2015-08-04 10:07:23,953 - WARNING - Terminating Instance:i-4b5ff382...
2015-08-04 10:07:24,208 - WARNING - Done.
2015-08-04 10:07:24,209 - WARNING - Terminating Instance:i-5c04a895...
2015-08-04 10:07:24,505 - WARNING - Done.
2015-08-04 10:07:24,505 - WARNING - Terminating Instance:i-38a20dfd...
2015-08-04 10:07:24,750 - WARNING - Done.
2015-08-04 10:07:24,750 - WARNING - Terminating Instance:i-ee06aa27...
2015-08-04 10:07:25,050 - WARNING - Done.
2015-08-04 10:07:25,051 - WARNING - Terminating Instance:i-505ff399...
2015-08-04 10:07:25,249 - WARNING - Done.
2015-08-04 10:07:25,249 - WARNING - Terminating Instance:i-4e5ff387...
2015-08-04 10:07:25,461 - WARNING - Done.
2015-08-04 10:07:25,461 - WARNING - Terminating Instance:i-090a6cc1...
2015-08-04 10:07:25,657 - WARNING - Done.
2015-08-04 10:07:25,657 - WARNING - Terminating Instance:i-cb0b6d03...
2015-08-04 10:07:25,838 - WARNING - Done.
2015-08-04 10:07:25,838 - WARNING - Terminating Instance:i-aaa30c6f...
2015-08-04 10:07:27,024 - WARNING - Done.
2015-08-04 10:07:27,024 - WARNING - Terminating Instance:i-d6218d1f...
2015-08-04 10:07:27,308 - WARNING - Done.
2015-08-04 10:07:27,309 - WARNING - Terminating Instance:i-e607ab2f...
2015-08-04 10:07:27,523 - WARNING - Done.
2015-08-04 10:07:27,523 - WARNING - Terminating Instance:i-65238fac...
2015-08-04 10:07:27,743 - WARNING - Done.
2015-08-04 10:07:27,743 - WARNING - Terminating Instance:i-d4086e1c...
2015-08-04 10:07:28,969 - WARNING - Done.
2015-08-04 10:07:28,969 - WARNING - Terminating Instance:i-efa9062a...
2015-08-04 10:07:29,167 - WARNING - Done.
2015-08-04 10:07:29,167 - WARNING - Terminating Instance:i-85309c4c...
2015-08-04 10:07:29,420 - WARNING - Done.
2015-08-04 10:07:29,420 - WARNING - Terminating Instance:i-0c5df1c5...
2015-08-04 10:07:29,658 - WARNING - Done.
2015-08-04 10:07:29,659 - WARNING - Terminating Instance:i-6a5ef2a3...
2015-08-04 10:07:30,368 - WARNING - Done.
2015-08-04 10:07:30,368 - WARNING - Terminating Instance:i-b6a80773...
2015-08-04 10:07:31,348 - WARNING - Done.
2015-08-04 10:07:31,348 - WARNING - Terminating Instance:i-9a05a953...
2015-08-04 10:07:31,618 - WARNING - Done.
2015-08-04 10:07:31,618 - WARNING - Terminating Instance:i-c50b6d0d...
2015-08-04 10:07:31,842 - WARNING - Done.
2015-08-04 10:07:31,842 - WARNING - Terminating Instance:i-3da30cf8...
2015-08-04 10:07:32,085 - WARNING - Done.
2015-08-04 10:07:32,085 - WARNING - Terminating Instance:i-7e268ab7...
2015-08-04 10:07:32,312 - WARNING - Done.
2015-08-04 10:07:32,312 - WARNING - Terminating Instance:i-1704a8de...
2015-08-04 10:07:36,325 - WARNING - Done.
2015-08-04 10:07:36,325 - WARNING - Terminating Instance:i-c858f401...
2015-08-04 10:07:36,545 - WARNING - Done.
2015-08-04 10:07:36,545 - WARNING - Terminating Instance:i-125df1db...
2015-08-04 10:07:37,234 - WARNING - Done.
2015-08-04 10:07:37,234 - WARNING - Terminating Instance:i-708629b5...
2015-08-04 10:07:37,468 - WARNING - Done.
2015-08-04 10:07:37,469 - WARNING - Terminating Instance:i-b0a20d75...
2015-08-04 10:07:37,692 - WARNING - Done.
2015-08-04 10:07:37,693 - WARNING - Terminating Instance:i-5b04a892...
2015-08-04 10:07:37,920 - WARNING - Done.
2015-08-04 10:07:37,920 - WARNING - Terminating Instance:i-c0389409...
2015-08-04 10:07:39,156 - WARNING - Done.
2015-08-04 10:07:39,157 - WARNING - Terminating Instance:i-d7218d1e...
2015-08-04 10:07:39,369 - WARNING - Done.
2015-08-04 10:07:39,369 - WARNING - Terminating Instance:i-dc852a19...
2015-08-04 10:07:39,603 - WARNING - Done.
2015-08-04 10:07:39,603 - WARNING - Terminating Instance:i-020a6cca...
2015-08-04 10:07:54,374 - WARNING - Done.
2015-08-04 10:07:54,375 - WARNING - Terminating Instance:i-c5218d0c...
2015-08-04 10:07:54,577 - WARNING - Done.
2015-08-04 10:07:54,577 - WARNING - Terminating Instance:i-ada30c68...
2015-08-04 10:07:54,796 - WARNING - Done.
2015-08-04 10:07:54,796 - WARNING - Terminating Instance:i-70aa05b5...
2015-08-04 10:07:55,064 - WARNING - Done.
2015-08-04 10:07:55,064 - WARNING - Terminating Instance:i-a75df16e...
2015-08-04 10:07:55,276 - WARNING - Done.
2015-08-04 10:07:55,277 - WARNING - Terminating Instance:i-5a3a9693...
2015-08-04 10:07:55,518 - WARNING - Done.
2015-08-04 10:07:55,518 - WARNING - Terminating Instance:i-5c852a99...
2015-08-04 10:07:55,723 - WARNING - Done.
2015-08-04 10:07:55,724 - WARNING - Terminating Instance:i-34278bfd...
2015-08-04 10:07:55,982 - WARNING - Done.
2015-08-04 10:07:55,982 - WARNING - Terminating Instance:i-b238947b...
2015-08-04 10:07:56,223 - WARNING - Done.
2015-08-04 10:07:56,223 - WARNING - Terminating Instance:i-eca90629...
2015-08-04 10:07:56,436 - WARNING - Done.
2015-08-04 10:07:56,437 - WARNING - Terminating Instance:i-e93a9620...
2015-08-04 10:07:56,651 - WARNING - Done.
2015-08-04 10:07:56,652 - WARNING - Terminating Instance:i-d6852a13...
2015-08-04 10:07:56,909 - WARNING - Done.
2015-08-04 10:07:56,909 - WARNING - Terminating Instance:i-3ba20dfe...
2015-08-04 10:07:57,274 - WARNING - Done.
2015-08-04 10:07:57,274 - WARNING - Terminating Instance:i-46319d8f...
2015-08-04 10:07:57,495 - WARNING - Done.
2015-08-04 10:07:57,495 - WARNING - Terminating Instance:i-555ff39c...
2015-08-04 10:07:57,746 - WARNING - Done.
2015-08-04 10:07:57,746 - WARNING - Terminating Instance:i-d5086e1d...
2015-08-04 10:07:57,960 - WARNING - Done.
2015-08-04 10:07:57,961 - WARNING - Terminating Instance:i-b2a80777...
2015-08-04 10:07:58,217 - WARNING - Done.
2015-08-04 10:07:58,217 - WARNING - Terminating Instance:i-513a9698...
2015-08-04 10:07:58,445 - WARNING - Done.
2015-08-04 10:07:58,445 - WARNING - Terminating Instance:i-430b6d8b...
2015-08-04 10:07:58,667 - WARNING - Done.
2015-08-04 10:07:58,668 - WARNING - Terminating Instance:i-b5268a7c...
2015-08-04 10:07:58,888 - WARNING - Done.
2015-08-04 10:07:58,888 - WARNING - Terminating Instance:i-53208c9a...
2015-08-04 10:07:59,215 - WARNING - Done.
2015-08-04 10:07:59,215 - WARNING - Terminating Instance:i-bea9067b...
2015-08-04 10:07:59,445 - WARNING - Done.
2015-08-04 10:07:59,445 - WARNING - Terminating Instance:i-58842b9d...
2015-08-04 10:07:59,705 - WARNING - Done.
2015-08-04 10:07:59,706 - WARNING - Terminating Instance:i-9905a950...
2015-08-04 10:08:00,120 - WARNING - Done.
2015-08-04 10:08:00,120 - WARNING - Terminating Instance:i-e507ab2c...
2015-08-04 10:08:08,217 - WARNING - Done.
2015-08-04 10:08:08,217 - WARNING - Terminating Instance:i-733995ba...
2015-08-04 10:08:08,449 - WARNING - Done.
2015-08-04 10:08:08,449 - WARNING - Terminating Instance:i-e4842b21...
2015-08-04 10:08:08,662 - WARNING - Done.

(aws_manager)[buildduty@aws-manager2.srv.releng.scl3.mozilla.com cloud-tools]$ python scripts/get_spot_amis.py
bld-linux64, us-east-1: ami-d73b9abc (spot-bld-linux64-2015-08-03-09-06, ebs)
try-linux64, us-east-1: ami-bd3796d6 (spot-try-linux64-2015-08-03-08-46, ebs)
tst-linux64, us-east-1: ami-03389968 (spot-tst-linux64-2015-08-03-09-06, ebs)
tst-linux32, us-east-1: ami-f9389992 (spot-tst-linux32-2015-08-03-09-11, ebs)
tst-emulator64, us-east-1: ami-5b3a9b30 (spot-tst-emulator64-2015-08-03-09-28, ebs)
bld-linux64, us-west-2: ami-5f2b226f (spot-bld-linux64-2015-08-02-09-11, ebs)
try-linux64, us-west-2: ami-99d6dfa9 (spot-try-linux64-2015-08-03-08-46, ebs)
tst-linux64, us-west-2: ami-95e9e0a5 (spot-tst-linux64-2015-08-03-09-06, ebs)
tst-linux32, us-west-2: ami-67b4bd57 (spot-tst-linux32-2015-08-03-09-11, ebs)
tst-emulator64, us-west-2: ami-5fe8e16f (spot-tst-emulator64-2015-08-03-09-28, ebs)
Blocks: 1189948
Looks like reverting the AMI worked. I've reopened Gaia, but I'm holding off on the other Gecko trees until more AWS instances have spun up to avoid huge backlogs off the bat.
Yeah, the correct fix here was to backout the change to the gtk tarball that added setup.sh, as the setup.sh script is broken.  Apparently that change landed before the tooltool changes were deployed, so the bug was hidden until the tooltool changes landed.

The change that should be backed out is for bug 1188780:
  http://hg.mozilla.org/mozilla-central/rev/a9d3df6e14e0
and we can re-land this puppet change once that's backed out.

I suspect that setup.sh just needs more linker paths set?
All trees reopened.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Let's repurpose this bug to fix the underlying issue that is that tooltool is not run in the mock environment.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Severity: blocker → normal
No longer blocks: 1180624
Component: Buildduty → General Automation
QA Contact: bugspam.Callek → catlee
Whiteboard: tooltool mock
Blocks: 1242641
Assignee: nobody → nthomas
Summary: Builds fail with Halting on failure while running ['sh', '/builds/slave/b2g-in-lx-00000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh' → Run tooltool inside mock
This is not tested, because that's what production is for (one less new year's resolution to stick to). There is the possibility that we'll be running mock_mozilla in the wrong cwd, and get the --cwd wrong, but I've tried to avoid that. List of affected builders to follow.
Attachment #8712421 - Flags: review?(bugspam.Callek)
The mac and windows builders change to MockCommand, but mock is disabled for them so they fall back to ShellCommand. Valgrind builds use ScriptFactory, otherwise it's all subclasses of MercurialBuildFactory.
Comment on attachment 8712421 [details] [diff] [review]
[buildbotcustom] Fix up buildbot style builds

Review of attachment 8712421 [details] [diff] [review]:
-----------------------------------------------------------------

Lets give this a shot
Attachment #8712421 - Flags: review?(bugspam.Callek) → review+
In production.
Followup fix for bogus use of mock_workdir which caused failures when not in mock:
https://hg.mozilla.org/build/buildbotcustom/rev/a4bc25f76f2d
https://hg.mozilla.org/build/buildbotcustom/rev/a212ce1d8529
This extends what we already have for some linux platforms across all linux and android (everywhere we have use_mock True).
Attachment #8712477 - Flags: review?(jlund)
Philor reported valgrinds hitting issues trying to run tooltool, ie
ERROR: chroot /builds/mock_mozilla/mozilla-centos6-x86_64/root/ not initialized!

Turns out we're trying to run tooltool in mock before setting up mock, hence this change. This should fix it up.
Attachment #8712478 - Flags: review?(jlund)
Comment on attachment 8712477 [details] [diff] [review]
[buildbot-configs] Copy in tooltool.py, and run it from in-mock location

Review of attachment 8712477 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +1821,2 @@
>              'tooltool_manifest_src': 'mobile/android/config/tooltool-manifests/android-x86/releng.manifest',
> +            'tooltool_script': ['/builds/tooltool.py'],

might be getting late for me but this looks like double entries in the same builder
Comment on attachment 8712478 [details] [diff] [review]
[buildbotcustom] Fix valgrind

Review of attachment 8712478 [details] [diff] [review]:
-----------------------------------------------------------------

sigh. :)
Attachment #8712478 - Flags: review?(jlund) → review+
Comment on attachment 8712487 [details] [diff] [review]
[buildbot-configs] Copy in tooltool.py, and run it from in-mock location (with TB too)

Review of attachment 8712487 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)
Attachment #8712487 - Flags: review?(jlund) → review+
Comment on attachment 8712487 [details] [diff] [review]
[buildbot-configs] Copy in tooltool.py, and run it from in-mock location (with TB too)

https://hg.mozilla.org/build/buildbot-configs/rev/10af9c3220aa
https://hg.mozilla.org/build/buildbot-configs/rev/99bb075221a7
Attachment #8712487 - Flags: checked-in+
Depends on: 1243251
This is working on Firefox m-b, and there are some running on m-r and m-esr38 and the comm branches to double check the status there is OK.

TODO 
- backport changes to mozharness-builds on m-a and newer
Blocks: 1245828
Morphing this to be buildbot-jobs only, as work for mozharness is happening in bug 1242641.
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Summary: Run tooltool inside mock → Run tooltool inside mock in buildbot jobs
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.