970552 - Do not use spot instances for some builders

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

We shouldn't use spot instances for some builders (PGO/release?) or/and branches (beta/release?).

Comment 1

[Rail Aliiev [:rail]]

Attachment: no_pgo_on_spots-buildbotcustom-2.diff

WCPGW?

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8374131 [details] [diff] [review]
no_pgo_on_spots-buildbotcustom-2.diff

https://hg.mozilla.org/build/buildbotcustom/rev/530a492013c9

Comment 3

[Rail Aliiev [:rail]]

in production

Comment 4

[Rail Aliiev [:rail]]

Attachment: tests.diff

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8374363 [details] [diff] [review]
tests.diff

https://hg.mozilla.org/build/buildbotcustom/rev/b7530258fc4d

Comment 6

[Rail Aliiev [:rail]]

2014-02-12 16:05:26-0800 [-] Error choosing next slave for builder 'release-mozilla-release-linux_repack_9/10', choosing randomly instead
2014-02-12 16:05:26-0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/python/context.py", line 37, in callWithContext
            return func(*args,**kw)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/twisted/enterprise/adbapi.py", line 429, in _runInteraction
            result = interaction(trans, *args, **kw)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_f23f5672becd_production_0.8-py2.7.egg/buildbot/process/builder.py", line 517, in _claim_buildreqs
            sb = self._choose_slave(available_slaves)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_f23f5672becd_production_0.8-py2.7.egg/buildbot/process/builder.py", line 548, in _choose_slave
            return self.nextSlave(self, available_slaves)
        --- <exception caught here> ---
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/misc.py", line 267, in _nextSlave
            return func(builder, available_slaves)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/misc.py", line 463, in _nextSlave_skip_spot
            valid.append(s)
        exceptions.IndexError: list index out of range

Additionally it would be great to avoid running any of release builds on spot instances because there may be no chance to get to the slave to debug some failure.

Comment 7

[Rail Aliiev [:rail]]

Attachment: [wip] nextSlave.diff

Comment 8

[Rail Aliiev [:rail]]

Attachment: nextSlave.diff

I think I found the issue. 

sorted(no_spot_slaves, _recentSort(builder))[-1] doesn't work for [], better to return None earlier.

Comment 9

[Armen [:armenzg]]

Live in production.

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8375265 [details] [diff] [review]
nextSlave.diff

https://hg.mozilla.org/build/buildbotcustom/rev/a03618388e2b

Comment 11

[Rail Aliiev [:rail]]

Attachment: non-unified.diff

+ non-unified

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8375658 [details] [diff] [review]
non-unified.diff

https://hg.mozilla.org/build/buildbotcustom/rev/5d08b50a1531

Comment 13

[Rail Aliiev [:rail]]

In production

Comment 14

[Chris AtLee [:catlee]]

Had to back this out. See https://bugzilla.mozilla.org/show_bug.cgi?id=980890#c11

Comment 15

[Rail Aliiev [:rail]]

(In reply to Chris AtLee [:catlee] from comment #14)
> Had to back this out. See
> https://bugzilla.mozilla.org/show_bug.cgi?id=980890#c11

http://hg.mozilla.org/build/buildbotcustom/rev/a55559e39a59

Comment 16

[Rail Aliiev [:rail]]

Attachment: kill-skip-spot.diff

Since the kill ratio for spot instances has been almost 0% since we landed the bidding improvements (see below), let's use spot instances everywhere except releases.

^bld-linux64
date, total jobs, jobs on spots, spot retries, o-d retries
2014-03-01, 1725, 1356 (78%), 2 (0%), 2 (0%)
2014-03-02, 1036, 762 (73%), 1 (0%), 0 (0%)
2014-03-03, 2564, 2046 (79%), 68 (3%), 0 (0%)
2014-03-04, 3263, 2636 (80%), 27 (1%), 1 (0%)
2014-03-05, 2987, 2306 (77%), 38 (1%), 2 (0%)
2014-03-06, 3456, 2688 (77%), 29 (1%), 1 (0%)
2014-03-07, 3003, 2425 (80%), 10 (0%), 1 (0%)
2014-03-08, 1303, 951 (72%), 0 (0%), 0 (0%)
2014-03-09, 998, 685 (68%), 0 (0%), 0 (0%)
2014-03-10, 2282, 1966 (86%), 15 (0%), 0 (0%)
2014-03-11, 2730, 2385 (87%), 2 (0%), 0 (0%)
2014-03-12, 2883, 2616 (90%), 9 (0%), 0 (0%)
2014-03-13, 3109, 2728 (87%), 3 (0%), 0 (0%)


It may sound blasphemous, but we can even reconsider our logic to avoid running retried jobs on spot instances! :)

Comment 17

[Rail Aliiev [:rail]]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=935533#c38

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 8391006 [details] [diff] [review]
kill-skip-spot.diff

Review of attachment 8391006 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, we could perhaps change it to run on spot if num_retries <= 1 instead of num_retries == 0

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8391006 [details] [diff] [review]
kill-skip-spot.diff

https://hg.mozilla.org/build/buildbotcustom/rev/1f3ed587680c

Comment 20

[Rail Aliiev [:rail]]

In production