1262239 - pulse_actions - Fail safely if the SETA information for a builder is not available

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Armen [:armenzg]]

This issue was probably hidden behind the issue that I currently fixed.

Apr 04 03:58:53 pulse-actions app/worker2.1: pulse_actions.handlers.treeherder_buildbot	 INFO:	 backfill action requested by [e-mail] on repo_name mozilla-inbound with job_id: 24909582 
Apr 04 03:58:54 pulse-actions app/worker2.1: mozci	 DEBUG:	 Buildername Windows 7 32-bit mozilla-inbound talos g1 is valid. 
Apr 04 03:58:54 pulse-actions app/worker2.1: Traceback (most recent call last): 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "pulse_actions/worker.py", line 79, in run_pulse 
Apr 04 03:58:54 pulse-actions app/worker2.1:     pulse.listen() 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 151, in listen 
Apr 04 03:58:54 pulse-actions app/worker2.1:     self._drain_events_loop() 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 198, in _drain_events_loop 
Apr 04 03:58:54 pulse-actions app/worker2.1:     self.connection.drain_events(timeout=self.timeout) 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events 
Apr 04 03:58:54 pulse-actions app/worker2.1:     return self.transport.drain_events(self.connection, **kwargs) 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events 
Apr 04 03:58:54 pulse-actions app/worker2.1:     return connection.drain_events(**kwargs) 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 326, in drain_events 
Apr 04 03:58:54 pulse-actions app/worker2.1:     return amqp_method(channel, args, content) 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/channel.py", line 1909, in _basic_deliver 
Apr 04 03:58:54 pulse-actions app/worker2.1:     fun(msg) 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 598, in _receive_callback 
Apr 04 03:58:54 pulse-actions app/worker2.1:     return on_m(message) if on_m else self.receive(decoded, message) 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 564, in receive 
Apr 04 03:58:54 pulse-actions app/worker2.1:     [callback(body, message) for callback in callbacks] 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "pulse_actions/worker.py", line 68, in handler_with_dry_run 
Apr 04 03:58:54 pulse-actions app/worker2.1:     event_handler(data, message, dry_run) 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/pulse_actions/handlers/route_functions.py", line 13, in route 
Apr 04 03:58:54 pulse-actions app/worker2.1:     treeherder_buildbot.on_buildbot_event(data, message, dry_run) 
Apr 04 03:58:54 pulse-actions app/worker2.1:   File "/app/pulse_actions/handlers/treeherder_buildbot.py", line 80, in on_buildbot_event 
Apr 04 03:58:54 pulse-actions app/worker2.1:     max_revisions=int(get_maxRevisions(buildername)), 
Apr 04 03:58:55 pulse-actions app/worker2.1:   File "/app/pulse_actions/utils/misc.py", line 19, in get_maxRevisions 
Apr 04 03:58:55 pulse-actions app/worker2.1:     return get_SETA_info(buildername)[0] 
Apr 04 03:58:55 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/mozci/platforms.py", line 143, in get_SETA_info 
Apr 04 03:58:55 pulse-actions app/worker2.1:     return SETA_DICT[buildername] 
Apr 04 03:58:55 pulse-actions app/worker2.1: KeyError: u'Windows 7 32-bit mozilla-inbound talos g1'

Comment 1

[Armen [:armenzg]]

Back in February we landed some code to determine the number of pushes SETA would skip for every given builder.
After fixing the last couple of issues this one became obvious.

The problem in here is that we did not have SETA values for certain builders (the case of talos).
I've refactored the code and fixed the issue in mozci:
https://github.com/mozilla/mozilla_ci_tools/commit/f2c31e15e19019e51598cd94e630c6dd64b470ad

I've also moved some logic from pulse_actions to mozci:
https://github.com/mozilla/pulse_actions/commit/fb9e1945bf1fd8386dfde90ab2b15737a392d77a

I've also included a new release of pushlog_client which is more reliable.

Here are all the failures we've seen in the last two days:
 Apr 04 15:35:07 pulse-actions app/worker2.1: Traceback (most recent call last): 
Apr 04 15:35:07 pulse-actions app/worker2.1: KeyError: u'WINNT 5.2 mozilla-inbound leak test spidermonkey_tier_1-plaindebug build' 
Apr 04 15:35:09 pulse-actions app/worker2.1: KeyError: u'WINNT 5.2 mozilla-inbound spidermonkey_tier_1-plain build' 
Apr 04 16:41:31 pulse-actions app/worker2.1: KeyError: u'Windows 7 32-bit mozilla-inbound talos g1' 
Apr 05 03:55:23 pulse-actions app/worker2.1: KeyError: u'Windows XP 32-bit mozilla-central pgo talos other-e10s' 
Apr 05 10:39:33 pulse-actions app/worker2.1: KeyError: u'Ubuntu HW 12.04 x64 mozilla-inbound talos dromaeojs' 
Apr 05 10:51:19 pulse-actions app/worker2.1: KeyError: u'Ubuntu HW 12.04 x64 mozilla-inbound talos dromaeojs' 
Apr 05 11:25:19 pulse-actions app/worker2.1: KeyError: u'Windows 7 32-bit mozilla-aurora pgo talos g1-e10s' 
Apr 05 14:52:29 pulse-actions app/worker2.1: KeyError: u'Windows 7 32-bit mozilla-inbound talos g1'