Closed Bug 1276914 Opened 9 years ago Closed 9 years ago

pulse_actions - Adding extra talos PGO logic is failing in certain occassions

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: armenzg, Assigned: martianwars)

Details

Attachments

(2 files)

mozci fix
53 bytes, text/x-github-pull-request
Details | Review
pulse_actions fix
48 bytes, text/x-github-pull-request
Details | Review
This bug doesn't make sense as Linux mozilla-inbound leak test build and Linux x86-64 try debug asan build are not new builders/ I don't know what's going on. In any case the raising of the exception has to have more information. It is hard to tell for which builder it failed. Maybe it was a badly downloaded allthethings.json? This also was a spur of failures: May 31 04:06:11 pulse-actions app/worker3.1: Traceback (most recent call last): May 31 04:06:11 pulse-actions app/worker3.1: raise MozciError May 31 04:06:11 pulse-actions app/worker3.1: MozciError May 31 04:06:48 pulse-actions app/worker3.1: Traceback (most recent call last): May 31 04:06:48 pulse-actions app/worker3.1: raise MozciError May 31 04:06:48 pulse-actions app/worker3.1: MozciError May 31 04:34:59 pulse-actions app/worker3.1: Traceback (most recent call last): May 31 04:34:59 pulse-actions app/worker3.1: raise MozciError May 31 04:34:59 pulse-actions app/worker3.1: MozciError May 31 05:31:13 pulse-actions app/worker3.1: Traceback (most recent call last): May 31 05:31:13 pulse-actions app/worker3.1: raise MozciError May 31 05:31:13 pulse-actions app/worker3.1: MozciError May 31 04:05:58 pulse-actions app/worker3.1: root INFO: Message {u'payload': {u'test_packages_url': u'http://archive.mozilla.org/pub/firefox/try-builds/jseward@mozilla.com-120a3e7b99a7027f715795b51673f26b0a21485c/try-linux64-asan-debug/firefox-49.0a1.en-US.linux-x86_64-asan.test_packages.json', u'slave': u'try-linux64-spot-430', u'tags': [], u'build_number': None, u'platform': u'linux64-asan', u'revision': u'120a3e7b99a7027f715795b51673f26b0a21485c', u'key': u'build.try-linux64-asan-debug.5577.log_uploaded', u'job_number': 5577, u'builddate': 1464662950, u'repack': None, u'status': 0, u'buildid': u'20160531024910', u'buildtype': u'debug', u'locale': u'en-US', u'timestamp': u'2016-05-31T11:05:56Z', u'locales': None, u'buildurl': u'http://archive.mozilla.org/pub/firefox/try-builds/jseward@mozilla.com-120a3e7b99a7027f715795b51673f26b0a21485c/try-linux64-asan-debug/firefox-49.0a1.en-US.linux-x86_64-asan.tar.bz2', u'version': None, u'release': u'120a3e7b99a7027f715795b51673f26b0a21485c', u'product': u'firefox', u'buildername': u'Linux x86-64 try debug asan build', u'symbols_url': None, u'testsurl': u'http://archive.mozilla.org/pub/firefox/try-builds/jseward@mozilla.com-120a3e7b99a7027f715795b51673f26b0a21485c/try-linux64-asan-debug/firefox-49.0a1.en-US.linux-x86_64-asan.gtest.tests.zip', u'logurl': u'http://archive.mozilla.org/pub/firefox/try-builds/jseward@mozilla.com-120a3e7b99a7027f715795b51673f26b0a21485c/try-linux64-asan-debug/try-linux64-asan-debug-bm78-try1-build5577.txt.gz', u'previous_buildid': None, u'tree': u'try'}, u'_meta': {u'serializer': u'json', u'exchange': u'exchange/build/normalized', u'sent': u'2016-05-31T04:05:57+01:00', u'routing_key': u'build.try.linux64-asan.debug.try-linux64-asan-debug'}}, took 0.000420093536377 seconds to execute May 31 04:06:11 pulse-actions app/worker3.1: Traceback (most recent call last): May 31 04:06:11 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 75, in run_listener May 31 04:06:11 pulse-actions app/worker3.1: consumer.listen() May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 151, in listen May 31 04:06:11 pulse-actions app/worker3.1: self._drain_events_loop() May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 198, in _drain_events_loop May 31 04:06:11 pulse-actions app/worker3.1: self.connection.drain_events(timeout=self.timeout) May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events May 31 04:06:11 pulse-actions app/worker3.1: return self.transport.drain_events(self.connection, **kwargs) May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events May 31 04:06:11 pulse-actions app/worker3.1: return connection.drain_events(**kwargs) May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 326, in drain_events May 31 04:06:11 pulse-actions app/worker3.1: return amqp_method(channel, args, content) May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/amqp/channel.py", line 1909, in _basic_deliver May 31 04:06:11 pulse-actions app/worker3.1: fun(msg) May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 598, in _receive_callback May 31 04:06:11 pulse-actions app/worker3.1: return on_m(message) if on_m else self.receive(decoded, message) May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 564, in receive May 31 04:06:11 pulse-actions app/worker3.1: [callback(body, message) for callback in callbacks] May 31 04:06:11 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 62, in handler_with_dry_run May 31 04:06:11 pulse-actions app/worker3.1: route(data, message, dry_run) May 31 04:06:11 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 52, in route May 31 04:06:11 pulse-actions app/worker3.1: talos.on_event(data, message, dry_run) May 31 04:06:11 pulse-actions app/worker3.1: File "/app/pulse_actions/handlers/talos.py", line 31, in on_event May 31 04:06:11 pulse-actions app/worker3.1: info = get_buildername_metadata(buildername) May 31 04:06:11 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozci/platforms.py", line 245, in get_buildername_metadata May 31 04:06:11 pulse-actions app/worker3.1: raise MozciError May 31 04:06:11 pulse-actions app/worker3.1: MozciError May 31 04:06:39 pulse-actions app/worker3.1: root INFO: Message {u'payload': {u'test_packages_url': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1464690936/firefox-49.0a1.en-US.linux-i686.test_packages.json', u'slave': u'bld-linux64-spot-1058', u'tags': [], u'build_number': None, u'platform': u'linux', u'product': u'firefox', u'logurl': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1464690936/mozilla-inbound-linux-debug-bm72-build1-build1717.txt.gz', u'key': u'build.mozilla-inbound-linux-debug.1717.log_uploaded', u'job_number': 1717, u'repack': None, u'buildername': u'Linux mozilla-inbound leak test build', u'status': 2, u'release': u'e33ed35dc9d98edf37f557fb24f9407522ab8a55', u'buildtype': u'debug', u'locale': u'en-US', u'timestamp': u'2016-05-31T11:06:38Z', u'locales': None, u'buildurl': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1464690936/firefox-49.0a1.en-US.linux-i686.tar.bz2', u'version': None, u'buildid': u'20160531033536', u'tree': u'mozilla-inbound', u'previous_buildid': None, u'symbols_url': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1464690936/firefox-49.0a1.en-US.linux-i686.crashreporter-symbols.zip', u'testsurl': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1464690936/firefox-49.0a1.en-US.linux-i686.gtest.tests.zip', u'revision': u'e33ed35dc9d98edf37f557fb24f9407522ab8a55', u'builddate': 1464665736}, u'_meta': {u'serializer': u'json', u'exchange': u'exchange/build/normalized', u'sent': u'2016-05-31T04:06:39+01:00', u'routing_key': u'build.mozilla-inbound.linux.debug.mozilla-inbound-linux-debug'}}, took 0.00992202758789 seconds to execute May 31 04:06:48 pulse-actions app/worker3.1: Traceback (most recent call last): May 31 04:06:48 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 75, in run_listener May 31 04:06:48 pulse-actions app/worker3.1: consumer.listen() May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 151, in listen May 31 04:06:48 pulse-actions app/worker3.1: self._drain_events_loop() May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 198, in _drain_events_loop May 31 04:06:48 pulse-actions app/worker3.1: self.connection.drain_events(timeout=self.timeout) May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events May 31 04:06:48 pulse-actions app/worker3.1: return self.transport.drain_events(self.connection, **kwargs) May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events May 31 04:06:48 pulse-actions app/worker3.1: return connection.drain_events(**kwargs) May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 326, in drain_events May 31 04:06:48 pulse-actions app/worker3.1: return amqp_method(channel, args, content) May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/amqp/channel.py", line 1909, in _basic_deliver May 31 04:06:48 pulse-actions app/worker3.1: fun(msg) May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 598, in _receive_callback May 31 04:06:48 pulse-actions app/worker3.1: return on_m(message) if on_m else self.receive(decoded, message) May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 564, in receive May 31 04:06:48 pulse-actions app/worker3.1: [callback(body, message) for callback in callbacks] May 31 04:06:48 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 62, in handler_with_dry_run May 31 04:06:48 pulse-actions app/worker3.1: route(data, message, dry_run) May 31 04:06:48 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 52, in route May 31 04:06:48 pulse-actions app/worker3.1: talos.on_event(data, message, dry_run) May 31 04:06:48 pulse-actions app/worker3.1: File "/app/pulse_actions/handlers/talos.py", line 31, in on_event May 31 04:06:48 pulse-actions app/worker3.1: info = get_buildername_metadata(buildername) May 31 04:06:48 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozci/platforms.py", line 245, in get_buildername_metadata May 31 04:06:48 pulse-actions app/worker3.1: raise MozciError May 31 04:06:48 pulse-actions app/worker3.1: MozciError
This is really strange. How often is allththings.json refreshed in the Heroku app?
Flags: needinfo?(armenzg)
Only when the file is newer.
Flags: needinfo?(armenzg)
With extra debugging this is what I'm seeing [1] At the moment I can only see allthethings.json having been fetched once. Maybe we need to add more debug messages to the main function: https://github.com/armenzg/mozilla_ci_tools/blob/master/mozci/sources/allthethings.py#L66 [1] May 31 07:53:30 pulse-actions app/worker3.1: root INFO: Console output logs DEBUG level messages. May 31 07:53:30 pulse-actions app/worker3.1: mozci DEBUG: Disable validations. May 31 07:53:30 pulse-actions app/worker3.1: replay.replay INFO: Listening on [u'exchange/build/normalized'], with topic [u'build.#.#'] May 31 07:53:30 pulse-actions app/worker3.1: mozci DEBUG: Fetching allthethings.json https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json May 31 07:53:34 pulse-actions app/worker3.1: Traceback (most recent call last): May 31 07:53:34 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 75, in run_listener May 31 07:53:34 pulse-actions app/worker3.1: consumer.listen() May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 151, in listen May 31 07:53:34 pulse-actions app/worker3.1: self._drain_events_loop() May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 198, in _drain_events_loop May 31 07:53:34 pulse-actions app/worker3.1: self.connection.drain_events(timeout=self.timeout) May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events May 31 07:53:34 pulse-actions app/worker3.1: return self.transport.drain_events(self.connection, **kwargs) May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events May 31 07:53:34 pulse-actions app/worker3.1: return connection.drain_events(**kwargs) May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 326, in drain_events May 31 07:53:34 pulse-actions app/worker3.1: return amqp_method(channel, args, content) May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/amqp/channel.py", line 1909, in _basic_deliver May 31 07:53:34 pulse-actions app/worker3.1: fun(msg) May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 598, in _receive_callback May 31 07:53:34 pulse-actions app/worker3.1: return on_m(message) if on_m else self.receive(decoded, message) May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 564, in receive May 31 07:53:34 pulse-actions app/worker3.1: [callback(body, message) for callback in callbacks] May 31 07:53:34 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 62, in handler_with_dry_run May 31 07:53:34 pulse-actions app/worker3.1: route(data, message, dry_run) May 31 07:53:34 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 52, in route May 31 07:53:34 pulse-actions app/worker3.1: talos.on_event(data, message, dry_run) May 31 07:53:34 pulse-actions app/worker3.1: File "/app/pulse_actions/handlers/talos.py", line 31, in on_event May 31 07:53:34 pulse-actions app/worker3.1: info = get_buildername_metadata(buildername) May 31 07:53:34 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozci/platforms.py", line 245, in get_buildername_metadata May 31 07:53:34 pulse-actions app/worker3.1: raise MozciError May 31 07:53:34 pulse-actions app/worker3.1: MozciError May 31 07:53:35 pulse-actions app/worker3.1: talos DEBUG: 'Linux mozilla-inbound build' with status 2. Nothing to be done. May 31 07:53:35 pulse-actions app/worker3.1: root INFO: Message {u'payload': {u'revision': u'0b6f1dcb1c219a5d4e1c421babc8ec9a0235656a', u'status': 2, u'slave': u'bld-linux64-spot-1062', u'platform': u'linux', u'logurl': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1464701925/mozilla-inbound-linux-bm72-build1-build1784.txt.gz', u'previous_buildid': None, u'locale': u'en-US', u'buildurl': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1464701925/firefox-49.0a1.en-US.linux-i686.tar.bz2', u'buildid': u'20160531063845', u'repack': None, u'test_packages_url': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1464701925/firefox-49.0a1.en-US.linux-i686.test_packages.json', u'product': u'firefox', u'locales': None, u'build_number': None, u'tags': [u''], u'release': u'0b6f1dcb1c219a5d4e1c421babc8ec9a0235656a', u'version': None, u'symbols_url': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1464701925/firefox-49.0a1.en-US.linux-i686.crashreporter-symbols.zip', u'timestamp': u'2016-05-31T14:53:34Z', u'testsurl': u'http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux/1464701925/firefox-49.0a1.en-US.linux-i686.gtest.tests.zip', u'job_number': 1784, u'key': u'build.mozilla-inbound-linux.1784.log_uploaded', u'buildtype': u'opt', u'buildername': u'Linux mozilla-inbound build', u'tree': u'mozilla-inbound', u'builddate': 1464676725}, u'_meta': {u'exchange': u'exchange/build/normalized', u'serializer': u'json', u'routing_key': u'build.mozilla-inbound.linux.opt.mozilla-inbound-linux', u'sent': u'2016-05-31T07:53:34+01:00'}}, took 0.00128889083862 seconds to execute
We started showing debug information on pulse_actions in order to discover what is going on. We normally fetch allthethings.json once a day [1] after the dynos reboot. The output seems alright. It seems that this exception is only being raised on worker3 (the process for adding talos PGO jobs) [2]. I've deployed a new version of mozci to give us more information as to what is happening. [1] May 31 07:53:30 pulse-actions app/worker3.1: mozci DEBUG: Fetching allthethings.json https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json Jun 01 08:24:54 pulse-actions app/worker3.1: mozci DEBUG: Fetching allthethings.json https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json Jun 02 09:03:28 pulse-actions app/worker3.1: mozci DEBUG: Fetching allthethings.json https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json Jun 03 09:23:01 pulse-actions app/worker3.1: mozci DEBUG: Fetching allthethings.json https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json Jun 03 11:34:54 pulse-actions app/worker3.1: mozci DEBUG: allthethings.json is not loaded in memory. Jun 03 11:34:54 pulse-actions app/worker3.1: mozci DEBUG: allthethings.json is not on disk. Jun 03 11:34:54 pulse-actions app/worker3.1: mozci DEBUG: Fetching allthethings.json https://secure.pub.build.mozilla.org/builddata/reports/allthethings.json Jun 03 11:34:57 pulse-actions app/worker3.1: mozci DEBUG: allthethings.json is valid. Jun 03 11:34:58 pulse-actions app/worker3.1: mozci DEBUG: allthethings.json seems to load good json data. [2] May 31 04:06:11 pulse-actions app/worker3.1: raise MozciError May 31 04:06:48 pulse-actions app/worker3.1: raise MozciError May 31 04:34:59 pulse-actions app/worker3.1: raise MozciError May 31 05:31:13 pulse-actions app/worker3.1: raise MozciError May 31 07:53:34 pulse-actions app/worker3.1: raise MozciError Jun 01 08:24:56 pulse-actions app/worker3.1: raise MozciError Jun 02 09:03:35 pulse-actions app/worker3.1: raise MozciError Jun 03 09:23:05 pulse-actions app/worker3.1: raise MozciError
I've got now new data [1]: > MozciError: Builder 'TB Linux x86-64 comm-esr38 nightly' is missing. All builders' lenght: 19160 That builder does not exist. It seems that we try to get the buildername metadata of an old builder: https://github.com/mozilla/pulse_actions/blob/master/pulse_actions/handlers/talos.py#L31 What assume is happening that the builder used to be valid. For some time we had disabled worker3, however, that left the pulse queue growing. While we were not processing I assume the builder got removed from the buildbot-configs, thus, making it an invalid builder. Once we resumed worker3, we processed the backlog of jobs and started choking with this invalid builder. We need to catch this exception and ignore it. martianwars: would you like to finish this up? or do you prefer me to? Did my explanation above make sense? [1] Jun 03 11:34:58 pulse-actions app/worker3.1: Traceback (most recent call last): Jun 03 11:34:58 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 75, in run_listener Jun 03 11:34:58 pulse-actions app/worker3.1: consumer.listen() Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 151, in listen Jun 03 11:34:58 pulse-actions app/worker3.1: self._drain_events_loop() Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 198, in _drain_events_loop Jun 03 11:34:58 pulse-actions app/worker3.1: self.connection.drain_events(timeout=self.timeout) Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events Jun 03 11:34:58 pulse-actions app/worker3.1: return self.transport.drain_events(self.connection, **kwargs) Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events Jun 03 11:34:58 pulse-actions app/worker3.1: return connection.drain_events(**kwargs) Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 326, in drain_events Jun 03 11:34:58 pulse-actions app/worker3.1: return amqp_method(channel, args, content) Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/amqp/channel.py", line 1909, in _basic_deliver Jun 03 11:34:58 pulse-actions app/worker3.1: fun(msg) Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 598, in _receive_callback Jun 03 11:34:58 pulse-actions app/worker3.1: return on_m(message) if on_m else self.receive(decoded, message) Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 564, in receive Jun 03 11:34:58 pulse-actions app/worker3.1: [callback(body, message) for callback in callbacks] Jun 03 11:34:58 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 62, in handler_with_dry_run Jun 03 11:34:58 pulse-actions app/worker3.1: route(data, message, dry_run) Jun 03 11:34:58 pulse-actions app/worker3.1: File "pulse_actions/worker.py", line 52, in route Jun 03 11:34:58 pulse-actions app/worker3.1: talos.on_event(data, message, dry_run) Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/pulse_actions/handlers/talos.py", line 31, in on_event Jun 03 11:34:58 pulse-actions app/worker3.1: info = get_buildername_metadata(buildername) Jun 03 11:34:58 pulse-actions app/worker3.1: File "/app/.heroku/python/lib/python2.7/site-packages/mozci/platforms.py", line 246, in get_buildername_metadata Jun 03 11:34:58 pulse-actions app/worker3.1: buildername, len(fetch_allthethings_data()['builders'])) Jun 03 11:34:58 pulse-actions app/worker3.1: MozciError: Builder 'TB Linux x86-64 comm-esr38 nightly' is missing. All builders' lenght: 19160 [2] May 28 04:07:32 pulse-actions app/worker3.1: root INFO: Message {u'_meta': {u'routing_key': u'build.comm-esr38.linux64.opt.nightly.comm-esr38-linux64-nightly', u'sent': u'2016-05-28T04:07:32+01:00', u'exchange': u'exchange/build/normalized', u'serializer': u'json'}, u'payload': {u'buildername': u'TB Linux x86-64 comm-esr38 nightly', u'testsurl': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464429731/thunderbird-38.8.0.en-US.linux-x86_64.web-platform.tests.zip', u'job_number': 36, u'symbols_url': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464429731/thunderbird-38.8.0.en-US.linux-x86_64.crashreporter-symbols.zip', u'previous_buildid': None, u'buildtype': u'opt', u'slave': u'bld-linux64-spot-387', u'completemarhash': u'408f41d3fab5eac871ac06fb465258af0b8a78039784a6137e8e96fe1a628c443f3856f22cc72169ddc488e810947c190029705a2fbedf1fcdc6306e390f8924', u'timestamp': u'2016-05-28T11:07:31Z', u'test_packages_url': None, u'locales': None, u'version': None, u'product': u'thunderbird', u'platform': u'linux64', u'status': 0, u'repack': None, u'logurl': u'http://archive.mozilla.org/pub/thunderbird/nightly/2016/05/2016-05-28-03-02-11-comm-esr38/comm-esr38-linux64-nightly-bm91-build1-build36.txt.gz', u'locale': u'en-US', u'tags': [u'nightly'], u'release': None, u'completemarurl': u'http://archive.mozilla.org/pub/thunderbird/nightly/2016/05/2016-05-28-03-02-11-comm-esr38/thunderbird-38.8.0.en-US.linux-x86_64.complete.mar', u'buildid': u'20160528030211', u'tree': u'comm-esr38', u'buildurl': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464429731/thunderbird-38.8.0.en-US.linux-x86_64.tar.bz2', u'key': u'build.comm-esr38-linux64-nightly.36.log_uploaded', u'build_number': None, u'revision': u'0c9ca47993a9571b58ce4b297ee13185d0e567f6', u'builddate': 1464404531}}, took 0.000459909439087 seconds to execute May 29 04:06:44 pulse-actions app/worker3.1: root INFO: Message {u'payload': {u'completemarhash': u'94ff2c2e1672f9b9002074394c3b8c865491ea0aea75afb66ecdf5a21d07ec97cf533011f009bec7c1f7c0f802c126c166bfc9eb15bd16d0eeb8bcf5b65ad214', u'buildid': u'20160529030221', u'tags': [u'nightly'], u'job_number': 14, u'status': 0, u'test_packages_url': None, u'platform': u'linux64', u'key': u'build.comm-esr38-linux64-nightly.14.log_uploaded', u'builddate': 1464490941, u'buildtype': u'opt', u'testsurl': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464516141/thunderbird-38.8.0.en-US.linux-x86_64.web-platform.tests.zip', u'revision': u'c4a371f2ce99b680aedb3786d2be5c57a35d7017', u'tree': u'comm-esr38', u'slave': u'bld-linux64-spot-383', u'product': u'thunderbird', u'previous_buildid': None, u'repack': None, u'locales': None, u'completemarurl': u'http://archive.mozilla.org/pub/thunderbird/nightly/2016/05/2016-05-29-03-02-21-comm-esr38/thunderbird-38.8.0.en-US.linux-x86_64.complete.mar', u'symbols_url': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464516141/thunderbird-38.8.0.en-US.linux-x86_64.crashreporter-symbols.zip', u'buildername': u'TB Linux x86-64 comm-esr38 nightly', u'buildurl': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464516141/thunderbird-38.8.0.en-US.linux-x86_64.tar.bz2', u'logurl': u'http://archive.mozilla.org/pub/thunderbird/nightly/2016/05/2016-05-29-03-02-21-comm-esr38/comm-esr38-linux64-nightly-bm74-build1-build14.txt.gz', u'build_number': None, u'release': None, u'timestamp': u'2016-05-29T11:06:43Z', u'version': None, u'locale': u'en-US'}, u'_meta': {u'serializer': u'json', u'exchange': u'exchange/build/normalized', u'sent': u'2016-05-29T04:06:44+01:00', u'routing_key': u'build.comm-esr38.linux64.opt.nightly.comm-esr38-linux64-nightly'}}, took 0.000328063964844 seconds to execute May 30 04:06:29 pulse-actions app/worker3.1: root INFO: Message {u'_meta': {u'serializer': u'json', u'sent': u'2016-05-30T04:06:28+01:00', u'exchange': u'exchange/build/normalized', u'routing_key': u'build.comm-esr38.linux64.opt.nightly.comm-esr38-linux64-nightly'}, u'payload': {u'product': u'thunderbird', u'slave': u'bld-linux64-spot-1052', u'tree': u'comm-esr38', u'job_number': 37, u'repack': None, u'locales': None, u'completemarurl': u'http://archive.mozilla.org/pub/thunderbird/nightly/2016/05/2016-05-30-03-02-09-comm-esr38/thunderbird-38.8.0.en-US.linux-x86_64.complete.mar', u'key': u'build.comm-esr38-linux64-nightly.37.log_uploaded', u'buildtype': u'opt', u'build_number': None, u'builddate': 1464577329, u'testsurl': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464602529/thunderbird-38.8.0.en-US.linux-x86_64.web-platform.tests.zip', u'revision': u'c4a371f2ce99b680aedb3786d2be5c57a35d7017', u'buildid': u'20160530030209', u'completemarhash': u'7616c07b6601d0f6981f12906946f19aa2c791f4d65c219e4e07163f2137ba5fc7d41fbf8f8860c5847698e06ed22d8af219e6bdb9724ab1c58ef3d690f51c63', u'status': 0, u'previous_buildid': None, u'tags': [u'nightly'], u'platform': u'linux64', u'test_packages_url': None, u'symbols_url': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464602529/thunderbird-38.8.0.en-US.linux-x86_64.crashreporter-symbols.zip', u'buildurl': u'http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-esr38-linux64/1464602529/thunderbird-38.8.0.en-US.linux-x86_64.tar.bz2', u'buildername': u'TB Linux x86-64 comm-esr38 nightly', u'logurl': u'http://archive.mozilla.org/pub/thunderbird/nightly/2016/05/2016-05-30-03-02-09-comm-esr38/comm-esr38-linux64-nightly-bm91-build1-build37.txt.gz', u'release': None, u'timestamp': u'2016-05-30T11:06:27Z', u'version': None, u'locale': u'en-US'}}, took 0.000508069992065 seconds to execute
Assignee: nobody → kalpeshk2011
Attached file mozci fix —
Attachment #8760046 - Flags: review?(armenzg)
Attached file pulse_actions fix —
The mozci fix would need to be released before this works
Attachment #8760047 - Flags: review?(armenzg)
Comment on attachment 8760047 [details] [review] pulse_actions fix Replied on GH.
Attachment #8760047 - Flags: review?(armenzg)
Attachment #8760046 - Flags: review?(armenzg)
Can we close this bug?
Flags: needinfo?(armenzg)
Yes, thank you!
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(armenzg)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: