720241 - self-serve retrigger requests just falling on the floor

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

https://build.mozilla.org/buildapi/self-serve/jobs/63885 is a request for a native Android build, at 11:28.

https://build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/14810f6dd6a7 is where that request is not shown as pending, not shown as running, not shown as completed, 20 minutes later.

I suspect the "nagios-sjc1: [82] buildbot-master10.build.sjc1:self service agent is CRITICAL: PROCS CRITICAL: 0 processes with args selfserve-agent.py" which has been going on since around the 4am hg bustage is related.

It's sad and pathetic, but particularly considering Android, the inability to retrigger is a blocker :|

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

2012-01-22 04:07:11,032 Loading masters from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json
Traceback (most recent call last):
  File "buildapi/buildapi/scripts/selfserve-agent.py", line 562, in <module>
    consumer.wait()
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/carrot/messaging.py", line 446, in wait
    it.next()
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/carrot/backends/pyamqplib.py", line 300, in consume
    self.channel.wait()
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 107, in wait
    return amqp_method(self, args, content)
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/amqplib/client_0_8/channel.py", line 1973, in _basic_deliver
    func(msg)
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/carrot/messaging.py", line 291, in _receive_callback
    self.receive(message.payload, message)
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/carrot/messaging.py", line 344, in receive
    callback(message_data, message)
  File "buildapi/buildapi/scripts/selfserve-agent.py", line 157, in receive_message
    self._refresh_masters()
  File "buildapi/buildapi/scripts/selfserve-agent.py", line 60, in _refresh_masters
    self.masters = json.load(urllib.urlopen(self.masters_url))
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/simplejson/__init__.py", line 329, in load
    use_decimal=use_decimal, **kw)
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/simplejson/__init__.py", line 385, in loads
    return _default_decoder.decode(s)
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/simplejson/decoder.py", line 402, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/builds/buildbot/selfserve-agent/lib/python2.6/site-packages/simplejson/decoder.py", line 420, in raw_decode
    raise JSONDecodeError("No JSON object could be decoded", s, idx)
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

I restarted the service and it processes a bunch of requests.