1163649 - ship-it is throwing 500 errors

Status: RESOLVED FIXED

(Release Engineering :: Applications: Shipit, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

2015-05-11 15:34:22,181 - kickoff.log_exception#1306: Exception on /releases.html [GET]
Traceback (most recent call last):
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.44.0.36/newrelic/hooks/framework_flask.py", line 98, in _nr_wrapper_Flask_handle_exception_
    return wrapped(*args, **kwargs)
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.44.0.36/newrelic/hooks/framework_flask.py", line 40, in _nr_wrapper_handler_
    return wrapped(*args, **kwargs)
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/views.py", line 83, in view
    return self.dispatch_request(*args, **kwargs)
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/views.py", line 150, in dispatch_request
    return meth(*args, **kwargs)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/views/releases.py", line 113, in get
    return render_template('releases.html', releases=sortedReleases(), form=form)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/views/releases.py", line 32, in sortedReleases
    return sorted(getReleases(), cmp=cmpReleases)

AFAICT we haven't had added any releases since this was working about 12 hours ago, but there may be some fallout from bug 1163472.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Oops, I truncated the traceback, it continues ...
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 224, in getReleases
    status = ReleaseEvents.getCurrentStatus(release.name)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 317, in getCurrentStatus
    status = cls.getStatus(name)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 311, in getStatus
    status[step] = status[step](name)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 354, in repackStatus
    for platform in cls.getEnUSPlatforms(name):
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 418, in getEnUSPlatforms
    return json.loads(release.enUSPlatforms)
  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.6/json/decoder.py", line 338, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Comment 2

[[e-mail] (:bhearsum)]

We fixed up a couple of rows that had an empty enUSPlatforms, now we're getting a new error:
2015-05-11 16:38:08,531 - kickoff.log_exception#1306: Exception on /releases.html [GET]
Traceback (most recent call last):
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.44.0.36/newrelic/hooks/framework_flask.py", line 98, in _nr_wrapper_Flask_handle_exception_
    return wrapped(*args, **kwargs)
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.44.0.36/newrelic/hooks/framework_flask.py", line 40, in _nr_wrapper_handler_
    return wrapped(*args, **kwargs)
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/views.py", line 83, in view
    return self.dispatch_request(*args, **kwargs)
  File "/data/www/ship-it.mozilla.org/release-kickoff/vendor/lib/python/flask/views.py", line 150, in dispatch_request
    return meth(*args, **kwargs)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/views/releases.py", line 113, in get
    return render_template('releases.html', releases=sortedReleases(), form=form)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/views/releases.py", line 32, in sortedReleases
    return sorted(getReleases(), cmp=cmpReleases)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 224, in getReleases
    status = ReleaseEvents.getCurrentStatus(release.name)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 317, in getCurrentStatus
    status = cls.getStatus(name)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 311, in getStatus
    status[step] = status[step](name)
  File "/data/www/ship-it.mozilla.org/release-kickoff/kickoff/model.py", line 358, in repackStatus
    if repacks['platforms'][repack.platform] != 1:
KeyError: u'linux64'

Comment 3

[[e-mail] (:bhearsum)]

Nick and I got this back up and running by adjusting a few columns in firefox_release:
 update firefox_release set enUSPlatforms='["linux", "linux64", "win32", "macosx64", "win64"]', ready=1, complete=1 where name="Firefox-38.0.5b1-build1";

I also updated the enUSPlatforms for Thunderbird:
update thunderbird_release set enUSPlatforms='["linux", "linux64", "win32", "macosx64"]' where enUSPlatforms="[]";

...but I'm not sure if that one needed it.

Nick has a theory on how we got in this state.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Indeed I do. Bug 1163472 meant release-runner couldn't update the release status in ship-it, in particular run mark_as_completed() at
 http://hg.mozilla.org/build/tools/file/de9b30d320f0/buildfarm/release/release-runner.py#l99

So enUSPlatforms wasn't set in the table, despite the sendchange running and the release automation doing its thing, and ship-it blows up trying to read the release state.