1441318 - /monitoring/crontabber/ is erroring out

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

https://crash-stats.mozilla.com/monitoring/crontabber/

is returning this as a response:

"""
{
"query_string": "",
"path": "/monitoring/crontabber/",
"error": "Internal Server Error"
}
"""

with an HTTP 500.

At the same time, /crontabber-state/ is showing data and looks fine.

This bug covers looking into what's going on.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm testing out a dry-run version of the fetch-adi-from-hive app and it's the first time it's run. Maybe the endpoint is dying because some of the data is missing like last_run and last_success?

Comment 2

[Miles Crabill [:miles]]

Here's a traceback from the webapp node that generated the 500:

Internal Server Error: /monitoring/crontabber/
Traceback (most recent call last):
File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/newrelic/hooks/framework_django.py", line 544, in wrapper
return wrapped(*args, **kwargs)
File "/data/socorro/webapp-django/crashstats/crashstats/utils.py", line 45, in wrapper
response = f(request, *args, **kw)
File "/data/socorro/webapp-django/crashstats/monitoring/views.py", line 35, in crontabber_status
most_recent_run = max(last_runs)
TypeError: can't compare datetime.datetime to NoneType
GET /monitoring/crontabber/ => generated 89 bytes in 48 msecs (HTTP/1.1 500) 6 headers in 557 bytes (1 switches on core 0)

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

As soon as my crontabber job was done, it filled in data for all the fields and now the endpoint returns ALLGOOD.

Between comment #2 and that fact, I'm 95% sure my theory in comment #1 is right.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I can reproduce this error in a local dev environment. It returns an HTTP 500 page--that's not wildly helpful, but it's fine. There's nothing in Sentry--I wrote up bug #1441400 to cover that concern.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

We're unlikely to see this again, but it's an easy fix and I was just fiddling with this code so it's in my head. Might as well fix it now.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4374: fix bug 1441318 - fix crontabber status error

Comment 7

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/5a26912211003ca7ec073fbda05ba22c967e7fa6
fix bug 1441318 - fix crontabber status error

If one of the jobs has a last_run of null (it was running for the first time),
then /monitoring/crontabber would error out with a TypeError. This fixes
that by changing that code to only look at non-None last_runs.

https://github.com/mozilla-services/socorro/commit/80507d86d72eb327ad0e8c139921d7b50dfc80c0
Merge pull request #4374 from willkg/1441318-crontabber-status

fix bug 1441318 - fix crontabber status error