1214016 - crontabber on stage not running

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Peter Bengtsson [:peterbe]]

Attachment: Screenshot 2015-10-12 15.48.37.png

We were wondering why the stage home page looked crazy. Turns out it was lacking ADI data. 

Turns out, we have raw ADI coming in just fine but the problem is that crontabber itself has not been running for 5 days. Look at the "Last success" column. They're all old! :)

Crontabber is simply not running.

Comment 1

[Peter Bengtsson [:peterbe]]

The problem is that crontabber runs with a consul log that spits out this error:

Traceback (most recent call last):
  File "/data/socorro/application/socorro/cron/crontabber_app.py", line 94, in <module>
    sys.exit(main(CronTabber))
  File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/crontabber/generic_app.py", line 220, in main
    config_manager_cls
  File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/crontabber/generic_app.py", line 294, in _do_main
    config_pathname=config_path
  File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/configman/config_manager.py", line 216, in __init__
    known_keys = self._overlay_expand()
  File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/configman/config_manager.py", line 627, in _overlay_expand
    an_option.set_value(an_option.default)
  File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/configman/option.py", line 117, in set_value
    new_value = self.from_string_converter(val)
  File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/crontabber/app.py", line 490, in class_list_converter
    class InnerClassList(RequiredConfig):
  File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/crontabber/app.py", line 520, in InnerClassList
    raise JobNotFoundError(class_list_element)
crontabber.app.JobNotFoundError: socorro.cron.jobs.matviews.NightlyBuildsCronApp|1d|05:00


That job no longer exists!

Comment 2

[Peter Bengtsson [:peterbe]]

This has now been removed from consul. Here's how I did it;

1) Run `consulate kv get socorro/crontabber/crontabber.jobs`
That spits out a massive string which I edited and removed the "socorro.cron.jobs.matviews.NightlyBuildsCronApp|1d|05:00" part.

2) Run `consulate kv set socorro/crontabber/crontabber.jobs "..."`
Where ... is the string from before but with one job removed. 

Crontabber on stage is currently working.

Comment 3

[Peter Bengtsson [:peterbe]]

Note-to-self; let's close this once stage looks healthy again [0].

https://crash-stats.allizom.org/crontabber-state/

Comment 4

[Peter Bengtsson [:peterbe]]

https://crash-stats.allizom.org/crontabber-state/ looks fine now.