1478080 - vendor and fork crontabber

Status: RESOLVED FIXED

(Socorro :: General, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Socorro engineering wrote a library called crontabber which manages coordinating and running scheduled and interdependent tasks. It has a variety of features some of which we still use.

This library is (probably) used by other people:

https://libraries.io/pypi/crontabber

We're in the process of simplifying our postgres situation, upgrading to Python 3, and simplifying Socorro in general. It will greatly minimize the work I need to do if we take this moment to vendor crontabber effectively forking it and then rewriting it in place in our repo.

This bug covers that.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Relatedly, if we can swing it, it'd be super to rewrite it such that all the bookkeeping is done in Django-land.

Making this a P2.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I want to do this in two stages:

Stage 1: vendor the existing crontabber and its tests and make sure that all works

Stage 2: (in a different bug) "rewrite" crontabber using Django models for bookkeeping, a new Job base class, and a Django management command to run things


This bug covers stage 1. Grabbing this to do soon because it'll make the switch to Python 3 easier and the switch from sqlalchemy to Django easier, too.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4605: fix bug 1478080: vendor crontabber

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

PR 4605 implements stage 1--vendoring the necessary bits of crontabber. It updates the code a smidge, removes some bits we don't use, reworks some of the code to use Socorro versions of things (TransactionExecutor, dbapi2_util, App, etc), and fixes some Python 3 issues.

It's not super great, but it's a step in the right direction and reduces future work. So there's that.

After this lands, I'll write up a bug for rewriting crontabber using Django models.

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/f4b26d0a485f7e6ec0306f82cd27441e7ecafea4
fix bug 1478080: vendor crontabber

This brings the necessary bits of crontabber back into Socorro and updates
it to pass linting and work with Python 3.

This doesn't bring in the tests since those are written for nose and there's
a lot of it and we really just want to test Socorro crontabber jobs--not an
entire library. If it turns out that was a bad decision, we can add (and
overhaul) them later.

https://github.com/mozilla-services/socorro/commit/1ef13d1a3d7b3a99feb7cd505d414bce194a23d0
Merge pull request #4605 from willkg/1478080-crontabber

fix bug 1478080: vendor crontabber