Closed
Bug 1493687
Opened 6 years ago
Closed 6 years ago
rewrite crontabber using django models and commands
Categories
(Socorro :: Webapp, task, P3)
Socorro
Webapp
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: willkg, Assigned: willkg)
References
Details
Attachments
(7 files)
53 bytes,
text/x-github-pull-request
|
Details | Review | |
53 bytes,
text/x-github-pull-request
|
Details | Review | |
53 bytes,
text/x-github-pull-request
|
Details | Review | |
53 bytes,
text/x-github-pull-request
|
Details | Review | |
53 bytes,
text/x-github-pull-request
|
Details | Review | |
53 bytes,
text/x-github-pull-request
|
Details | Review | |
53 bytes,
text/x-github-pull-request
|
Details | Review |
In bug #1478080, I pulled the code for the crontabber library (which itself was extracted out of Socorro years ago) back into Socorro. The next step for that is to convert it into a Django app.
Converting it into a Django app involves some or all of the following:
1. building Django models for the crontabber and crontabber_log tables
2. adding a django admin view for those tables
3. writing Django commands for the crontabber subcommands
4. reworking the existing crontabber jobs to work in this new context
This bug covers that work.
Assignee | ||
Comment 1•6 years ago
|
||
Making this a P2. It's something we want to do soon and it blocks the postgres cleanup work.
Blocks: 1361394
Priority: -- → P2
Assignee | ||
Comment 2•6 years ago
|
||
I'm going to generate the models and tweak the existing crontabber code to use them.
Step 2 will be a rewrite of the code.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
Assignee | ||
Comment 3•6 years ago
|
||
Comment 4•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/33009745d1a566cec62fa7466a95d9742f400439
bug 1493687: add Django cron models
https://github.com/mozilla-services/socorro/commit/3491562d0a536a395a39a786b21e8d6edc7770f6
bug 1493687: rework crontaber to use new Django model tables
https://github.com/mozilla-services/socorro/commit/dd42625017297b7b3a6865faf3a9aed510ca942f
Merge pull request #4668 from willkg/1493687-crontabber
bug 1493687: move crontabber tables to Django
Assignee | ||
Comment 5•6 years ago
|
||
Comment 6•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/22dbb93d1702841bbff5cc49f6eca97887b50b4b
bug 1493687: fix /crontabber-state
This nixes the CrontabberState middleware in favor of one that uses the
Django model.
This also removes the dependency visualization on the /crontabber-state
page. It doesn't like it when there are no dependencies and I figured
it's time to remove it.
Since it's the only use of d3-sankey, I removed that, too.
One minor irritant is that the depends_on field was a text[] in the
old models and a text (comma-separated string) in the new one. That's
causing some problems, so I hard-coded things. In the rewrite we won't
be redoing the depends_on, so it's not long for this world anyhow.
https://github.com/mozilla-services/socorro/commit/5fa797addebf8e8a6e98e364cc2f3a072c4dd215
Merge pull request #4669 from willkg/1493687-crontabber-state
bug 1493687: fix /crontabber-state
Assignee | ||
Comment 7•6 years ago
|
||
Comment 8•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/6c3cb4b2f562e0e8e3b09c3ef9a34e5773f1db81
bug 1493687: remove /api/CrontabberState and /crontabber-state
This removes the CrontabberState API endpoint and the /crontabber-state
view. We no longer support depends_on relationships between jobs, so
we don't need /crontabber-state anymore. This rewrites the one view
that looks at crontabber jobs to look at the database directly, so
we no longer need the API endpoint, either.
This also fixes the API tests to use a new NoOpMiddleware for testing
API infrastructure. That saves me the effort of rewriting those tests
every time I remove another API endpoint.
https://github.com/mozilla-services/socorro/commit/dc0137d1077c09176de23c3374c978235436fcdc
Merge pull request #4707 from willkg/1493687-crontabberstate
bug 1493687: remove /api/CrontabberState and /crontabber-state
Comment 9•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro
https://github.com/mozilla-services/socorro/commit/266014ae1e99ba1c4f4ebfe1020f4c63c5629df6
bug 1493687: fix dsn
https://github.com/mozilla-services/socorro/commit/57345cbc7ca246444c6116e21431dea14c51d6f0
Merge pull request #4725 from willkg/1493687-dsn
bug 1493687: fix dsn
Assignee | ||
Comment 10•6 years ago
|
||
Assignee | ||
Comment 11•6 years ago
|
||
Maybe instead of doing this, we rewrite the jobs using Huey:
https://huey.readthedocs.io/en/latest/
That probably requires Redis, but maybe we could write a Django-ORM storage backend:
https://huey.readthedocs.io/en/latest/contrib.html#sqlite-storage
Anyhow, definitely worth looking into and if we don't use it, there are some interesting API things that would be handy.
Assignee | ||
Comment 13•6 years ago
|
||
Assignee | ||
Comment 14•6 years ago
|
||
Assignee | ||
Comment 15•6 years ago
|
||
For the record, I reimplemented crontabber using Django commands. It's slimmed down and has additional tests. "Jobs" will be implemented as Django commands and then added to the JOBS schedule. It should work well and it means that "jobs" can be run as normal Django commands when testing and debugging without dealing with locking and schedules.
Assignee | ||
Comment 16•6 years ago
|
||
Assignee | ||
Comment 17•6 years ago
|
||
willkg merged PR #4873: "bug 1493687: add the tests" in ca75261.
Assignee | ||
Comment 18•6 years ago
|
||
One of the tests is failing intermittently:
___________ crashstats/cron/tests/test_utils.py::test_time_to_run _____________
Traceback (most recent call last):
File "/app/webapp-django/crashstats/cron/tests/test_utils.py", line 82, in test_time_to_run
assert time_to_run(job_spec, job) is False
File "/app/webapp-django/crashstats/cron/utils.py", line 87, in time_to_run
hh, mm = convert_time(time_)
File "/app/webapp-django/crashstats/cron/utils.py", line 61, in convert_time
raise TimeDefinitionError("Invalid definition of time %r" % value)
crashstats.cron.TimeDefinitionError: Invalid definition of time '24:06'
==================== 1 failed, 285 passed in 60.34 seconds =====================
Assignee | ||
Comment 19•6 years ago
|
||
Assignee | ||
Comment 20•6 years ago
|
||
Assignee | ||
Comment 21•6 years ago
|
||
Pushed this to prod just now. Will wait a little until I can verify it's working.
Assignee | ||
Comment 22•6 years ago
|
||
It ran in prod and produced LogEntries and IT IS GLORIOUS! Marking as FIXED.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Assignee | ||
Updated•6 years ago
|
You need to log in
before you can comment on or make changes to this bug.
Description
•