1329685 - update_runnable_jobs delays deployments and vagrant provision by 2 minutes

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P2)

Description

[Ed Morley [:emorley]]

Bug 1328618 added `call_command('update_runnable_jobs')` to the `load_initial_data` task (which is run on deploy and also during Vagrant provision).

This is a worthwhile change (making sure runnable jobs is up to date without having to wait 24 hours), however it wasn't clear at the time that the task takes over 2 minutes to run, which is a bit more of a delay than is desirable for every prod deploy and/or vagrant provision.

An alternative would be to schedule an async task, just like celerybeat does every 24 hours. eg in load_initial_data.py's handle() add instead:

fetch_allthethings.apply_async(...)

Comment 1

[Ed Morley [:emorley]]

In vagrant using `time ./manage.py load_initial_data`...
* load_initial_data just doing the `loaddata` = 1.8s
* load_initial_data `loaddata` + `load_preseed` = 1.8s
* load_initial_data `loaddata` + `load_preseed` + `update_runnable_jobs` = 2mins 30s

Comment 2

[Armen [:armenzg]]

My apologies about this. I did not realize how long it takes.
It takes a shorter time if run by hand.

Comment 3

[GitHub Autolander Bot]

Attachment: [treeherder] armenzg:seta_load_initial_data > mozilla:master

Comment 4

[Ed Morley [:emorley]]

Comment on attachment 8826339 [details] [review]
[treeherder] armenzg:seta_load_initial_data > mozilla:master

I've left a comment on the PR :-)

Comment 5

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:speed-up-load-initial-data > mozilla:master

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 8829657 [details] [review]
[treeherder] mozilla:speed-up-load-initial-data > mozilla:master

The 2 second -> 2 minute 30 second regression is pretty frustrating, so I think we should just fix it by reverting the call_command() added by bug 1328618, and I'll then file another bug for "improving the Vagrant development workflow for people working on SETA" (which was what bug 1328618 was trying to fix).

Comment 7

[William Lachance (:wlach)]

Comment on attachment 8829657 [details] [review]
[treeherder] mozilla:speed-up-load-initial-data > mozilla:master

This looks fine, adding :jmaher and :rwood to the bug so they have a head's up on the change. In the future, I'd direct SETA-related review requests to :rwood.

Comment 8

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/4f53e93d8e27a0c4bff9cbd38da4138e0596cba8
Bug 1329685 - Stop calling update_runnable_jobs during load_initial_data

Since it increases the runtime of `load_initial_data` from 2 seconds to
2 minutes 30 seconds, which delays both Vagrant provisions and prod
deploys.

This was added recently in bug 1328618, however another solution will
have to be found to improve the local development workflow instead.

Comment 9

[Ed Morley [:emorley]]

I've filed bug 1333243 for coming up with an alternative solution for improving local development workflow.