1508228 - Weekly intermittent failures task fails due to being interrupted by daily dyno restart

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The weekly report (as bug comment) how often a bug got used for classification hasn't been added to every bug today (script failure in flight?).

Last week 1204 bugs got updated by the bot on Monday, this week only 265

Compare https://bugzilla.mozilla.org/page.cgi?id=user_activity.html&action=run&who=orangefactor%40bots.tld
with https://bugzilla.mozilla.org/page.cgi?id=user_activity.html&action=run&who=orangefactor%40bots.tld&from=2018-11-11&to=2018-11-13&group=when

Comment 1

[Sarah Clements [:sclements]]

I've manually run the task so they'll be 1200 comments (with some duplicates). The papertrail alert shows this error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',). It looks like Heroku sends a SIGTERM to all processes when it's shutting down a dyno, which forces all the worker processes to immediately terminate. Not sure what a good work around is to prevent it from happening again - will look into it more.

Comment 2

[Sarah Clements [:sclements]]

Looking into this further, the Heroku Scheduler add-on could work as a replacement to the celery task. Since one of the options available is to run it daily, I could change the Commenter's default `weekly_mode` argument from False to a boolean check based on whether the current day is Monday (which will then run it in weekly or daily mode). The only caveat I see per the docs is: "Scheduler job execution is expected but not guaranteed. Scheduler is known to occasionally (but rarely) miss the execution of scheduled jobs." So then we'd still need to run it manually if that happened, assuming we'd get a papertrail alert. Ed, should we give this a try?

Comment 3

[Ed Morley [:emorley]]

Yeah I think it's the best option at the moment (and the move of other tasks to the scheduler addon in bug 1176492 has worked well so far).

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4300

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/24bda1ed838bf76011b227dde302d16a22d99ffb
Bug 1508228 - Remove Intermittents Commenter celery task (#4300)

Remove celery task and change Commenter weekly_mode default arg in
preparation for the move to the heroku scheduler

Comment 6

[Ed Morley [:emorley]]

This is now deployed to production. Could you set up the scheduler tasks for prototype/stage/prod?

Comment 7

[Sarah Clements [:sclements]]

Done!