Status

Cloud Services
Metrics: Pipeline
P1
normal
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: mdoglio, Assigned: mdoglio)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

a year ago
The current implementation only works on a single node deployment. The application installs a job in the crontab that simply checks every hour if there is something that needs to run. In a scenario with multiple web nodes multiple jobs will run, one per web node.
(Assignee)

Updated

a year ago
Blocks: 1248688
(Assignee)

Comment 1

a year ago
A possible solution is to have a node in the cluster dedicated to the scheduler.
No longer blocks: 1248688
(Assignee)

Updated

a year ago
Blocks: 1248688
(Assignee)

Comment 2

a year ago
On a second thought, it may be sufficient to have a single python command running every minute in a cron job. :robotblake is that doable in the dockerflow infrastructure?
Flags: needinfo?(bimsland)
That's do-able, but I feel like the right way to do it would be to use celery / huey / rq and run workers either as separate containers on each node or using supervisor (or similar) inside the containers.
Flags: needinfo?(bimsland)
:mdoglio and me talked about this a bit and found that based on our experience Celery would be way too complex for our use case. Instead I pitched RQ as a replacement since it's a nice compromise between the quality of the developer API and operational complexity (e.g. using Redis ElastiCache). We'd be using RQ-Scheduler (https://github.com/ui/rq-scheduler) for the periodic scheduling on top of RQ.
(Assignee)

Comment 5

a year ago
:jezdez can you please split the work required to configure rq into a set of bugs that block this one?
(Assignee)

Updated

a year ago
Depends on: 1302416
(Assignee)

Updated

a year ago
Summary: Find a way to schedule jobs in the ops infrastructure → Setup rq on ATMO V2

Comment 6

a year ago
Created attachment 8791539 [details] [review]
[telemetry-analysis-service] mozilla:bug-1302777-add-redis-instance > mozilla:master
(Assignee)

Updated

a year ago
Attachment #8791539 - Flags: review?(jezdez)
(Assignee)

Updated

a year ago
Duplicate of this bug: 1302777
(Assignee)

Updated

a year ago
Blocks: 1302761
(Assignee)

Comment 8

a year ago
This is on master now
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
Comment on attachment 8791539 [details] [review]
[telemetry-analysis-service] mozilla:bug-1302777-add-redis-instance > mozilla:master

This was reviewed on GitHub
Attachment #8791539 - Flags: review?(jezdez) → review+
You need to log in before you can comment on or make changes to this bug.