1720933 - Increase experimenter celery work instance size

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, task, P1)

Description

[Jared Lockhart [:jlockhart]]

We recently introduced some daily data processing celery tasks and we have seen that they will intermittently fail. Looking at the memory graph for the worker instance it's clear we're hitting the 1gb memory limit. Past tasks were extremely lightweight but now that we're dealing with telemetry we'll need more. Can we please increase the celery worker memory limit to 4-8gb? Thanks!

Comment 1

[Wesley Dawson [:whd]]

https://github.com/mozilla-services/cloudops-infra/pull/3248

https://console.cloud.google.com/kubernetes/deployment/us-west1/experimenter-prod-v1/prod-experimenter/experimenter-prod-experimenter-app-1-worker/overview?project=moz-fx-data-experim-prod-c281

I set the requested memory to 4GB. Any substantial increase beyond this will require a node pool change which will take significantly more effort to implement.

Comment 2

[Jared Lockhart [:jlockhart]]

Amazing thank you :whd!!