1408535 - Perhaps NOT store Celery results in Postgres

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[Peter Bengtsson [:peterbe]]

We currently store all Celery executions in Django [0]. 
Is this worth it? We have no API or UI where we query this, yet. Should we? 

At the time of writing, the only Celery task we have is the download from Microsoft. That task has its own state to give insight into attempts, failures and successes. 

If storing it "in Django", perhaps there are better places that have less overhead, like the Redis cache. 


[0] https://github.com/mozilla-services/tecken/blob/83e087b0c350d2eddc56d3abad033c7ac8926ed0/tecken/settings.py#L30

Comment 1

[Peter Bengtsson [:peterbe]]

One reason I'm filing this is because I noticed that every Celery task execution triggers 4 SQL queries to be executed:

SELECT "django_celery_results_taskresult"."id", "django_celery_results_taskresult"."task_id", "django_celery_results_taskresult"."status", "django_celery_results_taskresult"."content_type", "django_celery_results_taskresult"."content_encoding", "django_celery_results_taskresult"."result", "django_celery_results_taskresult"."date_done", "django_celery_results_taskresult"."traceback", "django_celery_results_taskresult"."hidden", "django_celery_results_taskresult"."meta" FROM "django_celery_results_taskresult" WHERE "django_celery_results_taskresult"."task_id" = '944cbd1e-e54c-442d-ba21-5c3a0ea49528';

INSERT INTO "django_celery_results_taskresult" ("task_id", "status", "content_type", "content_encoding", "result", "date_done", "traceback", "hidden", "meta") VALUES ('944cbd1e-e54c-442d-ba21-5c3a0ea49528', 'STARTED', 'application/json', 'utf-8', '{"pid": 11, "hostname": "celery@daf0403f029a"}', '2017-10-13T20:40:25.003867+00:00'::timestamptz, NULL, false, '{"children": []}') RETURNING "django_celery_results_taskresult"."id";

SELECT "django_celery_results_taskresult"."id", "django_celery_results_taskresult"."task_id", "django_celery_results_taskresult"."status", "django_celery_results_taskresult"."content_type", "django_celery_results_taskresult"."content_encoding", "django_celery_results_taskresult"."result", "django_celery_results_taskresult"."date_done", "django_celery_results_taskresult"."traceback", "django_celery_results_taskresult"."hidden", "django_celery_results_taskresult"."meta" FROM "django_celery_results_taskresult" WHERE "django_celery_results_taskresult"."task_id" = '944cbd1e-e54c-442d-ba21-5c3a0ea49528';

UPDATE "django_celery_results_taskresult" SET "task_id" = '944cbd1e-e54c-442d-ba21-5c3a0ea49528', "status" = 'SUCCESS', "content_type" = 'application/json', "content_encoding" = 'utf-8', "result" = 'null', "date_done" = '2017-10-13T20:40:25.385607+00:00'::timestamptz, "traceback" = NULL, "hidden" = false, "meta" = '{"children": []}' WHERE "django_celery_results_taskresult"."id" = 16099;

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/504

Comment 3

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/fcab8240f046275152e5a04910f693df16d1f19a
fixes bug 1408535 - Perhaps NOT store Celery results (#504)