901960 - BLOCKER: mdn caching issues b/w django & node.js services

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task, P2)

Description

[Luke Crouch [:groovecoder]]

This is a vague bug, so really we need someone to join us in #mdndev to help figure it out - then we can bump the severity back down.

KumaScript (node.js) is receiving random 404's from Kuma (django) which is causing all sorts of errors during writers' workflows. Force-refreshing in the browser ignores all caching between them, and that never has the issue.

So, there's something funny going on with cache between KumaScript and Kuma.

Comment 1

[Daniel Maher [:phrawzty]]

22:40:03 <@ubernostrum> phrawzty: last update I have is an email from groovecoder back during the afternoon (US Central time).
22:40:50 <@ubernostrum> phrawzty: seems like something going wrong with celery, and all the other errors are symptoms of it.
22:41:03 <@ubernostrum> And we're in an "observe and diagnose" mode on those.
22:41:42 < phrawzty> are the "random 404's" still occurring ?
22:42:13 <@ubernostrum> That... I don't know.
22:42:32 <@ubernostrum> Right now I just know we're asking the more prolific MDN editors to catalogue weird stuff they see whilst trying to save stuff.

Comment 2

[Daniel Maher [:phrawzty]]

See also :
https://groups.google.com/forum/#!topic/mozilla.dev.mdn/jCaPu1x6wTY

Which links :

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=901971
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=901960
[3] https://github.com/mozilla/kuma/pull/1258
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=866459
[5] https://bugzilla.mozilla.org/show_bug.cgi?id=866458 

Oy...

Comment 3

[Luke Crouch [:groovecoder]]

Correct - the root of the problem seems to be the overloaded celery queue. Closing this one now. Thanks for checking into it.