766003 - PRODUCTION: "kumawiki" waffle flag default/false is not preventing access to kuma wiki

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Editing, defect)

Description

[Les Orchard [:lorchard]]

When the "kumawiki" waffle flag is missing or false, this should be a 404:

    https://developer.mozilla.org/en-US/docs/en-US/IndexedDB$history

Instead, access is allowed to the Kuma wiki on *current* production, which doesn't have a good set of wiki data. Of course, if code for bug 754534 is pushed out, this URL will probably change to:

    https://developer.mozilla.org/en-US/docs/IndexedDB$history

Either way, this should be a 404.

It'll probably be a moot point after Kuma launch and bug 765719, but I suspect something in the order of decorators on wiki views is not allowing the waffle flag to work quite right.

Comment 2

[John Karahalis [:openjck]]

If this takes more than 1-point of work, stop and we will revisit it at the next planning meeting (maybe just via robots.txt).

Comment 3

[Les Orchard [:lorchard]]

This is really, really bizarre. We might want to spend more than just 1pt of time on this, because I suspect it's a symptom of a bigger problem. Like... do we maybe have some production web front ends pointing to the wrong databases? Or maybe to the wrong memcache?

Consider that there's no "kumawiki" waffle flag at all - I deleted it from admin. That should default to false. But, you can still see the "New Wiki Feature Preview", which should be hidden by a conditional in the template:

    https://developer.mozilla.org/en-US/docs/

Though, apparently there are no articles:
    https://developer.mozilla.org/en-US/docs/all

Somehow, despite that, you can see this one:
    https://developer.mozilla.org/en-US/docs/Firefox_3_for_developers

But, when you click Edit, per bug 767041, you get a 500 error:
    DoesNotExist: Document matching query does not exist.

I have no clue what's going on, but this seems very disturbing. And broken in production *right now*. Probably worth more than just a quick investigation

Comment 4

[Luke Crouch [:groovecoder]]

FWIW, I tried re-adding the kumawiki flag to superusers and a bunch of the kuma stuff went away from production.

https://developer.mozilla.org/en-US/docs/

I think the google bot found and indexed new kumawiki url's by scooping up the "Needs technical review" and "Needs editorial review" side boxes' content from the page, since the https://developer.mozilla.org/en-US/docs/all url was still inaccessible.

Jake,

Can you double-check the database and memcache(?) settings on developer.mozilla.org to make sure they are using the right database?

Comment 5

[Jake Maul [:jakem]]

The prod settings haven't changed since the dekiwiki cluster was moved to SCL3.

-rw-r--r--  1 root   root   2496 May  7 20:23 settings_local.py

The DB and memcache settings in it look normal to me. All 3 prod servers have identical DB and memcache servers.

Comment 6

[Luke Crouch [:groovecoder]]

All the production URL's are 404's now as expected.

If the settings look right I'm not sure what else to check except maybe access logs?

Jake can you check the production access logs for https://developer.mozilla.org/en-US/docs/Firefox_3_for_developers requests today in the last 2 hours. I've hit it once and got a 200, then again and got a 404.

I really hope that developer-new.mozilla.org isn't in the rotation as a web-head for prod yet? Can we make sure developer-new isn't fielding any other requests?

Comment 7

[Luke Crouch [:groovecoder]]

There is something going on for sure. I just made a waffle flag update on developer-new.mozilla.org and it affected developer.mozilla.org.

Comment 8

[Luke Crouch [:groovecoder]]

Jake did you check the -new settings_local.py file too? It seems like updated to developer-new.mozilla.org are somehow affecting developer.mozilla.org too. I just reproduced it again by changing a flag on developer-new.mozilla.org.

Comment 9

[Luke Crouch [:groovecoder]]

After poking a bunch I'm almost certain this is a cache issue. Are developer-new.mozilla.org and developer.mozilla.org sharing a memcache server pool?

Comment 10

[Luke Crouch [:groovecoder]]

Both sites are consistently using the flag value set on developer.mozilla.org and are ignoring flag values set on developer-new.mozilla.org.

So, I changed the developer.mozilla.org 'kumawiki' flag to superusers + staff + authenticated so we aren't leaking Kuma url's to the google bot on production, but auth users can still use developer-new for testing.

But this means authenticated users will see Kuma navigation points

developer.mozilla.org is getting cached results from developer-new.mozilla.org and vice-versa e.g.,

1. Go to https://developer.mozilla.org/en-US/docs/LorchardTestPage2 - 404
2. Go to https://developer-new.mozilla.org/en-US/docs/LorchardTestPage2 - 200
3. Go back to https://developer.mozilla.org/en-US/docs/LorchardTestPage2 - 200

1. Go to https://developer.mozilla.org/en-US/docs/all - no results
2. Go to https://developer-new.mozilla.org/en-US/docs/all - no results

1. Go to https://developer-new.mozilla.org/en-US/docs/User:lmorchard/TestSubpage - results
2. Go to https://developer.mozilla.org/en-US/docs/User:lmorchard/TestSubpage - results

Comment 11

[Les Orchard [:lorchard]]

(In reply to Luke Crouch [:groovecoder] from comment #9)
> After poking a bunch I'm almost certain this is a cache issue. Are
> developer-new.mozilla.org and developer.mozilla.org sharing a memcache
> server pool?

Ah, that would explain why certain parts of the code think there are pages in the DB, when there aren't. The cached data would be in the developer-new database.

I want to say Django has caching settings to specify prefix and version as part of keys, but I can't find the options in the v1.2 docs

Comment 12

[Les Orchard [:lorchard]]

This was fixed in an IT bug