Closed Bug 839341 Opened 11 years ago Closed 8 years ago

Demo files are not removed from server after demo is deleted

Categories

(developer.mozilla.org Graveyard :: Demo Studio / Dev Derby, defect)

Product:
developer.mozilla.org Graveyard
Component:
Demo Studio / Dev Derby
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: openjck, Unassigned)

Details

(Whiteboard: [specification][type:bug][triaged][might-expire]) 

What did you do?
================
1. Navigate to the Django admin panel
2. Navigate to Demos > Submissions
3. Check any demo
4. Update the Action to "Delete selected submissions"

What happened?
==============
The demo is removed from the Django admin panel and Demo Studio listings, but the demo files themselves can still be accessed directly.

What should have happened?
==========================
The demo should be removed from the Djano admin panel, the demo should be removed from Demo Studio listings, /and/ the demo files should be deleted from the server. This is very important when it comes to removing demos that infringe copyright.

Is there anything else we should know?
======================================
Les mentioned that there is code in place that should be doing this:
https://github.com/mozilla/kuma/blob/master/apps/demos/models.py#L466
Two more things to add here -- the same thing should happen if a demo is "censored" and in neither case do the files necessarily need to be deleted, just completely inaccessible to non-administrators.
Summary: Demo files are not removed from server after demo is deleted → Demo files are accessible even after a demo is removed or censored
(In reply to John Karahalis [:openjck] from comment #1)
> Two more things to add here -- the same thing should happen if a demo is
> "censored" and in neither case do the files necessarily need to be deleted,
> just completely inaccessible to non-administrators.

This will be very hard to do. In fact, I'm not even sure how to do it with our current infrastructure: The files are just served up from a dumb Apache server. There is no Django or Kuma in front of it to enforce the "censored" flag or any other access control.

Please file a separate bug for that issue, rather than expanding the scope of this one.
Summary: Demo files are accessible even after a demo is removed or censored → Demo files are not removed from server after demo is deleted
Yeah, this is really odd.  Could we instead rename files temporarily?  Ex: "index.html" -> "banned.index.html" ?
Actually, I was wrong about the dumb apache server thing: Demos are served up from the CDN, and I'm pretty sure there's no way to do authenticated access control from there to temporarily censor demos.

https://developer.cdn.mozilla.net/media/uploads/demos/l/m/lmorchard/acd1e7c4660faa00fe12169ae11a918a/pure-css3-clock_1308176462_demo_package/index.html

We could rename them to something obscure, but they'd still probably be discoverable somehow.
Could we replace the files? Maybe with zero-length files of the same name?
We could, but then the files are gone as effectively as if we'd deleted them, and the author would need to re-upload.
All censorship I have seen has been permanent. Maybe we should ditch the concept of temporary censorship altogether and just allow site admins to delete demos (which should in turn delete the files).
Can we place/generate a .htaccess file to block off the files?  That way we can keep the files in tact but not allow anyone to get to them.  Thoughts Les?
(In reply to David Walsh :davidwalsh from comment #8)
> Can we place/generate a .htaccess file to block off the files?  That way we
> can keep the files in tact but not allow anyone to get to them.  Thoughts
> Les?

No, this is on the CDN and not Apache. As far as I understand, there's no such thing as .htaccess on the CDN.

Best we could do is put an .htaccess on the origin server, but I don't think that affects the CDN once the files have already been proxied up.
On investigation, something has occurred to me: I think the issue here might be that our code *is* deleting the demo files from the *origin* server. Because, the code is indeed there to do that, and it does so on my dev VM.

But... I don't know what the retention policy is on the CDN. Once the CDN has picked up the files from the origin server, it doesn't go back to check on them. Reducing load on the origin server is the entire point of the CDN.

So... I wonder if there's some API call or something we need to do in order to force the CDN to flush its contents, ideally just for the deleted files?
I seem to remember this happening before the CDN. And it is also happening on developer-dev.allizom. I just deleted the demo BANANANANANANA, but it is still accessible here:

https://developer-dev.allizom.org/media/uploads/demos/l/m/lmorchard/926ca2fae63a9ab51505399d3f081057/bananananananana_1360705045_demo_package/index.html
And when I say "deleted", I mean:

* Censored the demo in the Django admin panel and...
* Deleted the demo from the Django admin panel
(In reply to John Karahalis [:openjck] from comment #11)
> I seem to remember this happening before the CDN. And it is also happening
> on developer-dev.allizom. I just deleted the demo BANANANANANANA, but it is
> still accessible here:
> 
> https://developer-dev.allizom.org/media/uploads/demos/l/m/lmorchard/
> 926ca2fae63a9ab51505399d3f081057/bananananananana_1360705045_demo_package/
> index.html

Bizarre. I could have sworn this worked on my dev VM when I tried it by hand. But, when I wrote a test for it, sure enough the files weren't getting deleted because it was trying to delete from the wrong path. Here's a fix for that:

    https://github.com/mozilla/kuma/pull/875

We'll still have the CDN issue though. At least, insofar as our Expires headers are set to be between 1 week - 1 year since last file access:

    https://github.com/mozilla/kuma/blob/master/media/.htaccess

That makes sense for general site resources, but not so much for uploads that might be deleted. 

But, since the uploads directory is actually a separate filesystem in production, there's nothing devs can do to put an .htaccess there to override the one for the rest of our media.
Filed bug 842751 to cover off on trying to get a new .htaccess file under uploads with much shorter Expires times.
Great news. Can you copy me on bug 842751?
Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/80dd0aa0843f2afeccece551683111539c83ea26
bug 839341: Properly delete demo files when submission record deleted

https://github.com/mozilla/kuma/commit/264bd8de0d904c515539704a0d2d67adfd4a996b
Merge pull request #875 from lmorchard/839341-demo-files-removed-on-delete

bug 839341: Properly delete demo files when submission record deleted
Whiteboard: [specification][type:bug] → [specification][type:bug][triaged][might-expire]
Demo Studio is being removed from MDN and archived as of end January 2016
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.