1402708 - Add "untrusted file serving" mode to Kuma

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, enhancement, P1)

Description

[John Whitlock [:jwhitlock]]

What problem would this feature solve?
======================================
In SCL3, Apache limits the URLs passed to Kuma for mdn.mozillademos.org, to the two types it is supposed to serve:

Live samples:
https://mdn.mozillademos.org/en-US/docs/Learn/CSS/Styling_text/Fundamentals$samples/Color

User attachments:

https://mdn.mozillademos.org/files/12984/web-font-example.png

We're attempting to not use Apache or NGINX in AWS, so we need to configure Kuma to run in a mode where is serves these endpoints (and possibly some others like robots.txt and a root page), and returns 404 for everything else.

Who has this problem?
=====================
Staff contributors to MDN

How do you know that the users identified above have this problem?
==================================================================
The staging instance of the untrusted domain is currently serving all pages.

How are the users identified above solving this problem now?
============================================================
We won't switch production to AWS until this issue is solved.

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
An environment variable could switch Kuma into "untrusted serving" mode, and restrict serving to the few endpoints needed.  This is my preferred option.

A middleware could detect that the request is for the untrusted domain, and return 404 for all but the few requests.

A lightweight NGINX configuration could duplicate the Apache 
configuration in AWS.

Is there anything else we should know?
======================================
Tracked in the MDN migration project:

https://github.com/mozmeao/infra/issues/529

Comment 1

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/2526865ecb6272baff58da03a36b0e7b3b2db95d
bug 1402708: restrict endpoints for requests with attachment host

https://github.com/mozilla/kuma/commit/9013533a05099ac5b952afa96bf07ef2f27cd1a2
Merge pull request #4446 from escattone/restrict-endpoints-2-1402708

Restrict endpoints (bug 1402708)

Comment 2

[John Whitlock [:jwhitlock]]

Kuma is serving mdn.mozillademos.org in AWS. It requires a lot more resources than serving web traffic, so there is probably follow-on work, but closing this bug.

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/ea9ffc134c685824d8eb82526215bbc4c92c1759
bug 1402708: Speed up tests w/ whitenoise 3.3.1

Set two config options in test that default to DEBUG (True in tests):

* WHITENOISE_AUTOREFRESH - When True, it stores the STATIC_ROOT path
  without further processing (fast). When False, it reads and processes all
  the files in the STATIC_ROOT with each call to the Django test client
  (very slow).
* WHITENOISE_USE_FINDERS - When True, it uses Django's static finders,
  which uses django-pipeline's test-optimized finder. When False, it
  scans the STATIC_ROOT path itself (slow).

This is a 10x speed-up for Docker for MacOS for me.

https://github.com/mozilla/kuma/commit/5d826647738545a86e59509089fa6871966c378f
Merge pull request #4481 from jwhitlock/faster-tests-1402708

bug 1402708: Speed up tests w/ whitenoise 3.3.1

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/d780a437be138e9c0417cbdc7f2613a75528a7cf
bug 1402708: add robots.txt endpoint to untrusted domains

https://github.com/mozilla/kuma/commit/ed5c121327c793cd744327369ba0e1b15637702f
Merge pull request #4485 from escattone/block-robots-for-untrusted-domains-1402708

bug 1402708: add robots.txt endpoint to untrusted domains