1362438 - Implement redirects, other features in Kuma instead of Apache

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, enhancement)

Description

[John Whitlock [:jwhitlock]]

What problem would this feature solve?
======================================
When features are implemented in Apache configuration files:

1) Development behaves differently than production, leading to unexpected results and difficulty in reproducing issues
2) We're tied to also using Apache in AWS

Who has this problem?
=====================
Staff contributors to MDN

How do you know that the users identified above have this problem?
==================================================================
Security efforts like bug 827398 and bug 1076893 rely on Apache configuration, and have been open for multiple years.  This is partially prioritization, and partially because of the cross-team coordination needed to update Apache configurations.

How are the users identified above solving this problem now?
============================================================
Open bugs like infrastructure bug 1353132 to request changes to the Apache configuration.  It is normal for these requests to take 1-2 weeks, because of the issues in cross-team collaboration, synchronizing with deployments, and monitoring that the site is not broken.

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
Bedrock has implemented redirects in Django, as part of the effort to move to AWS.  pmac has packaged that solution in a Django app [1].  The dual configuration can be shipped (both Apache and Django supporting redirects), tested in a Django-only deployment in AWS, and then the now-redundant Apache directives can be removed using bug requests on the normal schedule.

[1] https://github.com/pmac/django-redirect-urls

Is there anything else we should know?
======================================
AWS / SRE effort tracked in GitHub issue mozmar/infra#180

Comment 1

[[github robot]]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/f43e98b466c3589ec2b1d35d29b2c4ee5e5b2e2d
bug 1362438: Merge pull request #4217 from mozilla/dp_redirect_requirements

bug 1362438 - add deps for django-redirect-urls + tests

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/624ad698fc1740358ff4587748a3b9134abcdddd
bug 1362438: Merge pull request #4220 from mozilla/dp_httpd_redirects

implement SCL3 httpd redirects in Django

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/a82444819a5a5c8442b2ff300b0873cae214124a
bug 1362438: More redirects from SCL3 Apache

source:
https://github.com/mozilla/kuma/blob/master/configs/htaccess

https://github.com/mozilla/kuma/commit/7bbcbfab1d1f1447f4fac52d9a7efda3c83af818
bug 1362438: py.test asserts for assert_valid_url

https://github.com/mozilla/kuma/commit/17ec1297f2640425bcec9a4f9a2acdfa4902e4b4
bug 1362438: Refactor locale redirects

Add helper function to explicitly set locale_prefix and prepend_locale
to False for "regular" redirects, to avoid matching on non-locale
prefixes.

Add helper function locale_redirect to set locale_prefix to True, and
default prepend_locale to True. Adjust locale redirect patterns to rely
on locale matching in redirect_urls library.

The /Learn topic has moved to /docs/Learn, but this redirect is
enforced in Apache rules. Adding a TODO to change the redirect after AWS
move.

Adjusted some test patterns to use valid locales, so that tests will
pass against Apache in SCL3 and locally.

https://github.com/mozilla/kuma/commit/3b6807ee5f8ab1bf6ef1f2d944ea951f9410cd4a
Merge pull request #4231 from mozilla/dp_apache_redirects_part_2

Bug 1362438: additional redirects from SCL3 Apache to Django

Comment 4

[John Whitlock [:jwhitlock]]

I've analyzed the remaining redirects and Apache rules in:

https://github.com/mozilla/kuma/pull/4231#issuecomment-324359728

Some of the work will be done on this bug:

* Remove redirect rules for index.php, and return 404s. Over 90% are requests for recent changes RSS. These requests redirect to 404s or "Create a new page" (if logged in). We have new feeds [1], but I don't see any value in making these broken links suddenly work after 5-6 years. The remaining index.php requests are search bots refreshing old MediaWiki or DekiWiki links, or security scanners looking for Joomla or WordPress.
* Remove serving rules for patches, web-tech, and css. These served files that appear to have been created in 2009-2010, which are now no longer available.
* Redirect /contests to http://www.mozillalabs.com/
* Redirect /es4 to http://www.ecma-international.org/memento/TC39.htm

Other work will be done under bug 962148, to change some legacy locale URLs to redirects rather than serving the same content.

[1] https://developer.mozilla.org/en-US/docs/MDN/Contribute/Tools/Feeds

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/f8f1f6c92c742e90785f3154c5b95c1e06c06def
bug 1362438: Drop legacy index.php redirects

All of these redirects appear broken. Looking at the URLS:

* 90% are for the recent changes RSS feed from MediaWiki days (2009).
* 6% are requests for legacy pages, mostly from search engine crawlers
* 4% are "sniffing" attempts, looking for a Joomla or other site

None are from internal referrers.

https://github.com/mozilla/kuma/commit/990693851e17279e654adf468269e6d74b3f2e0b
bug 1362438: Drop legacy file serving

These paths (patches, web-tech, css) appear to have been active around
2008/2009, but are no longer served from SCL3. Instead, return a 404.

https://github.com/mozilla/kuma/commit/3a55ca4deab1e4ccf724ad1198c544f87c44b512
bug 1362438: Update legacy contests redirect

This appears to be for the 2007/2008 Extend Firefox contests:

https://blog.mozilla.org/labs/2008/03/extend-firefox-3-is-here/

Update redirect to the archived Mozilla Labs page.

https://github.com/mozilla/kuma/commit/267514a59a76c8f23d5f0c4da584f7af56216d2e
bug 1362438: Redirect the es4 requests

ES4 was the effort to specify the next version of JavaScript
(ECMAScript 4). It appears Mozilla hosted the wiki, or an archive of it,
at one time. Redirect to the ECMAScript task group.

https://github.com/mozilla/kuma/commit/a3200b9486a0a388594f73fdc2fe8f75a497ff4d
Merge pull request #4381 from jwhitlock/legacy-paths-1362438

bug 1362438: Change handling of some legacy paths

Comment 6

[John Whitlock [:jwhitlock]]

Fix pushed to production and stage. There are now either duplicate redirects in Apache and Kuma, or the Apache rule has been removed and now Kuma handles the redirect.

Comment 7

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/b3c9c4858ead126a0c6d7d87b4ff88aee09c3b6c
bug 1362438: add views for aws

* add views and tests for kuma and kumascript revision hashes
* add views and tests for sitemaps

https://github.com/mozilla/kuma/commit/a2a141b69a84a4a4eb8e11507bdb5e4b2b8d0c6e
Merge pull request #4401 from escattone/views-for-aws-1362438

bug 1362438: add misc views for aws