1431205 - Add status endpoint to allow dynamic functional tests

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Code Cleanup, enhancement)

Description

[John Whitlock [:jwhitlock]]

What problem would this feature solve?
======================================
A status endpoint at, for example, /_kuma_status, could return a machine-readable summary of the running Kuma service. Functional tests could read this and modify test behavior for the tested environment. 

Who has this problem?
=====================
Staff contributors to MDN

How do you know that the users identified above have this problem?
==================================================================
There are a few places where the caller needs to pass options to customize the tests for the target environment.

For example. the 404 tests fail against a development server, because the 404 response is different when DEBUG=True. If the test knew DEBUG=True, it could expect the debug version of the 404 page. It could also raise an error if a public server had DEBUG=True.

Other examples include maintenance mode tests, search result tests, and tests requiring a logged-in user.

How are the users identified above solving this problem now?
============================================================
Test parameters are used to include or exclude tests:

https://kuma.readthedocs.io/en/latest/tests-ui.html

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
A request to /_kuma_status could return some JSON, such as:

{
    "version": 1,
    "sites": {
        "web": {
            "url": "https://developer.mozilla.org/",
            "alternate_urls": [
                "https://prod.mdn.moz.works/"
            ],
            "indexed": true
        },
        "assets": {
            "url": "https://cdn.mdn.mozilla.net/",
            "alternate_urls": [],
            "indexed": true
        },
        "untrusted": {
            "url": "https://mdn.mozillademos.org/",
            "alternate_urls": [],
            "indexed": false
        }
    },
    "services": {
        "kumascript": {
            "active": true
        },
        "search": {
            "active": true,
            "populated": true
        },
        "database": {
            "active": true,
            "populated": true
        }
    },
    "configuration": {
        "debug": false,
        "maintenance_mode": false,
        "test_accounts": false
    },
    "request": {
        "host": "developer.mozilla.org",
        "site": "web"
    }
}

This could be requested and parsed once per test run as a pytest fixture or similar, and be used to determine what tests to run and what to skip.

Headless tests could be used to assert that production servers are configured as expected. For example, hostnames that should not be indexed would have a restrictive robots.txt, and "noindex" in <meta rel="robots"> header elements.

Is there anything else we should know?
======================================

Comment 1

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/67bfff93a36b5860ca65dcf997cc99fc7ba094e9
bug 1431205: Read REVISION_HASH in settings

https://github.com/mozilla/kuma/commit/29efd9f9be976abfcb556c6637f3cacf7a830945
bug 1431205: Add _kuma_status.json endpoint

https://github.com/mozilla/kuma/commit/49ab3f10c0c5f2e9a21f5948309fbb6484106354
bug 1431205: Fix 404 test when DEBUG=True

When DEBUG=True, the Django debug 404 page is displayed, not the
standard 404 page with MDN styling.

https://github.com/mozilla/kuma/commit/f397bc4e96ecb554c8bd42e1da824326e71ebc41
bug 1431205: Fix URLs with run_functional_tests.sh

When running as run_functional_tests.sh, the web server is requested at
http://web:8000/, and http://localhost:8000/ doesn't respond. Alter the
URLs returned by search to make it work.

https://github.com/mozilla/kuma/commit/162ad63b146387f924e0cfc7ad4a86b9934dd07b
bug 1431205: Expect less search results in dev

The sample database has about 1000 docs, and all the 'CSS' results are
returned by the default search filter. Production has more than 50K
docs, and some docs with 'CSS' are not in the default filter. Use the
search engine's doc count to try to detect the sample DB and change test
behaviour.

https://github.com/mozilla/kuma/commit/853773d2a66c7fd8b79f864146cf35d53123043f
bug 1431205: Handle ElasticSearch's NotFoundError

ElasticSearch returns NotFoundError when the requested index doesn't
exist. This can happen when restoring from a backup, or setting up the
development environment. Show available but not populated.

https://github.com/mozilla/kuma/commit/377abbf61b8db90b340e926ca5ab00530b3e9e38
Merge pull request #4637 from jwhitlock/kuma-status-1431205

bug 1431205: Add /_kuma_status.json, use for dynamic functional tests