2062034 - Add unit tests for docs/_addons/etp_matrix.py

Status: NEW

(Core :: Privacy: Anti-Tracking, task, P2)

Description

[Dan Mehic]

docs/_addons/etp_matrix.py is ~1100 lines, is almost entirely regex parsing of four source files, and has no tests at all. Its failure mode is a silently wrong cell on a published page rather than a build error, which makes it exactly the kind of code that needs them.

Bugs 2062028, 2062030 and 2062033 are all examples of defects that a test would have caught before publication.

Suggested coverage:

• parse_static_pref_list() and parse_firefox_js_overrides() against small fixtures, including #ifdef/#else blocks, @MACRO@ values and inline comments
• parse_content_blocking_prefs() against a representative switch, including a gated case
• a smoke assertion that every pref in FEATURES and OTHER_PRIVACY_PREFS resolves to a non-None value, so a literal None can never reach the page
• once bug 2062029 lands, the CATEGORY_PREFS coverage check

Comment 1

[Dan Mehic]

Some concrete starting material, since this is easier to argue for with something runnable attached.

Where the tests can live

etp_matrix.py imports only stdlib at module level (re, pathlib, urllib.parse) and imports Sphinx lazily inside generate_etp_matrix(), so it is importable and testable without a Sphinx environment. That keeps this simple.

docs/ has no moz.build, so the manifest needs a home in a directory that is already traversed. tools/moz.build already carries a PYTHON_UNITTEST_MANIFESTS list, and tools/moztreedocs/ is the docs-tooling directory, so:

tools/moztreedocs/test/python.toml
tools/moztreedocs/test/test_etp_matrix.py

with python.toml containing:

[DEFAULT]
subsuite = "moztreedocs"

["test_etp_matrix.py"]

and one line added to the existing list in tools/moz.build:

PYTHON_UNITTEST_MANIFESTS += [
    "code-coverage/tests/python/python.toml",
    "fuzzing/smoke/python.toml",
    "lint/test/python.toml",
+   "moztreedocs/test/python.toml",
    "tryselect/test/python.toml",
    "update-packaging/test/python.toml",
]

Run with ./mach python-test tools/moztreedocs/test/test_etp_matrix.py.

Result of running the suite below

I wrote and ran this against current main: 11 passed, 2 failed. Both failures are real, already-filed defects, which is a reasonable sign the suite is aimed at the right things:

• test_yaml_strips_inline_comment_from_value fails — bug 2062033. value: 0 # accept all cookies parses to '0 # accept all cookies'.
• test_features_covers_every_etp_controlled_pref fails — bug 2062028, listing exactly the three prefs: network.cookie.cookieBehavior.optInPartitioning, privacy.trackingprotection.allow_list.baseline.enabled, privacy.trackingprotection.allow_list.convenience.enabled.

Worth noting one asymmetry the suite surfaced: the unit test for inline comments fails while the page-level one (test_no_pref_value_contains_an_inline_comment) passes, because firefox.js currently overrides cookieBehavior and masks the bad YAML value. That is precisely the situation where a bug is latent on Desktop and becomes visible as soon as a platform without firefox.js in its chain is added, so both tests are worth keeping.

The tests

# Any copyright is dedicated to the Public Domain.
# http://creativecommons.org/publicdomain/zero/1.0/

import re
import sys
from pathlib import Path

import pytest

TOPSRCDIR = Path(__file__).parents[3]
sys.path.insert(0, str(TOPSRCDIR / "docs" / "_addons"))

import etp_matrix  # noqa: E402


# ---------------------------------------------------------------- fixtures


@pytest.fixture
def yaml_file(tmp_path):
    def _write(text):
        p = tmp_path / "StaticPrefList.yaml"
        p.write_text(text, encoding="utf-8")
        return p

    return _write


@pytest.fixture
def js_file(tmp_path):
    def _write(text):
        p = tmp_path / "prefs.js"
        p.write_text(text, encoding="utf-8")
        return p

    return _write


# ------------------------------------------------- parse_static_pref_list


def test_yaml_simple_value_and_comment(yaml_file):
    prefs = etp_matrix.parse_static_pref_list(
        yaml_file(
            """
# Enables the thing.
- name: privacy.example.enabled
  type: bool
  value: true
  mirror: always
"""
        )
    )
    assert prefs["privacy.example.enabled"]["value"] == "true"
    assert prefs["privacy.example.enabled"]["comment"] == "Enables the thing."


def test_yaml_ifdef_with_else_uses_release_value(yaml_file):
    """The #else branch is the release default and must win."""
    prefs = etp_matrix.parse_static_pref_list(
        yaml_file(
            """
- name: privacy.example.mode
  type: uint32_t
#ifdef NIGHTLY_BUILD
  value: 1
#else
  value: 3
#endif
  mirror: always
"""
        )
    )
    info = prefs["privacy.example.mode"]
    assert info["value"] == "3"
    assert "#ifdef NIGHTLY_BUILD" in info["ifdef_block"]


def test_yaml_ifdef_without_else_uses_the_only_value(yaml_file):
    prefs = etp_matrix.parse_static_pref_list(
        yaml_file(
            """
- name: privacy.example.nightlyonly
  type: bool
#ifdef NIGHTLY_BUILD
  value: true
#endif
  mirror: always
"""
        )
    )
    assert prefs["privacy.example.nightlyonly"]["value"] == "true"


def test_yaml_macro_value_is_recorded_as_macro(yaml_file):
    prefs = etp_matrix.parse_static_pref_list(
        yaml_file(
            """
- name: privacy.example.macro
  type: bool
  value: @IS_NIGHTLY_BUILD@
  mirror: always
"""
        )
    )
    assert prefs["privacy.example.macro"]["macro"] == "@IS_NIGHTLY_BUILD@"


def test_yaml_strips_inline_comment_from_value(yaml_file):
    """Regression test for bug 2062033."""
    prefs = etp_matrix.parse_static_pref_list(
        yaml_file(
            """
- name: network.cookie.cookieBehavior
  type: RelaxedAtomicInt32
  value: 0 # accept all cookies
  mirror: always
"""
        )
    )
    assert prefs["network.cookie.cookieBehavior"]["value"] == "0"


# --------------------------------------------- parse_firefox_js_overrides


def test_js_plain_prefs(js_file):
    prefs = etp_matrix.parse_firefox_js_overrides(
        js_file(
            """
pref("privacy.example.enabled", true);
pref("privacy.example.count", 5);
pref("privacy.example.name", "hello");
"""
        )
    )
    assert prefs["privacy.example.enabled"]["value"] == "true"
    assert prefs["privacy.example.count"]["value"] == "5"
    assert prefs["privacy.example.name"]["value"] == "hello"


def test_js_ifdef_with_else_uses_else_value(js_file):
    prefs = etp_matrix.parse_firefox_js_overrides(
        js_file(
            """
#ifdef NIGHTLY_BUILD
pref("privacy.example.enabled", true);
#else
pref("privacy.example.enabled", false);
#endif
"""
        )
    )
    assert prefs["privacy.example.enabled"]["value"] == "false"
    assert prefs["privacy.example.enabled"]["ifdef_block"] is not None


# ---------------------------------------------------- parse_feature_string


def test_feature_string_handles_negation_and_whitespace():
    parsed = etp_matrix.parse_feature_string("tp, tpPrivate ,-consentmanagerSkip,, qps")
    assert parsed == {
        "tp": True,
        "tpPrivate": True,
        "consentmanagerSkip": False,
        "qps": True,
    }


# ------------------------------------------------------- _get_pref_value


def test_pref_value_precedence_firefox_js_beats_all_js_beats_yaml():
    yaml_prefs = {"p": {"value": "yaml"}}
    all_js = {"p": {"value": "all"}}
    firefox_js = {"p": {"value": "firefox"}}
    assert etp_matrix._get_pref_value("p", yaml_prefs, firefox_js, all_js) == "firefox"
    assert etp_matrix._get_pref_value("p", yaml_prefs, {}, all_js) == "all"
    assert etp_matrix._get_pref_value("p", yaml_prefs, {}, {}) == "yaml"
    assert etp_matrix._get_pref_value("nope", yaml_prefs, {}, {}) is None


# --------------------------------------------- parse_content_blocking_prefs


def test_content_blocking_switch_assignments_and_gate(tmp_path):
    mjs = tmp_path / "ContentBlockingPrefs.sys.mjs"
    mjs.write_text(
        """
export const ContentBlockingPrefs = {
  PREF_LNA_ETP_ENABLED: "network.lna.etp.enabled",

  matchCBCategory(type, item) {
    switch (item) {
      case "tp":
        this.CATEGORY_PREFS[type]["privacy.trackingprotection.enabled"] = true;
        break;
      case "lna":
        if (Services.prefs.getBoolPref(this.PREF_LNA_ETP_ENABLED, false)) {
          this.CATEGORY_PREFS[type]["network.lna.blocking"] = true;
        }
        break;
      default:
        console.error("nope");
    }
  },
};
""",
        encoding="utf-8",
    )
    cases = etp_matrix.parse_content_blocking_prefs(mjs)

    assert cases["tp"] == [
        {
            "pref": "privacy.trackingprotection.enabled",
            "value_expr": "true",
            "gate_pref": None,
        }
    ]
    assert cases["lna"][0]["pref"] == "network.lna.blocking"
    assert cases["lna"][0]["gate_pref"] == "network.lna.etp.enabled"


# ------------------------------------------------------------ integration


@pytest.fixture(scope="module")
def real_sources():
    y = etp_matrix.parse_static_pref_list(
        TOPSRCDIR / "modules" / "libpref" / "init" / "StaticPrefList.yaml"
    )
    aj = etp_matrix.parse_firefox_js_overrides(
        TOPSRCDIR / "modules" / "libpref" / "init" / "all.js"
    )
    fj = etp_matrix.parse_firefox_js_overrides(
        TOPSRCDIR / "browser" / "app" / "profile" / "firefox.js"
    )
    return y, aj, fj


def _page_prefs():
    prefs = []
    for feature in etp_matrix.FEATURES:
        prefs += [
            p for p in (feature["pref_normal"], feature.get("pref_pb")) if p
        ]
    for features in etp_matrix.OTHER_PRIVACY_PREFS.values():
        for _name, normal, pbmode, _desc in features:
            prefs += [p for p in (normal, pbmode) if p]
    return prefs


def test_no_documented_pref_resolves_to_none(real_sources):
    """A None here becomes a literal `None` cell on the published page."""
    y, aj, fj = real_sources
    unresolved = [
        p for p in _page_prefs() if etp_matrix._get_pref_value(p, y, fj, aj) is None
    ]
    assert not unresolved, f"prefs would render as None: {unresolved}"


def test_no_pref_value_contains_an_inline_comment(real_sources):
    """Regression test for bug 2062033, at the page level."""
    y, aj, fj = real_sources
    leaked = [
        (p, v)
        for p in _page_prefs()
        if (v := etp_matrix._get_pref_value(p, y, fj, aj)) and "#" in str(v)
    ]
    assert not leaked, f"pref values leak a comment into the table: {leaked}"


def test_features_covers_every_etp_controlled_pref():
    """Regression test for bug 2062028 / coverage check from bug 2062029."""
    src = (
        TOPSRCDIR
        / "browser"
        / "components"
        / "protections"
        / "ContentBlockingPrefs.sys.mjs"
    ).read_text(encoding="utf-8")

    def field(name):
        m = re.search(rf'\b{name}\s*[:=]\s*"([^"]+)"', src)
        return m.group(1) if m else name

    block = re.search(r"strict:\s*\{(.*?)\n      \},", src, re.S).group(1)
    controlled = set(re.findall(r'"([a-z0-9_.]+)"\s*:', block, re.I))
    controlled |= {
        field("PREF_ALLOW_LIST_BASELINE"),
        field("PREF_ALLOW_LIST_CONVENIENCE"),
    }

    documented = set()
    for feature in etp_matrix.FEATURES:
        documented |= {
            p for p in (feature["pref_normal"], feature.get("pref_pb")) if p
        }

    missing = sorted(controlled - documented)
    assert not missing, f"ETP-controlled prefs missing from the ETP table: {missing}"

Feel free to take or discard any of it. The parser tests are the part I would prioritise: they are fast, need no tree access, and cover the #ifdef/#else release-value logic and the gate-pref extraction, which are the two subtlest pieces of the module.