1901662 - Perma TEST-UNEXPECTED-FAIL | No tests collected for subsuite 'featuregates' (Not in PYTHON_UNITTEST_MANIFESTS?)

Status: RESOLVED FIXED

(Toolkit :: FeatureGate, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: tszentpeteri [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=461688168&repo=mozilla-beta
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/XA_UdKsjRGaImgY8wziHfQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-06-10T15:32:49.964Z] Downloading pytest_mock-3.12.0-py3-none-any.whl (9.8 kB)
[task 2024-06-10T15:32:50.222Z] Installing collected packages: pytest-mock
[task 2024-06-10T15:32:50.235Z] Successfully installed pytest-mock-3.12.0
[task 2024-06-10T15:32:50.907Z] Collecting Flask==2.1.3
[task 2024-06-10T15:32:51.029Z]   Downloading Flask-2.1.3-py3-none-any.whl.metadata (3.9 kB)
[task 2024-06-10T15:32:51.173Z] Collecting Werkzeug>=2.0 (from Flask==2.1.3)
[task 2024-06-10T15:32:51.209Z]   Downloading werkzeug-3.0.3-py3-none-any.whl.metadata (3.7 kB)
[task 2024-06-10T15:32:51.213Z] Requirement already satisfied: Jinja2>=3.0 in ./third_party/python/Jinja2 (from Flask==2.1.3) (3.1.2)
[task 2024-06-10T15:32:51.328Z] Collecting itsdangerous>=2.0 (from Flask==2.1.3)
[task 2024-06-10T15:32:51.363Z]   Downloading itsdangerous-2.2.0-py3-none-any.whl.metadata (1.9 kB)
[task 2024-06-10T15:32:51.366Z] Requirement already satisfied: click>=8.0 in ./third_party/python/click (from Flask==2.1.3) (8.1.6)
[task 2024-06-10T15:32:51.368Z] Requirement already satisfied: importlib-metadata>=3.6.0 in ./third_party/python/importlib_metadata (from Flask==2.1.3) (6.0.0)
[task 2024-06-10T15:32:51.390Z] Requirement already satisfied: zipp>=0.5 in ./third_party/python/zipp (from importlib-metadata>=3.6.0->Flask==2.1.3) (3.4.1)
[task 2024-06-10T15:32:51.394Z] Requirement already satisfied: MarkupSafe>=2.0 in ./third_party/python/MarkupSafe/src (from Jinja2>=3.0->Flask==2.1.3) (2.0.1)
[task 2024-06-10T15:32:51.554Z] Collecting MarkupSafe>=2.0 (from Jinja2>=3.0->Flask==2.1.3)
[task 2024-06-10T15:32:51.590Z]   Downloading MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
[task 2024-06-10T15:32:51.640Z] Downloading Flask-2.1.3-py3-none-any.whl (95 kB)
[task 2024-06-10T15:32:51.691Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.6/95.6 kB 1.9 MB/s eta 0:00:00
[task 2024-06-10T15:32:51.726Z] Downloading itsdangerous-2.2.0-py3-none-any.whl (16 kB)
[task 2024-06-10T15:32:51.763Z] Downloading werkzeug-3.0.3-py3-none-any.whl (227 kB)
[task 2024-06-10T15:32:51.807Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.3/227.3 kB 5.3 MB/s eta 0:00:00
[task 2024-06-10T15:32:51.842Z] Downloading MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26 kB)
[task 2024-06-10T15:32:52.109Z] Installing collected packages: MarkupSafe, itsdangerous, Werkzeug, Flask
[task 2024-06-10T15:32:52.109Z]   Attempting uninstall: MarkupSafe
[task 2024-06-10T15:32:52.110Z]     Found existing installation: MarkupSafe 2.0.1
[task 2024-06-10T15:32:52.111Z]     Not uninstalling markupsafe at /builds/worker/checkouts/gecko/third_party/python/MarkupSafe/src, outside environment /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/python-test
[task 2024-06-10T15:32:52.111Z]     Can't uninstall 'MarkupSafe'. No files were found to uninstall.
[task 2024-06-10T15:32:52.285Z] Successfully installed Flask-2.1.3 MarkupSafe-2.1.5 Werkzeug-3.0.3 itsdangerous-2.2.0
[task 2024-06-10T15:32:52.722Z] Requirement already satisfied: MarkupSafe==2.0.1 in ./third_party/python/MarkupSafe/src (2.0.1)
[task 2024-06-10T15:32:53.670Z] Collecting werkzeug==2.1.2
[task 2024-06-10T15:32:53.795Z]   Downloading Werkzeug-2.1.2-py3-none-any.whl.metadata (4.4 kB)
[task 2024-06-10T15:32:53.842Z] Downloading Werkzeug-2.1.2-py3-none-any.whl (224 kB)
[task 2024-06-10T15:32:53.938Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.9/224.9 kB 2.4 MB/s eta 0:00:00
[task 2024-06-10T15:32:54.229Z] Installing collected packages: werkzeug
[task 2024-06-10T15:32:54.229Z]   Attempting uninstall: werkzeug
[task 2024-06-10T15:32:54.230Z]     Found existing installation: Werkzeug 3.0.3
[task 2024-06-10T15:32:54.237Z]     Uninstalling Werkzeug-3.0.3:
[task 2024-06-10T15:32:54.239Z]       Successfully uninstalled Werkzeug-3.0.3
[task 2024-06-10T15:32:54.349Z] Successfully installed werkzeug-2.1.2
[task 2024-06-10T15:32:55.092Z] Collecting zstandard==0.22.0
[task 2024-06-10T15:32:55.217Z]   Downloading zstandard-0.22.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.9 kB)
[task 2024-06-10T15:32:55.263Z] Downloading zstandard-0.22.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)
[task 2024-06-10T15:32:55.539Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 20.3 MB/s eta 0:00:00
[task 2024-06-10T15:32:55.845Z] Installing collected packages: zstandard
[task 2024-06-10T15:32:55.958Z] Successfully installed zstandard-0.22.0
[task 2024-06-10T15:32:59.339Z] TEST-UNEXPECTED-FAIL | No tests collected for subsuite 'featuregates' (Not in PYTHON_UNITTEST_MANIFESTS?)
[task 2024-06-10T15:32:59.359Z] Creating default state directory: /builds/worker/.mozbuild
[task 2024-06-10T15:32:59.359Z] Creating local state directory: /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69
[task 2024-06-10T15:32:59.359Z] Test configuration changed. Regenerating backend.
[task 2024-06-10T15:32:59.359Z] No build detected, test metadata may be incomplete.
[taskcluster 2024-06-10 15:33:00.505Z] === Task Finished ===
[taskcluster 2024-06-10 15:33:00.505Z] Unsuccessful task run with exit code: 1 completed in 197.11 seconds

Comment 1

[Geoff Brown [:gbrown]]

I see the same failure if I request featuregates in a push to try, from either central or beta:
https://treeherder.mozilla.org/jobs?repo=try&revision=50e22fb4d5e7e8545d60f6388b0043393d13b949
https://treeherder.mozilla.org/jobs?repo=try&revision=d376e1b35bc40bdb2d6b1803bf706b5c10f7e72b

Bug 1900963 recently made changes to exclude this directory for non-browser environments -- I wonder if that's related.

Comment 2

[Geoff Brown [:gbrown]]

If I revert the moz.build change in bug 1900963, the tests run and pass:
https://treeherder.mozilla.org/jobs?repo=try&revision=b4ccc876f96c946959476aad856cc5c16d048654

I suppose the source-tests run with a MOZ_BUILD_APP that is neither "mobile/android" nor "browser"?

Comment 3

[:Gijs (he/him)]

(In reply to Geoff Brown [:gbrown] from comment #2)

If I revert the moz.build change in bug 1900963, the tests run and pass:
https://treeherder.mozilla.org/jobs?repo=try&revision=b4ccc876f96c946959476aad856cc5c16d048654

I suppose the source-tests run with a MOZ_BUILD_APP that is neither "mobile/android" nor "browser"?

I'm really sorry but I don't know the answer to this, I'm afraid...

We should keep running the python tests on central, so it's a mistake that they are not included now.

But I don't know how best to accomplish that while also not shipping this dir for non-browser code (ie neither Android nor Thunderbird). I tried to look at the trypush, but it's not obvious what task / method either inside or outside the fg task is being used to collect the relevant manifests (and with what build-time defines like MOZ_BUILD_APP). I even traced back the commit that added PYTHON_UNITTEST_MANIFESTS but can't tell from that how it is being processed. But :ahal wrote it so maybe he can help?

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1900963

Comment 5

[Andrew Halberstadt [:ahal]]

Ah, yes this is a known issue.

When moz.build files are conditionally processed via things like MOZ_BUILD_APP, it often means we lose configuration we didn't mean to. As a workaround for this particular case, we often add the manifests here:
https://searchfox.org/mozilla-central/source/toolkit/toolkit.mozbuild#9

It's not ideal because the manifests aren't registered near where they live, but it'll get the tasks working at least. For a longer term solution, I wonder if ./mach python-test should be doing a file-system traversal of moz.build rather than a build config traversal.. maybe we depend on the build config to conditionally run some tests? I can't really remember.

Comment 6

[Intermittent Failures Robot]

11 failures in 4625 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 5
• autoland: 2
• mozilla-central: 2
• try: 2

Platform and build breakdown:

• linux1804-64: 6
  • opt: 6
• windows11-64: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1901662&startday=2024-06-10&endday=2024-06-16&tree=all

Comment 7

[Intermittent Failures Robot]

4 failures in 4577 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 4

Platform and build breakdown:

• linux1804-64: 2
  • opt: 2
• windows11-64: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1901662&startday=2024-06-17&endday=2024-06-23&tree=all

Comment 8

[Intermittent Failures Robot]

8 failures in 4116 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64: 4
  • opt: 4
• windows11-64: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1901662&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 9

[Mark Striemer [:mstriemer]]

Attachment: Bug 1901662 - Enable featuregates python tests on try r?ahal,Gijs

Comment 10

[Intermittent Failures Robot]

9 failures in 3199 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 5
• mozilla-esr128: 4

Platform and build breakdown:

• linux1804-64: 5
  • opt: 5
• windows11-64: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1901662&startday=2024-07-01&endday=2024-07-07&tree=all

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1d28d2f58293
Enable featuregates python tests on try r=ahal,Gijs

Comment 12

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/1d28d2f58293

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:mstriemer, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox129 to wontfix.

For more information, please visit BugBot documentation.

Comment 14

[Mark Striemer [:mstriemer]]

This feature isn't enabled on Beta 129 so likely no need to uplift

Comment 15

[Intermittent Failures Robot]

2 failures in 3780 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2

Platform and build breakdown:

• linux1804-64: 1
  • opt: 1
• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1901662&startday=2024-07-08&endday=2024-07-14&tree=all