Open Bug 1660870 Opened 1 year ago Updated 2 months ago

|./mach try fuzzy| tries to schedule wpt jobs for directories without web-platform-tests in it

Categories

(Release Engineering :: Applications: MozharnessCore, defect, P1)

Tracking

(Not tracked)

People

(Reporter: RyanVM, Unassigned)

Details

Attachments

(3 obsolete files)

Pretty sure this is a regression as this series of steps has successfully scheduled jobs in the past for me.

STR:
1.) Run ./mach configure
2.) Run ./mach try fuzzy browser/components/places/tests storage/test toolkit/components/places/tests testing/web-platform/tests/IndexedDB testing/web-platform/tests/storage testing/web-platform/tests/webstorage dom/indexedDB
3.) Filter on !qr !fis !gpu !verify and select all remaining jobs

Expected results:
Jobs from the listed directories will be scheduled for whatever test suite they happen to fall under.

Actual results:
WPT jobs all fail due to attempting to run jobs from dom/indexedDB as shown below:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313863264&repo=try&lineNumber=1296

Summary: |./mach try fuzzy| tries to schedule wpt jobs for directories with wpt → |./mach try fuzzy| tries to schedule wpt jobs for directories without web-platform-tests in it
Severity: -- → S2
Priority: -- → P1

Pretty sure this is caused by a bug in the web_platform_tests.py mozharness script:
https://searchfox.org/mozilla-central/rev/19c23d725f27d0989e4a60f36d64004cebb39736/testing/mozharness/scripts/web_platform_tests.py#309

It looks like we don't filter out non-WPT keys from MOZHARNESS_TEST_PATHS. I'm somewhat surprised that this is the first time I've seen this issue come up. I guess it isn't so common to pass both WPT and non-WPT paths at the same time..

Component: Mozbase → Applications: MozharnessCore
Product: Testing → Release Engineering
QA Contact: aki

Still an issue?

Flags: needinfo?(ryanvm)

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #2)

Still an issue?

Yes. Here's an updated log:
https://treeherder.mozilla.org/logviewer?job_id=350059364&repo=try&lineNumber=1424

Flags: needinfo?(ryanvm)

(In reply to Andrew Halberstadt [:ahal] from comment #1)

Pretty sure this is caused by a bug in the web_platform_tests.py mozharness script:
https://searchfox.org/mozilla-central/rev/19c23d725f27d0989e4a60f36d64004cebb39736/testing/mozharness/scripts/web_platform_tests.py#309

It looks like we don't filter out non-WPT keys from MOZHARNESS_TEST_PATHS. I'm somewhat surprised that this is the first time I've seen this issue come up. I guess it isn't so common to pass both WPT and non-WPT paths at the same time..

Hm, https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/web_platform_tests.py#395 errors out if we have a non-wpt path. should we just continue ?

Assignee: nobody → aki
Status: NEW → ASSIGNED

That would fix this bug, but it would be a bit better to filter out non-WPT suites in the first place. For context MOZHARNESS_TEST_PATHS should look something like:

{
    "web-platform-tests": [<paths>],
    "web-platform-tests-reftests": [<paths>],
    "mochitest-plain": [<paths>],
}

Ideally we'd still use fatal if there's a non-wpt path in the first two suites, but not in that third one. So we should instead ignore keys in MOZHARNESS_TEST_PATHS that don't correspond to a known WPT suite.

Attachment #9238712 - Attachment is obsolete: true
Attachment #9238736 - Attachment is obsolete: true
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2000fe6ec3bc
ignore non-wpt test paths in wpt. r=releng-reviewers,ahal DONTBUILD

Backed out changeset 2000fe6ec3bc (bug 1660870) for causing multiple wpt failures
Backout link: https://hg.mozilla.org/integration/autoland/rev/5d7a4d3e208088e0c771b7409bb20dbe92bb0f93
Push with failures, failure log.

Flags: needinfo?(aki)

Oops, I should have manually edited rather than a search/replace.

Repushed with the s,test_suites,test_types, fix, non-DONTBUILD. It should land after trees reopen.

Flags: needinfo?(aki)
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/69bd8bc9c511
ignore non-wpt test paths in wpt. r=releng-reviewers,ahal

Backed out changeset 69bd8bc9c511 (Bug 1660870) for causing wpt failures in block-page-break-inside-avoid-1-print.html
Backout link: https://hg.mozilla.org/integration/autoland/rev/6c6500b08fad53f895b6aa4650ad21106e3dbc14
Push with failures, failure log.

Flags: needinfo?(aki)

Ahal: any ideas? I'm ready to punt on this.

Flags: needinfo?(aki) → needinfo?(ahal)

I suspect this may have been caused by this change:

-                os.environ.get("MOZHARNESS_TEST_PATHS", '""')
+                os.environ.get("MOZHARNESS_TEST_PATHS", '"{}"')

Since if you json.loads the former you get '' (which evaluates to False), whereas the latter returns '{}' (which evaluates to True). So I think this patch was causing us to go into the following if statement even when MOZHARNESS_TEST_PATHS was unset. I'm pretty sure all the other WPT suites always pass in MOZHARNESS_TEST_PATHS, so that's why they didn't break as well.

Flags: needinfo?(ahal)
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c1aa00bd2722
ignore non-wpt test paths in wpt. r=releng-reviewers,ahal
Flags: needinfo?(aki)

Back in the pool.

Assignee: aki → nobody
Status: ASSIGNED → NEW
Flags: needinfo?(aki)
Attachment #9238798 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.