Closed Bug 1416828 Opened 8 years ago Closed 2 years ago

Common test manifest errors not covered by lint

Categories

(Firefox Build System :: General, enhancement, P3)

Product:
Firefox Build System
Component:
General
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: gbrown, Unassigned)

Details

:jmaher and I frequently update test manifests, typically to disable a test. Sometimes we make mistakes, which typically cause no failures on check-in but fail to produce the desired result. It would be nice if lint caught common problems in test manifests. I believe there is an existing test-manifest-lint job, but I think it only verifies that tests are not commented out.
In manifestparser manifests, a typical skip-if looks like: skip-if = os == 'win' but it is not uncommon to mistakenly write skip-if = os = 'win'
In reftest manifests, there is a precedence to skip-if/fail-if/fuzzy-if which can cause trouble if more than one annotation is applied to a test. See bug 1385288 for an example.
Priority: -- → P3
Product: Testing → Firefox Build System
Product: Firefox Build System → Developer Infrastructure
Severity: normal → S3

These sorts of failures are now covered by the build system when using the .toml format.

Status: NEW → RESOLVED
Closed: 2 years ago
Component: Lint and Formatting → General
Product: Developer Infrastructure → Firefox Build System
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.