Open Bug 1925646 Opened 26 days ago Updated 23 days ago

[meta] Remove outdated/redundant "implementation-status: backlog"

Categories

(Testing :: web-platform-tests, task)

task

Tracking

(Not tracked)

People

(Reporter: saschanaz, Unassigned)

References

Details

(Keywords: meta)

It downgrades tests from tier 1 to tier 2 if the corresponding feature is not implemented. If that changed, it should go away.

Depends on: 1925659

Would it make sense for wpt-sync to automatically remove the status for tests with non-zero success rate (or at least some percent of success rate)? Would it be doable?

Flags: needinfo?(james)

I agree this is unfortunate :/

Non-zero success rate wouldn't work; suites often have some percentage of tests that pass when they aren't implemented (typically checking the feature doesn't change some existing behaviour). That can be relatively high; I don't think 25% is unrealistic.

I think in theory the sync could remove implementation-status annotations when we reach some higher threshold e.g. 50%, but I think it's probably not quite trivial to implement. First we'd need to find everything marked as implementation-status: backlog and then record the pass rate for those tests. But at the moment the update code assumes that all the information it needs to decide whether to update something is in the log files i.e. we're not using the metadata files as input to decide whether to update something in the first place (and as a result get to skip reading them except when we already determined that something needs to be updated). Since AFAIK we don't have backlog status in the log files, (except in some very hacky way like via the job name), that doesn't quite work here.

Severity: -- → S3
Flags: needinfo?(james)
You need to log in before you can comment on or make changes to this bug.