2008815 - WPT sync bot sometimes leaves behind straggling bug annotations for no-longer-reproducing test failures

Status: NEW

(Testing :: web-platform-tests, defect)

Description

[Daniel Holbert [:dholbert]]

I noticed some cases where the wpt sync bot seems to have removed an expected-failure for a subtest, but left behind the string-match and bug annotation for the subtest.

This seems to have happened here, as one example:
https://hg-edge.mozilla.org/mozilla-central/rev/019a0c094d34348f5ffc8e2c0a1c6545a333b8cf#l54.2

   [generateKeyFrame(null) resolves for video sender, and throws for video receiver]
     bug: 1709960
-    expected:
-      if isolated_process: FAIL
 
   [generateKeyFrame throws NotAllowedError for invalid rid]
     bug: 1709960
-    expected:
-      if isolated_process: FAIL
 
   [generateKeyFrame throws NotFoundError for unknown rid]
     bug: 1709960
-    expected:
-      if isolated_process: FAIL

It looks like the bot cleans up cases where we have a subtest name, followed by an expectation, followed by a bug number (the bot cleaned up a bunch of these further down). But if the bug number comes before the expectation, then the bot fails to clean it up.

This has resulted in examples like the following where we've got non-functional annotations left behind in our .ini files:

  [generateKeyFrame(null) resolves for video sender, and throws for video receiver]
    bug: 1709960

...in files like these:
https://searchfox.org/firefox-main/rev/33bba5cfe4a89dda0ee07fa9fbac578353713fd3/testing/web-platform/meta/webrtc-encoded-transform/script-transform-sendKeyFrameRequest.https.html.ini#10-14

https://searchfox.org/firefox-main/rev/33bba5cfe4a89dda0ee07fa9fbac578353713fd3/testing/web-platform/meta/webrtc-encoded-transform/script-transform-generateKeyFrame.https.html.ini#12-25