Closed Bug 1543834 Opened 7 months ago Closed 7 months ago

Migrate one xul test document to html and confirm that we don't break intermittent tracking in treeherder

Categories

(Toolkit :: General, task)

task
Not set

Tracking

()

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: bgrins, Assigned: bgrins)

References

Details

Attachments

(1 file)

Let's find a single XUL test that has an intermittent failure to migrate to HTML in order to make sure the new test file gets tagged up properly in treeherder/bugzilla. From an earlier discussion about this:

Treeherder very crudely extracts search terms from each failure line [1] and then uses those to search against the bug summaries of bugs with the keyword intermittent-failure. If the rename affects the string that appears as the second to last token in the pipe-symbol delimited string (ie TEST-UNEXPECTED-FAIL | <test name> | <some message that isn't used for search term>), then it will affect the bug suggestions.

To prevent this, the bug summaries would need to be renamed, however older branches (eg mozilla-beta) will still be using the old test filename, so if preserving suggestions there was considered necessary (I'd ask the sheriffs), the bug summary will need to contain both the old and new test name, at least until the rename works it's way through the trains.

[1] https://github.com/mozilla/treeherder/blob/9778665a81d65b8eac0f5fd427a6059eece5afa0/treeherder/model/error_summary.py#L103-L160

From https://mozilla.logbot.info/treeherder/20190411#c16203988 it sounds like we can update the summary of https://bugzilla.mozilla.org/show_bug.cgi?id=1410000 from:

Intermittent toolkit/content/tests/chrome/test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

To something like:

Intermittent toolkit/content/tests/chrome/test_largemenu.html or toolkit/content/tests/chrome/test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

which would allow it to pick up intermittents from either m-c or beta. A few notes:

  1. It could be done manually here to confirm it works
  2. There is a 255 character limit on the summary which we wouldn't hit in this case but could hit with other files. Wes, would it be OK to drop the full file path from one of the filenames if that were to happen? Like "Intermittent toolkit/content/tests/chrome/test_largemenu.html test_largemenu.xul" or something?
Flags: needinfo?(wkocher)
Assignee: nobody → bgrinstead
Status: NEW → ASSIGNED
See Also: → 1410000
Pushed by bgrinstead@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2fb7a78d5bb5
Migrate test_largemenu.xul to test_largemenu.html to make sure intermittent tracking on treeherder survives a rename r=bdahl

Wes, would it be OK to drop the full file path from one of the filenames if that were to happen? Like "Intermittent toolkit/content/tests/chrome/test_largemenu.html test_largemenu.xul" or something?

It only needs to match the file name, and it's a substring search, so it can be in a comma separated list, and there doesn't even need to be spaces separating them. The paths can be omitted completely. You should be able to just list the old and new names.

In your example, it could be Intermittent test_largemenu.html,test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

Flags: needinfo?(wkocher)
Status: ASSIGNED → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
Type: defect → task
Blocks: 1548152
You need to log in before you can comment on or make changes to this bug.