1809200 - edge case where we successfully run a manifest of tests, but do not report to errorsummary.json

Status: RESOLVED FIXED

(Testing :: General, defect, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

this seems to be more common with xorig.

for example (xorig 2):
https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&selectedTaskRun=U4ajMMkpQsKTd7xrB7BnlA.0&test_paths=extensions%2Fformautofill%2Ftest%2Fmochitest&revision=8453a8c7c4fbec00850c934bef5be2c54fe30778&searchStr=mochitest%2Cdebug%2Cxorig%2Cosx

https://firefoxci.taskcluster-artifacts.net/U4ajMMkpQsKTd7xrB7BnlA/0/public/test_info/mochitest-plain_errorsummary.log

https://firefoxci.taskcluster-artifacts.net/U4ajMMkpQsKTd7xrB7BnlA/0/public/logs/live_backing.log

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

osx xorig:
TEST-START | extensions/formautofill/test/mochitest/test_address_level_1_submission.html

osx regular:
TEST-START | browser/extensions/formautofill/test/mochitest/test_address_level_1_submission.html

so we strip browser/ for xorig, that is probably somewhat related.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

ok, this seems to be when the path starts with browser/, all other instances seem to be showing up. I haven't found code that specifically strips this, but there are many layers in manifest parsing, url munging, and then reporting from the test -> js harness -> stdout -> python harness -> different log files.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

this appears to be fixed; I recall fixing something like this a few months ago, but cannot find the bug/change too easily.

Comment 5

[Marco Castelluccio [:marco]]

Any chance we could add some test case to make this no longer happen?
Can we easily find the changeset that fixed this by looking at past logs?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

this was fixed with Bug 1844880. As for unittests, the fix was:

     TEST_PATH_PREFIXES = [
         r"^/tests/",
-        r"^\w+://[\w\.]+(:\d+)?(/\w+)?/(tests?|a11y|chrome|browser)/",
+        r"^\w+://[\w\.]+(:\d+)?(/\w+)?/(tests?|a11y|chrome)/",
+        r"^\w+://[\w\.]+(:\d+)?(/\w+)?/(tests?|browser)/",
     ]

I am not sure if we could make a foolproof unittest for avoiding namespace conflicts.