Closed Bug 1582662 Opened 1 year ago Closed 1 year ago

test_dmd.js fails with artifact builds and breaks xpcshell suites on Try

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(firefox71 fixed)

RESOLVED FIXED
mozilla71
Tracking Status
firefox71 --- fixed

People

(Reporter: jdescottes, Assigned: nalexander)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file)

STRs:

  • build Firefox with artifacts enabled
  • run ./mach xpcshell-test memory/replace/dmd/test/test_dmd.js

Fails with

The details of the failure are as follows:

IOError: Included file '/Users/jdescottes/Development/hg/fx-team-artifact/objdir.noindex/_tests/xpcshell/memory/replace/dmd/test/xpcshell.ini' does not exist

  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/xpcshell/mach_commands.py", line 261, in run_xpcshell_test
    return xpcshell.run_test(**params)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/xpcshell/mach_commands.py", line 60, in run_test
    return self.run_suite(**kwargs)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/xpcshell/mach_commands.py", line 46, in run_suite
    return self._run_xpcshell_harness(**kwargs)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/xpcshell/mach_commands.py", line 136, in _run_xpcshell_harness
    result = xpcshell.runTests(filtered_args)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/xpcshell/runxpcshelltests.py", line 1348, in runTests
    options.get('verify'))
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/xpcshell/runxpcshelltests.py", line 891, in buildTestList
    mp = self.getTestManifest(self.manifest)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/xpcshell/runxpcshelltests.py", line 840, in getTestManifest
    return TestManifest([manifest], strict=True)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 780, in __init__
    ManifestParser.__init__(self, *args, **kwargs)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 88, in __init__
    self.read(*manifests)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 290, in read
    self._read(here, filename, defaults)
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 191, in _read
    include_file = read_file('include:')
  File "/Users/jdescottes/Development/hg/fx-team-artifact/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 117, in read_file
    raise IOError(message)

This is also impacting all try builds using artifacts and attempting to run xpcshell tests, eg
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e238ae5b92ce8104628918b5fa6401a5554502f2

Component: Memory Allocator → General
Product: Core → Firefox Build System

Seems to be a regression from Bug 1580996

Nick: could you take a look at this?

Edit: Some additional info. The expected behavior (before the patch from Bug 1580996) is that the test is skipped and there is no error when running ./mach xpcshell-test memory/replace/dmd/test/test_dmd.js

Flags: needinfo?(nalexander)
Regressed by: 1580996

(In reply to Julian Descottes [:jdescottes] from comment #1)

Seems to be a regression from Bug 1580996

Nick: could you take a look at this?

Yes. This is indeed a change caused by Bug 1580996; I'm not sure it's a regression.

Edit: Some additional info. The expected behavior (before the patch from Bug 1580996) is that the test is skipped and there is no error when running ./mach xpcshell-test memory/replace/dmd/test/test_dmd.js

You've reached an odd corner of the artifact build system. The offending change is, I think, https://hg.mozilla.org/mozilla-central/rev/a86b87abf6fd#l1.100, which I think is impacting the set of test .ini files in the object directory. Before the commit, the object directory would be populated with tests and test .ini files that weren't appropriate for an artifact build (or at least aren't marked as appropriate for an artifact build). After the commit, the object directory doesn't contain those download test .ini files, causing failures like what you see.

Overall, if it makes sense to, and it is helpful to be able to, run DMD tests from an artifact build, we can probably arrange to allow that. It's not that way now because the memory features are disabled unless you have a COMPILE_ENVIRONMENT, and that means the test files aren't installed in an artifact build.

But I wonder if you found this particular issue because some tests weren't skipped in a way you expected, and then you discovered this particular thing that clearly reproduced.

jdescottes: can you comment? I think I don't want to install tests and test .ini files from upstream artifacts directly; that's not particularly sensible. I'm going to flag Chris as well to see if my understanding is incorrect and we actually want that behaviour.

Flags: needinfo?(nalexander)
Flags: needinfo?(jdescottes)
Flags: needinfo?(cmanchester)

(In reply to Nick Alexander :nalexander [he/him] from comment #2)

But I wonder if you found this particular issue because some tests weren't skipped in a way you expected, and then you discovered this particular thing that clearly reproduced.

jdescottes: can you comment? I think I don't want to install tests and test .ini files from upstream artifacts directly; that's not particularly sensible. I'm going to flag Chris as well to see if my understanding is incorrect and we actually want that behaviour.

I am only interested in being able to run xpcshell test suites on try with artifact builds.
All the xpcshell suites are orange if you use artifact builds at the moment:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e238ae5b92ce8104628918b5fa6401a5554502f2

So that's the regression I'm really interested about here :) I thought finding root cause + changeset would help, sorry if it created some confusion.
I don't know anything about this test, but if the suite is not supposed to run on artifact builds, then it shouldn't try to run them on try?

Flags: needinfo?(jdescottes)
Summary: test_dmd.js fails with artifact builds → test_dmd.js fails with artifact builds and breaks xpcshell suites on Try

I was certainly surprised by a couple of things I found along the way here... but I guess my conclusion is that comment 2 is slightly backwards: we start installing ini manifests into the objdir from automation as a result of bug 1580996. The rule to skip them was already in place and prevented them from doing harm as a side effect when we started processing test archives for artifact builds. I'm surprised we still generate "master" test manifests at all, but I guess they're used by automation so I think we should just revert those lines from bug 1580996 and go back to skipping.

Flags: needinfo?(cmanchester)

(In reply to Chris Manchester (:chmanchester) from comment #4)

I was certainly surprised by a couple of things I found along the way here... but I guess my conclusion is that comment 2 is slightly backwards: we start installing ini manifests into the objdir from automation as a result of bug 1580996. The rule to skip them was already in place and prevented them from doing harm as a side effect when we started processing test archives for artifact builds. I'm surprised we still generate "master" test manifests at all, but I guess they're used by automation so I think we should just revert those lines from bug 1580996 and go back to skipping.

That makes more sense, and explains some things I could not. I'm happy to return the lines (with a comment!) and will get to it next week.

Assignee: nobody → nalexander
Status: NEW → ASSIGNED

Bug 1580996 cleaned up handling of {application,platform}.ini but
inadvertently populated .ini files from test archives into the
object directory. That causes issues, especially on try. Test INI
files should come from the local artifact build and not from the
upstream test archives.

Rather than re-instate the test at the time when processed test
archives are unpacked, the test is done as test archives are
packed/processed.

Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c1c08992d48b
Fix xpcshell test suites against artifact builds on try. r=chmanchester
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
You need to log in before you can comment on or make changes to this bug.