1380383 - Prevent sm-pkg from failing anytime it can't find some random unrelated file

Status: NEW

(Firefox Build System :: General, enhancement)

Description

[Andrew Halberstadt [:ahal]]

The sm-pkg task generates a package by picking and choosing various directories from the source (including moz.build) and then building with the included moz.build:
https://dxr.mozilla.org/mozilla-central/source/js/src/make-source-package.sh#69

Unfortunately, this means that if you add a file to any of those moz.builds (such as a test manifest) and neglect to remember to also copy that file in make-source-package.sh, you'll get backed out.

Is there some way we could do one of:
A) add a mode to the build system to not fail if stuff is missing
B) package this more sanely
C) automatically run sm-pkg on try anytime any of its included files are modified

I'd prefer A or B, but will gladly settle for C at this point.

Comment 1

[Andrew Halberstadt [:ahal]]

Comment 0 should read:
"by choosing various files and directories from the source"

Which leads to:

D) only copy entire directories into the package, never individual files

and while we're at it:

E) stop using a package and use a source checkout instead

Comment 2

[Mike Hommey [:glandium]]

F) Don't add gecko tests to moz.build in ways that affect non gecko builds.

But since you're not saying what you were doing, I can't tell you how you were doing it wrong. ;)

Comment 3

[Andrew Halberstadt [:ahal]]

In this case (bug 1380338) I added a python manifest:
https://hg.mozilla.org/try/diff/a6751ca9f7aa/taskcluster/moz.build

That isn't a gecko test and needs to be read even with a clobbered objdir. In order to land that patch, I need to add the entire taskcluster/taskgraph/test directory to the package.

Also moz.build is just an example. I've been backed out for moving a file out of one of the directories it includes as well.

Comment 4

[Andrew Halberstadt [:ahal]]

I'll admit I might be more likely to hit breakage here than anyone else due to the nature of the things I tend to modify. So maybe this isn't very high priority, but it's a valid complaint.

Comment 5

[Mike Hommey [:glandium]]

Well, the whole problem is that the taskcluster directory shouldn't be added to DIRS unconditionally in the first place.

Comment 6

[Andrew Halberstadt [:ahal]]

Some python manifests (like the one in taskcluster/moz.build) need to be parsed unconditionally, they don't depend on any build state. E.g this needs to work:

$ ./mach clobber
$ ./mach python-test

But that's not the whole problem, there are several ways we can break sm-pkg:

1. Move a python module that's added to build/virtualenv_packages.txt
2. Add a test to an unconditional manifest like: [../path/to/test]
3. Reference a file from an included makefile

In my experience, sm-pkg is very fragile and it's hard to remember to run it on try.