1029924 - Allow to install additional packages in mock

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Mike Hommey [:glandium]]

I had this idea to abuse try to build clang, gcc, and gtk3 packages for tooltool, which would be a more reliable way to do those builds than relying on people doing the right thing (which requires knowledge they might not have).

But building those require a few packages that aren't in our Firefox build environments. It would be nice to have a way to install additional packages in the mock environment.

What I was thinking was to add a build step that would take a manifest file in the tree (path of which would be defined in config.py), and run mock install with the list of packages that manifest contains.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Allow to install additional packages in mock

I don't feel like going through all the hurdles of testing this on staging, but as long as there is no syntax error, this is a no-op until there is a buildbot-config change, and we can test that by enabling it on a barely used try target (like, shell root analysis)

Comment 2

[Mike Hommey [:glandium]]

Attachment: Allow to install additional packages in mock on try

Comment 3

[Chris AtLee [:catlee]]

tempted to WONTFIX this given our efforts to migrate firefox builds to mozharness. it's much easier to do there.

is there an immediate need for this?

Comment 4

[Mike Hommey [:glandium]]

I wanted to setup regular builds of gcc trunk.

Comment 5

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #4)
> I wanted to setup regular builds of gcc trunk.

and it would have helped me when I had to update gtk for gtk3 builds.

Can we go forward here?

Comment 6

[Jordan Lund (:jlund)]

(In reply to Mike Hommey [:glandium] from comment #5)
> (In reply to Mike Hommey [:glandium] from comment #4)
> > I wanted to setup regular builds of gcc trunk.
> 
> and it would have helped me when I had to update gtk for gtk3 builds.
> 
> Can we go forward here?

mentioned in irc:

1) if we only care about the try case, we can put in additional mozharness args via the try commit message:

bug ref https://bugzil.la/967630

essentially doing this[1] here[2]

2) have mozharness look for an in-tree config that has its own list of mock packages (I think this is better)

essentially doing something like this totally untested pseudo code:

diff --git a/mozharness/mozilla/mock.py b/mozharness/mozilla/mock.py
index f8587c0..27cfc63 100644
--- a/mozharness/mozilla/mock.py
+++ b/mozharness/mozilla/mock.py
@@ -191,7 +191,16 @@ class MockMixin(object):
             old_packages_hash = None

         if mock_packages is None:
-            mock_packages = list(c.get('mock_packages'))
+            in_tree_mock_pkgs_file = os.path.join(self.query_abs_dirs()['abs_src_dir'],
+                                                  c.get('in_tree_mock_packages_path'))
+            if os.path.exists(in_tree_mock_pkgs_file):
+                # right now setup_mock is called before we download src (at least in the desktop
+                # ff case) so we would have to reverse that or else have in_tree_mock_packages_path
+                # be a raw file url that we download ahead of src download e.g.
+                "https://hg.mozilla.org/{branch}/raw-file/{rev}/browser/config/mock_packages.py'
+                mock_packages = self.parse_config_file(in_tree_mock_pkgs_file)['mock_packages']
+            else:
+                mock_packages = list(c.get('mock_packages'))

         package_list_hash = hashlib.new('sha1')
         if mock_packages:

would that solve your issue?
    

[1] http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/testing/talos.py?rev=6412b5ff437e#199
[2] http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/mock.py#194

Comment 7

[Mike Hommey [:glandium]]

The latter seems more interesting, because it could allow to more generally move the list in-tree, which would allow to unclutter those lists, too.

Comment 8

[Chris AtLee [:catlee]]

we should be using task cluster for gtk3.

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 8445606 [details] [diff] [review]
Allow to install additional packages in mock

Review of attachment 8445606 [details] [diff] [review]:
-----------------------------------------------------------------

I don't want more stuff going into buildbot factories that we don't require.

Comment 10

[Mike Hommey [:glandium]]

Note that when this bug was filed and the patch attached a year ago, buildbot factories were the only place where this could be hooked.