Make bundleclone extension available inside mock

RESOLVED WONTFIX

Status

Release Engineering
General Automation
RESOLVED WONTFIX
2 years ago
8 months ago

People

(Reporter: catlee, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments)

(Reporter)

Description

2 years ago
We have bundleclone installed at the host level, and we refer to it from our hgrc. However, it's not available inside mock which leads to confusing log messages e.g.

command: START
command: hg parent --template {node|short}
command: cwd: mozilla-beta
command: env: {'HGPLAIN': '1'}
*** failed to import extension bundleclone from /usr/local/lib/hgext/bundleclone.py: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
command: END (0.50s elapsed)

Also some jobs clone large repositories from inside mock, and would benefit from the speed increase here.

I can think of two approaches:
- Update the various lists of files we specify to copy into the mock env to include bundleclone
- Update the base mock environments to include bundleclone and any required configuration
(Reporter)

Comment 1

2 years ago
This looks interesting...

# config_opts['files']['path/name/no/leading/slash'] = "put file contents here."

Comment 2

2 years ago
Per IRC conversation with catlee, this is almost certainly the cause of the high load to hg.mozilla.org this morning. As part of the release process coupled with an increase in the amount of concurrent l10n builders, full clones of mozilla-release and mozilla-esr38 contributed significant load to hg.mozilla.org. We definitely flirted with a few capacity limits. Had these clones been serviced from S3-backed bundles, we wouldn't have had a problem.
(Reporter)

Comment 3

2 years ago
simply adding to mock_files fails because the leading directories don't exist:

11:33:01     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 --copyin --unpriv /usr/local/lib/hgext/bundleclone.py /usr/local/lib/hgext/bundleclone.py
11:33:01     INFO -  INFO: mock_mozilla.py version 1.0.3 starting...
11:33:01     INFO -  State Changed: init plugins
11:33:01     INFO -  INFO: selinux disabled
11:33:01     INFO -  State Changed: start
11:33:01     INFO -  State Changed: lock buildroot
11:33:01     INFO -  Mock Version: 1.0.3
11:33:01     INFO -  INFO: Mock Version: 1.0.3
11:33:01     INFO -  INFO: copying /usr/local/lib/hgext/bundleclone.py to /builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py
11:33:01     INFO -  ERROR: [Errno 2] No such file or directory: '/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py'
11:33:01     INFO -  Traceback (most recent call last):
11:33:01     INFO -    File "/usr/sbin/mock_mozilla", line 862, in <module>
11:33:01     INFO -      main(retParams)
11:33:01     INFO -    File "/usr/sbin/mock_mozilla", line 823, in main
11:33:01     INFO -      shutil.copy(src, dest)
11:33:01     INFO -    File "/usr/lib64/python2.6/shutil.py", line 84, in copy
11:33:01     INFO -      copyfile(src, dst)
11:33:01     INFO -    File "/usr/lib64/python2.6/shutil.py", line 51, in copyfile
11:33:01     INFO -      with open(dst, 'wb') as fdst:
11:33:01     INFO -  IOError: [Errno 2] No such file or directory: '/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py'
11:33:01    ERROR - Return code: 1
(Reporter)

Comment 4

2 years ago
Created attachment 8645152 [details]
Dockerfile to build bundleclone rpm
(Reporter)

Comment 5

2 years ago
Created attachment 8645153 [details]
spec file for bundleclone rpm
(Reporter)

Comment 6

2 years ago
if anybody wants to pick this up while I'm away, you can use this spec file / dockerfile to generate an rpm that will install bundleclone.py into /usr/local/lib/hgext.

I've put up a generated copy here: http://people.mozilla.org/~catlee/mercurial-bundleclone-1.0-1.el6.noarch.rpm

Still TODO:
- write puppet patches to install this rpm as part of the base mock package (modify mock configs)
- spec file cleanup
- publishing rpm to rpm repos

Comment 7

2 years ago
Since bundleclone canonically lives in version-control-tools, we can put the RPM generation in it. Although, we'll need to pin the install location, and /usr/local may not be right for all scenarios. So maybe this is releng-specific enough to not live in v-c-t.
You can work around the leading directories issue (comment #3) by specifying /usr/local/lib/hgext (and recursive copy takes care of the rest). I'm testing that out for releases with https://hg.mozilla.org/users/stage-ffxbld/buildbot-configs/rev/ac01212b5048.
Created attachment 8661536 [details] [diff] [review]
[buildbot-configs] Copy in bundleclone when setting up mock environment

As we've discussed, this should resolve the Traceback's we're getting with Fennec 41.0 build1 multi-locale. It works in staging with the slave's .hgrc set to traceback on errors.
Attachment #8661536 - Flags: review?(jlund)
Comment on attachment 8661536 [details] [diff] [review]
[buildbot-configs] Copy in bundleclone when setting up mock environment

Review of attachment 8661536 [details] [diff] [review]:
-----------------------------------------------------------------

r+
Attachment #8661536 - Flags: review?(jlund) → review+
Comment on attachment 8661536 [details] [diff] [review]
[buildbot-configs] Copy in bundleclone when setting up mock environment

https://hg.mozilla.org/build/buildbot-configs/rev/0b08becdf2b5
https://hg.mozilla.org/build/buildbot-configs/rev/65ef8a7712e3

Build masters reconfiged.
Attachment #8661536 - Flags: checked-in+
Created attachment 8661557 [details] [diff] [review]
150915_1191859_make_bundleclone_available_within_mock-mozharness.patch

the mozharness equivalent.

some of these changes will almost certainly be required to fix android nightlies
Attachment #8661557 - Flags: review?(nthomas)
Attachment #8661557 - Flags: review?(nthomas) → review+

Comment 13

2 years ago
https://hg.mozilla.org/mozilla-central/rev/3e8dde8f8c17
Comment on attachment 8661557 [details] [diff] [review]
150915_1191859_make_bundleclone_available_within_mock-mozharness.patch

https://hg.mozilla.org/mozilla-central/rev/3e8dde8f8c17
https://hg.mozilla.org/releases/mozilla-aurora/rev/7484c6088282

That should sort preempt bustage of Android nightlies. We'll have to wait and see if the release-style single-locale repacks blow up, and possibly uplift further.
Attachment #8661557 - Flags: checked-in+
(Reporter)

Updated

8 months ago
Status: NEW → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.