Closed
Bug 1191859
Opened 9 years ago
Closed 7 years ago
Make bundleclone extension available inside mock
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: catlee, Unassigned)
References
Details
Attachments
(4 files)
726 bytes,
text/plain
|
Details | |
760 bytes,
text/plain
|
Details | |
19.08 KB,
patch
|
jlund
:
review+
nthomas
:
checked-in+
|
Details | Diff | Splinter Review |
23.58 KB,
patch
|
nthomas
:
review+
nthomas
:
checked-in+
|
Details | Diff | Splinter Review |
We have bundleclone installed at the host level, and we refer to it from our hgrc. However, it's not available inside mock which leads to confusing log messages e.g. command: START command: hg parent --template {node|short} command: cwd: mozilla-beta command: env: {'HGPLAIN': '1'} *** failed to import extension bundleclone from /usr/local/lib/hgext/bundleclone.py: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py' command: END (0.50s elapsed) Also some jobs clone large repositories from inside mock, and would benefit from the speed increase here. I can think of two approaches: - Update the various lists of files we specify to copy into the mock env to include bundleclone - Update the base mock environments to include bundleclone and any required configuration
Reporter | ||
Comment 1•9 years ago
|
||
This looks interesting... # config_opts['files']['path/name/no/leading/slash'] = "put file contents here."
Comment 2•9 years ago
|
||
Per IRC conversation with catlee, this is almost certainly the cause of the high load to hg.mozilla.org this morning. As part of the release process coupled with an increase in the amount of concurrent l10n builders, full clones of mozilla-release and mozilla-esr38 contributed significant load to hg.mozilla.org. We definitely flirted with a few capacity limits. Had these clones been serviced from S3-backed bundles, we wouldn't have had a problem.
Reporter | ||
Comment 3•9 years ago
|
||
simply adding to mock_files fails because the leading directories don't exist: 11:33:01 INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 --copyin --unpriv /usr/local/lib/hgext/bundleclone.py /usr/local/lib/hgext/bundleclone.py 11:33:01 INFO - INFO: mock_mozilla.py version 1.0.3 starting... 11:33:01 INFO - State Changed: init plugins 11:33:01 INFO - INFO: selinux disabled 11:33:01 INFO - State Changed: start 11:33:01 INFO - State Changed: lock buildroot 11:33:01 INFO - Mock Version: 1.0.3 11:33:01 INFO - INFO: Mock Version: 1.0.3 11:33:01 INFO - INFO: copying /usr/local/lib/hgext/bundleclone.py to /builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py 11:33:01 INFO - ERROR: [Errno 2] No such file or directory: '/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py' 11:33:01 INFO - Traceback (most recent call last): 11:33:01 INFO - File "/usr/sbin/mock_mozilla", line 862, in <module> 11:33:01 INFO - main(retParams) 11:33:01 INFO - File "/usr/sbin/mock_mozilla", line 823, in main 11:33:01 INFO - shutil.copy(src, dest) 11:33:01 INFO - File "/usr/lib64/python2.6/shutil.py", line 84, in copy 11:33:01 INFO - copyfile(src, dst) 11:33:01 INFO - File "/usr/lib64/python2.6/shutil.py", line 51, in copyfile 11:33:01 INFO - with open(dst, 'wb') as fdst: 11:33:01 INFO - IOError: [Errno 2] No such file or directory: '/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/local/lib/hgext/bundleclone.py' 11:33:01 ERROR - Return code: 1
Reporter | ||
Comment 4•9 years ago
|
||
Reporter | ||
Comment 5•9 years ago
|
||
Reporter | ||
Comment 6•9 years ago
|
||
if anybody wants to pick this up while I'm away, you can use this spec file / dockerfile to generate an rpm that will install bundleclone.py into /usr/local/lib/hgext. I've put up a generated copy here: http://people.mozilla.org/~catlee/mercurial-bundleclone-1.0-1.el6.noarch.rpm Still TODO: - write puppet patches to install this rpm as part of the base mock package (modify mock configs) - spec file cleanup - publishing rpm to rpm repos
Comment 7•9 years ago
|
||
Since bundleclone canonically lives in version-control-tools, we can put the RPM generation in it. Although, we'll need to pin the install location, and /usr/local may not be right for all scenarios. So maybe this is releng-specific enough to not live in v-c-t.
Comment 8•9 years ago
|
||
You can work around the leading directories issue (comment #3) by specifying /usr/local/lib/hgext (and recursive copy takes care of the rest). I'm testing that out for releases with https://hg.mozilla.org/users/stage-ffxbld/buildbot-configs/rev/ac01212b5048.
Comment 9•9 years ago
|
||
As we've discussed, this should resolve the Traceback's we're getting with Fennec 41.0 build1 multi-locale. It works in staging with the slave's .hgrc set to traceback on errors.
Attachment #8661536 -
Flags: review?(jlund)
Comment 10•9 years ago
|
||
Comment on attachment 8661536 [details] [diff] [review] [buildbot-configs] Copy in bundleclone when setting up mock environment Review of attachment 8661536 [details] [diff] [review]: ----------------------------------------------------------------- r+
Attachment #8661536 -
Flags: review?(jlund) → review+
Comment 11•9 years ago
|
||
Comment on attachment 8661536 [details] [diff] [review] [buildbot-configs] Copy in bundleclone when setting up mock environment https://hg.mozilla.org/build/buildbot-configs/rev/0b08becdf2b5 https://hg.mozilla.org/build/buildbot-configs/rev/65ef8a7712e3 Build masters reconfiged.
Attachment #8661536 -
Flags: checked-in+
Comment 12•9 years ago
|
||
the mozharness equivalent. some of these changes will almost certainly be required to fix android nightlies
Attachment #8661557 -
Flags: review?(nthomas)
Updated•9 years ago
|
Attachment #8661557 -
Flags: review?(nthomas) → review+
Comment 14•9 years ago
|
||
Comment on attachment 8661557 [details] [diff] [review] 150915_1191859_make_bundleclone_available_within_mock-mozharness.patch https://hg.mozilla.org/mozilla-central/rev/3e8dde8f8c17 https://hg.mozilla.org/releases/mozilla-aurora/rev/7484c6088282 That should sort preempt bustage of Android nightlies. We'll have to wait and see if the release-style single-locale repacks blow up, and possibly uplift further.
Attachment #8661557 -
Flags: checked-in+
Reporter | ||
Updated•7 years ago
|
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
Assignee | ||
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•