Closed
Bug 1268294
Opened 9 years ago
Closed 9 years ago
Remove support for Mercurial bundles from hgtool
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: gps, Assigned: gps)
Details
Attachments
(2 obsolete files)
hgtool has support for cloning from bundles. This is no longer necessary as Mercurial itself will clone from bundles where available. And any high volume repo we clone from should be advertising bundles.
Assignee | ||
Comment 1•9 years ago
|
||
Now that we're running Mercurial 3.7.3 everywhere in automation and
Mercurial 3.7.3 supports cloning from bundles transparently, we no
longer have a need for supporting cloning from bundles in hgtool. So we
remove that code.
The command line and API arguments for specifying a bundle are preserved,
as callers in the wild may still use them. However, they now no-op.
Review commit: https://reviewboard.mozilla.org/r/49349/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/49349/
Attachment #8746298 -
Flags: review?(catlee)
Assignee | ||
Updated•9 years ago
|
Assignee: nobody → gps
Status: NEW → ASSIGNED
Assignee | ||
Comment 2•9 years ago
|
||
Now that we're running a Mercurial in production that knows how to
natively consume server-advertised bundles, we no longer need bundle
support in mozharness.
This patch removes most of it. There are still references to bundles in
the vendored hgtool.
Review commit: https://reviewboard.mozilla.org/r/49353/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/49353/
Attachment #8746304 -
Flags: review?(catlee)
Comment 3•9 years ago
|
||
Looks fine after an initial inspection, but what are we going to do about try? Without bundle support, hgtool will do a plain `hg clone -r <rev> ...` to get try checked out on the build machine, which doesn't use the advertised bundles from hg.
Assignee | ||
Comment 4•9 years ago
|
||
Do we seed the AMIs with Mercurial repo snapshots? If so, I have an idea that will handle Try.
Comment 5•9 years ago
|
||
(In reply to Gregory Szorc [:gps] from comment #4)
> Do we seed the AMIs with Mercurial repo snapshots? If so, I have an idea
> that will handle Try.
Yes, but there is code in hgtool that will blow away the /builds/hg-shared/try directory in some cases.
Assignee | ||
Comment 6•9 years ago
|
||
I was planning on adding the magical "auto share" functionality to hgtool next. Perhaps I need to do it as one shot.
I'm curious under what circumstances led to us blowing away the stores in /builds/hg-shared. Those should /never/ get corrupted. Only time I've seen that happen is if a machine loses power during/after a pull or has such an operation aborted forcefully (e.g. SIGKILL).
Comment 7•9 years ago
|
||
It's generally caused by problems pulling in a revision, or it thinking that the remote path has changed for some reason.
If you're up to it, you could also remove support for mirrors in the code - that hasn't been used in ages.
Assignee | ||
Comment 8•9 years ago
|
||
More reason to move forward on this. From http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32/1462211024/mozilla-inbound-win32-bm73-build1-build1691.txt.gz:
11:15:55 INFO - Mercurial Distributed SCM (version 3.7.3)
11:15:55 INFO - command: END (0.09s elapsed)
11:15:55 INFO - Attempting to initialize clone with bundles
11:15:55 INFO - command: START
11:15:55 INFO - command: hg init C:\builds\hg-shared\integration\mozilla-inbound
11:15:55 INFO - command: cwd: c:\builds\moz2_slave\m-in-w32-000000000000000000000
11:15:55 INFO - command: output:
11:15:55 INFO - command: END (0.10s elapsed)
11:15:55 INFO - Trying to use bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg
11:15:55 INFO - command: START
11:15:55 INFO - command: hg unbundle --traceback https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg
11:15:55 INFO - command: cwd: C:\builds\hg-shared\integration\mozilla-inbound
11:15:55 INFO - command: env: {'HGPLAIN': '1'}
11:37:01 INFO - Hit exception running hg:
11:37:01 INFO - Traceback (most recent call last):
11:37:01 INFO - File "c:\builds\moz2_slave\m-in-w32-000000000000000000000\build\tools\buildfarm\utils\../../lib/python\util\hg.py", line 99, in get_hg_output
11:37:01 INFO - return get_output(['hg'] + cmd, timeout=timeout, env=env, **kwargs)
11:37:01 INFO - File "c:\builds\moz2_slave\m-in-w32-000000000000000000000\build\tools\buildfarm\utils\../../lib/python\util\commands.py", line 207, in get_output
11:37:01 INFO - raise error
11:37:01 INFO - CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg']' returned non-zero exit status 255
11:37:01 INFO - Using _rmtree_windows ...
11:43:38 INFO - Using bundles failed; falling back to clone
11:43:38 INFO - command: START
11:43:38 INFO - command: hg clone --traceback -U https://hg.mozilla.org/integration/mozilla-inbound C:\builds\hg-shared\integration\mozilla-inbound
11:43:38 INFO - command: cwd: c:\builds\moz2_slave\m-in-w32-000000000000000000000
11:43:38 INFO - command: env: {'HGPLAIN': '1'}
11:52:21 INFO - command: END (523.75s elapsed)
11:52:21 INFO - command: output:
11:52:21 INFO - applying clone bundle from https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/integration/mozilla-inbound/1d99e6c5a2bf41b983658369b97c5d152649695b.packed1.hg
11:52:21 INFO - 248287 files to transfer, 1.73 GB of data
11:52:21 INFO - transferred 1.73 GB in 440.7 seconds (4.01 MB/sec)
21 minutes wasted for unknown reasons :/
Looking at http://activedata.allizom.org/tools/query.html#query_id=Km7Uryqo, there are plenty of excessively long "checkout-sources" mozharness steps in win32 builders. This is a massive time sink for our Windows automation :/
Assignee | ||
Updated•9 years ago
|
Attachment #8746298 -
Attachment is obsolete: true
Attachment #8746298 -
Flags: review?(catlee)
Assignee | ||
Updated•9 years ago
|
Attachment #8746304 -
Attachment is obsolete: true
Attachment #8746304 -
Flags: review?(catlee)
Assignee | ||
Comment 9•9 years ago
|
||
I'm going to take a different approach that ignores hgtool altogether.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
Assignee | ||
Comment 10•9 years ago
|
||
For posterity, the different approach was bug 1270317.
Updated•7 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•