Closed Bug 1268294 Opened 9 years ago Closed 9 years ago

Remove support for Mercurial bundles from hgtool

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: gps, Assigned: gps)

Details

Attachments

(2 obsolete files)

hgtool has support for cloning from bundles. This is no longer necessary as Mercurial itself will clone from bundles where available. And any high volume repo we clone from should be advertising bundles.
Now that we're running Mercurial 3.7.3 everywhere in automation and Mercurial 3.7.3 supports cloning from bundles transparently, we no longer have a need for supporting cloning from bundles in hgtool. So we remove that code. The command line and API arguments for specifying a bundle are preserved, as callers in the wild may still use them. However, they now no-op. Review commit: https://reviewboard.mozilla.org/r/49349/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/49349/
Attachment #8746298 - Flags: review?(catlee)
Assignee: nobody → gps
Status: NEW → ASSIGNED
Now that we're running a Mercurial in production that knows how to natively consume server-advertised bundles, we no longer need bundle support in mozharness. This patch removes most of it. There are still references to bundles in the vendored hgtool. Review commit: https://reviewboard.mozilla.org/r/49353/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/49353/
Attachment #8746304 - Flags: review?(catlee)
Looks fine after an initial inspection, but what are we going to do about try? Without bundle support, hgtool will do a plain `hg clone -r <rev> ...` to get try checked out on the build machine, which doesn't use the advertised bundles from hg.
Do we seed the AMIs with Mercurial repo snapshots? If so, I have an idea that will handle Try.
(In reply to Gregory Szorc [:gps] from comment #4) > Do we seed the AMIs with Mercurial repo snapshots? If so, I have an idea > that will handle Try. Yes, but there is code in hgtool that will blow away the /builds/hg-shared/try directory in some cases.
I was planning on adding the magical "auto share" functionality to hgtool next. Perhaps I need to do it as one shot. I'm curious under what circumstances led to us blowing away the stores in /builds/hg-shared. Those should /never/ get corrupted. Only time I've seen that happen is if a machine loses power during/after a pull or has such an operation aborted forcefully (e.g. SIGKILL).
It's generally caused by problems pulling in a revision, or it thinking that the remote path has changed for some reason. If you're up to it, you could also remove support for mirrors in the code - that hasn't been used in ages.
More reason to move forward on this. From http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32/1462211024/mozilla-inbound-win32-bm73-build1-build1691.txt.gz: 11:15:55 INFO - Mercurial Distributed SCM (version 3.7.3) 11:15:55 INFO - command: END (0.09s elapsed) 11:15:55 INFO - Attempting to initialize clone with bundles 11:15:55 INFO - command: START 11:15:55 INFO - command: hg init C:\builds\hg-shared\integration\mozilla-inbound 11:15:55 INFO - command: cwd: c:\builds\moz2_slave\m-in-w32-000000000000000000000 11:15:55 INFO - command: output: 11:15:55 INFO - command: END (0.10s elapsed) 11:15:55 INFO - Trying to use bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg 11:15:55 INFO - command: START 11:15:55 INFO - command: hg unbundle --traceback https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg 11:15:55 INFO - command: cwd: C:\builds\hg-shared\integration\mozilla-inbound 11:15:55 INFO - command: env: {'HGPLAIN': '1'} 11:37:01 INFO - Hit exception running hg: 11:37:01 INFO - Traceback (most recent call last): 11:37:01 INFO - File "c:\builds\moz2_slave\m-in-w32-000000000000000000000\build\tools\buildfarm\utils\../../lib/python\util\hg.py", line 99, in get_hg_output 11:37:01 INFO - return get_output(['hg'] + cmd, timeout=timeout, env=env, **kwargs) 11:37:01 INFO - File "c:\builds\moz2_slave\m-in-w32-000000000000000000000\build\tools\buildfarm\utils\../../lib/python\util\commands.py", line 207, in get_output 11:37:01 INFO - raise error 11:37:01 INFO - CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg']' returned non-zero exit status 255 11:37:01 INFO - Using _rmtree_windows ... 11:43:38 INFO - Using bundles failed; falling back to clone 11:43:38 INFO - command: START 11:43:38 INFO - command: hg clone --traceback -U https://hg.mozilla.org/integration/mozilla-inbound C:\builds\hg-shared\integration\mozilla-inbound 11:43:38 INFO - command: cwd: c:\builds\moz2_slave\m-in-w32-000000000000000000000 11:43:38 INFO - command: env: {'HGPLAIN': '1'} 11:52:21 INFO - command: END (523.75s elapsed) 11:52:21 INFO - command: output: 11:52:21 INFO - applying clone bundle from https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/integration/mozilla-inbound/1d99e6c5a2bf41b983658369b97c5d152649695b.packed1.hg 11:52:21 INFO - 248287 files to transfer, 1.73 GB of data 11:52:21 INFO - transferred 1.73 GB in 440.7 seconds (4.01 MB/sec) 21 minutes wasted for unknown reasons :/ Looking at http://activedata.allizom.org/tools/query.html#query_id=Km7Uryqo, there are plenty of excessively long "checkout-sources" mozharness steps in win32 builders. This is a massive time sink for our Windows automation :/
Attachment #8746298 - Attachment is obsolete: true
Attachment #8746298 - Flags: review?(catlee)
Attachment #8746304 - Attachment is obsolete: true
Attachment #8746304 - Flags: review?(catlee)
I'm going to take a different approach that ignores hgtool altogether.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
For posterity, the different approach was bug 1270317.
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: