1181597 - Deploy HG bundleclone extension to Windows

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Mark Cornmesser [:markco] OOO 2024/04/15]

https://bugzilla.mozilla.org/show_bug.cgi?id=1144872

Comment 1

[Gregory Szorc [:gps]]

Making the summary a little more search friendly, since "bundleclone" is the name of the extension.

Comment 2

[Mark Cornmesser [:markco] OOO 2024/04/15]

From looking at the documents that are included in the exe install. We may be able to install the bundleclone extension without out installing hg as a python package. It would require adding the path to the file under the extension header in the mercurial.ini file.

Comment 3

[Justin Wood (:Callek)]

In hg you can add extensions in the exe install *however* those extensions can't use core python features NOT used in core hg itself. Basically if core hg isn't using the module (from core python) it gets stripped out when the py2exe magic happens.

Which is why bundleclone can't work this way.

Comment 4

[Gregory Szorc [:gps]]

bundleclone should not use any Python modules not used by Mercurial itself, so it should be safe to use with the py2exe Mercurial. If it doesn't work with a py2exe Python, please file a bug and I'll make it work.

https://hg.mozilla.org/hgcustom/version-control-tools/file/db2299809957/hgext/bundleclone/__init__.py#l156

FWIW, someone told me the Mercurial package rework wasn't required for bundleclone: bundleclone was merely providing a convenient excuse to do proper Mercurial packaging on Windows.

Comment 5

[Chris AtLee [:catlee]]

FWIW, we are successfully using other extensions with the existing hg exe.

bundleclone does require managing the mercurial.ini file though, to make sure we're using bundles from the proper s3 region.

Comment 6

[Chris AtLee [:catlee]]

Mercurial 3.6 includes bundleclone, so no need to deploy it as a separate extension if we upgrade. We do still need to manage the configuration file, however.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

How would we implement this now that we have AWS builders ? Might it be easier if we're generating golden images like we do from linux, backed by puppet ? We'd have to see what we're doing to manage the config file on linux though (eg prefers in modules/mercurial/templates/hgrc.erb)

I'm interested because I'd like to deprecate generating bundles ourselves (bug 1229532).

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

We hit this in a release build today:

'hg' 'clone' 'https://hg.mozilla.org/build/tools' 'scripts'
 in dir c:\builds\moz2_slave\rel-m-beta-w32_rpk_4-000000000\. (timeout 1320 secs)
(ignoring URL on server that requires SNI)
(your Python is older than 2.7.9 and does not support modern and secure SSL/TLS; please consider upgrading your Python to a secure version)
downloading bundle https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/build/tools/0947372c292f8d3e0a4c5fd7b7bbc4276e3f7067.stream-legacy.hg
Traceback (most recent call last):
  File "mercurial\dispatch.pyc", line 140, in _runcatch
  File "mercurial\dispatch.pyc", line 850, in _dispatch
  File "mercurial\dispatch.pyc", line 611, in runcommand
  File "mercurial\dispatch.pyc", line 941, in _runcommand
  File "mercurial\dispatch.pyc", line 912, in checkargs
  File "mercurial\dispatch.pyc", line 847, in <lambda>
  File "mercurial\util.pyc", line 677, in check
  File "mercurial\commands.pyc", line 1371, in clone
  File "mercurial\hg.pyc", line 423, in clone
  File "C:/mozilla-build/hg/bundleclone.py", line 560, in clone
Abort: error fetching bundle: [Errno 8] _ssl.c:510: EOF occurred in violation of protocol
abort: error fetching bundle: [Errno 8] _ssl.c:510: EOF occurred in violation of protocol
(consider contacting the server operator if this error persists)
program finished with exit code 255

Same error on three separate slaves actually:
http://archive.mozilla.org/pub/firefox/candidates/43.0b9-candidates/build2/logs/release-mozilla-beta-win32_repack_1-bm72-build1-build1.txt.gz
http://archive.mozilla.org/pub/firefox/candidates/43.0b9-candidates/build2/logs/release-mozilla-beta-win32_repack_4-bm72-build1-build1.txt.gz
http://archive.mozilla.org/pub/firefox/candidates/43.0b9-candidates/build2/logs/release-mozilla-beta-win32_repack_7-bm72-build1-build1.txt.gz

So have we already got a hg+bundleclone, or hg >= 3.6 ? Then we hit some error which looks like ssl certs.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Those 3 autoretried OK on other instances, and the other 17 jobs of the same class seem to have not hit that error. I would say it was something that was fixed in a new golden AMI but b-2008-spot-167 is using ami-9bbeacfa, and that's the same as the most recently launched instances. The other two failures were on older instances: b-2008-spot-125 is on ami-f22b3993, and b-2008-spot-196 on ami-4cfcee2d.

Worth noting that the three failing jobs were us-west-2 slaves getting a bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/bu..., so looks like the current state is that we have bundleclone but not the config file which keeps transfers in-region.

</end bugblurt>

Comment 10

[Mark Cornmesser [:markco] OOO 2024/04/15]

We have had bundle clone in use since we started building in AWS. I missed updating the bug here. I suspect maybe something was happening with the hg caching we are doing. 

grenade: any thoughts on the failures above? And can we do something in regards to the keep the transfers in region through the user data. Maybe set the region as a variable in the mercurial.ini file and have the userdata set the value for the variable?

Comment 11

[Gregory Szorc [:gps]]

SSL cert issues smells like using a) old hg b) Python <2.7.9 c) bundleclone before bug 1201993.

Comment 12

[Rob Thijssen [:grenade (EET/UTC+0300)]]

just checked versions:

C:\Users\Administrator>python --version
Python 2.7.5

C:\Users\Administrator>hg --version
Mercurial Distributed SCM (version 3.2.1)

bundleclone diff with latest, shows it's quite old.

Comment 13

[Gregory Szorc [:gps]]

Please upgrade bundleclone to latest version ASAP.

Comment 14

[Mark Cornmesser [:markco] OOO 2024/04/15]

We are on the verge of switching over to Puppet managing the configuration of the golden AMIs. Which means shortly this version of bundleclone.py will be live on the builders: http://hg.mozilla.org/build/puppet/file/tip/modules/mercurial/files/bundleclone.py 

I am not sure if that is the latest version or not. If it is not should would update it for POSIX as well?

Comment 15

[Gregory Szorc [:gps]]

The bundleclone.py checked into puppet is ancient. Please sync it with https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/bundleclone/__init__.py. This will get you:

* Proper detection (and avoidance of) SNI
* Compatibility with Mercurial 3.5 and 3.6

Comment 16

[Mark Cornmesser [:markco] OOO 2024/04/15]

With HG 3.7.3 this is not needed.