1436767 - Install mercurial + robustcheckout on OSX generic-worker hosts

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Andrew Halberstadt [:ahal]]

I'm trying to get run-task [1] based tasks working with taskcluster's generic-worker. But the OSX hosts don't have mercurial (or robustcheckout [2]) installed.

I was told this would be configured in puppetagain.

[1] https://searchfox.org/mozilla-central/source/taskcluster/docker/recipes/run-task
[2] https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hgext/robustcheckout

Comment 1

[Kendall Libby [:fubar] (he/him)]

We definitely have mercurial installed; and we had talked about removing robustcheckout in the latest mercurial update (bug 1390493) but I don't actually see us removing it in any of the patches. I'll have Dragos help you out (when you're back).

Comment 2

[Andrew Halberstadt [:ahal]]

Ah, maybe it's just not in the PATH of the worker user. I'll dig out my patches and do a fresh try run.

Comment 3

[Dragos Crisan [:dragrom]]

:ahal Is the issue still present?

Comment 4

[Andrew Halberstadt [:ahal]]

Just re-pushed my old patch series, and this is the error:
https://taskcluster-artifacts.net/L_7RAhZKS8iv1F7K_uGtGg/0/public/logs/live_backing.log

So if hg is installed, it's not on the PATH. Would it be easy to add to the path? I'd prefer not having to hardcode it in the run-task script. (This is from https://treeherder.mozilla.org/#/jobs?repo=try&revision=8a003fcffa8eeb89673e87d8704a72df0d2beaee)

Comment 5

[Dragos Crisan [:dragrom]]

I saw this comment on a bug for update mercurial, it refers to robustcheckout: https://bugzilla.mozilla.org/show_bug.cgi?id=1390493#c15. If the robustcheckout need to be installed, please open a separate  bug for this.

Comment 6

[Andrew Halberstadt [:ahal]]

I don't have access to that bug, could you CC me?

Comment 7

[Dragos Crisan [:dragrom]]

I added to CC:

Comment 8

[Andrew Halberstadt [:ahal]]

Thanks.

Yeah, so looks like it was removed just to keep things clean. Sorry, we'll need it added back again. If you like we could morph this bug to be just about robustcheckout since hg is already installed.

(Though on that note, how can I find where hg is?)

Comment 9

[Dragos Crisan [:dragrom]]

Attachment: Bug_1436767_Install_robustchexkout.patch

Install robustcheckout extension to OSX workers

Comment 10

[Dragos Crisan [:dragrom]]

:ahal I installed robustcheckout extension on t-yosemite-r7-380.test.releng.mdc1.mozilla.com machine. Please run tests on it and let me know if everything is OK

Comment 11

[Andrew Halberstadt [:ahal]]

Hey Dragos, sorry I'm not sure how to connect to that. If there's a default user/password I'm not privy to them and if there's a way to point taskcluster at it I'm not sure how. My normal workflow is to just push to try and read the logs.

Though to be honest, there's not really much to test. If it exists somewhere on the machine, I can point mercurial at it and everything should work out fine. I'd just need to know the installation path.

Comment 12

[Gregory Szorc [:gps]]

For the record, `hg robustcheckout` should be used for *all* clones and checkouts in CI. Robustcheckout ensures that clones and checkouts are performed optimally. It prevents us from using patterns that can result in accidental DoS against hg.mozilla.org. These optimal patterns also reduce the overhead of VCS operations in CI, making tasks complete faster. Robustcheckout also retries certain operations, thus reducing intermittent failures.

I consider any use of `hg clone` and `hg update` in Firefox CI to be a bug. `hg robustcheckout` should be used instead.

Comment 13

[Jake Watkins [:dividehex]]

Comment on attachment 8960478 [details] [diff] [review]
Bug_1436767_Install_robustchexkout.patch

Review of attachment 8960478 [details] [diff] [review]:
-----------------------------------------------------------------

I would like see a README file in the modules/mercurial/files dir that at least gives the original location of the rebustcheckout source.  If someone needs to update this extension to the latest version then they will have a link to refer to.

I believe the source is in the VCT repo.
https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hgext/robustcheckout

::: modules/mercurial/manifests/ext/robustcheckout.pp
@@ +4,5 @@
> +class mercurial::ext::robustcheckout {
> +    include mercurial::ext::common
> +
> +    file {
> +        "$mercurial::settings::hgext_dir/robustcheckout.py":

Please use curly braces to enclose the variable name.  eg. "${variable}/string_constant"

Comment 14

[Dragos Crisan [:dragrom]]

Attachment: Bug_1436767_Install_robustcheckout.patch

Install robustcheckout on OSX workers. I corrected the robustcheckout class and added a README file with the roubustcheckout source

Comment 15

[Dragos Crisan [:dragrom]]

I tested roubustcheckout on t-yosemite-r7-380.test.releng.mdc1.mozilla.com. I received the following errors:

[[e-mail] ~]$ hg robustcheckout --sharebase 'checkouts/hg-store' --upstream 'https://hg.mozilla.org/mozilla-unified' --revision '8a003fcffa8eeb89673e87d8704a72df0d2beaee' https://hg.mozilla.org/try  checkouts/gecko
ensuring https://hg.mozilla.org/try@8a003fcffa8eeb89673e87d8704a72df0d2beaee is available at checkouts/gecko
Traceback (most recent call last):
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/scmutil.py", line 154, in callcatch
    return func()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 303, in _runcatchfunc
    return _dispatch(req)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 940, in _dispatch
    cmdpats, cmdoptions)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 701, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 948, in _runcommand
    return cmdfunc()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 937, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 1183, in check
    return func(*args, **kwargs)
  File "/usr/local/lib/hgext/robustcheckout.py", line 149, in robustcheckout
    sharebase)
  File "/usr/local/lib/hgext/robustcheckout.py", line 159, in _docheckout
    destvfs = scmutil.vfs(dest, audit=False, realpath=True)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/hgdemandimport/demandimportpy2.py", line 142, in __call__
    raise TypeError("%s object is not callable" % repr(self))
TypeError: <unloaded module 'vfs'> object is not callable
** Unknown exception encountered with possibly-broken third-party extension bundleclone
** which supports versions 3.6 of Mercurial.
** Please disable bundleclone and try your action again.
** If that fixes the bug please report it to https://bugzilla.mozilla.org/enter_bug.cgi?product=Developer%20Services&component=Mercurial%3A%20bundleclone
** Python 2.7.3 (default, Oct 14 2014, 13:07:22) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)]
** Mercurial Distributed SCM (version 4.4.1)
** Extensions loaded: share, rebase, strip, mq, purge, bundleclone, robustcheckout
Traceback (most recent call last):
  File "/usr/local/bin/hg", line 41, in <module>
    dispatch.run()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 85, in run
    status = (dispatch(req) or 0) & 255
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 173, in dispatch
    ret = _runcatch(req)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 313, in _runcatch
    return _callcatch(ui, _runcatchfunc)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 321, in _callcatch
    return scmutil.callcatch(ui, func)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/scmutil.py", line 154, in callcatch
    return func()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 303, in _runcatchfunc
    return _dispatch(req)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 940, in _dispatch
    cmdpats, cmdoptions)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 701, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 948, in _runcommand
    return cmdfunc()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 937, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 1183, in check
    return func(*args, **kwargs)
  File "/usr/local/lib/hgext/robustcheckout.py", line 149, in robustcheckout
    sharebase)
  File "/usr/local/lib/hgext/robustcheckout.py", line 159, in _docheckout
    destvfs = scmutil.vfs(dest, audit=False, realpath=True)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/hgdemandimport/demandimportpy2.py", line 142, in __call__
    raise TypeError("%s object is not callable" % repr(self))
TypeError: <unloaded module 'vfs'> object is not callable

:ahal Can you help me with this error?

Comment 16

[Jake Watkins [:dividehex]]

Comment on attachment 8960979 [details] [diff] [review]
Bug_1436767_Install_robustcheckout.patch

Review of attachment 8960979 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM, Thanks for the corrections. :-)

Comment 17

[Andrew Halberstadt [:ahal]]

Re-directing to :gps, I'm not really familiar with robustcheckout and I'm not sure what's going on there :/

Comment 18

[Gregory Szorc [:gps]]

Remove the bundleclone extension. It is deprecated, as the functionality is now in modern versions of Mercurial.

Comment 19

[Dragos Crisan [:dragrom]]

I removed bundleclone extension, and I received this error:

[[e-mail] ~]$ hg robustcheckout --sharebase 'checkouts/hg-store' --upstream 'https://hg.mozilla.org/mozilla-unified' --revision '8a003fcffa8eeb89673e87d8704a72df0d2beaee' https://hg.mozilla.org/try  checkouts/gecko
ensuring https://hg.mozilla.org/try@8a003fcffa8eeb89673e87d8704a72df0d2beaee is available at checkouts/gecko
Traceback (most recent call last):
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/scmutil.py", line 154, in callcatch
    return func()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 303, in _runcatchfunc
    return _dispatch(req)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 940, in _dispatch
    cmdpats, cmdoptions)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 701, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 948, in _runcommand
    return cmdfunc()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 937, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 1183, in check
    return func(*args, **kwargs)
  File "/usr/local/lib/hgext/robustcheckout.py", line 149, in robustcheckout
    sharebase)
  File "/usr/local/lib/hgext/robustcheckout.py", line 159, in _docheckout
    destvfs = scmutil.vfs(dest, audit=False, realpath=True)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/hgdemandimport/demandimportpy2.py", line 142, in __call__
    raise TypeError("%s object is not callable" % repr(self))
TypeError: <unloaded module 'vfs'> object is not callable
** unknown exception encountered, please report by visiting
** https://mercurial-scm.org/wiki/BugTracker
** Python 2.7.3 (default, Oct 14 2014, 13:07:22) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)]
** Mercurial Distributed SCM (version 4.4.1)
** Extensions loaded: share, rebase, strip, mq, purge, robustcheckout
Traceback (most recent call last):
  File "/usr/local/bin/hg", line 41, in <module>
    dispatch.run()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 85, in run
    status = (dispatch(req) or 0) & 255
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 173, in dispatch
    ret = _runcatch(req)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 313, in _runcatch
    return _callcatch(ui, _runcatchfunc)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 321, in _callcatch
    return scmutil.callcatch(ui, func)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/scmutil.py", line 154, in callcatch
    return func()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 303, in _runcatchfunc
    return _dispatch(req)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 940, in _dispatch
    cmdpats, cmdoptions)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 701, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 948, in _runcommand
    return cmdfunc()
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 937, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 1183, in check
    return func(*args, **kwargs)
  File "/usr/local/lib/hgext/robustcheckout.py", line 149, in robustcheckout
    sharebase)
  File "/usr/local/lib/hgext/robustcheckout.py", line 159, in _docheckout
    destvfs = scmutil.vfs(dest, audit=False, realpath=True)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/hgdemandimport/demandimportpy2.py", line 142, in __call__
    raise TypeError("%s object is not callable" % repr(self))
TypeError: <unloaded module 'vfs'> object is not callable

Comment 20

[Gregory Szorc [:gps]]

That error is due to using an ancient version of robustcheckout. This issue was fixed in bug 1359959.

Please vendor the latest version of robustcheckout, which can always be downloaded from https://hg.mozilla.org/hgcustom/version-control-tools/raw-file/@/hgext/robustcheckout/__init__.py.

Comment 21

[Dragos Crisan [:dragrom]]

Updated the roubustcheckout version. Now everything looks fine.

[[e-mail] ~]$ hg robustcheckout --sharebase 'checkouts/hg-store' --upstream 'https://hg.mozilla.org/mozilla-unified' --revision '8a003fcffa8eeb89673e87d8704a72df0d2beaee' https://hg.mozilla.org/try  checkouts/gecko
(using Mercurial 4.4.1)
ensuring https://hg.mozilla.org/try@8a003fcffa8eeb89673e87d8704a72df0d2beaee is available at checkouts/gecko
warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info
(cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
(sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info
applying clone bundle from https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-unified/6e032627821402c26a1cacf6043ba48c260b409c.zstd-max.hg
warning: connecting to s3-us-west-2.amazonaws.com using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info
adding changesets
adding manifests                                                                                                                                                                                            
adding file changes                                                                                                                                                                                         
added 456359 changesets with 2639695 changes to 430786 files (+10 heads)                                                                                                                                    
finished applying clone bundle
warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info
searching for changes
adding changesets
adding manifests                                                                                                                                                                                            
adding file changes
added 2 changesets with 2 changes to 2 files
new changesets 653ca8a2d6ae:fe7964ba2561
warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0); see https://mercurial-scm.org/wiki/SecureConnections for more info
searching for changes
no changes found

Comment 22

[Dragos Crisan [:dragrom]]

Attachment: Bug_1436767_Install_robustcheckout.patch

This patch install roubustcheckout on OSX workers. Changes from the previews patch:
- updatet robustcheckout.py script with the latest version
- removed bundleclone mercurial extension (https://bugzilla.mozilla.org/show_bug.cgi?id=1436767#c18)

Comment 23

[Gregory Szorc [:gps]]

Those "warning: connecting to hg.mozilla.org using legacy security technology (TLS 1.0)" messages mean that the Python being used to run Mercurial doesn't support modern crypto. i.e. it is using the system's /usr/bin/python or the Python it is using is older than 2.7.9 or isn't compiled against OpenSSL. I consider that a security bug and would encourage this to be addressed. But in a different bug though.

Comment 24

[Gregory Szorc [:gps]]

We just committed some changes to robustcheckout. Namely https://hg.mozilla.org/hgcustom/version-control-tools/rev/f8e922e5539f will help us debug intermittent failures in CI. So taking the latest version from version-control-tools would be much appreciated. It can be done as a follow-up change easily enough though. I don't want to delay this bug any further.

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8961767 [details] [diff] [review]
Bug_1436767_Install_robustcheckout.patch

Review of attachment 8961767 [details] [diff] [review]:
-----------------------------------------------------------------

This seems reasonable to me.

Comment 26

[Dragos Crisan [:dragrom]]

Attachment: Bug_1436767_Install_robustcheckout.patch

This patch install roubustcheckout on OSX workers. Changes from the previews patch:
- updatet robustcheckout.py script with the latest version
- removed bundleclone mercurial extension (https://bugzilla.mozilla.org/show_bug.cgi?id=1436767#c18)
- Updated the robustcheckout to the latest version (https://bugzilla.mozilla.org/show_bug.cgi?id=1436767#c24)

Comment 27

[Dragos Crisan [:dragrom]]

Comment on attachment 8962333 [details] [diff] [review]
Bug_1436767_Install_robustcheckout.patch

Pushed to default: https://hg.mozilla.org/build/puppet/rev/0e136d97b74e355bc26f157ae9b76aa0b488c4e8
Pushed to production: https://hg.mozilla.org/build/puppet/rev/268b1cd9f2df8c0c68c444602ad16e56e065a234

Comment 28

[Andrew Halberstadt [:ahal]]

Thanks Dragos! I'll give my work in bug 1436037 another stab and let you know if I run into problems.