1252572 - SM builds are busted on ESR45 due to lack of hg bundles

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.allizom.org/logviewer.html#?job_id=2194&repo=mozilla-esr45

 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-esr45.hg']' returned non-zero exit status 255
 Using bundles failed; falling back to clone
 command: START
 command: hg clone --traceback -U -r 342ccad2c0576f7220ecdbcdbb873eebedf0b88b https://hg.mozilla.org/releases/mozilla-esr45 /builds/hg-shared/releases/mozilla-esr45
 command: cwd: /builds/slave/m-esr45_lx-d_sm-arm-sim-000000
 command: env: {'HGPLAIN': '1'}
 process, is taking too long: 1800.12192011s (timeout 1800s).Terminating
 Hit exception running hg:
 Traceback (most recent call last):
   File "/builds/slave/m-esr45_lx-d_sm-arm-sim-000000/scripts/buildfarm/utils/../../lib/python/util/hg.py", line 99, in get_hg_output
     return get_output(['hg'] + cmd, timeout=timeout, env=env, **kwargs)
   File "/builds/slave/m-esr45_lx-d_sm-arm-sim-000000/scripts/buildfarm/utils/../../lib/python/util/commands.py", line 217, in get_output
     raise error
CalledProcessError: <unprintable CalledProcessError object>

Comment 1

[Steve Fink [:sfink] [:s:]]

Attachment: Stop passing --mirror and --bundle

We want to move over to clonebundles, so these options just cause problems now.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I'm going to create an esr45 bundle and put it up on archive.m.o, as RyanVM says this is affecting other builds too. I have some work underway to stop using these bundles but it's blocked on a issue for windows try builds.

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8725349 [details] [diff] [review]
Stop passing --mirror and --bundle

lgtm

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

http://archive.mozilla.org/pub/firefox/bundles/mozilla-esr45.hg is up now (ftp-ssl.mozilla.org is effectively a CNAME to that). RESOLVED FIXED ?

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Indeed, retriggers are running successfully now and the logs are showing us using the newly-created bundle.

Steve, since we use hg 3.2.1 + bundleclone.py in automation (and therefore, cloning from the CDN should Just Work), I think we should close this bug out without patching spidermonkey.sh and fix things on the buildbot side to not use --mirror and --bundle anymore. Sound reasonable to you?

Comment 6

[Gregory Szorc [:gps]]

hg.mozilla.org is now configured to produce esr45 bundles as well. They should appear within a few hours.

Comment 7

[Gregory Szorc [:gps]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #5)
> Steve, since we use hg 3.2.1 + bundleclone.py in automation (and therefore,
> cloning from the CDN should Just Work), I think we should close this bug out
> without patching spidermonkey.sh and fix things on the buildbot side to not
> use --mirror and --bundle anymore. Sound reasonable to you?

Yes. We should be using clonebundles/bundleclone + the auto pooling feature of the "share" extension everywhere. The latter will require Mercurial 3.5, which we don't appear to have widely deployed yet.

There are a crap ton of cloning related performance improvements in modern versions of hg. If you update to 3.7.2 (should be released later today), I can almost guarantee clones will become dozens of seconds if not minutes faster on all machines.

Comment 8

[Steve Fink [:sfink] [:s:]]

(oops, found an orphaned tab with this uncommitted comment)

(In reply to Gregory Szorc [:gps] from comment #7)
> (In reply to Ryan VanderMeulen [:RyanVM] from comment #5)
> > Steve, since we use hg 3.2.1 + bundleclone.py in automation (and therefore,
> > cloning from the CDN should Just Work), I think we should close this bug out
> > without patching spidermonkey.sh and fix things on the buildbot side to not
> > use --mirror and --bundle anymore. Sound reasonable to you?

Sure, fine with me. I'll probably want to land the patch sometime anyway, though, since I think it's just dead code now. But that can happen later, when there's no risk of timing things in the wrong order.

> Yes. We should be using clonebundles/bundleclone + the auto pooling feature
> of the "share" extension everywhere. The latter will require Mercurial 3.5,
> which we don't appear to have widely deployed yet.
> 
> There are a crap ton of cloning related performance improvements in modern
> versions of hg. If you update to 3.7.2 (should be released later today), I
> can almost guarantee clones will become dozens of seconds if not minutes
> faster on all machines.

From what I can tell, this would require building a new mozilla-python27-mercurial RPM with the updated hg. I don't know who does those?

Comment 9

[Steve Fink [:sfink] [:s:]]

Not planning on doing anything here.