699407 - Use hg mirrors wherever possible

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Lots of related bugs here, but I couldn't find one that was for actually using the mirrors...so here it is!

Comment 1

[Chris AtLee [:catlee]]

Attachment: Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 571644 [details] [diff] [review]
Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.

Most of this is refactoring some of the hgtool steps into a function called makeHgToolStep which knows how to handle bundles and mirrors.

I also set HG_SHARE_BASE_DIR for l10n jobs if we have it available in the platform's environment.

Comment 3

[Chris AtLee [:catlee]]

Attachment: Set mirror/bundle urls, and turn on shared checkouts for more platforms

Comment 4

[Justin Wood (:Callek)]

Comment on attachment 571644 [details] [diff] [review]
Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.


>+    def makeHgtoolStep(self, name='hg_update', repo_url=None, wc=None,
>+            mirrors=None, bundles=None, env=None,
>+            clone_by_revision=False, rev=None, workdir='build',
>+            use_properties=True, locks=None):
>+
>+        if not env:
>+            env = self.env
>+        env = env.copy()
>+        if use_properties:
>+            env['PROPERTIES_FILE'] = 'buildprops.json'

Nit, do env.copy() only if we plan to set any new env vars. (which is only this one place, so easy to nest)

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 571646 [details] [diff] [review]
Set mirror/bundle urls, and turn on shared checkouts for more platforms

Review of attachment 571646 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/staging_config.py
@@ +68,5 @@
>      'tinderbox_tree': 'MozillaTest',
>      'mobile_tinderbox_tree': 'MobileTest',
>      'hg_username': 'stage-ffxbld',
> +    'base_mirror_urls': ['http://hg.build.scl1.mozilla.com'],
> +    'base_bundle_urls': ['http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles'],

Are you sure you want to be using ftp.m.o here? Seems like dev-stage01 should be used instead.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 571644 [details] [diff] [review]
Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.

Review of attachment 571644 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with the below change.

::: process/factory.py
@@ +641,5 @@
> +
> +        if locks is None:
> +            locks = []
> +
> +        return ShellCommand(

Per IRC, we should use RetryingShellCommand here.

Comment 7

[Chris AtLee [:catlee]]

Attachment: Don't capture output in retry.py unless we need to

hgtool's output is uglified by retry.py since stderr and stdout are separated. this fixes that problem since we're not grepping for anything in hgtool's output.

Comment 8

[Chris AtLee [:catlee]]

Attachment: Set mirror/bundle urls, and turn on shared checkouts for more platforms

Comment 9

[Armen [:armenzg]]

I merge a change(s) from this bug to the production branches and reconfigured the masters around 1630 PDT.