Use hg mirrors wherever possible

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: catlee, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [hg])

Attachments

(3 attachments, 1 obsolete attachment)

Lots of related bugs here, but I couldn't find one that was for actually using the mirrors...so here it is!
Created attachment 571644 [details] [diff] [review]
Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.
Comment on attachment 571644 [details] [diff] [review]
Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.

Most of this is refactoring some of the hgtool steps into a function called makeHgToolStep which knows how to handle bundles and mirrors.

I also set HG_SHARE_BASE_DIR for l10n jobs if we have it available in the platform's environment.
Attachment #571644 - Attachment description: Set → Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.
Attachment #571644 - Flags: review?(bhearsum)
Created attachment 571646 [details] [diff] [review]
Set mirror/bundle urls, and turn on shared checkouts for more platforms
Attachment #571646 - Flags: review?(bhearsum)
Blocks: 699028
Blocks: 687064
Comment on attachment 571644 [details] [diff] [review]
Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.


>+    def makeHgtoolStep(self, name='hg_update', repo_url=None, wc=None,
>+            mirrors=None, bundles=None, env=None,
>+            clone_by_revision=False, rev=None, workdir='build',
>+            use_properties=True, locks=None):
>+
>+        if not env:
>+            env = self.env
>+        env = env.copy()
>+        if use_properties:
>+            env['PROPERTIES_FILE'] = 'buildprops.json'

Nit, do env.copy() only if we plan to set any new env vars. (which is only this one place, so easy to nest)
Comment on attachment 571646 [details] [diff] [review]
Set mirror/bundle urls, and turn on shared checkouts for more platforms

Review of attachment 571646 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/staging_config.py
@@ +68,5 @@
>      'tinderbox_tree': 'MozillaTest',
>      'mobile_tinderbox_tree': 'MobileTest',
>      'hg_username': 'stage-ffxbld',
> +    'base_mirror_urls': ['http://hg.build.scl1.mozilla.com'],
> +    'base_bundle_urls': ['http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles'],

Are you sure you want to be using ftp.m.o here? Seems like dev-stage01 should be used instead.
Comment on attachment 571644 [details] [diff] [review]
Set base mirror, base bundle urls; use hg share for l10n jobs; add makeHgToolStep and use it.

Review of attachment 571644 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with the below change.

::: process/factory.py
@@ +641,5 @@
> +
> +        if locks is None:
> +            locks = []
> +
> +        return ShellCommand(

Per IRC, we should use RetryingShellCommand here.
Attachment #571644 - Flags: review?(bhearsum) → review+
Created attachment 572052 [details] [diff] [review]
Don't capture output in retry.py unless we need to

hgtool's output is uglified by retry.py since stderr and stdout are separated. this fixes that problem since we're not grepping for anything in hgtool's output.
Attachment #572052 - Flags: review?(bhearsum)
Attachment #572052 - Flags: review?(bhearsum) → review+
Created attachment 572901 [details] [diff] [review]
Set mirror/bundle urls, and turn on shared checkouts for more platforms
Attachment #571646 - Attachment is obsolete: true
Attachment #571646 - Flags: review?(bhearsum)
Attachment #572901 - Flags: review?(bhearsum)
Attachment #572901 - Flags: review?(bhearsum) → review+
Attachment #572052 - Flags: checked-in+
Attachment #571644 - Flags: checked-in+
Attachment #572901 - Flags: checked-in+
Flags: needs-reconfig?

Comment 9

6 years ago
I merge a change(s) from this bug to the production branches and reconfigured the masters around 1630 PDT.
Flags: needs-reconfig?
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.