1179476 - use in gecko tree mozharness across all buildbot based jobs

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jordan Lund (:jlund)]

this bug will be broken up into a few parts

part 1 - enable mh in-tree for c-i jobs that are using mozharness pinning on a project branch (ash)

part 2 - enable mh in-tree for c-i jobs that are using mozharness pinning across all branches

part 3 - enable mh in-tree for all factory based jobs that clone mh as a build step (e.g. android nightly builds)

part 4 - enable mh in-tree for all desktop and mobile l10n jobs (right now they are not using mh pinning)

part 5 - enable mh in-tree for all release jobs

Comment 1

[Jordan Lund (:jlund)]

Attachment: 150707_bug_1179476_MozillaBuildFactory_support_for_mozharness_in_tree-bbotcustom.patch

this is to support the few edge cases where we download mozharness in factory based jobs. e.g. b2g32 gecko builds

Comment 2

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #1)
> Created attachment 8630725 [details] [diff] [review]
> 150707_bug_1179476_MozillaBuildFactory_support_for_mozharness_in_tree-
> bbotcustom.patch
> 
> this is to support the few edge cases where we download mozharness in
> factory based jobs. e.g. b2g32 gecko builds

this addresses part 3 from comment 0

part 1 and 2 is being covered in https://bugzilla.mozilla.org/show_bug.cgi?id=1154801

Comment 3

[Jordan Lund (:jlund)]

Attachment: 150707_bug_1179476_l10n_support_for_mozharness_in_tree-bbotcustom.patch

this is part 4, desktop_l10n and mobile_l10n support.

These builders were missing the required repo_path property that is used in the archiver client call so I added it.

Comment 4

[Jordan Lund (:jlund)]

Attachment: 150708_bug_1179476_release_support_for_mozharness_in_tree-bbotcustom.patch

this is part 5 of this bug

Rail, I was wondering if you could help me with this. Or maybe point me to someone who can. I am quite unfamiliar with release.py and releases themselves.

my logic here was:

  1) find all instances where we clone mozharness in release.py.
  2) see if there was a rev that I could use to get mozharness from. In this case, I found that we use tags (the same tag across the gecko tree and our build repos).
  3) find the release.py equivalent of repo_path. I found sourceRepoInfo['clonePath']
  4) add support for tags in ScriptFactory and archiver_client.py[1]

This code is untested. What's the normal procedure here for patching release.py? Do release folks run staging builds before every release? Do you have any current release builds you could run in staging with this?

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1181331#c4

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8630776 [details] [diff] [review]
150707_bug_1179476_l10n_support_for_mozharness_in_tree-bbotcustom.patch

feel free to bounce this review. There are not many mozharness l10n folks in releng anymore :)

this is part 4 of this bug. See https://bugzilla.mozilla.org/show_bug.cgi?id=1179476#c3

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8631085 [details] [diff] [review]
150708_bug_1179476_release_support_for_mozharness_in_tree-bbotcustom.patch

Review of attachment 8631085 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +908,5 @@
>                          scriptName='scripts/mobile_l10n.py',
>                          extra_args=extra_args,
>                          use_credentials_file=True,
>                          env=env,
> +                        relengapi_archiver_repo_path=sourceRepoInfo['clonePath'],

sourceRepoInfo['clonePath'] won't work, it's a staging only variable.

sourceRepoInfo['path'] will work for firefox and fennec, but not for TB.

Something like this may work:

relengapi_archiver_repo_path = sourceRepoInfo['path']
if sourceRepoKey == "comm":
    relengapi_archiver_repo_path = releaseConfig['sourceRepositories']['mozilla']['path']

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8630776 [details] [diff] [review]
150707_bug_1179476_l10n_support_for_mozharness_in_tree-bbotcustom.patch

Review of attachment 8630776 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 8

[Jordan Lund (:jlund)]

Attachment: 150708_bug_1179476_release_support_for_mozharness_in_tree-bbotcustom2.patch

thanks for your help thus far

interdiff: http://people.mozilla.org/~jlund/150708_mh-in-tree_release_support-bbotcustom-interdiff.patch

Comment 9

[Jordan Lund (:jlund)]

Attachment: 150709_bug_1154801_scriptfactory_gecko_mozharness-test_fix-bbotcustom.patch

all instances where ScriptFactory was used in release.py was patched however I missed ReleaseBuildFactory (which inherits from MercurialBuildFactory).

We need to pass a --tag to those instances as they do not have a revision property. This solution is akin to ScriptFactory solution:
https://bugzilla.mozilla.org/show_bug.cgi?id=1179476#c4
https://bugzilla.mozilla.org/show_bug.cgi?id=1179476#c8

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8631897 [details] [diff] [review]
150709_bug_1154801_scriptfactory_gecko_mozharness-test_fix-bbotcustom.patch

wrong patch. sorryf for the noise

Comment 11

[Jordan Lund (:jlund)]

Attachment: 150709_bug_1179476_release_support_for_mozharness_in_tree-MBF_fix-bbotcustom.patchb

same applies as https://bugzilla.mozilla.org/show_bug.cgi?id=1179476#c9

sorry, I need about 50 context lines to get the class in release.py (it is ReleaseBuildFactory)

Comment 12

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/5eb51c675b71

Comment 13

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/375f5b855039

Comment 14

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/4095a3b801e1

Comment 15

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/2bab32c607c1

Comment 16

[Jordan Lund (:jlund)]

this has been live and used everywhere since last week. Resolving