1154796 - make a copy of mozharness and put it in gecko tree

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jordan Lund (:jlund)]

this will initially be for internal testing (so it does not have to be mirrored)

context: http://jordan-lund.ghost.io/mozharness-steps-into-the-forest/

Comment 1

[Jordan Lund (:jlund)]

this ticket tracks:

part 1 - putting a copy of mozharness on ash project repo

part 2 - putting a copy of mozharness on trunk (m-c) related repos

part 3 - putting a copy of mozharness across all stabilizing/release repos

Comment 2

[Jordan Lund (:jlund)]

part 1 is done:
   https://hg.mozilla.org/projects/ash/file/8ca917a6b1e2/testing/mozharness
   noop change: https://treeherder.mozilla.org/#/jobs?repo=ash&revision=8ca917a6b1e2

Comment 3

[Jordan Lund (:jlund)]

Attachment: 150702_bug_1154796_in_gecko_tree_mozharness_copy-m-c.patch

as discussed over irc yesterday. This is a straight up copy of hg.m.o/build/mozharness with two differences:

1) there is no README.md
2) I modified the existing README.txt to point out that this is a copy and based off of: http://hg.mozilla.org/build/mozharness/rev/50903ae436e5

I left mozharness.json in there for now so that as we transition, I don't break existing jobs that are still using the pin.

this patch will be a no-op until we turn it on everywhere.

Comment 4

[Michael Shal [:mshal]]

Comment on attachment 8629158 [details] [diff] [review]
150702_bug_1154796_in_gecko_tree_mozharness_copy-m-c.patch

Looks good! Sorry for already bitrotting you in bug 1179382 - feel free to bring in the recent production branch and carry r+ forward.

Comment 5

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9e6987be9b33

Comment 6

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/9e6987be9b33

Comment 7

[Jordan Lund (:jlund)]

Attachment: 150702_bug_1154796_in_gecko_tree_mozharness_sync_43f7600b8c80.patch

this is asking for two things:

for a review of an update to the in-tree copy of mozharness to sync with latest prod mh repo rev: 43f7600b8c80

and to be able to uplift this across all the branches with r=mshal, a=testing

again, it will be a no-op until I land: https://bugzilla.mozilla.org/attachment.cgi?id=8630722

Comment 8

[Michael Shal [:mshal]]

Comment on attachment 8630741 [details] [diff] [review]
150702_bug_1154796_in_gecko_tree_mozharness_sync_43f7600b8c80.patch

It looks like a few files were removed in the latest production rev of mozharness, but are still in-tree with this patch:

Only in ./configs: update_tests
Only in ./configs/vcs_sync: staging_build-repos.py
Only in ./configs/vcs_sync: staging_l10n.py

Comment 9

[Jordan Lund (:jlund)]

Attachment: 150708_bug_1154796_in_gecko_tree_mozharness_sync_43f7600b8c80.patch

thanks for the catch. verified this time by diffing deleted files.

Comment 10

[Michael Shal [:mshal]]

Comment on attachment 8631113 [details] [diff] [review]
150708_bug_1154796_in_gecko_tree_mozharness_sync_43f7600b8c80.patch

Looks good!

Comment 11

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/e25de9972a77

Comment 12

[Jordan Lund (:jlund)]

Attachment: 150709_1154796_sync_to_4a31e6739409.patch

verified locally with (only dotfiles differ):

jlund@Hastings163:~/devel/mozilla/cleanRepos/mozilla-central/testing/mozharness
> diff -r . $DOWNLOADS/mozharness-4a31e6739409 > /tmp/diff

Comment 13

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/95a40a305e1b

Comment 14

[Jordan Lund (:jlund)]

Comment on attachment 8631685 [details] [diff] [review]
150709_1154796_sync_to_4a31e6739409.patch

thanks

https://hg.mozilla.org/mozilla-central/rev/95a40a305e1b

Comment 15

[Jordan Lund (:jlund)]

Attachment: 150716_1154796_sync_to_239491bc393a.patch

Comment 16

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/3d800e3a022e
https://hg.mozilla.org/mozilla-central/rev/e89a57634b17

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/43020b934bba
https://hg.mozilla.org/releases/mozilla-aurora/rev/0e2b4433dfcd

Comment 18

[Ryan VanderMeulen [:RyanVM]]

On second thought, we decided on IRC to go with mozharness rev cd537fcb6ff0 as the final base rev for the release branches.

https://hg.mozilla.org/releases/mozilla-aurora/rev/91c53b279514

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/527271c3a4bc

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr38/rev/c19fd7e5e4cc

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Jordan, what is the canonical location of mozharness development at the moment? Is it still https://hg.mozilla.org/build/mozharness/ or https://hg.mozilla.org/mozilla-central/testing/mozharness?

I ask now because I got the information that I should land all my patches to mozilla-central and not in the other repository given that it is not used anymore. But yesterday I have seen that several test harnesses report a script_revlink of mozharness to Treeherder which still points to /build/mozharness. Als checking this repository I see active development! I'm kinda confused now about where to land changes.

Comment 22

[Armen [:armenzg]]

That script_revlink should be dropped.

The Firefox OS team was landing there since they had not yet moved to the in-tree mozharness.
We should make the original place read/only.

Comment 23

[Jordan Lund (:jlund)]

(In reply to Henrik Skupin (:whimboo) from comment #21)
> Jordan, what is the canonical location of mozharness development at the
> moment? Is it still https://hg.mozilla.org/build/mozharness/ or
> https://hg.mozilla.org/mozilla-central/testing/mozharness?

https://hg.mozilla.org/mozilla-central/testing/mozharness

there is still a small tail of things that need cleaning up: https://bugzil.la/1180060 and I just dep'd a small bug specifically to remove script_revlink: bug 1208117

all mh automation scripts bar some releng services (b2g_bumper, vcs_sync) should be using the gecko-tree based mh. please land there. we can't make the original read/only as b2g_bumper/vcs_sync still need to push there. It is forked. We could rename the build/mozharness to clear up confusion.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Jordan. Good to know that all of our code landed correctly then.