714063 - SingleSourceFactory should use intree mozconfigs if they exist

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Rail Aliiev [:rail]]

ATM we use mozconfigs from buildbot-configs.

Comment 1

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 2

[Aki Sasaki (not active)]

I may be doing this in bug 748796.

Comment 3

[Aki Sasaki (not active)]

Not actively working on this.

Comment 4

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-buildbotcustom.diff

While I'm touching some release automation, I'll take a look at this one too.

Comment 5

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-buildbot-configs.diff

Comment 6

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-buildbotcustom-1.diff

v2, with tooltool enabled

Comment 7

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-buildbotcustom-2.diff

Moar cleanup, make mozconfig required

Comment 8

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-mozharness.diff

Looks like this is dead too

Comment 9

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-buildbot-configs.diff

refresh

Comment 10

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-tools.diff

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 8558720 [details] [diff] [review]
SingleSourceFactory-buildbot-configs.diff

Review of attachment 8558720 [details] [diff] [review]:
-----------------------------------------------------------------

diediediedideideideie

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 8558719 [details] [diff] [review]
SingleSourceFactory-mozharness.diff

Review of attachment 8558719 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/multi_locale/mozilla-aurora_android.json
@@ -25,5 @@
>      "hg_l10n_tag": "default",
>      "l10n_dir": "mozilla-aurora",
>      "merge_locales": true,
> -    "mozilla_dir": "build",
> -    "mozconfig": "build/configs/mozilla2/mobile/android/mozilla-aurora/nightly/mozconfig"

Why is mozconfig no longer needed?

Comment 13

[Rail Aliiev [:rail]]

(In reply to Ben Hearsum [:bhearsum] from comment #12)
> Comment on attachment 8558719 [details] [diff] [review]
> SingleSourceFactory-mozharness.diff
> 
> Review of attachment 8558719 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: configs/multi_locale/mozilla-aurora_android.json
> @@ -25,5 @@
> >      "hg_l10n_tag": "default",
> >      "l10n_dir": "mozilla-aurora",
> >      "merge_locales": true,
> > -    "mozilla_dir": "build",
> > -    "mozconfig": "build/configs/mozilla2/mobile/android/mozilla-aurora/nightly/mozconfig"
> 
> Why is mozconfig no longer needed?

I have no idea. Probably we forgot to delete them after switching to in-tree mozconfigs. I tested the multilocale builds in staging to be sure, and they worked fine.

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 8558698 [details] [diff] [review]
SingleSourceFactory-buildbotcustom-2.diff

https://hg.mozilla.org/build/buildbotcustom/rev/10f70016ef77

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 8559090 [details] [diff] [review]
SingleSourceFactory-tools.diff

https://hg.mozilla.org/build/tools/rev/5ab442bd20a0

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 8558720 [details] [diff] [review]
SingleSourceFactory-buildbot-configs.diff

https://hg.mozilla.org/build/buildbot-configs/rev/673c56fcf766

Comment 17

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/673c56fcf766

Comment 18

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/10f70016ef77

Comment 19

[Rail Aliiev [:rail]]

Failed for android:
cp: cannot stat `mobile/config/mozconfigs/linux64/release': No such file or directory

Comment 20

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-buildbotcustom.diff

Add a way to override source_mozconfig (for thunderbird). I checked, android builds used to use the same mozconfig for this builder as firefox and xulrunner.

Comment 21

[Rail Aliiev [:rail]]

Attachment: SingleSourceFactory-buildbot-configs.diff

Overrides for thunderbird

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 8559792 [details] [diff] [review]
SingleSourceFactory-buildbotcustom.diff

Review of attachment 8559792 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +501,5 @@
>      if not releaseConfig.get('skip_source'):
>          pf = branchConfig['platforms']['linux64']
> +        mozconfig = releaseConfig.get(
> +            'source_mozconfig',
> +            'browser/config/mozconfigs/linux64/release')

Shouldn't this be %(appName)s instead of browser?

Comment 23

[Rail Aliiev [:rail]]

(In reply to Ben Hearsum [:bhearsum] from comment #22)
> Comment on attachment 8559792 [details] [diff] [review]
> SingleSourceFactory-buildbotcustom.diff
> 
> Review of attachment 8559792 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: process/release.py
> @@ +501,5 @@
> >      if not releaseConfig.get('skip_source'):
> >          pf = branchConfig['platforms']['linux64']
> > +        mozconfig = releaseConfig.get(
> > +            'source_mozconfig',
> > +            'browser/config/mozconfigs/linux64/release')
> 
> Shouldn't this be %(appName)s instead of browser?

It won't work for android with different directory layout. Additionally we always used browser/linux mozconfigs for android to generate source tarballs (of course!)

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8558719 [details] [diff] [review]
SingleSourceFactory-mozharness.diff

https://hg.mozilla.org/build/mozharness/rev/f2ace537d36e

Comment 25

[Rail Aliiev [:rail]]

Additionally I filed bug 1129927 to have a mach target for this.

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 8559792 [details] [diff] [review]
SingleSourceFactory-buildbotcustom.diff

Review of attachment 8559792 [details] [diff] [review]:
-----------------------------------------------------------------

Please add a comment to explain this, r=me with that.

Comment 27

[Rail Aliiev [:rail]]

Comment on attachment 8559793 [details] [diff] [review]
SingleSourceFactory-buildbot-configs.diff

https://hg.mozilla.org/build/buildbot-configs/rev/7861f20788f2

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 8559792 [details] [diff] [review]
SingleSourceFactory-buildbotcustom.diff

https://hg.mozilla.org/build/buildbotcustom/rev/34b38c777aae

Comment 29

[Rail Aliiev [:rail]]

Green!

Comment 30

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/7861f20788f2

Comment 31

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/34b38c777aae

Comment 32

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/mozharness/rev/f2ace537d36e

Comment 33

[Rail Aliiev [:rail]]

It turns out that we also add the tooltool files to the tarball. Boooo.

Comment 34

[Rail Aliiev [:rail]]

Attachment: source_no_tooltool-buildbotcustom.diff

http://hg.mozilla.org/mozilla-central/file/6608e0605dfc/build/unix/mozconfig.linux#l12 has an option to avoid using tootool.

Tested in staging for firefox release, fennec release and thunderbird-esr31. The sizes are back to normal.

Comment 35

[Rail Aliiev [:rail]]

Comment on attachment 8569265 [details] [diff] [review]
source_no_tooltool-buildbotcustom.diff

https://hg.mozilla.org/build/buildbotcustom/rev/a8fcb52a5b8f

Comment 36

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/a8fcb52a5b8f

Comment 37

[Rail Aliiev [:rail]]

I regenerated the sources, ran the checksums builder and rsynced manually the files.