1173459 - Stop generating partial MAR files and publishing complete MARs to balrog as a part of nightly automation

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

because we can use Funsize!

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I got here via bug 874885, and I have no idea what funsize is. I don't see any link to any kind of documentation either. Rail, can you please shed some more information on it? What does it mean for updates of nightly builds, and details as included in pulse notifications?

Comment 2

[Rail Aliiev [:rail]]

We have a wiki page! :) https://wiki.mozilla.org/ReleaseEngineering/Funsize

Comment 3

[Rail Aliiev [:rail]]

Attachment: MOZ_AUTOMATION_UPDATE_PACKAGING.diff

It looks like we can get rid of MOZ_AUTOMATION_UPDATE_PACKAGING which controls partial updates only. We are not ready to turn this of, but it'd be better to be ready in advance.

Comment 4

[Michael Shal [:mshal]]

What am I looking at here? It's just a list of files but no diff.

Also just to clear up any confusion, the MOZ_AUTOMATION_UPDATE_PACKAGING symbol also controls whether or not the automation/build target runs 'make update-packaging'. IOW, it drives both completes and partials. To turn off just partial generation, we'll just want to delete the $(if) statement from this line:

https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/Makefile.in#34

and the automation-partial-patch target:

https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/Makefile.in#81

(and I guess the partial-patch target if it's now unused?)

Comment 5

[Rail Aliiev [:rail]]

haha, wrong command :) let me attach a new one

Comment 6

[Rail Aliiev [:rail]]

Attachment: disable-partials.diff

This one is better. :)

This is going to be landed whenever we decide to switch to funsize.

Comment 7

[Michael Shal [:mshal]]

Comment on attachment 8634232 [details] [diff] [review]
disable-partials.diff

>diff --git a/build/macosx/mozconfig.common b/build/macosx/mozconfig.common
>--- a/build/macosx/mozconfig.common
>+++ b/build/macosx/mozconfig.common
>@@ -2,17 +2,16 @@
> # License, v. 2.0. If a copy of the MPL was not distributed with this
> # file, You can obtain one at http://mozilla.org/MPL/2.0/.
> 
> MOZ_AUTOMATION_L10N_CHECK=0
> 
> if [ "x$IS_NIGHTLY" = "xyes" ]; then
>   # Some nightlies (eg: Mulet) don't want these set.
>   MOZ_AUTOMATION_UPLOAD_SYMBOLS=${MOZ_AUTOMATION_UPLOAD_SYMBOLS-1}
>-  MOZ_AUTOMATION_UPDATE_PACKAGING=${MOZ_AUTOMATION_UPDATE_PACKAGING-1}

I think you left this in accidentally? I don't see why we'd want to remove this for mac and not linux/windows. (I think we still want it there for all 3 to get the completes built)

And does funsize use the partial-patch target or can we get rid of that too?

Comment 8

[Rail Aliiev [:rail]]

d'oh, that was a leftover from the previous patch.

Funsize doesn't use any make targets, but we probably should keep that target around - I'm not sure it it's still used by release automation. Also it's a good example on how to create partials.

Thanks for the f+, I'll prep a patch whenever we are ready to switch.

Comment 9

[Michael Shal [:mshal]]

Sounds good. The rest of it looks awesome! I love patches that just delete ugly code.

Comment 10

[Rail Aliiev [:rail]]

Attachment: disable_partials.diff

Comment 11

[Michael Shal [:mshal]]

Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

\o/

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f106756af2a4

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/f106756af2a4

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/f106756af2a4

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

I'd like to uplift this change and stop generating partial updates as a part of the build. Instead they will be generated by Funsize. Even now, we overwrite partials in balrog with data submitted from funsize. This will be a noop from users point of view.

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

nm for now, looks like l10n is special and may need more love...

Comment 17

[Rail Aliiev [:rail]]

the balrog step fails even though `make upload` uploads the files, but probably doesn't set some variables properly, see http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/08/2015-08-15-03-02-08-mozilla-central-l10n/mozilla-central-linux-l10n-nightly-1-unknown-bm77-build1-build13.txt.gz


06:01:24     INFO - #####
06:01:24     INFO - ##### Running submit-to-balrog step.
06:01:24     INFO - #####
06:01:24     INFO - Running main action method: submit_to_balrog
06:01:24     INFO - Reading buildbot build properties...
06:01:24     INFO - Using buildbot properties:
06:01:24     INFO - {
06:01:24     INFO -     "properties": {
06:01:24     INFO -         "buildnumber": 13, 
06:01:24     INFO -         "product": "firefox", 
06:01:24     INFO -         "script_repo_revision": "production", 
06:01:24     INFO -         "builddir": "mozilla-central-linux-l10n-nightly-1", 
06:01:24     INFO -         "repository": "", 
06:01:24     INFO -         "buildername": "Firefox mozilla-central linux l10n nightly-1", 
06:01:24     INFO -         "buildid": "20150815030208", 
06:01:24     INFO -         "stage_platform": "linux", 
06:01:24     INFO -         "basedir": "/builds/slave/m-cen-lx-l10n-ntly-1-000000000", 
06:01:24     INFO -         "project": "", 
06:01:24     INFO -         "platform": "linux", 
06:01:24     INFO -         "master": "http://buildbot-master77.bb.releng.use1.mozilla.com:8001/", 
06:01:24     INFO -         "slavebuilddir": "m-cen-lx-l10n-ntly-1-000000000", 
06:01:24     INFO -         "scheduler": "Linux mozilla-central nightly l10n", 
06:01:24     INFO -         "branch": "mozilla-central", 
06:01:24     INFO -         "repo_path": "mozilla-central", 
06:01:24     INFO -         "slavename": "bld-linux64-spot-041", 
06:01:24     INFO -         "builduid": "1836627e3bec4917a8ba966ee0c9c4cd", 
06:01:24     INFO -         "revision": "2ddfc9180971"
06:01:24     INFO -     }, 
06:01:24     INFO -     "sourcestamp": {
06:01:24     INFO -         "repository": "", 
06:01:24     INFO -         "hasPatch": false, 
06:01:24     INFO -         "project": "", 
06:01:24     INFO -         "branch": "mozilla-central", 
06:01:24     INFO -         "changes": [], 
06:01:24     INFO -         "revision": "2ddfc9180971"
06:01:24     INFO -     }
06:01:24     INFO - }
06:01:24     INFO - Setting buildbot property branch to mozilla-central
06:01:24     INFO - Setting buildbot property appName to Firefox
06:01:24     INFO - Setting buildbot property hashType to sha512
06:01:24     INFO - Setting buildbot property platform to linux
06:01:24     INFO - Setting buildbot property buildid to 20150815030208
06:01:24     INFO - Setting buildbot property appVersion to 43.0a1
06:01:24    FATAL - Couldn't find complete mar url in config or package_urls
06:01:24    FATAL - Running post_fatal callback...
06:01:24    FATAL - Exiting -1
program finished with exit code 255

Comment 18

[Rail Aliiev [:rail]]

looks like it's https://bugzilla.mozilla.org/show_bug.cgi?id=1194950#c1

Comment 19

[Rail Aliiev [:rail]]

False alarm, not related to the original patch. Let's try to uplift again.

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

I'd like to uplift this change and stop generating partial updates as a part of the build. Instead they will be generated by Funsize. Even now, we overwrite partials in balrog with data submitted from funsize. This will be a noop from users point of view.

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

https://hg.mozilla.org/releases/mozilla-aurora/rev/60302e9a0a6d

Comment 22

[Rail Aliiev [:rail]]

All done here. No more partials as a part of builds!