Stop generating partial MAR files and publishing complete MARs to balrog as a part of nightly automation

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: rail, Assigned: rail)

Tracking

(Blocks: 1 bug)

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox42 fixed, firefox43 fixed)

Details

Attachments

(1 attachment, 2 obsolete attachments)

(Assignee)

Description

2 years ago
because we can use Funsize!
(Assignee)

Updated

2 years ago
Depends on: 1176550
I got here via bug 874885, and I have no idea what funsize is. I don't see any link to any kind of documentation either. Rail, can you please shed some more information on it? What does it mean for updates of nightly builds, and details as included in pulse notifications?
Flags: needinfo?(rail)
(Assignee)

Comment 2

2 years ago
We have a wiki page! :) https://wiki.mozilla.org/ReleaseEngineering/Funsize
Flags: needinfo?(rail)
Blocks: 725726
(Assignee)

Comment 3

2 years ago
Created attachment 8634217 [details] [diff] [review]
MOZ_AUTOMATION_UPDATE_PACKAGING.diff

It looks like we can get rid of MOZ_AUTOMATION_UPDATE_PACKAGING which controls partial updates only. We are not ready to turn this of, but it'd be better to be ready in advance.
Attachment #8634217 - Flags: feedback?(mshal)
What am I looking at here? It's just a list of files but no diff.

Also just to clear up any confusion, the MOZ_AUTOMATION_UPDATE_PACKAGING symbol also controls whether or not the automation/build target runs 'make update-packaging'. IOW, it drives both completes and partials. To turn off just partial generation, we'll just want to delete the $(if) statement from this line:

https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/Makefile.in#34

and the automation-partial-patch target:

https://dxr.mozilla.org/mozilla-central/source/tools/update-packaging/Makefile.in#81

(and I guess the partial-patch target if it's now unused?)
Flags: needinfo?(rail)
(Assignee)

Comment 5

2 years ago
haha, wrong command :) let me attach a new one
Flags: needinfo?(rail)
(Assignee)

Comment 6

2 years ago
Created attachment 8634232 [details] [diff] [review]
disable-partials.diff

This one is better. :)

This is going to be landed whenever we decide to switch to funsize.
Attachment #8634217 - Attachment is obsolete: true
Attachment #8634217 - Flags: feedback?(mshal)
Attachment #8634232 - Flags: review?(mshal)
Comment on attachment 8634232 [details] [diff] [review]
disable-partials.diff

>diff --git a/build/macosx/mozconfig.common b/build/macosx/mozconfig.common
>--- a/build/macosx/mozconfig.common
>+++ b/build/macosx/mozconfig.common
>@@ -2,17 +2,16 @@
> # License, v. 2.0. If a copy of the MPL was not distributed with this
> # file, You can obtain one at http://mozilla.org/MPL/2.0/.
> 
> MOZ_AUTOMATION_L10N_CHECK=0
> 
> if [ "x$IS_NIGHTLY" = "xyes" ]; then
>   # Some nightlies (eg: Mulet) don't want these set.
>   MOZ_AUTOMATION_UPLOAD_SYMBOLS=${MOZ_AUTOMATION_UPLOAD_SYMBOLS-1}
>-  MOZ_AUTOMATION_UPDATE_PACKAGING=${MOZ_AUTOMATION_UPDATE_PACKAGING-1}

I think you left this in accidentally? I don't see why we'd want to remove this for mac and not linux/windows. (I think we still want it there for all 3 to get the completes built)

And does funsize use the partial-patch target or can we get rid of that too?
Attachment #8634232 - Flags: review?(mshal) → feedback+
(Assignee)

Comment 8

2 years ago
d'oh, that was a leftover from the previous patch.

Funsize doesn't use any make targets, but we probably should keep that target around - I'm not sure it it's still used by release automation. Also it's a good example on how to create partials.

Thanks for the f+, I'll prep a patch whenever we are ready to switch.
Sounds good. The rest of it looks awesome! I love patches that just delete ugly code.
(Assignee)

Comment 10

2 years ago
Created attachment 8646908 [details] [diff] [review]
disable_partials.diff
Attachment #8634232 - Attachment is obsolete: true
Attachment #8646908 - Flags: review?(mshal)
Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

\o/
Attachment #8646908 - Flags: review?(mshal) → review+

Comment 12

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/f106756af2a4
(Assignee)

Comment 13

2 years ago
Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/f106756af2a4
Attachment #8646908 - Flags: checked-in+
https://hg.mozilla.org/mozilla-central/rev/f106756af2a4
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox43: --- → fixed
Resolution: --- → FIXED
(Assignee)

Comment 15

2 years ago
Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

I'd like to uplift this change and stop generating partial updates as a part of the build. Instead they will be generated by Funsize. Even now, we overwrite partials in balrog with data submitted from funsize. This will be a noop from users point of view.
Attachment #8646908 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 16

2 years ago
Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

nm for now, looks like l10n is special and may need more love...
Attachment #8646908 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 17

2 years ago
the balrog step fails even though `make upload` uploads the files, but probably doesn't set some variables properly, see http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/08/2015-08-15-03-02-08-mozilla-central-l10n/mozilla-central-linux-l10n-nightly-1-unknown-bm77-build1-build13.txt.gz


06:01:24     INFO - #####
06:01:24     INFO - ##### Running submit-to-balrog step.
06:01:24     INFO - #####
06:01:24     INFO - Running main action method: submit_to_balrog
06:01:24     INFO - Reading buildbot build properties...
06:01:24     INFO - Using buildbot properties:
06:01:24     INFO - {
06:01:24     INFO -     "properties": {
06:01:24     INFO -         "buildnumber": 13, 
06:01:24     INFO -         "product": "firefox", 
06:01:24     INFO -         "script_repo_revision": "production", 
06:01:24     INFO -         "builddir": "mozilla-central-linux-l10n-nightly-1", 
06:01:24     INFO -         "repository": "", 
06:01:24     INFO -         "buildername": "Firefox mozilla-central linux l10n nightly-1", 
06:01:24     INFO -         "buildid": "20150815030208", 
06:01:24     INFO -         "stage_platform": "linux", 
06:01:24     INFO -         "basedir": "/builds/slave/m-cen-lx-l10n-ntly-1-000000000", 
06:01:24     INFO -         "project": "", 
06:01:24     INFO -         "platform": "linux", 
06:01:24     INFO -         "master": "http://buildbot-master77.bb.releng.use1.mozilla.com:8001/", 
06:01:24     INFO -         "slavebuilddir": "m-cen-lx-l10n-ntly-1-000000000", 
06:01:24     INFO -         "scheduler": "Linux mozilla-central nightly l10n", 
06:01:24     INFO -         "branch": "mozilla-central", 
06:01:24     INFO -         "repo_path": "mozilla-central", 
06:01:24     INFO -         "slavename": "bld-linux64-spot-041", 
06:01:24     INFO -         "builduid": "1836627e3bec4917a8ba966ee0c9c4cd", 
06:01:24     INFO -         "revision": "2ddfc9180971"
06:01:24     INFO -     }, 
06:01:24     INFO -     "sourcestamp": {
06:01:24     INFO -         "repository": "", 
06:01:24     INFO -         "hasPatch": false, 
06:01:24     INFO -         "project": "", 
06:01:24     INFO -         "branch": "mozilla-central", 
06:01:24     INFO -         "changes": [], 
06:01:24     INFO -         "revision": "2ddfc9180971"
06:01:24     INFO -     }
06:01:24     INFO - }
06:01:24     INFO - Setting buildbot property branch to mozilla-central
06:01:24     INFO - Setting buildbot property appName to Firefox
06:01:24     INFO - Setting buildbot property hashType to sha512
06:01:24     INFO - Setting buildbot property platform to linux
06:01:24     INFO - Setting buildbot property buildid to 20150815030208
06:01:24     INFO - Setting buildbot property appVersion to 43.0a1
06:01:24    FATAL - Couldn't find complete mar url in config or package_urls
06:01:24    FATAL - Running post_fatal callback...
06:01:24    FATAL - Exiting -1
program finished with exit code 255
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 18

2 years ago
looks like it's https://bugzilla.mozilla.org/show_bug.cgi?id=1194950#c1
(Assignee)

Updated

2 years ago
See Also: → bug 1194950
(Assignee)

Comment 19

2 years ago
False alarm, not related to the original patch. Let's try to uplift again.
(Assignee)

Comment 20

2 years ago
Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

I'd like to uplift this change and stop generating partial updates as a part of the build. Instead they will be generated by Funsize. Even now, we overwrite partials in balrog with data submitted from funsize. This will be a noop from users point of view.
Attachment #8646908 - Flags: approval-mozilla-aurora?
Attachment #8646908 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
(Assignee)

Comment 21

2 years ago
Comment on attachment 8646908 [details] [diff] [review]
disable_partials.diff

https://hg.mozilla.org/releases/mozilla-aurora/rev/60302e9a0a6d
(Assignee)

Updated

2 years ago
Blocks: 1196836
(Assignee)

Comment 22

2 years ago
All done here. No more partials as a part of builds!
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
status-firefox42: --- → fixed

Updated

a year ago
Duplicate of this bug: 1162148
You need to log in before you can comment on or make changes to this bug.