1195365 - Stop publishing complete MARs to balrog as a part of build/l10n repacks

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

ATM we submit complete updates to balrog and let funsize generate partials. Funsize also copies the completes to a separate bucket and publishes their new location to balrog.

We should let funsize publish the initial complete MAR to avoid moments when we have only complete MARs published.

Funsize uses balrog to get information about both from and to MAR files. With this scenario we should use something other than balrog for the "to" MAR information. This could be either completeMarUrl buildbot property or indexed artifacts. Probably using the latter is preferable, even thought the routes are not stable yet.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given that there is no information in the pulse message anymore which builds will get the complete updates served, we can no longer test the update process with Firefox UI Tests for builds older than 4 days. So this is a blocker for our testing matrix.

Comment 2

[Rail Aliiev [:rail]]

Hmmm, there is no real "from" build ID for complete updates. It supposed to fit all builds, except the ones with partials. I don't think you can use the "from" build IDs that we build partials for - they will get partial updates.

In other words you need to have a build ID, which is old enough to get a complete MAR and be downloadable at the same time. Is this assumption correct?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Yes, this is totally correct. Best would be last partial - 1 day. I don't think that we should put more focus on older builds.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

FYI we currently do not test any of the full update mar files which will be used for builds older then 4 days. But we indirectly check them with the fallback update tests, which gets a full mar after failing a former download.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are actually not blocked on testing the complete mar files. We already do that as side-effect of the partial fallback update tests which run updates going back 4 days. In those cases the complete update gets downloaded via the old software update dialog and not the about window, but I don't think that this is that important here, given that the functionality of the about window is tested with the partial update downloads.

Comment 6

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog.

Review commit: https://reviewboard.mozilla.org/r/84576/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/84576/

Comment 7

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Match trailing comma coding guideline

Review commit: https://reviewboard.mozilla.org/r/84592/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/84592/

Comment 8

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8799388 [details]
As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/84576/diff/1-2/

Comment 9

[Rail Aliiev [:rail]]

Looks like you have been working on this bug ;)

Comment 10

[Justin Wood (:Callek)]

Comment on attachment 8799388 [details]
As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog.

https://reviewboard.mozilla.org/r/84574/#review83198

Based on https://treeherder.mozilla.org/#/jobs?repo=try&revision=d09a20ea63102e51a5c39fc779ff61229d9fca46&selectedJob=28895159 this shouldn't be an issue.

Comment 11

[Justin Wood (:Callek)]

Comment on attachment 8799388 [details]
As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog.

https://reviewboard.mozilla.org/r/84576/#review83200

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/be9b0021a857
As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog. r=Callek

Comment 13

[Justin Wood (:Callek)]

Marking leave-open because the pushed code doesn't fully fix this bug. (though bugs are cheap, so we can modify this bug and open a new one if :sfraser deems it appropriate)

Comment 14

[Simon Fraser [:sfraser] ⌚️GMT]

I'm happy to use this one as the central bug and fire off new sub-tasks for other components of the fix

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/be9b0021a857

Comment 16

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8799388 [details]
As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog.

Approval Request Comment
[Feature/regressing bug #]: 1195365
[User impact if declined]: n/a
[Describe test coverage new/current, TreeHerder]: 
[Risks and why]: We want to improve the publishing of complete and partial MARs, and that requires an extra field for l10n repacks. Risk low as tested on mozilla-central, change is not destructive. 
[String/UUID change made/needed]:

Comment 17

[Justin Wood (:Callek)]

(In reply to Simon Fraser [:sfraser] from comment #16)
> Comment on attachment 8799388 [details]
> As part of the work on bug 1195365 we need funsize to known about the MAR
> URLs for the latest build without that information coming from balrog.
> 
> Approval Request Comment
> [Feature/regressing bug #]: 1195365
> [User impact if declined]: n/a
> [Describe test coverage new/current, TreeHerder]: 
> [Risks and why]: We want to improve the publishing of complete and partial
> MARs, and that requires an extra field for l10n repacks. Risk low as tested
> on mozilla-central, change is not destructive. 
> [String/UUID change made/needed]:

I'll make the call that this is releng-infra-only of a change and doesn't need a relman approval. Sheriffs can you help Simon land this on aurora please.

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/8fa339af80ea

Comment 19

[Gerry Chang [:gchang]]

Comment on attachment 8799388 [details]
As part of the work on bug 1195365 we need funsize to known about the MAR URLs for the latest build without that information coming from balrog.

Per comment #17, take it in 51 aurora.

Comment 20

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bump version number for bug 1195365, fix previous version pinning

Review commit: https://reviewboard.mozilla.org/r/87178/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/87178/

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8802912 [details]
Bump version number for bug 1195365, fix previous version pinning

https://reviewboard.mozilla.org/r/87178/#review86236

Comment 22

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8802912 [details]
Bump version number for bug 1195365, fix previous version pinning

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/87178/diff/1-2/

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 8802912 [details]
Bump version number for bug 1195365, fix previous version pinning

https://reviewboard.mozilla.org/r/87178/#review86294

Comment 24

[Simon Fraser [:sfraser] ⌚️GMT]

We'll need to confirm where we should turn off the uploads to balrog, and publishing of releases. 

Funsize is only interested in Windows/Mac/Linux builders, and only does nightlies. Since it's not getting the latest release information from Balrog any more, we can turn off the balrog updates earlier in the workflow.  

desktop_l10n.py is an obvious candidate, and removing https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py#182

mobile_l10n.py is not, since funsize doesn't look at those builders.  

en-US is presumably https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/building/buildbase.py#2050 and making sure that self.submit_balrog_updates() isn't called. 

Before I submit a patch, can we confirm that those are:
a) The right two files (desktop_l10n.py and buildbase.py)
b) The only two files we'll need to change

Comment 25

[Justin Wood (:Callek)]

(In reply to Simon Fraser [:sfraser] from comment #24)
> We'll need to confirm where we should turn off the uploads to balrog, and
> publishing of releases. 
> 
> Funsize is only interested in Windows/Mac/Linux builders, and only does
> nightlies. Since it's not getting the latest release information from Balrog
> any more, we can turn off the balrog updates earlier in the workflow.  
> 
> desktop_l10n.py is an obvious candidate, and removing
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py#182

That should be one way, but better may be to remove balrog_servers from the config... https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/balrog/production.py#2

Note that funsize is not used for Thunderbird of course, so we want to make sure we don't break that.

> mobile_l10n.py is not, since funsize doesn't look at those builders.  
> 
> en-US is presumably
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/building/buildbase.py#2050 and making sure that
> self.submit_balrog_updates() isn't called. 

As you note, we don't want to get rid of this whole call, if nothing else because we need https://dxr.mozilla.org/mozilla-central/rev/944cb0fd05526894fcd90fbe7d1e625ee53cd73d/testing/mozharness/mozharness/mozilla/building/buildbase.py#2050,2063-2069 for the taskcluster nightly stuff..

> Before I submit a patch, can we confirm that those are:
> a) The right two files (desktop_l10n.py and buildbase.py)

They are indeed right for at least one approach :-)

> b) The only two files we'll need to change

Depending on the exact goal/approach that sounds accurate. If you go with the "config files will modify behavior" instead you can use some of the stuff in https://dxr.mozilla.org/mozilla-central/search?q=path%3Amozharness+path%3Asingle_locale&redirect=true :-)

Comment 26

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: bug 1195365 more rework to add site directories instead of plain path modifications r?

Review commit: https://reviewboard.mozilla.org/r/90088/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90088/

Comment 27

[Rail Aliiev [:rail]]

Comment on attachment 8806756 [details]
bug 1195365 more rework to add site directories instead of plain path modifications r?

https://reviewboard.mozilla.org/r/90088/#review89708

Comment 28

[Jordan Lund (:jlund)]

ping - is this bug still a release affecting issue?

Comment 29

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #28)
> ping - is this bug still a release affecting issue?

unping - sorry, there was some confusion, this was linked to a failed step in balrog submission. I believe that part of this has been resolved

Comment 31

[Simon Fraser [:sfraser] ⌚️GMT]

Fixed as a result of Bug 1342392

Comment 32

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.