1430246 - devedition 59.0b1 fixes

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Comment 1

[Aki Sasaki (not active)]

Attachment: mar_tools_url.diff

https://public-artifacts.taskcluster.net/VVIXmEsxTxu3G4Yov-L2hQ/0/public/logs/live_backing.log

[task 2018-01-12T21:00:57.190Z] 21:00:57     INFO -  'MOZHARNESS_CONFIG': 'single_locale/tc_linux32.py taskcluster_nightly.py',
[task 2018-01-12T21:00:57.190Z] 21:00:57     INFO -  'MOZHARNESS_OPTIONS': 'environment-config=single_locale/production.py branch-config=single_locale/mozilla-beta.py platform-config=single_locale/linux_devedition.py locale=ja:f30c0c2f2bc4 locale=ka:6d3ba6a8c7cb locale=kab:6b3c6e707013 locale=kk:c4125a6f69b4 locale=km:d934d1e2fbba',

So we're using mozilla-beta.py not mozilla-beta_devedition.py .

Comment 2

[Aki Sasaki (not active)]

Attachment: fix devedition source signing

Comment 3

[Aki Sasaki (not active)]

Attachment: fix devedition source signing 2

Comment 4

[Aki Sasaki (not active)]

https://hg.mozilla.org/releases/mozilla-beta/rev/bb73f0842a6cb7677d8886bfe5b9f16caa96b3aa
https://hg.mozilla.org/releases/mozilla-beta/rev/fbde5320505049a521f2e4f5d535eefb56c25ded

Now for 59.0b1 build5, then grafting to inbound.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d758848c0327
add mar_tools_url to m-b and m-r. r=callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/4880c627fc41
fix devedition source signing scope. r=bhearsum

Comment 6

[Aki Sasaki (not active)]

Attachment: import-os.diff

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 8942369 [details] [diff] [review]
import-os.diff

irc r+.
https://hg.mozilla.org/releases/mozilla-beta/rev/9ed6a296ba50cb886ef304f685ecd1822d997ea4

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/335474870968
import os in beta/release l10n mozharness configs. r=jlund DONTBUILD

Comment 9

[Aki Sasaki (not active)]

Attachment: fix-aurora.diff

This should hopefully fix build7.

Comment 10

[Aki Sasaki (not active)]

https://hg.mozilla.org/releases/mozilla-beta/rev/0f677284ab9bbb7663f3d47f464c43bb98ad0ead

Comment 11

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/d758848c0327
https://hg.mozilla.org/mozilla-central/rev/4880c627fc41
https://hg.mozilla.org/mozilla-central/rev/335474870968
https://hg.mozilla.org/mozilla-central/rev/b0fcf7b156b7

Comment 12

[Cristina Coroiu [:ccoroiu]]

Pushed by [e-mail]
https://hg.mozilla.org/integration/mozilla-inbound/rev/b0fcf7b156b7a004ff89e5626ebb5dc55155dabb
fix beta devedition updates (s,mozilla-aurora,mozilla-beta,). r=jlund

Comment 13

[Andreea Pavel [:apavel]]

Hi, the UV jobs from the latest push are orange.
failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=156133920&repo=mozilla-beta&lineNumber=613

:aki can you take a look, please?

Comment 14

[Aki Sasaki (not active)]

Yes, this is known, for Devedition 59.0b1. We're likely going to need a build 8 to release it.

Comment 15

[Jordan Lund (:jlund)]

FTR - notes from aki on how to start a new release using newest in-tree config but with old builds from previous rev: https://trello.com/c/u6MHrz8y/286-solution-to-run-promote-phase-from-a-newer-revision-while-promoting-builds-from-an-older-revision-from-shipit

Comment 16

[Jordan Lund (:jlund)]

all UV is failing to apply partials.

Platform wide. By eyeball, does not seem to be a specific locale.

Getting 4 failures while applying patches. See comment 13 for link to UV. Will look into root cause on Mon

Ben, any thoughts?

example of a fail: https://irccloud.mozilla.com/pastebin/vneY5cKP/ from https://tools.taskcluster.net/groups/T2bpwYAcQii8rzakQqzC7w/tasks/AMxWf9vtTwSPvQ5WbujRtQ/runs/0/artifacts

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: debugging notes, fwiw

Tried to get to the bottom of this and filed bug 1430535. 

I agree it's all locales but only partials, and we get a CRC mismatch trying to PATCH the existing file. AFAICT funsize is using the same files to generate the partial as we are trying to apply it to (installer vs complete mar, but have same contents in 58.0b16 win64 en-US). I think it has to be a funsize issue because we are running the updater executable from the previous build, and we're also seeing this on nightly.

Mitigations - since it's "only" partials failing we could remove them from the release blob in order to ship. On win64 it's 12MB vs 40MB for the file sizes.

Comment 18

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #17)
> Created attachment 8942606 [details]
> debugging notes, fwiw
> 
> Tried to get to the bottom of this and filed bug 1430535. 
> 
> I agree it's all locales but only partials, and we get a CRC mismatch trying
> to PATCH the existing file. AFAICT funsize is using the same files to
> generate the partial as we are trying to apply it to (installer vs complete
> mar, but have same contents in 58.0b16 win64 en-US). I think it has to be a
> funsize issue because we are running the updater executable from the
> previous build, and we're also seeing this on nightly.
> 
> Mitigations - since it's "only" partials failing we could remove them from
> the release blob in order to ship. On win64 it's 12MB vs 40MB for the file
> sizes.

On IRC this morning, everyone seemed to think that bug 1418425 was to blame, and that we need to fix ASAP in order to have 59.0b1 and RC builds ready for Softvision's tomorrow morning. Because of that, we had CloudOps manually overwrite the mar/mbsdiff that funsize uses (bug 1430597) with the last known good ones (https://ftp.mozilla.org/pub/firefox/nightly/2018/01/2018-01-10-10-02-24-mozilla-central/mar-tools/linux64/), and started builds after we verified that was done.

In parallel, bug 1418425 got backed out from central to make sure the next set of nightlies don't override with a bad version again.

Comment 19

[Jordan Lund (:jlund)]

Hm, we are still hitting similar failures on build8.

https://public-artifacts.taskcluster.net/EE8KTAcATr6ajiUyvuEBMQ/0/public/logs/live_backing.log

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

build10 had the funsize bucket purged (bug 1430741) and had a green promote graph. build11 being done for app code changes.

Comment 21

[Mihai Tabara [:mtabara]⌚️GMT]

I see no blocking bugs nor leftovers mentioned in the last few comments. Moreover, by looking in Balrog, I see we're currently serving Devedition-59.0b1-build11 at 100% so I guess we can close this bug. 

Thanks all for the help with this!

Comment 22

[Aki Sasaki (not active)]

\o/

Comment 23

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.