Closed Bug 1430246 Opened 2 years ago Closed 2 years ago

devedition 59.0b1 fixes

Categories

(Release Engineering :: Release Automation: Other, enhancement)

enhancement
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Assigned: aki)

References

Details

Attachments

(5 files, 1 obsolete file)

No description provided.
https://public-artifacts.taskcluster.net/VVIXmEsxTxu3G4Yov-L2hQ/0/public/logs/live_backing.log

[task 2018-01-12T21:00:57.190Z] 21:00:57     INFO -  'MOZHARNESS_CONFIG': 'single_locale/tc_linux32.py taskcluster_nightly.py',
[task 2018-01-12T21:00:57.190Z] 21:00:57     INFO -  'MOZHARNESS_OPTIONS': 'environment-config=single_locale/production.py branch-config=single_locale/mozilla-beta.py platform-config=single_locale/linux_devedition.py locale=ja:f30c0c2f2bc4 locale=ka:6d3ba6a8c7cb locale=kab:6b3c6e707013 locale=kk:c4125a6f69b4 locale=km:d934d1e2fbba',

So we're using mozilla-beta.py not mozilla-beta_devedition.py .
Attachment #8942322 - Flags: review?(bugspam.Callek)
Attachment #8942322 - Flags: review?(bugspam.Callek) → review+
Attached patch fix devedition source signing (obsolete) — Splinter Review
Attachment #8942334 - Flags: review?(bhearsum)
Attachment #8942334 - Attachment is obsolete: true
Attachment #8942334 - Flags: review?(bhearsum)
Attachment #8942343 - Flags: review?(bhearsum)
Attachment #8942343 - Flags: review?(bhearsum) → review+
Depends on: 1430228
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d758848c0327
add mar_tools_url to m-b and m-r. r=callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/4880c627fc41
fix devedition source signing scope. r=bhearsum
Keywords: leave-open
Attached patch import-os.diffSplinter Review
Attachment #8942369 - Flags: review?(jlund)
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/335474870968
import os in beta/release l10n mozharness configs. r=jlund DONTBUILD
Attached patch fix-aurora.diffSplinter Review
This should hopefully fix build7.
Attachment #8942407 - Flags: review?(jlund)
Attachment #8942407 - Flags: review?(jlund) → review+
Pushed by asasaki@mozilla.com
https://hg.mozilla.org/integration/mozilla-inbound/rev/b0fcf7b156b7a004ff89e5626ebb5dc55155dabb
fix beta devedition updates (s,mozilla-aurora,mozilla-beta,). r=jlund
Flags: needinfo?(aki)
Flags: needinfo?(aki)
Hi, the UV jobs from the latest push are orange.
failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=156133920&repo=mozilla-beta&lineNumber=613

:aki can you take a look, please?
Flags: needinfo?(aki)
Yes, this is known, for Devedition 59.0b1. We're likely going to need a build 8 to release it.
Flags: needinfo?(aki)
FTR - notes from aki on how to start a new release using newest in-tree config but with old builds from previous rev: https://trello.com/c/u6MHrz8y/286-solution-to-run-promote-phase-from-a-newer-revision-while-promoting-builds-from-an-older-revision-from-shipit
all UV is failing to apply partials.

Platform wide. By eyeball, does not seem to be a specific locale.

Getting 4 failures while applying patches. See comment 13 for link to UV. Will look into root cause on Mon

Ben, any thoughts?

example of a fail: https://irccloud.mozilla.com/pastebin/vneY5cKP/ from https://tools.taskcluster.net/groups/T2bpwYAcQii8rzakQqzC7w/tasks/AMxWf9vtTwSPvQ5WbujRtQ/runs/0/artifacts
Flags: needinfo?(bhearsum)
Attached file debugging notes, fwiw
Tried to get to the bottom of this and filed bug 1430535. 

I agree it's all locales but only partials, and we get a CRC mismatch trying to PATCH the existing file. AFAICT funsize is using the same files to generate the partial as we are trying to apply it to (installer vs complete mar, but have same contents in 58.0b16 win64 en-US). I think it has to be a funsize issue because we are running the updater executable from the previous build, and we're also seeing this on nightly.

Mitigations - since it's "only" partials failing we could remove them from the release blob in order to ship. On win64 it's 12MB vs 40MB for the file sizes.
(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #17)
> Created attachment 8942606 [details]
> debugging notes, fwiw
> 
> Tried to get to the bottom of this and filed bug 1430535. 
> 
> I agree it's all locales but only partials, and we get a CRC mismatch trying
> to PATCH the existing file. AFAICT funsize is using the same files to
> generate the partial as we are trying to apply it to (installer vs complete
> mar, but have same contents in 58.0b16 win64 en-US). I think it has to be a
> funsize issue because we are running the updater executable from the
> previous build, and we're also seeing this on nightly.
> 
> Mitigations - since it's "only" partials failing we could remove them from
> the release blob in order to ship. On win64 it's 12MB vs 40MB for the file
> sizes.

On IRC this morning, everyone seemed to think that bug 1418425 was to blame, and that we need to fix ASAP in order to have 59.0b1 and RC builds ready for Softvision's tomorrow morning. Because of that, we had CloudOps manually overwrite the mar/mbsdiff that funsize uses (bug 1430597) with the last known good ones (https://ftp.mozilla.org/pub/firefox/nightly/2018/01/2018-01-10-10-02-24-mozilla-central/mar-tools/linux64/), and started builds after we verified that was done.

In parallel, bug 1418425 got backed out from central to make sure the next set of nightlies don't override with a bad version again.
Flags: needinfo?(bhearsum)
Depends on: 1415678
build10 had the funsize bucket purged (bug 1430741) and had a green promote graph. build11 being done for app code changes.
I see no blocking bugs nor leftovers mentioned in the last few comments. Moreover, by looking in Balrog, I see we're currently serving Devedition-59.0b1-build11 at 100% so I guess we can close this bug. 

Thanks all for the help with this!
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
\o/
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Depends on: 1443036
No longer depends on: 1443036
You need to log in before you can comment on or make changes to this bug.