Closed Bug 1279722 Opened 4 years ago Closed 3 years ago

Thunderbird 45.2b1 uses ACCEPTED_MAR_CHANNEL_IDS=thunderbird-comm-release

Categories

(Release Engineering :: Release Requests, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rail, Unassigned)

Details

Attachments

(1 file)

See 47.0b1 update verification failures in http://ftp.mozilla.org/pub/thunderbird/candidates/47.0b1-candidates/build1/logs/

Found updater at updater.exe
PATCH DIRECTORY ../../update
INSTALLATION DIRECTORY .
WORKING DIRECTORY .
failed: 22
calling QuitProgressUI
FAIL: update status was not successful: failed: 22
FAIL: check_updates returned failure for WINNT_x86-msvc-x86 downloads/Thunderbird Setup 45.2b1.exe vs. downloads/Thunderbird Setup 47.0b1.exe: 1

Exit code 22 corresponds to MAR_CHANNEL_MISMATCH_ERROR, see https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/common/errors.h#42

I verified the error:

$ wget 'http://archive.mozilla.org/pub/mozilla.org/thunderbird/releases/45.2b1/win32/is/Thunderbird Setup 45.2b1.exe'

$ 7z x Thunderbird\ Setup\ 45.2b1.exe core/update-settings.ini

$ cat core/update-settings.ini
; If you modify this file updates may fail.
; Do not modify this file.

[Settings]
ACCEPTED_MAR_CHANNEL_IDS=thunderbird-comm-release


Probably it's related to the fact that we used comm-esr45 for 45 betas, and this setting was set with releases in mind.

This means that we may need to create a custom MAR with thunderbird-comm-release channel, which would update update-settings.ini.
I don't really know what to do to create a custom MAR. Is that something that I can do with guidance, or is it something that releng (probably rail) needs to do?
Flags: needinfo?(rail)
Sorry for the tardy reply. In bug 1255245 we had similar issue in Firefox.
Flags: needinfo?(rail)
I just talked to Nick, and looks like we would need to rebuild 47.0b1 with fixes similar to bug 1255245.
Cause investigation - 45.2b1 was built with
mozilla commit: https://hg.mozilla.org/releases/mozilla-esr45/rev/4ed86d63c09a
comm commit: https://hg.mozilla.org/releases/comm-beta/rev/THUNDERBIRD452b1_2016051723_RELBRANCH

http://hg.mozilla.org/releases/comm-beta/file/THUNDERBIRD_45_2b1_RELEASE/mail/confvars.sh has
ACCEPTED_MAR_CHANNEL_IDS=thunderbird-comm-release
MAR_CHANNEL_ID=thunderbird-comm-release

This should have been
ACCEPTED_MAR_CHANNEL_IDS=thunderbird-comm-beta,thunderbird-comm-release
MAR_CHANNEL_ID=thunderbird-comm-beta

The http://hg.mozilla.org/releases/comm-beta/rev/b1a324ae1b972.

Resolution - I think we'll need to rebuild 47.0b1 with a change to confvars.sh which sets 
 MAR_CHANNEL_ID=thunderbird-comm-release
so that all beta versions accept it. This would only be for this beta. And also attachment 8731048 [details] [diff] [review], so that update-settings.ini is include in the mar file, so that 45.2b1 users are fixed up.
Plus a rule in Balrog that forces 45.2b1 to update to 47.0b1, which will cause some havoc with update testing in future releases (failures because 45.2b1 updates to 47.0b1 instead of the newly built beta). We don't have to deal with this until after 47.0b1 ships though.

BTW, the fixes will be cross-platform, but we're only having failures on Windows. That's the only platform that's verifying channel-id and mar signing at the time the update is applied.
I pushed https://hg.mozilla.org/releases/comm-beta/rev/8d3a0c9fd8df to a branch to be used to build a Thunderbird 47.0b2 with the suggested changes from comment 4.
I pushed http://hg.mozilla.org/releases/mozilla-beta/rev/77c04596623e to a branch to be used to build Thunderbird 47.0b2, as suggested in the second half of comment 4.
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #8)
> 45.2 update_verify for win32 also fail
> https://archive.mozilla.org/pub/thunderbird/candidates/45.2-candidates/
> build1/logs/release-comm-esr45-win32_release_update_verify_1-bm91-build1-
> build4.txt.gz

hmm, but they are exit code 1
not exit code 22
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #9)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #8)
> > 45.2 update_verify for win32 also fail
> > https://archive.mozilla.org/pub/thunderbird/candidates/45.2-candidates/
> > build1/logs/release-comm-esr45-win32_release_update_verify_1-bm91-build1-
> > build4.txt.gz
> 
> hmm, but they are exit code 1
> not exit code 22

This is -- odd -- Scanning log for failures
-------------------------
FAIL: complete from https://aus4.mozilla.org/update/3/Thunderbird/38.8.0/20160503054940/WINNT_x86-msvc/nn-NO/release-localtest/default/default/default/update.xml?force=1 wrong size
FAIL: update.xml size: 42957205
FAIL: actual size: 42957160
FAIL: download_mars returned non-zero exit code: 1
-------------------------
This run has failed, see the above log


I'm going to re-run and try and see if it was a one-off issue, failing that I have no idea why only that one failed...
(In reply to Justin Wood (:Callek) from comment #10)
> I'm going to re-run and try and see if it was a one-off issue, failing that
> I have no idea why only that one failed...

Successful with: https://archive.mozilla.org/pub/thunderbird/candidates/45.2-candidates/build1/logs/release-comm-esr45-win32_release_update_verify_1-bm91-build1-build5.txt.gz
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Looks like we forgot to add an update rule for 45.2b1 to 47.0b1. Do we have any ADI data which shows that there are a significant amount of ADI on 45.2b1 ?
Flags: needinfo?(aleth)
(In reply to Nick Thomas [:nthomas] from comment #12)
> Looks like we forgot to add an update rule for 45.2b1 to 47.0b1. Do we have
> any ADI data which shows that there are a significant amount of ADI on
> 45.2b1 ?

rkent might know about this?
Flags: needinfo?(aleth) → needinfo?(rkent)
For some reason crash stats won't directly report on ADI for 45.2b1  But it will give individual crashes, and extrapolating from those values I would estimate 10,000 - 15,000 ADI for 45.2b1  That would make it around 1/3 of beta users, so yes this is significant (at least for us).

Wayne might be better at this than I am.
Flags: needinfo?(rkent)
(In reply to Kent James (:rkent) from comment #14)
> For some reason crash stats won't directly report on ADI for 45.2b1 

Assuming you are trying to get ADI from daily [1]  ... versions age off the daily crash-stats UI after 2-3 cycles. 

> But it will give individual crashes, and extrapolating from those values I would
> estimate 10,000 - 15,000 ADI for 45.2b1  That would make it around 1/3 of
> beta users, so yes this is significant (at least for us).

I'm not sure 10k-15k for 45.2b1 is good estimate - but hard to say without knowing how that figure was derived. 

My math ... IIRC normal ADI for the most current version is ~25k, and 47.0b1 ADI is ~23k per [1] - a k difference during summertime is insignificant.  [2] can be used as a substitute for the daily query (and also gives much older versions), which shows 45.2b1 doesn't have an extraordinarly high crash volume, and suggests an ADI of about 1k.  (However, 44.0b1 is about 2x higher than I would expect - something I noted a few weeks ago.)

[1] crashes per day - https://crash-stats.mozilla.com/crashes-per-day/?p=Thunderbird&v=45.2&v=51.0a1&v=50.0a2&v=47.0b2&hang_type=any&os=Windows&os=Mac+OS+X&os=Linux&date_start=2016-08-02&date_end=2016-08-16&submit=Generate

[2] hack at crashes by version - https://crash-stats.mozilla.com/search/?product=Thunderbird&release_channel=beta&_sort=-date&_facets=signature&_facets=version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-version 


(In reply to Nick Thomas [:nthomas] from comment #12)
> Looks like we forgot to add an update rule for 45.2b1 to 47.0b1. Do we have
> any ADI data which shows that there are a significant amount of ADI on 45.2b1 ?

If my crash report math above is close, then something else helped 45.2b1 users get to 47.0b1
OK I think I missed a week/day conversion

45 crashes in the last week.

Crash rate approximately 0.003 per day (from 45.2.0 crash rate)

Total ADI: (45/7)/(0.003) is around 2100
(In reply to Nick Thomas [:nthomas] from comment #12)
> Looks like we forgot to add an update rule for 45.2b1 to 47.0b1. 

I've added a rule to force 45.2b1 to update to 47.0b2, since the latter is where the fixes are. We should not generate any partial updates from 45.2b1 to <new release>, as they won't be used.
The important change here is remove <45.b2b1> and the matching past-update line from mozBeta-thunderbird-branch-patcher2.cfg, so that we don't get update failures when 45.2b1 updates to 47.0b2 instead of <latest build>.

Helpfully I've obscured that by removing anything older then 40.0 from the beta and release configs, to improve runtime of the update verify tests.
Attachment #8786549 - Flags: review?(rail)
Comment on attachment 8786549 [details] [diff] [review]
[tools] Trim 45.2b1 out of patcher config, and also cull old versions

Review of attachment 8786549 [details] [diff] [review]:
-----------------------------------------------------------------

wheeeee!
Attachment #8786549 - Flags: review?(rail) → review+
Comment on attachment 8786549 [details] [diff] [review]
[tools] Trim 45.2b1 out of patcher config, and also cull old versions

I dropped the change to mozRelease-branch-patcher2.cfg because 'whoops, how did that get there ?!?' -  https://hg.mozilla.org/build/tools/rev/464671cced63
Attachment #8786549 - Flags: checked-in+
(In reply to Nick Thomas [:nthomas] (PTO 9-11 Oct) from comment #17)
> (In reply to Nick Thomas [:nthomas] from comment #12)
> > Looks like we forgot to add an update rule for 45.2b1 to 47.0b1. 
> 
> I've added a rule to force 45.2b1 to update to 47.0b2, since the latter is
> where the fixes are.

It looks like this rule only got applied to the beta-localtest channel (https://aus5.mozilla.org/api/v1/rules/414) although it is perhaps irrelevant at this point. (I noticed this when trying to figure out why there was an extra rule on beta-localtest compared the other beta* channels).
You need to log in before you can comment on or make changes to this bug.