wrong config file used for release config bump when doing betas off of mozilla-release

RESOLVED FIXED

Status

Release Engineering
Release Automation
P1
normal
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: sylvestre, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

3 years ago
Fails to build with:
Traceback (most recent call last):
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    ConfigLoader(configFileName=configFileName)
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile, check_synchronously_only=True)
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/master.py", line 652, in loadConfig
    exec f in localDict
  File "/builds/buildbot/build1/master/master.cfg", line 140, in <module>
    secrets=getattr(passwords, 'secrets', None)
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/process/release.py", line 1267, in generateReleaseBranchObjects
    mar_channel_ids=updateConfig.get("marChannelIds", []),
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/process/factory.py", line 3935, in __init__
    self.partialUpdates.keys())
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/common.py", line 181, in getPreviousVersion
    return str(max(StrictVersion(v) for v in partialVersions if v != version))
ValueError: max() arg is an empty sequence 


The second argument of the getPreviousVersion is empty ([]) but the second is correct (38.0b7)

See that this line:
https://mxr.mozilla.org/build/source/buildbotcustom/process/release.py#1243
is getting empty partials.

It might be related to the fact that we are starting build from m-r for a beta build ?!
I am trying to debug this issue on buildbot-master94; and the value of updateConfig before the exception (
https://mxr.mozilla.org/build/source/buildbotcustom/process/release.py#1229) is:

{'verifyConfigs': {'macosx64': 'mozRelease-firefox-mac64.cfg', 'win32': 'mozRelease-firefox-win32.cfg', 'linux64': 'mozRelease-firefox-linux64.cfg', 'linux': 'mozRelease-firefox-linux.cfg'}, 'patcherConfig': 'mozRelease-branch-patcher2.cfg', 'versionRegex': '^\\d+\\.\\d+(\\.\\d+)?$', 'ruleId': 145, 'testChannels': {'release-cdntest': {'ruleId': 57}, 'release-localtest': {'ruleId': 56}}, 'cdnTestChannel': 'release-cdntest', 'localTestChannel': 'release-localtest', 'partialUpdates': {}}

partialUpdates is an empty dictionary
Bumping priority, any reconfig fails
Priority: -- → P1
08:47 <~bhearsum> oh, this is fun
08:48 <~bhearsum> we use the repo name to find the release config: https://github.com/mozilla/build-tools/blob/master/lib/python/release/info.py#L58
08:48 <~bhearsum> so it's trying to build a beta with release-firefox-mozilla-release.py
08:48 -!- kmoir-afk is now known as kmoir
08:48 <~bhearsum> it's a good thing this errored, otherwise all sorts of stuff would've gotten messed up
08:49 -!- jmaher|afk is now known as jmaher
08:50 <@mgerva|buildduty> Sylvestre: ^
08:50 <~bhearsum> probably the best thing to do is to add a hack to that function to return the right thing
08:50 <~bhearsum> similar to https://github.com/mozilla/build-tools/commit/c991947e85cbb4d78bc6232245bddd969b4e46ee
Created attachment 8597250 [details] [diff] [review]
try to hardcode release config name for betas off of release

So, the bumps that were done before were done against the mozilla-release config. I backed them out and this _should_ get release runner to do them against the correct file. Once this is landed and I have updated the release runner code we should be able to try again.
Attachment #8597250 - Flags: review?(mgervasini)
Attachment #8597250 - Flags: review?(mgervasini) → review+
Summary: Firefox & Thunderbird cannot build because of a partial issue → wrong config file used for release config bump when doing betas off of mozilla-release
Attachment #8597250 - Flags: checked-in+
This seems to be working. We should back it out later. If we're going to continue to do crazy shit like this for "spring" releases we should find a better fix.
Duplicate of this bug: 1160234

Updated

3 years ago
Blocks: 1159993
Created attachment 8600123 [details] [diff] [review]
[tools] Handle Fennec too, add tests

version becomes a mandatory argument to getReleaseConfigName, but we always pass it anyway so no big deal.
Attachment #8600123 - Flags: review?(rail)
Attachment #8600123 - Flags: review?(rail) → review+
Comment on attachment 8600123 [details] [diff] [review]
[tools] Handle Fennec too, add tests

https://hg.mozilla.org/build/tools/rev/2fa0b22073d2

Updated bm81:/builds/releaserunner/tools from rev 763d644f85e2 to 2fa0b22073d2, and restarted it with supervisorctl.
Attachment #8600123 - Flags: checked-in+
Verified that we bumped the right release config for 38.0b10, eg 
  http://hg.mozilla.org/build/buildbot-configs/rev/737156c9cef0
And also a lot of |-DMOZ_UPDATE_CHANNEL='beta'| in the logs, so bug 1160234 is fixed now.

Unfortunately the repacks are failing with
18:57:40     INFO - Updating /builds/slave/rel-m-beta-and-api-11_rpk_1-00/build/mozilla-beta revision FENNEC_38_0b10_RELEASE.
18:57:40     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'update', '-C', '-r', 'FENNEC_38_0b10_RELEASE'] in /builds/slave/rel-m-beta-and-api-11_rpk_1-00/build/mozilla-beta
18:57:40     INFO - Copy/paste: hg --config ui.merge=internal:merge update -C -r FENNEC_38_0b10_RELEASE
18:58:11    ERROR -  abort: unknown revision 'FENNEC_38_0b10_RELEASE'!

ie trying to use mozilla-beta, which is hardcoded in the mh config.

Fix the configs, retag, and backout (respectively):
https://hg.mozilla.org/build/mozharness/rev/d9f4f2430fe1
https://hg.mozilla.org/build/mozharness/rev/3211ed508293
https://hg.mozilla.org/build/mozharness/rev/a7e7233ac59b
That fixed the en-US issue, but it turns out we tagged locales in mozilla-beta (a bug!), eg
 https://hg.mozilla.org/releases/l10n/mozilla-beta/de/rev/10a37089af9b

So hg_l10n_base turned out to be wrong:
20:21:25     INFO - Updating /builds/slave/rel-m-beta-and-api-11_rpk_1-00/build/mozilla-release/an revision FENNEC_38_0b10_RELEASE.
20:21:25     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'update', '-C', '-r', 'FENNEC_38_0b10_RELEASE'] in /builds/slave/rel-m-beta-and-api-11_rpk_1-00/build/mozilla-release/an
20:21:25     INFO - Copy/paste: hg --config ui.merge=internal:merge update -C -r FENNEC_38_0b10_RELEASE
20:21:26    ERROR -  abort: unknown revision 'FENNEC_38_0b10_RELEASE'!

So fix better, tag, backout again:
 https://hg.mozilla.org/build/mozharness/rev/5dbeb9668b35
 https://hg.mozilla.org/build/mozharness/rev/86184e217679
 https://hg.mozilla.org/build/mozharness/rev/05a0f4e993f7
(Reporter)

Comment 11

2 years ago
I think this is now fixed
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.