1133250 - Missing updates on nightly-esr31 channel for builds on Feb 14th

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Our Mozmill test caught the problem that no updates are available on the nightly-esr31 channel across platforms:

https://aus3.mozilla.org/update/3/Firefox/31.4.0esrpre/20150213000505/WINNT_x86-msvc/en-US/nightly-esr31/Windows_NT%206.1/default/default/update.xml?force=1

Builds do exist:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr31/

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Btw we will stop running tests with mozmill for esr31 nightly builds soon. For more information see bug 1102971.

Comment 2

[Rail Aliiev [:rail]]

It sounds related to bug 1129550. We stopped generating and pushing aus2 (the old update server) snippets, but the load balancer for some reason still serves the updates of off  aus2.mozilla.org:

curl -I 'https://aus3.mozilla.org/update/3/Firefox/31.4.0esrpre/20150213000505/WINNT_x86-msvc/en-US/nightly-esr31/Windows_NT%206.1/default/default/update.xml?force=1'
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: pp-app-dist04
Cache-Control: no-store, must-revalidate, post-check=0, pre-check=0, private
Content-Type: text/xml;
Date: Sun, 15 Feb 2015 21:43:02 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Set-Cookie: aus2a=64.228.94.154.1424036582.5216; expires=Sun, 16-Feb-2020 02:46:52 GMT; path=/; domain=aus2.mozilla.org
X-Powered-By: PHP/5.1.6
X-Cache-Info: caching

Comment 3

[[e-mail] (:bhearsum)]

Wow, I thought these had been moved over for ages. Looks like they're not in the current traffic script. We're planning to redirect ALL requests to aus4 this week, so this will be fixed soon.

Thanks for catching this.

Comment 4

[[e-mail] (:bhearsum)]

Looks like there might be something more going on here, even https://aus4.mozilla.org/update/3/Firefox/31.4.0esrpre/20150213000505/WINNT_x86-msvc/en-US/nightly-esr31/Windows_NT%206.1/default/default/update.xml?force=1 returns no update. I'll dig into more once the redirect is in place.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #4)
> Looks like there might be something more going on here, even
> https://aus4.mozilla.org/update/3/Firefox/31.4.0esrpre/20150213000505/
> WINNT_x86-msvc/en-US/nightly-esr31/Windows_NT%206.1/default/default/update.
> xml?force=1 returns no update. I'll dig into more once the redirect is in
> place.

I'm guessing that this has to do with the "31.5.0esrpre" versions in the esr blobs. I don't think our version class copes with that: https://github.com/mozilla/balrog/blob/master/auslib/util/versions.py

I'd really love to drop these silly tags from esr now that they're unique compared to beta/release...

Comment 6

[[e-mail] (:bhearsum)]

I dug into the history of the "esrpre" style versions and it looks like it boils down to bug 732375, which talks about the versions being the same on release and esr. Now that ESRs have their own unique version style (except for the very first release of a new ESR branch), I think we should go back to being consistent with every other branch and drop the "esrpre" part of the version number. Henrik, you originally filed bug 732375, any objection to this?

We'll still need some sort of change to Balrog to support all of the builds out there that currently have that version, but at least we can relegate it to support for old crud, rather than current crud this way.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: deal with "esrpre" style versions by stripping them away early

Rather than adding another hack into ClientRequestView I was able to add a route that stripped "esrpre" away before the request is dispatched. For reasons I don't understand, this doesn't work for the "x86 " case w/ Avast URLs.

Comment 8

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/7bcba31a27b659e118b27225268571b379811ca9
bug 1133250: Missing updates on nightly-esr31 channel for builds on Feb 14th - strip away esrpre from version versions to make esr nightlies work. r=nthomas

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 8566096 [details] [diff] [review]
deal with "esrpre" style versions by stripping them away early

This patch will likely go into production early next week, but maybe sooner.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The difference for the esr31 builds to those on the Nightly and Aurora channel are that for ESR we also have releases. We do not have them for the latter. So how would it be possible to differentiate a nightly ESR build from a release ESR build?

Comment 11

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) from comment #10)
> The difference for the esr31 builds to those on the Nightly and Aurora
> channel are that for ESR we also have releases. We do not have them for the
> latter. So how would it be possible to differentiate a nightly ESR build
> from a release ESR build?

Only release builds have "esr" set as the channel. CI builds have "default", nightly builds have "nightly-esr". There's also application.ini, which contains an exact changeset. Version number should not be considered a reliable identifier for builds on any channel IMO.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Anyway I think I'm happy with that change, especially if it makes things lesser broken. No need to ship something special only for automation purposes. So lets get this done.

Do you have an ETA when those updates will work again?

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) from comment #12)
> Anyway I think I'm happy with that change, especially if it makes things
> lesser broken. No need to ship something special only for automation
> purposes. So lets get this done.
> 
> Do you have an ETA when those updates will work again?

Waiting on a Balrog push for the existing patch in this bug. However, we're talking about shutting them off in bug 1135024, so they may not work for very long.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks Ben! I also talked with David Burns a couple of minutes ago and we decided to really shutdown esr31 nightly tests, and only keep the tests for releases.

Comment 15

[[e-mail] (:bhearsum)]

Since we *are* turning off ESR nightlies in bug 1135024 the only thing left to do here is to wait for the Balrog change to go live, even though it isn't strictly necessary anymore. There's no pointing in adjusting the esr31 version number, since it makes no difference. I made a note in bug 1135702 to not use "esrpre" versions when we create that branch, though.

Comment 16

[[e-mail] (:bhearsum)]

The balrog patch is in prod.