make aus3 and aus4 responses consistent

RESOLVED FIXED

Status

Release Engineering
Release Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: bhearsum, Assigned: bhearsum)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 3 obsolete attachments)

(Assignee)

Description

3 years ago
Created attachment 8463963 [details] [diff] [review]
stop using /nightly/ for firefox/thunderbird

I ran an update comparison test between aus3 and aus4 today. One of the major failures that came up was some results pointing at /pub/mozilla.org/$product/nightly/... instead of /pub/mozilla.org/$product/candidates. The $version-candidates dirs in nightly/ are just symlinks to the candidates/ ones these days, so we should just point at them instead.

Some of the older releases for Thunderbird no longer exist in "candidates", but we don't generate partials for things that old anyways, so this should be okay.

"git diff | dwdiff --diff-input -c -d /" highlights these changes much better (if you have dwdiff". "git diff --word-diff=porcelain" helps a bit too.
Attachment #8463963 - Flags: review?(nthomas)
(Assignee)

Comment 1

3 years ago
I found a few other obvious wasy to improve those - making this bug more generic.
Summary: use "candidates" instead of "nightly" in snippets to make comparsions to balrog easier → make aus3 and aus4 responses consistent
(Assignee)

Comment 2

3 years ago
Created attachment 8464007 [details] [diff] [review]
always print completes before partials

Silly diff, doesn't realize that the order of these doesn't matter! AFAICT, aus3 hardcodes complete before partial (http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/index.php#196), so we should do the same.
Assignee: nobody → bhearsum
Status: NEW → ASSIGNED
Attachment #8464007 - Flags: review?(nthomas)
(Assignee)

Comment 3

3 years ago
Created attachment 8464010 [details] [diff] [review]
use /candidates/ for files and mozilla.org for detailsUrls

www.mozilla.com redirects to www.mozilla.org already, so this should be pretty safe:
➜  tools  curl -I http://www.mozilla.com/en-US/firefox/31.0/releasenotes/
HTTP/1.1 301 Moved Permanently
Server: Apache
X-Backend-Server: pp-web03
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
Date: Tue, 29 Jul 2014 14:28:54 GMT
Location: https://www.mozilla.org/en-US/firefox/31.0/releasenotes/
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Cache-Info: caching
Attachment #8463963 - Attachment is obsolete: true
Attachment #8463963 - Flags: review?(nthomas)
Attachment #8464010 - Flags: review?(nthomas)
Attachment #8464007 - Flags: review?(nthomas) → review+
Comment on attachment 8464010 [details] [diff] [review]
use /candidates/ for files and mozilla.org for detailsUrls

Review of attachment 8464010 [details] [diff] [review]:
-----------------------------------------------------------------

I'm a confused by this. The changes will fix any future snippets, but not help you in the short term except for Firefox beta. Unless you're planning to generate some new ones ?

::: release/patcher-config-bump.pl
@@ +293,4 @@
>            version => $version,
>            oldVersion => $partialVersion
>          );
> +        $partialUpdate->{'path'} = catfile($product, 'candidates', $version .

Wondering why you've left $partialUpdate->{'betatest-url'}, and other uses of 'nightly' which end up in snippets. AFAIK path doesn't. r- for this issue.

::: release/patcher-configs/mozEsr24-branch-patcher2.cfg
@@ +9,4 @@
>                  path   firefox/nightly/24.7.0esr-candidates/build1/update/%platform%/%locale%/firefox-24.7.0esr.complete.mar
>                  url   http://download.mozilla.org/?product=firefox-24.7.0esr-complete&os=%bouncer-platform%&lang=%locale%
>              </complete>
> +            details   https://www.mozilla.org/%locale%/firefox/24.7.0/releasenotes/

Harmless, but you could just wait until the next esr24/esr31/release and let the bumper do this ?
Attachment #8464010 - Flags: review?(nthomas) → review-
(Assignee)

Comment 5

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #4)
> Comment on attachment 8464010 [details] [diff] [review]
> use /candidates/ for files and mozilla.org for detailsUrls
> 
> Review of attachment 8464010 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I'm a confused by this. The changes will fix any future snippets, but not
> help you in the short term except for Firefox beta. Unless you're planning
> to generate some new ones ?

Hm. I was assuming that the next Beta will fix up all of the snippets. I guess that's not true for the watershed updates to 29.0b8, though. How do you feel about me generating new updates for that? It feels a touch risky, but it should be pretty easy to verify that I'm only changing detailsUrl (for i in `rsync -n -av | grep blah`; do diff ...; done).

> ::: release/patcher-config-bump.pl
> @@ +293,4 @@
> >            version => $version,
> >            oldVersion => $partialVersion
> >          );
> > +        $partialUpdate->{'path'} = catfile($product, 'candidates', $version .
> 
> Wondering why you've left $partialUpdate->{'betatest-url'}, and other uses
> of 'nightly' which end up in snippets. AFAIK path doesn't. r- for this issue.

Whoops, that's just an oversight...sorry!

> ::: release/patcher-configs/mozEsr24-branch-patcher2.cfg
> @@ +9,4 @@
> >                  path   firefox/nightly/24.7.0esr-candidates/build1/update/%platform%/%locale%/firefox-24.7.0esr.complete.mar
> >                  url   http://download.mozilla.org/?product=firefox-24.7.0esr-complete&os=%bouncer-platform%&lang=%locale%
> >              </complete>
> > +            details   https://www.mozilla.org/%locale%/firefox/24.7.0/releasenotes/
> 
> Harmless, but you could just wait until the next esr24/esr31/release and let
> the bumper do this ?

Sure, can do.

Comment 6

3 years ago
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/d891d8d65468ad3d8bbf8e8d3fc7f727206ca5b0
bug 1045583: make aus3 and aus4 responses consistent - always print completes before partials (like aus3 does). r=nthomas
(Assignee)

Comment 7

3 years ago
Comment on attachment 8464007 [details] [diff] [review]
always print completes before partials

I pushed this, but I'm going to hold off asking for a push until we either have more Balrog patches queued up, or the other patch in this bug is landed.
Attachment #8464007 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Depends on: 1049524
(Assignee)

Comment 8

3 years ago
Created attachment 8471777 [details] [diff] [review]
change detailsUrl; make balrog use "nightly" instead of "candidates"

I had to dig up old IRC logs, but it looks we agreed to switch Balrog to "nightly" for now rather than changing existing snippets. This patch should do most of that, though we may need to go back and change the blobs for the watershed updates by hand.

I also fixed a missing import (it's in a code path we don't hit, which is why it hasn't caused a problem).
Attachment #8464010 - Attachment is obsolete: true
Attachment #8471777 - Flags: review?(nthomas)
Comment on attachment 8471777 [details] [diff] [review]
change detailsUrl; make balrog use "nightly" instead of "candidates"

>diff --git a/lib/perl/Release/Patcher/Config.pm b/lib/perl/Release/Patcher/Config.pm
>--- a/lib/perl/Release/Patcher/Config.pm
>+++ b/lib/perl/Release/Patcher/Config.pm
>@@ -31,7 +31,7 @@ sub GetProductDetails {
>                $appVersion . $endOfUrl;
>     }
>     else {
>-        return 'https://www.mozilla.com/%locale%/' . $product . '/' .
>+        return 'https://www.mozilla.org/%locale%/' . $product . '/' .

Was expecting this change in python code (tools/lib/python/release/info.py's getProductDetails()), or missed something about what you're trying to do here. The other chunk looks fine. Leaving r? until you respond.

If you want to check the beta channel, I looked at https://aus3.mozilla.org/update/3/Firefox/32.0/20140807212602/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/beta/Darwin%2013.1.0/default/default/update.xml vs aus4 and noticed the capitalization is different in the bouncer products.
(Assignee)

Comment 10

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #9)
> Comment on attachment 8471777 [details] [diff] [review]
> change detailsUrl; make balrog use "nightly" instead of "candidates"
> 
> >diff --git a/lib/perl/Release/Patcher/Config.pm b/lib/perl/Release/Patcher/Config.pm
> >--- a/lib/perl/Release/Patcher/Config.pm
> >+++ b/lib/perl/Release/Patcher/Config.pm
> >@@ -31,7 +31,7 @@ sub GetProductDetails {
> >                $appVersion . $endOfUrl;
> >     }
> >     else {
> >-        return 'https://www.mozilla.com/%locale%/' . $product . '/' .
> >+        return 'https://www.mozilla.org/%locale%/' . $product . '/' .
> 
> Was expecting this change in python code (tools/lib/python/release/info.py's
> getProductDetails()), or missed something about what you're trying to do
> here. The other chunk looks fine. Leaving r? until you respond.

Ah. You didn't comment on this part specifically, so I assumed it was okay to make this change to aus3 bits. I think we're supposed to be doing it anyways...at least that's what bug 679077 thinks. Still, I could s/.org/.com/ for now instead.

> If you want to check the beta channel, I looked at
> https://aus3.mozilla.org/update/3/Firefox/32.0/20140807212602/Darwin_x86_64-
> gcc3-u-i386-x86_64/en-US/beta/Darwin%2013.1.0/default/default/update.xml vs
> aus4 and noticed the capitalization is different in the bouncer products.

Good catch - do you want me to switch the Balrog products, or the aus3 ones?
Comment on attachment 8471777 [details] [diff] [review]
change detailsUrl; make balrog use "nightly" instead of "candidates"

We spoke on vidyo - the plan is to modify Balrog blobs and submission code to mimic what we have in snippets (details url on mozilla.com, nightly instead of candidates in test urls for test channels, lower case products in urls for beta/release channels). Once we've fully transitioned to aus4.m.o we can fix the first two to point to directly to the actual locations.
Attachment #8471777 - Attachment is obsolete: true
Attachment #8471777 - Flags: review?(nthomas)
(Assignee)

Comment 12

3 years ago
Created attachment 8473693 [details] [diff] [review]
adjust candidates dir, bouncer products, details url

I had another look through my diff and couldn't find anything else that wasn't a watershed moment. There might be something else buried in there - it'll be easier to spot with this landed.
Attachment #8473693 - Flags: review?(nthomas)
Attachment #8473693 - Flags: review?(nthomas) → review+
(Assignee)

Updated

3 years ago
Attachment #8473693 - Flags: checked-in+
(Assignee)

Comment 13

3 years ago
I landed this and moved the tags so that it would be used in Firefox 32.0b8 today. The submission appears to have worked fine, based on looking at https://aus4-admin.mozilla.org/releases/Firefox-32.0b8-build1/data. I'm going to do some more comparisons to double check if there's more changes we should make.
(Assignee)

Comment 14

3 years ago
I reran the snippet comparison tests. I had to work around hashFunction differences -- according to bug 853036 we use different casing for nightlies and releases on aus3. I post processed the snippets to fix that (thanks for the suggestion Nick!) because I don't think it's worthwhile making balrog inconsistent for nightlies+releases.

Things look much better now. AFAICT, Firefox snippets are the same, with the exception of 28.0 and earlier betas (because of missing watershed points).

Thunderbird looks mostly fine too, except the detailsURL is vastly different:
FAIL: Unmatched snippets on /update/3/Thunderbird/31.0/20140613002318/Linux_x86_64-gcc3/ru/betatest/GTK%202.18./default/default/update.xml?force=1:
--- https://aus3.mozilla.org//update/3/Thunderbird/31.0/20140613002318/Linux_x86_64-gcc3/ru/betatest/GTK%202.18./default/default/update.xml?force=1
+++ https://aus4.mozilla.org//update/3/Thunderbird/31.0/20140613002318/Linux_x86_64-gcc3/ru/betatest/GTK%202.18./default/default/update.xml?force=1
@@ -1,6 +1,6 @@
 <?xml version="1.0"?>
 <updates>
-    <update type="minor" displayVersion="32.0 Beta 1" appVersion="32.0" platformVersion="32.0" buildID="20140813074326" detailsURL="http://live.mozillamessaging.com/thunderbird/releasenotes?locale=ru&amp;platform=linux-x86_64&amp;version=32.0">
-        <patch type="complete" URL="http://stage.mozilla.org/pub/mozilla.org/thunderbird/nightly/32.0b1-candidates/build1/update/linux-x86_64/ru/thunderbird-32.0b1.complete.mar" hashFunction="sha512" hashValue="31f4445773c309ad424866901828b47367ab3feb23137e7d33365f960a6a916eb93db03ff1fa45d30d9880cb751a7cbf3f7dba0a90a794f200612124ac1a3dc3" size="37138464"/>
+    <update type="minor" displayVersion="32.0 Beta 1" appVersion="32.0" platformVersion="32.0" buildID="20140813074326" detailsURL="https://www.mozilla.org/ru/thunderbird/32.0/releasenotes/">
+        <patch type="complete" URL="http://stage.mozilla.org/pub/mozilla.org/thunderbird/candidates/32.0b1-candidates/build1/update/linux-x86_64/ru/thunderbird-32.0b1.complete.mar" hashFunction="sha512" hashValue="31f4445773c309ad424866901828b47367ab3feb23137e7d33365f960a6a916eb93db03ff1fa45d30d9880cb751a7cbf3f7dba0a90a794f200612124ac1a3dc3" size="37138464"/>

(The candidates dir difference can be ignored, because we need to wait for another Thunderbird beta to test that again.)

I'm going to look into this to see what the right thing to do is.
(Assignee)

Comment 15

3 years ago
#tbdrivers tells me that mozillamessaging.com is deprecated, so this is the same situation as mozilla.com. Unfortunately, the URL that aus3 uses contains the platform. Balrog doesn't support platform substitution, and it's probably not worthwhile adding just for this.

I went over the log another time, and there's three differences:
* Lack of watershed updates for Firefox - fixed by bug 1049550
* Thunderbird detailsURLs - not fixing per above
* Thunderbird candidates dir - fixed in 33.0b1

Given all of that, I'm calling this fixed, and we'll cope with the Thunderbird detailsURLs being different.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.