986988 - (balrog-release) Tracking bug for work required to make Balrog usable for Releases & ESR

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This is the release equivalent of bug 933414 (betas).

Comment 1

[[e-mail] (:bhearsum)]

After adding an OS blocking rule for release*, my quick snippet comparison test is as follows:
Tested 1776 paths.
Pass count: 666
Fail count: 1110
Error count: 0

Going through the logs, the failures I've noticed are:
* Missing a lot of extended attributes:
FAIL: Unmatched snippets on /update/3/Firefox/10.0.1/20120208060813/Darwin_x86_64-gcc3-u-i386-x86_64/de/release/Darwin%2010/default/default/update.xml?force=1:
--- https://aus3.mozilla.org//update/3/Firefox/10.0.1/20120208060813/Darwin_x86_64-gcc3-u-i386-x86_64/de/release/Darwin%2010/default/default/update.xml?force=1
+++ https://aus4.mozilla.org//update/3/Firefox/10.0.1/20120208060813/Darwin_x86_64-gcc3-u-i386-x86_64/de/release/Darwin%2010/default/default/update.xml?force=1
@@ -1,6 +1,6 @@
 <?xml version="1.0"?>
 <updates>
-    <update type="minor" displayVersion="32.0.3" appVersion="32.0.3" platformVersion="32.0.3" buildID="20140923175406" detailsURL="https://www.mozilla.com/de/firefox/32.0.3/releasenotes/" actions="showURL" openURL="https://www.mozilla.org/de/firefox/32.0.3/whatsnew/?oldversion=%OLD_VERSION%">
+    <update type="minor" displayVersion="32.0.3" appVersion="32.0.3" platformVersion="32.0.3" buildID="20140923175406" detailsURL="https://www.mozilla.com/de/firefox/32.0.3/releasenotes/">
         <patch type="complete" URL="http://download.mozilla.org/?product=firefox-32.0.3-complete&amp;os=osx&amp;lang=de&amp;force=1" hashFunction="sha512" hashValue="b35081c3fd4d9ef8817f0a2ad6fe0d66fde87cebd8c1eaa53bbd3b2af443d20a9ad75f55827cee1092ecc67b6cc732b779ee2e1416cd2adc03650dd9d8239865" size="63415956"/>
     </update>
 </updates>
---- end of diff

* Missing 31.0 -> 31.0.3 partial:
FAIL: Unmatched snippets on /update/3/Firefox/31.0/20140716183446/Darwin_x86_64-gcc3-u-i386-x86_64/de/release/Darwin%2010/default/default/update.xml?force=1:
--- https://aus3.mozilla.org//update/3/Firefox/31.0/20140716183446/Darwin_x86_64-gcc3-u-i386-x86_64/de/release/Darwin%2010/default/default/update.xml?force=1
+++ https://aus4.mozilla.org//update/3/Firefox/31.0/20140716183446/Darwin_x86_64-gcc3-u-i386-x86_64/de/release/Darwin%2010/default/default/update.xml?force=1
@@ -1,7 +1,6 @@
 <?xml version="1.0"?>
 <updates>
-    <update type="minor" displayVersion="32.0.3" appVersion="32.0.3" platformVersion="32.0.3" buildID="20140923175406" detailsURL="https://www.mozilla.com/de/firefox/32.0.3/releasenotes/" actions="showURL" openURL="https://www.mozilla.org/de/firefox/32.0.3/whatsnew/?oldversion=%OLD_VERSION%">
+    <update type="minor" displayVersion="32.0.3" appVersion="32.0.3" platformVersion="32.0.3" buildID="20140923175406" detailsURL="https://www.mozilla.com/de/firefox/32.0.3/releasenotes/">
         <patch type="complete" URL="http://download.mozilla.org/?product=firefox-32.0.3-complete&amp;os=osx&amp;lang=de&amp;force=1" hashFunction="sha512" hashValue="b35081c3fd4d9ef8817f0a2ad6fe0d66fde87cebd8c1eaa53bbd3b2af443d20a9ad75f55827cee1092ecc67b6cc732b779ee2e1416cd2adc03650dd9d8239865" size="63415956"/>
-        <patch type="partial" URL="http://download.mozilla.org/?product=firefox-32.0.3-partial-31.0&amp;os=osx&amp;lang=de&amp;force=1" hashFunction="sha512" hashValue="da4d7a181f5a8d10b28f021d2b59a0cd238aa305b173ea9403e1df6cadf2bbc87d6dbd079742e3fa992bb512460d83128dad647497f2031a402f12ee8db910d7" size="23276626"/>
     </update>
 </updates>
---- end of diff

* No updates served for Thunderbird:
FAIL: Unmatched snippets on /update/3/Thunderbird/10.0.1/20120208124956/Linux_x86_64-gcc3/de/release/GTK%202.18./default/default/update.xml?force=1:
--- https://aus3.mozilla.org//update/3/Thunderbird/10.0.1/20120208124956/Linux_x86_64-gcc3/de/release/GTK%202.18./default/default/update.xml?force=1
+++ https://aus4.mozilla.org//update/3/Thunderbird/10.0.1/20120208124956/Linux_x86_64-gcc3/de/release/GTK%202.18./default/default/update.xml?force=1
@@ -1,6 +1,3 @@
 <?xml version="1.0"?>
 <updates>
-    <update type="minor" displayVersion="31.1.2" appVersion="31.1.2" platformVersion="31.1.2" buildID="20140923203151" detailsURL="http://live.mozillamessaging.com/thunderbird/releasenotes?locale=de&amp;platform=linux-x86_64&amp;version=31.1.2">
-        <patch type="complete" URL="http://download.mozilla.org/?product=thunderbird-31.1.2-complete&amp;os=linux64&amp;lang=de&amp;force=1" hashFunction="sha512" hashValue="df7f7142326593077cbc44e3ea1cae3b299c4a7157bc64407b9d9f754d383d0bf0ffd2c8c237d7fa9198238130c299d97865781308f7ee32768f123bdb829cc9" size="32400292"/>
-    </update>
 </updates>
---- end of diff

There's probably something else buried in the log there but it's hard to see through all the existing noise. I also want to do a bigger comparison test, but I'll wait until after these are cleared up to do that.

Comment 2

[[e-mail] (:bhearsum)]

Oops, the Thunderbird failures were caused by not having the "release" channel rule in Balrog pointing at anything. After fixing that they failed again because of detailsUrl differences. This happened for Beta too, because the snippet generating code uses really old detailsURLs, but the balrog client code uses newer (more correct) code. After fixing that, the results are:
Tested 1776 paths.
Pass count: 1161
Fail count: 615
Error count: 0

But we have a few new failures where we're missing partials for some Thunderbird requests:
FAIL: Unmatched snippets on /update/3/Thunderbird/24.6.0/20140610001341/WINNT_x86-msvc/ru/release/Windows_NT%205.1/default/default/update.xml?force=1:
--- https://aus3.mozilla.org//update/3/Thunderbird/24.6.0/20140610001341/WINNT_x86-msvc/ru/release/Windows_NT%205.1/default/default/update.xml?force=1
+++ https://aus4.mozilla.org//update/3/Thunderbird/24.6.0/20140610001341/WINNT_x86-msvc/ru/release/Windows_NT%205.1/default/default/update.xml?force=1
@@ -2,6 +2,5 @@
 <updates>
     <update type="minor" displayVersion="31.1.2" appVersion="31.1.2" platformVersion="31.1.2" buildID="20140923203151" detailsURL="blah">
         <patch type="complete" URL="http://download.mozilla.org/?product=thunderbird-31.1.2-complete&amp;os=win&amp;lang=ru&amp;force=1" hashFunction="sha512" hashValue="e0bf63113211d5dfb2fbc9850fef7bdcfeeb8af2be636b243c1739f3802723b72501c8f72a7e3e0ecf9f4b01d81ecafb17a6d8be8867b1913fd0cb9ecbbfccd8" size="32106382"/>
-        <patch type="partial" URL="http://download.mozilla.org/?product=thunderbird-31.1.2-partial-24.6.0&amp;os=win&amp;lang=ru&amp;force=1" hashFunction="sha512" hashValue="d5989c7d7c9c8994596d35cd12e549da187128c5ae0d84565af930fba4f2b9eb943815d3d2b5083dc94b91d6f7799a1822d7130db75b1baf383d67183660a60c" size="20091777"/>
     </update>
 </updates>
---- end of diff


I suspect this is rooted in the same thing as the missing Firefox partials.

Comment 3

[[e-mail] (:bhearsum)]

I ended up doing a much larger snippet comparison sooner than I expected. It covered all platforms+locales all the way back to 10.0 and found no new failure cases compared to the quick test, hooray! I won't be rerunning that set again until small set of URLs fully pass.

Comment 4

[[e-mail] (:bhearsum)]

As a temporary workaround, I manually added "actions" and "openURL" to the Firefox 32.0.3 blob. There was only 45 failures in the short url list after that, all of which are the missing partial (bug 1074970). This is very encouraging!

Comment 5

[[e-mail] (:bhearsum)]

We switched over today.