1261637 - add stubbed out versions of releases recently deleted from balrog

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

I haven't checked other releases, but this issue was caught by 46.0b7 update verify:

https://aus5.mozilla.org/update/3/Firefox/46.0/20160316065941/Linux_x86-gcc3/zh-TW/beta-localtest/default/default/default/update.xml?force=1

Serves the following update:
<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="46.0 Beta 7" appVersion="46.0" platformVersion="46.0" buildID="20160401021843" detailsURL="https://www.mozilla.org/zh-TW/firefox/46.0/releasenotes/">
        <patch type="complete" URL="http://archive.mozilla.org/pub/firefox/candidates/46.0b7-candidates/build2/update/linux-i686/zh-TW/firefox-46.0b7.complete.mar" hashFunction="sha512" hashValue="46759d7a2504c1b57ef3b50a8c838e032f31c4af2b416054b026152bb4ebb44b52330cab29fdd5ab74ab3fc7f73ff36ae192cda0e80cebff6dda1c0c1cefcc45" size="53225072"/>
        <patch type="partial" URL="http://archive.mozilla.org/pub/firefox/candidates/46.0b7-candidates/build2/update/linux-i686/zh-TW/firefox-46.0b5-46.0b7.partial.mar" hashFunction="sha512" hashValue="583c73bc4a70463efe9fa457a4c7e5e6a06c938131c280f101c98d832c2fe503b7c901b554e73052880e6aa889fb5e116ee5c811042a47365105c46cf7f0f7dd" size="4367169"/>
    </update>
</updates>

According to http://hg.mozilla.org/build/tools/file/cb50be544cd9/release/patcher-configs/mozBeta-branch-patcher2.cfg#l1609 build ID 20160316065941 is 46.0b2, not 46.0b5. Actually, changing build ID doesn't make any difference.

I checked Firefox-46.0b7-build2, and the beta-localtest entry looks ok to me:

    "beta-localtest": {
      "partials": {
        "Firefox-46.0b2-build3": "http://archive.mozilla.org/pub/firefox/candidates/46.0b7-candidates/build2/update/%OS_FTP%/%LOCALE%/firefox-46.0b2-46.0b7.partial.mar",
        "Firefox-46.0b4-build2": "http://archive.mozilla.org/pub/firefox/candidates/46.0b7-candidates/build2/update/%OS_FTP%/%LOCALE%/firefox-46.0b4-46.0b7.partial.mar",
        "Firefox-46.0b5-build2": "http://archive.mozilla.org/pub/firefox/candidates/46.0b7-candidates/build2/update/%OS_FTP%/%LOCALE%/firefox-46.0b5-46.0b7.partial.mar"
      },

Also, 20160316065941 is in Firefox-46.0b2-build3 as expected.

Comment 1

[Rail Aliiev [:rail]]

Nick, Ben, can you help debugging this? So far I have no clue what is going on here. Maybe related to recent changes in Balrog?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Bug 1074362 and we built a partial for a release that wasn't shipped or got culled ?

Comment 3

[Rail Aliiev [:rail]]

Oh, maybe Firefox-46.0b5-build2 was deleted from the DB. I see http://ftp.mozilla.org/pub/firefox/releases/46.0b5/ in place, but not in Balrog.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Yes, it was. I've reproduced the issue in a dev environment, and we'll get this error whenever 
* when a partial should be served, but 46.0b5 appears before the right version in the 'partials' list for that locale
* all completes

I have a patch to attach to bug 1074362, but the quickest fix is to probably to resurrect the Firefox-46.0b5-build2 blob and trim almost everything out. Just leave buildIDs and whatever else is mandatory but make sure we can't serve with it by removing hashes/sizes/urls etc.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Firefox-46.0b5-build2-trimmed.json

Balrog admin will accept this in dev, and it fixes the wrong partial issue for the URL in the initial comment, as well as for 46.0b4 (wrong partial) and 45.0b9 (complete only). Ben, OK to push this back into prod ?

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #5)
> Created attachment 8737660 [details]
> Firefox-46.0b5-build2-trimmed.json
> 
> Balrog admin will accept this in dev, and it fixes the wrong partial issue
> for the URL in the initial comment, as well as for 46.0b4 (wrong partial)
> and 45.0b9 (complete only). Ben, OK to push this back into prod ?

Yeah, that looks sensible to me. We also culled b6 IIRC, though I'm not sure it was ever shipped?

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #6)
> (In reply to Nick Thomas [:nthomas] from comment #5)
> > Created attachment 8737660 [details]
> > Firefox-46.0b5-build2-trimmed.json
> > 
> > Balrog admin will accept this in dev, and it fixes the wrong partial issue
> > for the URL in the initial comment, as well as for 46.0b4 (wrong partial)
> > and 45.0b9 (complete only). Ben, OK to push this back into prod ?
> 
> Yeah, that looks sensible to me. We also culled b6 IIRC, though I'm not sure
> it was ever shipped?

Here's the full list of Releases that were culled:
Fennec-46.0b6-build1 (probably don't care about this one, we haven't been updating beta channel Fennec rules)
Firefox-46.0b5-build1
Firefox-46.0b5-build2
Firefox-46.0b6-build1
Thunderbird-38.7.1-build1
Thunderbird-45.0b3-build1
Thunderbird-45.0b3-build2

We probably need to stub out all of the Firefox and Thunderbird releases that were shipped, and probably best to stub out the unshipped build1's as well.

Comment 8

[Rail Aliiev [:rail]]

I added the Firefox-46.0b5-build2 release blob and testing it using one of the failed update verify builders.

Comment 9

[Rail Aliiev [:rail]]

Sounds like it's fixed now \o/

Thank you Nick for the help!

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #7)
> (In reply to Ben Hearsum (:bhearsum) from comment #6)
> > (In reply to Nick Thomas [:nthomas] from comment #5)
> > > Created attachment 8737660 [details]
> > > Firefox-46.0b5-build2-trimmed.json
> > > 
> > > Balrog admin will accept this in dev, and it fixes the wrong partial issue
> > > for the URL in the initial comment, as well as for 46.0b4 (wrong partial)
> > > and 45.0b9 (complete only). Ben, OK to push this back into prod ?
> > 
> > Yeah, that looks sensible to me. We also culled b6 IIRC, though I'm not sure
> > it was ever shipped?
> 
> Here's the full list of Releases that were culled:
> Fennec-46.0b6-build1 (probably don't care about this one, we haven't been
> updating beta channel Fennec rules)
> Firefox-46.0b5-build1
> Firefox-46.0b5-build2
> Firefox-46.0b6-build1
> Thunderbird-38.7.1-build1
> Thunderbird-45.0b3-build1
> Thunderbird-45.0b3-build2
> 
> We probably need to stub out all of the Firefox and Thunderbird releases
> that were shipped, and probably best to stub out the unshipped build1's as
> well.

I added back stubbed out versions of everything except Fennec, generated with a script. I'll attach that here momentarily for posterity.

Comment 11

[[e-mail] (:bhearsum)]

Attachment: script to generated stubbed release from full one