769156 - 12->13.0.1 updates on releasetest fail due to 13.0.2build1 staging

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As reported by Anthony on bug 769090 yesterday currently updates on releasetest are broken and can't be applied. This happens for all releases across platforms. I will attach an update log which hopefully will help.

As what I can see is that we finish downloading the patches but finally see the error:

*** AUS:SVC Downloader:onStopRequest - non-verification failure
*** AUS:SVC getStatusTextFromCode - transfer error: Failed (unknown reason), default code: 2152398849
*** AUS:SVC Downloader:onStopRequest - setting state to: download-failed

We shouldn't put the updates onto the release channel until this problem has been fixed.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So the problem here is, when you install a 12.0 release of Firefox and check for updates on the releasetest channel you will get offered an update to 13.0.1 instead of 13.0.2 with a size of about 10.0MB. The regular size of the 12.0->13.0.1 update is 29MB when on the release channel.

The downloaded update is getting applied correctly and when you check another time for updates you finally will get the 13.0.2 offered. The patch size is of about 1.2MB on Windows. Starting the download fails immediately.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: mozmill results with updater log

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So this also fails for 13.0.1->13.0.2 updates on releasetests. The interim update to 13.0.1 could be a side-effect of the real issue.

I don't think I can do more for now and would move this over to Releng now so we can get this fixed.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

This is a consequence of the current unusual situation: 
* 13.0.1 is the current release for end users
* last weekend we built 13.0.2 build1, but it's unreleased and on hold
* we want a 12.0 -> 13.0.1 partial update to try to help uptake to 13.0.1

So from the 2nd point we have releasetest pointing to 13.0.2 build1 for everything 10.0 and higher, but the files aren't on the mirrors so bouncer returns 404s, and the updater reports error 2152398849. The exception is that 12.0 -> 13.0.1 is live on the test channels, and happens to be on the mirrors, so releasetest works there.

So Mozmill only needs to test 12.0 -> 13.0.1 here, and not go onwards. I think this boils down to INVALID.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Nick Thomas [:nthomas] from comment #4)
> So Mozmill only needs to test 12.0 -> 13.0.1 here, and not go onwards. I
> think this boils down to INVALID.

Anthony, were you aware of this situation? If not then we have to fix the information coming from the releng / release driver side what we have to test.

Comment 6

[u279076]

As I understand it, there should be no interim 13.0.1; in other words, 13.0.1 is the end-point and no 13.0.2 should be offered. Unfortunately that does not appear to be the case. Currently on releasetest there is 13.0.2build1 and 13.0.1 which is confusing Mozmill. I don't think we'll encounter this problem when we push to release because 13.0.2 snippets were never pushed there. 

Here are the options as I see it:
1) Do the sign-off fully manual; this means testing much less over a longer QA period.
2) Remove 13.0.2 from releasetest and retest with automation
3) Just push to release knowing the snippets were okay on betatest

Comment 7

[u279076]

A forth option Henrik pointed out to me is that the results show a successful 13.0.1 update, it's just the secondary check for the 404'd 13.0.2 update which is failing.

In other words:
1. Update 12.0 -> 13.0.1
2. Firefox restarts
3. Check for update again (not expecting another update)
4. Update to 13.0.2 offered but snippet is 404 (fail)

The reports show this:
http://mozmill-ondemand.blargon7.com/#/update/report/23e194f1171aa4baf774928b2604140f

Build pre:
Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0 (zh-TW, 20120420145725)

Build post:
Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0.1 (zh-TW, 20120614114901)

So a forth option is to ignore that final point of failure and move forward knowing 12->13.0.1 works and that we shouldn't encounter this when on release.

Comment 8

[Alex Keybl [:akeybl]]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #6)
> 2) Remove 13.0.2 from releasetest and retest with automation

I'm in favor of this option since it reflects reality, and we don't currently have plans to ship the 13.0.2 we have in hand. Ben let us know it wouldn't be too difficult.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Alex Keybl [:akeybl] from comment #8)
> (In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #6)
> > 2) Remove 13.0.2 from releasetest and retest with automation
> 
> I'm in favor of this option since it reflects reality, and we don't
> currently have plans to ship the 13.0.2 we have in hand. Ben let us know it
> wouldn't be too difficult.

I repushed the silent 13.0.1 snippets. Henrik/Anthony, you should be good to go here now.

Comment 10

[u279076]

I'm still hitting the same problem.

Comment 11

[u279076]

Example snippet after updating to 13.0.1:
https://aus3.mozilla.org/update/3/Firefox/13.0.1/20120614114901/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/releasetest/Darwin%2010.8.0/default/default/update.xml?force=1

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #11)
> Example snippet after updating to 13.0.1:
> https://aus3.mozilla.org/update/3/Firefox/13.0.1/20120614114901/
> Darwin_x86_64-gcc3-u-i386-x86_64/en-US/releasetest/Darwin%2010.8.0/default/
> default/update.xml?force=1

Sorry, I forgot that we needed to delete these 13.0.1 snippets by hand. Just did that now, you should be good to go.

Comment 13

[u279076]

Thanks Ben -- this appears to have fixed it. Our automation is looking good now:
http://mozmill-ondemand.blargon7.com/#/update/detail?branch=13.0&channel=releasetest&from=2012-06-28&to=2012-06-28&target=13.0.1

Note that I'm not aware of an easy way to filter out the previous failures from the report. I'll add context for those failures to the testplan.