687231 - Beta-N -> Beta fallback updates failing on releasetest for some locales

Status: RESOLVED INCOMPLETE

(Core :: Networking: HTTP, defect)

Description

[u279076]

Attachment: Update log

7.0b5 ja, ko, ru, zh-TW fallback updates on releasetest are failing to be found, but only under automation. I can fallback update these builds just fine if I test them manually.

I've attached a copy of the update log.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Something is wrong here by downloading the fallback patch:

*** AUS:UI gUpdates:onLoad - setting current page to startpage errorpatching
*** AUS:SVC Downloader:onStartRequest - original URI spec: http://download.mozilla.org/?product=firefox-7.0b6-complete&os=osx&lang=zh-TW&force=1, final URI spec: http://mozilla.3c.stonekitty.net/firefox/releases/7.0b6/update/mac/zh-TW/firefox-7.0b6.complete.mar
*** AUS:SVC Downloader:onStopRequest - original URI spec: http://download.mozilla.org/?product=firefox-7.0b6-complete&os=osx&lang=zh-TW&force=1, final URI spec: http://mozilla.3c.stonekitty.net/firefox/releases/7.0b6/update/mac/zh-TW/firefox-7.0b6.complete.mar, status: 2147549183
*** AUS:SVC Downloader:onStopRequest - non-verification failure
*** AUS:SVC getStatusTextFromCode - transfer error: 失敗 (不明原因), default code: 2152398849
*** AUS:SVC Downloader:onStopRequest - setting state to: download-failed
*** AUS:SVC Downloader:onStopRequest - all update patch downloads failed

This behavior only happens for a fallback of a partial patch. When I run the tests against 7.0b4 everything is fine. Same for 7.0b5 updates for en-US.

Rob, is there anything we could do to get more details about this unknown error? I could try to create a HTTP log tomorrow. But would be nice to hear if you have other ideas too. Thanks.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> Rob, is there anything we could do to get more details about this unknown
> error? I could try to create a HTTP log tomorrow. But would be nice to hear
> if you have other ideas too. Thanks.

I should also CC Rob.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: HTTP log

So not sure what's going on here but we fail while downloading the complete mar file from the server. See the full log attached.

1880710336[100116260]: nsHttpChannel::OnStartRequest [this=100c73800 request=121af6680 status=0]
1880710336[100116260]: nsHttpChannel::ProcessResponse [this=100c73800 httpStatus=206]
1880710336[100116260]: nsHttpHandler::NotifyObservers [chan=c73850 event="http-on-examine-response"]
1880710336[100116260]: nsHttpChannelAuthProvider::CheckForSuperfluousAuth? [this=11a14e040 channel=100c73ad0]
1880710336[100116260]:   continuation state has been reset
1880710336[100116260]: nsHttpChannel::ProcessNormal [this=100c73800]
1880710336[100116260]:   calling mListener->OnStartRequest
1880710336[100116260]: nsHttpChannel::OnStopRequest [this=100c73800 request=121af6680 status=8000ffff]
1880710336[100116260]:   calling OnStopRequest
95973376[100140320]: nsHttpConnection::OnSocketReadable [this=688660]
95973376[100140320]: nsHttpConnection::CloseTransaction[this=688660 trans=22102720 reason=8000ffff]
95973376[100140320]: nsHttpTransaction::Close [this=22102720 reason=8000ffff]
95973376[100140320]: nsHttpConnectionMgr::ReclaimConnection [conn=688660]
95973376[100140320]: nsHttpTransaction::DeleteSelfOnConsumerThread [this=22102720]
95973376[100140320]: proxying delete to consumer thread...
95973376[100140320]: nsHttpConnection::Close [this=688660 reason=8000ffff]
95973376[100140320]: nsHttpConnectionMgr::OnMsgReclaimConnection [conn=100688660]
95973376[100140320]:   connection cannot be reused; closing connection

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I don't think it's a fault in our automation scripts. It's probably triggered by the increased speed of the test. Moving over to Core:Network.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Steps to reproduce:

1. Get the Mozmill environment for your platform: https://mozqa.com/mozmill-env/
2. Execute setup.sh to install all dependencies
3. Clone the mozmill-automation repository: run.sh hg clone http://hg.mozilla.org/qa/mozmill-automation
4. Download the 7.0b5 zh-TW locale: run.sh download.py -v 7.0b5 -l zh-TW
5. Run the update test: run.sh testrun_update.py --channel=releasetest firefox-7.0b5.zh-TW.mac.dmg

Comment 7

[u279076]

So it looks like this only happens for BetaN-1 to BetaN where N is the most current version in the candidate cycle.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #7)
> So it looks like this only happens for BetaN-1 to BetaN where N is the most
> current version in the candidate cycle.

And only for releasetest. Could that be related to internal mirrors? Ben or Nick, does one of you have an idea?

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) from comment #8)
> (In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #7)
> > So it looks like this only happens for BetaN-1 to BetaN where N is the most
> > current version in the candidate cycle.
> 
> And only for releasetest. Could that be related to internal mirrors? Ben or
> Nick, does one of you have an idea?

Based on comment#1, I would bet that this isn't related to internal mirrors (mozilla.3c.stonekitty.net isn't one of them).

It's hard to say much more because this example happened with now-overwritten snippets. If you can repro again with the current snippets I can look at things in a bit more depth.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ben, please check bug 692626 which Anthony has filed earlier today. It also happens for updates from 8.0b1 -> 8.0b2

Comment 11

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) from comment #10)
> Ben, please check bug 692626 which Anthony has filed earlier today. It also
> happens for updates from 8.0b1 -> 8.0b2

Hmmm, I don't see anything weird for the 8.0b1 -> 8.0b2 zh-TW windows update. I'm not sure what to tell you =\.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I was still able to reproduce it earlier today with 8.0b1 ru and the steps given in comment 5 from our machine in the QA lab.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Just tried it again on OS X 10.7 with 8.0b1 ru and it is still failing.