1216737 - TEST-UNEXPECTED-FAIL | test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Available update has been found (transfer error: Update XML file not found (404) - request.status: 503)

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Today we had a failure for our update tests because the update.xml file didn't exist. I don't see how this can happen given that we run our tests after we have received the funsize Pulse message. I assumed that at this time everything is in place for testing. But looks like it's not always the case?

Here the gecko log of the problem:

05:55:13 *** AUS:SVC UpdateManager:_loadXMLFileIntoArray: XML file does not exist
05:55:15 *** AUS:SVC Checker: checkForUpdates, force: true
05:55:15 *** AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/3/Firefox/44.0a1/20151017030224/Linux_x86_64-gcc3/en-US/nightly/Linux%203.13.0-53-generic%20(GTK%203.10.8)/default/default/update.xml?force=1
05:55:15 *** AUS:SVC Checker:checkForUpdates - sending request to: https://aus5.mozilla.org/update/3/Firefox/44.0a1/20151017030224/Linux_x86_64-gcc3/en-US/nightly/Linux%203.13.0-53-generic%20(GTK%203.10.8)/default/default/update.xml?force=1
05:55:16 *** AUS:SVC Checker:onLoad - request completed downloading document
05:55:16 *** AUS:SVC Checker:_updates get - unexpected node name!
05:55:16 *** AUS:SVC Checker:onLoad - there was a problem checking for updates. Exception: Error: Unexpected node name, expected: updates, got: parsererror
05:55:16 *** AUS:SVC Checker:onLoad - request.status: 503
05:55:16 *** AUS:SVC getStatusTextFromCode - transfer error: Update XML file not found (404), default code: 404

Rail, what do you think could be the reason for it?

Comment 1

[Rail Aliiev [:rail]]

Could be related to the FTP->S3 migration.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hm, but the update.xml file does not live on FTP but comes from Balrog.

Comment 3

[Rail Aliiev [:rail]]

Oh, was it an actual 404 for the update link? That's weird... Balrog usually doesn't return 404, instead it returns a properly xml-formed empty update. Could be the load balancer?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I cannot see more in the above log as that it comes from here:
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#667

I assume Rob can give us more details.

Comment 5

[Robert Strong (they/them - no direct email)]

That means the update.xml was not present.

I just tried opening the url in comment #0 and got the following error which would account for this.

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Comment 7

[Chris AtLee [:catlee]]

I think this line is confusing:
05:55:16 *** AUS:SVC getStatusTextFromCode - transfer error: Update XML file not found (404), default code: 404

If it's really getting a 503 from the server.

Comment 8

[Robert Strong (they/them - no direct email)]

OK. That function is about the text it is going to return and the previous line which is the status code is
05:55:16 *** AUS:SVC Checker:onLoad - request.status: 503

Comment 9

[Robert Strong (they/them - no direct email)]

BTW: a very long time ago I wanted the text returned in the UI to be the exact error that was encountered and UX didn't want that. So, the request.status has to be used.

Also, you can see what I got when just opening the url by clicking the link in comment #5 (I copied / pasted what was in the browser into this bug). It is opening now but it definitely appears that something weird is going on the server side.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We had this problem again for Nightly builds lately. For example see:
https://treeherder.mozilla.org/logviewer.html#?job_id=2891879&repo=mozilla-central

There is also a HTTP log attached if it helps you to better determine the issue.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Looks like this was a infra issue and got fixed via bug 1199721. Moving this bug back into our test component and marking it as fixed. I will keep an eye on the results over the next days.