920453 - Aurora 64b Linux fails to update (due to empty update.xml)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Andrei Eftimie]

We failed to update Aurora on Linux 64b:
http://mozmill-daily.blargon7.com/#/update/report/837c1e0f361ac93453ee697219613d87

The update.xml is empty:
https://aus3.mozilla.org/update/3/Firefox/26.0a2/20130923004006/Linux_x86_64-gcc3/en-US/aurora/Linux%203.2.0-52-generic%20(GTK%202.24.10)/default/default/update.xml?force=1

I'll update the AUS update.xml for reference.

Firefox build that fails: 20130923004006

Tested manually (thanks Andreea) on 64b Linux, and it doesn't report any available updates, yet we have them available on the FTP server:
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-09-24-00-40-01-mozilla-aurora/

Comment 1

[Andrei Eftimie]

Attachment: update.xml

Ehm, not sure if this will be any help...

Empty update.xml file from:
https://aus3.mozilla.org/update/3/Firefox/26.0a2/20130923004006/Linux_x86_64-gcc3/en-US/aurora/Linux%203.2.0-52-generic%20(GTK%202.24.10)/default/default/update.xml?force=1

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It sounds like another symptom of bug 858953.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ups. I didn't wanted to close it.

Comment 4

[Armen [:armenzg]]

I don't know how to fix this.
I need help on what to do or even better if someone could grab this!

Sequence of events
- 63a505ec015c checked in on Saturday - ffxbld – Sat Sep 21 03:23:54 2013 PDT
- Nightly build starts on Sunday Sep 22 2013 00:40:30
-- for push 63a505ec015c (correct changeset)
- No nightly build on Monday since no code was checked in from the last nightly
- 07a29b018edc checked in on Monday - [e-mail] – Mon Sep 23 04:04:02 2013 PDT
- Many more check-ins on Monday
- b34384409be6 checked in on Monday - ffxbld – Mon Sep 23 16:30:43 2013 PDT
- Nightly build starts on Tuesday - Sep 24 2013 00:51:18 - 03:48
-- for push b34384409be6 (correct changeset)
-- it finished before the next nightly was to be triggered
- Nightly build starts on Tuesday - Sep 24 2013 05:56:06
-- for push 63a505ec015c (__older__ changeset!!)

The last nightly should have not happened IIUC.

All 3 nightly builds say this:
 The Nightly scheduler named 'mozilla-aurora nightly' triggered this build 
http://buildbot-master61.srv.releng.use1.mozilla.com:8001/builders/Linux%20x86-64%20mozilla-aurora%20nightly/builds/4
http://buildbot-master58.srv.releng.usw2.mozilla.com:8001/builders/Linux%20x86-64%20mozilla-aurora%20nightly/builds/31
http://buildbot-master63.srv.releng.use1.mozilla.com:8001/builders/Linux%20x86-64%20mozilla-aurora%20nightly/builds/19

[1] https://tbpl.mozilla.org/php/getParsedLog.php?id=28205526&tree=Mozilla-Aurora&full=1
buildid: 20130922004001
[2] https://tbpl.mozilla.org/php/getParsedLog.php?id=28278063&tree=Mozilla-Aurora&full=1
buildid: 20130924004001
[3] https://tbpl.mozilla.org/php/getParsedLog.php?id=28287311&tree=Mozilla-Aurora&full=1
buildid: 20130923004006 -- buildid of the *23rd*?

The upload dates show this as well. Here are the last 4 nightly partial mar files in *descendent* order:
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09/?C=M;O=D

https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09/2013-09-25-00-40-04-mozilla-aurora/firefox-26.0a2.en-US.linux-x86_64.partial.20130923004006-20130925004004.mar
25-Sep-2013 10:34
23th -> 25th (2 days jump!)

https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09/2013-09-23-00-40-06-mozilla-aurora/firefox-26.0a2.en-US.linux-x86_64.partial.20130924004001-20130923004006.mar
24-Sep-2013 15:16
24th -> 23rd (backwards!)

https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09/2013-09-24-00-40-01-mozilla-aurora/firefox-26.0a2.en-US.linux-x86_64.partial.20130922004001-20130924004001.mar
24-Sep-2013 10:45
22nd -> 24th (2 days jump!)

https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/09/2013-09-22-00-40-01-mozilla-aurora/firefox-26.0a2.en-US.linux-x86_64.partial.20130921004001-20130922004001.mar
22-Sep-2013 09:55
21st -> 22nd (normal)

I can verify that I can update from the 22nd (complete update to the 23rd???) but I can't update from the build of neither the 23rd nor the 24th.

Comment 5

[Armen [:armenzg]]

I have tested that we're good for now.

I opened the build from the 22th and updated to the 26th.

We nevertheless have to fix this for real.