984163 - Updates not found since 2014-03-13 Firefox for Android Nightly/Aurora

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[Richard Newman [:rnewman]]

Just forced a check, no result. 

I/UpdateService(24817): next update will be at: Sun Mar 16 17:34:28 PDT 2014
I/UpdateService(24817): opening connection with url: https://aus4.mozilla.org/update/4/Fennec/30.0a1/20140313030202/Android_arm-eabi-gcc3/en-US/nightly/4.4.2/default/default/30.0a1/update.xml

=>

<?xml version="1.0"?>
<updates>
</updates>

but:

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/

shows

[DIR]	2014-03-16-03-02-02-mozilla-central-android-armv6/	16-Mar-2014 12:33 	- 	 
[DIR]	2014-03-16-03-02-02-mozilla-central-android-l10n/	16-Mar-2014 11:33 	- 	 
[DIR]	2014-03-16-03-02-02-mozilla-central-android-x86/	16-Mar-2014 11:28 	- 	 
[DIR]	2014-03-16-03-02-02-mozilla-central-android/	16-Mar-2014 12:08 	-

Comment 1

[Richard Newman [:rnewman]]

Guessing this belongs in AUS…

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I'm on 2014-03-13 on my phone too. Moving this to the component for the next-gen update server.

Comment 3

[Bill Gianopoulos [:WG9s]]

When this has happened before, then updating manually to a later version by going to aurora.mozilla.org for aurora and nightly.mozilla.org has always got it working again doing this i am still stuck on March 15th.

Comment 4

[Bill Gianopoulos [:WG9s]]

And just in case you have not figured this out, it would be really nice to get people to the latest versions before the uplift which is tomorrow.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Turns out to not be what you suggested.

From the 20140316030202 armv7 build:

Balrog request to https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-20140316030202/builds/Android_arm-eabi-gcc3/en-US
Data sent: {'alias': 'null', 'product': u'Fennec', 'version': u'30.0a1', 'hashFunction': u'sha512', 'copyTo': '["Fennec-mozilla-central-nightly-latest"]', 'data': '{"buildID": "20140316030202", "appv": "30.0a1", "complete": {"fileUrl": "http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014/03/2014-03-16-03-02-02-mozilla-central-android/fennec-30.0a1.multi.android-arm.apk", "hashValue": "dde0763198b39795f54512c4378181a0f23aa2f353c20fffc4708f1e53301b7173faa19c8f5d2faeaf6a06c617120fe69053d36f875c5bcc177034edb20557b1", "from": "*", "filesize": "33425682"}, "extv": "30.0a1"}'}
Starting new HTTPS connection (3): aus4-admin.mozilla.org
"PUT /releases/Fennec-mozilla-central-nightly-20140316030202/builds/Android_arm-eabi-gcc3/en-US HTTP/1.1" 401 79
Caught HTTPError: ffxbld is not allowed to access /releases/:name/builds/:platform/:locale by PUT

This is fallout from bug 749297, which enhanced our permissions model so we could restrict accounts to operate on certain products (deployed in bug 983135, on March 13 after the nightlies happened that day). We didn't allow ffxbld access to Fennec due to an oversight.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

There's a patch on bug 984166 which will fix the backend so we can let ffxbld have access to Firefox + Fennec permissions. Or we could set up a separate account for Fennec submission. What do you prefer Ben ?

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #6)
> There's a patch on bug 984166 which will fix the backend so we can let
> ffxbld have access to Firefox + Fennec permissions. Or we could set up a
> separate account for Fennec submission. What do you prefer Ben ?

We use ffxbld accounts everywhere else for Fennec so I think it makes sense to do the same here. Thanks for digging into this, I'll try to get that fixed landed this morning.

Comment 8

[Aaron Train [:aaronmt]]

Aurora too (e.g, from my 03/13 build: https://aus4.mozilla.org/update/4/Fennec/29.0a2/20140313004000/Android_arm-eabi-gcc3/en-US/aurora/4.4.2/default/default/29.0a2/update.xml)

Comment 9

[[e-mail] (:bhearsum)]

The patch that Nick posted in bug 984166 works fine in my testing. I've asked for it to be pushed to production in bug 984391. Once that's done I'll try to rerun all of the failed Fennec nightly submissions. Most of the builds are done on spot instances these days, though, so there's no guarantee I'll be able to resubmit them all.

Comment 10

[[e-mail] (:bhearsum)]

I filed bug 984392 to help us catch this sort of issue more quickly in the future.

Comment 11

[[e-mail] (:bhearsum)]

OK, this is fixed now. I resubmitted what I could, but the spot instances for the following builds already died, so they'll have to wait for tomorrow's Nightly:
* Nightly x86
* Aurora armv6, armv7

Thanks for reporting this!