806343 - Aurora update not offered since 10/23

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Cristian Nicolae (:xti)]

Firefox 18.0a2 (2012-10-23)
Device: Galaxy S2
OS: Android 4.0.3

Steps to reproduce:
1. Install an Aurora build since 10/23 or later
2. Open Aurora
3. Go to about:firefox
4. Tap on Check for Updates button

Expected result:
The new update is downloaded (if the WiFi connection is active)

Actual result:
No updates message is displayed after step 4.

Comment 1

[Aaron Train [:aaronmt]]

I/UpdateService( 2540): no update available

Comment 2

[Aaron Train [:aaronmt]]

ARMv6 is good, ARMv7 no good

Comment 3

[[e-mail] (:bhearsum)]

Well, something is happening:

complete snippet: |version=1\ntype=complete\nurl=http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/10/2012-10-29-04-20-12-mozilla-aurora-android/fennec-18.0a2.multi.android-arm.apk\nhashFunction=sha512\nhashValue=cc726a92f3f36df3617ecbd26a75787d3c6b8b1770badd3f8b0790bed4694a8668705ce461f4d9dfe7d692819c72bfb7e0cb05f467a875480c7cd39d26dfbd2f\nsize=26049005\nbuild=20121029042012\nappv=18.0a2\nextv=18.0a2|
command: START
command: ssh -l ffxbld -i /home/cltbld/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US
command: cwd: /builds/slave/m-aurora-andrd-ntly/build
command: output:
command: END (0.62s elapsed)

command: START
command: scp -i /home/cltbld/.ssh/auspush complete.txt partial.txt [e-mail]:/opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US
command: cwd: /builds/slave/m-aurora-andrd-ntly/build
command: output:
command: END (0.72s elapsed)

command: START
command: ssh -l ffxbld -i /home/cltbld/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US
command: cwd: /builds/slave/m-aurora-andrd-ntly/build
command: output:
command: END (0.46s elapsed)

command: START
command: ssh -l ffxbld -i /home/cltbld/.ssh/auspush aus3-staging.mozilla.org 'touch /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US/complete.txt /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US/partial.txt /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US'
command: cwd: /builds/slave/m-aurora-andrd-ntly/build
command: output:
command: END (0.47s elapsed)

But I also see the Balrog client failing. That doesn't directly affect production updates, but the error might be indicative of the root problem:
Traceback (most recent call last):
  File "/builds/slave/m-aurora-andrd-ntly/tools/scripts/updates/balrog-client.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-aurora-andrd-ntly/tools/lib/python/balrog/client/cli.py", line 71, in run
    data = self.generate_data()
  File "/builds/slave/m-aurora-andrd-ntly/tools/lib/python/balrog/client/cli.py", line 54, in generate_data
    'filesize': props['completeMarSize'],

Still looking into this.

Comment 4

[[e-mail] (:bhearsum)]

Things look OK on the RelEng side. I get valid-looking updates from various URLs:
https://aus3.mozilla.org/update/4/Fennec/18.0a1/20121023042012/Android_arm-eabi-gcc3-armv6/en-US/nightly/Linux%202.6.32.9-27227-g3c98b0d/default/default/14.0a1/update.xml
https://aus3.mozilla.org/update/4/Fennec/18.0a1/20121028042012/Android_arm-eabi-gcc3-armv6/en-US/nightly/Linux%202.6.32.9-27227-g3c98b0d/default/default/14.0a1/update.xml

Do builds from 10/22 and earlier get updates? If so, this sounds like a client problem.

Comment 5

[[e-mail] (:bhearsum)]

This smells like client side issues. bug 802396 landed between the 22nd and 23rd.

Comment 6

[[e-mail] (:bhearsum)]

Per comments 4 and 5, I'm pretty sure this is a regression from bug 802396.

Comment 7

[Cristian Nicolae (:xti)]

The build from 10/22 it's good. I got the update

Comment 8

[[e-mail] (:bhearsum)]

I shut off updates for all of Fennec on mozilla-aurora as a precaution.

Comment 9

[[e-mail] (:bhearsum)]

I get "no update available" on a 10/24 mozilla-aurora armv6 build, too. I don't think this is just armv7

Comment 10

[Aaron Train [:aaronmt]]

Aurora ARMv6 (Oct 25th build) - I/UpdateService( 4828): no update available

Comment 11

[[e-mail] (:bhearsum)]

I also see bug 774618 landing in my guessed regression range (https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=733109bd4ed1&tochange=636b289899a4)

Rob, Brian - any idea if that bug could've broken android updates on aurora (but not on nightly)

Comment 12

[[e-mail] (:bhearsum)]

11:31 < snorp> https://aus2.mozilla.org/update/4/Fennec/18.0a2/%BUILDID%/Android_arm-eabi-gcc3-/%LOCALE%/aurora/%
OS_VERSION%/default/default/18.0a2/update.xml
11:31 < snorp> it's using that
11:31 < snorp> buildid is replaced with 20121029042012
11:31 < snorp> locale with whatever update.locale has
11:32 < snorp> and os version with the android version
11:32 < snorp> I already see the problem
11:32 < bhearsum|buildduty> ok, so 
https://aus3.mozilla.org/update/4/Fennec/18.0a2/20121028042012/Android_arm-eabi-gcc3-/en-US/aurora/Linux/default/default/18.0a2/update.xml has no update
11:32 < snorp> bhearsum|buildduty: MOZ_PKG_SPECIAL is apparently defined
11:32 < snorp> but it's blank?
11:32 < snorp> the - after gcc3 indicates that
11:32 < bhearsum|buildduty> ah
11:33 -!- hwine-commute is now known as hwine
11:33 < snorp> I would say rogue env var in the build somehow
11:33 < snorp> hmm
11:33 < snorp> how could this possibly work in nightly
11:33 < snorp> crap
11:34 < snorp> we always do -DMOZ_PKG_SPECIAL=$(MZO_PKG_SPECIAL) in the makefile
11:34 < snorp> we should probably not do that.


The reason Nightly wasn't affected by this is because over in bug 772045 we added a symlink for the "Android_arm-eabi-gcc3-" target a few months ago. I added the same for Aurora to unbork those users.

Comment 13

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: Avoid empty define for MOZ_PKG_SPECIAL

Comment 14

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 676159 [details] [diff] [review]
Avoid empty define for MOZ_PKG_SPECIAL


>diff --git a/mobile/android/base/UpdateService.java b/mobile/android/base/UpdateService.java

>+            Log.i(LOGTAG, "Checking for updates using " + url);
>+

Does this need to be in our production code? Does it leak private data? I'd rather remove it.

Comment 15

[Cristian Nicolae (:xti)]

I cannot reproduce this issue on the yesterday's Nightly build.

Comment 16

[[e-mail] (:bhearsum)]

(In reply to Mark Finkle (:mfinkle) from comment #14)
> Comment on attachment 676159 [details] [diff] [review]
> Avoid empty define for MOZ_PKG_SPECIAL
> 
> 
> >diff --git a/mobile/android/base/UpdateService.java b/mobile/android/base/UpdateService.java
> 
> >+            Log.i(LOGTAG, "Checking for updates using " + url);
> >+
> 
> Does this need to be in our production code? Does it leak private data? I'd
> rather remove it.

We have equivalent logging in Desktop, when app.update.log = True, for what it's worth.

Comment 17

[Brian R. Bondy [:bbondy]]

(In reply to Ben Hearsum [:bhearsum] from comment #11)
> I also see bug 774618 landing in my guessed regression range
> (https://hg.mozilla.org/releases/mozilla-aurora/
> pushloghtml?fromchange=733109bd4ed1&tochange=636b289899a4)
> 
> Rob, Brian - any idea if that bug could've broken android updates on aurora
> (but not on nightly)

I don't think so, but are either of these 2 LOG messages displayed int he app.udpate.log -> true console log?

1)
 LOG("UpdateService:_postUpdateProcessing - previous patch failed " +
		"and no patch available");

2)
 LOG("UpdateManager:get activeUpdate - channel has changed, " +
// partial downloads		"reloading default preferences to workaround bug 802022");


If neither show up in the log it's not related.

Comment 18

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Ben Hearsum [:bhearsum] from comment #16)
> (In reply to Mark Finkle (:mfinkle) from comment #14)
> > Comment on attachment 676159 [details] [diff] [review]
> > Avoid empty define for MOZ_PKG_SPECIAL
> > 
> > 
> > >diff --git a/mobile/android/base/UpdateService.java b/mobile/android/base/UpdateService.java
> > 
> > >+            Log.i(LOGTAG, "Checking for updates using " + url);
> > >+
> > 
> > Does this need to be in our production code? Does it leak private data? I'd
> > rather remove it.
> 
> We have equivalent logging in Desktop, when app.update.log = True, for what
> it's worth.

I'd be fine with making logging something that can be enabled, but is not on by default.

Comment 19

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Mark Finkle (:mfinkle) from comment #18)
> (In reply to Ben Hearsum [:bhearsum] from comment #16)
> > (In reply to Mark Finkle (:mfinkle) from comment #14)
> > > Comment on attachment 676159 [details] [diff] [review]
> > > Avoid empty define for MOZ_PKG_SPECIAL
> > > 
> > > 
> > > >diff --git a/mobile/android/base/UpdateService.java b/mobile/android/base/UpdateService.java
> > > 
> > > >+            Log.i(LOGTAG, "Checking for updates using " + url);
> > > >+
> > > 
> > > Does this need to be in our production code? Does it leak private data? I'd
> > > rather remove it.
> > 
> > We have equivalent logging in Desktop, when app.update.log = True, for what
> > it's worth.
> 
> I'd be fine with making logging something that can be enabled, but is not on
> by default.

It's pretty painful to add pref-driven stuff for the update service because it doesn't have direct access to them. I think it's fine to log this all the time (and would have saved a lot of time while tracking down this bug), but as it's not related to the fix for this bug I'll remove it.

Comment 20

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: Avoid empty define for MOZ_PKG_SPECIAL

Comment 21

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 676167 [details] [diff] [review]
Avoid empty define for MOZ_PKG_SPECIAL

[Approval Request Comment]
Fixes updater bug. Releng can work around it, but it would be nice to have the fix in anyways. Low risk.

Comment 22

[[e-mail] (:bhearsum)]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #21)
> Comment on attachment 676167 [details] [diff] [review]
> Avoid empty define for MOZ_PKG_SPECIAL
> 
> [Approval Request Comment]
> Fixes updater bug. Releng can work around it, but it would be nice to have
> the fix in anyways. Low risk.

Just to be clear about this, we _can_ work around it (like we can work around many things), but it's the not something we're going to leave live for very long - it risks giving users the wrong build if we have some additional bug that sends armv6 users to the Android_arm-eabi-gcc3- channel, or something else silly. I'm going to be removing the workaround in about a month (bug 806393).

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 676167 [details] [diff] [review]
Avoid empty define for MOZ_PKG_SPECIAL

Looks like we only need this uplifted as far as Aurora (18).

Comment 24

[James Willcox (:snorp) ([e-mail]) (he/him)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/4021cc0039bb

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/2d8d8ceb7a6a

Comment 26

[Paul Feher]

Verified on:
Samsung Galaxy R (Android 2.3.4)
Aurora 18.0a2 (2012-11-18)