Closed Bug 806343 Opened 7 years ago Closed 7 years ago

Aurora update not offered since 10/23

Categories

(Firefox for Android :: General, defect, critical)

18 Branch
ARM
Android
defect
Not set
critical

Tracking

()

VERIFIED FIXED
Firefox 19
Tracking Status
firefox18 + verified
firefox19 + verified

People

(Reporter: xti, Assigned: snorp)

Details

Attachments

(1 file, 1 obsolete file)

Firefox 18.0a2 (2012-10-23)
Device: Galaxy S2
OS: Android 4.0.3

Steps to reproduce:
1. Install an Aurora build since 10/23 or later
2. Open Aurora
3. Go to about:firefox
4. Tap on Check for Updates button

Expected result:
The new update is downloaded (if the WiFi connection is active)

Actual result:
No updates message is displayed after step 4.
I/UpdateService( 2540): no update available
Component: General → Release Engineering
Product: Firefox for Android → mozilla.org
Version: Firefox 18 → other
Summary: Aurora update is not working since 10/23 → (mobile) Aurora update not offered since 10/23
Assignee: nobody → bhearsum
ARMv6 is good, ARMv7 no good
Summary: (mobile) Aurora update not offered since 10/23 → (mobile/ARMv7) Aurora update not offered since 10/23
Well, something is happening:

complete snippet: |version=1\ntype=complete\nurl=http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/10/2012-10-29-04-20-12-mozilla-aurora-android/fennec-18.0a2.multi.android-arm.apk\nhashFunction=sha512\nhashValue=cc726a92f3f36df3617ecbd26a75787d3c6b8b1770badd3f8b0790bed4694a8668705ce461f4d9dfe7d692819c72bfb7e0cb05f467a875480c7cd39d26dfbd2f\nsize=26049005\nbuild=20121029042012\nappv=18.0a2\nextv=18.0a2|
command: START
command: ssh -l ffxbld -i /home/cltbld/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US
command: cwd: /builds/slave/m-aurora-andrd-ntly/build
command: output:
command: END (0.62s elapsed)

command: START
command: scp -i /home/cltbld/.ssh/auspush complete.txt partial.txt ffxbld@aus3-staging.mozilla.org:/opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US
command: cwd: /builds/slave/m-aurora-andrd-ntly/build
command: output:
command: END (0.72s elapsed)

command: START
command: ssh -l ffxbld -i /home/cltbld/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US
command: cwd: /builds/slave/m-aurora-andrd-ntly/build
command: output:
command: END (0.46s elapsed)

command: START
command: ssh -l ffxbld -i /home/cltbld/.ssh/auspush aus3-staging.mozilla.org 'touch /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US/complete.txt /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US/partial.txt /opt/aus2/incoming/2/Fennec/mozilla-aurora/Android_arm-eabi-gcc3/20121029042012/en-US'
command: cwd: /builds/slave/m-aurora-andrd-ntly/build
command: output:
command: END (0.47s elapsed)

But I also see the Balrog client failing. That doesn't directly affect production updates, but the error might be indicative of the root problem:
Traceback (most recent call last):
  File "/builds/slave/m-aurora-andrd-ntly/tools/scripts/updates/balrog-client.py", line 36, in <module>
    runner.run()
  File "/builds/slave/m-aurora-andrd-ntly/tools/lib/python/balrog/client/cli.py", line 71, in run
    data = self.generate_data()
  File "/builds/slave/m-aurora-andrd-ntly/tools/lib/python/balrog/client/cli.py", line 54, in generate_data
    'filesize': props['completeMarSize'],

Still looking into this.
This smells like client side issues. bug 802396 landed between the 22nd and 23rd.
Per comments 4 and 5, I'm pretty sure this is a regression from bug 802396.
Component: Release Engineering → General
Product: mozilla.org → Firefox for Android
Version: other → Firefox 18
Assignee: bhearsum → nobody
The build from 10/22 it's good. I got the update
I shut off updates for all of Fennec on mozilla-aurora as a precaution.
I get "no update available" on a 10/24 mozilla-aurora armv6 build, too. I don't think this is just armv7
Summary: (mobile/ARMv7) Aurora update not offered since 10/23 → Aurora update not offered since 10/23
Aurora ARMv6 (Oct 25th build) - I/UpdateService( 4828): no update available
I also see bug 774618 landing in my guessed regression range (https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=733109bd4ed1&tochange=636b289899a4)

Rob, Brian - any idea if that bug could've broken android updates on aurora (but not on nightly)
Severity: major → critical
11:31 < snorp> https://aus2.mozilla.org/update/4/Fennec/18.0a2/%BUILDID%/Android_arm-eabi-gcc3-/%LOCALE%/aurora/%
OS_VERSION%/default/default/18.0a2/update.xml
11:31 < snorp> it's using that
11:31 < snorp> buildid is replaced with 20121029042012
11:31 < snorp> locale with whatever update.locale has
11:32 < snorp> and os version with the android version
11:32 < snorp> I already see the problem
11:32 < bhearsum|buildduty> ok, so 
https://aus3.mozilla.org/update/4/Fennec/18.0a2/20121028042012/Android_arm-eabi-gcc3-/en-US/aurora/Linux/default/default/18.0a2/update.xml has no update
11:32 < snorp> bhearsum|buildduty: MOZ_PKG_SPECIAL is apparently defined
11:32 < snorp> but it's blank?
11:32 < snorp> the - after gcc3 indicates that
11:32 < bhearsum|buildduty> ah
11:33 -!- hwine-commute is now known as hwine
11:33 < snorp> I would say rogue env var in the build somehow
11:33 < snorp> hmm
11:33 < snorp> how could this possibly work in nightly
11:33 < snorp> crap
11:34 < snorp> we always do -DMOZ_PKG_SPECIAL=$(MZO_PKG_SPECIAL) in the makefile
11:34 < snorp> we should probably not do that.


The reason Nightly wasn't affected by this is because over in bug 772045 we added a symlink for the "Android_arm-eabi-gcc3-" target a few months ago. I added the same for Aurora to unbork those users.
Comment on attachment 676159 [details] [diff] [review]
Avoid empty define for MOZ_PKG_SPECIAL


>diff --git a/mobile/android/base/UpdateService.java b/mobile/android/base/UpdateService.java

>+            Log.i(LOGTAG, "Checking for updates using " + url);
>+

Does this need to be in our production code? Does it leak private data? I'd rather remove it.
I cannot reproduce this issue on the yesterday's Nightly build.
(In reply to Mark Finkle (:mfinkle) from comment #14)
> Comment on attachment 676159 [details] [diff] [review]
> Avoid empty define for MOZ_PKG_SPECIAL
> 
> 
> >diff --git a/mobile/android/base/UpdateService.java b/mobile/android/base/UpdateService.java
> 
> >+            Log.i(LOGTAG, "Checking for updates using " + url);
> >+
> 
> Does this need to be in our production code? Does it leak private data? I'd
> rather remove it.

We have equivalent logging in Desktop, when app.update.log = True, for what it's worth.
(In reply to Ben Hearsum [:bhearsum] from comment #11)
> I also see bug 774618 landing in my guessed regression range
> (https://hg.mozilla.org/releases/mozilla-aurora/
> pushloghtml?fromchange=733109bd4ed1&tochange=636b289899a4)
> 
> Rob, Brian - any idea if that bug could've broken android updates on aurora
> (but not on nightly)

I don't think so, but are either of these 2 LOG messages displayed int he app.udpate.log -> true console log?

1)
 LOG("UpdateService:_postUpdateProcessing - previous patch failed " +
		"and no patch available");

2)
 LOG("UpdateManager:get activeUpdate - channel has changed, " +
// partial downloads		"reloading default preferences to workaround bug 802022");


If neither show up in the log it's not related.
(In reply to Ben Hearsum [:bhearsum] from comment #16)
> (In reply to Mark Finkle (:mfinkle) from comment #14)
> > Comment on attachment 676159 [details] [diff] [review]
> > Avoid empty define for MOZ_PKG_SPECIAL
> > 
> > 
> > >diff --git a/mobile/android/base/UpdateService.java b/mobile/android/base/UpdateService.java
> > 
> > >+            Log.i(LOGTAG, "Checking for updates using " + url);
> > >+
> > 
> > Does this need to be in our production code? Does it leak private data? I'd
> > rather remove it.
> 
> We have equivalent logging in Desktop, when app.update.log = True, for what
> it's worth.

I'd be fine with making logging something that can be enabled, but is not on by default.
(In reply to Mark Finkle (:mfinkle) from comment #18)
> (In reply to Ben Hearsum [:bhearsum] from comment #16)
> > (In reply to Mark Finkle (:mfinkle) from comment #14)
> > > Comment on attachment 676159 [details] [diff] [review]
> > > Avoid empty define for MOZ_PKG_SPECIAL
> > > 
> > > 
> > > >diff --git a/mobile/android/base/UpdateService.java b/mobile/android/base/UpdateService.java
> > > 
> > > >+            Log.i(LOGTAG, "Checking for updates using " + url);
> > > >+
> > > 
> > > Does this need to be in our production code? Does it leak private data? I'd
> > > rather remove it.
> > 
> > We have equivalent logging in Desktop, when app.update.log = True, for what
> > it's worth.
> 
> I'd be fine with making logging something that can be enabled, but is not on
> by default.

It's pretty painful to add pref-driven stuff for the update service because it doesn't have direct access to them. I think it's fine to log this all the time (and would have saved a lot of time while tracking down this bug), but as it's not related to the fix for this bug I'll remove it.
Attachment #676159 - Attachment is obsolete: true
Attachment #676159 - Flags: review?(blassey.bugs)
Attachment #676167 - Flags: review?(blassey.bugs) → review+
Comment on attachment 676167 [details] [diff] [review]
Avoid empty define for MOZ_PKG_SPECIAL

[Approval Request Comment]
Fixes updater bug. Releng can work around it, but it would be nice to have the fix in anyways. Low risk.
Attachment #676167 - Flags: approval-mozilla-beta?
Attachment #676167 - Flags: approval-mozilla-aurora?
(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) from comment #21)
> Comment on attachment 676167 [details] [diff] [review]
> Avoid empty define for MOZ_PKG_SPECIAL
> 
> [Approval Request Comment]
> Fixes updater bug. Releng can work around it, but it would be nice to have
> the fix in anyways. Low risk.

Just to be clear about this, we _can_ work around it (like we can work around many things), but it's the not something we're going to leave live for very long - it risks giving users the wrong build if we have some additional bug that sends armv6 users to the Android_arm-eabi-gcc3- channel, or something else silly. I'm going to be removing the workaround in about a month (bug 806393).
Comment on attachment 676167 [details] [diff] [review]
Avoid empty define for MOZ_PKG_SPECIAL

Looks like we only need this uplifted as far as Aurora (18).
Attachment #676167 - Flags: approval-mozilla-beta?
Attachment #676167 - Flags: approval-mozilla-beta-
Attachment #676167 - Flags: approval-mozilla-aurora?
Attachment #676167 - Flags: approval-mozilla-aurora+
https://hg.mozilla.org/mozilla-central/rev/2d8d8ceb7a6a
Assignee: nobody → snorp
Status: NEW → RESOLVED
Closed: 7 years ago
Flags: in-testsuite-
Resolution: --- → FIXED
Target Milestone: --- → Firefox 19
Verified on:
Samsung Galaxy R (Android 2.3.4)
Aurora 18.0a2 (2012-11-18)
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.