1508600 - Startup Crash in java.lang.IllegalStateException: at org.mozilla.gecko.fxa.authenticator.AndroidFxAccount.getState(AndroidFxAccount.java)

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: Firefox Accounts, defect, P2)

Description

[Pascal Chevrel:pascalc]

This bug was filed from the Socorro interface and is
report bp-280fe71a-e510-4fcf-9b98-65c4a0181120.
=============================================================

Java stack trace:

java.lang.IllegalStateException
	at org.mozilla.gecko.fxa.authenticator.AndroidFxAccount.getState(AndroidFxAccount.java:754)
	at org.mozilla.gecko.fxa.FirefoxAccountsUtils.separateAccountAndShowNotification(FirefoxAccountsUtils.java:100)
	at org.mozilla.gecko.fxa.FirefoxAccountsUtils.optionallySeparateAccountsDuringFirstRun(FirefoxAccountsUtils.java:66)
	at org.mozilla.gecko.fxa.FirefoxAccounts.getFirefoxAccounts(FirefoxAccounts.java:55)
	at org.mozilla.gecko.fxa.FirefoxAccounts.firefoxAccountsExist(FirefoxAccounts.java:36)
	at org.mozilla.gecko.activitystream.homepanel.ActivityStreamPanel.<init>(ActivityStreamPanel.java:91)
	at org.mozilla.gecko.activitystream.homepanel.ActivityStreamHomeScreen.<init>(ActivityStreamHomeScreen.java:28)
	at java.lang.reflect.Constructor.newInstance0(Native Method)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:334)
	at android.view.LayoutInflater.createView(LayoutInflater.java:647)
	at android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:790)
	at android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:730)
	at android.view.LayoutInflater.inflate(LayoutInflater.java:492)
	at android.view.LayoutInflater.inflate(LayoutInflater.java:423)
	at org.mozilla.gecko.activitystream.homepanel.ActivityStreamHomeFragment.onCreateView(ActivityStreamHomeFragment.java:67)
	at android.support.v4.app.Fragment.performCreateView(Fragment.java:2354)
	at android.support.v4.app.FragmentManagerImpl.moveToState(FragmentManager.java:1419)
	at android.support.v4.app.FragmentManagerImpl.moveFragmentToExpectedState(FragmentManager.java:1740)
	at android.support.v4.app.FragmentManagerImpl.moveToState(FragmentManager.java:1809)
	at android.support.v4.app.BackStackRecord.executeOps(BackStackRecord.java:799)
	at android.support.v4.app.FragmentManagerImpl.executeOps(FragmentManager.java:2580)
	at android.support.v4.app.FragmentManagerImpl.executeOpsTogether(FragmentManager.java:2367)
	at android.support.v4.app.FragmentManagerImpl.removeRedundantOperationsAndExecute(FragmentManager.java:2322)
	at android.support.v4.app.FragmentManagerImpl.execSingleAction(FragmentManager.java:2199)
	at android.support.v4.app.BackStackRecord.commitNowAllowingStateLoss(BackStackRecord.java:651)
	at android.support.v4.app.FragmentStatePagerAdapter.finishUpdate(FragmentStatePagerAdapter.java:167)
	at android.support.v4.view.ViewPager.populate(ViewPager.java:1236)
	at android.support.v4.view.ViewPager.populate(ViewPager.java:1084)
	at android.support.v4.view.ViewPager.setAdapter(ViewPager.java:534)
	at com.booking.rtlviewpager.RtlViewPager.setAdapter(RtlViewPager.java:131)
	at org.mozilla.gecko.home.HomePager.updateUiFromConfigState(HomePager.java:427)
	at org.mozilla.gecko.home.HomePager.access$600(HomePager.java:38)
	at org.mozilla.gecko.home.HomePager$ConfigLoaderCallbacks.onLoadFinished(HomePager.java:516)
	at org.mozilla.gecko.home.HomePager$ConfigLoaderCallbacks.onLoadFinished(HomePager.java:507)
	at android.support.v4.app.LoaderManagerImpl$LoaderInfo.callOnLoadFinished(LoaderManager.java:476)
	at android.support.v4.app.LoaderManagerImpl$LoaderInfo.onLoadComplete(LoaderManager.java:444)
	at android.support.v4.content.Loader.deliverResult(Loader.java:126)
	at org.mozilla.gecko.home.HomeConfigLoader.deliverResult(HomeConfigLoader.java:41)
	at org.mozilla.gecko.home.HomeConfigLoader.deliverResult(HomeConfigLoader.java:13)
	at android.support.v4.content.AsyncTaskLoader.dispatchOnLoadComplete(AsyncTaskLoader.java:255)
	at android.support.v4.content.AsyncTaskLoader$LoadTask.onPostExecute(AsyncTaskLoader.java:80)
	at android.support.v4.content.ModernAsyncTask.finish(ModernAsyncTask.java:487)
	at android.support.v4.content.ModernAsyncTask$InternalHandler.handleMessage(ModernAsyncTask.java:504)
	at android.os.Handler.dispatchMessage(Handler.java:106)
	at android.os.Looper.loop(Looper.java:164)
	at android.app.ActivityThread.main(ActivityThread.java:6499)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:442)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)

Comment 1

[Liz Henry (:lizzard) (relman/hg->git project)]

Low volume crash affecting beta and release.

Comment 2

[Marta Stańczak]

(In reply to Pascal Chevrel:pascalc from comment #0)
> This bug was filed from the Socorro interface and is
> report bp-280fe71a-e510-4fcf-9b98-65c4a0181120.
> =============================================================
> 
> Java stack trace:
> 
> java.lang.IllegalStateException
> 	at
> org.mozilla.gecko.fxa.authenticator.AndroidFxAccount.
> getState(AndroidFxAccount.java:754)
> 	at
> org.mozilla.gecko.fxa.FirefoxAccountsUtils.
> separateAccountAndShowNotification(FirefoxAccountsUtils.java:100)
> 	at
> org.mozilla.gecko.fxa.FirefoxAccountsUtils.
> optionallySeparateAccountsDuringFirstRun(FirefoxAccountsUtils.java:66)
> 	at
> org.mozilla.gecko.fxa.FirefoxAccounts.getFirefoxAccounts(FirefoxAccounts.
> java:55)
> 	at
> org.mozilla.gecko.fxa.FirefoxAccounts.firefoxAccountsExist(FirefoxAccounts.
> java:36)
> 	at
> org.mozilla.gecko.activitystream.homepanel.ActivityStreamPanel.<init>
> (ActivityStreamPanel.java:91)
> 	at
> org.mozilla.gecko.activitystream.homepanel.ActivityStreamHomeScreen.<
> init>(ActivityStreamHomeScreen.java:28)
> 	at java.lang.reflect.Constructor.newInstance0(Native Method)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:334)
> 	at android.view.LayoutInflater.createView(LayoutInflater.java:647)
> 	at android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:790)
> 	at android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:730)
> 	at android.view.LayoutInflater.inflate(LayoutInflater.java:492)
> 	at android.view.LayoutInflater.inflate(LayoutInflater.java:423)
> 	at
> org.mozilla.gecko.activitystream.homepanel.ActivityStreamHomeFragment.
> onCreateView(ActivityStreamHomeFragment.java:67)
> 	at android.support.v4.app.Fragment.performCreateView(Fragment.java:2354)
> 	at
> android.support.v4.app.FragmentManagerImpl.moveToState(FragmentManager.java:
> 1419)
> 	at
> android.support.v4.app.FragmentManagerImpl.
> moveFragmentToExpectedState(FragmentManager.java:1740)
> 	at
> android.support.v4.app.FragmentManagerImpl.moveToState(FragmentManager.java:
> 1809)
> 	at
> android.support.v4.app.BackStackRecord.executeOps(BackStackRecord.java:799)
> 	at
> android.support.v4.app.FragmentManagerImpl.executeOps(FragmentManager.java:
> 2580)
> 	at
> android.support.v4.app.FragmentManagerImpl.
> executeOpsTogether(FragmentManager.java:2367)
> 	at
> android.support.v4.app.FragmentManagerImpl.
> removeRedundantOperationsAndExecute(FragmentManager.java:2322)
> 	at
> android.support.v4.app.FragmentManagerImpl.execSingleAction(FragmentManager.
> java:2199)
> 	at
> android.support.v4.app.BackStackRecord.
> commitNowAllowingStateLoss(BackStackRecord.java:651)
> 	at
> android.support.v4.app.FragmentStatePagerAdapter.
> finishUpdate(FragmentStatePagerAdapter.java:167)
> 	at android.support.v4.view.ViewPager.populate(ViewPager.java:1236)
> 	at android.support.v4.view.ViewPager.populate(ViewPager.java:1084)
> 	at android.support.v4.view.ViewPager.setAdapter(ViewPager.java:534)
> 	at com.booking.rtlviewpager.RtlViewPager.setAdapter(RtlViewPager.java:131)
> 	at
> org.mozilla.gecko.home.HomePager.updateUiFromConfigState(HomePager.java:427)
> 	at org.mozilla.gecko.home.HomePager.access$600(HomePager.java:38)
> 	at
> org.mozilla.gecko.home.HomePager$ConfigLoaderCallbacks.
> onLoadFinished(HomePager.java:516)
> 	at
> org.mozilla.gecko.home.HomePager$ConfigLoaderCallbacks.
> onLoadFinished(HomePager.java:507)
> 	at
> android.support.v4.app.LoaderManagerImpl$LoaderInfo.
> callOnLoadFinished(LoaderManager.java:476)
> 	at
> android.support.v4.app.LoaderManagerImpl$LoaderInfo.
> onLoadComplete(LoaderManager.java:444)
> 	at android.support.v4.content.Loader.deliverResult(Loader.java:126)
> 	at
> org.mozilla.gecko.home.HomeConfigLoader.deliverResult(HomeConfigLoader.java:
> 41)
> 	at
> org.mozilla.gecko.home.HomeConfigLoader.deliverResult(HomeConfigLoader.java:
> 13)
> 	at
> android.support.v4.content.AsyncTaskLoader.
> dispatchOnLoadComplete(AsyncTaskLoader.java:255)
> 	at
> android.support.v4.content.AsyncTaskLoader$LoadTask.
> onPostExecute(AsyncTaskLoader.java:80)
> 	at
> android.support.v4.content.ModernAsyncTask.finish(ModernAsyncTask.java:487)
> 	at
> android.support.v4.content.ModernAsyncTask$InternalHandler.
> handleMessage(ModernAsyncTask.java:504)
> 	at android.os.Handler.dispatchMessage(Handler.java:106)
> 	at android.os.Looper.loop(Looper.java:164)
> 	at android.app.ActivityThread.main(ActivityThread.java:6499)
> 	at java.lang.reflect.Method.invoke(Native Method)
> 	at
> com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:
> 442)
> 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)

Comment 3

[Ryan VanderMeulen [:RyanVM]]

This startup crash is still trending in the 65.0b4 release.

Comment 4

[:sdaswani]

Liz, should this be P1'ed?

Comment 5

[Liz Henry (:lizzard) (relman/hg->git project)]

Crash volume doesn't look high to me, only 2 crashes in 65.0b4. Ryan what do you mean by trending?

Comment 7

[Kevin Brosnan [Ex-Mozilla]]

Possible str from bug 1349147 comment 9.

> I was able to reproduce the issue (4/4) following this STR:
> 1. Go to Settings -> Sign in, and sign in with valid credentials.
> 2. Go to Settings -> Disconnect from your FxAccount
> 3. Nightly crashes

Comment 8

[Andrei Lazar]

Attachment: Bug 1508600 Startup Crash in java.lang.IllegalStateException: at org.mozilla.gecko.fxa.authenticator.AndroidFxAccount.getState(AndroidFxAccount.java) r=sdaswani

Signing in and then signing out fast enough will cause a crash, and that's because we are
dropping the account before the transition managed to end.
If we drop the account from the manager while married will causes a crash when handling the final.

Comment 9

[:Grisha Kruglov]

Bug 1505720 is about state misalignment that can be triggered by user action.

This bug is a startup crash that's triggered within the "require re-authentication if application data was cleared" code path. Bug 1429735 landed that functionality.

Comment 10

[:Grisha Kruglov]

Reading relevant code a bit more along with the stack traces, it seems that Bug 1429735 is connected only incidentally, in a sense that it's making a certain error condition more likely to show up (since we're hitting that code path more often now).

The specific error condition is misalignment of account's 'state' information with what we expect to see. Our expectation is that an account that we get from an Android Account manager will have a stateLabel & state json dump, which are necessary to recreate this account's FxA state machine state. We assume we'll have this state because we set it explicitly during account creation, never delete it and only update it. See [0] for when state is created initially.

However, when we actually query the account manager, at least one of those is null, meaning that we can't recreate an account state accurately. In fact, since this situation appears entirely not possible given our expectations above, we actually bail out at this point and throw a runtime exception, which is what those stack traces are.

Digging into the stacktraces a bit, 70% of these crashes happen on Huawei phones, mostly running API26. It's certainly not an even distribution over the user base, and so likely the cause of these crashes is some bad interaction with the accounts system on these specific devices. Perhaps the accounts system is buggy in these cases, or modified in a way that doesn't follow common patterns (wouldn't be a surprise to me).

There are two main directions to go here.

1. investigate what exactly happens on these specific devices.

• That is: How can we create an android account, pass in userdata via addAccountExplicitly, make sure it's really set by also iterating over userdata and setting it again (see [1]), and then still get back 'null' when we query it later on?

2. provide some kind of an escape hatch once we get into this state.

• This could look like
  -- re-initializing an account and hoping it'll work this time,
  -- returning a "fake" state and trying to persist that state in the background
  -- deleting an account entirely, which in practice means that the FxA system just won't work at all for the affected devices
  -- something else..? how can we do a sensible thing here when our core expectations aren't met?

I have a feeling that none of these options are going to be particularly easy or straightforward, but they're worth investigating. A key hurdle to get around might be that we're hitting this code path from many different entry points. There isn't a single owner of the account state, and so whoever queries it first after the problem starts to manifest itself will hit this problem. This makes our life harder, since it means that by the time we're in getState and crashing, we're deep in one of many top-level interactions.

Back when I actually worked on this codebase, the plan was to unify account management under a long-lived singleton-like object with a very explicit lifecycle, so that we have a single point of entry and ownership of these states, and thus a much easier recovery paths. Alas, that hasn't happened.

[0] https://dxr.mozilla.org/mozilla-central/source/mobile/android/services/src/main/java/org/mozilla/gecko/fxa/authenticator/AndroidFxAccount.java?q=path%3AAndroidFxAccount.java&redirect_type=single#579
[1] https://dxr.mozilla.org/mozilla-central/source/mobile/android/services/src/main/java/org/mozilla/gecko/fxa/authenticator/AndroidFxAccount.java?q=path%3AAndroidFxAccount.java&redirect_type=single#601

Comment 11

[Mira Lobontiu (Android QA)]

Hi all,

I encountered this crash today on Beta 66.0b11 ARM64 build (https://hg.mozilla.org/releases/mozilla-beta/rev/b08f44691ec3b9cdfb99bfd1a09c464e0a35785b), on Google Pixel (Android 9.0).

It was reproducible 2/3 times.
Not reproducible on Beta 66.0b11 from Playstore.

Should I file a new bug?

Crash signature:
java.lang.NullPointerException: at org.mozilla.gecko.fxa.authenticator.AndroidFxAccount.renameAccountIfNecessary(AndroidFxAccount.java)

Comment 12

[Chris Peterson [:cpeterson]]

Andrei, what are the next steps for this bug? The overall crash volume is pretty low, but it sounds like the crash is reproducible.

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

It sounds like this is not only reproducible but is a code path many users are likely to hit. Bumping the priority to P1.

Mirabela, did you have any clear STR to hit this bug?

Comment 14

[Mira Lobontiu (Android QA)]

Hi Liz,

The STR are:
Steps to reproduce:

1. Go to Settings -> Sign in, and sign in with valid credentials.
2. Go to Settings to your FxA and Disconnect
3. Beta crashes

Thanks!

Comment 15

[Andrei Lazar]

(In reply to Chris Peterson [:cpeterson] from comment #12)

Andrei, what are the next steps for this bug? The overall crash volume is pretty low, but it sounds like the crash is reproducible.

I tried a fix for this issue but it looks like I didn't covered all the cases and Grisha suggested another approach for this issue which requires far more resources, but Susheel said we shouldn't invest in maintaining this component unless it's high crash volume.

(In reply to miralobontiu from comment #14)

Hi Liz,

The STR are:
Steps to reproduce:

1. Go to Settings -> Sign in, and sign in with valid credentials.
2. Go to Settings to your FxA and Disconnect
3. Beta crashes

Thanks!

This is low volume because it happens only if you disconnect fast enough after you just signed in (~3 seconds, depends on internet connection).
I would add this description to STR.

Mira, can you confirm please?

Comment 16

[Mira Lobontiu (Android QA)]

Hi Andrei,
Yes, I confirm that Beta crashes only if I disconnect really fast.
If I wait for about 10 seconds, Beta doesn`t crash.
Thanks!

Comment 17

[Liz Henry (:lizzard) (relman/hg->git project)]

Stefan, it sounds like this startup crash in Accounts may increase in volume for Fennec users running ARM builds in the future, though it isn't high volume now. I'm not sure who might investigate further. Possibly glandium? What do you think?

Comment 18

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Why is this related to ARM builds?

Comment 19

[Chris Peterson [:cpeterson]]

(In reply to Andrei Lazar from comment #15)

(In reply to Chris Peterson [:cpeterson] from comment #12)

Andrei, what are the next steps for this bug? The overall crash volume is pretty low, but it sounds like the crash is reproducible.

I tried a fix for this issue but it looks like I didn't covered all the cases and Grisha suggested another approach for this issue which requires far more resources, but Susheel said we shouldn't invest in maintaining this component unless it's high crash volume.

That's a good point. This is Fennec FxA code and is probably not reused in Fenix, so this bug's priority is not critical.

(In reply to Stefan Arentz [:st3fan] from comment #18)

Why is this related to ARM builds?

This bug is not ARM-specific. This Java exception indicates that the cached FxA bundle does not exist or its JSON data is malformed. There are a few x86 reports, too.

java.lang.IllegalStateException: stateLabelString and stateString must not be null, but: (stateLabelString == null) = true and (stateString == null) = true

https://hg.mozilla.org/mozilla-central/annotate/dfdec6abcc4d/mobile/android/services/src/main/java/org/mozilla/gecko/fxa/authenticator/AndroidFxAccount.java#l753

Comment 20

[Stefan Deiac]

Hi, I tried to reproduce this issue on the latest version of Nightly 68.0a1 (2019-03-27) using Google Pixel (Android 9) using the information from comment 14 and comment 16 but I wasn't able.

Comment 21

[Marcia Knous [:marcia]]

This crash is still in 68, but the volume is extremely low. Let's see what happens when 68 is released.

Comment 22

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 23

[Marcia Knous [:marcia]]

Still present in 68 betas, but volume is fairly minimal.

Comment 24

[Vasilica Tamas, QA [:vtamas]]

I was able to reproduce this crash 2 times consecutively on Fennec 68.1.1 using Google Pixel (Android 8.0.0):
bp-0c514db7-481c-4e9b-8172-939980191007
bp-0d9d45ff-c532-4021-ab3f-e23c60191007

Comment 25

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.