1601729 - Leanplum Open URL deeplink is unpredictable when multiple versions of Firefox are installed

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: General, defect, P1)

Description

[laustin]

The push notification to Android Lapsed and Dormant users sent on December 05th failed to exxhibit the expected behavior of resolving a Firefox blog post (link dependant on the language) on click through.

Reports of both the EN and IT links indicated users were sent to resource:///localization/

Links in question:
EN: firefox://open?url=https://blog.mozilla.org/firefox/block-cryptominers-with-firefox/&uid={{User ID}}
IT: firefox://open?url=https://blog.mozilla.org/firefox/it/firefox-ha-due-modi-per-mettere-al-sicuro-il-tuo-shopping-natalizio/&uid={{User ID}}

These links follow the accepted format. All links were extensively QA'ed by the Marketing team. They have been unable to reproduce the reported behavior on available Android devices.

Full incident report can be found here: https://docs.google.com/document/d/10lvbxjvwkpKfv4ltR0GGLtlsJOWBJnpDZOENuRudI4E/edit?usp=sharing

The team suspects there is an issue isolated to the One Plus device, as both reports came from this device.

Comment 1

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

After a lot of testing and investigation we have found one concrete issue that is problematic:

If you have multiple copies of Firefox installed, like a combination of Production, Beta and Nightly, the firefox:// deeplinks do not work properly:

• The notification always properly arrives
• In the failure case, tapping the notification results in Firefox Production being opened, but the deeplink not being processed
• The behaviour is random. For some people it works, for others it does not. It may depend on an internal ordering Android does, or maybe the order in which you installed the different Firefox flavors.
• Failure happens on both firefox://open?url=... links and links that go to a specific section of the app.

In the failure case, we observed we observed the following intent being processed after tapping the notification:

12-13 08:55:15.355  1475  4710 I ActivityTaskManager: START u0 {act=android.intent.action.VIEW 
   dat=firefox://open?url=https://blog.mozilla.org/firefox/block-cryptominers-with-firefox/&uid=b5a439f9-70cb-459e-b93e-1cfe3d3014c2
   flg=0x10000000 pkg=org.mozilla.firefox_beta cmp=org.mozilla.firefox_beta/.App} from uid 10231

Note that although this was a Leanplum campaign sent to a Device ID registered as Firefox Production, the intent mentions pkg=org.mozilla.firefox_beta and cmp=org.mozilla.firefox_beta/.App.

After removing Firefox Beta from the device, the campaign was run again and the following Intent was ultimately processed:

12-13 09:06:29.843  1475 11076 I ActivityTaskManager: START u0 {act=android.intent.action.VIEW 
   dat=firefox://open?url=https://blog.mozilla.org/firefox/block-cryptominers-with-firefox/&uid=b5a439f9-70cb-459e-b93e-1cfe3d3014c2
   flg=0x10000000 pkg=org.mozilla.firefox cmp=org.mozilla.firefox/.App} from uid 10231

This attempt also loaded a new tab with the URL from the deeplink.

From a Leanplum SDK perspective, it looks like opening the deeplink is handled in:

https://searchfox.org/mozilla-esr68/source/mobile/android/thirdparty/com/leanplum/messagetemplates/OpenURL.java

Can we audit that code and find out if it does the correct thing?

Addition thoughts:

• Could this have anything to do with the shared user id between Fennec Flavors

What is a good mitigation for this issue? Can we fix this or should we introduce multiple URL schemes like firefox://, firefox-nightly:// and 'firefox-beta://`. (This is a possibility but I would prefer to not do that if we can instead fix this in some other way.)

Comment 2

[Sorina Florean [:sflorean]]

We investigated this bug using OnePlus 5T (Android 9) and we encountered different behaviors. Since we don't have access to the Leanplum dashboard for Firefox release our testing was based on Beta and Nightly and we find different behaviors.

• privacy menu opened after the push notification was tapped (we couldn't reproduce the second time);
• push notification not received if both Beta and Nightly were targeted
• push notification not received at all if we performed "send preview".

If we can help with other details, let us know.

Comment 3

[Vlad Baicu]

Attachment: Bug 1601729 - Change deeplink scheme for nightly.r=AndreiLazar

Comment 4

[Vlad Baicu]

Attachment: Bug 1601729 - Change deeplink scheme for beta.r=AndreiLazar

Comment 5

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

I filed bug 1606937 to request a FENNEC_BETA define just like we have for FENNEC_NIGHTLY.

Comment 6

[Petru-Mugurel Lingurar [:petru]]

Attachment: Bug 1601729 - Use different deeplink schemes depending on app channel; r?VladBaicu,RyanVM

This would allow us to send targeted push messages to specific applications
and so protecting against the case in which the user has more apps installed,
all trying to consume the just received push message which may not be intended
for the one that actually got to consume it.

Comment 7

[Petru-Mugurel Lingurar [:petru]]

Reworked the branching in the above patch.
Now we'll have 4 possible schemas, one for each release channel:

• firefox-dev for local, developer builds. Probably this wouldn't be used. Maybe just for internal tests.
• firefox-nightly for official nightly builds.
• firefox-beta for official beta builds.
• firefox for official release builds. This is the old schema previously used by all channels. Using this still in order to not break compatibility with previously set campaigns.

If we want to target users of any release channel we must build our deeplink to use the exact schema from any of the above.

‎
This deeplink schemas are dynamically set at compile time by checking for specific flags from mozconfig following this branching strategy:

• no MOZILLA_OFFICIAL flag -> local dev build -> firefox-dev
• MOZILLA_OFFICIAL and FENNEC_NIGHTLY flags -> official nightly build -> firefox-nightly
• MOZILLA_OFFICIAL and FENNEC_BETA flags -> official beta build -> firefox-beta
• MOZILLA_OFFICIAL (no FENNEC_NIGHTLY or FENNEC_BETA flags) -> official release build -> firefox

Comment 8

[Petru-Mugurel Lingurar [:petru]]

Attachment: Bug 1601729 - Validate the new schemes in code; r?VladBaicu

Now that we have more deeplink schemes we need to make sure these are not
rejected when validating a received Intent as a deeplink.

Comment 9

[Petru-Mugurel Lingurar [:petru]]

Tested locally all configurations for mozconfig and saw all 4 expected schemes in manifest as expected.
Also here is a try build for nightly and beta containing my patches which shows the "firefox-beta" for the beta apk and "firefox-nightly" for the aurora apk
https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa9640624154947034e7adb2dbf5743ff9b921f6

Comment 10

[Petru-Mugurel Lingurar [:petru]]

Comment on attachment 9119143 [details]
Bug 1601729 - Use different deeplink schemes depending on app channel; r?VladBaicu,RyanVM

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Change needed for highly important marketing campaigns
• User impact if declined: Misbehaving deeplinks in marketing campaigns
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Small change
• String or UUID changes made by this patch:

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9119143 [details]
Bug 1601729 - Use different deeplink schemes depending on app channel; r?VladBaicu,RyanVM

Needed for Fennec->Fenix migration work. Approved for 68.5b3.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/27375dd57676
https://hg.mozilla.org/releases/mozilla-esr68/rev/f92856bebbf1

Comment 13

[Cristina Coroiu [:ccoroiu]]

Backed out changeset f92856bebbf1 (bug 1601729) for findbugs bustage at withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION

Backout: https://hg.mozilla.org/releases/mozilla-esr68/rev/92fc182ab07a770ec60c37cdc6eba1e7629a3057

Failure push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr68&revision=3edf801b9317700e2937f57161e15735e7dad587

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284460377&repo=mozilla-esr68&lineNumber=3564

[task 2020-01-10T21:34:59.609Z] 21:34:59 INFO - BUILD FAILED in 41s
[task 2020-01-10T21:34:59.609Z] 21:34:59 INFO - 48 actionable tasks: 1 executed, 47 up-to-date
[task 2020-01-10T21:35:00.123Z] 21:35:00 INFO - TinderboxPrint: report<br/><a href="https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RZaM0BSOTTiWmYsjYEu3qA/runs/0/artifacts/public/android/findbugs/findbugs-withoutGeckoBinariesDebug-output.html">HTML withoutGeckoBinariesDebug report</a>, visit "Inspect Task" link for details
[task 2020-01-10T21:35:00.123Z] 21:35:00 INFO - TinderboxPrint: report<br/><a href="https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RZaM0BSOTTiWmYsjYEu3qA/runs/0/artifacts/public/android/findbugs/findbugs-withoutGeckoBinariesDebug-output.xml">XML withoutGeckoBinariesDebug report</a>, visit "Inspect Task" link for details
[task 2020-01-10T21:35:00.124Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | android-findbugs | Findbugs found issues in the project. See the report at: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RZaM0BSOTTiWmYsjYEu3qA/runs/0/artifacts/public/android/findbugs/findbugs-withoutGeckoBinariesDebug-output.html
[task 2020-01-10T21:35:00.124Z] 21:35:00 INFO - SUITE-START | android-findbugs | withoutGeckoBinariesDebug
[task 2020-01-10T21:35:00.124Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | org.mozilla.gecko.deeplink.DeepLinkContract
[task 2020-01-10T21:35:00.124Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | <BugInstance abbrev="MS" category="MALICIOUS_CODE" priority="1" rank="16" type="MS_MUTABLE_COLLECTION">
[task 2020-01-10T21:35:00.124Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | <Class classname="org.mozilla.gecko.deeplink.DeepLinkContract">
[task 2020-01-10T21:35:00.124Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | <SourceLine classname="org.mozilla.gecko.deeplink.DeepLinkContract" end="16" sourcefile="DeepLinkContract.java" sourcepath="org/mozilla/gecko/deeplink/DeepLinkContract.java" start="12" />
[task 2020-01-10T21:35:00.124Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | </Class>
[task 2020-01-10T21:35:00.124Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | <Field classname="org.mozilla.gecko.deeplink.DeepLinkContract" isStatic="true" name="DEEP_LINK_SCHEMES" signature="Ljava/util/List;">
[task 2020-01-10T21:35:00.125Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | <SourceLine classname="org.mozilla.gecko.deeplink.DeepLinkContract" sourcefile="DeepLinkContract.java" sourcepath="org/mozilla/gecko/deeplink/DeepLinkContract.java" />
[task 2020-01-10T21:35:00.125Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | </Field>
[task 2020-01-10T21:35:00.125Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | <SourceLine classname="org.mozilla.gecko.deeplink.DeepLinkContract" end="16" endBytecode="27" sourcefile="DeepLinkContract.java" sourcepath="org/mozilla/gecko/deeplink/DeepLinkContract.java" start="16" startBytecode="27" />
[task 2020-01-10T21:35:00.125Z] 21:35:00 INFO - TEST-UNEXPECTED-FAIL | withoutGeckoBinariesDebug:MS_MUTABLE_COLLECTION | </BugInstance>
[task 2020-01-10T21:35:00.125Z] 21:35:00 INFO - SUITE-END | android-findbugs | withoutGeckoBinariesDebug
[task 2020-01-10T21:35:00.147Z] 21:35:00 ERROR - Return code: 1
[task 2020-01-10T21:35:00.147Z] 21:35:00 ERROR - 1 not in success codes: [0]
[task 2020-01-10T21:35:00.147Z] 21:35:00 WARNING - setting return code to 2
[task 2020-01-10T21:35:00.147Z] 21:35:00 FATAL - Halting on failure while running ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', 'mach', '--log-no-times', 'android', 'findbugs']
[task 2020-01-10T21:35:00.147Z] 21:35:00 FATAL - Running post_fatal callback...
[task 2020-01-10T21:35:00.147Z] 21:35:00 FATAL - Exiting 2
[task 2020-01-10T21:35:00.147Z] 21:35:00 INFO - [mozharness: 2020-01-10 21:35:00.147934Z] Finished build step (failed)
[task 2020-01-10T21:35:00.148Z] 21:35:00 INFO - Running post-run listener: _parse_build_tests_ccov
[task 2020-01-10T21:35:00.148Z] 21:35:00 INFO - Running post-run listener: _shutdown_sccache
[task 2020-01-10T21:35:00.148Z] 21:35:00 INFO - Running post-run listener: _summarize
[task 2020-01-10T21:35:00.148Z] 21:35:00 ERROR - # TBPL FAILURE #

Comment 14

[Petru-Mugurel Lingurar [:petru]]

Thanks Cristina.
Updated the patch to account for this. Can you please merge it again?

Comment 15

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/be5614869c3a

Comment 16

[Petru-Mugurel Lingurar [:petru]]

I see the recent nightly and beta builds from our repo - http://archive.mozilla.org/pub/mobile/ already using the new schemes, firefox-nightly and firefox-beta.
Please help test that with having multiple Firefox apps installed Leanplum deeplinks using a particular scheme open the intended app.
I tested locally with adb

adb shell am start -a android.intent.action.VIEW -d "firefox://fxa-signin"
adb shell am start -a android.intent.action.VIEW -d "firefox-nightly://fxa-signin"
adb shell am start -a android.intent.action.VIEW -d "firefox-beta://fxa-signin"

and saw that depending on the deeplink scheme the right application is opened.

Comment 17

[Sorina Florean [:sflorean]]

Verified as fixed on the latest 68 ESR - Nightly and Beta build. Devices:

• Google Pixel 2 (Android Q)
• OnePlus 5T (Android 9)