1645181 - Conditioned profiles for Geckoview has some issues

Status: NEW

(Testing :: Condprofile, defect, P3)

Description

[Greg Mierzwinski [:sparky]]

Opening this bug since there seems to be some issues with the conditioned profiles and it's not clear what may be causing them.

On Raptor-webext, geckoview is perma-failing with the conditioned-profiles so we'll have to temporarily disable them: https://bugzilla.mozilla.org/show_bug.cgi?id=1645135

While on Browsertime Fennec68, I am hitting some failures where the webpage load starts but nothing ever gets displayed on Firefox (see the videos, I see the same locally on P2 and ) - we'll be disabling conditioned-profiles for fennec when we land the migration: https://treeherder.mozilla.org/#/jobs?repo=try&revision=15bce2e2554e86588637851fb4a4f42d54bd2d63

Comment 1

[Tarek Ziadé (:tarek)]

i took the last working conditioned profiles that worked and one of the latest and compared them.

1/ profile used in last working run: https://firefoxci.taskcluster-artifacts.net/S-JFqGxRTpGoeF9OA2lPZg/0/public/condprof/profile-p2_aarch64-geckoview_example-settled-default.tgz

2/ profile used in broken run : https://firefoxci.taskcluster-artifacts.net/SXwRV4MASvaoXebikEAojA/0/public/condprof/profile-p2_aarch64-geckoview_example-settled-default.tgz

The only difference I can see is that the one that does not work contains an extra folder, called "extension-store" that contains a k-v database
that's used here: https://searchfox.org/mozilla-central/source/toolkit/components/extensions/ExtensionPermissions.jsm#59

If you have a local P2 environment, could you try running profile 2/ by removing its "extension-store" directory first. It it does work we can
focus our investigation in its area

Comment 2

[Greg Mierzwinski [:sparky]]

:tarek, you found it! After removing the extensions-store directory I was able to run the tests again with the conditioned-profile.

This sounds like a geckoview/fennec68 regression then right?

Comment 3

[Tarek Ziadé (:tarek)]

Could be! Let's ask James

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

Agi added the new extension store, so over to him.

Comment 5

[[ex-Mozilla] Agi Sferro | :agi]

I have a few questions here:

• is this a problem with GeckoView from mozilla-central failing to connect to the raptor extension, or is it a problem with Fennec?
• how do we generate this profile? do we share it between Fennec and GV?
• Are there drive restrictions on the device, I see a few I/O errors on the logs, not sure if they're related or not:

06-09 05:23:20.142  6178  6178 I org.mozilla.geckoview_example: type=1400 audit(0.0:60): avc: denied { setattr } for comm=4267494F546872656164506F6F6C20 name="data.mdb" dev="sda45" ino=1441921 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0 tclass=file permissive=1

• How do we install the raptor extension? can I have a link to the code that does that?
• How can I run this locally? I tried running ./mach raptor --test raptor-tp6m-1 --app=geckoview --binary="org.mozilla.geckoview_example" and it fails with

18:14:20    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
18:14:20    ERROR -   File "/Users/asferro/workspace/mozilla-central/testing/mozharness/mozharness/base/script.py", line 2083, in run_action
18:14:20    ERROR -     method(action, success=success and self.return_code == 0)
18:14:20    ERROR -   File "/Users/asferro/workspace/mozilla-central/testing/mozharness/mozharness/base/python.py", line 515, in _start_resource_monitoring
18:14:20    ERROR -     self.activate_virtualenv()
18:14:20    ERROR -   File "/Users/asferro/workspace/mozilla-central/testing/mozharness/mozharness/base/python.py", line 442, in activate_virtualenv
18:14:20    ERROR -     exec(open(activate).read(), dict(__file__=activate))
18:14:20    ERROR - IOError: [Errno 2] No such file or directory: u'/Users/asferro/workspace/mozilla-central/objdir/testing/raptor-venv/bin/activate_this.py'
18:14:20     INFO - [mozharness: 2020-06-12 18:14:20.731471Z] Finished create-virtualenv step (success)
18:14:20    FATAL - Aborting due to failure in post-action listener.
18:14:20    FATAL - Running post_fatal callback...
18:14:20    FATAL - Exiting -1
1

Comment 6

[Tarek Ziadé (:tarek)]

Hey Agi, I'll answer to some of your questions, Greg can fill in,

This problem is not related to raptor directly, but rather to the "conditioned profile", which is a job that runs in the ci to create reusable pre-fillede profiles. Here's an example of a 'conditioned profile' used by Raptor: https://firefox-ci-tc.services.mozilla.com/tasks/auFYB6u1ToiEdpl2pukw7Q

it does the follwoing:

• downloads a profile from yesterday
• starts the browser with that profile
• leave it open for a few minutes
• close it
• upload the profile as an artifact (and that will be the one used in step #1)

this process of opening/closing the browser with the profile containing the new dir works, which means the issue occurs only when
Raptor is using it. So it could be a pref difference or how the raptor extension is installed maybe.

do we share it between Fennec and GV?

no, we have one for each "browser"

Comment 7

[[ex-Mozilla] Agi Sferro | :agi]

In the broken conditioned profile the lmdb database present at extension-store seems to be corrupted, which would explain why something is broken. I'll do some manual testing.

Comment 8

[Greg Mierzwinski [:sparky]]

(In reply to Agi Sferro | :agi | ⏰ PST | he/him from comment #5)

I have a few questions here:

• is this a problem with GeckoView from mozilla-central failing to connect to the raptor extension, or is it a problem with Fennec?

On geckoview, the browser seems to fail to install/start the raptor extension. On Fennec, we test with browsertime now so there is no webextension but the profile causes the browser to open to the standard homepage view (when it should go to an about:blank-like page) and when we load a URL the page never refreshes and it continues to show the homepage view.

• how do we generate this profile? do we share it between Fennec and GV?

I think Tarek answered this one (they are two different profiles, we don't share them).

• Are there drive restrictions on the device, I see a few I/O errors on the logs, not sure if they're related or not:

06-09 05:23:20.142  6178  6178 I org.mozilla.geckoview_example: type=1400 audit(0.0:60): avc: denied { setattr } for comm=4267494F546872656164506F6F6C20 name="data.mdb" dev="sda45" ino=1441921 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0 tclass=file permissive=1

There are no drive restrictions that I know of but for the moment, we have to use a rooted phone to test. bc's been working on removing this restriction.

• How do we install the raptor extension? can I have a link to the code that does that?

https://searchfox.org/mozilla-central/source/testing/raptor/raptor/webextension/base.py#196-211

• How can I run this locally? I tried running ./mach raptor --test raptor-tp6m-1 --app=geckoview --binary="org.mozilla.geckoview_example" and it fails with

18:14:20    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
18:14:20    ERROR -   File "/Users/asferro/workspace/mozilla-central/testing/mozharness/mozharness/base/script.py", line 2083, in run_action
18:14:20    ERROR -     method(action, success=success and self.return_code == 0)
18:14:20    ERROR -   File "/Users/asferro/workspace/mozilla-central/testing/mozharness/mozharness/base/python.py", line 515, in _start_resource_monitoring
18:14:20    ERROR -     self.activate_virtualenv()
18:14:20    ERROR -   File "/Users/asferro/workspace/mozilla-central/testing/mozharness/mozharness/base/python.py", line 442, in activate_virtualenv
18:14:20    ERROR -     exec(open(activate).read(), dict(__file__=activate))
18:14:20    ERROR - IOError: [Errno 2] No such file or directory: u'/Users/asferro/workspace/mozilla-central/objdir/testing/raptor-venv/bin/activate_this.py'
18:14:20     INFO - [mozharness: 2020-06-12 18:14:20.731471Z] Finished create-virtualenv step (success)
18:14:20    FATAL - Aborting due to failure in post-action listener.
18:14:20    FATAL - Running post_fatal callback...
18:14:20    FATAL - Exiting -1
1

This error is related to your local configuration - you may have to clobber/build and run mach bootstrap to solve it.

Comment 9

[Greg Mierzwinski [:sparky]]

Attachment: 1.mp4

This is what we see in fennec when it uses the conditioned profile.

Comment 10

[[ex-Mozilla] Agi Sferro | :agi]

I copied over the profile to my local build and I got this error:

06-15 12:27:05.355  6771  6789 E GeckoConsole: [JavaScript Error: "migrate error: IndeterminateBitDepth" {file: "resource://gre/modules/ExtensionPermissions.jsm" line: 66}]
06-15 12:27:05.355  6771  6789 E GeckoConsole: _init@resource://gre/modules/ExtensionPermissions.jsm:66:10

so there's definitely something wrong with the db in the profile.

Comment 11

[[ex-Mozilla] Agi Sferro | :agi]

I wonder if this is just a aarch64/x86_64 problem. I'll try in my phone.

Comment 12

[Greg Mierzwinski [:sparky]]

:tarek, could you check to see why the fennec profiles aren't working with browsertime? The fennec profiles don't have the extension-store directory so it's related to something else.

Comment 13

[[ex-Mozilla] Agi Sferro | :agi]

(In reply to Agi Sferro | :agi | ⏰ PST | he/him from comment #11)

I wonder if this is just a aarch64/x86_64 problem. I'll try in my phone.

I can reproduce with the above profile on my phone too, so not related to different architecture.

Victor do you have any idea why we would be getting a IndeterminateBitDepth error in Comment 10? The profile in Comment 1 has the rkv database in the extension-store folder.

We should probably also handle that in our code and not fail like that.

Comment 14

[Greg Mierzwinski [:sparky]]

Making this bug for geckoview only - tarek, the ni? is moved to bug 1645885.

Comment 15

[Greg Mierzwinski [:sparky]]

Duplicate message.

Comment 16

[[ex-Mozilla] Agi Sferro | :agi]

So I'm fairly confident now that the problem is some sort of permission error with the device:

06-09 05:23:20.142  6178  6178 I org.mozilla.geckoview_example: type=1400 audit(0.0:60): avc: denied { setattr } for comm=4267494F546872656164506F6F6C20 name="data.mdb" dev="sda45" ino=1441921 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0 tclass=file permissive=1
06-09 05:23:20.145  6178  6197 I GeckoConsole: 1591680200145	addons.xpi-utils	WARN	addMetadata: Add-on [e-mail] is invalid: I/O error: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }

If you see this log, the code is trying to write to disk and getting a PermissionDenied error. The problem has started now because previously the corresponding write to disk was done in the background and nothing waited on it. Could it be that some files are owned by root by mistake? that would cause this to fail.

Comment 17

[Greg Mierzwinski [:sparky]]

:bc, is it possible for us to have file permission issues on the P2 phones for where the profile is stored? The location of where the profile is stored was changed recently and here geckoview added a new folder extension-store to the profile.

Comment 18

[Bob Clary [:bc] (inactive)]

I'll leave the NI open to remind me. We recently moved the raptor profile from the sdcard to /data/local/tmp which prevents issues with scoped storage and allows real, modifiable permissions and it may be the root of this issue. I'm progressing on allowing unrooted devices but am having issues with particular versions of Android which I am investigating now. I'm out part of the day today and will take a closer look this evening when I get back.

Comment 19

[Bob Clary [:bc] (inactive)]

My rootless patch didn't help. See the "after" push. It looks like android is killing the browser?

Comment 20

[Victor Porof [:vporof][:vp]]

(In reply to Agi Sferro | :agi | ⏰ PST | he/him from comment #13)

(In reply to Agi Sferro | :agi | ⏰ PST | he/him from comment #11)

I wonder if this is just a aarch64/x86_64 problem. I'll try in my phone.

I can reproduce with the above profile on my phone too, so not related to different architecture.

Victor do you have any idea why we would be getting a IndeterminateBitDepth error in Comment 10? The profile in Comment 1 has the rkv database in the extension-store folder.

We should probably also handle that in our code and not fail like that.

I recall there used to be issues when using RKV with LMDB, where database files weren't usable across architectures, so stores created on 32bit must be manually migrated to 64bit and vice versa as they aren't compatible with each other. See bug 1557161 and this explanation[0].

I'm thinking Nan might also have some more info here, as he's worked with myk on this back when it was an issue.

So you might get this when migrating between 32bit and 64bit, and the error is thrown here[1]. Try checking the list of limitations here[2] and see if any of them apply.

Note that all of this will go away once we switch kvstore away from LMDB and to safe-mode.

[0] https://github.com/mozilla/rkv/blob/abd08f409aac4d8ff3d537e5ad8887590c44171d/src/migrate.rs#L11
[1] https://github.com/mozilla/rkv/blob/abd08f409aac4d8ff3d537e5ad8887590c44171d/src/migrate.rs#L526
[2] https://github.com/mozilla/rkv/blob/abd08f409aac4d8ff3d537e5ad8887590c44171d/src/migrate.rs#L29

Comment 21

[Greg Mierzwinski [:sparky]]

EDIT: Nevermind, wrong platform was being tested.

Comment 22

[Greg Mierzwinski [:sparky]]

So I ran a test where I forced a complete refresh of the profile, and the tests are now passing on the P2 device: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=243c322711a8db7788de9f24a3ea3fb925f17c10

Based on this, the issue with geckoview occurs when existing profiles are updated with the new extension-store folder.

Comment 23

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:tarek, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 24

[BugBot [:suhaib / :marco]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:sparky, since the bug doesn't have a severity set, could you please set the severity or close the bug?

For more information, please visit auto_nag documentation.