1552519 - For geckoview browser apps startup use local configuration file instead of command line args

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Robert Wood [:rwood]]

Long story short, Bug 1533385 added support for setting geckoview browser prefs/env vars using a local on-device config file. The current method of using the command line arg/intent to pass in the profile [0] will be made obsolete.

This blocks [1] so we should make this Raptor change soon.

[0] https://searchfox.org/mozilla-central/rev/94c6b5f06d2464f6780a52f32e917d25ddc30d6b/testing/raptor/raptor/raptor.py#889

[1] https://github.com/mozilla-mobile/fenix/issues/1350

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a task, but please change it back in case of error.

Comment 2

[Robert Wood [:rwood]]

From https://hg.mozilla.org/integration/autoland/rev/b978367ef7bb:

The configuration file format is YAML and looks like:

prefs:
  foo.bar.boolean: true
  foo.bar.string: "string"
  foo.bar.int: 500
env:
  MOZ_LOG: nsHttp:5
args: [--marionette]

By default, if the consuming App is debuggable, GeckoView will read
configuration from /data/local/tmp/$PACKAGE-geckoview-config.yaml at
startup.

Comment 3

[Robert Wood [:rwood]]

I have this working locally now on my GP2 with the geckoview example app. Raptor writes this on the device:

walleye:/data/local/tmp # cat org.mozilla.geckoview_example-geckoview-config.yaml
args:
- --profile
- /sdcard/raptor/profile
- use_multiprocess
- true
env:
LOG_VERBOSE: 1
R_LOG_LEVEL: 6
walleye:/data/local/tmp #

Comment 4

[Robert Wood [:rwood]]

Attachment: Bug 1552519 - For geckoview browser apps startup use local configuration file instead of command line args

Comment 5

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7fc54e748eb04f1f22ced84682a63cbaf25e7242

Comment 6

[Robert Wood [:rwood]]

05-23 13:40:26.518 20967 20967 I GeckoRuntime: Adding debug configuration from: /data/local/tmp/org.mozilla.geckoview_example-geckoview-config.yaml
05-23 13:40:26.518 20967 20967 D GeckoDebugConfig: Adding environment variables from debug config: {LOG_VERBOSE=1, R_LOG_LEVEL=6}
05-23 13:40:26.518 20967 20967 D GeckoDebugConfig: Adding arguments from debug config: [--profile, /sdcard/raptor/profile, use_multiprocess, true]
05-23 13:40:26.518 20967 20967 D GeckoThread: State changed to LAUNCHED

Comment 7

[Robert Wood [:rwood]]

Also try push with update (added unit test) and jobs for reference browser and fenix:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d3c5f5405386e5b4a2c066844156c0b162f5bde

Comment 8

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #7)

Also try push with update (added unit test) and jobs for reference browser and fenix:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d3c5f5405386e5b4a2c066844156c0b162f5bde

Looks like the *-config.yaml is not being picked up by reference browser or fenix. The file is being written (i.e. for Fenix):

3:47:10 INFO - raptor-utils writing {'args': ['--profile', '/sdcard/raptor/profile', 'use_multiprocess', True], 'env': {'R_LOG_LEVEL': 6, 'LOG_VERBOSE': 1}} to /tmp/tmp72nyCD/org.mozilla.fenix.raptor-geckoview-config.yaml
03:47:10 INFO - raptor-main copying /tmp/tmp72nyCD/org.mozilla.fenix.raptor-geckoview-config.yaml to device: data/local/tmp/org.mozilla.fenix.raptor-geckoview-config.yaml
03:47:10 INFO - adb command_output: adb -s HT7B21A02889 wait-for-device push /tmp/tmp72nyCD/org.mozilla.fenix.raptor-geckoview-config.yaml data/local/tmp/org.mozilla.fenix.raptor-geckoview-config.yaml, timeout: None, timedout: None, exitcode: 0, output: /tmp/tmp72nyCD/org.mozilla.fenix.raptor-geckoview-config.yaml: 1 file pushed. 0.0 MB/s (108 bytes in 0.004s)

:nalexander, :csadilek, any ideas why this isn't working for Fenix and Reference browser? It works great for geckoview example. Maybe I need a different config.yaml filename for those apps? Thanks!

Comment 9

[Nick Alexander :nalexander [he/him]]

(In reply to Robert Wood [:rwood] from comment #8)

(In reply to Robert Wood [:rwood] from comment #7)

Also try push with update (added unit test) and jobs for reference browser and fenix:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d3c5f5405386e5b4a2c066844156c0b162f5bde

Looks like the *-config.yaml is not being picked up by reference browser or fenix. The file is being written (i.e. for Fenix):

3:47:10 INFO - raptor-utils writing {'args': ['--profile', '/sdcard/raptor/profile', 'use_multiprocess', True], 'env': {'R_LOG_LEVEL': 6, 'LOG_VERBOSE': 1}} to /tmp/tmp72nyCD/org.mozilla.fenix.raptor-geckoview-config.yaml
03:47:10 INFO - raptor-main copying /tmp/tmp72nyCD/org.mozilla.fenix.raptor-geckoview-config.yaml to device: data/local/tmp/org.mozilla.fenix.raptor-geckoview-config.yaml
03:47:10 INFO - adb command_output: adb -s HT7B21A02889 wait-for-device push /tmp/tmp72nyCD/org.mozilla.fenix.raptor-geckoview-config.yaml data/local/tmp/org.mozilla.fenix.raptor-geckoview-config.yaml, timeout: None, timedout: None, exitcode: 0, output: /tmp/tmp72nyCD/org.mozilla.fenix.raptor-geckoview-config.yaml: 1 file pushed. 0.0 MB/s (108 bytes in 0.004s)

:nalexander, :csadilek, any ideas why this isn't working for Fenix and Reference browser? It works great for geckoview example. Maybe I need a different config.yaml filename for those apps? Thanks!

Yes -- the Raptor APKs (and presumably the r-b APKs) are release builds, which don't read the configuration by default. See https://mozilla.github.io/geckoview/tutorials/automation.html#enabling-reading-configuration-from-a-file-for-release-builds. However, what we really want is Bug 1554256, which is how I envisioned addressing this but have not yet gotten to.

Comment 10

[Nick Alexander :nalexander [he/him]]

Clearing NI for both of us.

Comment 11

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2bf8a850c8de9d789768c898c1de95f307d7fa0d

Comment 12

[Robert Wood [:rwood]]

Attachment: Bug 1552519 - For geckoview browser apps startup use local configuration file instead of command line args

Comment 13

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf0d3f3ef29b5be74b11ac30c69a956d26788db5

Comment 14

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ef6b74f9ddb8bc8269a3a16e68a864a554137304

Comment 15

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=5e29e193ae5803f0ffcd7f20a181e7a51009f413

Comment 16

[Robert Wood [:rwood]]

With the addition of setting the debug-app flag (so release apk's will read the config.yml):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5d2267dc00077888288a457fbbdf22f3d944f43b

Comment 17

[Robert Wood [:rwood]]

Working on gve, fennec, and refbrow, but for some reason not fenix:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5d2267dc00077888288a457fbbdf22f3d944f43b

Comment 18

[Robert Wood [:rwood]]

Hey Christian,

So using the config.yaml file on android works for geckoview example and reference browser, but doesn't work for fenix. Would you mind having a quick look at my patch? As far as I can tell I am writing the file correctly. Here is a sample log out put from my latest try run [0] (the file is written but when starting Fenix it's not picking up the config.yaml settings so Raptor times out):

09:28:25 INFO - raptor-utils Info: writing {'args': ['--profile', '/sdcard/raptor/profile', 'use_multiprocess', True], 'env': {'R_LOG_LEVEL': 6, 'LOG_VERBOSE': 1}} to /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml
09:28:25 INFO - raptor-main Info: copying /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml to device: data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml
09:28:25 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell rm -r data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
09:28:26 INFO - adb command_output: adb -s HT7AT1A01275 wait-for-device push /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml, timeout: None, timedout: None, exitcode: 0, output: /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml: 1 file pushed. 0.1 MB/s (108 bytes in 0.002s)
09:28:26 INFO - adb chmod: path=data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml, recursive=True, mask=777, root=True
09:28:26 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell su -c "chmod -R 777 data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml"; echo adb_returncode=$?, timeout: None, root: True, timedout: None, exitcode: 0, output:
09:28:26 INFO - raptor-main Info: starting fenix
09:28:26 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell am force-stop org.mozilla.fenix.performancetest; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
09:28:26 INFO - adb Granting important runtime permissions to org.mozilla.fenix.performancetest
09:28:26 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell pm grant org.mozilla.fenix.performancetest android.permission.WRITE_EXTERNAL_STORAGE; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
09:28:27 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell pm grant org.mozilla.fenix.performancetest android.permission.READ_EXTERNAL_STORAGE; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
09:28:27 [taskcluster:error] Aborting task...

[0] https://treeherder.mozilla.org/#/jobs?repo=try&revision=95aae805307201b0c6e1c219fa8e0a5866fe0798

Comment 19

[Nick Alexander :nalexander [he/him]]

(In reply to Robert Wood [:rwood] from comment #18)

Hey Christian,

So using the config.yaml file on android works for geckoview example and reference browser, but doesn't work for fenix. Would you mind having a quick look at my patch? As far as I can tell I am writing the file correctly. Here is a sample log out put from my latest try run [0] (the file is written but when starting Fenix it's not picking up the config.yaml settings so Raptor times out):

09:28:25 INFO - raptor-utils Info: writing {'args': ['--profile', '/sdcard/raptor/profile', 'use_multiprocess', True], 'env': {'R_LOG_LEVEL': 6, 'LOG_VERBOSE': 1}} to /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml
09:28:25 INFO - raptor-main Info: copying /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml to device: data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml
09:28:25 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell rm -r data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
09:28:26 INFO - adb command_output: adb -s HT7AT1A01275 wait-for-device push /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml, timeout: None, timedout: None, exitcode: 0, output: /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml: 1 file pushed. 0.1 MB/s (108 bytes in 0.002s)
09:28:26 INFO - adb chmod: path=data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml, recursive=True, mask=777, root=True
09:28:26 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell su -c "chmod -R 777 data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml"; echo adb_returncode=$?, timeout: None, root: True, timedout: None, exitcode: 0, output:
09:28:26 INFO - raptor-main Info: starting fenix
09:28:26 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell am force-stop org.mozilla.fenix.performancetest; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
09:28:26 INFO - adb Granting important runtime permissions to org.mozilla.fenix.performancetest
09:28:26 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell pm grant org.mozilla.fenix.performancetest android.permission.WRITE_EXTERNAL_STORAGE; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
09:28:27 INFO - adb shell_output: adb -s HT7AT1A01275 wait-for-device shell pm grant org.mozilla.fenix.performancetest android.permission.READ_EXTERNAL_STORAGE; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
09:28:27 [taskcluster:error] Aborting task...

[0] https://treeherder.mozilla.org/#/jobs?repo=try&revision=95aae805307201b0c6e1c219fa8e0a5866fe0798

I expected this to be Bug 1554256, but I fetched the APK and it is debuggable.

So my next guess is that you're running a Fenix built against GeckoView 68 (beta), but even that is supposed to have Bug 1533385 applied.

So my guess after that is there's a legit bug here, which will be best diagnosed by substituting a local GeckoView into Fenix; or that the missing leading slash in these commands actually matters:

 09:28:26 INFO - adb command_output: adb -s HT7AT1A01275 wait-for-device push /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml data/local/tmp/org.mozilla.fenix.performancetest-geckoview-config.yaml, timeout: None, timedout: None, exitcode: 0, output: /tmp/tmpZcYeBq/org.mozilla.fenix.performancetest-geckoview-config.yaml: 1 file pushed. 0.1 MB/s (108 bytes in 0.002s)

(Note the data/....) Is the file in the correct place? If you edit the /data/local/tmp/... file manually, does it work?

Comment 20

[Robert Wood [:rwood]]

Thanks. No adding the forward-slash to the path doesn't help. On the device the config.yaml exists (in the same folder / along-side the *-config.yamls that worked with geckoview and refbrow). Here's the contents of the fenix one (same as the others):

walleye:/data/local/tmp # ls
org.mozilla.fenix.performancetest-geckoview-config.yaml
walleye:/data/local/tmp # cat org.mozilla.fenix.performancetest-geckoview-config.yaml
args:

• --profile
• /sdcard/raptor/profile
• use_multiprocess
• true
  env:
  LOG_VERBOSE: 1
  R_LOG_LEVEL: 6
  walleye:/data/local/tmp #

Comment 21

[Christian Sadilek [:csadilek]]

Nick, Rob,

Proguard strikes again! The reason this doesn't work in Fenix is that org.mozilla.gecko.util.DebugConfig gets optimized away. When parsing the .yaml file we get a java.lang.InstantiationException for this class.

I'll put up a PR with an exclusion rule shortly.

Comment 22

[Christian Sadilek [:csadilek]]

Here's a Fenix PR that fixes the problem: https://github.com/mozilla-mobile/fenix/pull/3911

Comment 23

[Robert Wood [:rwood]]

Another try push after the Fenix PR was merged:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a55f143a73c803675a4e2cd9c6176051ef2805df

Comment 24

[Robert Wood [:rwood]]

Updated. Changed fenix activity name as the org.mozilla.fenix.browser.BrowserPerformanceTestActivity will eventually be removed now that moving to on-device config.yaml. Mozbase adb.py requires an activity name, so using org.mozilla.fenix.IntentReceiverActivity so don't need to change the mozharness/mozdevice module).

https://treeherder.mozilla.org/#/jobs?repo=try&revision=17e5e3cc024fd42747ce8aad659bb5612e2478ca

Comment 25

[Robert Wood [:rwood]]

After updates (thanks for the review :sparky!):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6b567499f0836f2a5ad92e007511d7dffea40d77

Comment 26

[Robert Wood [:rwood]]

Try run in comment 25 is still queued up on android (except for the power jobs which are green); here's another in case the remaining jobs time out:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=76a3752305d6f8a2fd26bae1de5068c022a7d3e0

Comment 27

[Robert Wood [:rwood]]

Try run in comment 26 timed out; new try run:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2d38de4723b5cc5c5a3fb8d876f8cdbc11fb2fbf

Comment 28

[Robert Wood [:rwood]]

Try run in comment 27 is still pending, here's another in case ^ times out:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0a9851a13008b344ef71eb06b3bcd8112d309110

Comment 29

[Robert Wood [:rwood]]

Comment 28 run pending, here's another back-up one:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=91f526e124e805cdc04adcf7a70cf196f2095e64

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac23331d8e58
For geckoview browser apps startup use local configuration file instead of command line args r=perftest-reviewers,sparky

Comment 31

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/ac23331d8e58