1543369 - Raptor cold page-load tests on Firefox android fennec

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Add Raptor support to run the tp6m page-load test suites on the android fennec app in cold-load mode

Comment 1

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1543369 Patch for fixing fennec try builds. ONLY FOR GATHERING RESULTS. Can't be landed.

Comment 2

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=695d0d116328bb1a6c2b4027cc5586d68c7b8f27

Comment 3

[Robert Wood [:rwood]]

I don't think we should expand the cold page-load tests yet until we determine exactly what cold page-load process to use, i.e. see Bug 153794.

Comment 4

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #3)

I don't think we should expand the cold page-load tests yet until we determine exactly what cold page-load process to use, i.e. see Bug 153794.

Bug typo, see Bug Bug 1537941.

Comment 5

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #4)

(In reply to Robert Wood [:rwood] from comment #3)

I don't think we should expand the cold page-load tests yet until we determine exactly what cold page-load process to use, i.e. see Bug 153794.

Bug typo, see Bug Bug 1537941.

As per discussion on IRC we will go ahead with the current page-load implementation for now, as Bug 1537941 won't be resolved quickly.

Comment 6

[Robert Wood [:rwood]]

We are going to land the Raptor test definitions and taskcluster configs for cold page-load on Fenix first (Bug 1545175), and then we will mirror those test definitions here for Fennec.

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1543369 Raptor cold page-load tests on Firefox android fennec

Comment 8

[Robert Wood [:rwood]]

Note: Raptor support for cold page-load tests for Fennec already exists; this bug is for adding the actual Raptor test INIs/taskcluster configs to turn this on in production where desired.

Comment 9

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Rob, I am working on bug 1549669 which needs to run every warm/cold test from raptor apps (fennec, geckoview, refbrow, fenix) to make sure there's no more regression. How can we synchronize with the code? I assume you only have locally the tasks you picked up from me.

Comment 10

[Robert Wood [:rwood]]

(In reply to Alexandru Ionescu from comment #9)

Rob, I am working on bug 1549669 which needs to run every warm/cold test from raptor apps (fennec, geckoview, refbrow, fenix) to make sure there's no more regression. How can we synchronize with the code? I assume you only have locally the tasks you picked up from me.

Sorry Alex I don't understand your question... I'm not working on this (I did try pushes to get numbers on Fenix v64 if that's what you mean, but they were just try pushes and not anything landed).

If you need fennec configs to test out your fix in bug 1549669 then you could make the patch here for the fennec configs/tests, make a new bookmark, apply your fennec configs/tests patch from this bug, and also apply your fix in bug 1549669 and push that to try in one push?

Comment 11

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1543369 Raptor cold page-load tests on Firefox android fennec

Comment 12

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Pushed this try build https://treeherder.mozilla.org/#/jobs?repo=try&revision=f50524475413560319940172f95f2cb82dfdc220&selectedJob=246613944
tp6m-1-fennec-cold on G5: my test passed but failed due to timeout... see the very bottom of this log https://taskcluster-artifacts.net/falXyZgBTTqz35B3RHDKIQ/0/public/logs/live_backing.log

Same test, on P2 Aarch64:
15:01:37 INFO - raptor-main finished
15:01:37 ERROR - Traceback (most recent call last):
15:01:37 INFO - File "/builds/task_1557932368/workspace/build/tests/raptor/raptor/raptor.py", line 1219, in <module>
15:01:37 INFO - main()
15:01:37 INFO - File "/builds/task_1557932368/workspace/build/tests/raptor/raptor/raptor.py", line 1182, in main
15:01:37 INFO - raptor.run_test(next_test, timeout=int(next_test['page_timeout']))
15:01:37 INFO - File "/builds/task_1557932368/workspace/build/tests/raptor/raptor/raptor.py", line 962, in run_test
15:01:37 INFO - self.run_test_teardown()
15:01:37 INFO - File "/builds/task_1557932368/workspace/build/tests/raptor/raptor/raptor.py", line 181, in run_test_teardown
15:01:37 INFO - self.check_for_crashes()
15:01:37 INFO - File "/builds/task_1557932368/workspace/build/tests/raptor/raptor/raptor.py", line 1102, in check_for_crashes
15:01:37 INFO - remote_dir = posixpath.join(self.device_profile, 'minidumps')
15:01:37 INFO - AttributeError: 'RaptorAndroid' object has no attribute 'device_profile'
15:01:37 INFO - certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.
15:01:37 ERROR - Return code: 1
15:01:37 WARNING - setting return code to 1

Comment 13

[Bob Clary [:bc] (inactive)]

On successful test runs I see 30 * pausing 30 seconds to let browser settle... * messages which is 15 minutes of waiting. If you are going to run so many iterations of the cold tests you need to adjust the max run time to account for the waiting.

Please, Please Someone needs to take care of the test tear down so that it doesn't access properties that were not completely initialized, like device_profile, if the test fails prematurely before all of the properties are defined.

Comment 12 is mitmprox failure: Bug 1549209, Bug 1549882

Comment 14

[Robert Wood [:rwood]]

Wow yeah so fennec is even slower, so the jobs are exceeding the max time we have set (not the page cycle timeout). We can't really reduce the browser cycles down b/c of data noise.

Since this is only landing for try only (or is this supposed to be on central and try??) I'd say increase the overall taskcluster time limit for these jobs is fine. If you look at the logs they are very close to being finished the last browser cycle in most cases. So I'd suggest setting:

max-run-time: 2700

You'll need to do that for each job / in each raptor.yml section you're adding because we don't want to change the default (1800) at the top.

ps. If you get a 'Task aborted - max run time exceeded' error please don't retrigger the job, it will still get that same error in most cases so retriggers are not necessary.

Comment 15

[Alexandru Ionescu (needinfo me) [:alexandrui]]

:rwood, this is supposed to be scheduled on CI actually, to get a comparison of the pageload fennec v64 vs fenix.

Comment 16

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Latest try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f5992dd0e51d20daa1a023bd098020dc99afbfe4

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2f20a0d737e1
Raptor cold page-load tests on Firefox android fennec r=Bebe

Comment 18

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Same as https://bugzilla.mozilla.org/show_bug.cgi?id=1546323#c8 for bing-restaurants. I only got to land this. I can file a bug for disabling this test.

Comment 19

[Robert Wood [:rwood]]

(In reply to Alexandru Ionescu from comment #18)

Same as https://bugzilla.mozilla.org/show_bug.cgi?id=1546323#c8 for bing-restaurants. I only got to land this. I can file a bug for disabling this test.

This bug is for fennec (latest) not fennec v64, so not sure what you mean here. If this is a serious intermittent (on latest fennec) then you may want to consider disabling this particular test - depends on the frequency, your call. Thanks!

Comment 20

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Oh, sorry, it's the latest. Must have mixed the up a bit.

Comment 21

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/2f20a0d737e1