Closed Bug 1529207 Opened 5 years ago Closed 5 years ago

Add new Raptor tests in tp6m-4

Categories

(Testing :: Raptor, enhancement, P1)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
enhancement

Tracking

(firefox67 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla67
Tracking Flags:
Tracking Status
firefox67 --- fixed

People

(Reporter: Bebe, Assigned: marauder)

References

Details

Attachments

(1 file)

Bug 1529207 - Add new Raptor tests in tp6m-4
47 bytes, text/x-phabricator-request
Details | Review
Blocks: 1513847
Assignee: nobody → marian.raiciof

Recording files can be found here :
https://drive.google.com/drive/folders/1GJIgJc2qW53vwsVf2ILzrgTDY9xU5sSt?usp=sharing

There are some errors when testing these files on mozilla-central's raptor:

15:34:06 INFO - raptor-output error: no raptor test results found for raptor-tp6m-cnn-geckoview
15:34:06 INFO - raptor-output error: no summarized raptor results found for raptor-tp6m-cnn-geckoview
15:34:06 INFO - raptor-control-server shutting down control server
15:34:06 INFO - raptor-main removing reverse socket connections
15:34:06 INFO - adb command_output: adb -s ZY3228PRPF wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
15:34:06 INFO - raptor-main finished
15:34:06 INFO - raptor-main TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-tp6m-cnn-geckoview
15:34:06 ERROR - Return code: 1
15:34:06 WARNING - setting return code to 1
15:34:06 INFO - Killing logcat pid 31085.
15:34:06 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.

I tested different scenarios and i believe these errors are caused by the manifest files for booking and cnn that are currently empty.

Status: NEW → ASSIGNED

I have an update on this one :

The tests are working well now on raptor from mozilla-central.

I talked to Florin a while back when i encountered some issues with raptor tests
and he told me at that time to add the URL of that website to
/mozilla-central/testing/raptor/webext/raptor/manifest.json

I did that for booking.com and cnn.com - now all is working without issues.

Tested with geckoview example app on my GP2 device and the playback works great; uploaded to tooltool and emailed the manifests.

(In reply to Robert Wood [:rwood] from comment #5)

Pushed to try on behalf of :marauder

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7ac4067a5887e918ee262e95175056d23b05ef9d

One of the test pages (edition.cnn.com) fails on the Moto G5 but passes on the GP2:

raptor-main TEST-UNEXPECTED-FAIL: test 'raptor-tp6m-cnn-geckoview' timed out loading test page: https://edition.cnn.com/

I don't have a Moto G5 - can someone who has one please apply this patch locally and try to look into this? Thanks!

(In reply to Robert Wood [:rwood] from comment #6)

(In reply to Robert Wood [:rwood] from comment #5)

I don't have a Moto G5 - can someone who has one please apply this patch locally and try to look into this? Thanks!

I've replicated the timeout - it looks like this is a genuinely show site. Could we try increasing the timeout?

let's re-upload CNN
https://drive.google.com/drive/folders/15dvkXYrfvltJJo1efaB4TEwSpcFdi96Y?usp=sharing

Flags: needinfo?(rwood)
Flags: needinfo?(dave.hunt)

After talking with :davehunt we decided to not re upload

Updated page_timeout = 100000

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8913416f1340a12549e781bf6938b25a3034d36d

Flags: needinfo?(rwood)
Flags: needinfo?(dave.hunt)

Re-triggering to make sure we have no intermittent

(In reply to Florin Strugariu [:Bebe] from comment #12)

Re-triggering to make sure we have no intermittent

4 out of 11 retry have failed

(In reply to Florin Strugariu [:Bebe] from comment #14)

Try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dff84d1a3a801252ab29fcb54236ef3cc7bd908a

raptor-tp6m-cnn-ampstories-geckoview timed out in the very first run, can we disable that one too?

I have created a new bug for raptor-tp6m-cnn-ampstories-geckoview
https://bugzilla.mozilla.org/show_bug.cgi?id=1533666

I'll also update the phabricator differential here:
https://phabricator.services.mozilla.com/D20852

(In reply to Florin Strugariu [:Bebe] from comment #17)

New try build
https://treeherder.mozilla.org/#/jobs?repo=try&revision=75eafd063610a277e90ed21b1a9940179f0ac30d

In the future please retrigger new suites on one of the mobile platforms; I retriggered on GP2 x64 so we will have 6 runs total.

Android Raptor bitbar jobs are backed up quite a bit in production and on try.

:aerickson, do we have capacity if we land this? On try this new tp6m-4 chunk is showing up as about 6 minute run-time. However it will be expanded in the future to about 15-20 min when the rest of the tests in this chunk are turned on.

Flags: needinfo?(aerickson)

We are at capacity now and we're barely keeping up with production and have high wait times for try. We are expecting more devices to become online next month that will roughly double our capacity (confirmed with :jmaher). Adding the test to mozilla-central is probably doable now, but autoland, inbound, and try should wait otherwise developers will likely start experiencing long wait time for try pushes.

Flags: needinfo?(aerickson)

https://hg.mozilla.org/mozilla-central/rev/18fa4e4ce035

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox67: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: