Closed Bug 1536168 Opened 6 years ago Closed 6 years ago

Raptor tests are failing to run on fenix

Categories

(Testing :: Raptor, defect, P1)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rwood, Unassigned)

References

Details

Bug 1522100 landed support for running the Raptor performance tests (speedometer and tp6m-1) on the android components reference browser. At the time of that landing, the tests worked great.

However now the tests are failing on a more recent reference browser (tried on the latest build on Mar-18) - looks like the Raptor webext may no longer be loading successfully in the reference browser.

:csadilek have there been any changes to the reference browser that would cause Raptor to no longer work?
:jlorenzo have you seen this with your work to stand up CI support for reference browser with Raptor?

Flags: needinfo?(jlorenzo)
Flags: needinfo?(csadilek)

The last error I've seen was: https://tools.taskcluster.net/groups/BqrINI6KSXmbk8nMl3U3mQ/tasks/VPmtXCirSs-lCrHS7AqEkg/runs/0/logs/public%2Flogs%2Flive_backing.log#L1051

I'm not too sure if it's the same error, or not.

Flags: needinfo?(jlorenzo)

:rwood, what type of failures are you seeing? can you paste a log or some text from a local run that shows the error? Also how do you get the browser and what steps do you take to get this error?

Flags: needinfo?(rwood)

I'm hitting these in my try runs for fenix/refbrow from mozilla-central:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=12adab3d7008618ff1f4596fbe0757f6af13bf94

Summary: Raptor tests are failing to run on the reference browser → Raptor tests are failing to run on the reference browser and fenix

Raptor is not 'officially' supported on Fenix yet, only the cmd line supported was added so that dev could try it out when the Fenix was ready (i.e. supported loading the webext, command line arg for the profile, etc). There are no Raptor tests configured to run on Fenix yet.

As for Raptor on the reference browser it used to work (support is there for speedometer and tp6m-1 only) but has been broken since support initially landed. It looks to me like the Raptor webext isn't even loading but I need to verify and find out more when I get a chance.

I confirm I still hit the error in comment 2. After looking more closely at the logs, this is the same error than comment 4: Raptor times out, waiting for an activity:

15:49:44     INFO -  adb shell_output: adb -s ZY322M59S3 wait-for-device shell am start -W -n org.mozilla.reference.browser/org.mozilla.reference.browser.BrowserTestActivity -a android.intent.action.Main --ez use_multiprocess True --es args "-profile /sdcard/raptor/profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6" -d about:blank; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: Starting: Intent { act=android.intent.action.Main dat=about:blank cmp=org.mozilla.reference.browser/.BrowserTestActivity (has extras) }
15:49:44     INFO -  Status: ok
15:49:44     INFO -  Activity: org.mozilla.reference.browser/.BrowserActivity
15:49:44     INFO -  ThisTime: 876
15:49:44     INFO -  TotalTime: 1198
15:49:44     INFO -  WaitTime: 1213
15:49:44     INFO -  Complete
16:05:14     INFO -  raptor-main application timed out after 933 seconds

https://tools.taskcluster.net/groups/HM15OhFeQJqKMcVHPvATpQ/tasks/DPtknpPTS9Cwg_GFkn1-zw/runs/0/logs/public%2Flogs%2Flive_backing.log#L1016

we need to determine if this is a geckoview or a fenix/refbrow issue, :ahal will try to narrow this down to see if geckoview is the root cause

I've bisected reference browser and found the regression. Filed an issue and working on a fix: https://github.com/mozilla-mobile/reference-browser/issues/667

Flags: needinfo?(csadilek)

(In reply to Christian Sadilek [:csadilek] from comment #8)

I've bisected reference browser and found the regression. Filed an issue and working on a fix: https://github.com/mozilla-mobile/reference-browser/issues/667

Thanks very much Christian!

Flags: needinfo?(rwood)
See Also: → 1537989

Raptor tests are now running again on latest reference-browser (master and upcoming nightly build).

Thanks, I have them running again in my try pushes.

Fenix builds are still failing, but I haven't seen them passing yet, so I'm not sure if that's the same issue here, or if it's something else. I'll do some investigating.

It still has the same failure log as the reference browser did before the fix:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235848880&repo=try&lineNumber=1405

The application times out waiting for an activity as in comment 6. I looked at the commit that caused the regression in the reference browser and it looks like Fenix doesn't have any similar commits landing around the same time, so the issue here must be slightly different. Johan, are you able to run fenix builds locally? Maybe I'm just setting up the task wrong.

Flags: needinfo?(jlorenzo)
Summary: Raptor tests are failing to run on the reference browser and fenix → Raptor tests are failing to run on fenix

I'm sorry, I've never tried to run jobs locally. Similarly, I've just focused on r-b, so far.

Flags: needinfo?(jlorenzo)

:ahal Ah yes, Fenix is different. Raptor is disabled by default in Fenix release builds (and enabled in debug builds) as we didn't want to expose the BrowserPerformanceTestActivity.

So, right now you'd have to use the Debug build for Fenix or we can create a special Raptor release one:
https://github.com/mozilla-mobile/fenix/pull/670

Thanks Christian, makes sense! At this point I don't mind using debug builds to get up and running, though I imagine running performance tests on debug builds will be a problem down the road.

I'm currently grabbing fenix builds from the taskcluster index here:
https://tools.taskcluster.net/index/project.mobile.fenix.signed-nightly.nightly/latest

Would it be possible to add debug builds to the index as well? Or is there some other auto-updating URL I can use?

Flags: needinfo?(csadilek)

Should be possible, yes, but I'd defer to Johan and team to define what's best e.g. create a TC index for debug builds or start publishing a separate release build variant for Fenix with Raptor enabled and index that?

Flags: needinfo?(csadilek) → needinfo?(jlorenzo)

what's best e.g. create a TC index for debug builds or start publishing a separate release build variant for Fenix with Raptor enabled and index that?

The former looks cleaner to me. We can easily expose a debug build from master. I don't think we should clutter the nightly graph with debug builds. I'll take care of it, this week.

Flags: needinfo?(jlorenzo)

We've had a PI request for comparing Fennec v64 with Fenix based on GeckoView v67. I'm concerned that a debug build of Fenix will not be a suitable comparison. Do we have a suitable Fenix build for to use with Raptor?

Flags: needinfo?(csadilek)

:davehunt - It's no problem to configure a release build variant with Raptor enabled.

:jlorenzo - Looks like we need to add a new release variant and expose it on TC. It's basically just an entry to specify a build type e.g. raptor or staging in https://github.com/mozilla-mobile/fenix/blob/master/app/build.gradle#L34:
staging {
shrinkResources true
minifyEnabled true
proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
manifestPlaceholders.isRaptorEnabled = "true"
}

We need to decide how to best expose this on TC though?

Flags: needinfo?(jlorenzo)

Nice! I'm working on https://github.com/mozilla-mobile/fenix/pull/1220, which will expose every build type on pushes to master. So, if this variant is output by ./gradlew printBuildVariants, then exposing it will be automatic once #1220 lands.

Flags: needinfo?(jlorenzo)

:jlorenzo, this sounds great to expose all variants to be accessible, looking forward to that landing.

my understanding is this variant doesn't exist, but it sounds easy to add, who will take care of adding the new release-test variant ?

:jmaher I'll try to get that done today.

Flags: needinfo?(csadilek)

(In reply to Johan Lorenzo [:jlorenzo] from comment #20)

I'm working on https://github.com/mozilla-mobile/fenix/pull/1220

This PR is ready for review, if you're interested, :csadilek.

Thanks :jlorenzo, will take a look. I've put up the raptor release build variant as well: https://github.com/mozilla-mobile/fenix/pull/1245

Raptor is running successfully on Fenix now.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.