Closed Bug 1025961 Opened 8 years ago Closed 6 years ago

B2G emulator reftest runtime on v2.0/master is double what it is on v1.4

Categories

(Testing :: Reftest, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(tracking-b2g:backlog)

RESOLVED INCOMPLETE
tracking-b2g backlog

People

(Reporter: RyanVM, Unassigned)

References

Details

I noticed last week that B2G emulator reftests seem to have severely degraded in performance at some point over the past couple development cycles, with the runtime nearly doubling where they used to be. This can be seen by comparing results on b2g30 (v1.4) and trunk, where the runtime is nearly double on average.

I'm working on bisecting this down on Try, and it appears so far that there were multiple regressions along the way. I'll provide more information as I get it.
It does appear that we enabled a lot more tests at one point, so that at least explains some of the regression.
Right now, it appears there are 3 different spikes in my Try runs:
* One relatively early in the Gecko 31 cycle where we went from ~30min average to ~42min average.
* One relatively early in the Gecko 32 cycle where we went from ~42min average to ~48min average.
* One later in Gecko 32 where we went from ~48min average to ~55min average.

Earlier Try experiments suggest that enabling OOP on reftests may explain the later regression, but the earlier ones still need bisecting.
The first regression (30->42) is almost certainly from bug 986409, which enabled multiple directories that were previously being skipped. I'm going to run with that and focus on the latter two regressions.
https://hg.mozilla.org/mozilla-central/rev/c704812d3776
Of particular note:
(In reply to David Baron [:dbaron] (busy May 31-June 15) (UTC-7) from bug 986409 comment #5)
> The push prior to this ran 5123 reftests on B2G ICS Emulator Opt; this push
> ran 9724.
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #2)
> * One later in Gecko 32 where we went from ~48min average to ~55min average.
> 
> Earlier Try experiments suggest that enabling OOP on reftests may explain
> the later regression, but the earlier ones still need bisecting.

Bisection agrees that OOP was indeed behind the third ~7min average slowdown per chunk. Still trying to narrow down the middle regression.
Depends on: 986409, 922680
Regression #2 appears to have narrowed down to revision b437c670e090 (bug 1001683).
Depends on: 1001683
So in summary:

"Regression" #1 (~30min -> ~42min) was from bug 986409, which nearly doubled the number of reftests run. Clearly an increase would have been expected from such a change, so this looks to be safe to ignore.

Regression #2 (~42min -> ~48min, ~25% slowdown) was from bug 1001683, which converted a bunch of ReadPixelsIntoImageSurface callers to ReadPixelsIntoDataSurface callers. It's unclear as to why that would have regressed run time, but it seems worth investigating for real-world perf regressions.

Regression #3 (~48min -> ~55min, ~15% slowdown) was from bug 922680, which enabled OOP for B2G reftests. I guess this amounts to IPC overhead? Still seems like something that merits investigating to see if there are any wins to be had that could improve things on real-world phones too.

Not sure who should own this bug from here, but I believe I've taken it as far as I can at this point.
jgriffin - Is this causing the reftests to not be possible to be ran?

milan - Does this have any end user implied impact?
Flags: needinfo?(milan)
Flags: needinfo?(jgriffin)
We're still running the tests, they're just running less efficiently than they used to, which increases end-to-end time on TBPL.
Flags: needinfo?(jgriffin)
I don't have any bugs that I've seen that show the end user impact, but that doesn't mean there isn't one.

Jonathan/Matt, would you expect the slowdown from bug 1001683 (see comment 7?)

CJ, would you expect the slowdown from OOP bug 922680 (see comment 7?)
Flags: needinfo?(milan)
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(jwatt)
Flags: needinfo?(cku)
(In reply to Milan Sreckovic [:milan] from comment #10)
> I don't have any bugs that I've seen that show the end user impact, but that
> doesn't mean there isn't one.
> 
> Jonathan/Matt, would you expect the slowdown from bug 1001683 (see comment
> 7?)

Nope, not expected at all. I can't see anything in the patch that would cause this either. It's probably the CompositorOGL readback though, which is code that only affects reftests and snapshots.
Flags: needinfo?(matt.woodrow)
(In reply to Milan Sreckovic [:milan] from comment #10)
> CJ, would you expect the slowdown from OOP bug 922680 (see comment 7?)
IPC may slow it down. Meanwhile, we also disable some test cases(bug 973835), font-inflation/ XUL for example. This should make it faster.

I think it's worth to look into this problem for perf downgrade after enable OOP. No because of rendering performance, but to reduce resource of try servers occupied by reftest.

PS:
After bug 1000722 land, it's going to be even slower.
Flags: needinfo?(cku)
Given that this isn't a test blocker & there isn't proof of end user impact of this bug, not blocking.
blocking-b2g: 2.0? → backlog
I don't have any more info to add to comment 11.
Flags: needinfo?(jwatt)
blocking-b2g: backlog → ---
These branches are EOL and the discussion in this bug strongly implies that nothing else is going to be done here anyway.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.