We need to get test coverage for opengl reftest on android.
Trying a test on android to get an idea for what state we are at for reftest + talos:
We also made some test to see where we stand for talos. We didn't notice any significant regression. Some results were +2% which are on the high end of the regression.
The results are suspicious so I ran a patch to force a regression to make sure the test were being executed properly. This appears to be the case as test show a ~270% regression on tp4mobile no chrome and fennecbench zoom. I'm not sure why we didn't see a regression with tp4 mobile.
Everything looks normal with enable OGL and no regression being forced:
(If viewing >30 days from now change the displayrange)
Ran into remote-reftest misreporting success (doesn't block this):
Created attachment 560996 [details] [diff] [review]
For now I flagged all the failing test so that we can enable test coverage for OGL. Of importance is that all canvas/css-gradients/image-element/svg are disable for now.
We will work towards supporting all android reftest but for the time being I suggest that we ignore these test for this configuration and start automated testing to prevent further regressions.
Follow up task will be to remove all occurrences of |Android&&layersOpenGL| in **/reftest.list files.
Comment on attachment 560996 [details] [diff] [review]
Review of attachment 560996 [details] [diff] [review]:
We should definitely have at least two followup bugs; I don't think we can release with any skip-if(Android&&layersOpenGL), especially not if they are skipped because they crash, but we should also try to fix all the failures before releasing.
I presume those follow-up bugs were filed?