1272878 - Autophone - Mochitest Skia failures

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: Android 4.2 / Nexus 4 Errors

We currently run Mochitest Skia (Msk) on mozilla-central

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=2afd8fa9bb5df5577e5566468bb423b76c63cc77&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=autophone

and have errors the following errors

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: Android 4.4 / Nexus 5 Errors

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: Android 6.0.1 / Nexus 6p Errors

Comment 3

[Bob Clary [:bc] (inactive)]

snorp: If we aren't able to fix these issues with Msk or if it is not a priority, I'll remove Msk from the non-try production runs so we will not be in a perma-orange state. Let me know how you'd like me to handle these.

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

Milan, do you think you could get someone to look at these failures?

Comment 5

[Milan Sreckovic [:milan] (needinfo for best results)]

A bit of background here - these used to run, but started failing, or never ran, or we updated the version of Android and then it started failing, or...?
George, can you find out some of the history on these?

Comment 6

[Bob Clary [:bc] (inactive)]

Snorp asked for these ages ago where they have been available on try but we never got them in shape to run regularly partly due to perma-orange.

In an attempt to find out which tests we could run without failures I added them to fx-team mozilla-central mozilla-beta mozilla-aurora mozilla-release in March but removed them from anything but mozilla-central in April since they were causing load issues due to timeouts/perma-orange.

If we don't care about these I intend to remove them from mozilla-central before the end of June and just keep them available on try.

Comment 7

[Milan Sreckovic [:milan] (needinfo for best results)]

I'm going to assume we care or :snorp wouldn't have tagged us, so, George, let's see what more we can find out about these failures.

Comment 8

[George Wright (:gw280) (needinfo me!)]

For some reason I never knew this existed in the first place. As far as I can tell, it looks like these have been around for nearly 3 years? The original bug is bug 893611. It's just a normal mochitest canvas test run on Android but with skiagl preffed on. What puzzles me is that the prefs used for this are the same as the default on android anyway (https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/mobile.js#819), and we don't have any perma-orange on our normal mochitest runs, so why is this failing?

Looking at the log, there are only three actual test failures:

452 INFO TEST-UNEXPECTED-FAIL | dom/canvas/test/test_imagebitmap_cropping.html | pixel 100,100 of  is 255,255,255,247; expected 255,255,255,255 +/- 5 
207 INFO TEST-UNEXPECTED-FAIL | dom/canvas/test/test_imagebitmap.html | pixel 384,0 of  is 4,251,0,255; expected 0,255,0,255 +/- 1 
538 INFO TEST-UNEXPECTED-FAIL | dom/canvas/test/test_offscreencanvas_dynamic_fallback.html | after dynamic fallback, screenshots should be the same 

But we do get an EGL error:

05-13 18:21:23.220 E/libEGL  ( 3540): eglMakeCurrent:775 error 3002 (EGL_BAD_ACCESS)
05-13 18:21:23.220 W/Adreno-EGL( 3540): <qeglDrvAPI_eglMakeCurrent:2919>: EGL_BAD_ACCESS

This may be harmless though.

Bob, are there any differences between our "normal" android mochitests run on m-c, and the autophone-based Mochitest-Skia here? Are the devices themselves different?

Comment 9

[Bob Clary [:bc] (inactive)]

gbrown can provide more detail I think, but my understanding is the "normal" android mochitests run on emulators while the autophone tests run on actual devices.

If you click on the job symbol, you will see the device name in the left panel. We currently have Nexus 4, Nexus 5, Nexus 6, Nexus 6P, Nexus 9 and a smattering of Nexus S which are being phased out as Android 2.3 support leaves the station.

I would guess that the different screen sizes might come into play here.

Comment 10

[Geoff Brown [:gbrown]]

The "normal" tier-1 mochitests on treeherder run on Android 4.3.1 (slight variant of JLS36I), on the android emulator with a screen size of 800 wide x 1280 high. This bug is about autophone, running on various real devices.

Comment 11

[George Wright (:gw280) (needinfo me!)]

Attachment: 0001-Bug-1272878-Fuzz-test_bitmaprenderer.html-because-of.patch

bitmaprenderer uses an ImageLayer instead of a CanvasLayer, which takes a totally different codepath to render images than canvas2d, and result in some anti-aliasing differences at the edge of the image.

Both codepaths apply some anti-aliasing (canvas2d in this case has 254,255,254 as the edge pixels, and bitmaprenderer has 240,255,240), so it's not a case of AA vs. non-AA, which I think would be problematic.

It seems reasonable to fuzz this as AA differences when using different codepaths isn't something we've traditionally been fussed about.

Comment 12

[George Wright (:gw280) (needinfo me!)]

leave open for now, as there are a couple of other issues that need resolving.

Comment 13

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8779044 [details] [diff] [review]
0001-Bug-1272878-Fuzz-test_bitmaprenderer.html-because-of.patch

Review of attachment 8779044 [details] [diff] [review]:
-----------------------------------------------------------------

Great, thanks for fix.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e2d1e50ec9f5
Fuzz test_bitmaprenderer.html because of differences in our two codepaths with antialiasing r=Morris

Comment 15

[George Wright (:gw280) (needinfo me!)]

Bob, once this hits central we can probably enable Msk on Android 5 and 6:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9916bcddfd85&exclusion_profile=false

Comment 16

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/b19e073039d9e50db74f7d065ac5e8b698ea7b94

deployed 2016-08-10 ~00:00 PDT

I expect the next mozilla-central build will have the merge. /me crosses fingers.

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/e2d1e50ec9f5

Comment 18

[Bob Clary [:bc] (inactive)]

I messed up and enabled Nexus 5 which of course is still failing. When the current push on m-c finishes, I'll update the manifest to disable it again. We'll be running Msk on Nexus 6P only.

Comment 19

[Bob Clary [:bc] (inactive)]

disable Nexus 5 on Msk
https://github.com/mozilla/autophone/commit/3a2e6bd5f2261d20e59917b3afbf272cad072870
deployed 2016-08-10 ~08:55:00 PDT.

Comment 20

[Bob Clary [:bc] (inactive)]

filed bug 1294491 for the assertion errors.

Comment 21

[Bob Clary [:bc] (inactive)]

-> fixed

Comment 22

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.