737427 - remove the need for adjusting screen resolution on tegras

Status: RESOLVED DUPLICATE of bug 746260

(Release Engineering :: General, defect, P4)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Our tegras are fragile and the more we do with them the higher chance of failure.  When running reftests, we adjust the resolution (now to 1600x1200 since widthx1080 was not enough to meet the minimum 1000 height of the viewport) for a reftest and adjust it back down.  This requires two reboots which take on average 5 minutes per reboot.  With each checkin, we run 18 reftest based jobs, or 180 minutes of reboots related to adjusting the screen resolution.

In addition to the time saved, the tegras have a shared memory model with the video card.  This means that the higher the resolution, the less memory is available to fennec.  I have noticed running mochitest or robocop on 1600x1200 we hang or crash with OOM and then changing back to 1024x768 the tests work just fine.  The chance that our lack of memory for fennec is causing a lot of the failures on tegras is very high, reducing our memory needs is a good practice in general.

I took a look at which reftests require the 1000 pixel minimum.  It appears there are just a handful (I believe 10, although it was really hard to determine this based off a try server run).  Oddly enough we do NOT run those 10 tests on android currently.  Local runs and a tegra staging run yield the same results if we keep a 1024x768 resolution.

I suggest we fix this in 3 steps:
1) Adjust the sut_tools code to check for a 1024x768 resolution (leaving the code in place for dynamic adjustment)
2) Add the --ignore-window-size flag to the buildbot steps for launching reftest based tests on android
3) Add code in reftest to support a manifest condition for resolution and add conditions for the tests which are known to be larger. (this is not a releng item)

Comment 1

[Armen [:armenzg]]

I guess I can already start working on #1 and #2, right?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

yeah, 1+2 will resolve this bug, #3 is for completeness!

Comment 3

[Armen [:armenzg]]

Attachment: [sut_tools/config.py] don't change to 1600x1200

Comment 4

[Armen [:armenzg]]

Attachment: [buildbotcustom] add --ignore-window-size

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 607557 [details] [diff] [review]
[sut_tools/config.py] don't change to 1600x1200

Review of attachment 607557 [details] [diff] [review]:
-----------------------------------------------------------------

looks great.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 607558 [details] [diff] [review]
[buildbotcustom] add --ignore-window-size

Review of attachment 607558 [details] [diff] [review]:
-----------------------------------------------------------------

so simple.

Comment 7

[Aki Sasaki (not active)]

I have wanted to ditch the screen resolution changes for forever. jmaher++ armenzg++

Comment 8

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

We've discussed this so many times ...  It's not just tests that fail with too-low resolution.  It's tests that will *pass*, when they should fail, because we're not comparing all the pixels.  It should be possible to automatically identify those tests by instrumenting the reftest harness, but it's not trivial.

This is going to result in reftest failures going undetected on android.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

we need to identify all the tests which require >1000 pixels and mark them that way in a manifest.  As it stands we are getting identical results with and without the screen resolution change.

Comment 10

[Armen [:armenzg]]

I will deploy this tomorrow as I would like to make sure that it doesn't go in without updating the foopies.

Comment 11

[Armen [:armenzg]]

talos svg for *XUL* has failed twice. I have triggered 4 more jobs to determine if it is a real regression:
http://tinderbox.mozilla.org/showlog.cgi?log=MobileTest/1332267423.1332270182.3086.gz&fulltext=1

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

all talos jobs run as 1024x768, so I doubt that is related.  If it fails 4 times in a row, then remove the changes and try again to see if these changes cause the failure.

Comment 13

[Armen [:armenzg]]

It was not related. 2 of the jobs succeeded and 2 of them failed.

We're still on track to enable this tomorrow morning.

Comment 14

[Armen [:armenzg]]

Comment on attachment 607557 [details] [diff] [review]
[sut_tools/config.py] don't change to 1600x1200

https://hg.mozilla.org/build/tools/rev/a6462804d946

Comment 15

[Armen [:armenzg]]

Comment on attachment 607558 [details] [diff] [review]
[buildbotcustom] add --ignore-window-size

http://hg.mozilla.org/build/buildbotcustom/rev/febf2d311f1b

Comment 16

[Armen [:armenzg]]

buildbot-master19 went live around  9:45AM PDT and
buildbot-master20 went live around 10:25AM PDT

Comment 17

[Armen [:armenzg]]

We're backing out due to some new perma-orange that appeared:
https://tbpl.mozilla.org/php/getParsedLog.php?id=10246645&full=1&branch=mozilla-inbound

Comment 18

[Armen [:armenzg]]

Great. If I trigger something from yesterday we get green. If I trigger a change from today we go orange.

BTW if you let me work on bug 715193 we can make this bug to be testable on try (hopefully - man, I should stop promising heaven).

Comment 19

[Armen [:armenzg]]

I will know in few days when I can get to this and adjust back the priority to P2 when working on it. P3 means I am not touching it and might be grabbing it back in 2 weeks.

Comment 20

[Chris AtLee [:catlee]]

armen, please back this out from the default branch as well.

Comment 21

[Armen [:armenzg]]

I backed it out from default as well.
http://hg.mozilla.org/build/buildbotcustom/rev/579525331a9c