1309086 - run linux64 reftests on Ubuntu 16.04

Status: RESOLVED FIXED

(Testing :: General, defect)

Comment 1

[Karl Tomlinson (:karlt)]

Attachment: bug 1309086 run taskcluster desktop reftests and crashtests on Ubuntu 16.04

Review commit: https://reviewboard.mozilla.org/r/84742/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/84742/

Comment 2

[Karl Tomlinson (:karlt)]

These are all currently hidden due to bug 1223198, which will be fixed by moving off GTK 3.4.

There are some new failures due theme and font differences:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c330de71e670bc92bee559dd1d6d6561dd33e8c8&exclusion_profile=false&selectedJob=28924857

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8799604 [details]
bug 1309086 run taskcluster desktop reftests and crashtests on Ubuntu 16.04

https://reviewboard.mozilla.org/r/84742/#review83374

this is great, thanks for pushing this forward.  Any indication of crashtest or jsreftest could be moved over as well?  I assume you tested on opt/debug/asan on your try pushes?

Comment 4

[Karl Tomlinson (:karlt)]

Comment on attachment 8799604 [details]
bug 1309086 run taskcluster desktop reftests and crashtests on Ubuntu 16.04

https://reviewboard.mozilla.org/r/84742/#review83374

I added the same change to crashtests.  It probably makes sense for jsreftest too, but I'm limiting this to testsuites about which I know something.

Passing on taskcluster, but not on buildbot opt tests, which will not be unhidden because bug 1223198 will not be fixed on buildbot.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=61310e5e5bea92484677c0e6b44bdbed63a53ce4&exclusion_profile=false

Comment 5

[Karl Tomlinson (:karlt)]

Attachment: bug 1309086 reftest annotation changes for move to Ubuntu 16.04

"random" is used where the bug is likely in the test, or the test was already
random on other platforms because the feature was not complete.

"fails" is used where it looks like the bug is in the product, so that we know
to re-enable the test with the product is fixed.

Review commit: https://reviewboard.mozilla.org/r/85160/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/85160/

Comment 6

[Karl Tomlinson (:karlt)]

Comment on attachment 8799604 [details]
bug 1309086 run taskcluster desktop reftests and crashtests on Ubuntu 16.04

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/84742/diff/1-2/

Comment 7

[Daniel Holbert [:dholbert]]

Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

https://reviewboard.mozilla.org/r/85160/#review85210

r=me, with one bug-number annotation added:

::: layout/reftests/css-grid/reftest.list:82
(Diff revision 1)
>  == grid-item-align-003.html grid-item-align-003-ref.html
>  == grid-item-justify-001.html grid-item-justify-001-ref.html
>  == grid-item-justify-002.html grid-item-justify-002-ref.html
>  == grid-item-stretch-001.html grid-item-stretch-001-ref.html
>  == grid-item-self-baseline-001.html grid-item-self-baseline-001-ref.html
> -skip-if(!gtkWidget) == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms
> +random == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms

I filed bug 1310792 on these.

Please add "bug 1310792" to the end of the message on the first line here.

Comment 8

[Mats Palmgren (inactive)]

> +random == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms

Are there NO Linux platforms at all where these tests still pass?

Comment 9

[Mats Palmgren (inactive)]

Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

I object to the layout/reftests/css-grid/reftest.list changes until
comment 8 has been answered.

Comment 10

[Daniel Holbert [:dholbert]]

(same; my r+ was based on the assumption that this affected all linux tinderboxen.  If there are some which are unaffected & still pass, we should absolutely craft a more specific annotation that keeps them running [and still link to bug 1310792 in the manifest, so we've got the failure-everywhere-else tracked].)

Comment 11

[Karl Tomlinson (:karlt)]

(In reply to Mats Palmgren (:mats) from comment #8)
> Are there NO Linux platforms at all where these tests still pass?

They still pass on Ubuntu 12.04 platforms.  Some tinderboxen would still be
running reftests on 12.04 with buildbot after changes here, but their results are
already hidden behind "Excluded jobs".

There are no linux32 reftests running on taskcluster (pending resolution of
either bug 1253315 or bug 1209932), and it is easy enough to annotate these
reftests as passing on linux32.  Would you be happy to live with that for the
sake of getting some (i.e. taskcluster) reftests unhidden on linux64, where
all reftests are also currently hidden?

https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-e6a4ea63fc2642452e48c26cd108c67fa8597af8/try-linux/try_ubuntu32_vm_test-reftest-1-bm05-tests1-linux32-build95.txt.gz
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e6a4ea63fc2642452e48c26cd108c67fa8597af8&exclusion_profile=false&selectedJob=29303317

Are you also happy to run these reftests on other platforms to catch any
assertion failures should they start occurring?

Comment 12

[Karl Tomlinson (:karlt)]

Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/85160/diff/1-2/

Comment 13

[Karl Tomlinson (:karlt)]

Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

(In reply to Daniel Holbert [:dholbert] from comment #10)
> (same; my r+ was based on the assumption that this affected all linux
> tinderboxen.  If there are some which are unaffected & still pass, we should
> absolutely craft a more specific annotation that keeps them running

There is not currently anything available in the sandbox to distinguish between Ubuntu 12.04 and 16.04 or between buildbot and taskcluster, but I assume the intent is to decommission builtbot tests where they have moved to taskcluster, so any new attribute would become obsolete.  Therefore this patch implements the proposal of comment 11 to annotate differently on i686 where there are no taskcluster tests.

Requesting review again because there are hidden tinderboxen unaffected by the test system changes proposed here.

Comment 14

[Daniel Holbert [:dholbert]]

(In reply to Karl Tomlinson (:karlt) from comment #11)
> There are no linux32 reftests running on taskcluster [...]

Boo. And it looks like the (non-taskcluster) Linux32 reftest jobs on mozilla-inbound are currently hidden (behind "excluded") for some reason.

> Would you be happy to live with that for the
> sake of getting some (i.e. taskcluster) reftests unhidden on linux64, where
> all reftests are also currently hidden?

Based on your logs, it does indeed look like that keeps this test running on (hidden) Linux32 reftest -- which is better than the current state of things, I think.

> Are you also happy to run these reftests on other platforms to catch any
> assertion failures should they start occurring?

The switch from "skip-if" to "random-if", you mean?  That seems fine, yeah.

Comment 15

[Daniel Holbert [:dholbert]]

Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

Your Try run from comment 11 shows "unexpected-pass" for all non-taskcluster Linux x64 opt reftest runs, it looks like.  So, some of the annotations still need a bit of adjustment, it seems.

Comment 16

[Mats Palmgren (inactive)]

css-grid/reftest.list looks better now, thanks!

Comment 17

[Karl Tomlinson (:karlt)]

(In reply to Daniel Holbert [:dholbert] from comment #15)
> Your Try run from comment 11 shows "unexpected-pass" for all non-taskcluster
> Linux x64 opt reftest runs, it looks like.

Yes.  Those are hidden and will remain hidden because of bug 1223198.  The fix for that bug is to use an OS without the bug, which taskcluster enables us to do.
With these tests running on taskcluster, there will be no need to keep running them on buildbot.

> So, some of the annotations still need a bit of adjustment, it seems.

There is not currently anything available in the sandbox to distinguish between Ubuntu 12.04 and 16.04 or between buildbot and taskcluster.  When taskclaster and buildbot are running the same tests on different 64-bit OSes, the results are different.  The taskcluster tests are the ones I'm aiming to unhide, and so the annotations are designed for those machines.

I guess an attribute could be added, but I don't know how its value would be determined, and it seems unnecessary when the buildbot linux64 tests are obsoleted by the taskcluster tests.

Comment 18

[Daniel Holbert [:dholbert]]

Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

OK, just wanted to make sure those oranges weren't an oversight. r=me, thanks!

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b25ba1be8195
run taskcluster desktop reftests and crashtests on Ubuntu 16.04 r=jmaher
https://hg.mozilla.org/integration/autoland/rev/4a6af5f7d9bb
reftest annotation changes for move to Ubuntu 16.04 r=dholbert

Comment 20

[(not currently active) Ted Mielczarek]

(In reply to Karl Tomlinson (:karlt) from comment #17)
> I guess an attribute could be added, but I don't know how its value would be
> determined, and it seems unnecessary when the buildbot linux64 tests are
> obsoleted by the taskcluster tests.

FWIW, if we wanted to add this we could presumably just read the bits out of /etc/lsb-release:
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"

mozinfo sets an `os_version` attribute for most platforms, but it's using `platform.linux_distribution()` on Linux which seems to be returning useless results (the contents of /etc/debian_version on my Ubuntu).

Comment 21

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/b25ba1be8195
https://hg.mozilla.org/mozilla-central/rev/4a6af5f7d9bb

Comment 22

[Ryan VanderMeulen [:RyanVM]]

Looks like things are much happier now, other than the 1302389-scrolled-rect* tests.
https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=linux%20x64%20reftest&group_state=expanded&filter-tier=1&filter-tier=2&filter-tier=3&exclusion_profile=false

Comment 24

[Geoff Brown [:gbrown]]

I'm trying to run 32-bit linux reftests on 16.04 and hitting many of the same problems encountered in this bug; I'll need to relax some of the annotations to skip/fail on 32-bit linux also.