Closed Bug 1309086 Opened 3 years ago Closed 3 years ago

run linux64 reftests on Ubuntu 16.04

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(firefox52 fixed)

RESOLVED FIXED
mozilla52
Tracking Status
firefox52 --- fixed

People

(Reporter: karlt, Assigned: karlt)

References

(Depends on 8 open bugs)

Details

Attachments

(2 files)

No description provided.
These are all currently hidden due to bug 1223198, which will be fixed by moving off GTK 3.4.

There are some new failures due theme and font differences:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c330de71e670bc92bee559dd1d6d6561dd33e8c8&exclusion_profile=false&selectedJob=28924857
Depends on: 1309095
Depends on: 1309103
Depends on: 1309107
Depends on: 1309113
Comment on attachment 8799604 [details]
bug 1309086 run taskcluster desktop reftests and crashtests on Ubuntu 16.04

https://reviewboard.mozilla.org/r/84742/#review83374

this is great, thanks for pushing this forward.  Any indication of crashtest or jsreftest could be moved over as well?  I assume you tested on opt/debug/asan on your try pushes?
Attachment #8799604 - Flags: review?(jmaher) → review+
Depends on: 1309405
Depends on: 1309425
Depends on: 1309426
Depends on: 1309428
Depends on: 1309429
Depends on: 1309430
Comment on attachment 8799604 [details]
bug 1309086 run taskcluster desktop reftests and crashtests on Ubuntu 16.04

https://reviewboard.mozilla.org/r/84742/#review83374

I added the same change to crashtests.  It probably makes sense for jsreftest too, but I'm limiting this to testsuites about which I know something.

Passing on taskcluster, but not on buildbot opt tests, which will not be unhidden because bug 1223198 will not be fixed on buildbot.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=61310e5e5bea92484677c0e6b44bdbed63a53ce4&exclusion_profile=false
Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

https://reviewboard.mozilla.org/r/85160/#review85210

r=me, with one bug-number annotation added:

::: layout/reftests/css-grid/reftest.list:82
(Diff revision 1)
>  == grid-item-align-003.html grid-item-align-003-ref.html
>  == grid-item-justify-001.html grid-item-justify-001-ref.html
>  == grid-item-justify-002.html grid-item-justify-002-ref.html
>  == grid-item-stretch-001.html grid-item-stretch-001-ref.html
>  == grid-item-self-baseline-001.html grid-item-self-baseline-001-ref.html
> -skip-if(!gtkWidget) == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms
> +random == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms

I filed bug 1310792 on these.

Please add "bug 1310792" to the end of the message on the first line here.
Attachment #8800168 - Flags: review?(dholbert) → review+
> +random == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms

Are there NO Linux platforms at all where these tests still pass?
Flags: needinfo?(karlt)
Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

I object to the layout/reftests/css-grid/reftest.list changes until
comment 8 has been answered.
Attachment #8800168 - Flags: review-
(same; my r+ was based on the assumption that this affected all linux tinderboxen.  If there are some which are unaffected & still pass, we should absolutely craft a more specific annotation that keeps them running [and still link to bug 1310792 in the manifest, so we've got the failure-everywhere-else tracked].)
(In reply to Mats Palmgren (:mats) from comment #8)
> Are there NO Linux platforms at all where these tests still pass?

They still pass on Ubuntu 12.04 platforms.  Some tinderboxen would still be
running reftests on 12.04 with buildbot after changes here, but their results are
already hidden behind "Excluded jobs".

There are no linux32 reftests running on taskcluster (pending resolution of
either bug 1253315 or bug 1209932), and it is easy enough to annotate these
reftests as passing on linux32.  Would you be happy to live with that for the
sake of getting some (i.e. taskcluster) reftests unhidden on linux64, where
all reftests are also currently hidden?

https://archive.mozilla.org/pub/firefox/try-builds/ktomlinson@mozilla.com-e6a4ea63fc2642452e48c26cd108c67fa8597af8/try-linux/try_ubuntu32_vm_test-reftest-1-bm05-tests1-linux32-build95.txt.gz
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e6a4ea63fc2642452e48c26cd108c67fa8597af8&exclusion_profile=false&selectedJob=29303317

Are you also happy to run these reftests on other platforms to catch any
assertion failures should they start occurring?
Flags: needinfo?(karlt) → needinfo?(mats)
Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

(In reply to Daniel Holbert [:dholbert] from comment #10)
> (same; my r+ was based on the assumption that this affected all linux
> tinderboxen.  If there are some which are unaffected & still pass, we should
> absolutely craft a more specific annotation that keeps them running

There is not currently anything available in the sandbox to distinguish between Ubuntu 12.04 and 16.04 or between buildbot and taskcluster, but I assume the intent is to decommission builtbot tests where they have moved to taskcluster, so any new attribute would become obsolete.  Therefore this patch implements the proposal of comment 11 to annotate differently on i686 where there are no taskcluster tests.

Requesting review again because there are hidden tinderboxen unaffected by the test system changes proposed here.
Attachment #8800168 - Flags: review+ → review?(dholbert)
Flags: needinfo?(mats)
(In reply to Karl Tomlinson (:karlt) from comment #11)
> There are no linux32 reftests running on taskcluster [...]

Boo. And it looks like the (non-taskcluster) Linux32 reftest jobs on mozilla-inbound are currently hidden (behind "excluded") for some reason.

> Would you be happy to live with that for the
> sake of getting some (i.e. taskcluster) reftests unhidden on linux64, where
> all reftests are also currently hidden?

Based on your logs, it does indeed look like that keeps this test running on (hidden) Linux32 reftest -- which is better than the current state of things, I think.

> Are you also happy to run these reftests on other platforms to catch any
> assertion failures should they start occurring?

The switch from "skip-if" to "random-if", you mean?  That seems fine, yeah.
Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

Your Try run from comment 11 shows "unexpected-pass" for all non-taskcluster Linux x64 opt reftest runs, it looks like.  So, some of the annotations still need a bit of adjustment, it seems.
Flags: needinfo?(karlt)
css-grid/reftest.list looks better now, thanks!
(In reply to Daniel Holbert [:dholbert] from comment #15)
> Your Try run from comment 11 shows "unexpected-pass" for all non-taskcluster
> Linux x64 opt reftest runs, it looks like.

Yes.  Those are hidden and will remain hidden because of bug 1223198.  The fix for that bug is to use an OS without the bug, which taskcluster enables us to do.
With these tests running on taskcluster, there will be no need to keep running them on buildbot.

> So, some of the annotations still need a bit of adjustment, it seems.

There is not currently anything available in the sandbox to distinguish between Ubuntu 12.04 and 16.04 or between buildbot and taskcluster.  When taskclaster and buildbot are running the same tests on different 64-bit OSes, the results are different.  The taskcluster tests are the ones I'm aiming to unhide, and so the annotations are designed for those machines.

I guess an attribute could be added, but I don't know how its value would be determined, and it seems unnecessary when the buildbot linux64 tests are obsoleted by the taskcluster tests.
Flags: needinfo?(karlt)
Comment on attachment 8800168 [details]
bug 1309086 reftest annotation changes for move to Ubuntu 16.04

OK, just wanted to make sure those oranges weren't an oversight. r=me, thanks!
Attachment #8800168 - Flags: review?(dholbert) → review+
Pushed by ktomlinson@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b25ba1be8195
run taskcluster desktop reftests and crashtests on Ubuntu 16.04 r=jmaher
https://hg.mozilla.org/integration/autoland/rev/4a6af5f7d9bb
reftest annotation changes for move to Ubuntu 16.04 r=dholbert
(In reply to Karl Tomlinson (:karlt) from comment #17)
> I guess an attribute could be added, but I don't know how its value would be
> determined, and it seems unnecessary when the buildbot linux64 tests are
> obsoleted by the taskcluster tests.

FWIW, if we wanted to add this we could presumably just read the bits out of /etc/lsb-release:
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"

mozinfo sets an `os_version` attribute for most platforms, but it's using `platform.linux_distribution()` on Linux which seems to be returning useless results (the contents of /etc/debian_version on my Ubuntu).
https://hg.mozilla.org/mozilla-central/rev/b25ba1be8195
https://hg.mozilla.org/mozilla-central/rev/4a6af5f7d9bb
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
Depends on: 1311604
Blocks: 1311916
Duplicate of this bug: 1281159
I'm trying to run 32-bit linux reftests on 16.04 and hitting many of the same problems encountered in this bug; I'll need to relax some of the annotations to skip/fail on 32-bit linux also.
Flags: needinfo?(gbrown)
See Also: → 1287099
See Also: → 1328771
Flags: needinfo?(gbrown)
You need to log in before you can comment on or make changes to this bug.