624851 - Enable {crash,ref}test-ipc for Mac machines on Try

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

See also bug 623613.  AFAIK we have all the infra support we need for this already, just need to flip the switch to enable these test suites when the platform deps are ready.

Comment 1

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Forgot to note that for the time being, we'll need to run these tests with GPU rendering *disabled*, because our multi-process gfx code doesn't understand D3D yet.  (This caused lots of problems on tryserver! ;) )

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

For my records, tests initially failing in my VM

REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/modules/libpr0n/test/reftest/pngsuite-ancillary/ccwn2c08.png | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/modules/libpr0n/test/reftest/pngsuite-ancillary/ccwn3p08.png | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/modules/libpr0n/test/reftest/apng/delaytest.html?bug546272.png | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/modules/libpr0n/test/reftest/generic/accept-image-catchall.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/289480.html#top | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/564991-1.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/579985-1.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/pixel-rounding/background-image-tiling.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/as-image/background-image-rect-1png.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/as-image/list-simple-1.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/image/image-fill-01.svg | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/image/image-svg-inline-01.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/smil/anim-image-href-01.svg | image comparison (==)

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(Also for my reference, tests failing in-process in my VM

REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/289480.html#top | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/579985-1.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/gfx/tests/reftest/611498-1.html | image comparison (==)

)

Comment 4

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Good news:

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1294962549.1294965381.21061.gz
reftest: 5568/20/304 L-FAIL

Of the failures, the vast majority are differences I can't distinguish even on one pixel visually, even zoomed in fully.  Not sure what's up, wouldn't block on those.

One is a legitimate failure probably caused by bug 605618.  Should be easy to fix.

The rest are apparently failures to load the test plugin from the content process.  Not sure what's up with that.  Very much willing to disable those tests, since they're a sideshow of a sideshow.

Not sure what's up with the alleged leak.

Comment 5

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Oh, the "leak" is probably

TEST-UNEXPECTED-FAIL | tab process 3284 | automationutils.processLeakLog() | missing output line for total leaks!

We _exit() in content processes when the chrome process shuts down.  Probably need a LogTerm() before doing that, for unit tests.

Comment 6

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

The reftests are mainly failing for niggly pixel differences due to the duct-tape-and-rubber-band rendering setup.  We're never going to ship out-of-process content with this code, so I don't think it's worth investing time sorting out these failures.

I *do* think it's worth running the crashtests so that we know of big problems caused by platform changes.

Comment 7

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

I think this should block because it's going to be our only automated way to catch broken windows nightlies of fennec, which extension developers are expected to use.

Comment 8

[Chris AtLee [:catlee]]

Same question as in bug 623613:

Do you want this pref set for the existing reftest / crashtest runs, or do you
want an additional set of runs with this pref set?

Which branches should this be applied to?

Comment 9

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

This needs to be an additional run for now, say CIPC (crashtest IPC) for short.

Re: branches, let's say
 - m-c
 - tracemonkey
 - places
 - project twigs
 - electrolysis

in that order of importance.

Comment 10

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

On my last try push (263d877928ac)
 - opt crashtest-ipc passed on both win7 and winXP
 - debug failed for the same reasons as in bug 623613 comment 32:

s: talos-r3-w7-004
REFTEST TEST-UNEXPECTED-PASS | file:///c:/talos-slave/test/build/reftest/tests/editor/libeditor/html/crashtests/615015-1.html | assertion count 0 is less than expected 6 assertions
REFTEST TEST-UNEXPECTED-PASS | file:///c:/talos-slave/test/build/reftest/tests/layout/generic/crashtests/586806-1.html | assertion count 0 is less than expected 11 assertions
Bug 590426 - Intermittent 586806-1.html | timed out waiting for reftest-wait to be removed (after onload fired) TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 1648 bytes during test execution
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 1 instance of AtomImpl with size 20 bytes
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 5 instances of FontTableBlobData with size 12 bytes each (60 bytes total)
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 2 instances of PuppetWidget with size 216 bytes each (432 bytes total)
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 1 instance of gfxASurface with size 24 bytes
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 1 instance of gfxFont with size 112 bytes


So the opt windows crashtest-ipc are ready to be enabled! :)

Comment 11

[Armen [:armenzg]]

cjones what is your expected schedule? and your expected priority?
Is this something still being actively worked on? For which version of Firefox do you want this?
I have my hands full so I would like to get things out in the right order depending on urgency.

Comment 12

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

I think this leak has been fixed so we could probably turn these tests on.  I'm not 100% sure though.  Fixing them up is somewhere near the middle of my priority queue.

For FF6 on desktop, this is bug very-good-to-have but not blocking priority.  This is a hard blocker for the FF7 timeframe.

This bug would be useful for fennec, to test the fennec-windows-desktop builds shipped for extension developers, but I can't comment on the priority of that.

Comment 13

[Armen [:armenzg]]

The current load on Windows machines is insane.
What I could do is look into enabling it but with jobs of low priority until we do buy the new testing infrastructure pool that would have more capacity.

Comment 14

[Aki Sasaki (not active)]

Attachment: enable reftest-ipc and crashtest-ipc for windows and mac

Should fix bug 846146, bug 625058, and bug 846145.

Comment 15

[Aki Sasaki (not active)]

Attachment: mac builderlist diff

Comment 16

[Aki Sasaki (not active)]

Attachment: win builderlist diff

linux has no changes.

Comment 17

[Aki Sasaki (not active)]

Comment on attachment 719313 [details] [diff] [review]
enable reftest-ipc and crashtest-ipc for windows and mac

http://hg.mozilla.org/build/buildbot-configs/rev/977a79f18af3

Comment 18

[Aki Sasaki (not active)]

I see Ripc and Cipc for osx and windows in inbound.

Comment 19

[Aki Sasaki (not active)]

Attachment: actually have ipc test configs for win+mac in mozharness

Copying from the linux unittest configs.

Comment 20

[Aki Sasaki (not active)]

Comment on attachment 721447 [details] [diff] [review]
actually have ipc test configs for win+mac in mozharness

http://hg.mozilla.org/build/mozharness/rev/b12acff27f2f

Ran unit.sh, merged to production.

Comment 21

[Aki Sasaki (not active)]

Per Philor, these are largely broken.

Comment 22

[Aki Sasaki (not active)]

Attachment: only schedule win+mac ipc tests on try

I'll attach the builderlist diffs shortly.

Comment 23

[Aki Sasaki (not active)]

Attachment: mac builderlist diff

Just removing ipc mac tests

Comment 24

[Aki Sasaki (not active)]

Attachment: win builderlist diff

Same.

Comment 25

[Aki Sasaki (not active)]

These tests are currently hidden.  Who is going to get them green?

Comment 26

[Aki Sasaki (not active)]

Comment on attachment 721542 [details] [diff] [review]
only schedule win+mac ipc tests on try

http://hg.mozilla.org/build/buildbot-configs/rev/e8077c9ee77d

Comment 27

[Armen [:armenzg]]

Merged and reconfiguration completed.

Comment 28

[Aki Sasaki (not active)]

These are now enabled on Try only.

How should we track fixing these tests on Try?

Comment 29

[Phil Ringnalda (:philor)]

And to make that question even more questionable, between the time you pushed and the time of the reconfig the Mac ones (which were only rather bent, unlike the Windows ones that are totally broken) got greened up, so they're running green and hidden on try.

Comment 30

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage. 

Not clear if this is still needed, or what current state is. Anyone?

Comment 31

[Phil Ringnalda (:philor)]

Status:

Somebody needed to have filed bug 906382 to run Mac crashtest-ipc everywhere, and somebody needs to fix it.

Somebody needed to have filed bug 906383 to green up Mac reftest-ipc, and somebody needs to fix it, so somebody can file a bug to turn on Mac reftests everywhere.

Windows was shut off again on try, so it needs somebody new to care who will file a new turn-it-on-on-try bug, and will green it up.