Closed Bug 624851 Opened 12 years ago Closed 9 years ago

Enable {crash,ref}test-ipc for Mac machines on Try

Categories

(Release Engineering :: General, defect, P3)

x86
macOS
defect

Tracking

(fennec-)

RESOLVED FIXED
Tracking Status
fennec - ---

People

(Reporter: cjones, Assigned: aki)

References

Details

(Whiteboard: [unittest][new suite])

Attachments

(5 files, 2 obsolete files)

See also bug 623613.  AFAIK we have all the infra support we need for this already, just need to flip the switch to enable these test suites when the platform deps are ready.
Whiteboard: [unittest][new suite]
Forgot to note that for the time being, we'll need to run these tests with GPU rendering *disabled*, because our multi-process gfx code doesn't understand D3D yet.  (This caused lots of problems on tryserver! ;) )
For my records, tests initially failing in my VM

REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/modules/libpr0n/test/reftest/pngsuite-ancillary/ccwn2c08.png | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/modules/libpr0n/test/reftest/pngsuite-ancillary/ccwn3p08.png | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/modules/libpr0n/test/reftest/apng/delaytest.html?bug546272.png | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/modules/libpr0n/test/reftest/generic/accept-image-catchall.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/289480.html#top | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/564991-1.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/579985-1.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/pixel-rounding/background-image-tiling.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/as-image/background-image-rect-1png.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/as-image/list-simple-1.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/image/image-fill-01.svg | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/image/image-svg-inline-01.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/svg/smil/anim-image-href-01.svg | image comparison (==)
(Also for my reference, tests failing in-process in my VM

REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/289480.html#top | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/layout/reftests/bugs/579985-1.html | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/cjones/mozilla/mozilla-central/gfx/tests/reftest/611498-1.html | image comparison (==)

)
Good news:

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1294962549.1294965381.21061.gz
reftest: 5568/20/304 L-FAIL

Of the failures, the vast majority are differences I can't distinguish even on one pixel visually, even zoomed in fully.  Not sure what's up, wouldn't block on those.

One is a legitimate failure probably caused by bug 605618.  Should be easy to fix.

The rest are apparently failures to load the test plugin from the content process.  Not sure what's up with that.  Very much willing to disable those tests, since they're a sideshow of a sideshow.

Not sure what's up with the alleged leak.
Oh, the "leak" is probably

TEST-UNEXPECTED-FAIL | tab process 3284 | automationutils.processLeakLog() | missing output line for total leaks!

We _exit() in content processes when the chrome process shuts down.  Probably need a LogTerm() before doing that, for unit tests.
The reftests are mainly failing for niggly pixel differences due to the duct-tape-and-rubber-band rendering setup.  We're never going to ship out-of-process content with this code, so I don't think it's worth investing time sorting out these failures.

I *do* think it's worth running the crashtests so that we know of big problems caused by platform changes.
Summary: Enable reftests+crashtests for out-of-process content on Windows machines → Enable crashtests for out-of-process content on Windows machines
I think this should block because it's going to be our only automated way to catch broken windows nightlies of fennec, which extension developers are expected to use.
tracking-fennec: --- → ?
Same question as in bug 623613:

Do you want this pref set for the existing reftest / crashtest runs, or do you
want an additional set of runs with this pref set?

Which branches should this be applied to?
This needs to be an additional run for now, say CIPC (crashtest IPC) for short.

Re: branches, let's say
 - m-c
 - tracemonkey
 - places
 - project twigs
 - electrolysis

in that order of importance.
tracking-fennec: ? → 2.0-
Priority: -- → P3
Hardware: x86_64 → x86
On my last try push (263d877928ac)
 - opt crashtest-ipc passed on both win7 and winXP
 - debug failed for the same reasons as in bug 623613 comment 32:

s: talos-r3-w7-004
REFTEST TEST-UNEXPECTED-PASS | file:///c:/talos-slave/test/build/reftest/tests/editor/libeditor/html/crashtests/615015-1.html | assertion count 0 is less than expected 6 assertions
REFTEST TEST-UNEXPECTED-PASS | file:///c:/talos-slave/test/build/reftest/tests/layout/generic/crashtests/586806-1.html | assertion count 0 is less than expected 11 assertions
Bug 590426 - Intermittent 586806-1.html | timed out waiting for reftest-wait to be removed (after onload fired) TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 1648 bytes during test execution
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 1 instance of AtomImpl with size 20 bytes
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 5 instances of FontTableBlobData with size 12 bytes each (60 bytes total)
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 2 instances of PuppetWidget with size 216 bytes each (432 bytes total)
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 1 instance of gfxASurface with size 24 bytes
TEST-UNEXPECTED-FAIL | tab process 1096 | automationutils.processLeakLog() | leaked 1 instance of gfxFont with size 112 bytes


So the opt windows crashtest-ipc are ready to be enabled! :)
cjones what is your expected schedule? and your expected priority?
Is this something still being actively worked on? For which version of Firefox do you want this?
I have my hands full so I would like to get things out in the right order depending on urgency.
OS: Linux → Windows 7
Priority: P3 → P4
I think this leak has been fixed so we could probably turn these tests on.  I'm not 100% sure though.  Fixing them up is somewhere near the middle of my priority queue.

For FF6 on desktop, this is bug very-good-to-have but not blocking priority.  This is a hard blocker for the FF7 timeframe.

This bug would be useful for fennec, to test the fennec-windows-desktop builds shipped for extension developers, but I can't comment on the priority of that.
The current load on Windows machines is insane.
What I could do is look into enabling it but with jobs of low priority until we do buy the new testing infrastructure pool that would have more capacity.
Priority: P4 → P3
Whiteboard: [unittest][new suite] → [unittest][new suite][win7] ffx6 good to have; ffx7 hardblocker
Summary: Enable crashtests for out-of-process content on Windows machines → Enable crashtest-ipc for Windows machines
Attached patch mac builderlist diff (obsolete) — Splinter Review
Attached patch win builderlist diff (obsolete) — Splinter Review
linux has no changes.
Attachment #719313 - Flags: review?(armenzg) → review+
Comment on attachment 719313 [details] [diff] [review]
enable reftest-ipc and crashtest-ipc for windows and mac

http://hg.mozilla.org/build/buildbot-configs/rev/977a79f18af3
Attachment #719313 - Flags: checked-in+
I see Ripc and Cipc for osx and windows in inbound.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Copying from the linux unittest configs.
Attachment #721447 - Flags: review?(armenzg)
Attachment #721447 - Flags: review?(armenzg) → review+
Comment on attachment 721447 [details] [diff] [review]
actually have ipc test configs for win+mac in mozharness

http://hg.mozilla.org/build/mozharness/rev/b12acff27f2f

Ran unit.sh, merged to production.
Attachment #721447 - Flags: checked-in+
Per Philor, these are largely broken.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Enable crashtest-ipc for Windows machines → Enable {crash,ref}test-ipc for Windows and Mac machines on Try
Whiteboard: [unittest][new suite][win7] ffx6 good to have; ffx7 hardblocker → [unittest][new suite]
I'll attach the builderlist diffs shortly.
Attachment #721542 - Flags: review?(armenzg)
Just removing ipc mac tests
Attachment #719314 - Attachment is obsolete: true
Same.
Attachment #719315 - Attachment is obsolete: true
These tests are currently hidden.  Who is going to get them green?
Attachment #721542 - Flags: review?(armenzg) → review+
Merged and reconfiguration completed.
These are now enabled on Try only.

How should we track fixing these tests on Try?
Assignee: nobody → cjones.bugs
And to make that question even more questionable, between the time you pushed and the time of the reconfig the Mac ones (which were only rather bent, unlike the Windows ones that are totally broken) got greened up, so they're running green and hidden on try.
Found in triage. 

Not clear if this is still needed, or what current state is. Anyone?
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee
Product: mozilla.org → Release Engineering
Depends on: 906382
Depends on: 906383
Status:

Somebody needed to have filed bug 906382 to run Mac crashtest-ipc everywhere, and somebody needs to fix it.

Somebody needed to have filed bug 906383 to green up Mac reftest-ipc, and somebody needs to fix it, so somebody can file a bug to turn on Mac reftests everywhere.

Windows was shut off again on try, so it needs somebody new to care who will file a new turn-it-on-on-try bug, and will green it up.
Assignee: cjones.bugs → aki
Status: REOPENED → RESOLVED
Closed: 10 years ago9 years ago
OS: Windows 7 → Mac OS X
Resolution: --- → FIXED
Summary: Enable {crash,ref}test-ipc for Windows and Mac machines on Try → Enable {crash,ref}test-ipc for Mac machines on Try
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.