554934 - (fedora-oranges) [tracking bug] Fedora unit tests oranges

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P4)

Description

[Armen [:armenzg]]

I am going to give a stab at the failures seen on staging for running unit test on Fedora talos machines.

I believe ctalbert is interested on helping with this and I CCed him and will be in touch with him.

First a list of the overall situation:
- GREEN  - Fedora 12 mozilla-central opt test mochitests-1/5
- GREEN  - Fedora 12 mozilla-central opt test mochitests-2/5 (last build was orange)
- GREEN  - Fedora 12 mozilla-central opt test mochitests-3/5
- ORANGE - Fedora 12 mozilla-central opt test mochitests-4/5
- GREEN  - Fedora 12 mozilla-central opt test mochitests-5/5
- ORANGE - Fedora 12 mozilla-central opt test mochitest-other
- ORANGE - Fedora 12 mozilla-central opt test reftest
- GREEN  - Fedora 12 mozilla-central opt test crashtest
- ORANGE - Fedora 12 mozilla-central opt test xpcshell
- GREEN  -  Fedora 12 mozilla-central opt test jsreftest

Now I will break down each one of the oranges that we have.

Comment 1

[Armen [:armenzg]]

> - GREEN  - Fedora 12 mozilla-central opt test mochitests-2/5 (last build was
> orange)
Nothing to worry about this one time orange. Moving on to next builder.

remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]

Comment 2

[Armen [:armenzg]]

Attachment: mochitests 4/5 - tests/layout/generic/test/test_backspace_delete.xul

Comment 3

[Armen [:armenzg]]

Attachment: mochitest-chrome - chrome://mochikit/content/chrome/toolkit/content/tests/chrome/test_autocomplete2.xul

This is broken down into:
- ORANGE - Fedora 12 mozilla-central opt test mochitest-other
   * ORANGE - mochitest-chrome
   * ORANGE - mochitest-browser-chrome
   * ORANGE - mochitest-a11y
   * GREEN  - mochitest-ipcplugins

The attachment if for the first one.

Comment 4

[Armen [:armenzg]]

Attachment: mochitest-browser-chrome - chrome://mochikit/content/browser/browser/components/places/tests/browser/browser_library_panel_leak.js & browser-test.js

There are two failures in here. First one is a test failure like the previous ones and the second one is a time-out for browser-test.js.

I have checked few logs and I also got intermittent time outs forbrowser_privatebrowsing_beforeunload.js. Note that this time out happens just before our usual browser-test.js time out.

Comment 5

[Armen [:armenzg]]

Attachment: mochitest-a11y - chrome://mochikit/content/a11y/accessible/attributes/test_text.html

Comment 6

[Armen [:armenzg]]

Attachment: reftest - many failures

Comment 7

[Armen [:armenzg]]

Attachment: xpcshell - test_jsctypes.js and xpcshell/head.js

Comment 8

[Armen [:armenzg]]

I have gone through all the test suites and I will verify that if I run it manually I will hit the same failures.

At first my guts tell me that these are all valid failures since we have never run unit tests on Fedora. That Firefox behaves as expected on CentOS doesn't mean that it should behave correctly on Fedora. Anyways, I will leave conclusion for later.

Comment 9

[Armen [:armenzg]]

Attachment: talos-rev3-fed-001's /etc/X11/xorg.conf

Comment 10

[cmtalbert]

(In reply to comment #6)
> Created an attachment (id=434882) [details]
> reftest - many failures

I'll apologize up front for the spam, but it's easiest to look at these log by log.

Looking over the reftests, it looks like most of these are anti-ailiasing and/or color depth issues.  I see small pixel differences on corners, especially rounded corners is the problem with most of these tests.  I'd make sure that the xvfb settings between fedora and centos are exactly identical.  However, there is a whole class of tests that should fail if those are different and none of those are in this log.  So, it might be something even more small than that.

The really big issue here are the font-face tests that have failed.  These look like the fonts didn't download at all.  My first thought was that it had issues connecting to the local webserver to download the font.  However, if it had blanket issues getting to that server we'd have seen many more of the font tests failing.  Perhaps we are conflicting here with an installed font on the Fedora system that the centos system didn't have?

TODOs:
* verify your color settings/resolution
* verify installed fonts, that you can actively hit the local httpd.js webserver

Comment 11

[cmtalbert]

(In reply to comment #7)
> Created an attachment (id=434883) [details]
> xpcshell - test_jsctypes.js and xpcshell/head.js

This test is failing to load a specific system library.  Perhaps libm.so is called something else on fedora or is not in the path?
See:
http://mxr.mozilla.org/mozilla-central/source/js/ctypes/tests/unit/test_jsctypes.js.in#1786

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

We are going to try adding
    DefaultDepth  24
to the xorg.conf Screen section as a diagnostic

Comment 13

[cmtalbert]

(In reply to comment #5)
> Created an attachment (id=434879) [details]
> mochitest-a11y -
> chrome://mochikit/content/a11y/accessible/attributes/test_text.html

Ok, here's a clue as to what's happening with reftest.  If you go look at this test: http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/attributes/test_text.html?force=1 you will see that this thing really doesn't do a good job of setting a default known state before tickling a bunch of font properties that all will be affected by the system's default settings.  

Digging some ancient (and possibly incorrect) information out of my head, I recall there is a default CSS used for each OS in the browser and in lieu of any user CSS direction those values are used.  They are used to set things like default font, font size, font family etc.  So, if those default settings aren't what they are on centos then this test could do something unexpected.  This one is really a fault of the test because it doesn't appear (in my quick scan) to set all the defaults to the known states that it is expecting.  If we were to change those default CSS values then we'd also break this test, I think.

And that might be related to what's going on with those reftest font-face tests.

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #11)
> (In reply to comment #7)
> > Created an attachment (id=434883) [details] [details]
> > xpcshell - test_jsctypes.js and xpcshell/head.js
> 
> This test is failing to load a specific system library.  Perhaps libm.so is
> called something else on fedora or is not in the path?
> See:
> http://mxr.mozilla.org/mozilla-central/source/js/ctypes/tests/unit/test_jsctypes.js.in#1786

$ yum -q whatprovides */libm.so
glibc-devel-2.11-2.i686 : Object files for development using standard C
                        : libraries.
Repo        : fedora
Matched from:
Filename    : /usr/lib/libm.so

$ rpm -qi glibc-devel
package glibc-devel is not installed

It looks like we do not have libm.so on the system.  Instead of installing the development package, we could copy that library from the rpm using yumdownloader and rpm2cpio.

Comment 15

[cmtalbert]

(In reply to comment #4)
> Created an attachment (id=434876) [details]
> mochitest-browser-chrome -
> chrome://mochikit/content/browser/browser/components/places/tests/browser/browser_library_panel_leak.js
> & browser-test.js
> 
> There are two failures in here. First one is a test failure like the previous
> ones and the second one is a time-out for browser-test.js.
> 
> I have checked few logs and I also got intermittent time outs
> forbrowser_privatebrowsing_beforeunload.js. Note that this time out happens
> just before our usual browser-test.js time out.
I'll bet you're hitting a version of bug 541404 here.

Comment 16

[cmtalbert]

(In reply to comment #2)
> Created an attachment (id=434871) [details]
> mochitests 4/5 - tests/layout/generic/test/test_backspace_delete.xul

Looks like bug 474068

Comment 17

[Jonathan Kew [:jfkthame]]

(In reply to comment #10)
> The really big issue here are the font-face tests that have failed.  These look
> like the fonts didn't download at all.

No, the @font-face failures I see in the log are:
  src-list-local-full.html
  src-list-local-full-quotes.html
  src-list-local-fallback.html
These are all tests that do NOT download fonts, they use @font-face to access locally-installed fonts on the host system. These are inherently somewhat fragile as they depend on choosing font names that are actually available. The tests try several names in order to cover fonts that are expected on Mac, Windows, and some Linux systems, but apparently they're not finding anything on the Fedora boxes.

The fix is to identify a suitable font name that IS available on the Fedora boxes and add this to the list of font names to try; maybe DejaVu Sans would work, if they don't have FreeSans or Bitstream Vera Sans. Could someone verify what fonts are available there? (E.g., run "fc-list | sort" and report what it lists.)

Comment 18

[Armen [:armenzg]]

As per meeting, I am putting this bug on the side for few days since I have to help out on getting Snow Leopard and Leopard out first.

Clint, the Fedora machine will still be reporting to http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest in case there is some tests that you are going to start fixing.

We will get to the depth issues as soon as we can.
Thanks for looking into this!

Comment 19

[Armen [:armenzg]]

Attachment: Fedora's fonts

Jonathan here is the list of fonts.

For things as easy as running commands for you I can do it easily but for other more dedicated tasks I will take on it next week.

Let me know if there is anything I can run for you in the machine.

Comment 20

[cmtalbert]

(In reply to comment #17)
> (In reply to comment #10)
Ah thanks, Jonathan, I only looked at these quickly.  Then this further supports my hypothesis w.r.t. that a11y test that the basic defaults available on Fedora are not our usual defaults that we are expecting.

Comment 21

[Armen [:armenzg]]

Status update per ORANGE suites

(In reply to comment #0)
> - mochitests-4/5
       (In reply to comment #16)
       Looks like bug 474068
> - mochitest-other
>  * mochitest-chrome
>  * mochitest-browser-chrome
       (In reply to comment #15)
        I'll bet you're hitting a version of bug 541404 here.
>  * mochitest-a11y
> - reftest
    (In reply to comment #17)
    > The fix is to identify a suitable font name that IS available on the Fedora
    > boxes and add this to the list of font names to try;
> - xpcshell

We have 6 suites that are permanent ORANGE, we have identified 2 possible bugs (added as dependency) and 1 bug to be file (the fonts one - can we please file bug?).

Clint and Jonathan, what is the approach we should have for this bug to move forward?
What can releng do on our side to fix these oranges?
Can I change the summary of this bug to "tracking bug for Fedora unit tests oranges"?

Comment 22

[Armen [:armenzg]]

Any updates on this?

Comment 23

[cmtalbert]

(In reply to comment #5)
> Created an attachment (id=434879) [details]
> mochitest-a11y -
> chrome://mochikit/content/a11y/accessible/attributes/test_text.html

Dbolter, MarcoZ: can you comment on this test?  The part of the test that is failing is default font size of the input control (area 8).  In this line: http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/attributes/test_text.html?force=1#65 you state that it should be 8pt on MAC, 10pt on all other platforms.  Is it possible that on Fedora, this defaults to 8pt as well?  I'm not exactly sure how to correct this problem without breaking your test.  We don't have a check for specific versions of linux here, and I'm worried that setting a specific font style on the control itself invalidates what the test is trying to do.

Comment 24

[cmtalbert]

Attachment: Patch to fix the reftests

Jonathan,  would you mind taking a look at this?  I think this should fix the failing reftests here.  I ran it on Windows and it worked, I'm still building on my mac.

Armen: I don't have access to a Fedora box, so if you'd be so kind as to try this out there and see if it fixes things, I'd appreciate your feedback too.

Comment 25

[Jonathan Kew [:jfkthame]]

Comment on attachment 437434 [details] [diff] [review]
Patch to fix the reftests

I don't have a fedora box to actually test this personally, but the patch looks fine to me. Provided Fedora has those DejaVu fonts, it ought to do the job.

Comment 26

[cmtalbert]

(In reply to comment #25)
> (From update of attachment 437434 [details] [diff] [review])
> I don't have a fedora box to actually test this personally, but the patch looks
> fine to me. Provided Fedora has those DejaVu fonts, it ought to do the job.

They were in the font list Armen provided, so let's cross our fingers.  I've pushed this reftest fix to m-c, tests passed locally on windows and mac, and we'll see if they make it on Fedora when Armen pushes to staging later next week.

Changeset: e24d24f3ceed

Comment 27

[alice nodelman [:alice] [:anode]]

Fedora unit tests are now on mozilla-central and burning, so I'd say that the fix didn't work. :)

Comment 28

[Armen [:armenzg]]

I have disabled the unit tests that were red (11 of the tests suites from both platforms). At first, it seems that they are failing at an automation step (not the test steps) and that brings the redness.
Lukas filed bug 558240 for debugging it.

(In reply to comment #27)
> Fedora unit tests are now on mozilla-central and burning, so I'd say that the
> fix didn't work. :)
Clint you will have to wait for me to fix bug 558240 to see if the patch fixed one of the many reftests issues.

Comment 29

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #28)
> Clint you will have to wait for me to fix bug 558240 to see if the patch fixed
> one of the many reftests issues.

No need to wait; look at http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox&noignore=1


I've ensured that the builds that are perma-orange are hidden:
http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/thread/30a90df5d56a78e9

But I re-showed the ones that are red the first cycle only, since that should go away after another few cycles.

Comment 30

[(no longer active)]

If I can get access to one of the Fedora boxes on which bug 541404 is perma-orange, I can gelp debug it.

Comment 31

[Armen [:armenzg]]

(In reply to comment #30)
> If I can get access to one of the Fedora boxes on which bug 541404 is
> perma-orange, I can gelp debug it.

I will help you with that.

Comment 32

[Armen [:armenzg]]

Status update:

* We have 4 tests suites hidden on tinderbox: M0, M4, Reftest & Xpcshell
** This is for Fed32/Fed64 & opt/debug builds
* I have filed bugs for all permanent oranges that are seen
* I would like to enable these hidden tests now that we have everything filed.

Comment 33

[(no longer active)]

It would be great if someone who monitors the fedora boxes can verify if the issue in bug 541404 has really been fixed on those machines, now that the patch for that bug landed.  I checked the test run on that revision, and the problem wasn't showing up.

Comment 34

[Phil Ringnalda (:philor)]

Doesn't update in realtime (rumor has it that it's twice a day), but http://brasstacks.mozilla.com/topfails/test?name=chrome://mochikit/content/browser/browser/components/privatebrowsing/test/browser/browser_privatebrowsing_openlocation.js would be happy to monitor that for you.

Comment 35

[Armen [:armenzg]]

Status update:

* M4:
 * hidden because of bug 474068 (pango bug)
 * I have requested the test to be marked as KNOWN-FAIL
* MO:
 * mochitest-chrome:
  * bug 558908 (test_autocomplete2.xul)
  * no one assigned
 * mochitest-browser-chrome:
  * only 1 unexpected fail
  * bug 560878 (browser_bug435788.js)
 * mochitest-a11y:
  * bug 560888 (test_text.html)
* Reftest:
 * bug 560882. Tracking bugs for reftests. Too many.
 * I want to move all reftest efforts from QA/devs into that bug
* Xpcshell:
 * bug 558911 (libm.so)
 * My next bug to work on


Intermittent oranges:
* M2:
 * bug 558697 and bug 558699. test_Prototype.html. Affects Fed12X64 so far from comments
 * no one assigned
* debug M1:
 * bug 560784
 * this happened only once http://talos-master.mozilla.org:8012/builders/Rev3%20Fedora%2012%20mozilla-central%20debug%20test%20mochitests-1%2F5/builds/173

Comment 36

[Armen [:armenzg]]

Status update:

 * mochitest-browser-chrome:
   * Only affects 32 bit machines and the optimized build
   * bug 560878 (browser_bug435788.js)
 * mochitest-a11y:
   * bug 560888 (test_text.html)
 * reftests - tracking bug 560882  
   * Bug 558910 - antialiasing
   * Bug 558977 - reftests/svg/objectBoundingBox-and-fePointLight-01.svg
   * Bug 558981 - reftests/text-decoration/underline-block-propagation-2-standards.html
   * Bug 559125 - reftests/bugs/180085-1.html and reftests/bugs/359903-1.html
   * Bug 559426 - layout/reftests/box-shadow/boxshadow-inner-basic.html and reftests/bugs/456219-1c.html

Fixed since last time:
 * M4 - bug 474068
 * MO - mochitest-chrome - bug 558908
 * Xpcshell - bug 558911

Intermittent oranges:
 * bug 558697
 * bug 558699
 * bug 560784

Comment 37

[Marco Bonardo [:mak]]

Armen, is browser_library_panel_leak.js still an issue? i don't see it filed. IF it is you could file a bug and try to assign it to me.

Comment 38

[Armen [:armenzg]]

Hi Marco,
I have checked around and it is only an issue for:
- Rev3 Fedora 12x64 mozilla-central opt test mochitest-other

oooooo I like this topfails thingy:
http://brasstacks.mozilla.com/topfails/test/Firefox?name=chrome://mochikit/content/browser/browser/components/places/tests/browser/browser_library_panel_leak.js

I have filed bug 562663 to track this.

Comment 39

[Armen [:armenzg]]

Status update:

* mochitest-browser-chrome:
  * bug 560878 (browser_bug435788.js)
  * bug 562663
* mochitest-a11y:
  * bug 560888 (test_text.html)
* reftests - tracking bug 560882  
  * Bug 558910 - antialiasing
  * Bug 558977 - reftests/svg/objectBoundingBox-and-fePointLight-01.svg
  * Bug 559125 - reftests/bugs/180085-1.html and reftests/bugs/359903-1.html
  * Bug 559426 - layout/reftests/box-shadow/boxshadow-inner-basic.html and reftests/bugs/456219-1c.html
 
Fixed since last time:
* reftests:
  * bug 558981

Intermittent oranges:
* bug 558697
* bug 558699
* bug 560784

Comment 40

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #39)
> Intermittent oranges:
> * bug 558697
> * bug 558699
> * bug 560784

These three appear to be intermittent oranges in tests that are already enabled, so they don't block turning anything on.  They're typical (relatively low frequency) random oranges, and don't appear to block anything as far as getting the Fedora 12 unit tests fully enabled.

Comment 41

[Armen [:armenzg]]

(In reply to comment #40)
> (In reply to comment #39)
> > Intermittent oranges:
> > * bug 558697
> > * bug 558699
> > * bug 560784
> 
> These three appear to be intermittent oranges in tests that are already
> enabled, so they don't block turning anything on.  They're typical (relatively
> low frequency) random oranges, and don't appear to block anything as far as
> getting the Fedora 12 unit tests fully enabled.

That's correct. I just drag them over in case anybody wonders about "what are the other bugs that you have listed as dependent?"

Comment 42

[Alex Vincent [:WeirdAl]]

FYI, please note bug 403783 - this has a known unit-test failure for Fedora, which was disabled soon after check-in.

Comment 43

[Alex Vincent [:WeirdAl]]

Disregard my previous comment - that was CentOS, not Fedora.  Sorry for the bugspam. :(

Comment 44

[Armen [:armenzg]]

Status update:

=== BLOCKERS ===
* mochitest-browser-chrome:
  * bug 487717 - browser_bug435788.js times out (bug 560878 got DUPED)
  * bug 562663 - chrome://mochikit/content/browser/browser/components/places/tests/browser/browser_library_panel_leak.js
* reftests - tracking bug 560882  
  * bug 558910 - antialiasing
  * bug 559426 - layout/reftests/box-shadow/boxshadow-inner-basic.html and
reftests/bugs/456219-1c.html
  * *unfortunately* I see many more reftest failures that I will have to file

=== Times outs (intermitents) ===
* mochitest-browser-chrome:
  * bug 487717 - browser_bug435788.js times out (bug 560878 got DUPED)

Fixed since comment #39:
* mochitest-a11y:
  * bug 560888 (test_text.html)
* reftests:
  * bug 558977 - reftests/svg/objectBoundingBox-and-fePointLight-01.svg
  * bug 559125 - reftests/bugs/180085-1.html and reftests/bugs/359903-1.html

Intermittent oranges:
* bug 558697
* bug 558699
* bug 560784

Comment 45

[John Daggett (:jtd)]

The reftests patch attached here is not correct, you need to adjust the reference pages also to include Deja Vu Sans.

Comment 46

[Armen [:armenzg]]

Status update:

=== perma-orange ===
* mochitest-browser-chrome:
  * bug 562663 - [test patch]          browser/browser_library_panel_leak.js
* reftests - tracking bug 560882  
  * bug 558910 - [what right plan is?] antialiasing (.fonts.conf)
  * bug 559426 - [test patch]          box-shadow/boxshadow-inner-basic.html 
  * bug 564274 - [test patch]          456219-1{a,b,c}.html

=== Times outs (frequent) ===
* mochitest-browser-chrome:
  * bug 487717 - [extra logging landed] browser_bug435788.js times out
 
Fixed since comment #44:
* Intermittent orange:
  * bug 558697

Intermittent oranges:
 * bug 558699
 * bug 560784

Comment 47

[Armen [:armenzg]]

Hidden builders:
* [few greens]   Rev3 Fedora 12x64 mozilla-central opt test mochitest-other
* [perma-orange] Rev3 Fedora 12x64 mozilla-central opt test reftest
* [perma-orange] Rev3 Fedora 12x64 mozilla-central debug test reftest
* [few greens]   Rev3 Fedora 12 mozilla-central opt test mochitest-other
* [perma-orange] Rev3 Fedora 12 mozilla-central opt test reftest
* [perma-orange] Rev3 Fedora 12 mozilla-central debug test reftest

Made visible since last time:
* Rev3 Fedora 12x64 mozilla-central debug test mochitest-other
* Rev3 Fedora 12 mozilla-central debug test mochitest-other

Comment 48

[Armen [:armenzg]]

And this bug (it was in the dependency but not my update summary):
* bug 520384 - [no patch] places/tests/browser/browser_library_infoBox.js
   * http://brasstacks.mozilla.com/topfails/test/Firefox?name=chrome://mochikit/content/browser/browser/components/places/tests/browser/browser_library_infoBox.js
   * affects both Fedora 32 and 64 bit
   * this is intermittent
   * it affects mochitest-browser-chrome

Comment 49

[Armen [:armenzg]]

* sporadic bug:
  * bug 527099

Comment 50

[Armen [:armenzg]]

* sporadic crash:
  * bug 505016 - nsXPCWrappedJS::QueryInterface

Comment 51

[Daniel Holbert [:dholbert]]

Filed Bug 565458 on a Fedora 12 timeout in browser_privatebrowsing_beforeunload.js.

Comment 52

[(no longer active)]

(In reply to comment #51)
> Filed Bug 565458 on a Fedora 12 timeout in
> browser_privatebrowsing_beforeunload.js.

FWIW, I don't think that bug 565458 is a Fedora specific issue.  It looks a lot like bug 543278.  I posted a patch on that bug which _might_ fix this problem.

Comment 53

[Armen [:armenzg]]

All test-suites for Fedora are fully enabled and visible. There are no more perma-oranges and we are now in good shape to start disabling unit tests on CentOS on mozilla-central.

Status update:

perma-oranges ====> ZARRO!  

Intermittent/timeouts:
 * bug 487717 - browser_bug435788.js times out
 * bug 560784 - test_playback.html | bug461281.ogg
 * bug 527099 - sporadic timeout in dom/tests/mochitest/chrome/test_focus.xul
 * bug 505016 - nsXPCWrappedJS::QueryInterface
 * bug 565458 - Timeout - browser_privatebrowsing_beforeunload.js

Fixed since comment #46:
* mochitest-browser-chrome:
  * bug 562663 - browser/browser_library_panel_leak.js
  * bug 520384 - [no patch] places/tests/browser/browser_library_infoBox.js
* reftests
  * bug 558910 - antialiasing (.fonts.conf)
  * bug 559426 - box-shadow/boxshadow-inner-basic.html 
  * bug 564274 - 456219-1{a,b,c}.html
* Intermittent orange:
  * bug 558699

Comment 54

[Armen [:armenzg]]

Nothing for me to do in this tracking bug besides just tracking the intermittent bugs.