564234 - [TopFails] do we track reftest failures?

Status: RESOLVED INVALID

(Testing :: General, defect)

Description

[Armen [:armenzg]]

I don't see any reftest mentions in:
http://brasstacks.mozilla.com/topfails/tests/Firefox
and this makes it difficult for me to see if the perma-oranges for Fedora are gone or not.

Reftest failures look like this:

REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-debug-u-reftest/build/reftest/tests/layout/reftests/box-shadow/boxshadow-inner-basic.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-debug-u-reftest/build/reftest/tests/layout/reftests/bugs/456219-1c.html |

Comment 1

[Murali Nandigama [:murali]]

We track mochi, ref and crashtests along with xpcshell.

I am working on few enhancements as we speak in parsing build log. After that, I will clean and rescrape the data and remove Fedora stuff.

Comment 2

[Murali Nandigama [:murali]]

 difficult for me to see if the perma-oranges for Fedora are
> gone or not.


Would you like me to permanently ignore any build that starts with "Rev3 Fedora" or "Feroda" ?

Comment 3

[Murali Nandigama [:murali]]

Also do you want me to ignore "Rev3 MacOSX" as well ?

Comment 4

[Armen [:armenzg]]

I think what you are saying is to scrape everything and start again for all the Rev3 machines?

Leopard had few perma-oranges in the beginning but it has been without perma-oranges for few weeks:
- Rev3 MacOSX Leopard 10.5.8 mozilla-central {opt,debug} test .*

I don't know what the current state of Snow Leopard is but I know that it still has many oranges. We can completely ignore it for now.
- Rev3 MacOSX Snow Leopard 10.6.2 mozilla-central {opt,debug} test .*

Still many perma-oranges for tow of its test suites:
- Rev3 Fedora 12 mozilla-central {opt,debug} test {mochitest-other,reftest}
- Rev3 Fedora 12x64 mozilla-central {opt,debug} test {mochitest-other,reftest}

Does this answer your questions?

Comment 5

[cmtalbert]

Murali, there is a list of machines in tinderbox that are configured to be "ignored" at the moment.  Can you simply read that list and ignore those builds for inclusion by the scraper?  I think that would be a far better solution and when the next set of builds are being worked on and are in an "ignored state" you don't have to update your code to accommodate that. This way when these "rev 3" builds stop being "ignored" they will automatically be included in the topfails analysis.

Comment 6

[Murali Nandigama [:murali]]

I have already added proper logic to code to check builds that have noignore=1.
That way I get to ignore all junk builds.