Closed Bug 564234 Opened 10 years ago Closed 7 years ago

[TopFails] do we track reftest failures?

Categories

(Testing :: General, defect)

x86
macOS
defect
Not set

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: armenzg, Assigned: k0scist)

References

(Blocks 1 open bug)

Details

I don't see any reftest mentions in:
http://brasstacks.mozilla.com/topfails/tests/Firefox
and this makes it difficult for me to see if the perma-oranges for Fedora are gone or not.

Reftest failures look like this:

REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-debug-u-reftest/build/reftest/tests/layout/reftests/box-shadow/boxshadow-inner-basic.html |
REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/mozilla-central-fedora-debug-u-reftest/build/reftest/tests/layout/reftests/bugs/456219-1c.html |
We track mochi, ref and crashtests along with xpcshell.

I am working on few enhancements as we speak in parsing build log. After that, I will clean and rescrape the data and remove Fedora stuff.
 difficult for me to see if the perma-oranges for Fedora are
> gone or not.


Would you like me to permanently ignore any build that starts with "Rev3 Fedora" or "Feroda" ?
Also do you want me to ignore "Rev3 MacOSX" as well ?
I think what you are saying is to scrape everything and start again for all the Rev3 machines?

Leopard had few perma-oranges in the beginning but it has been without perma-oranges for few weeks:
- Rev3 MacOSX Leopard 10.5.8 mozilla-central {opt,debug} test .*

I don't know what the current state of Snow Leopard is but I know that it still has many oranges. We can completely ignore it for now.
- Rev3 MacOSX Snow Leopard 10.6.2 mozilla-central {opt,debug} test .*

Still many perma-oranges for tow of its test suites:
- Rev3 Fedora 12 mozilla-central {opt,debug} test {mochitest-other,reftest}
- Rev3 Fedora 12x64 mozilla-central {opt,debug} test {mochitest-other,reftest}

Does this answer your questions?
Assignee: nobody → mnandigama
Murali, there is a list of machines in tinderbox that are configured to be "ignored" at the moment.  Can you simply read that list and ignore those builds for inclusion by the scraper?  I think that would be a far better solution and when the next set of builds are being worked on and are in an "ignored state" you don't have to update your code to accommodate that. This way when these "rev 3" builds stop being "ignored" they will automatically be included in the topfails analysis.
I have already added proper logic to code to check builds that have noignore=1.
That way I get to ignore all junk builds.
Assignee: murali.nandigama → jhammel
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INVALID
Component: Infrastructure → General
You need to log in before you can comment on or make changes to this bug.