525163 - mochitest-browser-chrome randomly turns tree orange without reporting any failures

Status: RESOLVED WORKSFORME

(Testing :: Mochitest, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256784432.1256787026.4781.gz&fulltext=1
OS X 10.5.2 mozilla-central test everythingelse on 2009/10/28 19:47:12  


In this log, mochitest-browser-chrome reported buildbot status "warnings", but there are no failures logged.

It's not clear to me whether this is a build-automation-side bug or a runtests.py bug; I'm not sure how the status is set.

Comment 1

[Serge Gautherie (:sgautherie)]

Most probably due to bug 473506 "again": I'm trying to check this locally...

Comment 2

[Serge Gautherie (:sgautherie)]

I copied the whole mochitest-browser-chrome log and locally ran
{{
  log = open("steps__in.log", "r")
  failIdent = r"^\t(Fail|Failed): 0"
  if not re.search(failIdent, log.read(), re.MULTILINE) or \
     re.search("^TEST-UNEXPECTED-", log.read(), re.MULTILINE):
      print "WARNINGS"
  else:
      print "PASS"
  log.close()
}}
which passes.

I have no idea about what is going wrong on the box :-(

Comment 3

[Serge Gautherie (:sgautherie)]

[
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256788103.1256790593.11034.gz&fulltext=1
OS X 10.5.2 mozilla-central test opt everythingelse on 2009/10/28 20:48:23

WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!
]
is green.
The test details have a "few" differences though both builds reports '3885/0/5'.
Neither this nor the 'no refcount' warning should make a difference.

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256786847.1256789290.29506.gz&fulltext=1
Linux mozilla-central test opt everythingelse on 2009/10/28 20:27:27
is green too.

Like comment 2, this would mean bug 473506 might not be involved after all.
Yet, I have even less idea what the cause could be:
over to RelEng, unless someone else has an idea.

Comment 4

[Chris Cooper [:coop] (he/him)]

I ran the same script Serge did locally (on Mac) and it passed for me as well.

I've backed out the offending patch in bug 473506 and reconfig-ed the production-masters. We'll try to figure this out in staging.

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to comment #4) 
> I've backed out the offending patch in bug 473506 and reconfig-ed the
> production-masters. We'll try to figure this out in staging.

Er, I meant bug 473259.

Comment 6

[Serge Gautherie (:sgautherie)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256793537.1256795953.4162.gz
OS X 10.5.2 mozilla-central test everythingelse on 2009/10/28 22:18:57
is green.

Then, I would be 95% sure it's unrelated to bug 473506.
And not have any clue about what happened on comment 0 build. :-(

***

(In reply to comment #4)
> I've backed out the offending patch in bug 473506 and reconfig-ed the
> production-masters.

I didn't want that (yet)...

> We'll try to figure this out in staging.

That's more like what I expected (ftb).

Comment 7

[Daniel Holbert [:dholbert]]

This looks like another instance of this:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256797391.1256799289.7491.gz
WINNT 5.2 mozilla-central test opt everythingelse on 2009/10/28 23:23:11

Comment 8

[Daniel Holbert [:dholbert]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256796017.1256798109.27384.gz
OS X 10.5.2 mozilla-central test everythingelse on 2009/10/28 23:00:17
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256796041.1256798139.27702.gz
OS X 10.5.2 mozilla-central test everythingelse on 2009/10/28 23:00:41

Comment 9

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This looks like it's happening on the debug unit tester on Windows too:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256819432.1256823850.30495.gz
WINNT 5.2 mozilla-central test debug everythingelse on 2009/10/29 05:30:32

Comment 10

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

It seems that http://hg.mozilla.org/mozilla-central/rev/2bf3249fe4bb should fix this.

Comment 11

[Serge Gautherie (:sgautherie)]

(In reply to comment #7)
(In reply to comment #8)
(In reply to comment #9)

No, these are a fallout of bug 473259 comment 55 "mess".

(In reply to comment #10)

This additional backout works around the added mess,
but we still have no explanation about comment 0.

Comment 12

[Serge Gautherie (:sgautherie)]

No more reports than the initial one.
I have no idea what caused the latter.
But it looks improbable to be bug 473506, which is now landed on all 3 branches (without complains).

Comment 13

[(not currently active) Ted Mielczarek]

Is this still a problem?

Comment 14

[Phil Ringnalda (:philor)]

No.