603539 - Mac builds sometimes exceed the current leakFailureThreshold, unrelated to the patch that triggered the build

Status: RESOLVED DUPLICATE of bug 603015

(Testing :: General, defect)

Description

[Phil Ringnalda (:philor)]

Attachment: moar magic

Two recent Mac builds, http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1286571025.1286572241.15631.gz and http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1286839556.1286843321.3555.gz have exceeded the leakThreshold that http://hg.mozilla.org/build/buildbotcustom/file/1351003e0957/steps/test.py#l203 uses to decide whether or not to warn about an increase in the Lk: number.

That's really ugly orange, since nobody (other than dbaron) even knows how to find why the log was orange, and nobody (other than possibly bhearsum, who apparently ported it from somewhere I can't find in CVS, in bug 422296) knows why the magic number 7261838 is a problem to exceed, when typical numbers are ~2MB on Mac, 1MB on Linux, and 400KB on Windows, and particularly since the first "failure" was on a patch that added a line of text to license.html, so now that we've failed, nobody has any idea why (or even what), since it's clearly unrelated to anything. (And then there's the bug 603015 ugliness, where a build which does not exceed the threshold will be orange only because the build before it did exceed it.)

We could file a bug in Core: General about how at utterly random times we exceed a magic number, which we would certainly, unquestionably, ignore completely, or we can just diminish our expectations, and pick a new higher magic number.

Comment 1

[[e-mail] (:bhearsum)]

Comment on attachment 482443 [details] [diff] [review]
moar magic

Is there a bug or other place you can link to that explains why it's set to what it is?

r=me if dbaron thinks this is the correct threshold.

Comment 2

[Phil Ringnalda (:philor)]

See the second clause of the second paragraph in comment 0: to my eyes, the number sprang into being while you were porting leak tests from whatever ran them in the pre-buildbot days. Do you remember what that was, what repo it would have been in, and whether or not that was mxr'ed? I couldn't find it in a half hour of looking last night, but if I ever knew how that code worked and was arranged, I've forgotten since.

Comment 3

[[e-mail] (:bhearsum)]

http://mxr.mozilla.org/mozilla/source/tools/tinderbox/ is where it would've been in a past life, as far as I know.

I was more wondering if there was a bug that caused the increase, but now that I've read comment #0, I see that there isn't.

Comment 4

[Phil Ringnalda (:philor)]

Uh oh. This needs a morph, and it's probably going to be beyond my can't-actually-test-it abilities.

http://mxr.mozilla.org/mozilla/source/tools/tinderbox/tinder-defaults.pl#88 did indeed have a $LeakFailureThreshold, which http://mxr.mozilla.org/mozilla/source/tools/tinderbox/build-seamonkey-util.pl#3493 used, to tell whether *RLk* had passed the threshold. We're supposed to keep RLk at 0 (which I don't think we're currently testing at all), not supposed to keep Lk below some large number.

Comment 5

[Phil Ringnalda (:philor)]

Actually, not as bad as I thought: CompareBloatLogs does check for RLk > 0, though if it's possible to put something in the log that would catch the errorparser's attention before http://hg.mozilla.org/build/buildbotcustom/file/31a22bd3816e/steps/test.py#l157 that would be a whole lot nicer.

But since dbaron was at first pretty sure that we didn't have anything that would turn a build orange if Lk increased, and we don't seem to have had any such thing in the old days, and we've apparently never hit it before, and because of the intermittent huge numbers on Mac we have to have a threshold that's something like 18 times the normal Windows numbers, it looks to me like we ought to just rip out all the leakFailureThreshold stuff in CompareLeakLogs, rather than going to all the trouble of adding per-platform thresholds, and telling it that not everything is resultSet = 'new', and giving it a message besides "warnings", for something we neither trust nor particularly want.

Comment 6

[Phil Ringnalda (:philor)]

But since we just removed RLk, which I thought was the one true number, I pretty clearly have absolutely no idea what to do here.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Is this bug related (or perhaps a dupe of) bug 603015?

Comment 8

[Phil Ringnalda (:philor)]

If either one gets morphed into "remove CompareLeakLogs and its call sites" then the other is a dupe of the one which lands the patch and gets marked fixed, yeah. Otherwise, that depends on this because nobody reasonable would fix that before finding out what this is going to do.

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1287698915.1287703197.18660.gz

Comment 9

[Phil Ringnalda (:philor)]

Starting to rethink obsoleting that sweep it under the carpet for now with a higher threshold patch, though.

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1287911512.1287913439.30551.gz

Comment 10

[Mike Taylor [:bear]]

If the possible solutions all appear to be tweaking of either the test or the threshold, then I'm punting to the test group for their input.

Comment 11

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1291754965.1291756776.19933.gz
OS X 10.5.2 mozilla-central leak test build

Comment 12

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1292187719.1292189287.30249.gz

Comment 13

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1294337202.1294340865.14375.gz

Comment 14

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1294442328.1294444543.31575.gz

Comment 15

[Phil Ringnalda (:philor)]

or maybe I mean http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1294439664.1294444807.32755.gz, I can't tell current from previous any better than CompareLeakLogs can.

Comment 16

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1294700461.1294702774.18854.gz

Comment 17

[Phil Ringnalda (:philor)]

Amusingly enough, comment 16 is not this, it's a genuine leak exceeding the threshold, but this cries wolf so much I didn't believe it.

Comment 18

[Phil Ringnalda (:philor)]

http://hg.mozilla.org/mozilla-central/rev/2147912ec8e1

Comment 19

[Matt Brubeck (:mbrubeck)]

http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Aurora/1302902069.1302905708.10749.gz