437391 - 1700% Tgfx regression on all linux boxes

Status: RESOLVED INCOMPLETE

(Core Graveyard :: GFX, defect)

Description

[Dave Townsend [:mossop]]

On Mozilla2 all linux talos boxes show a massive regression in Tgfx. No other platforms seem to and there appears to be just one changeset in the range that simply adjusts a reftest: http://hg.mozilla.org//mozilla-central/index.cgi/rev/1a969abc8556

Comment 1

[alice nodelman [:alice] [:anode]]

Remember that regression ranges should go back a few hours from the reported time of the suspicious point on the graph server (to be fixed with bug 419487).

I have not made any changes to these machines once they were pushed to the main mozilla-central waterfall.  I just had a quick look at them and they appear to be in a normal state (correct settings, responsive, etc).

Comment 2

[Dave Townsend [:mossop]]

I was basing my reading on the starttime of the talos run which I assumed still lines up with the start time of the build it is testing on the tinderbox. Is this no longer the case? If not I don't have a clue how to work out when the regression might have occured.

Comment 3

[alice nodelman [:alice] [:anode]]

The time reported to the graph server for talos tests has _always_ been talos test start time, not build start time - which is why we ended up with bug 419487.

Comment 4

[Dave Townsend [:mossop]]

I meant the start time as reported to the tinderbox, which always mystically lines up with the start time of a build on the main builder for that platform and does not match the time on the graphserver display.

Comment 5

[alice nodelman [:alice] [:anode]]

Okay, sorry for the confusion.  Individual talos runs line up with the build start time for the build that they are testing - that is correct.

Comment 6

[alice nodelman [:alice] [:anode]]

These numbers are still elevated for linux mozilla-central tgfx.  I'm not seeing any increase in ts/tp for the same machines.  I would expect some generalized problem with the numbers if the machine configuration was incorrect.

On the talos side I'm not sure if there is much more that I can do.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

All other talos machines are looking good here, which makes us feel this is not a talos infrastructure problem. 

What code changes were landed in this regression window? Could any of those changes have caused this regression?

Comment 8

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

Assuming that the talos builds are lining up correctly in the tinderbox display with the machines that produced the builds, then the last known good changeset was http://hg.mozilla.org//mozilla-central/index.cgi/rev/b0bc6be7db45 and the first known bad changeset was http://hg.mozilla.org//mozilla-central/index.cgi/rev/1a969abc8556

Comment 9

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

Which gives exactly one commit in the regression window:

changeset:   15238:1a969abc8556
user:        L. David Baron <[e-mail]>
date:        Tue Jun 03 18:15:05 2008 -0700
summary:     Mark reftest added for bug 433640 as failing on Mac, due to bug 379317.

A change to a reftest manifest isn't going to cause a performance regression.  So I think that makes it clear that either:
 * the talos builds are not lining up correctly, or
 * this is an infrastructure problem.

Comment 10

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

You can see the relevant tinderbox display, with a decent window around both sides, at http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla2&maxdate=1212555600

Comment 11

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

I'd note that it would take only one talos run lining up incorrectly by one build to add the following (earlier) changesets into the regression window:

changeset:   15234:553480dceb26
user:        Vladimir Vukicevic <[e-mail]>
date:        Tue Jun 03 17:00:32 2008 -0700
summary:     b=430906; add moz-opaque attribute to <canvas>; r+sr=roc

changeset:   15235:be00592555a0
user:        Vladimir Vukicevic <[e-mail]>
date:        Tue Jun 03 17:00:37 2008 -0700
summary:     b=433086, improve nsDOMWindowUtils::Redraw; r+sr=roc

changeset:   15236:378707d75b14
user:        Vladimir Vukicevic <[e-mail]>
date:        Tue Jun 03 17:01:27 2008 -0700
summary:     b=423913; check for -fshort-wchar correctness; r=bsmedberg

changeset:   15237:b0bc6be7db45
user:        Vladimir Vukicevic <[e-mail]>
date:        Tue Jun 03 17:01:32 2008 -0700
summary:     b=426563, put arm xptstubs shared assembly correctly in text section; r=bsmedberg

These seem a good bit more likely.

Comment 12

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

In particular, after following http://wiki.mozilla.org/Mozilla2:Trender to bug 313812 to view-source:https://bugzilla.mozilla.org/attachment.cgi?id=201407 it looks like nsDOMWindowUtils::Redraw is used by the Tgfx test.  This makes me think this is reasonably likely that the regression:
 * was caused by bug 433086
 * is a testing artefact and not a real regression

Is it worth getting a bug on file to understand why the talos builds aren't lining up?

Comment 13

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

(In reply to comment #12)
> Is it worth getting a bug on file to understand why the talos builds aren't
> lining up?

I filed bug 438752.

Comment 14

[alice nodelman [:alice] [:anode]]

From my look at comment #10, you can see that Changeset: 1a969abc8556 produced a build with the time stamp "1212542296, Linux mozilla-central build" (see top of the log for that build).  The talos machine that first showed the regression pulled a build from :

18:52:34 URL:http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1212542296/firefox-3.1a1pre.en-US.linux-i686.tar.bz2 [8955064/8955064] -> "firefox-3.1a1pre.en-US.linux-i686.tar.bz2" [1]

As you can see, the time stamp generated for the build matches the directory that talos pulls the build from.  This makes me confident that talos is testing what it claims to be testing.

Comment 15

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

The build that I'm claiming is misaligned is the one on "Linux talos mozilla-central qm-plinux-trunk02" that claims to have used the build starting at 17:37:07.  That build is labeled as having "Changeset: b0bc6be7db45", which has vlad's checkins above, but that talos run still reports a pre-regression time (2.30).

Comment 16

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Tgfx was quite horked on linux before and was reporting completely bogus numbers.  Before, XSync wasn't being called at all due to a wrong #ifdef; now XSync is being correctly used.  (I wouldn't trust Tgfx at all quite yet, but it should be somewhat better; the test itself needs some more tweaking to give useful numbers overall.)

Comment 17

[Shawn Wilsher :sdwilsh]

I'm going through and marking old performance regression bugs as INCOMPLETE that are likely too old to be valid or get any traction on them.

Please re-open if you have more information or can demonstrate the regression still exists.

Comment 18

[Jeffrey Baker]

Was a 6-month-old bug so ancient as to be intractable?  Whether this bug is in gecko or in the Tgfx test itself, the fact remains that Tgfx shows Linux to be 40x slower than the other platforms.  I think this should stay open.

Comment 19

[Shawn Wilsher :sdwilsh]

(In reply to comment #18)
> Was a 6-month-old bug so ancient as to be intractable?  Whether this bug is in
> gecko or in the Tgfx test itself, the fact remains that Tgfx shows Linux to be
> 40x slower than the other platforms.  I think this should stay open.
I closed this bug primarily because of comment 16.  Also, six months is pretty old for a perf regression bug because a lot of code changes happen during that time.