1108636 - 10% WinXP|7 A11y regression on Inbound (v.36) around November 18 on PGO only

Status: RESOLVED WORKSFORME

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

it appears we missed some regressions (usually 1 or 2) and alerts were not sent out.  These regressions come from PGO only and are seen on win7 and winxp platforms.

It looks like there was a similar issue on linux 32+64, but it was backed out or fixed.

Here is the graphs:
http://graphs.mozilla.org/graph.html#tests=%5B%5B223,63,37%5D,%5B223,63,25%5D,%5B223,63,35%5D,%5B223,63,33%5D%5D&sel=1415028065922.2522,1418058255034&displayrange=90&datatype=running

I will need to spend some time backfilling data and tracking this down.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

jmaher, can you find the push which caused this!

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

* linux32 has a regression from bug 1100513 (https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=mozilla-inbound%20pgo%20talos%20other&fromchange=8c37c5083952&tochange=8e604ebfc313&rev=ea2d1aa200e4)
* linux*|win* has another regression a few pushes later from bug 1096280 (https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=mozilla-inbound%20pgo%20talos%20other&fromchange=8c37c5083952&tochange=8e604ebfc313&rev=a3f7920495b4)

So the linux/win regression is backed out, but the windows regressions remained (do we need to reset pgo somehow?)

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

here is the range and retriggers  Iwas doing: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=mozilla-inbound%20pgo%20talos%20other&fromchange=8c37c5083952&tochange=8e604ebfc313

Comment 4

[Jan de Mooij [:jandem]]

(In reply to Joel Maher (:jmaher) from comment #2)
> * linux32 has a regression from bug 1100513
> (https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=mozilla-
> inbound%20pgo%20talos%20other&fromchange=8c37c5083952&tochange=8e604ebfc313&r
> ev=ea2d1aa200e4)

Hm this is pretty weird. Was this also PGO only?

> * linux*|win* has another regression a few pushes later from bug 1096280
> (https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=mozilla-
> inbound%20pgo%20talos%20other&fromchange=8c37c5083952&tochange=8e604ebfc313&r
> ev=a3f7920495b4)
> 
> So the linux/win regression is backed out, but the windows regressions
> remained (do we need to reset pgo somehow?)

So the backout didn't change this at all? We don't have to reset anything and there should have been plenty of clobber builds since then. The first patch in that bug made some small changes and wasn't backed out, but it was mostly just moving some files into its own directory... Strange.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

yes, this is just pgo only.

The Win7 regression is sort of resolved Dec 17th, but winxp stays high.  Maybe it is related to another patch.  The numbers are what they are.  Unfortunately tbpl doesn't go back more than 30 days in history, so we don't have a lot of luck.  treeherder goes back much further, but retriggers won't work past 30 days (due to ftp space, etc.)

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I do not see this winxp regression on mozilla-beta, can we close this?

Comment 7

[Jan de Mooij [:jandem]]

(In reply to Joel Maher (:jmaher) from comment #6)
> I do not see this winxp regression on mozilla-beta, can we close this?

The offending patch was backed out so it was pretty mysterious in the first place. Closing is fine with me.