1229632 - Autophone Talos - failed to get measurement s for tsvg, tpn

Status: RESOLVED INVALID

(Firefox for Android Graveyard :: General, defect)

Description

[Bob Clary [:bc] (inactive)]

Beginning with Tue Dec 1, 15:41:41

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=f6e234ee862b&exclusion_profile=false&filter-searchStr=autophone

Bug 1229299 - StateMirroring/StateWatching calling the mirror callback out of order 

tsvg and tpn are failing to get any measurements. I don't believe this is related to the recent deployment of bug 1214812 since the deployment occurred at bug 1214812 comment 49 at 2015-12-01 14:50:11

The patch does appear to be unrelated though.

Comment 1

[Jean-Yves Avenard [:jya]]

what does this test actually measure?

AFAIK, the StateMirror are only used at this stage by the media playback stack.

It could indicate a bug in the way the measurement were made that only got exposed because of the problem in State Mirroring logic. But the fix is sound and I doubt it could introduce a regression.

Comment 2

[Bob Clary [:bc] (inactive)]

The relevant test is implemented in Java though I'm not familiar with how it works:
https://dxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/src/org/mozilla/gecko/tests/testCheck2.java

The test stopped failing with https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=bb12e45c0dd8&filter-searchStr=autophone&exclusion_profile=false

glandium: Any idea if your patches would affect this?

Comment 3

[Jean-Yves Avenard [:jya]]

sounds more like a temporary failure as it stopped working for 4 commits, and I see another failure, this time in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=6df8e622a79d.

Those talos keep throwing errors and regression with totally irrelevant commits, this keeps happening.

I'm getting more and more convinced that they serve no purpose but make the developers waste time tracking shadows.

Comment 4

[Mike Hommey [:glandium]]

(In reply to Bob Clary [:bc:] from comment #2)
> glandium: Any idea if your patches would affect this?

Shouldn't.

Comment 5

[Bob Clary [:bc] (inactive)]

(In reply to Jean-Yves Avenard [:jya] from comment #3)
> sounds more like a temporary failure as it stopped working for 4 commits,
> and I see another failure, this time in
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> inbound&revision=6df8e622a79d.
> 

That is a null pointer crash for tpn in the browser and is probably an unrelated failure. The other failures for tpn previously were also either null pointer crashes or a flash crash.

This failure is about failing to get a measurement at all for both tsvg and tpn when no crash occurred. It is possible there was a crash in these cases but that no minidump was generated so we didn't detect it.

> Those talos keep throwing errors and regression with totally irrelevant
> commits, this keeps happening.
> 
> I'm getting more and more convinced that they serve no purpose but make the
> developers waste time tracking shadows.

I can't tell you how discouraging that statement is. I don't like wasting your time nor mine, so if you really feel this way please contact your manager and let's stop running these tests altogether.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Talos finds 7-10 real regression/month- there is no need to make such sweeping negative statements, I would be happy to explain the top reasons why we have false alerts.  

As for why we had 4 builds in a row with no measurements detected, I thought I would look into this.  The previous build, I retriggered the two tests and they were green.  For the given revision above, I did the same process and they are also green.  This indicates something environmental is most likely the cause.  Not sure if this is just a hiccup on the device, server, or other infrastructure related bits.

Comment 7

[Jean-Yves Avenard [:jya]]

(In reply to Joel Maher (:jmaher) from comment #6)
> Talos finds 7-10 real regression/month- there is no need to make such
> sweeping negative statements, I would be happy to explain the top reasons
> why we have false alerts.  

And for how many false positive?

I'm sorry that you think I'm making a sweeping negative statement or that they are perceived as being discouraging.

The variation in results between runs is often enormous, both showing either an increase or a decrease of performance.

The increase, I can personally live with of course, but talking about my particular case where nothing I do should ever have an impact on talos results, I have received repetidly emails on how following one of my change there was an increase of x%. Last time was something like +13% something (which was adding comments on some code, no functional change whatsoever)

For the "regression", when you have spent days, even weeks working on a particular problem only to receive the strongest worded message, when you *know* it can't ever be responsible or remotely related to what talos tests. (Like "*** Please let us know your plans by Monday, or the offending patch will be backed out! ***").

Only to then to spend days arguing back and forth that really the results don't make sense (and not once was a regression ever present in my case).
How discouraging do you think it is for the person on the wrong end of the talos stick to receive such message?

When a test has such a great variance with its result and report so many false positive: I naturally question the validity of the tests. And unfortunately, as time goes by, the sentiment that it doesn't make sense has only grown.

Now of course we all want to genuinely find a solution that provides valid and accurate results. And I hope that this will happen some day.

Comment 8

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment #6 seems to indicate that this is an autophone problem. Please renom if this is not the case.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

this might be a wfm if we cannot determine what happened to the environment during that blip of time.

Comment 10

[Bob Clary [:bc] (inactive)]

This only occurred from Tue Dec 1, 15:41:41 to Tue Dec 1, 17:53:25. jmaher proved it was an intermittent issue by retriggering the jobs though it is unknown what the root cause of the problem was. -> invalid.